技能说明
name: neo4j-spark-skill
description: Use when reading from or writing to Neo4j with Apache Spark or Databricks using the
Neo4j Connector for Apache Spark 6.0 (org.neo4j.connectors:spark) or 5.x
(org.neo4j:neo4j-connector-apache-spark). Covers SparkSession
setup, DataFrame reads via labels/Cypher/relationship scan, DataFrame writes with SaveMode,
node.keys for MERGE, relationship write mapping, partition and batch tuning, PySpark and Scala
examples, Databricks cluster config, Databricks secrets for credentials, Delta Lake to Neo4j
pipelines. Does NOT handle Cypher authoring — use neo4j-cypher-skill. Does NOT handle the Python
bolt driver — use neo4j-driver-python-skill. Does NOT handle GDS algorithms — use neo4j-gds-skill.
version: 1.0.3
allowed-tools: Bash WebFetch
Neo4j Connector for Apache Spark
When to Use
- Reading Neo4j nodes/relationships into Spark DataFrames
- Writing Spark DataFrames to Neo4j as nodes or relationships
- Databricks notebooks connecting to Neo4j
- Delta Lake → Neo4j ingestion pipelines
- Partitioned parallel reads from large Neo4j graphs
When NOT to Use
- Python bolt driver / execute_query →
neo4j-driver-python-skill
- Cypher query writing →
neo4j-cypher-skill
- GDS graph algorithms →
neo4j-gds-skill
- Spring Boot + Neo4j →
neo4j-spring-data-skill
Version Matrix
| Connector | Spark | Scala | Java | Databricks Runtime | Neo4j | Maven coordinate |
|---|
| 6.0.x | 4.0, 4.1 | 2.13 | 17+ | 17.3 LTS | 5.x, 2025.x, 2026.x | org.neo4j.connectors:spark:6.0.0-s_2.13 |
| 5.5.x / 5.4.x | 3.4, 3.5 | 2.12, 2.13 | 8+ | 14.3–16.4 LTS | 4.4, 5.x, 2025.x, 2026.x | org.neo4j:neo4j-connector-apache-spark_2.13:5.5.0_for_spark_3 |
Group ID changed in 6.0 — org.neo4j:neo4j-connector-apache-spark_<scala> is now a relocation POM pointing at org.neo4j.connectors:spark. On Spark 3.x stay on 5.5.x.
6.0 breaking changes
| Change | Migration |
|---|
| Spark baseline 3.5 → 4.0/4.1; Scala 2.12 and Java 8–11 dropped | Upgrade to 5.5.0 first, then Spark 4.x + Scala 2.13 + Java 17 |
Maven coordinate org.neo4j.connectors:spark:<version>-s_2.13 | Replace old _for_spark_3 coordinate |
schema.optimization.type removed | schema.optimization.node.keys, schema.optimization.relationship.keys, schema.optimization |
$stream.offset in partitioned reads removed | Use partitions + query.count |
;-separated multi-statement script removed | script.1, script.2, … script.N — executed in numbered order |
relationship.save.strategy default native → keys | Set .option("relationship.save.strategy", "native") explicitly to keep old behaviour |
query option rewritten for Data Source V2 predicate push-down | No action; verify plans on upgrade |
Setup
Standalone Spark (PySpark)
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.appName("neo4j-app")
.config("spark.jars.packages",
"org.neo4j.connectors:spark:6.0.0-s_2.13") # Spark 3.x: org.neo4j:neo4j-connector-apache-spark_2.13:5.5.0_for_spark_3
.config("neo4j.url", "neo4j+s://xxxx.databases.neo4j.io")
.config("neo4j.authentication.type", "basic")
.config("neo4j.authentication.basic.username", "neo4j")
.config("neo4j.authentication.basic.password", "password")
.getOrCreate())
Standalone Spark (Scala)
val spark = SparkSession.builder
.appName("neo4j-app")
.config("spark.jars.packages",
"org.neo4j.connectors:spark:6.0.0-s_2.13")
.config("neo4j.url", "neo4j+s://xxxx.databases.neo4j.io")
.config("neo4j.authentication.type", "basic")
.config("neo4j.authentication.basic.username", "neo4j")
.config("neo4j.authentication.basic.password", "password")
.getOrCreate()
Databricks — Cluster Installation
- Cluster → Libraries → Install New → Maven
- Coordinate
org.neo4j.connectors:spark:6.0.0-s_2.13 on DBR 17.3 LTS; org.neo4j:neo4j-connector-apache-spark_2.13:5.5.0_for_spark_3 on DBR 14.3–16.4 LTS
- Cluster → Advanced Options → Spark tab — add config:
neo4j.url neo4j+s://xxxx.databases.neo4j.io
neo4j.authentication.type basic
neo4j.authentication.basic.username {{secrets/neo4j/username}}
neo4j.authentication.basic.password {{secrets/neo4j/password}}
- Use Single user access mode (Unity Catalog shared mode not supported)
Databricks — Secrets (preferred over plaintext)
# Store credentials once:
# databricks secrets create-scope --scope neo4j
# databricks secrets put --scope neo4j --key url
# databricks secrets put --scope neo4j --key username
# databricks secrets put --scope neo4j --key password
neo4j_url = dbutils.secrets.get(scope="neo4j", key="url")
neo4j_user = dbutils.secrets.get(scope="neo4j", key="username")
neo4j_pass = dbutils.secrets.get(scope="neo4j", key="password")
spark.conf.set("neo4j.url", neo4j_url)
spark.conf.set("neo4j.authentication.type", "basic")
spark.conf.set("neo4j.authentication.basic.username", neo4j_user)
spark.conf.set("neo4j.authentication.basic.password", neo4j_pass)
Key Configuration Options
| Option | Description | Default |
|---|
neo4j.url | Bolt/Neo4j URI | — (required) |
neo4j.authentication.type | none, basic, kerberos, bearer | basic |
neo4j.authentication.basic.username | Username | driver default |
neo4j.authentication.basic.password | Password | driver default |
neo4j.authentication.bearer.token | Bearer token | — |
neo4j.database | Target database | driver default |
neo4j.access.mode | read or write | read |
neo4j.encryption.enabled | TLS (ignored with +s/+ssc URI) | false |
neo4j.db.transaction.timeout | Transaction timeout (ms) | driver default |
neo4j.db.transaction.metadata.<key> | Custom transaction metadata surfaced in que | |
阅读完整来源 ↗