Run PySpark on Fabric Without Notebooks

Where this fits

You're building an AI agent that helps data engineers. A user says: "Run this quick ETL on my Fabric lakehouse, but don't create a notebook—I just want the result." Normally that means spinning up a Livy session manually, wrestling with auth tokens, and cleaning up artifacts. The executing-spark skill removes that friction: it lets your agent execute arbitrary PySpark or Python code on Fabric Spark compute via ephemeral Livy sessions, with full read/write access to Delta tables—no notebook artifact persisted.

Why agents benefit

• Zero artifact overhead: Sessions are ephemeral; no notebook files to create, list, or delete. Your agent can focus on the task, not housekeeping.
• Direct Delta access: Spark SQL works seamlessly against lakehouse tables, so agents can read, transform, and write data without extra configuration.
• Automated authentication: The skill pulls a token from az account get-access-token (not fab auth), which is the only token that works for OneLake storage inside the session. No manual token juggling.
• Natural language triggers: It activates on phrases like "run PySpark in Fabric" or "execute Python on Fabric compute"—your agent knows when to use it without explicit instructions.
• No dependency on notebook infrastructure: Your agent can run ad-hoc analysis, test snippets, or ETL jobs directly against Spark compute, making it ideal for iterative workflows.

Practical scenarios

Quick data validation

A user suspects a Delta table has nulls in a critical column. Instead of opening a notebook, your agent fires off a short PySpark script to count nulls and returns the result in seconds.

Ad-hoc ETL for a one-off report

A stakeholder needs a CSV export from a lakehouse table. Your agent runs a Spark job that reads the table, applies filters, and writes the output to a file—all without leaving a trace.

Testing a new transformation logic

Before committing to a full pipeline, a developer wants to test a complex window function. Your agent submits the code, captures the output, and shares it—ephemeral, fast, and safe.

Add it to your agent workflow

Install the skill with:

npx skills add data-goblin/power-bi-agentic-development --skill executing-spark

Then, in your agent's configuration, ensure it has the prerequisite: an authenticated Azure CLI session and a lakehouse in the target workspace. When a user asks to run Spark code, the agent will invoke the skill automatically. For example:

# Your agent can now do this:
code = "df = spark.sql('SELECT * FROM my_lakehouse.my_table LIMIT 10')"
result = agent.run_skill('executing-spark', code=code)
print(result)

Compare before adopting

Before integrating, evaluate the skill against your existing approach. Check how actively it's maintained on GitHub (stars, recent commits) and whether it fits your agent's toolchain. Alternatives include using the Fabric API directly or managing Livy sessions yourself—more code, more failure points. Also consider whether your agents already have a notebook-based execution path that would conflict with ephemeral sessions.

Why it is worth tracking

With 893 stars and a focused scope, executing-spark is gaining traction among developers building Fabric-aware agents. Its core value—ephemeral, artifact-free execution—solves a real pain point. As Fabric adoption grows, this skill's utility will likely increase. Review it when you need to extend your agent's data operations beyond read-only queries or when notebook management becomes a bottleneck. For a live overview of the skill and its updates, see the OpenAgentSkill listing.