AI Agent Skills

Apache Spark - A unified analytics engine for large-scale data processing

Official Dockerfile for Apache Spark

Security library for Sparkjava: OpenID Connect, SAML2, CAS, OAuth, LDAP, JWT...

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.

End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics work…

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent…

A simplified, lightweight ETL Framework based on Apache Spark

📈 A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to…

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

A playground for experimenting ideas that may apply to Spark SQL/Catalyst