Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Data & analytics

omh-data-pipelines

[omh] Data pipeline work -- an ETL or streaming job, a backfill or replay, duplicate events, a schema change downstream, a lineage question, a data-quality regression: make every rerun idempotent, bound every replay, and gate each load on observed checks. Use when the user says: data-pipelines, data pipeline, data pipelines, etl, elt, etl pipeline, etl job, etl backfill.

rlaope · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add rlaope/oh-my-hermes --skill omh-data-pipelines

About this skill


name: "omh-data-pipelines" description: "[omh] Data pipeline work -- an ETL or streaming job, a backfill or replay, duplicate events, a schema change downstream, a lineage question, a data-quality regression: make every rerun idempotent, bound every replay, and gate each load on observed checks. Use when the user says: data-pipelines, data pipeline, data pipelines, etl, elt, etl pipeline, etl job, etl backfill." metadata: hermes: tags: [workflow, oh-my-hermes, planning] category: planning phase: data-pipelines role: planner quality_tier: idempotent-replay-gated

Data Pipelines

This is an OMH data-pipelines workflow skill, projected for Agent Skills hosts (Claude Code, Codex, Cursor, opencode, OpenClaw, pi).

Why This Exists

data-pipelines exists because pipeline work had no owner: backend owns a service's schema migration, data-analysis analyzes data it is handed, and relational-db owns a database's locks and indexes, while a backfill that duplicated events or a schema change with unknown readers reached memory and event lanes with no idempotency contract or replay bound at all.

First Steps

  • Ask what makes a row unique at the sink before planning any rerun.
  • Bound the window and the targets before ordering any replay or backfill step.

Do Not Use When

  • The ask is a service's own database migration, API, or queue design; use backend.
  • The ask is analyzing, charting, or summarizing a dataset that was handed over; use data-analysis.
  • The ask is a slow query, an index, or DDL locking a live table; use relational-db.
  • The ask is remembering or syncing what the assistant knows about the user; use memory-sync.

Examples

Good example:

  • Prompt: our airflow etl backfill is producing duplicate events
  • Expected behavior: Find the sink's unique key, name the append that duplicated rows, write the idempotency contract (event-id dedupe or partition overwrite), then bound the backfill window and gate it on key uniqueness and row count against the prior window.
  • Why: Rerunning an appending backfill doubles the duplicates it was meant to fix.

Bad example:

  • Prompt: just delete the duplicates and rerun the whole history
  • Expected behavior: Refuse the unbounded rerun: fix the write to be idempotent first, then backfill a bounded window behind a quality gate.
  • Why: Deleting duplicates without fixing the write guarantees the next rerun duplicates again.

Completion Checklist

  • The sink's unique key and the idempotency contract are stated.
  • Every replay or backfill is bounded by window and target.
  • Every downstream reader of a schema change is named with its impact.
  • Every load names its data-quality gate and the value that stops it.
  • OMH ran nothing, and every count cites observed output or is marked unverified.

Recovery Notes

  • If no unique key exists at the sink, the first step is defining one; say so before any rerun.
  • If lineage is unavailable, list readers found by search and mark the map incomplete.

Use When

Use when a batch or streaming data pipeline needs planning or repair: an ETL, ELT, Airflow, dbt, Spark or Kafka job; a backfill or a replay of past events; duplicate or missing rows; a schema change whose downstream readers are unknown; a lineage question; or a data-quality regression. The output is the lineage, the schema change's downstream impact, an idempotency contract, a bounded replay or backfill plan, and the data-quality gate each load must pass; OMH runs no job and reads no warehouse.

Strong routing signals: `data-pipelines`, `data pipeline`, `data pipelines`, `etl`, `elt`, `etl pipeline`, `etl job`, `etl backfill`, `airflow dag`, `airflow etl`, `airflow backfill`, `dagster`, `dbt model`, `dbt run`, `spark job`, `kafka topic`, `kafka events`, `kafka consumer`, `backfill`, `data backfill`, `replay events`, `replay the events`, `event replay`, `idempotent`, `idempotency`, `exactly once`, `exactly-once`, `duplicate events`, `lineage`, `data lineage`, `data quality`, `data quality check`, `schema evolution`, `late arriving data`, `dead letter queue`, `batch job`

Catalog Metadata

Category: planning Phase: data-pipelines Quality tier: idempotent-replay-gated Reasoning demand: standard

Quality bar:

  • Find what makes a row unique at the sink before proposing any rerun.
  • Load references/pipeline-method.md for the idempotency patterns, the schema compatibility table, the replay and backfill procedure, and the quality checks instead of recalling them.
  • Map lineage from the orchestrator's graph first and mark anything found only by search.
  • Treat duplicates as an idempotency defect, not a cleanup task: fix the write, then repair the rows.
  • Keep prepared, run, and verified as separate states for every load and check.

Required inputs:

  • the pipeline: its orchestrator, its sources, its sinks, and its schedule or trigger
  • the unit of the problem: the table, topic, or model, and the time window affected
  • what makes a row unique at the sink: the natural key, the event id, or the partition
  • the downstream readers already known: models, dashboards, exports, services
  • observed counts, job logs, or check results for any claim about what was loaded

Expected outputs:

  • lineage_map/v1
  • schema_change_impact/v1
  • idempotency_contract/v1
  • replay_backfill_plan/v1
  • data_quality_gate/v1

Artifact expectations:

  • lineage_map/v1 names each upstream source and each downstream reader of the affected table, topic, or model, from the orchestrator's graph or a lineage record, and marks readers found by search rather than by the graph
  • schema_change_impact/v1 classifies the change as additive, widening, or breaking for each downstream reader, and names the reader that breaks and the order that avoids it
  • idempotency_contract/v1 names the key that makes a rerun safe -- a natural key upsert, an event-id dedupe window, or a partition overwrite -- and what happens to a row written twice
  • re
Read the complete source ↗