Idempotent Ingestion: The SOP Your RAG Pipeline Is Missing

Where this fits

You've scheduled a nightly re-index of your document corpus. By week two, retrieval returns five near-identical chunks for the same source, and deleted PDFs still show up in results. The root cause: your ingestion pipeline isn't idempotent. This skill gives agents a concrete SOP for making re-runs a no-op — content hashes decide insert/update/skip, so unchanged docs never re-embed, and deletions propagate to the index.

Why agents benefit

• Detects idempotency violations in existing pipelines by looking for missing docstore or hash-based upsert logic.
• Implements the fix using LlamaIndex's IngestionPipeline + DocstoreStrategy or LangChain's index() + RecordManager, choosing the right pattern for the framework in use.
• Handles delete propagation — the skill explains why a docstore alone isn't enough and how to sync removals to the vector index.
• Debugs duplicate-chunk reports by tracing whether the pipeline re-embeds unchanged content or fails to deduplicate at the source.
• Provides cross-framework equivalents — including a manual hash table approach for custom pipelines — so agents aren't locked to one library.

Practical scenarios

Scheduled re-index with a changing corpus

Your agent reviews a daily job that re-ingests a folder of markdown files. It spots missing doc_id checks and rewrites the pipeline to use content hashes, so only new or modified files are embedded.

CI re-ingest after code changes

A team's CI pipeline re-runs ingestion on every commit. The agent patches it to skip unchanged docs, cutting embedding costs and preventing index bloat.

"Retrieval shows deleted docs" bug

Users delete files from the source, but chunks remain in the vector store. The agent applies the skill's delete-propagation guidance, ensuring the docstore's removal also triggers index cleanup.

Add it to your agent workflow

Install the skill via npx:

npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion

Then instruct your agent: "Check the ingestion pipeline for idempotency. If it lacks doc-hash upserts, apply the agentsop-idempotent-ingestion SOP." The agent will inspect the code, identify the gap, and propose a concrete fix using the skill's patterns.

Compare before adopting

Before integrating this skill into your agent's toolset, compare:

• Quality signals: Does the skill include real code examples and edge-case handling (e.g., hash collisions, partial failures)?
• Maintenance freshness: Is the repo active? Check the GitHub commit history and issue responses.
• Alternatives: LlamaIndex and LangChain docs cover idempotency, but this skill condenses it into an actionable SOP with cross-framework notes.
• Workflow fit: Does your agent already use LlamaIndex? The skill is an ENHANCE overlay over the base [[llamaindex]] skill, so it slots in naturally.

Why it is worth tracking

With 364 stars and a clear, focused scope, this skill addresses a recurring production pain point. Its trigger keywords cover the exact phrases you'd use when debugging duplicate or stale retrieval. Track it if you're building RAG pipelines that must survive repeated ingestion runs — and evaluate it early, because retrofitting idempotency later is costly. For a broader view of agent skills, see OpenAgentSkill's listing.