agentsop-idempotent-ingestion
Re-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — **ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op** — plus the docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` + `DocstoreStrategy`), the delete-propagation problem, and cross-framework equivalents (LangChain `index()` + `RecordManager`, manual hash table). ENHANCE overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but the re-ingest-correctness contract is not surfaced.
agentsope · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion技能说明
name: agentsop-idempotent-ingestion
version: 0.1.0
description: |
Re-ingest-correctness SOP for production RAG. Activate when a calling agent
builds, reviews, or debugs an ingestion pipeline that runs more than once over
a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or
a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule —
ingestion must be idempotent: a document's content hash decides
insert/update/skip, so re-running over unchanged docs is a no-op — plus the
docstore + doc-hash upsert machinery (LlamaIndex IngestionPipeline +
DocstoreStrategy), the delete-propagation problem, and cross-framework
equivalents (LangChain index() + RecordManager, manual hash table). ENHANCE
overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but
the re-ingest-correctness contract is not surfaced.
trigger_keywords:
- "idempotent ingestion"
- "re-ingest"
- "re-index"
- "docstore"
- "doc hash"
- "upsert"
- "duplicate chunks"
- "IngestionPipeline"
- "RecordManager"
- "incremental update"
- "scheduled reindex" when_to_use:
- "any ingestion pipeline that will run more than once (cron, CI, webhook, manual re-run)"
- "a corpus where source documents are added, edited, or deleted over time"
- "debugging duplicate / stale chunks polluting retrieval after a re-run"
- "reviewing a PR that calls VectorStoreIndex.from_documents / pipeline.run without a docstore"
- "designing the ingestion side of a production RAG before first deploy" when_not_to_use:
- "a one-shot index built once and never refreshed (static corpus, throwaway notebook)"
- "the corpus is small enough to fully rebuild from scratch in seconds and rebuild-on-every-run is genuinely acceptable (measure first)"
- "pure retrieval / query-time work with no ingestion path"
Idempotent Ingestion · Re-Ingest-Correctness SOP
Third-person operating model for a coder agent that owns ingestion correctness across repeated runs. The audience is the LLM agent writing or reviewing the pipeline code — not the end user.
One sentence: The interesting run is the second one. A correct pipeline hashes each document, looks the hash up in a docstore, and inserts / updates / skips accordingly — so re-running over unchanged docs changes nothing.
This skill is an ENHANCE overlay over [[llamaindex]]. The base skill names
IngestionPipeline with docstore + UPSERTS_AND_DELETE as a hardening
must-have (OP-08, anti-pattern A10) but stops at "use it". This overlay
surfaces the correctness contract — what idempotency actually means, how the
hash decides the action, and how it generalises beyond LlamaIndex.
1. 何时激活 (Activation Rules)
Activate this skill whenever any of the following holds:
- The codebase contains an ingestion call (
VectorStoreIndex.from_documents,pipeline.run(documents=...),vectorstore.add_documents, a custom embed- upsert loop) and that call will execute more than once over a corpus that can change between runs.
- The user mentions any of: re-ingest, re-index, scheduled / nightly index
refresh, incremental document updates, docstore, doc hash, upsert,
RecordManager, "keep the index in sync with the source". - A bug report says "I see the same answer chunk twice", "retrieval returns duplicates", "deleted a file but it still shows up in answers", "the index keeps growing every night even though nothing changed".
- PR review: new ingestion code that builds the index with no docstore / no hash-keyed dedup and the corpus is live (LlamaIndex base skill A10).
- A production RAG is about to ship and the ingestion side has only ever been tested as a cold first run.
Do not activate when:
- The index is built once from a static corpus and never refreshed — there is no second run to make idempotent.
- The corpus is tiny and rebuilding from scratch each run is measurably cheaper than maintaining a docstore (rare; verify, don't assume).
- The task is purely query-time (retrieval, rerank, synthesis) with no ingestion path in scope.
2. 核心心智模型 (Core Mental Model)
Three principles. Violating any one means a second run can duplicate or corrupt the index — regardless of how good the retrieval stack downstream is.
Principle 1 — Ingestion must be idempotent; the hash is the decision
Re-running the pipeline over an unchanged document is a no-op. Run it once, run it a thousand times — same index. The mechanism: each document is reduced to a stable content hash (and a stable doc id); that hash is looked up in a docstore (a key→hash record store separate from the vector store); the lookup result decides the action.
doc_id seen? hash changed?
│ │
┌────────────┴───────────┐ ┌──────┴──────┐
no yes no yes
│ │ │ │
INSERT (embed + upsert) │ SKIP UPDATE
└──────────────►(delete old chunks,
re-embed, upsert)
The hash is the whole game. Without it, the pipeline cannot tell "I have seen
this exact content" from "this is new" — so it re-embeds and re-adds
everything, and the vector store accumulates duplicates. (LlamaIndex base skill
failure #3 / A10; developers.llamaindex.ai/.../loading/ingestion_pipeline/.)
Principle 2 — The docstore is a separate ledger, not the vector store
The vector store holds embeddings keyed by node id. The docstore holds, per
document, {doc_id → content_hash} (and the node ids derived from it). They are
two stores with two jobs:
| Store | Holds | Answers |
|---|---|---|
| Docstore | doc_id → hash, doc→node mapping | "have I seen this content before?" |
| Vector store | node_id → embedding + metadata | "what is semantically ne |