跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

agentsop-idempotent-ingestion

Re-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — **ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op** — plus the docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` + `DocstoreStrategy`), the delete-propagation problem, and cross-framework equivalents (LangChain `index()` + `RecordManager`, manual hash table). ENHANCE overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but the re-ingest-correctness contract is not surfaced.

agentsope · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion

技能说明


name: agentsop-idempotent-ingestion version: 0.1.0 description: | Re-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op — plus the docstore + doc-hash upsert machinery (LlamaIndex IngestionPipeline + DocstoreStrategy), the delete-propagation problem, and cross-framework equivalents (LangChain index() + RecordManager, manual hash table). ENHANCE overlay over [[llamaindex]]: the IngestionPipeline exists in the base skill but the re-ingest-correctness contract is not surfaced. trigger_keywords:

  • "idempotent ingestion"
  • "re-ingest"
  • "re-index"
  • "docstore"
  • "doc hash"
  • "upsert"
  • "duplicate chunks"
  • "IngestionPipeline"
  • "RecordManager"
  • "incremental update"
  • "scheduled reindex" when_to_use:
  • "any ingestion pipeline that will run more than once (cron, CI, webhook, manual re-run)"
  • "a corpus where source documents are added, edited, or deleted over time"
  • "debugging duplicate / stale chunks polluting retrieval after a re-run"
  • "reviewing a PR that calls VectorStoreIndex.from_documents / pipeline.run without a docstore"
  • "designing the ingestion side of a production RAG before first deploy" when_not_to_use:
  • "a one-shot index built once and never refreshed (static corpus, throwaway notebook)"
  • "the corpus is small enough to fully rebuild from scratch in seconds and rebuild-on-every-run is genuinely acceptable (measure first)"
  • "pure retrieval / query-time work with no ingestion path"

Idempotent Ingestion · Re-Ingest-Correctness SOP

Third-person operating model for a coder agent that owns ingestion correctness across repeated runs. The audience is the LLM agent writing or reviewing the pipeline code — not the end user.

One sentence: The interesting run is the second one. A correct pipeline hashes each document, looks the hash up in a docstore, and inserts / updates / skips accordingly — so re-running over unchanged docs changes nothing.

This skill is an ENHANCE overlay over [[llamaindex]]. The base skill names IngestionPipeline with docstore + UPSERTS_AND_DELETE as a hardening must-have (OP-08, anti-pattern A10) but stops at "use it". This overlay surfaces the correctness contract — what idempotency actually means, how the hash decides the action, and how it generalises beyond LlamaIndex.


1. 何时激活 (Activation Rules)

Activate this skill whenever any of the following holds:

  1. The codebase contains an ingestion call (VectorStoreIndex.from_documents, pipeline.run(documents=...), vectorstore.add_documents, a custom embed
    • upsert loop) and that call will execute more than once over a corpus that can change between runs.
  2. The user mentions any of: re-ingest, re-index, scheduled / nightly index refresh, incremental document updates, docstore, doc hash, upsert, RecordManager, "keep the index in sync with the source".
  3. A bug report says "I see the same answer chunk twice", "retrieval returns duplicates", "deleted a file but it still shows up in answers", "the index keeps growing every night even though nothing changed".
  4. PR review: new ingestion code that builds the index with no docstore / no hash-keyed dedup and the corpus is live (LlamaIndex base skill A10).
  5. A production RAG is about to ship and the ingestion side has only ever been tested as a cold first run.

Do not activate when:

  • The index is built once from a static corpus and never refreshed — there is no second run to make idempotent.
  • The corpus is tiny and rebuilding from scratch each run is measurably cheaper than maintaining a docstore (rare; verify, don't assume).
  • The task is purely query-time (retrieval, rerank, synthesis) with no ingestion path in scope.

2. 核心心智模型 (Core Mental Model)

Three principles. Violating any one means a second run can duplicate or corrupt the index — regardless of how good the retrieval stack downstream is.

Principle 1 — Ingestion must be idempotent; the hash is the decision

Re-running the pipeline over an unchanged document is a no-op. Run it once, run it a thousand times — same index. The mechanism: each document is reduced to a stable content hash (and a stable doc id); that hash is looked up in a docstore (a key→hash record store separate from the vector store); the lookup result decides the action.

            doc_id seen?           hash changed?
                │                       │
   ┌────────────┴───────────┐    ┌──────┴──────┐
   no                       yes  no            yes
   │                         │   │              │
INSERT (embed + upsert)      │  SKIP        UPDATE
                             └──────────────►(delete old chunks,
                                              re-embed, upsert)

The hash is the whole game. Without it, the pipeline cannot tell "I have seen this exact content" from "this is new" — so it re-embeds and re-adds everything, and the vector store accumulates duplicates. (LlamaIndex base skill failure #3 / A10; developers.llamaindex.ai/.../loading/ingestion_pipeline/.)

Principle 2 — The docstore is a separate ledger, not the vector store

The vector store holds embeddings keyed by node id. The docstore holds, per document, {doc_id → content_hash} (and the node ids derived from it). They are two stores with two jobs:

StoreHoldsAnswers
Docstoredoc_id → hash, doc→node mapping"have I seen this content before?"
Vector storenode_id → embedding + metadata"what is semantically ne
阅读完整来源 ↗