agentsop-llm-artifact-versioning
Enhancement overlay — version the WHOLE deployable LLM-app artifact as one bundle: prompts + compiled programs + model snapshot pins + retrieval config + eval-set version, versioned together so a deploy is reproducible and rollback is atomic. Activate when preparing to deploy an LLM app, when asking "what exactly is running in prod right now?", when a deploy must be reproducible months later, or when an incident needs a clean rollback. The core reframe: an LLM app artifact is NOT an ML model — it is a manifest over many independently-mutable parts, not one weights file. Do NOT activate for one-off prompt edits with no deploy, for a single-component demo, or where a vendor owns the whole prompt lifecycle. For versioning ONE compiled prompt use [[agentsop-per-model-artifacts]]; for the CI comparison mechanism use [[agentsop-regression-gate]]. Search keywords: prompt versioning, reproducible deploy, what is running in prod, rollback LLM app, model pinning, prompt registry, version prompts
agentsope · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add agentsope/SkillAlchemy --skill agentsop-llm-artifact-versioning技能说明
name: agentsop-llm-artifact-versioning version: 0.1.0 description: >- Enhancement overlay — version the WHOLE deployable LLM-app artifact as one bundle: prompts
- compiled programs + model snapshot pins + retrieval config + eval-set version, versioned together so a deploy is reproducible and rollback is atomic. Activate when preparing to deploy an LLM app, when asking "what exactly is running in prod right now?", when a deploy must be reproducible months later, or when an incident needs a clean rollback. The core reframe: an LLM app artifact is NOT an ML model — it is a manifest over many independently-mutable parts, not one weights file. Do NOT activate for one-off prompt edits with no deploy, for a single-component demo, or where a vendor owns the whole prompt lifecycle. For versioning ONE compiled prompt use [[agentsop-per-model-artifacts]]; for the CI comparison mechanism use [[agentsop-regression-gate]]. Search keywords: prompt versioning, reproducible deploy, what is running in prod, rollback LLM app, model pinning, prompt registry, version prompts and config.
Versioned, Reproducible LLM Artifact — Overlay SOP
"Prompts are effectively the weights of an LLM application." — DSPy core philosophy [arxiv.org/abs/2310.03714] (R1 §1)
"Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile." — dspy-sop SKILL, Dilemma Case B (R1 §2)
This is an enhancement overlay, not a framework SOP. It sits on top of whatever stack you use (DSPy, LangChain, raw API) and adds one discipline: define, pin, and version the entire deployable bundle as a unit. It is the broad sibling of [[agentsop-per-model-artifacts]] — that skill versions one compiled prompt; this one versions everything that ships together.
1. 何时激活 (When to activate)
Activate when any of these appears in the user's intent, codebase, or workflow:
| Trigger | Signal |
|---|---|
| Preparing to deploy | "ship this to prod", a Dockerfile/deploy.yaml/serving entrypoint wrapping an LLM app, a release checklist |
| "What is running in prod?" | Nobody can name the exact prompt text + model snapshot + retriever config currently serving traffic |
| Reproducibility need | "reproduce the deploy from last quarter", an audit, a regulator asking what produced an output |
| Rollback need | Incident: prod behavior changed and the team needs the last known-good combination of components back |
| Drift symptoms | Score moved, no code change merged; or "we updated the prompt but forgot which model it was tuned for" |
| Multi-component apps | RAG + reranker + synthesizer + judge, each naming its own model/config, none bundled |
| Cross-skill bridges | DSPy save_program produced a compiled program → it is one component of the bundle; pin the rest. Per-prompt lifecycle handled by [[agentsop-per-model-artifacts]] → wrap as a bundle component here. |
Do NOT activate when:
- One-off prompt edit with no deploy and no reproduction requirement — just edit and run.
- A single-component proof-of-concept where there is exactly one prompt, one model, no
retrieval, and it will never be reproduced. Flat
v1.jsonis fine. - A vendor owns the whole lifecycle (OpenAI Prompt Library, Anthropic Workbench managed prompts, fully vendor-managed RAG) — let them version it.
- The team rewrites the app daily during exploration — bundle versioning has no shelf life until the app shape stabilizes (same boundary as dspy-sop "signature still changing").
2. 核心心智模型 (Core mental model)
An LLM app artifact ≠ an ML model
The single most common mistake is reasoning about an LLM app the way you reason about a trained model. They are not the same shape.
| ML model | LLM app artifact |
|---|---|
One weights file (.pt, .safetensors) | A manifest over many parts |
| Identity = file hash | Identity = hash of the whole bundle |
| Mutates only on retrain | Each part mutates independently and silently |
| Versioned by a model registry | Versioned by a bundle manifest + tag |
The deployable artifact is:
┌──────────────── DEPLOYABLE BUNDLE (one tag) ────────────────┐
│ │
prompts compiled model pins retrieval eval-set
(text + programs (snapshot id config version
hashes) (program.json) per call site) (index ptr, (sha256)
│ embed model, │
│ top_k, reranker) │
│ │
└── version them TOGETHER, or you can't reproduce a deploy ──┘
The load-bearing claim: the deployable artifact = prompts + compiled programs + model pins
- retrieval config + eval-set version. Version them together or you cannot reproduce a deploy — and you cannot roll back without producing a never-tested combination.
Why "together" is non-negotiable
Each part can change without touching the others (R1 §3, §4, §6):
- A model alias rolls forward server-side (
gpt-4o→ new snapshot) — prompt unchanged, behavior changed. - A model snapshot gets a silent server-side patch (R1 §4) — pin unchanged, behavior changed.
- A retriever knob moves in a dashboard (top_k 5→10) — no commit, behavior changed.
- The eval set gains 50 examples — the "same" score now means something different (R1 §5).
If these are versioned separately, "the deploy" is not a thing you can name. If they are versioned as one bundle with one id, the deploy is reproducible and rollback is atomic (R1 §8).
The registry analogy and its limit (R1 §7)
MLflow's Model Registry gives the right primitives: versioning, stage transitions (Staging/Production/Archived), reproduce-from-config, compare-versions [`~/.claude/skills/mlflow/SK
阅读完整来源 ↗