跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

agentsop-llm-artifact-versioning

Enhancement overlay — version the WHOLE deployable LLM-app artifact as one bundle: prompts + compiled programs + model snapshot pins + retrieval config + eval-set version, versioned together so a deploy is reproducible and rollback is atomic. Activate when preparing to deploy an LLM app, when asking "what exactly is running in prod right now?", when a deploy must be reproducible months later, or when an incident needs a clean rollback. The core reframe: an LLM app artifact is NOT an ML model — it is a manifest over many independently-mutable parts, not one weights file. Do NOT activate for one-off prompt edits with no deploy, for a single-component demo, or where a vendor owns the whole prompt lifecycle. For versioning ONE compiled prompt use [[agentsop-per-model-artifacts]]; for the CI comparison mechanism use [[agentsop-regression-gate]]. Search keywords: prompt versioning, reproducible deploy, what is running in prod, rollback LLM app, model pinning, prompt registry, version prompts

agentsope · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentsope/SkillAlchemy --skill agentsop-llm-artifact-versioning

技能说明


name: agentsop-llm-artifact-versioning version: 0.1.0 description: >- Enhancement overlay — version the WHOLE deployable LLM-app artifact as one bundle: prompts

  • compiled programs + model snapshot pins + retrieval config + eval-set version, versioned together so a deploy is reproducible and rollback is atomic. Activate when preparing to deploy an LLM app, when asking "what exactly is running in prod right now?", when a deploy must be reproducible months later, or when an incident needs a clean rollback. The core reframe: an LLM app artifact is NOT an ML model — it is a manifest over many independently-mutable parts, not one weights file. Do NOT activate for one-off prompt edits with no deploy, for a single-component demo, or where a vendor owns the whole prompt lifecycle. For versioning ONE compiled prompt use [[agentsop-per-model-artifacts]]; for the CI comparison mechanism use [[agentsop-regression-gate]]. Search keywords: prompt versioning, reproducible deploy, what is running in prod, rollback LLM app, model pinning, prompt registry, version prompts and config.

Versioned, Reproducible LLM Artifact — Overlay SOP

"Prompts are effectively the weights of an LLM application." — DSPy core philosophy [arxiv.org/abs/2310.03714] (R1 §1)

"Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile." — dspy-sop SKILL, Dilemma Case B (R1 §2)

This is an enhancement overlay, not a framework SOP. It sits on top of whatever stack you use (DSPy, LangChain, raw API) and adds one discipline: define, pin, and version the entire deployable bundle as a unit. It is the broad sibling of [[agentsop-per-model-artifacts]] — that skill versions one compiled prompt; this one versions everything that ships together.


1. 何时激活 (When to activate)

Activate when any of these appears in the user's intent, codebase, or workflow:

TriggerSignal
Preparing to deploy"ship this to prod", a Dockerfile/deploy.yaml/serving entrypoint wrapping an LLM app, a release checklist
"What is running in prod?"Nobody can name the exact prompt text + model snapshot + retriever config currently serving traffic
Reproducibility need"reproduce the deploy from last quarter", an audit, a regulator asking what produced an output
Rollback needIncident: prod behavior changed and the team needs the last known-good combination of components back
Drift symptomsScore moved, no code change merged; or "we updated the prompt but forgot which model it was tuned for"
Multi-component appsRAG + reranker + synthesizer + judge, each naming its own model/config, none bundled
Cross-skill bridgesDSPy save_program produced a compiled program → it is one component of the bundle; pin the rest. Per-prompt lifecycle handled by [[agentsop-per-model-artifacts]] → wrap as a bundle component here.

Do NOT activate when:

  • One-off prompt edit with no deploy and no reproduction requirement — just edit and run.
  • A single-component proof-of-concept where there is exactly one prompt, one model, no retrieval, and it will never be reproduced. Flat v1.json is fine.
  • A vendor owns the whole lifecycle (OpenAI Prompt Library, Anthropic Workbench managed prompts, fully vendor-managed RAG) — let them version it.
  • The team rewrites the app daily during exploration — bundle versioning has no shelf life until the app shape stabilizes (same boundary as dspy-sop "signature still changing").

2. 核心心智模型 (Core mental model)

An LLM app artifact ≠ an ML model

The single most common mistake is reasoning about an LLM app the way you reason about a trained model. They are not the same shape.

ML modelLLM app artifact
One weights file (.pt, .safetensors)A manifest over many parts
Identity = file hashIdentity = hash of the whole bundle
Mutates only on retrainEach part mutates independently and silently
Versioned by a model registryVersioned by a bundle manifest + tag

The deployable artifact is:

            ┌──────────────── DEPLOYABLE BUNDLE (one tag) ────────────────┐
            │                                                             │
  prompts       compiled        model pins        retrieval       eval-set
  (text +       programs        (snapshot id      config          version
   hashes)      (program.json)   per call site)   (index ptr,     (sha256)
            │                    embed model,                             │
            │                    top_k, reranker)                        │
            │                                                             │
            └── version them TOGETHER, or you can't reproduce a deploy ──┘

The load-bearing claim: the deployable artifact = prompts + compiled programs + model pins

  • retrieval config + eval-set version. Version them together or you cannot reproduce a deploy — and you cannot roll back without producing a never-tested combination.

Why "together" is non-negotiable

Each part can change without touching the others (R1 §3, §4, §6):

  • A model alias rolls forward server-side (gpt-4o → new snapshot) — prompt unchanged, behavior changed.
  • A model snapshot gets a silent server-side patch (R1 §4) — pin unchanged, behavior changed.
  • A retriever knob moves in a dashboard (top_k 5→10) — no commit, behavior changed.
  • The eval set gains 50 examples — the "same" score now means something different (R1 §5).

If these are versioned separately, "the deploy" is not a thing you can name. If they are versioned as one bundle with one id, the deploy is reproducible and rollback is atomic (R1 §8).

The registry analogy and its limit (R1 §7)

MLflow's Model Registry gives the right primitives: versioning, stage transitions (Staging/Production/Archived), reproduce-from-config, compare-versions [`~/.claude/skills/mlflow/SK

阅读完整来源 ↗