跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
法律与合规

verbalized-sampling

Generate diverse outputs by prompting for a probability distribution instead of a single response. Implements Verbalized Sampling (VS) from Zhang et al. 2025 — a training-free technique that counteracts LLM mode collapse caused by typicality bias in alignment data. Use when the task needs genuine diversity: creative writing, brainstorming/ideation, synthetic data generation, persona/dialogue simulation, adversarial examples, open-ended QA with multiple valid answers, or any situation where "generate 5 ideas" keeps returning the same cluster. Do NOT use for: single correct answer tasks, factual lookup, strict format compliance.

gnurio · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add gnurio/nurijanian-skills --skill verbalized-sampling

技能说明


name: verbalized-sampling description: > Generate diverse outputs by prompting for a probability distribution instead of a single response. Implements Verbalized Sampling (VS) from Zhang et al. 2025 — a training-free technique that counteracts LLM mode collapse caused by typicality bias in alignment data.

Use when the task needs genuine diversity: creative writing, brainstorming/ideation, synthetic data generation, persona/dialogue simulation, adversarial examples, open-ended QA with multiple valid answers, or any situation where "generate 5 ideas" keeps returning the same cluster.

Do NOT use for: single correct answer tasks, factual lookup, strict format compliance.

Verbalized Sampling

Universal VS Template

[Task description with rich context]

Generate {k} responses. Return in JSON format with key "{output_key}" (list of dicts). Each dict:
• text: [output specification]
• probability: estimated probability (0.0–1.0) of this response given the input

{Distribution constraint}

Output ONLY the JSON object.

Distribution constraints — pick one:

  • Sample from the full distribution. — balanced, moderate diversity
  • Sample from the tails of the distribution, with each probability below 0.10. — high diversity
  • Sample from the tails of the distribution, with each probability below 0.01. — maximum diversity

Variant Selection

VariantWhen to useTrade-off
VS-StandardStraightforward tasks, speed priorityBest balance
VS-CoTComplex tasks needing quality + diversitySlight diversity cost, higher quality
VS-MultiMaximum diversity, token cost acceptableBest diversity, 2× token cost

VS-CoT: add "reasoning": "step-by-step thought process" as the first field in each dict.

VS-Multi: Turn 1 generates k/2 responses. Turn 2: "Generate k alternative responses to the original prompt — do not repeat ideas from Turn 1."

Context-First Phase (run before VS)

VS outputs are only as good as the problem framing going in. Before constructing the VS prompt:

Step 1 — Decompose into subproblems: Break the task into 3–5 distinct subproblems or angles. Example: "improve sales for a B2B SaaS" → (1) acquisition channels, (2) conversion from trial, (3) pricing/packaging, (4) referral/word-of-mouth, (5) partnerships.

Step 2 — Load context for each subproblem:

  • What are the real constraints? (time, budget, team size, org politics, market saturation)
  • What do others in this space actually do? (base rates — what approaches are common, what have failed)
  • What has already been tried? (avoid re-suggesting)

Step 3 — Inject context into the VS prompt: Compress answers from Step 2 into the prompt preamble. Name the subproblems as explicit coverage requirements: "Cover at least one idea addressing each of: [subproblem 1], [subproblem 2], ..."

If you cannot answer Step 2 without asking the user, ask first before generating. Generic outputs caused by thin context are the primary failure mode for brainstorming tasks (FM-2).

Critique-and-Improve Loop (run after VS, before presenting)

After generating VS output, run a self-critique pass before presenting results. See references/critique-framework.md for the full 6-dimension framework and prompt templates.

Quick pass: For each output item, check:

  1. Is this naive/obvious? (would it appear in a top-10 listicle?)
  2. Is this actionable? (could execution start Monday without further research?)
  3. Does this require magical thinking? (assumes steps will "just work" with no mechanism)
  4. Does this ignore base rates? (approaches with known low success rates presented as good bets)

If 2+ items fail 2+ checks:

  • Flag the specific failures with reasons
  • Generate 2–3 improved variants that directly address the flagged weaknesses
  • Present: original outputs + critique summary + improved variants

For automated quality scoring of VS outputs, see references/judges.md for LLM-as-Judge prompts.

Output Mode Selection

JSON mode (default for agent pipelines — pipe-able, machine-readable):

  • Use when outputs feed into downstream processing, storage, or evaluation
  • Return raw JSON as-is

Readable mode (in-chat or external sharing):

  • Format using scripts/format_vs_output.py (see below), or render inline as numbered markdown
  • Group by diversity tier: High (p < 0.05), Moderate (p 0.05–0.15), Low/Common (p > 0.15)
  • Show probabilities inline as (p=0.07)

To format manually in-chat:

## High diversity (p < 0.05)
1. [text] (p=0.03)

## Moderate diversity (p 0.05–0.15)
2. [text] (p=0.08)

CLI formatting: echo '<json>' | python ~/.cursor/skills/verbalized-sampling/scripts/format_vs_output.py

Probability Threshold Quick Reference

ThresholdUse case
Full distributionGeneral brainstorm, want common + uncommon mix
p < 0.15Moderate novelty — avoids top-5 obvious answers
p < 0.10High diversity — noticeably non-obvious outputs
p < 0.05Aggressive — expect surprising, niche ideas
p < 0.01Maximum — edge cases, stress testing, adversarial

Failure Modes (from empirical evals)

FM-1: Overfit Topic Collapse High-frequency training topics (weight loss, productivity, exercise) resist VS even at p<0.01. The tail of the model's distribution is still inside the well-known solution cluster. The paper's 1.6-2.1× diversity gains apply to creative and niche domains — not saturated self-help topics.

Mitigation: Add explicit exclusion constraints: "Exclude any idea covered in mainstream [domain] journalism. Prioritize ideas from adjacent fields or underrepresented subcultures."

FM-2: Context Starvation → Generic Gravity Thin prompt context ("Xero + retention") produces generic-category outputs even at tail sampling. The more proprietary and specific the context, the better VS performs.

Mitigation: Load rich context before the VS prompt — company stage, current c

阅读完整来源 ↗