跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

ml4t-multi-agent-forecasting

Multi-agent probability forecasting with diversity, aggregation, and debate controls. Use when combining several agent forecasts or evaluating forecast ensembles.

ml4t · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add ml4t/skills --skill ml4t-multi-agent-forecasting

技能说明


name: ml4t-multi-agent-forecasting description: "Multi-agent probability forecasting with diversity, aggregation, and debate controls. Use when combining several agent forecasts or evaluating forecast ensembles." when_to_use: "Use when building multi-agent research, Neyman aggregation, adversarial debate, or resolved-question forecast evaluation" dependencies: [agent-state-memory] metadata: book_chapters: "24" library: "" paths: ["/multi_agent.py", "/forecasting_pipeline.py", "**/adversarial_debate.py"]

Multi-Agent Forecasting

Multiple agents help only when they add independent information or structured disagreement. Re-running the same prompt at higher temperature usually produces correlated forecasts, not a useful ensemble.

The Problem

Forecast pipelines often report "agent consensus" from several identical agents. On well-specified macro questions, those agents read the same evidence and return nearly identical probabilities. Averaging correlated forecasts gives false confidence unless the system measures diversity, calibrates probabilities, and stress-tests the consensus with opposing arguments.

The Pattern

WRONG

forecasts = [agent.run(question, temperature=0.7) for _ in range(5)]
p_yes = sum(f.p_yes for f in forecasts) / len(forecasts)
print(f"consensus={p_yes:.2%}")

CORRECT

import math
from statistics import mean


def logit(p: float) -> float:
    p = min(max(p, 1e-6), 1 - 1e-6)
    return math.log(p / (1 - p))


def inv_logit(x: float) -> float:
    return 1 / (1 + math.exp(-x))


def neyman_aggregate(probs: list[float], diversity: float) -> float:
    avg_logit = mean(logit(p) for p in probs)
    return inv_logit(avg_logit * diversity)


forecasts = [
    bull_agent.run(question),
    bear_agent.run(question),
    base_rate_agent.run(question),
]
divergence = max(f.p_yes for f in forecasts) - min(f.p_yes for f in forecasts)
aggregate = neyman_aggregate([f.p_yes for f in forecasts], diversity=1.2)

if divergence < 0.05:
    aggregate = run_adversarial_debate(question, forecasts).p_yes

Forecast Controls

  • Use role, evidence-source, or method diversity; do not rely on temperature alone
  • Preserve each forecast's evidence, confidence, rationale, and uncertainty list
  • Aggregate in logit space when probabilities are far from 50%
  • Evaluate resolved questions with Brier score, log score, calibration, and sharpness
  • Run ablations: no debate, no supervisor, simple mean, weighted aggregation

Guardrails

  • Identical-agent ensemble - check probability spread before claiming diversity
  • Consensus without calibration - low disagreement is not the same as accuracy
  • Free-text handoff - downstream aggregation needs typed p_yes fields
  • Leaky evaluation - only score questions resolved after the forecast timestamp

Checklist

  • Forecast artifacts contain probability, confidence, evidence, and timestamp
  • Agent diversity is structural, not only sampling noise
  • Aggregation method and diversity factor are recorded
  • Debate or supervisor stages are evaluated with ablations
  • Resolved-question scoring uses proper scoring rules