跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

policyengine-calibration-diagnostics

Sensitivity registry for PolicyEngine microsim results — maps {program x deviation signature} to the calibration target or imputed variable most likely driving a mismatch, and reads the live per-target diagnostics for the current Microcosm release. The knowledge base behind the calibration-diagnostics agent and /analyze-policy Stage 5.6. Load when investigating why a microsim result differs from a prior score, or when reviewing whether a reform classification is calibration-sensitive. Triggers: "why does my reform not match", "policyengine cost off", "calibration mismatch", "imputed variable", "takeup rate", "non-filer", "itemizer share", "small state variance", "calibration target", "Microcosm (formerly Populace) target", "calibration dashboard", "target-diagnostics", "relative_error", "diagnose mismatch", "deviation signature".

PolicyEngine · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add PolicyEngine/policyengine-claude --skill policyengine-calibration-diagnostics

技能说明


name: policyengine-calibration-diagnostics description: | Sensitivity registry for PolicyEngine microsim results — maps {program x deviation signature} to the calibration target or imputed variable most likely driving a mismatch, and reads the live per-target diagnostics for the current Microcosm release. The knowledge base behind the calibration-diagnostics agent and /analyze-policy Stage 5.6. Load when investigating why a microsim result differs from a prior score, or when reviewing whether a reform classification is calibration-sensitive. Triggers: "why does my reform not match", "policyengine cost off", "calibration mismatch", "imputed variable", "takeup rate", "non-filer", "itemizer share", "small state variance", "calibration target", "Microcosm (formerly Populace) target", "calibration dashboard", "target-diagnostics", "relative_error", "diagnose mismatch", "deviation signature". metadata: category: data

PolicyEngine calibration diagnostics

Converts tribal "I'd check the takeup rate first" knowledge into a structured sensitivity registry, and pairs it with the live per-target calibration API so hypotheses are ranked against real relative_error numbers rather than assumptions. When an /analyze-policy comparison returns INVESTIGATE, this skill supplies the ranked candidate causes.

The calibrated microdata is now a Microcosm build (see the policyengine-data skill for how targets, weights, and L0 sparsity work). Calibration targets live in the Microcosm build's target set — not in a hand-maintained loss file — and their fit is queryable per release from the dashboard API below.

When to use

  • Stage 5.6 / Stage 6 of /analyze-policy — invoked by the calibration-diagnostics agent.
  • Code review of microsim PRs where the headline number differs from priors.
  • Designing or auditing a Microcosm calibration target.
  • Debugging why a state-level run looks volatile.

Top-level architecture

PolicyEngine microsim results depend on three layers:

  1. Country model logic (policyengine-us, policyengine-uk, policyengine-canada) — formulas, parameters.
  2. Calibrated microdata (Microcosm) — survey weights + imputations matched to administrative targets.
  3. Behavioral assumptions (takeup rates, labor-supply elasticities) — usually parameters but easy to overlook.

A magnitude mismatch is almost always rooted in layer 2 or 3, not layer 1 (layer-1 mismatches show up as outright simulation errors, not magnitude drift).

Live calibration API (check this first)

Per-target fit for the current Microcosm release, no auth, reads the release from Hugging Face:

BASE = https://calibration-diagnostics.vercel.app/calibration/dashboard/api/populace
GET {BASE}/target-diagnostics?source=<source>     # every target for a source, with relative_error
GET {BASE}/target-investigation?target=<id>        # full investigation packet for one target
GET {BASE}/releases                                # release ids for pinning

Sources map to reform domains: Social Security → ssa; Medicaid/ACA/Medicare → cms_medicaid / cms_aca / cms_medicare; TANF → hhs_acf_tanf; income tax / credits → irs_soi, jct, cbo; state income tax → state_income_*. A target already >10% off in the release diagnostics is a stronger hypothesis than any prior. (Dashboard UI: calibration-diagnostics.vercel.app.)

The three-ring reading method (align with /analyze-policy Stage 5.6)

Checking only the reform's primary marginal is the classic miss — a reform's cost usually depends on the joint distribution of its variable with other income, and the primary marginal can calibrate perfectly while the interaction is off. Check three rings outward:

  1. Ring 1 — the primary variable's marginals. Map the reform's domain to a calibration source and fetch its targets.
  2. Ring 2 — the mechanism's other inputs. Every variable that enters the formula the reform changes. SS-benefit taxation depends on combined income, so dividends, taxable interest, and pensions (irs_soi) are load-bearing; a CTC phase-in depends on earnings; a SNAP change on rents and deductions.
  3. Ring 3 — the interacted quantity itself (best single check). Search for the downstream quantity the reform directly reprices — taxable_social_security_amount for SS-taxation, eitc_amount for EITC, itemizer counts for deduction caps. When such a target exists it validates the joint distribution end-to-end through the actual formula, and its relative_error bounds the data-side bias of the score. Weight it above every marginal. When no ring-3 target exists, say so — "marginals within tolerance; joint distribution untargeted" is materially weaker evidence.

Worked example (HR 904, SS-taxation reform): ssa total benefits calibrate to −0.2% (ring 1 fine) while irs_soi ... taxable_social_security_amount runs −10.9% (ring 3) — the score's operative base is ~11% low in the data even though the headline marginal is nearly exact.

Interpretation is informative, not a gate: a poorly-calibrated but reform-relevant target does not flip the comparator verdict by itself, but it belongs in the report's uncertainty discussion. Escalate to INVESTIGATE when a directly-load-bearing target (the primary variable OR the ring-3 interacted quantity) is off by |relative_error| > 25%, even if headline numbers match anchors.

Sensitivity table — top programs

The rows below are the paradigm-independent knowledge: which calibration input moves which result, and in which direction. They hold regardless of which build produced the weights — use them to decide which targets to pull from the API above.

EITC

Calibration inputDirection of effectNotes
Childless-adult takeup rateHigher → more cost, more poverty reduction in deciles 1-3Historically ~65% childless vs ~80% with-kids. Uniform-takeup models over-state expansions. Verify ag
阅读完整来源 ↗