policyengine-data
Load for PolicyEngine's data layer — how the microdata behind population microsimulations is built, calibrated, versioned, and named. Covers the Microcosm stack (Frame kernel, microcosm-fit conditional models, microcosm-calibrate weights with L0 sparsity, build/release gates), the certified datasets that flow into policyengine bundles (populace_us_2024 sparse ~57k default, populace_us_2024_acs_local ~1.6M local-area, populace_uk_2023 private), the "one national dataset filtered by geography" local-area philosophy, the calibration diagnostics dashboard, and where data work goes now that policyengine-us-data is archived. Triggers: Microcosm (formerly Populace), Frame, microcosm-fit, microcosm-calibrate, calibration target, survey weights, reweighting, imputation, QRF, quantile loss, L0 sparsity, sparse 57k, local-area data, ACS, FRS, WAS, enhanced microdata, DEFAULT_DATASET. NOT for: running simulations (see policyengine) or diagnosing a specific score mismatch (see policyengine-calibrat
PolicyEngine · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add PolicyEngine/policyengine-claude --skill policyengine-data技能说明
name: policyengine-data description: | Load for PolicyEngine's data layer — how the microdata behind population microsimulations is built, calibrated, versioned, and named. Covers the Microcosm stack (Frame kernel, microcosm-fit conditional models, microcosm-calibrate weights with L0 sparsity, build/release gates), the certified datasets that flow into policyengine bundles (populace_us_2024 sparse ~57k default, populace_us_2024_acs_local ~1.6M local-area, populace_uk_2023 private), the "one national dataset filtered by geography" local-area philosophy, the calibration diagnostics dashboard, and where data work goes now that policyengine-us-data is archived. Triggers: Microcosm (formerly Populace), Frame, microcosm-fit, microcosm-calibrate, calibration target, survey weights, reweighting, imputation, QRF, quantile loss, L0 sparsity, sparse 57k, local-area data, ACS, FRS, WAS, enhanced microdata, DEFAULT_DATASET. NOT for: running simulations (see policyengine) or diagnosing a specific score mismatch (see policyengine-calibration-diagnostics). metadata: category: data
PolicyEngine data
How the microdata behind PolicyEngine population runs is built, calibrated, versioned, and
named. For using datasets in a simulation, see the policyengine skill (this skill is about
where the data comes from). For diagnosing why one score disagrees with a benchmark, see
policyengine-calibration-diagnostics.
The current data stack is Microcosm (repo PolicyEngine/microcosm, local mirror
~/PolicyEngine/microcosm; read its README.md + DESIGN.md). It replaced the technique-named
packages of the previous stack (microdf / microimpute / microcalibrate / L0 /
policyengine-us-data), which shared no datatype and had their worst bugs at the seams between
flat DataFrames.
The Microcosm architecture
Microcosm is one kernel datatype — the Frame — with packages as operators on it. It is a
PEP 420 namespace (microcosm.*) shipped as independently-installable shard distributions, so an
analyst doing imputation never has to install torch and vice versa. Microcosm releases pin the
shards as a constellation.
| Package | Import | Role | Succeeds |
|---|---|---|---|
microcosm-frame | microcosm.frame | the kernel: Frame, typed weights, strata, links, weighted accounting, unit structure, the RulesEngine protocol | microdf, microunit |
microcosm-fit | microcosm.fit | conditional models (weight-aware by construction) | ad-hoc imputation scripts |
microcosm-calibrate | microcosm.calibrate | targets → calibrated weights (APG / L0) | microcalibrate |
microcosm-build | microcosm.build | build plans, donor graphs, release gates, country build stages | one-off build drivers |
microcosm-data | microcosm.data | published population registry + lazy engine loaders | country-specific data packages |
Key design facts (from DESIGN.md) that change how you reason about the data:
- The
Frameis a weighted sampling frame of entity tables. Person + group-entity tables with explicitperson_<group>_idlinkage established once at assembly — no operator re-derives person↔unit attachment from a flat frame. It carries typed weights (design | importance | calibrated, one vector per weighted entity) with conservation invariants the kernel enforces (strata mass sums; no silent zeroing; no NaN/negative), and strata giving every record explicit provenance (cps_passthrough,synthetic_conditional,tail_verbatim, ...). Generation owns support (oversample where it is scarce); calibration owns representation. - The rules engine is an adapter, not a dependency.
microcosm.frame.rules.RulesEngineis a Protocol (variable_entity,variable_dtype,entity_schema,materialize,export_contract,write_dataset). Today's adapter ispolicyengine_us; the Axiomrulespec-usadapter is written against the same protocol so the swap is a new adapter, not a migration. microcosm-fitis weight-aware by construction — fits read the frame's typed weights; there is no unweighted default. Canonical model: regime-gated, chained quantile forests with weights materialized by weighted bootstrap.microcosm-calibrateis the only place calibrated weights are produced. Sparse target-matrix compilation + APG / L0 pruning is the core, not an option — "generate big then prune" is the intended design (300k → 3M → 30M candidate pools pruned to a compact frame). Its longitudinal rule: one weight per trajectory (multi-period targets stack as(target, period)constraint rows over one weight vector).- Process rules are as binding as the architecture: behavioral contract tests in CI from day one (weighted fits shift draws toward the weighted truth; calibration conserves declared mass; unit assignment partitions exactly); constellation versioning (consumers pin the constellation, not git SHAs); artifacts embed a certificate of the rules-engine + package versions that produced them; stage manifests are versioned artifacts with invariant checks.
The long-run goal in DESIGN.md ("The commons") is a communal, continuously-improving synthetic
population where the three contribution types are the package decomposition — records
(new strata at honest weights, frame), conditional structure (fitted P(y|x) models,
fit — the only way private sources contribute), and facts (targets with standard errors,
calibrate — Chronicle's lane). A contribution merges iff it improves the population's score on
held-out, rotated evidence without degrading a protected target family beyond tolerance.
Certified releases → policyengine bundles
Builds that pass the release gates are published to Hugging Face (policyengine/populace-us,
policyengine/populace-uk-private) and referenced by name in the certified bundle manifest that
ships inside the policyengine package. Verified in policyengine 4.2