跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

policyengine-data

Load for PolicyEngine's data layer — how the microdata behind population microsimulations is built, calibrated, versioned, and named. Covers the Microcosm stack (Frame kernel, microcosm-fit conditional models, microcosm-calibrate weights with L0 sparsity, build/release gates), the certified datasets that flow into policyengine bundles (populace_us_2024 sparse ~57k default, populace_us_2024_acs_local ~1.6M local-area, populace_uk_2023 private), the "one national dataset filtered by geography" local-area philosophy, the calibration diagnostics dashboard, and where data work goes now that policyengine-us-data is archived. Triggers: Microcosm (formerly Populace), Frame, microcosm-fit, microcosm-calibrate, calibration target, survey weights, reweighting, imputation, QRF, quantile loss, L0 sparsity, sparse 57k, local-area data, ACS, FRS, WAS, enhanced microdata, DEFAULT_DATASET. NOT for: running simulations (see policyengine) or diagnosing a specific score mismatch (see policyengine-calibrat

PolicyEngine · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add PolicyEngine/policyengine-claude --skill policyengine-data

技能说明


name: policyengine-data description: | Load for PolicyEngine's data layer — how the microdata behind population microsimulations is built, calibrated, versioned, and named. Covers the Microcosm stack (Frame kernel, microcosm-fit conditional models, microcosm-calibrate weights with L0 sparsity, build/release gates), the certified datasets that flow into policyengine bundles (populace_us_2024 sparse ~57k default, populace_us_2024_acs_local ~1.6M local-area, populace_uk_2023 private), the "one national dataset filtered by geography" local-area philosophy, the calibration diagnostics dashboard, and where data work goes now that policyengine-us-data is archived. Triggers: Microcosm (formerly Populace), Frame, microcosm-fit, microcosm-calibrate, calibration target, survey weights, reweighting, imputation, QRF, quantile loss, L0 sparsity, sparse 57k, local-area data, ACS, FRS, WAS, enhanced microdata, DEFAULT_DATASET. NOT for: running simulations (see policyengine) or diagnosing a specific score mismatch (see policyengine-calibration-diagnostics). metadata: category: data

PolicyEngine data

How the microdata behind PolicyEngine population runs is built, calibrated, versioned, and named. For using datasets in a simulation, see the policyengine skill (this skill is about where the data comes from). For diagnosing why one score disagrees with a benchmark, see policyengine-calibration-diagnostics.

The current data stack is Microcosm (repo PolicyEngine/microcosm, local mirror ~/PolicyEngine/microcosm; read its README.md + DESIGN.md). It replaced the technique-named packages of the previous stack (microdf / microimpute / microcalibrate / L0 / policyengine-us-data), which shared no datatype and had their worst bugs at the seams between flat DataFrames.

The Microcosm architecture

Microcosm is one kernel datatype — the Frame — with packages as operators on it. It is a PEP 420 namespace (microcosm.*) shipped as independently-installable shard distributions, so an analyst doing imputation never has to install torch and vice versa. Microcosm releases pin the shards as a constellation.

PackageImportRoleSucceeds
microcosm-framemicrocosm.framethe kernel: Frame, typed weights, strata, links, weighted accounting, unit structure, the RulesEngine protocolmicrodf, microunit
microcosm-fitmicrocosm.fitconditional models (weight-aware by construction)ad-hoc imputation scripts
microcosm-calibratemicrocosm.calibratetargets → calibrated weights (APG / L0)microcalibrate
microcosm-buildmicrocosm.buildbuild plans, donor graphs, release gates, country build stagesone-off build drivers
microcosm-datamicrocosm.datapublished population registry + lazy engine loaderscountry-specific data packages

Key design facts (from DESIGN.md) that change how you reason about the data:

  • The Frame is a weighted sampling frame of entity tables. Person + group-entity tables with explicit person_<group>_id linkage established once at assembly — no operator re-derives person↔unit attachment from a flat frame. It carries typed weights (design | importance | calibrated, one vector per weighted entity) with conservation invariants the kernel enforces (strata mass sums; no silent zeroing; no NaN/negative), and strata giving every record explicit provenance (cps_passthrough, synthetic_conditional, tail_verbatim, ...). Generation owns support (oversample where it is scarce); calibration owns representation.
  • The rules engine is an adapter, not a dependency. microcosm.frame.rules.RulesEngine is a Protocol (variable_entity, variable_dtype, entity_schema, materialize, export_contract, write_dataset). Today's adapter is policyengine_us; the Axiom rulespec-us adapter is written against the same protocol so the swap is a new adapter, not a migration.
  • microcosm-fit is weight-aware by construction — fits read the frame's typed weights; there is no unweighted default. Canonical model: regime-gated, chained quantile forests with weights materialized by weighted bootstrap.
  • microcosm-calibrate is the only place calibrated weights are produced. Sparse target-matrix compilation + APG / L0 pruning is the core, not an option — "generate big then prune" is the intended design (300k → 3M → 30M candidate pools pruned to a compact frame). Its longitudinal rule: one weight per trajectory (multi-period targets stack as (target, period) constraint rows over one weight vector).
  • Process rules are as binding as the architecture: behavioral contract tests in CI from day one (weighted fits shift draws toward the weighted truth; calibration conserves declared mass; unit assignment partitions exactly); constellation versioning (consumers pin the constellation, not git SHAs); artifacts embed a certificate of the rules-engine + package versions that produced them; stage manifests are versioned artifacts with invariant checks.

The long-run goal in DESIGN.md ("The commons") is a communal, continuously-improving synthetic population where the three contribution types are the package decomposition — records (new strata at honest weights, frame), conditional structure (fitted P(y|x) models, fit — the only way private sources contribute), and facts (targets with standard errors, calibrate — Chronicle's lane). A contribution merges iff it improves the population's score on held-out, rotated evidence without degrading a protected target family beyond tolerance.

Certified releases → policyengine bundles

Builds that pass the release gates are published to Hugging Face (policyengine/populace-us, policyengine/populace-uk-private) and referenced by name in the certified bundle manifest that ships inside the policyengine package. Verified in policyengine 4.2

阅读完整来源 ↗