跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

experiment-diagnostics

Diagnostics for randomized experiments: assignment integrity, attrition and missingness, intent-to-treat estimation, prespecified outcome families with multiplicity control, randomization inference, and noncompliance handling. Invoke from `code-and-data` whenever a target result rests on randomized assignment.

isitcredible · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add isitcredible/research_skills --skill experiment-diagnostics

技能说明


name: experiment-diagnostics description: >- Diagnostics for randomized experiments: assignment integrity, attrition and missingness, intent-to-treat estimation, prespecified outcome families with multiplicity control, randomization inference, and noncompliance handling. Invoke from code-and-data whenever a target result rests on randomized assignment.

Experiment diagnostics

Randomization buys identification at baseline and nothing after it. Check everything that happens after: attrition, noncompliance, outcome multiplicity, and analysis-unit mismatches can each quietly rebuild the selection problems randomization removed, and each has its own test.

Trigger: randomized assignment behind any target result. NOT_APPLICABLE for observational designs.

Method cribs (source-audited): ../../references/methods/young_2019_channelling_fisher.md and ../../references/methods/morgan_rubin_2012_rerandomization.md — restricted-assignment-set RI; a screened/rerandomized design analyzed with pure-randomization theory gets conservative inference (safe direction, but still the wrong distribution).

Core checks

  • Assignment and analysis-unit match. Randomized at the clinic, analyzed at the patient, is a clustering error; inference belongs at the level of assignment, with the cluster count reported. The whether-to- cluster decision is a design question, not a residual-correlation test (../../references/methods/abadie_athey_imbens_wooldridge_2023_clustering.md): cluster-level assignment is what forces the cluster adjustment, and in a fully unit-randomized trial clustering the SEs is a costly choice, not a default — check which variance component the design actually carries.
  • Collapse to the assignment unit (point estimate). The unit-match check fixes inference; when treatment is assigned at a coarser level than the analysis rows, also collapse to one row per assigned unit (average the replicate subsamples) and confirm the coefficient itself, not only its interval, is read at that resolution. See ../../references/design-matrix.md, "Collapse-test record".
  • Assignment integrity. Balance on predetermined covariates, and the randomization procedure as actually implemented — batch, stratified, re-randomized — reflected in the analysis.
  • Post-treatment conditioning. Check whether the headline spec controls for, or subsets on, a post-randomization variable (a mediator, an endogenous screen, an adjustment set chosen on results). An experiment re-imports bias exactly here; where present, hand to controls-samples-diagnostics with the model named.
  • Attrition and missingness. Overall and differential by arm; when the differential is non-trivial (say, the arms' attrition rates differ beyond a stated tolerance, or attrition correlates with baseline outcomes), run Lee bounds or a stated worst-case analysis. A differential that is not statistically significant is not clearance: the bounds show whether it matters beside the effect. Attrition is the experiment's selection problem readmitted through the back door.
  • Intent-to-treat estimate. The ITT is the design's protected quantity; report it before any per-protocol or treatment-on-treated number.
  • Prespecified outcome family and multiplicity adjustment. Compare the outcomes reported against the outcomes collected (registry, pre-analysis plan, survey instrument), define the family from the claim structure, and adjust. An experiment with twenty outcomes and one star is the multiplicity failure in its purest form. Where a pre-analysis plan or registry entry exists, cite ../../references/methods/olken_2015_preanalysis.md for the credibility limits of pre-registration; where the multiplicity is visible only in the paper's own spec table, report the pile-up of p-values just below the threshold descriptively — correlated specifications cluster around the baseline by construction, so the pile-up is not by itself evidence of search; price it with the specification sweep — and do not run the p-curve tests of ../../references/methods/elliott_kudrin_wuthrich_2022_phacking.md on one paper's table: they are across-study tests. (Humps at cutoffs are what non-increasingness tests detect; the "no power" case is p-hacking that shifts mass left without humps, which the paper's bounds tests target.)
  • Randomization inference where feasible — the design's own exact test, using the actual assignment mechanism. Name the permuted statistic: the t-statistic permutation under the true mechanism is the standard object (Young's randomization-t). Coefficient permutations (the randomization-c) test the same sharp null, but prefer the -t: it stays exact in joint tests and robust under heterogeneous effects, where the -c does not (Young 2019, §III.B). The permutation must respect the mechanism — permute within strata for a stratified design, condition on the re-randomization criterion where the design used one — or the RI p-value is invalid. Permuting arm labels while leaving arm-derived columns (treatment dummies, control flags, cell memberships) at their observed values is not a permutation of the assignment — it permutes only the margins whose columns are actually re-derived, and an "invariance" or degeneracy rate measured under it is a harness artifact. Re-derive every assignment-descended column from the permuted labels, and validate the generator before trusting it: rebuilding from the observed labels must reproduce the shipped columns exactly. The per-coefficient test inherits the omnibus caveat: it is a sharp-null test. Where RI is infeasible, record why and what was run instead. A degenerate or invariant direct statistic is NOT infeasibility — when the natural statistic is mechanically invariant under valid permutations (a pooled or selected cell whose membership barely moves), the obliga
阅读完整来源 ↗