challenge
Fourth review stage, run by a fresh agent: test the review's evidence in both directions — where the review errs against the authors and where it errs in their favour — rebuild it from the pristine deposit and the printed page, and rule on each element: STANDS, STANDS WITH CHANGES, WRONG or ADD, with proposed wording and every adverse result the stage's own scripts produced. Rules on evidence, never on severity.
isitcredible · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add isitcredible/research_skills --skill challenge技能说明
name: challenge description: >- Fourth review stage, run by a fresh agent: test the review's evidence in both directions — where the review errs against the authors and where it errs in their favour — rebuild it from the pristine deposit and the printed page, and rule on each element: STANDS, STANDS WITH CHANGES, WRONG or ADD, with proposed wording and every adverse result the stage's own scripts produced. Rules on evidence, never on severity.
Challenge
You test the review's evidence, not its tone, and you rule on evidence, never on severity: the verdict stage decides what the surviving evidence establishes. Reviews built to find fault err against the authors: in the first ten such tests every review had errors, nearly all against them, and three headline verdicts were wrong. A stage built only to defend the authors errs the other way: when it removed invalid attacks, valid ones went with them, and estimators the review's own logs showed to be uninformative were written up as identified. Argue the authors' case, then the reader's, and rule on the evidence. STANDS on a sound finding is as useful a ruling as WRONG on a bad one.
Inputs (filled in by the orchestrator)
- Evidence under test: every claim-ledger row, verbatim, with its holds, kernel, status, and the findings and evidence paths it rests on in both directions — what the review counted against the claim and what it counted for it (where the ledger has one undirected evidence column, sort it into the two directions yourself); every finding in the stage and module reports; and every calculation the review built itself that a finding rests on.
- Packet: {job root}: the published article, its appendix, and the pristine
04_replication_*/. - Work folder:
work/challenge/, holding scriptss1…, each with a log of the same name. A large run is split by group of result families, one agent per group, each writingreview/challenge-REPORT-<group>.md.
Rules
- YOU MAY READ THE REVIEW'S SCRIPTS AND LOGS TO SEE WHAT IT DID. YOU MAY NEVER RUN
ITS SCRIPTS, OR CITE ITS
results.csv, LOGS OR DERIVED DATA AS EVIDENCE. Re-running a flawed regression reproduces the flaw. Build every number from the pristine deposit and the printed page, and reproduce the paper's anchor cell before any variant. - A finding you did not rebuild stays in force. A ruling may weaken a row only on evidence you rebuilt. List every finding on the row that you did not rebuild, in either direction, under (d); it carries into the verdict stage unchanged. Removing an invalid argument against the paper does not remove the valid ones beside it.
- Every number in your report cites
script:linein your own log or a printed page and cell. A ruling with neither is not a ruling. Thereviewrules bind you (../SKILL.md, especially 4, 6 and 8). - Check each PDF's title page before citing it; cite printed page numbers.
Where reviews err against the authors
Test every element for each of these:
- A wrong design treated as the honest test: a variant the design's literature rules out, such as a shift-share shock leave-out or residualised variant without the exposure-sum control (for a leave-out, the sum of the remaining shares — with complete shares, the dropped shock's share). Find the correct version; the authors have often printed it. Reproduce it from the deposit before it rescues the claim; the wrong-design finding falls either way.
- Our repair changing their result: run the unmodified deposit and diff.
- A test with no power read as a refutation: simulate it under the paper's own estimate. If it would seldom reject when the paper is right, its non-rejection is no evidence. If it cannot reject any value, it is degenerate. A correction of the paper's own estimate (rule 8's definition) is not such a test: low power does not restore support the correction removed. A variant relabelled a correction is.
- An uninformative estimate read as a null: an interval that contains both zero and the paper's estimate refutes nothing. "Fails", "collapses" or "does not survive" needs an estimate that excludes the paper's. A claim whose evidence is uninformative is stated as not established. That is the finding, not a softer one, because the burden is the paper's. Say what the data cannot say, plainly, with the interval.
- An attack on a claim the paper does not make: find every quoted claim on the printed page and read the whole sentence for qualifiers ("In our model", "of final goods"). A quote that is not on the page, or a significance claim the paper never makes, is no target.
- A choice the paper states and justifies, counted as a failure. A stated justification answers an omission charge; it does not end the argument, and a review that rebuts the justification itself stands.
- The authors' own disclosure presented as our discovery: credit them.
Where reviews err in the authors' favour
Test every element for each of these:
- Evidence for the claim held to a lower bar than evidence against it. Rule 8 binds both directions. A first stage called strong, an estimate called precise, a result called robust, or a test the paper "survives" must hold at the inference level the design's literature requires (the shock level for shift-share, the level of the identifying variation for clustering), and the test must have been able to fail.
- A magnitude carried by an uninformative estimator. When the weak-identification-robust set (Anderson–Rubin, conditional LR, or a null-imposed shift-share interval) is unbounded, or the first stage is insignificant at the design's level, the estimator measures no magnitude: say so, and say what still carries the sign (usually the reduced form). Decode the output: a set printed with its lower endpoint above its upper, or with an infinite standard error, is unbounded.