human-eval-handoff-repair
Use when validating, repairing, or mapping human-evaluation handoff packages, filled annotation CSVs, rebuttal annotation UIs, or reviewer annotation returns across package versions. Applies to checking row alignment, detecting cross-snapshot contamination, converting old labels to current schemas, generating refill/import CSVs, and deciding whether filled annotations are safe for formal aggregation.
yha9806 · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add yha9806/academic-writing-toolkit --skill human-eval-handoff-repairAbout this skill
name: human-eval-handoff-repair description: Use when validating, repairing, or mapping human-evaluation handoff packages, filled annotation CSVs, rebuttal annotation UIs, or reviewer annotation returns across package versions. Applies to checking row alignment, detecting cross-snapshot contamination, converting old labels to current schemas, generating refill/import CSVs, and deciding whether filled annotations are safe for formal aggregation. allowed-tools: Read, Glob, Grep, Bash, Write
Human Eval Handoff Repair
Use this skill when the user asks to inspect, repair, migrate, or validate human-evaluation packages or filled annotation CSVs, especially when multiple package versions exist.
First Principle
Never trust row number, filename, or visible ID alone. Treat a filled CSV as usable only after it is aligned to the target public package by stable task-specific keys and its editable labels pass schema checks.
Do not fix labels by guessing. If a row cannot be matched safely, leave its annotation fields blank and produce a refill/import template plus an unmatched reference file.
Inputs To Locate
- Target public handoff package folder or ZIP.
- Filled CSVs from annotators.
- Any older package claimed as the source version.
- Current task schemas from the target UI data or target task CSVs.
- If relevant, QC reports from the target package.
Prefer a user-specified output directory. If none is specified, write generated reports and repaired files under codex_outputs/ in the current workspace. Avoid synced personal document folders unless the user explicitly asks for them.
Task Types And Stable Keys
Use these stable keys before copying any labels:
- Claim Warrant Audit:
item_id,tradition,medium,period,layer,dimension_id,claim_text. - Release Suitability Audit:
item_id,source_bucket,pipeline_mode,option_a_source,option_b_source,governed_option,option_a_text,option_b_text. - Card Faithfulness Spot Audit:
audit_field_id,tradition,card_type,dimension_id,field_name,field_value,source_reference_excerpt.
If the exact stable key does not match, do not formally aggregate the row. Text-only fuzzy candidates may be generated for manual review, but must not be treated as validated mappings.
Editable Label Columns
Claim Warrant Audit editable columns:
support_sourceoverreachcertainty_calibrationrecommended_actionannotator_confidencenotes
Release Suitability Audit editable columns:
suitability_decisionreason_primaryrisk_in_worse_optionannotator_confidencenotes
Card Faithfulness Spot Audit editable columns:
field_statusfield_provenancenotes
Only copy editable label columns. Never copy old evidence text, image paths, metadata fields, or UI-only columns into a new package.
Label Schema Checks
Claim legal values:
support_source:visual_observation,metadata_context,card_or_expert_context,b0_excerpt,mixed,noneoverreach:no_overreach,minor_overreach,major_overreach,unsupportedcertainty_calibration:well_calibrated,slightly_overstated,clearly_overstated,uncertain_or_not_checkablerecommended_action:accept,revise,reject,filter_or_abstainannotator_confidence: usually1to5
Release legal values:
suitability_decision:A_better,B_better,tie_both_ok,tie_both_bad,abstain_requiredreason_primary: target package enum only; validate from the target CSV or UI schema.risk_in_worse_option: target package enum only; validate from the target CSV or UI schema.annotator_confidence: usually1to5
Card legal values:
field_status:faithful,over_specific,unsupported,wrong,unclearfield_provenance:expert_critique,metadata,visual_observation,accepted_domain_context,none,unclear
Normalize harmless spelling and casing only when the mapping is deterministic. Otherwise flag the value and leave it blank.
Required QC Procedure
For every filled CSV:
- Count rows against the target task row count.
- Check required label columns are present.
- Check all non-empty labels are legal enum values.
- Compare non-label/source columns with the target blank CSV.
- Compute exact stable-key overlap with the target package.
- Identify whether the filled CSV matches the current package, an older package, or neither.
- Run simple logic checks.
- Produce a QC JSON and Markdown summary.
Important logic flags:
- Claim:
support_source=nonewithrecommended_action=accept. - Claim:
overreach=unsupportedwithrecommended_action=accept. - Claim:
recommended_action=filter_or_abstainwithoverreach=no_overreach. - Claim:
recommended_action=rejectwhilesupport_sourceis notnoneandoverreach=no_overreach. - Release:
abstain_requiredwith a confident preference reason. - Release:
A_betterorB_betterwith notes saying both options are unusable. - Card:
field_status=faithfulwithfield_provenance=none. - Card:
field_status=wrongwith strong provenance unless notes justify it.
Logic flags are review warnings, not automatic invalidation.
Decision Categories
PASS: exact target alignment, legal labels, no serious logic issues.PASS_WITH_MINOR_QC: exact alignment and legal labels, but has small note or logic warnings requiring spot check.PARTIAL_PASS: some rows safely map by stable key; use only mapped rows, refill unmatched rows.REFERENCE_ONLY: old package or fuzzy/text-only mapping; useful for guidance but not formal aggregation.FAIL_REDO: no safe stable alignment, major contamination, missing required labels, or wrong task schema.
Mapping And Repair Rules
When migrating old filled data to a current package:
- Start from the current target blank task CSV.
- Build exact stable-key lookup for current rows.
- For each old filled row, copy only editable labe