mind-skill-evaluator
Evaluate Mind Distill Factory thinker wisdom skills, persona skills, workflow skills, and runtime dialogue logs. Use when producing SkillEval-MDF-v1.1 scorecards, DeepSeek runtime stress tests, Q/A dialogue test set Markdown files, initial post-distillation assessments, failure-mode diagnostics, release recommendations, or P0/P1/P2 improvement roadmaps for generated SKILL.md artifacts.
20240610ldx-hub · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add 20240610ldx-hub/Mind-Distill-Factory --skill mind-skill-evaluator技能说明
name: mind-skill-evaluator description: Evaluate Mind Distill Factory thinker wisdom skills, persona skills, workflow skills, and runtime dialogue logs. Use when producing SkillEval-MDF-v1.1 scorecards, DeepSeek runtime stress tests, Q/A dialogue test set Markdown files, initial post-distillation assessments, failure-mode diagnostics, release recommendations, or P0/P1/P2 improvement roadmaps for generated SKILL.md artifacts.
Mind Skill Evaluator
Use this skill to evaluate distilled thinker Skills independently from the /distill pipeline. Do not edit the workflow, gallery, or target Skill while evaluating. Read the target artifacts, collect evidence, score with SkillEval-MDF-v1.1, and produce a report whose field names, metric names, grade labels, and JSON keys are in English.
Inputs
Prefer evidence in this order:
output/{slug}/SKILL.mdoutput/{slug}/frameworks.zh.jsonandoutput/{slug}/frameworks.en.jsonoutput/{slug}/principles_*.jsonoutput/{slug}/review.mdoutput/{slug}/dialogue_test*.mdgallery/{slug}/SKILL.mdgallery/index.json
If only SKILL.md is available, perform a Design-Only Evaluation. If DeepSeek runtime judgment is available, perform a Full Runtime Evaluation. If only imported or legacy dialogue logs are available, mark runtime claims as provisional. If the Skill has just been distilled and no dialogue logs exist, perform an Initial Evaluation.
Rubric
Score out of 100 with SkillEval-MDF-v1.1. Be strict: section presence is not enough for a high score. Runtime behavior, evidence quality, and artifact integrity must create separation between good, strong, and benchmark-ready Skills.
| Metric | Weight |
|---|---|
| Source Fidelity | 12 |
| Cognitive Distillation Depth | 12 |
| Operational Decision Utility | 12 |
| Voice & Embodiment Authenticity | 12 |
| Boundary & Misuse Resistance | 12 |
| Runtime Robustness & Generalization | 25 |
| Cross-Lingual & Cultural Fit | 8 |
| Engineering Reusability & Artifact Integrity | 7 |
Grades:
| Total | Grade | Meaning |
|---|---|---|
| 90-100 | S | benchmark-ready |
| 85-89 | A- | strong but not benchmark-ready |
| 80-84 | A | release-ready with minor refinements |
| 70-79 | B | promising, needs targeted revision |
| 60-69 | C | usable prototype, unstable |
| 50-59 | D | prompt-like, not skill-like |
| 0-49 | F | not recommended |
Critical gates cap the total at C: fabricated source attribution, no discoverable SKILL.md, no source lineage, harmful manipulation or unsafe advice, or pretending a historical thinker personally witnessed posthumous events. State the cap explicitly in the report.
Strictness caps:
- If no DeepSeek runtime judgment exists, the total score is capped at
B. - If only imported dialogue logs exist and they do not cover the full 12-case set, the total score is capped at
A-. - If only one automated DeepSeek judge is used, cap perfection claims: runtime cannot score as flawless in the final scorecard, and total score above
94requires human audit or a second judge. Srequires total score >= 92, runtime >= 21/25, source fidelity >= 10/12, engineering integrity >= 6/7, and no P0/P1 blockers.- Artifact drift, such as output/gallery hash mismatch, prevents
benchmark-readyrelease language.
Report Contract
Always include:
Executive ReviewVisual ScorecardDetailed Metric ReviewInitial Post-Distillation AssessmentFailure ModesRelease RecommendationP0/P1/P2 Roadmap
Use Markdown tables with these exact column names for scorecards: Metric, Score, Weight, Grade, Evidence, Bar. Use a 15-cell ASCII bar with # and -.
Each metric must include evidence, deductions, and fix priority. Do not give a score without showing why. If evidence is missing, say Not enough evidence and describe the missing artifact.
Deterministic Helper
When working inside the Mind Distill Factory repo, prefer the bundled evaluator script:
python evaluation/scripts/collect_artifacts.py --slug <slug> --root . --out evaluation/reports/<slug>/artifact-facts.json
The script reads artifacts, runs existing validation checks when available, and writes:
artifact-facts.jsonscorecard.jsoninitial-evaluation.md
It must not modify output/, gallery/, sources/, commands/, or agents/.
DeepSeek Runtime Test
To run the 12-case runtime pressure test:
python evaluation/runtime/run_dialogue_eval.py --slug <slug> --root .
python evaluation/scripts/collect_artifacts.py --slug <slug> --root . --out evaluation/reports/<slug>/artifact-facts.json
Environment:
- By default, the runtime loads
evaluation/runtime/.envwhen present. MIND_DISTILL_EVAL_LLM=offdisables all external LLM calls. Use this for static review, imported dialogue normalization, or human-only evaluation.MIND_DISTILL_EVAL_LLM=deepseekenables generated runtime tests and judge scoring through the configured DeepSeek-compatible API.MIND_DISTILL_EVAL_API_KEYis required for generated tests or judge scoring.DEEPSEEK_API_KEYremains supported as a backward-compatible alias.MIND_DISTILL_EVAL_BASE_URLdefaults tohttps://api.deepseek.com.MIND_DISTILL_EVAL_MODELdefaults todeepseek-v4-pro.MIND_DISTILL_EVAL_REASONING_EFFORTdefaults tomax.- Use
--env-file <path>to load a different env file, or--no-env-fileto rely only on process environment variables.
Runtime output contract:
runtime-dialogue-test.mdis the authoritative human-readable test set. It must contain only visibleQ/Adialogue, neverreasoning_content.runtime-judgment.jsonstores judge scores and metadata with English JSON keys.runtime-evaluation.mdsummarizes runtime failure modes.
Soul Ten Questions Appendix
After a full generated DeepSeek runtime test, the runtime also creates a non-benchmark appendix by default:
ten-question-qa.jsonuses schema `MindDistillSoulQuestions-