Grade Your Agent's SOUL.md with soul-grader

Where this fits

You've just deployed a Hermes agent that's been drifting — its responses feel off, it's ignoring its own guardrails, and you suspect the SOUL.md has decayed into a vague wishlist. You need a rigorous, repeatable way to grade that file before you trust it again. soul-grader gives you a rubric grounded in the SOUL.md field-guide research, not gut feeling, so you can review, rewrite, or approve agent identity files with confidence.

Why agents benefit

• Applies a normative rubric: Uses the field-guide research artifacts as the only source of truth, so grading is consistent across every SOUL.md you touch — no vibes, no personal biases.
• Detects companion-doc contradictions: Automatically flags when a SOUL.md conflicts with its sidecar documents, catching issues that manual review often misses.
• Classifies active vs. retired agents: Distinguishes between living personalities and legacy files, so you don't waste effort polishing something that's already archived.
• Produces durable, shareable reports: Generates structured grading output that fits into your existing fleet review workflow, making it easy to track improvements over time.
• Handles non-Hermes services safely: Knows when a file isn't meant for Hermes and avoids forcing irrelevant criteria, keeping your entire fleet honest.

Practical scenarios

Pre-deployment audit for a new agent

You're about to ship a customer-facing support agent. Instead of eyeballing its SOUL.md, you run soul-grader in CI. It scores the file against the rubric, flags missing safety constraints, and suggests concrete rewrites before the agent ever sees production traffic.

Quarterly fleet-wide identity refresh

Your team maintains 20+ agents, and you've noticed some have drifted from their original purpose. You run the fleet grading workflow from the skill's references, classify which agents are still active, and identify which SOUL.md files need a phase-coupled refresh to match their current role.

Post-incident SOUL.md review

After a production incident where an agent ignored its escalation path, you use soul-grader to review the SOUL.md and its companion docs. The skill's contradiction checks reveal that the escalation policy was overridden in a sidecar file — something you'd never have spotted manually.

Add it to your agent workflow

Install the skill with a single command:

npx skills add cobibean/soul-grader-skill --skill soul-grader

Then invoke it from your agent toolchain to grade a SOUL.md file. For example, in a Hermes agent prompt:

Run soul-grader on `agents/support/SOUL.md` and output a report with scores, contradictions, and suggested rewrites.

The skill will follow the linked workflow references to produce a structured, actionable review. For more context on how this skill fits into the broader ecosystem, see the OpenAgentSkill listing.

Compare before adopting

Before you commit, compare soul-grader against other grading approaches on four axes:

• Quality signals: Does it use a real research-backed rubric or just generic checklists? Check the linked references for evidence.
• Maintenance freshness: Is the skill actively updated? Check the repo's commit history and issue tracker.
• Alternatives: Could you build the same with a custom prompt? Only if you have the time to codify your own rubric — soul-grader saves that effort.
• Workflow fit: Does it integrate with your existing fleet review process? The skill assumes a certain file structure and grading workflow, so verify it matches yours.

Why it is worth tracking

soul-grader sits at a rare intersection: it's both opinionated (normative rubric) and practical (fleet-grade workflows). With 58 stars and an active author, it's showing real community traction. The skill's focus on research-backed grading and production handoff patterns suggests it's built by someone who's actually run agent fleets, not just written a toy. Keep an eye on it if you're serious about maintaining agent identity quality — and evaluate it when you next audit your own SOUL.md review process.