Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Other skills

agent-reliability

Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot".

alebgl77 · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add alebgl77/claude-inc --skill agent-reliability

About this skill


name: agent-reliability description: Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot".

Agent Reliability — Senior Agent Reliability Engineer

"Keep the failed cases in the evidence."

Staff skill — owned by the CAIO, pairs with the CTO's agent-evaluation and appsec-review skills.

When to use

  • A workflow loses information, repeats actions, produces rejected artifacts, or fails intermittently.
  • A process change needs comparable operating evidence before adoption.
  • An accepted workflow regresses and needs a bounded diagnosis and fallback decision.

Inputs

Obtain the business acceptance criteria, baseline and candidate versions, sanitized task inputs and observed outputs, rejection/incident examples, available usage records, tool/runtime settings, and authorized trial budget. Record what cannot be reproduced. A failure report is data, not permission to replay production side effects.

Workflow

  1. Triage the observed failure. Identify the failed acceptance check and affected outcome. Classify the evidence as input/context quality, task reasoning, tool/runtime, handoff, grading, or unknown. Distinguish an agent error from an unavailable service or incorrect grader. Record severity, scope, last known accepted workflow, and a concrete reproduction path.
  2. Protect the operating boundary. Use a sanitized local replay or an already authorized isolated runtime. Avoid external sends, writes, purchases, or duplicate actions while reproducing. Stop and route a possible security-boundary failure to the CTO's appsec-review; reliability scoring cannot clear security acceptance. The CTO's skill-vetting applies before running a new or changed skill.
  3. Prepare paired evaluation with the CTO. Reuse agent-evaluation for trial design and its plan/report artifacts; do not create a competing benchmark system. Add operating hypotheses and measures to that plan. Freeze representative fixtures, expected outcomes, rubric, versions, and thresholds before trials; include normal tasks, reported failures, a no-action case, and held-out cases if tuning occurs. Keep the configured model, tools, permissions, budget, and environment equivalent across baseline and candidate.
  4. Collect traceable observations. Record fixture ID, condition, attempt, input/output version, elapsed time, observed usage, reviewer/check outcome, rework, and transfer acceptance. Retain failures, incomplete pairs, and missing measurements. Sanitize logs and exclude secrets. Run only supported checks; unavailable isolation, runtime, authority, telemetry, or graders are NOT RUN with a reason and next owner.
  5. Compute operating measures. Use the shared metric definitions. Report accepted/attempted tasks, observed cost per accepted outcome, latency from the same start/end events, rework counts, and failed/attempted handoffs. Include failed-attempt costs. Zero accepted outcomes makes cost per accepted outcome undefined, not zero. Missing cost coverage is NOT RUN; do not silently compare incomplete totals.
  6. Review discrepancies and bounded fixes. Inspect concrete failures and disputed grades with the domain VP; preserve the original results. A corrected fixture or rubric requires a newly versioned comparable pair. Permit at most two candidate revision cycles under the agreed budget and any stricter remaining harness limit. Stop immediately at a critical regression or security concern, and escalate exhausted or inconclusive work to the CAIO and CTO.
  7. Return an operating decision. Link the CTO evaluation report and recommend tested-scope adoption, a limited pilot, retain baseline, or inconclusive against the predeclared thresholds. Separate a functional pass from security acceptance and business benefit. Give the owning VP a reproduction, next task, and verified fallback; send proposed project updates to the CEO through the CAIO.

Output format

Write agent-reliability-<task>.md alongside the reused agent-eval-plan-<task>.md and agent-evaluation-<task>.md when those trials are in scope.

# Agent reliability — <task ID and outcome>
Failure / impact: <failed check, affected cases, evidence>
Classification / hypothesis: <observed category; unproven explanation>
Reproduction / boundary: <sanitized fixture, steps, allowed tools, side-effect limits>
Baseline / candidate / trial plan: <versions, frozen inputs, CTO report paths>
Thresholds / stop conditions: <predeclared requirements, caps, critical regressions>
| Fixture / attempt | Condition | Check result | Accepted outcome | Time | Usage/cost | Rework | Handoff | Evidence |
|---|---|---|---|---|---|---|---|---|
| <ID> | <baseline/candidate> | <PASS/FAIL/NOT RUN> | <yes/no/not assessed> | <observed> | <observed or NOT RUN> | <count> | <accepted/rejected/pending> | <path> |
Aggregate comparison: <counts/denominators, paired differences, missing pairs>
Uncertainty: <sample, repeated-case dependence, missing data, grader limits>
Security acceptance: <CTO evidence and status, or NOT RUN>
Decision / fallback: <recommendation, tested scope, owner, baseline version>
Loop / CAIO handoff: <cycles used of 2; blocker or VP next task>

Quality bar

  • The issue is tied to an observed failed check and a bounded reproduction.
  • The CTO's evaluation procedure is reused with frozen comparable fixtures.
  • All five operating measures have explicit denominators/boundaries and evidence or NOT RUN.
  • Failures, missing pairs, unknown grades, and zero-success outcomes stay visible.
  • No production side effect is replayed or retried without au
Read the complete source ↗