qa-metrics
Define, track, and act on QA metrics: test coverage percentage, flakiness rate, defect escape rate, MTTR, test execution time trends, automation ROI, quality gates, and SLAs for test suites. Includes metric formulas, realistic targets by company stage, and the action to take when each metric goes red. Use when: "QA metrics," "test metrics," "quality KPIs," "test health," "flakiness rate," "defect escape rate." Not for: building the dashboard UI (Allure/Grafana) — use qa-dashboard; measuring coverage gaps and mutation score — use coverage-analysis. Related: qa-dashboard, coverage-analysis, ci-cd-integration, release-readiness, quality-postmortem.
kindlmann · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add petrkindlmann/qa-skills --skill qa-metricsAbout this skill
name: qa-metrics description: >- Define, track, and act on QA metrics: test coverage percentage, flakiness rate, defect escape rate, MTTR, test execution time trends, automation ROI, quality gates, and SLAs for test suites. Includes metric formulas, realistic targets by company stage, and the action to take when each metric goes red. Use when: "QA metrics," "test metrics," "quality KPIs," "test health," "flakiness rate," "defect escape rate." Not for: building the dashboard UI (Allure/Grafana) — use qa-dashboard; measuring coverage gaps and mutation score — use coverage-analysis. Related: qa-dashboard, coverage-analysis, ci-cd-integration, release-readiness, quality-postmortem. license: MIT metadata: author: kindlmann version: "2.0" category: metrics
Quick Route
| Your situation | Metric to reach for | Section / reference |
|---|---|---|
| No metrics yet, where to start | code coverage, flakiness, defect escape | Core Principles + references/rollout.md |
| Bugs escaping despite high coverage | defect escape rate + mutation score | Coverage Metrics, Defect Metrics |
| Tests flaky, devs ignore CI | flakiness rate, pass rate | Test Health Metrics |
| Slow pipeline, stale feedback | suite duration, parallelism efficiency | Execution Metrics |
| Justifying automation spend | automation ROI | Process Metrics |
| Picking targets for our stage | company-stage table | Setting Realistic Targets |
| Leadership wants delivery + quality | DORA + escape rate | Engineering Quality Metrics (DORA) |
| Building the dashboard view | (go to qa-dashboard) | — |
Discovery Questions
Check .agents/qa-project-context.md first — if it exists, use it as the foundation and skip anything already answered there. Then:
Current State
- What metrics are you tracking today? (Even "we glance at CI pass rate sometimes" counts.)
- Where does your test data live? (CI system, coverage reports, bug tracker, spreadsheets, nowhere)
- Do you have any dashboards already? Who looks at them, and how often?
- What tooling runs your CI pipeline? (GitHub Actions, GitLab CI, Jenkins, CircleCI)
Stakeholders
- Who are the stakeholders for quality metrics? (engineering, product, leadership, customers)
- What does each care about? Engineers want flakiness; leadership wants escape trends; product wants release confidence.
- Who will own each metric? If nobody owns it, nobody acts on it.
Quality Problems
- What quality problems need visibility? (regressions, slow pipelines, flaky tests, coverage gaps, incidents)
- What broke in production recently? Would a metric have caught it earlier?
- What decisions are you making without data today?
Goals
- What does "healthy test suite" mean for your team?
- Any compliance or contractual quality requirements? (SLAs, SOC2, ISO)
- Quick wins vs. full observability — what is the appetite for metrics infrastructure?
Core Principles
1. Metrics Should Drive Action, Not Just Dashboards
A metric without an action plan is decoration. For every metric, define: what threshold triggers action, what the action is, and who takes it. If flakiness crosses 5%, the on-call engineer investigates the top 3 flaky tests that week. No ambiguity.
2. Leading vs Lagging — Track Both, Act on Leading
Defect escape rate and MTTR are lagging: you learn after users were hurt. Flakiness, coverage delta, and skipped-test count are leading: they predict escapes before they happen. Lagging metrics tell leadership whether quality is moving; leading metrics are where engineers spend their daily attention because they can still change the outcome.
3. Trend Over Snapshot
A single number is nearly useless. Coverage at 72% means nothing; coverage trending 68% → 72% over three sprints tells a story. Display metrics as time series and evaluate direction, not absolute position.
4. Every Metric Needs a Target and an Owner
A target makes a metric actionable; an owner makes it accountable. Without both it becomes background noise. Set targets from your team's maturity (see the targets table) and assign owners who can actually move the number.
5. Vanity Metrics Waste Everyone's Time
"We have 4,000 tests" sounds impressive until 800 are disabled and 200 flaky. Count what matters: tests that run, pass reliably, and catch real bugs. The ultimate metric is whether users hit bugs that hurt the business — work backward from there. Defect escape rate connects directly to user experience; lines of test code connects to nothing.
Essential QA Metrics
Each metric: definition, formula, recommended target, why it matters, and the action when it goes red.
Test Coverage Metrics
How much of our system is verified by automated tests? (leading)
Code Coverage Percentage
The percentage of code exercised by automated tests (line, branch, or statement level).
Line coverage = (lines executed by tests / total executable lines) × 100
Branch coverage = (branches executed by tests / total branches) × 100
Targets: Line: 70-85% for app code. Branch: 60-75%. Critical paths (payments, auth): 90%+.
Why it matters: Coverage identifies blind spots — code never exercised by tests is where bugs hide undetected.
When it goes red: Coverage drops on PR: block merge or flag. Low in critical module: create targeted tasks. Plateaus: check for dead code vs. genuinely untested logic.
Warning: Coverage measures execution, not assertion quality. Pair it with mutation testing (StrykerJS v9.6+, mutmut) for a truer picture. Stryker's Vitest runner tracks recent Vitest releases — check
Read the complete source ↗