跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

observability-driven-testing

Use production telemetry as INPUT to design new tests. Covers OpenTelemetry integration with tests, trace-based assertions, log-informed test creation, production-error analysis for coverage gaps, and telemetry-driven test prioritization. Use when: "trace-based testing," "design tests from logs," "OpenTelemetry assertions," "production errors point to test gaps," "telemetry-driven testing." Not for: safe rollout techniques (flags, canary) during release — use testing-in-production. Not for: scheduled post-deploy probes — use synthetic-monitoring. Not for: triaging CI failures — use ai-bug-triage. Related: testing-in-production, synthetic-monitoring, qa-metrics, ai-bug-triage.

kindlmann · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add petrkindlmann/qa-skills --skill observability-driven-testing

技能说明


name: observability-driven-testing description: >- Use production telemetry as INPUT to design new tests. Covers OpenTelemetry integration with tests, trace-based assertions, log-informed test creation, production-error analysis for coverage gaps, and telemetry-driven test prioritization. Use when: "trace-based testing," "design tests from logs," "OpenTelemetry assertions," "production errors point to test gaps," "telemetry-driven testing." Not for: safe rollout techniques (flags, canary) during release — use testing-in-production. Not for: scheduled post-deploy probes — use synthetic-monitoring. Not for: triaging CI failures — use ai-bug-triage. Related: testing-in-production, synthetic-monitoring, qa-metrics, ai-bug-triage. license: MIT metadata: author: kindlmann version: "2.1" category: production

Quick Route

SituationGo to
Make test execution emit traces correlated with the appOTel test-runner setup (references/trace-assertions.md)
Assert which services were called / no error spans / latencyTraces as Test Evidence
Turn a Sentry/Datadog error into a testProduction Error to Test Pipeline
Decide which endpoints need tests nextTelemetry-Driven Test Prioritization
A trace assertion is flaky or a span never arrivesFailure Modes

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Observability stack:

  • What APM/tracing tool is in place? (Datadog, New Relic, Honeycomb, Splunk Observability/SignalFx, ServiceNow Cloud Observability — formerly Lightstep, Dash0, Jaeger, Grafana Tempo, OpenTelemetry-native) — determines how you pull traces and which query syntax the diagnosis workflow uses.
  • Is OpenTelemetry instrumented in the application, and which services? — un-instrumented services are invisible and untestable via traces.
  • What logging infrastructure exists? (ELK, Loki, CloudWatch, Datadog Logs) — sets where log-by-trace-ID correlation happens.
  • Are structured logs used, or free-form text? — structured logs are parseable into test gaps; free-form needs a fingerprinting step first.

Tracing maturity:

  • Are distributed traces available across service boundaries? — without them, only single-service span assertions are possible.
  • What is the trace sampling rate? (100%, 10%, head-based, tail-based) — probabilistic sampling will randomly drop the trace a test asserts on; you must force-sample test traffic (see Failure Modes).
  • Can you search traces by error status, latency threshold, or custom attributes?
  • Are traces correlated with logs and metrics? — enables exemplars (metric → representative trace ID), which makes prioritization concrete.

Production error tracking:

  • What error tracking tool is used? (Sentry, SmartBear Insight Hub — formerly Bugsnag, Rollbar, Datadog Error Tracking, LaunchDarkly Observability — incl. session replay, formerly Highlight.io)
  • How are production errors triaged? (Automated, manual, ignored)
  • Is there a process for turning production errors into test cases?
  • What was the last production error that a test should have caught?

Test infrastructure:

  • Can tests emit telemetry? (Traces, custom metrics, structured logs)
  • Are test results correlated with application telemetry?
  • Do you have a test-to-code coverage mapping? — required to compute the error-rate-to-coverage matrix below.

Core Principles

1. Production data informs test priorities

The most valuable tests prevent real production errors — not theoretical edge cases, not contrived scenarios. Production error logs are a pre-prioritized backlog of tests you should have written, ordered by what real users actually hit.

2. Traces are test evidence

"The API returned 200" proves the endpoint responded. "The request hit the cache, skipped the database, and returned in <50ms" proves the system behaved correctly at every layer. Traces make tests deeper without making them more brittle.

3. Observability gaps equal test gaps

A code path with no traces, no logs, and no metrics is invisible — untestable in production and unverifiable during incidents. Observability coverage and test coverage are two views of the same problem.

4. Close the feedback loop

The complete cycle: error detected → analyzed → test written → deployed → recurrence prevented. If your team finds production errors but does not systematically create tests, the same class of error recurs.


Traces as Test Evidence

Pin @opentelemetry/semantic-conventions to an exact version and treat sem-conv bumps as breaking. Trace assertions reference attribute names by string; those names drift across releases and your assertions silently break. v1.41.0 (April 2026) shipped GenAI breaking changes and a process.executable entity split, and moved graphql.document from Recommended to Opt-In. Pin the literal version and bump deliberately:

// package.json — exact pin, no caret
"@opentelemetry/semantic-conventions": "1.41.1"
npm install --save-exact @opentelemetry/semantic-conventions@1.41.1

Do not introduce new OpenTracing shims. The OTel spec deprecated OpenTracing compatibility in March 2026 (removal no earlier than March 2027); new instrumentation should target native OTel APIs and OTLP.

Three patterns, all in `references/trace-assertio

阅读完整来源 ↗