跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

observability-and-instrumentation

Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing, or alerting. Use when shipping any feature that runs in production and you need evidence it works. Use when production issues are reported but you can't tell what happened from the available data.

addyosmani · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add addyosmani/agent-skills --skill observability-and-instrumentation

技能说明


name: observability-and-instrumentation description: Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing, or alerting. Use when shipping any feature that runs in production and you need evidence it works. Use when production issues are reported but you can't tell what happened from the available data.

Observability and Instrumentation

Overview

Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to Use

  • Building any feature that will run in production
  • Adding a new service, endpoint, background job, or external integration
  • A production incident took too long to diagnose ("we couldn't tell what happened")
  • Setting up or reviewing alerting rules
  • Reviewing a PR that adds I/O, retries, queues, or cross-service calls

NOT for:

  • Diagnosing a failure happening right now — use the debugging-and-error-recovery skill (observability is what makes that skill fast next time)
  • Profiling and optimizing measured slowness — use the performance-optimization skill
  • Launch-day monitoring checklists and rollback triggers — see the shipping-and-launch skill; this skill covers the instrumentation that feeds them

Process

1. Define "working" before instrumenting

Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.

2. Pick the right signal for each question

SignalAnswersCost profileExample
Structured log"What happened in this specific case?"Per-event; grows with trafficpayment_failed with provider error code
Metric"How often / how fast, in aggregate?"Fixed per series; cheap to queryp99 latency of provider calls
Trace"Where did time go across services?"Per-request; usually sampledOne slow checkout, broken down by hop

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels — use them consistently:

LevelMeaningOn-call action
errorInvariant broken; someone may need to actInvestigate
warnDegraded but handled (retry succeeded, fallback used)Watch for trends
infoSignificant business event (order placed, job finished)None
debugDiagnostic detailOff in production by default

Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:

// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

Never log secrets, tokens, passwords, or full PII. This is a hard rule from the security-and-hardening skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.

4. Metrics

For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.

As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' prom-client — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],  // '2xx', not '200'
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels — that belongs in logs and traces.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.

5. Distributed tracing

Use OpenTelemetry — it's the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with

阅读完整来源 ↗