跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

sre-monitoring-and-alerting

Use when designing or reviewing monitoring, dashboards and alerts — golden signals, white-box vs black-box, alert hygiene, and turning SLOs into multiwindow multi-burn-rate alerts (with Prometheus templates and a burn-rate calculator). Based on Google SRE practice.

paulip114 · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add paulip114/sre-practice-skills --skill sre-monitoring-and-alerting

技能说明


name: sre-monitoring-and-alerting description: Use when designing or reviewing monitoring, dashboards and alerts — golden signals, white-box vs black-box, alert hygiene, and turning SLOs into multiwindow multi-burn-rate alerts (with Prometheus templates and a burn-rate calculator). Based on Google SRE practice.

Monitoring and Alerting

Monitoring answers what is broken and why. Page humans only when the error budget is under real threat; everything else is a ticket or a dashboard.

Monitoring Design

Monitoring exists to answer two questions: what is broken (symptom) and why (cause). Build it so that a human is only interrupted for the first, and can find the second quickly.

Principles

  • Alert on symptoms, investigate with causes. Users feel "checkout is slow", not "DB CPU is 90%". Page on user-visible symptoms (ideally SLO burn — see Alerting on SLOs below); keep cause metrics for dashboards and debugging.
  • Every page must be urgent, actionable, and need human judgment. If a script could respond, automate it. If it can wait until morning, it's a ticket. If nobody acts on it, delete it.
  • Keep it simple. Rules that rarely fire, signals nobody looks at, and clever heuristics rot. Prefer a small set of well-understood signals and remove what isn't used (review quarterly).
  • Black-box + white-box. Black-box probes show what users see right now (and catch problems that never reach your servers). White-box metrics show internals and imminent problems. You need both.
  • Measure distributions, not averages. Use histograms/percentiles for latency; averages hide the tail. Choose bucket boundaries that match SLO thresholds.

The four golden signals (request-serving systems)

SignalWhat to measureWatch for
LatencyTime to serve, separately for successes and errorsFast errors hide in blended averages; slow errors are worse than fast ones
TrafficDemand: requests/s, sessions, transactions, bytesSudden drops can mean an upstream outage, not good news
ErrorsRate of failed requests: explicit (5xx), implicit (wrong content), policy (slower than promised)Some errors are only visible end-to-end
SaturationHow full the most constrained resource is (CPU, memory, I/O, threads, queue depth, connection pools)Latency increase is often the earliest sign; predict "full in N hours"

For pipelines add freshness, throughput and backlog; for storage add durability and capacity headroom.

Design procedure

  1. Start from SLIs (sre-slos-and-error-budgets). Those become the top of every dashboard and the source of pages.
  2. Instrument the golden signals at each tier (load balancer, service, critical dependencies). Label by endpoint/request class, response code, and version so you can slice by release.
  3. Add saturation metrics for the resource that runs out first. Know what that is from load tests (sre-capacity-and-system-design).
  4. Add black-box probes for each critical user journey, from outside your network.
  5. Decide metrics vs logs vs traces. Metrics for things you alert on and trend (cheap, aggregated). Logs for per-event detail and forensic debugging. Traces for latency breakdown across services. Don't page from log scraping if a metric can carry the signal.
  6. Build dashboards in layers:
    • Service overview: SLIs + budget remaining, golden signals, current release version.
    • Dependency view: each critical dependency's latency/errors as seen from the caller.
    • Resource view: saturation per component. Annotate deploys, config pushes and incidents on graphs.
  7. Write alert rules with: a clear name, severity (page/ticket), the user impact in the description, and a runbook link with first diagnostic steps.
  8. Test the monitoring. Verify alerts fire by injecting failures in staging or by unit-testing rules. Monitor the monitoring (a dead-man's-switch alert that fires when the pipeline stops sending).

Alert hygiene checklist

For every existing page:

  • Did it detect something users would notice (or will soon)?
  • Was action required immediately, and did a human need to decide what to do?
  • Has it fired spuriously in the last quarter? How often was it ignored?
  • Does it duplicate another alert for the same symptom?
  • Does its runbook exist and still match reality?

Anything failing these: convert to ticket, automate the response, merge, or delete. Track pages per shift as a health metric (sre-on-call-and-toil).

Anti-patterns

  • Paging on every cause metric "just in case".
  • Static thresholds copied from another service without understanding.
  • Dashboards with dozens of graphs and no ordering by importance.
  • Monitoring that depends on the system it monitors (same cluster, same network path).
  • Long for: durations used to hide flapping instead of fixing the signal.

Alerting on SLOs

Page a human only when the error budget is under real threat. Everything else is a ticket or a dashboard.

Judge any alerting strategy on four properties

  • Precision — share of alerts that were significant.
  • Recall — share of significant events that alerted.
  • Detection time — how fast it fires.
  • Reset time — how long it keeps firing after recovery.

A "significant event" means it consumed a meaningful fraction of the error budget.

Why the simple options fail

ApproachProblem
Alert when the short-window (e.g. 10 min) error rate ≥ SLO thresholdPoor precision: can fire many times a day while the SLO is still met
Same threshold, long window (e.g. 36 h)Good precision, but it keeps firing for hours after the problem is fixed and is expensive to compute
Short window + for: durationDetection time ignores severity (a total outage waits as long as a tiny one); a flapping signal resets the timer and may never fire. Don't use durations for SLO alerts.
Single burn-rate alert (e.g
阅读完整来源 ↗