sre-monitoring-and-alerting
Use when designing or reviewing monitoring, dashboards and alerts — golden signals, white-box vs black-box, alert hygiene, and turning SLOs into multiwindow multi-burn-rate alerts (with Prometheus templates and a burn-rate calculator). Based on Google SRE practice.
paulip114 · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add paulip114/sre-practice-skills --skill sre-monitoring-and-alerting技能说明
name: sre-monitoring-and-alerting description: Use when designing or reviewing monitoring, dashboards and alerts — golden signals, white-box vs black-box, alert hygiene, and turning SLOs into multiwindow multi-burn-rate alerts (with Prometheus templates and a burn-rate calculator). Based on Google SRE practice.
Monitoring and Alerting
Monitoring answers what is broken and why. Page humans only when the error budget is under real threat; everything else is a ticket or a dashboard.
Monitoring Design
Monitoring exists to answer two questions: what is broken (symptom) and why (cause). Build it so that a human is only interrupted for the first, and can find the second quickly.
Principles
- Alert on symptoms, investigate with causes. Users feel "checkout is slow", not "DB CPU is 90%". Page on user-visible symptoms (ideally SLO burn — see Alerting on SLOs below); keep cause metrics for dashboards and debugging.
- Every page must be urgent, actionable, and need human judgment. If a script could respond, automate it. If it can wait until morning, it's a ticket. If nobody acts on it, delete it.
- Keep it simple. Rules that rarely fire, signals nobody looks at, and clever heuristics rot. Prefer a small set of well-understood signals and remove what isn't used (review quarterly).
- Black-box + white-box. Black-box probes show what users see right now (and catch problems that never reach your servers). White-box metrics show internals and imminent problems. You need both.
- Measure distributions, not averages. Use histograms/percentiles for latency; averages hide the tail. Choose bucket boundaries that match SLO thresholds.
The four golden signals (request-serving systems)
| Signal | What to measure | Watch for |
|---|---|---|
| Latency | Time to serve, separately for successes and errors | Fast errors hide in blended averages; slow errors are worse than fast ones |
| Traffic | Demand: requests/s, sessions, transactions, bytes | Sudden drops can mean an upstream outage, not good news |
| Errors | Rate of failed requests: explicit (5xx), implicit (wrong content), policy (slower than promised) | Some errors are only visible end-to-end |
| Saturation | How full the most constrained resource is (CPU, memory, I/O, threads, queue depth, connection pools) | Latency increase is often the earliest sign; predict "full in N hours" |
For pipelines add freshness, throughput and backlog; for storage add durability and capacity headroom.
Design procedure
- Start from SLIs (
sre-slos-and-error-budgets). Those become the top of every dashboard and the source of pages. - Instrument the golden signals at each tier (load balancer, service, critical dependencies). Label by endpoint/request class, response code, and version so you can slice by release.
- Add saturation metrics for the resource that runs out first. Know what that is from load tests (
sre-capacity-and-system-design). - Add black-box probes for each critical user journey, from outside your network.
- Decide metrics vs logs vs traces. Metrics for things you alert on and trend (cheap, aggregated). Logs for per-event detail and forensic debugging. Traces for latency breakdown across services. Don't page from log scraping if a metric can carry the signal.
- Build dashboards in layers:
- Service overview: SLIs + budget remaining, golden signals, current release version.
- Dependency view: each critical dependency's latency/errors as seen from the caller.
- Resource view: saturation per component. Annotate deploys, config pushes and incidents on graphs.
- Write alert rules with: a clear name, severity (page/ticket), the user impact in the description, and a runbook link with first diagnostic steps.
- Test the monitoring. Verify alerts fire by injecting failures in staging or by unit-testing rules. Monitor the monitoring (a dead-man's-switch alert that fires when the pipeline stops sending).
Alert hygiene checklist
For every existing page:
- Did it detect something users would notice (or will soon)?
- Was action required immediately, and did a human need to decide what to do?
- Has it fired spuriously in the last quarter? How often was it ignored?
- Does it duplicate another alert for the same symptom?
- Does its runbook exist and still match reality?
Anything failing these: convert to ticket, automate the response, merge, or delete. Track pages per shift as a health metric (sre-on-call-and-toil).
Anti-patterns
- Paging on every cause metric "just in case".
- Static thresholds copied from another service without understanding.
- Dashboards with dozens of graphs and no ordering by importance.
- Monitoring that depends on the system it monitors (same cluster, same network path).
- Long
for:durations used to hide flapping instead of fixing the signal.
Alerting on SLOs
Page a human only when the error budget is under real threat. Everything else is a ticket or a dashboard.
Judge any alerting strategy on four properties
- Precision — share of alerts that were significant.
- Recall — share of significant events that alerted.
- Detection time — how fast it fires.
- Reset time — how long it keeps firing after recovery.
A "significant event" means it consumed a meaningful fraction of the error budget.
Why the simple options fail
| Approach | Problem |
|---|---|
| Alert when the short-window (e.g. 10 min) error rate ≥ SLO threshold | Poor precision: can fire many times a day while the SLO is still met |
| Same threshold, long window (e.g. 36 h) | Good precision, but it keeps firing for hours after the problem is fixed and is expensive to compute |
Short window + for: duration | Detection time ignores severity (a total outage waits as long as a tiny one); a flapping signal resets the timer and may never fire. Don't use durations for SLO alerts. |
| Single burn-rate alert (e.g |