sre-slos-and-error-budgets
Use when defining, reviewing or refining SLIs and SLOs, writing or applying an error budget policy (release freezes, reliability prioritization, escalation), or quantifying incidents and projects in error-budget terms. Based on Google SRE practice.
paulip114 · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add paulip114/sre-practice-skills --skill sre-slos-and-error-budgets技能说明
name: sre-slos-and-error-budgets description: Use when defining, reviewing or refining SLIs and SLOs, writing or applying an error budget policy (release freezes, reliability prioritization, escalation), or quantifying incidents and projects in error-budget terms. Based on Google SRE practice.
SLOs and Error Budgets
Define what "reliable enough" means for users, then agree in advance what the organization does when the service falls short.
SLO Design
Design service level indicators (SLIs) and objectives (SLOs) that track what users experience, and that the organization will actually use to make decisions.
Core ideas
- An SLO is a target for "reliable enough". Above it, nearly all users are satisfied; below it, they start to complain or leave.
- 100% is never the target. Every extra nine costs more and is worth less to users, dependencies between you and the user add their own failures, and a 100% goal leaves no room for change, which is the main source of outages.
- Express every SLI as good events / valid events, a ratio between 0% and 100%. Then error budget = 100% − SLO, and one set of tooling (numerator, denominator, threshold) works for every SLI.
- Separate the SLI specification (what matters to the user, e.g. "home page loads in < 100 ms") from the SLI implementation (where and how it is measured). One spec can have several implementations that trade off fidelity, coverage and cost.
- SLOs need an owner empowered to trade velocity for reliability — usually the product owner.
Procedure
- Scope. Pick one application. Name the users whose happiness you are optimizing.
- Map the system. Sketch components, request flow, data flow and critical dependencies. Classify each component as request-driven, pipeline or storage (see
references/sli-menu.md). - List critical user journeys (e.g. search → add to cart → checkout). These are what SLIs should ultimately represent.
- Choose SLI specs. At most ~5 SLI types for the service. Use multiple thresholds for latency-like SLIs so you cover both the typical case and the tail (e.g. 90% < 100 ms and 99% < 400 ms).
- Choose implementations. Start with the cheapest data that exists (logs or load-balancer metrics) over probers or client instrumentation that would take weeks. Prefer measuring as close to the user as is practical, and note what each source misses (e.g. server logs miss requests that never arrive).
- Set starter targets. If nothing better is known, measure recent performance and round down to a manageable figure (e.g. two significant figures for availability, a coarse step for latency). Don't lock yourself into today's accidental excellence, but remember users come to expect whatever you deliver.
- Pick a window. Default: a 4-week rolling window (whole weeks keep weekend counts constant), with weekly summaries for prioritization and quarterly reports for planning. Use calendar windows only where business planning needs them.
- Compute the budget in absolute terms (allowed bad events per window) so people can reason in "this outage cost 13% of our budget".
- Get agreement from three parties: product (target is good enough for users), development (will act when budget runs out), operations (defensible without heroics or excessive toil). If any party won't sign, iterate on the SLI/SLO — don't ship an SLO without teeth.
- Document it using
references/slo-document-template.md, including the rationale and whether numbers are observational or ad hoc. Review monthly at first, quarterly once stable. - Pair with an error budget policy (Error Budget Policy below) and burn-rate alerts (
sre-monitoring-and-alerting).
Iterating on an existing SLO
Collect a cheap signal of user happiness (support tickets, manually detected outages, forum posts, surveys). Then compare it with budget burn:
- Real user pain that never dented the budget → poor recall: tighten the SLO or move/broaden the SLI implementation.
- Budget burn with no user pain → poor precision: loosen the SLO or fix the SLI.
- If moving the threshold either way gives too many false positives or negatives, the implementation is the problem, not the number.
- Need a tighter target than you can meet today? Track it as an aspirational SLO that the error budget policy explicitly does not enforce.
Use this matrix to decide what to do next:
| SLO | Toil | Users happy? | Action |
|---|---|---|---|
| Met | Low | Yes | Ship faster / relax release process, or move SRE effort elsewhere |
| Met | Low | No | Tighten SLO |
| Met | High | Yes | Reduce false-positive alerting or temporarily loosen SLO; fix product / automate mitigation |
| Met | High | No | Tighten SLO |
| Missed | Low | Yes | Loosen SLO |
| Missed | Low | No | Make alerting more sensitive |
| Missed | High | Yes | Loosen SLO |
| Missed | High | No | Offload toil; fix product / automate mitigation |
Advanced patterns
- Bucket requests by importance or expected responsiveness (e.g. premium vs free tier; interactive vs bulk export) and give each bucket its own SLO, rather than one SLO per endpoint.
- Dependencies: a component on the critical path of a high-value journey needs an SLO at least as strict as that journey. Don't multiply availabilities as if redundant copies were independent — shared dependencies and control planes break that math.
- Don't confuse SLOs with SLAs: an SLA is a contractual promise with penalties and should be looser than the internal SLO.
Output
When asked to design SLOs, produce: the component classification, a table of SLI specs → implementations (with known blind spots), proposed targets with rationale, window, budget in absolute numbers, and a filled SLO document.
Error Budget Policy
An SLO only changes behavior if the organization has agreed in advance what it will do when the budget runs low. The error budget policy is that agreement.