跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

sre-on-call-and-toil

Use when designing on-call rotations and operational workload (page limits, interrupts, onboarding, team overload), measuring and eliminating toil with safe automation, or starting and growing an SRE practice in an organization. Based on Google SRE practice.

paulip114 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add paulip114/sre-practice-skills --skill sre-on-call-and-toil

技能说明


name: sre-on-call-and-toil description: Use when designing on-call rotations and operational workload (page limits, interrupts, onboarding, team overload), measuring and eliminating toil with safe automation, or starting and growing an SRE practice in an organization. Based on Google SRE practice.

On-Call, Toil and SRE Teams

Keep operational work bounded and shrinking so engineers spend most of their time on engineering, and build the team practices that sustain that.

Sustainable On-Call and Operational Load

On-call works when pages are rare, meaningful and manageable, and the rest of the team's time goes to engineering that makes pages rarer.

Targets to aim for

  • Ops work ≤ 50% of the team's time (on-call, tickets, toil, manual operations). The rest is engineering. When ops exceeds this, redirect the overflow to the development team or pause launches until it's back under control.
  • On-call ≤ 25% of any individual's time, which implies a rotation of roughly 8 people at one site, or 6 per site across two sites for 24/7 primary + secondary coverage.
  • At most ~2 incidents per 12-hour shift on average. Each real incident (handling, cleanup, postmortem) takes hours; more than that and quality drops. Zero for long stretches means skills atrophy — run drills.
  • Response time matches the SLO: a tight SLO needs minutes-level acknowledgment; a loose one can tolerate longer.
  • Multi-site rotations ("follow the sun") avoid night shifts but add handoff and communication cost.
  • Compensate out-of-hours on-call (time off or pay) and cap it so it stays bounded.

Rotation design checklist

  • Primary and secondary, with clear rules for when secondary takes over.
  • Escalation paths to developer on-call and dependency teams.
  • Explicit handoff at shift change (open issues, ongoing incidents, risky pushes).
  • Playbooks/runbooks linked from every alert.
  • Every page reviewed weekly: was it actionable? duplicated? fixable by automation?
  • Metrics tracked: pages/shift, incidents/shift, time-to-ack, out-of-hours pages, % pages with no action taken.

Psychological safety under pressure

Stress pushes people toward fast, intuitive, unverified action. Counter it with: clear escalation (asking for help is expected, not a failure), written procedures, and a culture where on-callers are trusted to act and not blamed for reasonable decisions.

Interrupts: separate them from project work

Interrupts include pages, tickets, and ongoing operational responsibilities (rollouts, flag changes, requests from other teams). Context switching destroys focus, so:

  • Assign one person to own interrupts for the shift (tickets as well as pages), and protect everyone else's project time.
  • The interrupt-holder does no project work that week; they work on reducing interrupts (fixing the causes of tickets, writing automation, improving docs).
  • Hand over explicitly between interrupt-holders.
  • If ticket volume needs more than one person, add a second dedicated person rather than spreading interrupts across the team.
  • Review tickets regularly: recurring types are toil to eliminate (Toil Reduction and Automation below).

Onboarding new on-callers

  • Learn the system by layers: request path, then dependencies, then failure modes. Use past postmortems as reading.
  • Shadow on-call, then reverse-shadow (newcomer leads, experienced person backs up).
  • Run "Wheel of Misfortune" role-plays: a game master replays a past incident, the trainee responds using real tools/dashboards, then the team debriefs.
  • Have trainees improve playbooks and dashboards as they learn — fresh eyes find gaps.
  • Use checklists of knowledge areas; certify readiness explicitly before joining the rotation.

Recovering from operational overload

Signs: constant firefighting, pages ignored, postmortems without follow-up, team morale low, ops > 50%. Overload can also be perceived — sudden changes in scope, unclear priorities or loss of autonomy can make a team feel overloaded even when measured load is moderate. Measure the load (tickets, pages, toil hours) and also ask the team; fix both the workload and the sources of uncertainty (clear priorities, explicit ownership, saying no to unplanned work).

Approach (often with an experienced SRE embedded for a few weeks):

  1. Learn the service and its context; identify the biggest sources of stress (top pages, top ticket types, riskiest manual procedures).
  2. Stop the bleeding by fixing what's on fire, but focus on reducing future work, not just doing today's.
  3. Introduce SLOs and an error budget policy so reliability vs. velocity becomes a data-driven conversation (sre-slos-and-error-budgets).
  4. Ensure postmortems are written and action items done.
  5. Rank toil and automate the top items.
  6. Leave behind written practices (on-call review, ticket review, postmortem process) so the team keeps improving.

Toil Reduction and Automation

What toil is

Toil is operational work that is:

  • Manual — a human runs it.
  • Repetitive — done again and again.
  • Automatable — a machine could do it (or the need could be designed away).
  • Tactical — interrupt-driven, reactive.
  • No enduring value — the service is no better afterwards.
  • Scales with the service — grows linearly with users, traffic or instances.

Not toil: overhead (meetings, HR), and engineering work that has lasting value, even if it's tedious. Some toil is unavoidable and even fine in small amounts; the problem is when it grows unchecked and crowds out engineering.

Budget: keep toil (plus other ops work) under ~50% of team time — see Targets to aim for above.

Procedure

  1. Identify. Sources: tickets by type, pages that needed a manual fix, runbook steps, requests from other teams, recurring calendar tasks, release/deploy steps. Ask the people doing it — they know.
  2. Measure. For each item: frequency
阅读完整来源 ↗