sre-capacity-and-system-design
Use when designing or reviewing a distributed system for scale and resilience — NALSD capacity math, load balancing, load shedding, retries, deadlines, graceful degradation, cascading failures, simplicity, and distributed consensus for critical state. Based on Google SRE practice.
paulip114 · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add paulip114/sre-practice-skills --skill sre-capacity-and-system-design技能说明
name: sre-capacity-and-system-design description: Use when designing or reviewing a distributed system for scale and resilience — NALSD capacity math, load balancing, load shedding, retries, deadlines, graceful degradation, cascading failures, simplicity, and distributed consensus for critical state. Based on Google SRE practice.
Capacity, Overload and Reliable System Design
Design with real numbers, survive the loss of your largest failure domain, and make every component degrade gracefully instead of collapsing under load.
Overload and Cascading Failures
A cascading failure is one that grows by positive feedback: one part fails, its load moves to the rest, which then fails too. Most are driven by overload, so the defense is to make every service degrade gracefully instead of falling over.
How cascades happen
- Server overload → slower responses → more concurrent requests → more memory/CPU → slower still.
- Resource exhaustion: CPU (queues grow), memory (GC thrash, OOM kills), threads, file descriptors, connection pools.
- Retry amplification: every layer retries, multiplying load on the struggling backend.
- Crash loops / health-check churn: overloaded tasks fail health checks, get removed, pushing more load onto survivors.
- Slow startup / cold caches: restarted tasks can't take full load yet.
- Triggers: traffic spikes, a lost cluster/zone, bad releases, cache flushes, batch jobs, a planned change that reduces capacity.
Defenses (design time)
- Load-test to find the limit and how the service fails beyond it. Know which resource runs out first. A service should keep serving at its capacity under excess load, not collapse.
- Load shedding: reject excess requests early and cheaply (based on concurrent requests, queue length or CPU) with an explicit "overloaded" error. Keep queues short — a long queue just adds latency.
- Graceful degradation: serve cheaper results (cached, partial, lower quality) when overloaded. Make degraded mode easy to trigger and test it regularly.
- Request criticality: tag requests (e.g. critical-plus, critical, sheddable-plus, sheddable) and propagate the tag; shed least critical first. Batch and prefetch traffic should be sheddable.
- Retries done right:
- Exponential backoff with jitter.
- Per-request retry cap (e.g. 3 attempts).
- A retry budget per client (e.g. retries ≤ ~10% of requests); stop retrying when exceeded.
- Retry at one layer only; lower layers return "overloaded, don't retry" so the error doesn't fan out.
- Deadlines: every request carries a deadline, propagated (minus elapsed time) to downstream calls. Servers drop work whose deadline has already passed. Beware deadlines set far longer than useful.
- Client-side adaptive throttling: clients track requests vs. accepted responses over a recent window and reject locally with probability ≈ max(0, (requests − K·accepts) / (requests + 1)), with K ≈ 2. This stops wasted work at the source when the backend is rejecting.
- Load balancing: at the frontend, steer users by proximity and capacity (DNS, anycast, L7). In the datacenter, use subsetting (each client talks to a bounded subset of backends) and balancing on load signals (e.g. weighted round robin or least-loaded), not naive round robin.
- Capacity planning: provision for peak plus headroom for losing your largest failure domain (N+2 is common); forecast organic growth and launches; autoscaling with sensible min/max, scaling on the right signal, and awareness that scaling a frontend can overload a backend.
- Avoid dangerous behaviors: don't fail health checks just because you're busy; make startup fast and warm caches gradually; avoid synchronized clients (jitter cron and retries); test the full-cluster-loss scenario.
Recovery playbook (during an overload incident)
- Stop the feedback loop first: increase capacity if possible, but more importantly reduce load.
- Shed or block load: drop sheddable/batch traffic, rate-limit or block the abusive or misbehaving client, turn on degraded mode.
- Stop retries amplifying (disable client retries, raise backoff).
- Stop health-check churn if it is killing tasks that could serve.
- Restart carefully in stages, letting caches warm, rather than all at once.
- Add traffic back gradually, watching saturation.
- Afterwards: postmortem, with load test and degradation work as action items.
Review checklist for a service
- Known capacity per task and per cluster, from load tests, with the limiting resource identified?
- Behavior beyond capacity tested — sheds instead of collapsing?
- Deadlines set and propagated; expired work dropped?
- Retries bounded per request and per client, at one layer, with backoff + jitter?
- Request criticality defined and used for shedding?
- Degraded mode exists and is exercised?
- Capacity covers peak with loss of the largest failure domain?
- Health checks distinguish "dead" from "busy"?
Reliable System Design Review
Non-abstract large system design (NALSD)
Design iteratively and make every design concrete: put real numbers on it and check it would fit on real machines and networks. An elegant abstract design that needs 10× the hardware you have is not a design.
Procedure
- Clarify requirements. Users, operations (reads/writes), data size, request rate, latency and availability SLOs, consistency needs, growth over 1–3 years.
- Is it possible at all? Sketch the simplest design ignoring scale (one machine). Does it meet functional requirements?
- Can we do better? Improve the algorithm/data layout before adding machines (e.g. O(n) → O(log n), precompute, index).
- Make it scale. Distribute: shard, replicate, cache, batch. For each component, calculate:
- Peak QPS (incl. growth and a safety factor), per-request CPU, memory, disk and network cost.
- St