resilience-failure
This skill should be used when the user asks about "fault tolerance", "resilience", a "circuit breaker", "graceful degradation", "retry storm" or "thundering herd on recovery", "exponential backoff with jitter", "timeout", "bulkhead", a "single point of failure" (SPOF), "failover", or "rate limiting" (token bucket / leaky bucket / sliding window). Use it whenever a design must keep working through node crashes, slow dependencies, traffic spikes, or partial outages — i.e. any time the answer to "what happens when this breaks?" is missing, even if the user doesn't say "resilience".
proyecto26 · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add proyecto26/system-design-skills --skill resilience-failureAbout this skill
name: resilience-failure description: This skill should be used when the user asks about "fault tolerance", "resilience", a "circuit breaker", "graceful degradation", "retry storm" or "thundering herd on recovery", "exponential backoff with jitter", "timeout", "bulkhead", a "single point of failure" (SPOF), "failover", or "rate limiting" (token bucket / leaky bucket / sliding window). Use it whenever a design must keep working through node crashes, slow dependencies, traffic spikes, or partial outages — i.e. any time the answer to "what happens when this breaks?" is missing, even if the user doesn't say "resilience".
Resilience & Failure
Design the system so that when a part breaks — and it will — the failure is contained and the user still gets a useful (if degraded) answer instead of an error page or a cascading outage. Getting this wrong is the difference between a slow dependency and a total meltdown: the most common amplifier of an outage is the system's own reaction to it (retry storms, health-check stampedes).
When to reach for this
Any design with a remote dependency, a shared resource, or an SLA. Reach here to find single points of failure, decide what each call does when its dependency is slow or down, protect a service from being overwhelmed (rate limiting), and plan how a recovered service comes back without being crushed by the backlog.
When NOT to
Don't wrap a single in-process function or a best-effort batch job in circuit
breakers and bulkheads — that's machinery for cross-process/cross-network calls
(YAGNI). Don't add retries to a non-idempotent write without an idempotency key
first (→ api-design) — you'll duplicate side effects. The cheapest design that
meets the availability target wins; chasing an extra nine you don't need costs
real complexity (→ back-of-the-envelope for what a nine actually buys).
Clarify first
- Availability target — how many nines, and is it per-request or per-feature? (→
back-of-the-envelope.) - Blast radius — if this dependency dies, must the whole request fail, or can the feature degrade or hide?
- Idempotency — is the operation safe to retry? If not, what makes it safe (key, dedup)? (→
api-design.) - Latency budget — how long may a call wait before a timeout is better than waiting? (→
back-of-the-envelope.) - Limit dimension & policy — rate-limit per user / IP / API key / tenant? Hard (reject) or soft (queue/shape)? Burst tolerated?
The options
Layered defenses; most real designs combine several.
- Timeout — bound every remote call. Use everywhere; an unbounded wait is the root of most cascades.
- Retry with backoff + jitter — re-attempt transient failures with growing, randomized delays. Use for idempotent calls against blips; never naked retries.
- Circuit breaker — stop calling a dependency that's failing; fail fast and probe to recover. Use when a downstream is down or slow and retries would pile on.
- Bulkhead — isolate resources (thread pools, connection pools, queues) per dependency. Use so one slow dependency can't exhaust capacity shared by others.
- Graceful degradation — fall back to a cached/stale value, partial result, default, or hidden feature. Use when a usable-but-worse answer beats an error.
- Rate limiting / load shedding — cap inbound work; reject or shape excess. Use to protect a service from overload, abuse, or a stampeding caller.
- Redundancy / failover — run N>1 of every component; promote a standby on
failure. Use to remove SPOFs. (Health checks/LB failover live in
load-balancing.)
Rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window) and
the circuit-breaker state machine are detailed in references/deep-dive.md.
Trade-offs
| Option | What it solves | What it worsens | Change it when |
|---|---|---|---|
| Timeout | Bounds blocked threads; stops one slow call hanging the caller | Too tight → false failures; too loose → cascades | Tune to the dependency's p99, not a guess |
| Retry + backoff + jitter | Rides out transient blips | Multiplies load; duplicates non-idempotent writes | Add jitter + cap attempts + budget; require idempotency |
| Circuit breaker | Fails fast, gives a sick dependency room to recover | Adds state/tuning; can trip on a blip and over-shed | Flapping → tune thresholds / half-open probe rate |
| Bulkhead | Contains one failure to its own pool | Lower peak utilization; more pools to size | One noisy dependency starves others |
| Graceful degradation | Keeps the user served when a dependency dies | Serves stale/partial; more code paths to test | Correctness must be exact → fail closed instead |
| Rate limiting | Protects the service; bounds cost/abuse | Rejects legitimate bursts; needs shared state at scale | Limits too strict (valid drops) or too loose (overload) |
| Redundancy / failover | Removes SPOFs; survives node/region loss | Cost, replication lag, failover consistency risk | Failover drops un-replicated writes → consistency-coordination |
Behavior under stress
This block exists to stop the system from amplifying its own outage.
- Retry storm: a dependency slows, every caller retries, retries pile on the retries of callers upstream, and load multiplies geometrically. Mitigate: exponential backoff with jitter, a per-request retry budget (cap total attempts), and a circuit breaker so a dead dependency isn't retried at all.
- Thundering herd on recovery: a service comes back and every queued client
and expired cache entry hits it at once, knocking it over again. Mitigate:
half-open circuit breakers that admit a trickle, jittered client reconnect,
request coalescing, and slow-start ramp. (Cache-expiry stampede is
caching.) - Health-check stampede / accidental DDoS: aggressive health checks or load-balancer probes hammer a recovering instance. Mitigate: gentle probe intervals, fail-fast readiness, and draining. (Pro