chaos
Chaos engineering. failure injection, circuit breakers, game day, disaster recovery, resilience test, network failures.
arbazkhan971 · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add arbazkhan971/godmode --skill chaos技能说明
name: chaos description: Chaos engineering. failure injection, circuit breakers, game day, disaster recovery, resilience test, network failures.
Chaos — Chaos Engineering
Activate When
- User invokes
/godmode:chaos - User says "chaos test", "resilience test", "failure injection", "break it on purpose"
- User asks "what happens when X fails?" or "is this resilient?"
- User wants to plan a game day or disaster recovery drill
- Ship skill needs resilience validation before production deployment
- After a production incident, to prevent recurrence
Workflow
Step 1: Define Steady State
Before injecting failures, establish what "healthy" looks like:
STEADY STATE DEFINITION:
System: <service name / system boundary>
Architecture: <monolith | microservices | serverless>
Health indicators (must all be true for "steady state"):
- Response success rate: > <X>% (e.g., 99.9%)
- Response time P95: < <X>ms (e.g., 500ms)
- Error rate: < <X>% (e.g., 0.1%)
- Queue depth: < <N> messages (e.g., 1000)
- CPU usage: < <X>% (e.g., 80%)
- Memory usage: < <X>% (e.g., 85%)
- Active connections: < <N> (e.g., connection pool max)
...
Step 2: Identify Failure Domains
Map all the ways the system can fail:
FAILURE DOMAIN MAP:
| Category | Components | Impact if Failed |
|--|--|--|
| Network | Load balancer | Total outage |
| | DNS resolution | Total outage |
| | Inter-service network | Partial outage |
| | External API access | Feature degraded |
| Compute | Application process | Service restart |
| | Worker processes | Queue backlog |
| | Cron/scheduled jobs | Delayed tasks |
| | Container/VM host | Service relocation |
| Storage | Primary database | Read/write loss |
IF experiment crashes service: halt and rollback. WHEN steady-state violated: record finding.
Step 3: Design Chaos Experiments
Create specific, controlled experiments for each failure domain:
Experiment Template
CHAOS EXPERIMENT:
Name: <descriptive name>
Hypothesis: "When <failure condition>, the system will <expected behavior>"
Blast radius: <single request | single user | single service | entire system>
Duration: <how long to inject failure>
Rollback: <how to stop the experiment immediately>
Prerequisites:
- [ ] Steady state verified
- [ ] Monitoring dashboards open
- [ ] Rollback procedure tested
- [ ] Team notified (if production)
- [ ] Incident response team on standby (if production)
...
Network Failure Experiments
Experiment N1: Dependency Timeout
Hypothesis: "When the payment API responds slowly (5s+), the checkout
service returns a user-friendly error within 3 seconds and does not
block other requests."
Injection:
# Using tc (traffic control) to add latency
tc qdisc add dev eth0 root netem delay 5000ms
# Or using toxiproxy
toxiproxy-cli toxic add -n latency -t latency \
-a latency=5000 payment-api
...
Experiment N2: DNS Failure
Hypothesis: "System falls back to cached data when DNS fails." Injection: iptables -A OUTPUT -p udp --dport 53 -j DROP. Verify cached responses served, error messages shown for uncached.
Experiment N3: Packet Loss
Hypothesis: "With 10% packet loss, success rate stays >95%." Injection: tc qdisc add dev eth0 root netem loss 10%. Verify retry logic, SLO compliance, no pool exhaustion.
Process Failure Experiments
Experiment P1: Process Crash
Hypothesis: "When the application process crashes, it restarts within
30 seconds and no requests are dropped (load balancer removes unhealthy
instance)."
Injection:
# Kill application process
kill -9 $(pgrep -f "node server.js")
# Or in Kubernetes
kubectl delete pod <pod-name> --grace-period=0
Verify:
...
Experiment P2: Memory Pressure
Hypothesis: "At 90%+ memory, app sheds load gracefully." Injection: stress-ng --vm 1 --vm-bytes 80% --timeout 300s. Verify load shedding, no OOM kill, health check alive.
Experiment P3: CPU Saturation
Hypothesis: "At 95% CPU, health checks and critical paths prioritized." Injection: stress-ng --cpu $(nproc) --timeout 300s. Verify health check <1s, background deferred, autoscaling triggers.
Storage Failure Experiments
Experiment S1: Database Failover
Hypothesis: "When the primary database fails, the system fails over to
the replica within 30 seconds with < 1 second of write unavailability."
Injection:
# Stop primary database
docker stop postgres-primary
# Or in cloud — promote replica
aws rds failover-db-cluster --db-cluster-identifier <cluster>
Verify:
- Read traffic continues on replica immediately
...
Experiment S2: Cache Failure (Cold Cache)
Hypothesis: "When Redis is unavailable, the system falls back to direct
database queries with acceptable performance degradation (P95 < 2s
instead of < 200ms)."
Injection:
# Flush all cached data
redis-cli FLUSHALL
# Or kill Redis entirely
docker stop redis
Verify:
...
Experiment S3: Disk Full
Hypothesis: "When disk reaches 95%, the system stops non-critical writes,
alerts operators, and continues serving read traffic."
Injection:
# Fill disk to 95%
fallocate -l $(df --output=avail / | tail -1 | awk '{print int($1*0.90)}')k /tmp/fill-disk
Verify:
- Log rotation and temp file cleanup triggered
- Non-critical writes (analytics, logs) paused
- Critical writes (transactions) continue to reserved space
- Alert fires with disk usage percentage
...
Step 4: Circuit Breaker Validation
Specifically test circuit breaker behavior:
CIRCUIT BREAKER VALIDATION:
State Transitions
CLOSED ──(failures > threshold)──→ OPEN
| ▲ | |
| | | (timeout) |
| | ▼ |
└──(success)── HALF-OPEN ←─────────┘
CLOSED: Normal operation, requests flow through
OPEN: All requests fail fast (no network call)
HALF-OPEN: Limited requests to test recovery
Step 5: Game Day Planning
Organize a structured resilience testing exercise:
阅读完整来源 ↗