跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

chaos

Chaos engineering. failure injection, circuit breakers, game day, disaster recovery, resilience test, network failures.

arbazkhan971 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add arbazkhan971/godmode --skill chaos

技能说明


name: chaos description: Chaos engineering. failure injection, circuit breakers, game day, disaster recovery, resilience test, network failures.

Chaos — Chaos Engineering

Activate When

  • User invokes /godmode:chaos
  • User says "chaos test", "resilience test", "failure injection", "break it on purpose"
  • User asks "what happens when X fails?" or "is this resilient?"
  • User wants to plan a game day or disaster recovery drill
  • Ship skill needs resilience validation before production deployment
  • After a production incident, to prevent recurrence

Workflow

Step 1: Define Steady State

Before injecting failures, establish what "healthy" looks like:

STEADY STATE DEFINITION:
System: <service name / system boundary>
Architecture: <monolith | microservices | serverless>

Health indicators (must all be true for "steady state"):
  - Response success rate: > <X>% (e.g., 99.9%)
  - Response time P95: < <X>ms (e.g., 500ms)
  - Error rate: < <X>% (e.g., 0.1%)
  - Queue depth: < <N> messages (e.g., 1000)
  - CPU usage: < <X>% (e.g., 80%)
  - Memory usage: < <X>% (e.g., 85%)
  - Active connections: < <N> (e.g., connection pool max)
  ...

Step 2: Identify Failure Domains

Map all the ways the system can fail:

FAILURE DOMAIN MAP:
| Category | Components | Impact if Failed |
|--|--|--|
| Network | Load balancer | Total outage |
|  | DNS resolution | Total outage |
|  | Inter-service network | Partial outage |
|  | External API access | Feature degraded |
| Compute | Application process | Service restart |
|  | Worker processes | Queue backlog |
|  | Cron/scheduled jobs | Delayed tasks |
|  | Container/VM host | Service relocation |
| Storage | Primary database | Read/write loss |

IF experiment crashes service: halt and rollback. WHEN steady-state violated: record finding.

Step 3: Design Chaos Experiments

Create specific, controlled experiments for each failure domain:

Experiment Template

CHAOS EXPERIMENT:
Name: <descriptive name>
Hypothesis: "When <failure condition>, the system will <expected behavior>"
Blast radius: <single request | single user | single service | entire system>
Duration: <how long to inject failure>
Rollback: <how to stop the experiment immediately>
Prerequisites:
  - [ ] Steady state verified
  - [ ] Monitoring dashboards open
  - [ ] Rollback procedure tested
  - [ ] Team notified (if production)
  - [ ] Incident response team on standby (if production)
  ...

Network Failure Experiments

Experiment N1: Dependency Timeout

Hypothesis: "When the payment API responds slowly (5s+), the checkout
  service returns a user-friendly error within 3 seconds and does not
  block other requests."

Injection:
  # Using tc (traffic control) to add latency
  tc qdisc add dev eth0 root netem delay 5000ms

  # Or using toxiproxy
  toxiproxy-cli toxic add -n latency -t latency \
    -a latency=5000 payment-api

  ...

Experiment N2: DNS Failure Hypothesis: "System falls back to cached data when DNS fails." Injection: iptables -A OUTPUT -p udp --dport 53 -j DROP. Verify cached responses served, error messages shown for uncached.

Experiment N3: Packet Loss Hypothesis: "With 10% packet loss, success rate stays >95%." Injection: tc qdisc add dev eth0 root netem loss 10%. Verify retry logic, SLO compliance, no pool exhaustion.

Process Failure Experiments

Experiment P1: Process Crash

Hypothesis: "When the application process crashes, it restarts within
  30 seconds and no requests are dropped (load balancer removes unhealthy
  instance)."

Injection:
  # Kill application process
  kill -9 $(pgrep -f "node server.js")

  # Or in Kubernetes
  kubectl delete pod <pod-name> --grace-period=0

Verify:
  ...

Experiment P2: Memory Pressure Hypothesis: "At 90%+ memory, app sheds load gracefully." Injection: stress-ng --vm 1 --vm-bytes 80% --timeout 300s. Verify load shedding, no OOM kill, health check alive.

Experiment P3: CPU Saturation Hypothesis: "At 95% CPU, health checks and critical paths prioritized." Injection: stress-ng --cpu $(nproc) --timeout 300s. Verify health check <1s, background deferred, autoscaling triggers.

Storage Failure Experiments

Experiment S1: Database Failover

Hypothesis: "When the primary database fails, the system fails over to
  the replica within 30 seconds with < 1 second of write unavailability."

Injection:
  # Stop primary database
  docker stop postgres-primary

  # Or in cloud — promote replica
  aws rds failover-db-cluster --db-cluster-identifier <cluster>

Verify:
  - Read traffic continues on replica immediately
  ...

Experiment S2: Cache Failure (Cold Cache)

Hypothesis: "When Redis is unavailable, the system falls back to direct
  database queries with acceptable performance degradation (P95 < 2s
  instead of < 200ms)."

Injection:
  # Flush all cached data
  redis-cli FLUSHALL

  # Or kill Redis entirely
  docker stop redis

Verify:
  ...

Experiment S3: Disk Full

Hypothesis: "When disk reaches 95%, the system stops non-critical writes,
  alerts operators, and continues serving read traffic."

Injection:
  # Fill disk to 95%
  fallocate -l $(df --output=avail / | tail -1 | awk '{print int($1*0.90)}')k /tmp/fill-disk

Verify:
  - Log rotation and temp file cleanup triggered
  - Non-critical writes (analytics, logs) paused
  - Critical writes (transactions) continue to reserved space
  - Alert fires with disk usage percentage
  ...

Step 4: Circuit Breaker Validation

Specifically test circuit breaker behavior:

CIRCUIT BREAKER VALIDATION:
  State Transitions
  CLOSED ──(failures > threshold)──→ OPEN
| ▲ |  |
|  |  | (timeout) |
|  | ▼ |
  └──(success)── HALF-OPEN ←─────────┘
  CLOSED: Normal operation, requests flow through
  OPEN: All requests fail fast (no network call)
  HALF-OPEN: Limited requests to test recovery

Step 5: Game Day Planning

Organize a structured resilience testing exercise:

阅读完整来源 ↗