chaos-engineering
Validate system resilience through controlled fault injection. Covers hypothesis-driven chaos experiments, failure injection types (network, service, infrastructure, dependency), LitmusChaos/Chaos Mesh/AWS FIS/Gremlin/toxiproxy tooling, automated abort gating, game day planning, and progressive chaos adoption. Use when: "chaos engineering," "fault injection," "resilience test," "game day," "failure recovery," "system reliability," "blast radius." Not for: safe rollout flags/canary/dark launch during a release — use testing-in-production; designing new tests from production telemetry — use observability-driven-testing. Related: testing-in-production, observability-driven-testing, performance-testing, release-readiness, test-environments.
petrkindlmann · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add petrkindlmann/qa-skills --skill chaos-engineering技能说明
name: chaos-engineering description: >- Validate system resilience through controlled fault injection. Covers hypothesis-driven chaos experiments, failure injection types (network, service, infrastructure, dependency), LitmusChaos/Chaos Mesh/AWS FIS/Gremlin/toxiproxy tooling, automated abort gating, game day planning, and progressive chaos adoption. Use when: "chaos engineering," "fault injection," "resilience test," "game day," "failure recovery," "system reliability," "blast radius." Not for: safe rollout flags/canary/dark launch during a release — use testing-in-production; designing new tests from production telemetry — use observability-driven-testing. Related: testing-in-production, observability-driven-testing, performance-testing, release-readiness, test-environments. license: MIT metadata: author: kindlmann version: "2.0" category: knowledge
Quick Route
| Situation | Go to |
|---|---|
| First experiment ever, team is new | Starting Small → First Three Experiments |
| Designing one experiment | Chaos Experiment Workflow (5 steps) |
| Picking a tool for your environment | Tools → Choosing a tool decision tree |
| Running a team session | Game Day Planning |
| Need runnable injection commands/configs | references/fault-injection.md |
| Want the abort to fire without a human | references/fault-injection.md → Automated abort |
Discovery Questions
Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.
Environment and readiness:
- Where will chaos experiments run? (Pre-production only, production with approval, never production)
- What is the team's monitoring maturity? Can you detect problems in real time?
- Has the team practiced incident response? Is there a runbook?
- Is there executive buy-in for chaos engineering? (Important for production experiments)
Architecture:
- What is the architecture? (Monolith, microservices, serverless, hybrid)
- What are the critical dependencies? (Database, cache, message queue, third-party APIs)
- Are there single points of failure? (Single database, single region, no redundancy)
- What redundancy and failover mechanisms exist?
Current resilience practices:
- Do services have health checks? What do they check?
- Are there circuit breakers, retry logic, or timeout configurations?
- What happens when a dependency is unavailable? (Graceful degradation, hard failure, unknown)
- Have you experienced unexpected outages? What failed?
Team and culture:
- Is the team comfortable with controlled failure? (Anxiety is normal and should be addressed)
- Who would be the chaos engineering champion? (Needs someone to own the practice)
- What is the appetite for starting? (Start small or dive in)
Core Principles
1. Hypothesis-driven: define expected behavior before injecting
Every chaos experiment starts with a hypothesis: "We believe that if [failure X occurs], the system will [expected behavior Y]." Without a hypothesis, you are just breaking things. The hypothesis names concrete steady-state metrics (baseline metrics) — error rate, latency, throughput — and the bound each may move to.
Example hypothesis: "We believe that if the primary database becomes unavailable, the application will serve cached data for read requests and queue write requests for up to 5 minutes without user-visible errors. Blast radius: staging, one service. Steady-state baseline: error rate <0.1%, P95 latency <300ms."
2. Start small: one service, controlled blast radius
The first chaos experiment should not be "shut down production." It should be "add 200ms latency to one non-critical service in staging." Increase scope gradually as confidence and tooling mature.
3. Monitoring is a prerequisite
If you cannot detect problems in real time, you cannot safely inject failures. Chaos experiments without monitoring are just outages with extra steps. Verify dashboards, alerts, and on-call processes before running any experiment.
4. Game days build muscle memory
Running chaos experiments in automated pipelines is valuable, but game days -- scheduled sessions where the team runs experiments together and practices response -- build the human skills that matter during real incidents.
Chaos Experiment Workflow
Every chaos experiment follows this five-step process.
Step 1: Define steady state hypothesis
Identify the metrics that define "normal" and predict what should happen during the experiment.
Experiment: Database failover
Steady state:
- Error rate: < 0.1%
- P95 latency: < 300ms
- Successful orders per minute: > 50
Hypothesis: When the primary database fails over to the replica,
- Error rate will spike to < 2% for < 30 seconds
- P95 latency will increase to < 1s for < 60 seconds
- No orders will be permanently lost
- The application will recover without manual intervention
Step 2: Introduce the variable
Inject the failure in a controlled way with a clear scope and duration.
Injection:
Target: primary database (PostgreSQL)
Method: block TCP port 5432 on the primary instance
Scope: single database instance
Duration: 60 seconds
Blast radius: staging environment only (first run)
Abort conditions:
- Error rate > 10% for > 2 minutes
- Any data corruption detected
- Manual abort by experiment owner
Step 3: Observe
During the experiment, monitor all relevant metrics in real time. Assign observers to spec
阅读完整来源 ↗