跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

chaos-engineering

Validate system resilience through controlled fault injection. Covers hypothesis-driven chaos experiments, failure injection types (network, service, infrastructure, dependency), LitmusChaos/Chaos Mesh/AWS FIS/Gremlin/toxiproxy tooling, automated abort gating, game day planning, and progressive chaos adoption. Use when: "chaos engineering," "fault injection," "resilience test," "game day," "failure recovery," "system reliability," "blast radius." Not for: safe rollout flags/canary/dark launch during a release — use testing-in-production; designing new tests from production telemetry — use observability-driven-testing. Related: testing-in-production, observability-driven-testing, performance-testing, release-readiness, test-environments.

petrkindlmann · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add petrkindlmann/qa-skills --skill chaos-engineering

技能说明


name: chaos-engineering description: >- Validate system resilience through controlled fault injection. Covers hypothesis-driven chaos experiments, failure injection types (network, service, infrastructure, dependency), LitmusChaos/Chaos Mesh/AWS FIS/Gremlin/toxiproxy tooling, automated abort gating, game day planning, and progressive chaos adoption. Use when: "chaos engineering," "fault injection," "resilience test," "game day," "failure recovery," "system reliability," "blast radius." Not for: safe rollout flags/canary/dark launch during a release — use testing-in-production; designing new tests from production telemetry — use observability-driven-testing. Related: testing-in-production, observability-driven-testing, performance-testing, release-readiness, test-environments. license: MIT metadata: author: kindlmann version: "2.0" category: knowledge

Quick Route

SituationGo to
First experiment ever, team is newStarting Small → First Three Experiments
Designing one experimentChaos Experiment Workflow (5 steps)
Picking a tool for your environmentTools → Choosing a tool decision tree
Running a team sessionGame Day Planning
Need runnable injection commands/configsreferences/fault-injection.md
Want the abort to fire without a humanreferences/fault-injection.md → Automated abort

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Environment and readiness:

  • Where will chaos experiments run? (Pre-production only, production with approval, never production)
  • What is the team's monitoring maturity? Can you detect problems in real time?
  • Has the team practiced incident response? Is there a runbook?
  • Is there executive buy-in for chaos engineering? (Important for production experiments)

Architecture:

  • What is the architecture? (Monolith, microservices, serverless, hybrid)
  • What are the critical dependencies? (Database, cache, message queue, third-party APIs)
  • Are there single points of failure? (Single database, single region, no redundancy)
  • What redundancy and failover mechanisms exist?

Current resilience practices:

  • Do services have health checks? What do they check?
  • Are there circuit breakers, retry logic, or timeout configurations?
  • What happens when a dependency is unavailable? (Graceful degradation, hard failure, unknown)
  • Have you experienced unexpected outages? What failed?

Team and culture:

  • Is the team comfortable with controlled failure? (Anxiety is normal and should be addressed)
  • Who would be the chaos engineering champion? (Needs someone to own the practice)
  • What is the appetite for starting? (Start small or dive in)

Core Principles

1. Hypothesis-driven: define expected behavior before injecting

Every chaos experiment starts with a hypothesis: "We believe that if [failure X occurs], the system will [expected behavior Y]." Without a hypothesis, you are just breaking things. The hypothesis names concrete steady-state metrics (baseline metrics) — error rate, latency, throughput — and the bound each may move to.

Example hypothesis: "We believe that if the primary database becomes unavailable, the application will serve cached data for read requests and queue write requests for up to 5 minutes without user-visible errors. Blast radius: staging, one service. Steady-state baseline: error rate <0.1%, P95 latency <300ms."

2. Start small: one service, controlled blast radius

The first chaos experiment should not be "shut down production." It should be "add 200ms latency to one non-critical service in staging." Increase scope gradually as confidence and tooling mature.

3. Monitoring is a prerequisite

If you cannot detect problems in real time, you cannot safely inject failures. Chaos experiments without monitoring are just outages with extra steps. Verify dashboards, alerts, and on-call processes before running any experiment.

4. Game days build muscle memory

Running chaos experiments in automated pipelines is valuable, but game days -- scheduled sessions where the team runs experiments together and practices response -- build the human skills that matter during real incidents.


Chaos Experiment Workflow

Every chaos experiment follows this five-step process.

Step 1: Define steady state hypothesis

Identify the metrics that define "normal" and predict what should happen during the experiment.

Experiment: Database failover
Steady state:
  - Error rate: < 0.1%
  - P95 latency: < 300ms
  - Successful orders per minute: > 50

Hypothesis: When the primary database fails over to the replica,
  - Error rate will spike to < 2% for < 30 seconds
  - P95 latency will increase to < 1s for < 60 seconds
  - No orders will be permanently lost
  - The application will recover without manual intervention

Step 2: Introduce the variable

Inject the failure in a controlled way with a clear scope and duration.

Injection:
  Target: primary database (PostgreSQL)
  Method: block TCP port 5432 on the primary instance
  Scope: single database instance
  Duration: 60 seconds
  Blast radius: staging environment only (first run)

  Abort conditions:
    - Error rate > 10% for > 2 minutes
    - Any data corruption detected
    - Manual abort by experiment owner

Step 3: Observe

During the experiment, monitor all relevant metrics in real time. Assign observers to spec

阅读完整来源 ↗