executing-distributed-system-tests
Use when running a previously designed distributed-systems test plan against a real or simulated cluster — driving fault injection, workload, chaos scenarios, linearizability / consistency runs, durability, partition, crash-recovery, upgrade, performance/SLO runs, tenant isolation runs, boundary or authz runs, fairness / noisy-neighbor runs, or release validation. Also use when asked to "execute the plan", "reproduce a distributed bug", "run stability tests", "drive chaos", "validate a release end-to-end", "run the tenant isolation tests", or when a plan file exists at docs/testing-plans/ or any caller-specified location and needs to be run. Discovers and reuses the SUT's test toolbox rather than reinventing, captures nemesis landing evidence per scenario, runs the green-but-broken and weak-oracle audits before any PASS, and for boundary or fairness scenarios with §7.M.S arms runs each surface arm with its own verdict under a downgrade rule so the aggregate cannot fold an untested surf
shenli · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add shenli/distributed-system-testing --skill executing-distributed-system-tests技能说明
name: executing-distributed-system-tests description: Use when running a previously designed distributed-systems test plan against a real or simulated cluster — driving fault injection, workload, chaos scenarios, linearizability / consistency runs, durability, partition, crash-recovery, upgrade, performance/SLO runs, tenant isolation runs, boundary or authz runs, fairness / noisy-neighbor runs, or release validation. Also use when asked to "execute the plan", "reproduce a distributed bug", "run stability tests", "drive chaos", "validate a release end-to-end", "run the tenant isolation tests", or when a plan file exists at docs/testing-plans/ or any caller-specified location and needs to be run. Discovers and reuses the SUT's test toolbox rather than reinventing, captures nemesis landing evidence per scenario, runs the green-but-broken and weak-oracle audits before any PASS, and for boundary or fairness scenarios with §7.M.S arms runs each surface arm with its own verdict under a downgrade rule so the aggregate cannot fold an untested surface into a pass.
Executing Distributed-System Tests
Pairs with designing-distributed-system-tests. That skill produces
a plan; this skill runs it. The two communicate only through
filesystem artifacts: the plan file in and a session directory
plus findings report out.
The most common failure mode this skill is built to avoid: a run that produces a green checkmark without anyone having checked that the workload, the fault, and the oracle each did their job. The "green-but-broken" checks are not optional.
Process
1. Load the plan
If a plan file path was supplied, read it. If the user described a plan in conversation, extract the scenario list. If the plan is missing oracles or per-scenario budget tiers (Smoke / Hardening / Release, per the plan template's §7 scenario fields), halt — hand back to the design skill rather than improvise. Improvising an oracle in the moment is how green-but-broken results get produced.
2. Discover the SUT toolbox
Search the repo before writing any new code. Look for:
tools/,scripts/,bin/— drivers, workload generators, cluster bring-up scriptstests/integration/,tests/stability/,tests/chaos/docs/runbooks/,docs/testing/,docs/stability-test-plan.md- Makefile / justfile /
cargo xtasktargets that look like cluster commands - existing CI definitions that already wire fault-injection
Record what you found in the session log under "Toolbox discovered". This is required before any scenario runs — it prevents the skill from re-inventing tools that already exist.
2b. Probe environment capability and guide install if needed
Right after toolbox discovery, before running any scenario, check that the host can actually run the toolbox you just catalogued. If the plan has an "Environment requirements" section (the design skill emits one), treat it as the spec; otherwise infer the list from the selected techniques and the SUT toolbox.
Ask the user first; do not silently probe. Before running any
which / --version / docker ps checks, list the capabilities
the plan needs and ask the user: "what's available in your
environment, and what would you like me to skip?" Most operators
already know whether they have Docker, sudo for iptables, a Go
toolchain, etc. Asking up front saves a probe round, surfaces
substitution options the operator may know about (e.g. "I have
podman instead of Docker"), and respects their authority over
their own machine. After the user answers, verify with quick
probes and reconcile any gap between what they said and what's
actually present.
Probe categories:
- container runtime (docker / podman + compose)
- language toolchains at the version floors the plan declares
- backend services (Postgres, MinIO / S3-compatible, message brokers)
- fault-injection facilities (iptables, tc/netem, libfaketime, dm-flakey, Toxiproxy)
- kernel features (network namespaces for asymmetric partition, cgroups for IO throttling)
- observability stack referenced by the plan
For each capability, produce a row: requirement → present? → version → source. Record the full matrix in the session log; the findings report cites it per scenario.
For missing capabilities, do not silently mark INCONCLUSIVE. Two cases:
-
Trivially installable (a package the user can
apt-get install/brew installin seconds): surface the install command to the user with a one-line explanation of what it enables, and offer to proceed once they've installed (or to install for them if you have permission and the change is low-blast-radius). Do not runsudoor system-level installs without explicit user approval. -
Non-trivial to install (requires service setup, license, admin access, or careful configuration): explain what's missing, what scenarios depend on it, and what the user would gain by adding it. Then ask whether to (a) wait while they set it up, (b) proceed and mark dependent scenarios INCONCLUSIVE, or (c) substitute a degraded approximation (and document the substitution honestly in the findings).
Either way, INCONCLUSIVE is only the right verdict after the user has been told what's missing and either declined to install it or the install is genuinely out of reach. "Tried to run and silently no-opped" remains forbidden.
3. Establish a session directory
Create:
{{session_root}}/{{plan_slug}}/{{UTC_timestamp}}/
├── logs/
├── metrics/
├── artifacts/
└── findings/
Default session_root is ./test-sessions/ in the SUT repo, or
./test-sessions/ in the current working directory if the user
prefers not to write into the SUT repo. If the caller specified an
output root in the request (e.g. "produce a session directory and
findings report under /path/X/"), honor that path instead of the
default — do not silently relocate output. Place a copy of assets/session-log-template.md at
session-log.md inside this di