跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。

Your Agent Reran the Test. Did It Fix Anything?

Keep the first failure, distinguish retries from changed-code runs, and choose trace settings that preserve the evidence a coding agent needs to diagnose flaky tests.

2026-09-16

保留发布时的原文,未自动翻译。

A green retry is a different result

A checkout test fails. Your coding agent reruns it, gets a green result and reports that the problem is fixed. But no code changed. What exactly was fixed?

This guide is for maintainers delegating CI investigation to an agent. The deliverable should preserve the failure and explain the next experiment, not simply produce a successful command. A retry can help diagnose an intermittent problem; it cannot, by itself, establish that a defect was removed.

Methodology and scope

We reviewed the official Playwright retry and tracing guides alongside pytest's explanation of flaky tests on September 16, 2026. These sources cover result classification, captured evidence and possible causes. The checkout scenario below is an illustrative investigation plan, not a test we executed or a measured reliability improvement.

The procedure is our editorial recommendation. pytest supplies useful diagnostic considerations here, not APIs that should be pasted into a Playwright project. Repository-specific acceptance rules still belong to the maintainer.

Preserve three different identities

Playwright distinguishes an initial pass from a test that fails initially and passes on retry. It labels the latter flaky. On failure, its worker process and browser are replaced; with retries enabled, the failed test is retried in the new worker. That matters when interpreting a later success. See the Playwright retry guide.

Our proposed investigation record separates three identities:

  • The logical test: file, test name, browser project and selected scenario.
  • The run: commit plus uncommitted changes, dependency lock, configuration, command and relevant fixture identity.
  • The attempt: initial execution or retry, outcome, timestamp and evidence location.

A retry inside one run is not the same as starting a new CI job. Neither is equivalent to running after a patch. Preserve those distinctions in the agent's final report. Do not overwrite the first failure attachment with the last successful screenshot.

A fresh browser also does not establish that every external service was reset. List the services involved, including shared test accounts, queues or databases, and state which boundaries are actually controlled.

Capture the attempt you need to explain

The Trace Viewer documentation says that on-first-retry records the first retry, not the original attempt. It also documents retain-on-failure, which records tests and removes traces from successful runs. Traces can expose actions, snapshots and network activity.

The practical consequence is easy to miss: a trace of the successful retry does not show the earlier failing execution. Before saying the agent inspected the failure, check which attempt produced the attachment.

For a focused investigation, propose a recording policy that preserves the failing attempt, with a defined retention period. Avoid changing the whole project's tracing policy without review. Recording more evidence has storage and execution costs; it can also collect sensitive application data. Keep private traces in approved storage and redact evidence before sharing it externally.

Work one checkout hypothesis at a time

Consider an invented case: the test expects an order confirmation, but the first attempt still shows a processing state. A later attempt passes. The agent suspects that another test reused the same order fixture.

Ask for this sequence:

  1. Save the original assertion, run identity and available service logs before rerunning.
  2. State the hypothesis precisely: which shared identifier could affect which transition?
  3. Choose one discriminating observation, such as whether the order identifier already existed before the failing action.
  4. Propose one scoped change, such as generating an isolated fixture, while keeping the intended user-facing assertion.
  5. Record the new run separately and inspect the relevant neighboring tests, not only the isolated test.

This is a proposed experiment, not a conclusion about the hypothetical failure. If the evidence points to a genuine application race, changing the fixture may be irrelevant. If the required service logs are unavailable, report that gap rather than filling it with a confident explanation.

Set a stopping rule before rerunning

The pytest flaky-test discussion identifies uncontrolled state, order dependencies, cleanup and overly strict assertions as possible causes. It also warns that permanently quarantining a test can be dangerous. Intermittence deserves investigation; it is not proof that the infrastructure alone is at fault.

Give the agent an explicit diagnostic budget: the allowed commands, number of reruns, runtime budget and conditions requiring a human decision. Stop when that budget is exhausted or when the next step requires changing shared resources.

Do not let the agent repeatedly increase timeouts, weaken assertions or mark the test skipped merely to obtain green CI. Such changes need a separate rationale and approval. A temporary mitigation should retain an owner and a follow-up condition rather than being reported as a root-cause fix.

Limitations and the handoff

Repeated successful runs cannot prove that an intermittent failure is impossible. A bounded investigation may end with an unresolved hypothesis, and that can be an honest, useful result. This workflow does not replace the repository's release policy.

Require a short handoff containing the initial failure, every subsequent attempt, the tested hypothesis, the actual change if any, remaining uncertainty and the maintainer's next decision. Separate observed outcomes from explanations.

For execution boundaries, read our guide to worktrees versus agent sandboxes. When choosing a candidate from the skill directory, ask whether its debugging workflow preserves contradictory evidence. The most useful agent is not the one that can rerun fastest; it is the one that leaves a failure understandable.

0