Which Tests Ran? Check an Agent's pytest Selection Before Accepting Green

An agent repairs a checkout function and reports that the tests pass. Its command included a keyword filter. The test that exercises a declined payment was never selected.

Nothing in that story requires a flaky test or a broken test runner. The green result can accurately describe the selected tests while failing to answer the maintainer's question.

This workflow is for developers reviewing agent-generated pytest commands and CI changes. Before examining the outcome, establish which tests the run was supposed to include.

Methodology and scope

We reviewed pytest's invocation, marker, exit-code and integration documentation on October 12, 2026. The documentation establishes selection mechanisms and result meanings. The checkout example, membership record and acceptance rules below are our editorial recommendations, not an executed repository investigation.

This guide concerns test selection. It does not establish statement coverage, prove that assertions are meaningful or replace a project's release requirements.

Write the intended set before choosing a selector

Start with the behavior under repair. For an illustrative checkout change, the maintainer might require ordinary success, declined payment and duplicate submission cases. Identify their actual tests in the repository before giving the agent a command.

The pytest invocation guide documents selection by directory, module, node identifier and keyword expression. These choices express different scopes. A filename is not the same thing as one function, and a convenient keyword is not a formal description of a business requirement.

Prefer explicit node identifiers for a small diagnostic set. Use the repository's approved broader suite for release validation. Record why the diagnostic set is narrower rather than allowing its result to become an accidental claim about all tests.

A compact selection record should contain the source revision, working directory, active configuration, exact command, intended node identifiers and exclusions. If the repository has uncommitted changes, include that fact; the commit alone does not identify the tested source.

Compare collection with intention

For an authorized repository, a collection-only invocation can help inspect the selection before running the tests. The invocation documentation describes pytest --collect-only -q. Apply the same relevant paths and selectors that the proposed execution will use.

Do not describe this step as a sandboxed read of filenames. pytest's integration guidance discusses importing test modules. Collection can load project code and plugins. Review and constrain the execution environment before collecting tests from an unfamiliar repository; do not expose production secrets merely to obtain a list.

Compare the selected node identifiers with the intended cases. A total count is insufficient: replacing the declined-payment test with another success case can leave the count unchanged.

If a required case is absent, stop and investigate discovery rules, names, paths and filters. Do not automatically broaden permissions, create dummy tests or rename a test just to make a selector match.

Treat markers and keywords as reviewed inputs

The custom marker examples show how marker expressions include or exclude tests and how the run reports deselected items. A marker is metadata used for selection, not proof of the behavior a test exercises.

Ask the agent to explain both the selector and any changes it makes to markers. Removing a marker from a failing test can change which job runs it without correcting the application.

In the checkout example, compare the required cases with the collected set before and after a proposed CI edit. If the agent changes -m, -k, a test directory or discovery configuration, treat that as a change to the validation scope. It deserves review even if the application patch itself is tiny.

Keep excluded tests visible in the handoff. A selected test that is skipped and a deselected test are different situations; neither supplies a passing execution of the behavior you needed.

Preserve the runner's actual result

The pytest exit-code reference distinguishes successful execution, test failure, interruption, internal error, usage error and no collected tests. In particular, code 5 represents no tests collected.

Do not let a shell wrapper discard that signal merely to keep the CI job green. If an optional shard can legitimately be empty, define the exception in the repository's policy and report it as empty, not as tested successfully. A required regression case should not inherit an optional-shard exception.

Retain the exact invocation, exit code and summary together. A command that runs only collection establishes selection evidence, not successful test execution. A passing diagnostic subset establishes only the outcome of that subset.

A proposed acceptance checklist

Before accepting the agent's repair, review four questions:

1. Did the intended regression case appear in the selected node identifiers?
2. Did it actually execute rather than remain skipped or deselected?
3. Did the diagnostic command preserve its result without a wrapper masking failure?
4. Did the required wider suite run against the same intended source state?

The answer can be incomplete. If an external service is unavailable, report that the integration case was not executed and identify the remaining release decision. Do not substitute a unit test and silently claim equivalent coverage.

Keep test reports in the project's approved evidence location. Logs and attachments may contain tokens or customer-like fixtures, so inspect what is safe to share before attaching them to a public issue.

Limitations and next steps

Collection and execution can diverge when configuration, generated cases or environments change. Keep the interval bounded and retain the final execution summary. A correct selection is necessary for this workflow, but it does not prove that the selected assertions detect the defect.

This complements our guide to flaky retries and evidence: that article follows attempts after a test is selected; this one checks the membership of the run first. When reviewing a debugging Skill in the directory, ask for both.