Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Other skills

loop

Continuously improve an AI agent from traces, feedback and evaluations through repeated learning and experiments. Use improve for a single fix or improvement and eval for a one-off evaluation run. Excludes scheduled prompts and polling.

relai-ai · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add relai-ai/agent-eval-loop --skill loop

About this skill


name: loop description: Continuously improve an AI agent from traces, feedback and evaluations through repeated learning and experiments. Use improve for a single fix or improvement and eval for a one-off evaluation run. Excludes scheduled prompts and polling.

Loop: learn from new traces, fix, repeat

Inspect existing evidence and runners. Read relai/README.md, or the bundled terms when absent; use the start skill for needed managed records without rebuilding working infrastructure. This skill coordinates learning and verified improvement across the discover, eval and improve skills under one contract.

The flow below is a starting point. Adapt, reorder, repeat or skip activities when that better serves the request and the evidence, and load stage guidance only when relevant. The agreed scope, budget, authorization, evidence requirements and stage rules still bind.

Goal: useful learning and evidence-backed improvement with human judgment used where needed. Every kept change has comparable evidence and regression checks; every candidate decision is recorded with its reason. An investigation can produce useful knowledge without a code change. Maintain that knowledge when it helps future work under the shared policy; do not fill an iteration with artifact updates that add no durable value.

Two levels: learning iterations and experiments

  • An iteration is a learning cycle started by a credible learning opportunity: new inputs, an unresolved question, or a useful experiment using existing evidence. It need not include trace intake, new tests or code changes.
  • An experiment round happens inside an iteration (improve: improve-loop). It can produce learning without a patch. Accepted candidates become the incumbent; other exploration remains isolated.

A round cap counts experiment rounds, including diagnosis; report candidate attempts separately.

Done when

The paths below describe RELAI-managed records. Existing project contracts, iteration history, candidate decisions and reporting may serve the same purpose when authorization and evidence remain inspectable and recoverable; do not create duplicate state merely to match these filenames.

  • The contract was approved (below) and saved in relai/records/loop/state.json (relai/docs/files.md), with a line in decisions.md.
  • Before claiming an improvement, a comparable baseline exists for the objective and guardrails, measured or reused with a rationale. Investigation without a candidate records its findings and limits without an improvement claim.
  • Each iteration is appended to state.json under iterations; each attempted candidate has a relai/changes.md entry.
  • The loop stopped on a stopping condition, and you reported: the iterations run and what started each, experiments run, candidates tried and kept, the applicable objective/guardrail evidence and its limits, resource measurements, the regressions caught and changes rejected (and why), the user's decisions, total spend, benchmark coverage, and where the improved agent lives; the final split of a benchmark only when authorized. Relevant managed state or equivalent project reporting is current.

Establish the loop agreement

Reuse an existing contract covering these choices. Ask only for missing consequential choices or changed boundaries; establish a new contract before starting when none exists. The contract is the improvement agreement (relai/README.md, Terms) and supplies the authorization for the stage decisions it covers. It is not approval to skip an uncovered consequential decision.

  • Objective and guardrails: what improves (test pass rate, or cost or latency with scores held), and the guardrails with their bounds and evidence requirements.
  • Scope: what may change (prompts, tool descriptions, skills, memory or context construction, retry logic, agent code, model settings) and what is off-limits.
  • Inputs: the trace folder or stream to learn from, if any, the test suites in play, and any benchmarks (with any agreed held-out sample and batch size per iteration). When traces will be triaged by a model, the provider and model, and the user's authorization to send trace text to it (discover: trace-triage); with that and the budget, the loop runs triage beyond the pilot size (--yes for the example script).
  • Length: the most iterations to run, or "until there is nothing new"; and any experiment limit per iteration.
  • Budget and evidence: a ceiling for the loop and any per-iteration limits, covering agent trials, judges and triage. Measurement effort is chosen per decision (eval: measure).
  • When to ask the user (recommend: only these): user intent is genuinely ambiguous; two requirements conflict; a behavior is irreversible or safety-sensitive; a judge and the user's past verdicts disagree; a product-policy decision is needed. They are asked as relai/README.md says ("When the user is asked"); everything else proceeds and appears in the digest.
  • Approvals: delegated (default): fresh independent reviews stand in for scenario approval, grader previews and task lists, as delegated approvals describe; or supervised: the user approves each scenario table, grader preview and kept change individually. Inside the loop this setting answers the build skills' "how involved" question.
  • Check-ins: a short digest after every iteration (default), or only at the end.
  • Watch mode (optional): instead of stopping when nothing is new, re-check the trace source on a schedule the user sets up and start an iteration only when new traces or answers have arrived. A check th
Read the complete source ↗