loop
Continuously improve an AI agent from traces, feedback and evaluations through repeated learning and experiments. Use improve for a single fix or improvement and eval for a one-off evaluation run. Excludes scheduled prompts and polling.
relai-ai · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add relai-ai/agent-eval-loop --skill loop技能说明
name: loop description: Continuously improve an AI agent from traces, feedback and evaluations through repeated learning and experiments. Use improve for a single fix or improvement and eval for a one-off evaluation run. Excludes scheduled prompts and polling.
Loop: learn from new traces, fix, repeat
Inspect existing evidence and runners. Read relai/README.md, or the bundled
terms when absent; use the start skill
for needed managed records without rebuilding working infrastructure. This skill coordinates
learning and verified improvement across the discover, eval and improve skills
under one contract.
The flow below is a starting point. Adapt, reorder, repeat or skip activities when that better serves the request and the evidence, and load stage guidance only when relevant. The agreed scope, budget, authorization, evidence requirements and stage rules still bind.
Goal: useful learning and evidence-backed improvement with human judgment used where needed. Every kept change has comparable evidence and regression checks; every candidate decision is recorded with its reason. An investigation can produce useful knowledge without a code change. Maintain that knowledge when it helps future work under the shared policy; do not fill an iteration with artifact updates that add no durable value.
Two levels: learning iterations and experiments
- An iteration is a learning cycle started by a credible learning opportunity: new inputs, an unresolved question, or a useful experiment using existing evidence. It need not include trace intake, new tests or code changes.
- An experiment round happens inside an iteration (improve: improve-loop). It can produce learning without a patch. Accepted candidates become the incumbent; other exploration remains isolated.
A round cap counts experiment rounds, including diagnosis; report candidate attempts separately.
Done when
The paths below describe RELAI-managed records. Existing project contracts, iteration history, candidate decisions and reporting may serve the same purpose when authorization and evidence remain inspectable and recoverable; do not create duplicate state merely to match these filenames.
- The contract was approved (below) and saved in
relai/records/loop/state.json(relai/docs/files.md), with a line indecisions.md. - Before claiming an improvement, a comparable baseline exists for the objective and guardrails, measured or reused with a rationale. Investigation without a candidate records its findings and limits without an improvement claim.
- Each iteration is appended to
state.jsonunderiterations; each attempted candidate has arelai/changes.mdentry. - The loop stopped on a stopping condition, and you reported: the iterations run and what started each, experiments run, candidates tried and kept, the applicable objective/guardrail evidence and its limits, resource measurements, the regressions caught and changes rejected (and why), the user's decisions, total spend, benchmark coverage, and where the improved agent lives; the final split of a benchmark only when authorized. Relevant managed state or equivalent project reporting is current.
Establish the loop agreement
Reuse an existing contract covering these choices. Ask only for missing consequential choices
or changed boundaries; establish a new contract before starting when none exists. The contract is
the improvement agreement (relai/README.md, Terms) and supplies the authorization for the
stage decisions it covers. It is not approval to skip an uncovered consequential decision.
- Objective and guardrails: what improves (test pass rate, or cost or latency with scores held), and the guardrails with their bounds and evidence requirements.
- Scope: what may change (prompts, tool descriptions, skills, memory or context construction, retry logic, agent code, model settings) and what is off-limits.
- Inputs: the trace folder or stream to learn from, if any, the test suites in play, and any
benchmarks (with any agreed held-out sample and batch size per iteration). When traces will be
triaged by a model, the provider and model, and the user's authorization to send trace text to
it (discover: trace-triage); with that and the budget,
the loop runs triage beyond the pilot size (
--yesfor the example script). - Length: the most iterations to run, or "until there is nothing new"; and any experiment limit per iteration.
- Budget and evidence: a ceiling for the loop and any per-iteration limits, covering agent trials, judges and triage. Measurement effort is chosen per decision (eval: measure).
- When to ask the user (recommend: only these): user intent is genuinely ambiguous; two
requirements conflict; a behavior is irreversible or safety-sensitive; a judge and the user's
past verdicts disagree; a product-policy decision is needed. They are asked as
relai/README.mdsays ("When the user is asked"); everything else proceeds and appears in the digest. - Approvals:
delegated(default): fresh independent reviews stand in for scenario approval, grader previews and task lists, as delegated approvals describe; orsupervised: the user approves each scenario table, grader preview and kept change individually. Inside the loop this setting answers the build skills' "how involved" question. - Check-ins: a short digest after every iteration (default), or only at the end.
- Watch mode (optional): instead of stopping when nothing is new, re-check the trace source on a schedule the user sets up and start an iteration only when new traces or answers have arrived. A check th