jev-use
Build or adapt a bounded computer-use loop where Cua Driver observes and acts, TypeSafe Jev selects only from application-owned candidate IDs, and the caller validates and verifies every action. Use for the jev-use recipe or similar Jev integrations; do not use it to add model logic or credentials to Cua Driver.
trycua · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add trycua/cua --skill jev-useAbout this skill
name: jev-use description: Build or adapt a bounded computer-use loop where Cua Driver observes and acts, TypeSafe Jev selects only from application-owned candidate IDs, and the caller validates and verifies every action. Use for the jev-use recipe or similar Jev integrations; do not use it to add model logic or credentials to Cua Driver.
jev-use
Keep the decision layer above Cua Driver. Driver supplies observations and executes actions; the application constructs complete candidates; TypeSafe Jev returns one candidate ID. Never let Jev invent tool names, coordinates, refs, targets, delivery modes, or other arguments.
Use the example at libs/cua-driver/examples/jev-use/ as the runnable reference.
Keep TypeSafe request construction in the external Jev adapter rather than in
Driver or a Driver extension. The Python and TypeScript adapters must expose
equivalent mock and live behavior.
For a process boundary, use cua.jev_choice_request_v1 on stdin and require
cua.jev_choice_v1 on stdout. The request contains only a goal, capture ID,
compact regions, bounded history, and candidate IDs with descriptions; the
response contains only the selected ID, model identity, confidence, and
probabilities. Invoke the Python interpreter and absolute chooser path directly
without a shell.
For native desktop applications, use NativeAccessibilitySource and
cua.jev_choice_request_v2, which adds a per-candidate source (page, ax,
or visual), compact value-free elements, and optional progress counted
from the runner's own performed actions. Browser tasks keep sending v1.
Prefer browser DOM and semantic evidence. The optional visual adapter consumes
the public cua.visual_regions_v1 result only when Driver advertises both
parse_visual_regions and the capture-bound click.capture_id input.
Use the checked-in fixtures for deterministic development; do not add a model,
extension artifact, or Driver implementation detail to the recipe.
Decision loop
- State the goal and obtain a fresh Cua Driver observation through one persistent CLI or MCP session.
- Prefer an unambiguous fresh accessibility or browser DOM token.
- If visual grounding is needed, discover
parse_visual_regionsthrough the current MCP tool inventory. Validate its versioned result, capture ID, screenshot reference and dimensions, coordinate mapping, unique region IDs, bounds, content, confidence, and ambiguity. Build a pixel action only with the exact capture ID in the sameclickcall. Otherwise reobserve or abstain. - Construct a bounded candidate table. Each executable candidate contains the
complete Driver tool and arguments. Include
reobserveandabstainwhen evidence can be stale, incomplete, or ambiguous. - Send Jev only the goal, compact observation, recent history, and candidate
IDs with descriptions. Include typed visual regions and their
capture_idwhen the current observation has validated visual evidence; do not send extension internals or screenshot bytes. - Resolve the returned ID against the original immutable table. Reject an unknown, duplicate, malformed, denied, stale, or capture-mismatched choice, or a result below the caller's stated confidence policy.
- Execute at most one Driver action. Use background delivery by default; foreground delivery is an explicit escalation subject to the active Driver contract and user authorization.
- Reobserve and verify the postcondition before building another table.
Freshness and visual evidence
- Treat Driver page refs, accessibility tokens, screenshot IDs, and visual region IDs as observation-local. Never reuse them after the UI changes.
- Require visual bounds and centers to remain inside the exact screenshot coordinate space and tied to the same target and snapshot.
- If semantic and visual evidence disagree, or multiple regions are plausible,
offer
reobserveandabstainwithout inventing a mutation. - Never remove
capture_idor retry an expired, stale, or mismatched capture as an unbound coordinate action. - Use semantic evidence as authority when it is available. A visual label does not prove editability or interactivity.
Native accessibility candidates
- Build native candidates from one
get_window_statecall that returns the tree and the screenshot together, so element tokens andcapture_iddescribe the same moment. - Map raw AX, UIA, and AT-SPI roles through the role-class table in
native_roles.py/native_roles.ts, keyed by Driver'snormalized_role. Do not normalize roles in Driver. - Offer only enabled, on-screen, labeled, native elements; exclude
in_web_contentelements, window chrome, and labels equal to the value. - Derive candidate IDs from role class, label, and actionable-ancestor path,
never
element_index. Cap at 24 action candidates plusreobserveandabstain, and log how many were dropped. - Exclude delete, send, purchase, and close actions unless the task spec allows that risk. Text comes only from task parameters.
- A stale token or truncated tree leads to a reobserve, never to an unbound coordinate action.
- Verify completion through the task's independent oracle, such as the harness task-state file, not the accessibility tree the model saw.
Credentials and proof
The deterministic mock path must work without TYPESAFE_API_KEY. For live Jev,
read the key from the process environment or a secure interactive prompt; never
put it in source, command arguments, logs, artifacts, or messages. Verify task
completion from an independent application postcondition rather than a model
answer, action response, or screenshot alone.