跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

jev-computer-use

Use when driving a desktop GUI through a computer-use driver — windows, menus, native apps, OS dialogs. You build a table of safe actions; Jev picks the next one in about 0.4 seconds.

kerpopule · 站主发布

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add https://github.com/kerpopule/hermes-jev-skills/tree/e5918165f457717b388152efd26f1de32d587466/skills/jev-computer-use --skill "jev-computer-use"

技能说明


name: jev-computer-use description: Use when driving a desktop GUI through a computer-use driver — windows, menus, native apps, OS dialogs. You build a table of safe actions; Jev picks the next one in about 0.4 seconds. version: 0.1.0 license: MIT metadata: hermes: tags: [jev, typesafe, computer-use, gui, cua] related_skills: [jev-browser-use]

Computer use with Jev

You stay the planner and the hands. Jev is only the fast "which one next?" in the middle. It returns an id from a table you built, so it cannot invent coordinates, text, selectors or tool calls. The worst a wrong answer can do is pick another action you already judged safe.

Web pages belong to jev-browser-use. This skill is for desktop apps and OS surfaces, driven through whatever computer-use driver you have (CUA Driver over MCP, the platform's native computer-use tool, an accessibility bridge).

The loop

  1. Observe with your driver. Prefer accessibility/semantic state over pixels. Every ref, capture id and coordinate is good for this observation only.

  2. Build the candidate table locally. Each row is an opaque id plus one complete, prevalidated action. Always include:

    • reobserve: look again, change nothing
    • abstain: stop and ask for help
  3. Privacy gate. Nothing sensitive goes to Jev: no credentials, tokens, cookies, password-field contents, payment data, customer data, screenshots, files or unbounded page text. If the screen holds such content, abstain or handle it without Jev.

  4. Ask once:

    jev choose < request.json          # Hermes: the jev_choose_action tool, argument `request`
    
    {"schema": "jev.action_choice_request_v1",
     "goal": "Open Settings and select Appearance.",
     "observation_id": "capture-0042",
     "regions": [{"id": "r1", "role": "button", "label": "Appearance", "interactive": true}],
     "history": [{"selected_id": "open-settings", "outcome": "settings window opened"}],
     "candidates": [
       {"id": "select-appearance", "description": "Click the Appearance row in the Settings sidebar."},
       {"id": "reobserve", "description": "Take a fresh observation without changing anything."},
       {"id": "abstain", "description": "Do not act; ask the person for help."}]}
    

    Pass the JSON on stdin or from a temp file. Never interpolate it into a shell string.

  5. Run exactly the one action behind selected_id. Confidence under the floor (0.65, measured — see scripts/calibrate_choose.py), or any Jev failure, comes back as reobserve. Always send regions: they are Jev's evidence the element is really on screen, and the same request scored 0.60 without them and 1.00 with them. Never derive an action from anything but the id.

  6. Observe again and verify the postcondition yourself. A chosen id, a delivered click or a screenshot is not proof. Check application state before the next step. Stop after a bounded number of steps.

Authority

Driving a GUI gives you no new permissions. Sending, publishing, paying, purchasing, deleting, changing credentials or security settings, and anything touching customer data still need the person's explicit yes, exactly as they would without a GUI. Use your driver's standard permission mode; never an approval-bypass flag. The person does all sign-ins, 2FA and payment prompts themselves.

If the driver, the key or the target is unavailable: stop and say what is missing. Do not improvise another way to control the screen.

jev choose --mock answers reobserve with no network call, for testing your loop.

Bundled runner

The loop above is the contract. scripts/jev_gui_agent.py is a working implementation of it — the desktop counterpart to jev-browser-use's runner — so you do not have to rebuild the observe/choose/act cycle by hand:

python3 <this skill>/scripts/jev_gui_agent.py \
  --pid 26955 --window-id 46041 \
  --goal 'Open the Library page in YouTube Music' \
  --expect 'Library' --max-steps 12 --json

It drives cua-driver over MCP, builds the candidate table from the accessibility tree, sends jev.action_choice_request_v1, and performs only the action behind the returned id. Exit 0 verified, 4 unverified, 2 refused to start, 6 abstained. --max-regions defaults to 26 so the table stays inside the 32-candidate contract once reobserve and abstain are added.

If the driver binary is missing or does not speak MCP, the runner prints one FAIL: line and exits 2. Set CUA_DRIVER_BIN or install the driver; do not retry the same command.

If you cannot run it, fall back to the loop above by hand — but do not fall back to AppleScript UI scripting, xdotool or coordinate clicking. Stop and say what is missing.

--plan: a multi-step command in one run

python3 <this skill>/scripts/jev_gui_agent.py --plan \
  --goal 'Open System Settings, go to General and then open About' \
  --expect 'About' --json

Use it when the person gave a spoken-style command with several steps, above all one that starts by opening an app or a site. Without it you run the loop once per hop and spend a full turn of your own composing each command: measured, the loop took 8 seconds and the agent around it took 36. With --plan the same command is one run: about 1 second to plan, then each step.

Do not use it for a single navigation goal such as "open the Library page". The plain loop is already one step there, and a plan adds a model call and sends the command to one more service for nothing. Do not use it either when each hop needs its own --expect: a plan is verified once, at the end.

What it does:

  1. One call to a small text model with reasoning switched off (JEV_PLAN_MODEL, else TEXT_MODEL, at TEXT_MODEL_BASE_URL; key from TEXT_MODEL_API_KEY or OPENROUTER_API_KEY, else the OS secret store) turns the command into ordered steps from a closed vocabulary.
  2. Steps with no on-screen target run directly: open_app (`
阅读完整来源 ↗