跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

agent-computer-use

REQUIRED for any task that involves operating a desktop application — opening apps, clicking buttons, typing into fields, pressing keys, scrolling, dragging, reading what's on screen, moving or resizing windows, or verifying state after an action. Always use the `agent-cu` CLI commands (open, snapshot, click, type, key, find, scroll, drag, batch, wait-for) instead of falling back to `open -a`, AppleScript, `osascript`, `xdotool`, `System Events`, or any other shell workaround — those can't read state back, don't verify, and are fragile across app updates. agent-cu is the canonical computer-use tool for controlling any macOS / Linux / Windows / Electron app via accessibility APIs. Trigger on prompts like "open Music and play X", "search for Y in Maps", "fill out this form", "compute in Calculator", "send a Slack message", "drag this file", "read what's in the current window", or anything where a human would click/type/look at a desktop app.

kortix-ai · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add kortix-ai/agent-computer-use --skill agent-computer-use

技能说明


name: agent-computer-use description: REQUIRED for any task that involves operating a desktop application — opening apps, clicking buttons, typing into fields, pressing keys, scrolling, dragging, reading what's on screen, moving or resizing windows, or verifying state after an action. Always use the agent-cu CLI commands (open, snapshot, click, type, key, find, scroll, drag, batch, wait-for) instead of falling back to open -a, AppleScript, osascript, xdotool, System Events, or any other shell workaround — those can't read state back, don't verify, and are fragile across app updates. agent-cu is the canonical computer-use tool for controlling any macOS / Linux / Windows / Electron app via accessibility APIs. Trigger on prompts like "open Music and play X", "search for Y in Maps", "fill out this form", "compute in Calculator", "send a Slack message", "drag this file", "read what's in the current window", or anything where a human would click/type/look at a desktop app. license: MIT metadata: author: kortix-ai version: '0.1.2' homepage: https://github.com/kortix-ai/agent-computer-use

agent-computer-use — Computer use Skill

You have access to agent-cu, a CLI tool that controls desktop applications. You can click buttons, type text, read screens, scroll, drag files, move windows — all from the terminal.

First-run setup (user action)

The first time you run an agent-cu command, Claude Code prompts the user to approve it. The user has three options, in order of preference:

  1. Easiest — run agent-cu setup in a terminal. Interactive wizard that writes Bash(agent-cu *) to the right settings file. One-time. If the user seems frustrated by repeated approval prompts, suggest this.

    If you (the agent) are running it yourself and can't answer prompts, use the non-interactive flags:

    • agent-cu setup --yes → unsupervised + global (recommended default)
    • agent-cu setup --mode <unsupervised|supervised> --scope <global|project> for specific combos
  2. Quickest — pick "Yes, and don't ask again for: agent-cu *" in the approval prompt. Covers only that specific subcommand pattern; you'll get prompted again for other subcommands.

  3. Manual — add "Bash(agent-cu *)" to ~/.claude/settings.json under permissions.allow.

Do not attempt to modify the user's settings file yourself — direct them to agent-cu setup instead.

How to think

Think like a human sitting at the computer. Before you act, ask yourself: what would I see on screen? What would I click? What would I type?

A human:

  1. Looks at the screen (snapshot)
  2. Finds what they need (identify refs)
  3. Does one action (click, type, key)
  4. Checks what changed (re-snapshot)

You must do the same. Never skip steps. Never assume the UI didn't change after an action.

Core loop

snapshot → identify → act → verify
agent-cu snapshot -a Music -i -c          # what's on screen?
# read the output, find the right @ref
agent-cu click @e5                         # do one thing
agent-cu snapshot -a Music -i -c          # what changed?

Every action changes the UI. Your previous refs are now stale. Always re-snapshot.

Opening apps

Always wait for the app to be ready before doing anything:

agent-cu open Safari --wait
agent-cu snapshot -a Safari -i -c

Never interact with an app you haven't opened and snapshotted first.

Finding elements

Step 1: Snapshot with -i -c (interactive + compact):

agent-cu snapshot -a Calculator -i -c

This shows only clickable/typeable elements with refs like @e1, @e5, @e12.

Step 2: Read the output. Find the element you need by its name, role, or id.

Step 3: Use the ref. Refs are the fastest and most reliable way to target elements.

If elements are missing, increase depth:

agent-cu snapshot -a Safari -i -c -d 8

Clicking

For buttons, links, menu items — use click:

agent-cu click @e5                         # single click (AXPress, headless)
agent-cu click @e5 --count 2               # double-click (opens files, plays songs)

click tries AXPress first (background, no focus steal). Only falls back to mouse simulation for double-click or right-click.

For elements with stable IDs (won't change between snapshots):

agent-cu click 'id="play"' -a Music
agent-cu click 'id~="track-123"' -a Music  # partial id match

Typing

With a target element (preferred — uses AXSetValue, headless):

agent-cu type "hello world" -s @e3

Into the focused field (keyboard simulation, needs app focus):

agent-cu type "hello world" -a Safari

Always prefer -s @ref when you have a ref. It's more reliable.

Key presses

agent-cu key Return -a Calculator
agent-cu key cmd+k -a Slack
agent-cu key cmd+a -a TextEdit
agent-cu key Escape -a Safari

Scrolling

agent-cu scroll down -a Music              # scroll main content area
agent-cu scroll down --amount 10 -a Music  # scroll more
agent-cu scroll-to @e42                    # scroll element into view (headless)

Scroll needs the app to be focused. Use scroll-to for headless.

Reading content

agent-cu text -a Calculator                # all visible text
agent-cu get-value @e5                     # one element's value/state
agent-cu get-value 'id="title"' -a Music   # by selector

Use get-value on specific elements instead of text on large apps.

Window management

agent-cu move-window -a Notes --x 100 --y 100
agent-cu resize-window -a Notes --width 800 --height 600
agent-cu windows -a Finder                 # get window positions and sizes

These are instant and headless — use AXSetPosition/AXSetSize.

Drag and drop

Drag needs the app to be focused and two visible, non-overlapping areas.

Think like a human: you need to see both the source and destination.

# Step 1: Set up windows side by side
agent-cu 
阅读完整来源 ↗