agent-computer-use
REQUIRED for any task that involves operating a desktop application — opening apps, clicking buttons, typing into fields, pressing keys, scrolling, dragging, reading what's on screen, moving or resizing windows, or verifying state after an action. Always use the `agent-cu` CLI commands (open, snapshot, click, type, key, find, scroll, drag, batch, wait-for) instead of falling back to `open -a`, AppleScript, `osascript`, `xdotool`, `System Events`, or any other shell workaround — those can't read state back, don't verify, and are fragile across app updates. agent-cu is the canonical computer-use tool for controlling any macOS / Linux / Windows / Electron app via accessibility APIs. Trigger on prompts like "open Music and play X", "search for Y in Maps", "fill out this form", "compute in Calculator", "send a Slack message", "drag this file", "read what's in the current window", or anything where a human would click/type/look at a desktop app.
kortix-ai · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add kortix-ai/agent-computer-use --skill agent-computer-useAbout this skill
name: agent-computer-use
description: REQUIRED for any task that involves operating a desktop application — opening apps, clicking buttons, typing into fields, pressing keys, scrolling, dragging, reading what's on screen, moving or resizing windows, or verifying state after an action. Always use the agent-cu CLI commands (open, snapshot, click, type, key, find, scroll, drag, batch, wait-for) instead of falling back to open -a, AppleScript, osascript, xdotool, System Events, or any other shell workaround — those can't read state back, don't verify, and are fragile across app updates. agent-cu is the canonical computer-use tool for controlling any macOS / Linux / Windows / Electron app via accessibility APIs. Trigger on prompts like "open Music and play X", "search for Y in Maps", "fill out this form", "compute in Calculator", "send a Slack message", "drag this file", "read what's in the current window", or anything where a human would click/type/look at a desktop app.
license: MIT
metadata:
author: kortix-ai
version: '0.1.2'
homepage: https://github.com/kortix-ai/agent-computer-use
agent-computer-use — Computer use Skill
You have access to agent-cu, a CLI tool that controls desktop applications. You can click buttons, type text, read screens, scroll, drag files, move windows — all from the terminal.
First-run setup (user action)
The first time you run an agent-cu command, Claude Code prompts the user to approve it. The user has three options, in order of preference:
-
Easiest — run
agent-cu setupin a terminal. Interactive wizard that writesBash(agent-cu *)to the right settings file. One-time. If the user seems frustrated by repeated approval prompts, suggest this.If you (the agent) are running it yourself and can't answer prompts, use the non-interactive flags:
agent-cu setup --yes→ unsupervised + global (recommended default)agent-cu setup --mode <unsupervised|supervised> --scope <global|project>for specific combos
-
Quickest — pick "Yes, and don't ask again for:
agent-cu *" in the approval prompt. Covers only that specific subcommand pattern; you'll get prompted again for other subcommands. -
Manual — add
"Bash(agent-cu *)"to~/.claude/settings.jsonunderpermissions.allow.
Do not attempt to modify the user's settings file yourself — direct them to agent-cu setup instead.
How to think
Think like a human sitting at the computer. Before you act, ask yourself: what would I see on screen? What would I click? What would I type?
A human:
- Looks at the screen (snapshot)
- Finds what they need (identify refs)
- Does one action (click, type, key)
- Checks what changed (re-snapshot)
You must do the same. Never skip steps. Never assume the UI didn't change after an action.
Core loop
snapshot → identify → act → verify
agent-cu snapshot -a Music -i -c # what's on screen?
# read the output, find the right @ref
agent-cu click @e5 # do one thing
agent-cu snapshot -a Music -i -c # what changed?
Every action changes the UI. Your previous refs are now stale. Always re-snapshot.
Opening apps
Always wait for the app to be ready before doing anything:
agent-cu open Safari --wait
agent-cu snapshot -a Safari -i -c
Never interact with an app you haven't opened and snapshotted first.
Finding elements
Step 1: Snapshot with -i -c (interactive + compact):
agent-cu snapshot -a Calculator -i -c
This shows only clickable/typeable elements with refs like @e1, @e5, @e12.
Step 2: Read the output. Find the element you need by its name, role, or id.
Step 3: Use the ref. Refs are the fastest and most reliable way to target elements.
If elements are missing, increase depth:
agent-cu snapshot -a Safari -i -c -d 8
Clicking
For buttons, links, menu items — use click:
agent-cu click @e5 # single click (AXPress, headless)
agent-cu click @e5 --count 2 # double-click (opens files, plays songs)
click tries AXPress first (background, no focus steal). Only falls back to mouse simulation for double-click or right-click.
For elements with stable IDs (won't change between snapshots):
agent-cu click 'id="play"' -a Music
agent-cu click 'id~="track-123"' -a Music # partial id match
Typing
With a target element (preferred — uses AXSetValue, headless):
agent-cu type "hello world" -s @e3
Into the focused field (keyboard simulation, needs app focus):
agent-cu type "hello world" -a Safari
Always prefer -s @ref when you have a ref. It's more reliable.
Key presses
agent-cu key Return -a Calculator
agent-cu key cmd+k -a Slack
agent-cu key cmd+a -a TextEdit
agent-cu key Escape -a Safari
Scrolling
agent-cu scroll down -a Music # scroll main content area
agent-cu scroll down --amount 10 -a Music # scroll more
agent-cu scroll-to @e42 # scroll element into view (headless)
Scroll needs the app to be focused. Use scroll-to for headless.
Reading content
agent-cu text -a Calculator # all visible text
agent-cu get-value @e5 # one element's value/state
agent-cu get-value 'id="title"' -a Music # by selector
Use get-value on specific elements instead of text on large apps.
Window management
agent-cu move-window -a Notes --x 100 --y 100
agent-cu resize-window -a Notes --width 800 --height 600
agent-cu windows -a Finder # get window positions and sizes
These are instant and headless — use AXSetPosition/AXSetSize.
Drag and drop
Drag needs the app to be focused and two visible, non-overlapping areas.
Think like a human: you need to see both the source and destination.
# Step 1: Set up windows side by side
agent-cu
Read the complete source ↗