跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

operator

Drive any Windows GUI application on the owner's real PC and PROVE what was done. A "fake human" that opens apps, reads their UI Automation tree, clicks/types/scrolls through them, and journals every action frame-bound and replayable — the desktop generalization of the playtester skill (Unity via injected C# seams) to "any app via UIA + pixels". Triggers when the owner says "operate <app>", "drive <app> and test X", "open <app> and do Y", "computer-use this", or points at the .Codex/tools/operator toolkit. Owner-present/supervised sessions only in OP-A (host lane, no arming). SEAM-FIRST: if a programmatic API/MCP/headless recipe can do the step, use THAT, not pixels.

DITlieD · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add DITlieD/ELAI-archive --skill operator

技能说明


name: operator description: Drive any Windows GUI application on the owner's real PC and PROVE what was done. A "fake human" that opens apps, reads their UI Automation tree, clicks/types/scrolls through them, and journals every action frame-bound and replayable — the desktop generalization of the playtester skill (Unity via injected C# seams) to "any app via UIA + pixels". Triggers when the owner says "operate ", "drive and test X", "open and do Y", "computer-use this", or points at the .Codex/tools/operator toolkit. Owner-present/supervised sessions only in OP-A (host lane, no arming). SEAM-FIRST: if a programmatic API/MCP/headless recipe can do the step, use THAT, not pixels.

OPERATOR (frame-bound computer-use for the real Windows host)

WHAT THIS IS A loop where the model drives a real Windows application through its UI Automation tree + screen pixels, and every actuation is bound to the exact observation it was decided from (frame binding, fail-closed). It replaces "I clicked around and it looked fine" with a journal of intent -> frame-hash-before -> action -> frame-hash-after -> result rows plus before/after frames and crosshair evidence crops that an auditor can re-hash without trusting this session's narration. Same discipline as the playtester skill, generalized off Unity onto any GUI app.

SEAM-FIRST RULE (doctrine #1, non-negotiable) Before driving a single pixel, check for a PROGRAMMATIC SEAM. Pixel-driving a step a seam could have done is a FINDING, not a success (a "wrong-lane" finding: record it and reroute). Routing table (pack docs under .Codex/tools/operator/packs/ are the per-app seam inventory + GUI residue — read the named pack before driving that app): Unity -> unity-pack (packs/unity-pack.md): Unity MCP / unityMCP tools + batch_execute; playtester loop; GUI ONLY for native OS dialogs (Import Package / Save As / Preferences residue). IMGUI panes are UIA-dark — refuse GUI for scene/Inspector/Hierarchy content. MULTISEAT HARD: Hub uses machine-wide pipe Unity-hubIPCService — never kill Editor to free Hub for another seat; only one Hub on the PC at a time; project lists are per Windows user (projects-v1.json). Blender -> blender-pack (packs/blender-pack.md): blender --background --python <recipe>.py headless bpy first; catalog id blender / winget BlenderFoundation.Blender. GUI residue = addon dialogs / eyeball QC only. CAD -> cad-pack (packs/cad-pack.md): CadQuery / FreeCAD python console first; Fusion MCP = RECON_NEEDED until vendor-verified. GUI residue = sketch/constraint dialogs. no named seam -> generic-playbook (packs/generic-playbook.md): exploration protocol for Paint-class / unknown apps (UIA-rich → PATH-1; UIA-dark content → do not thrash). web / browser -> Playwright / CDP (AXTree-first) a CLI/API app -> call the CLI/API directly The GUI operator lane exists ONLY for the residue: GUI-only apps (no API), editor dialogs no MCP reaches, installers, and visual verification. Always open the named pack above before actuating that app; pack absence for a new app means write/extend a pack (docs+catalog row), do not invent a fork of op_*.py.

PRECONDITIONS (verify before starting; fail loud if missing)

  • OWNER-PRESENT session by default (OP-A host lane; --mode shared). Unattended full-control driving is --mode exclusive (owner away/asleep). Destructive action classes are DARK in OP-A in BOTH modes (see below); mode never relaxes that gate.
  • The venv exists: F:.ELAI_workflow.Codex\tools\operator.venv (python -m venv; deps in requirements.txt). Run every tool with that interpreter: set PY=F:.ELAI_workflow.Codex\tools\operator.venv\Scripts\python.exe and run the tools from their own directory (F:.ELAI_workflow.Codex\tools\operator) so op_common/op_uia import.
  • FOREGROUND is required only for the COORDINATE FALLBACK path (PATH 2). The PRIMARY path is UIA-pattern actuation (PATH 1) which needs NO foreground and NO mouse, so it drives an app while the owner uses a different window. Prefer an element id that carries a usable pattern (snapshot.json lists each element's actionable patterns); only pixel-only / no-pattern targets need the target foregrounded, and PATH 2 refuses FOREGROUND_MISMATCH rather than type into the wrong window.

THE TOOLKIT (every primitive; the skill IS the production entrypoint — a tool this skill never names is wire-dark) All under F:.ELAI_workflow.Codex\tools\operator\ :

  • op_run.py — run lifecycle. op_run.py new --slug -> prints a run dir under workflow/operator-runs/{ts}-{slug}/ (journal.jsonl, frames/, crops/, report.md, run.json). Capture that path; every other tool takes --run . op_run.py record-start --run [--region L,T,W,H] [--max-seconds N] -> start a bounded ffmpeg gdigrab recording. OFF by default; use ONLY on armed destructive actions (OP-B). Evidence posture in OP-A is before/after frames + crops, not video. op_run.py record-stop --run op_run.py finalize --run [--crystallize] -> writes the findings report (action table + summary) from the journal. With --crystallize (OP-E1) ALSO mints replay.json + manifest.json from OK journal rows. op_run.py crystallize --run -> OP-E1 alone: mint replay.json + manifest.json (no report rewrite). Equivalent to op_record.py crystallize --run .
  • op_record.py crystallize --run — OP-E1 recorder (also called from finalize --crystallize). Reads journal.jsonl + last snapshot.json; mints: replay.json ordered playable steps (UIA selector = role+name+runtime_id first; coords+frame-hash fallback; assertions from journal deltas; class/mode/destructive flag per step). Skips REFUSED/PRE_IMAGE rows; de-dupes menu-open when a following menu-select shares the parent. manifest.json monitor geometry, tool versions, catalog blake3, app window titles, taskspec_shape block for later OP-F. Schema version op-
阅读完整来源 ↗