AI agent skill · by @vellum-ai

Computer Use

Control a connected desktop

Archived 2026-10-23 · Source updated 2026-09-17 · License: MIT. Check the original repository for the latest version.

SKILL.md instructions

This skill provides the computer_use_* action tools for controlling a
connected desktop. CU tools run through the main agent loop via HostCuProxy.

The skill is internally preactivated for conversations with a connected desktop client.

Tools in this skill are proxy tools. Execution is forwarded to a connected
desktop client and is never handled locally by the assistant.

## Observations

Every computer-use step returns the accessibility tree. Every action also
returns a screenshot taken after it ran (one at the end of a
`computer_use_sequence`), so check it to see what your action did. An
observation comes with a screenshot on a desktop's first look, when it is
window-scoped, or when you pass `include_screenshot: true`. Ask for one whenever
the tree is not enough to act on: a canvas, a game, a custom-drawn view, few or
unlabeled controls, or a layout question.

The tree is walked to a limited depth to keep steps fast, and says when it was
cut off. If the element you need is not in it, call `computer_use_observe` with
`full_tree: true`.

## Scripting apps (macOS)

Reach for `computer_use_run_applescript` first when an app can be driven by
script. It does not take the cursor.

**Try the app's own scripting dictionary before anything else.** Ask it for the
state you want, not for the clicks that would produce that state. Finder, Mail,
Music, Notes, Safari, Terminal and many third-party apps have one, and where
there is a dictionary the whole task is usually a sentence with no window to
open, no tree to read and nothing to click:

```applescript
tell application "Finder"
  set target of front window to desktop
  set current view of front window to list view
  set sort column of list view options of front window to name column
end tell
```

**System Events menu clicking is the fallback**, for apps with no dictionary
entry for what you need: `click menu item "Split Clip" of menu "Modify" of menu
bar 1` inside `tell application "System Events" to tell process "iMovie"`. It is
UI automation in a script's clothes: it still depends on the menu sitting where
you expect and on the app being frontmost. A menu item that needs a selection or
a playhead position does nothing when that context is missing, so set it up
first, and read `enabled of menu item` when unsure.

Click and type for everything a script cannot reach. `host_bash` is for shell
commands, not for driving apps.

## Typing is not sending

Pressing enter in a chat, email or form usually sends or submits it, and that
cannot be taken back. Send only when the user asked you to send, post or
submit. When they asked you to type, write or draft something, type it and stop
before pressing enter; tell them it is ready to send.

## Batching known steps (macOS)

When you already know the next few actions and none depends on seeing the
result of the one before, send them as one `computer_use_sequence` call, for
example opening a new window, typing a URL and pressing enter. Act one step at
a time whenever the next action depends on what the screen shows.

## Window-scoped observation (macOS)

`computer_use_observe` accepts optional `capture_window_id`, a native macOS
CGWindowID (not a browser tab ID or accessibility element ID). Obtain the ID
from a current native window inventory before using it; never guess one.
The native helper captures that window and its accessibility tree without
including secondary windows, even when another app covers the selected window.
A missing window must not be replaced with a desktop capture.

This is a **single observation**, not a session-wide privacy boundary: normal
click/type/scroll and other action tools still observe the whole desktop. Do not promise app-only capture for a whole control session.
The desktop must explicitly advertise `host_cu_window_capture` support on its
connection; the daemon rejects older or unsupported clients before requesting
any capture. Other desktop platforms reject this option.

The screenshot is window-relative, while action coordinates are screen points;
do not scale it using full-display dimensions. Prefer accessibility element IDs
and ensure the intended window is focused before a later action. Window-scoped
observation does not focus the window or confine subsequent input to that app.