AI agent skill · by @A-EVO-Lab

Screenshot Analysis

How to interpret accessibility tree elements and correlate them with screenshot regions for accurate GUI interaction.

Archived 2026-10-23 · Source updated 2026-08-22. Check the original repository for the latest version.

SKILL.md instructions

# Screenshot & Accessibility Tree Analysis

## Reading the Accessibility Tree
The a11y_tree is your primary guide for finding interactive elements:
```
[role=window] "Document - LibreOffice Calc"
  [role=menubar]
    [role=menuitem] "File"
    [role=menuitem] "Edit"
  [role=toolbar]
    [role=button] "Save" (x=120, y=45)
  [role=table] "Sheet"
    [role=cell] "A1" (x=80, y=200)
```

## Coordinate mapping
- a11y_tree provides (x, y) center coordinates for each element
- Use these for pyautogui.click(x, y)
- If coordinates seem wrong, the element may be scrolled off-screen

## Element roles cheat sheet
- `button`: clickable action (Save, OK, Cancel)
- `menuitem`: menu entry (click to open submenu or execute)
- `textbox`/`text`: editable text field (click then type)
- `combobox`: dropdown (click to open, then select)
- `checkbox`: toggle (click to check/uncheck)
- `cell`: spreadsheet cell (click to select, then type)
- `tab`: tab selector (click to switch)
- `dialog`: modal window (must handle before other actions)

## When a11y_tree is insufficient
- Some custom widgets don't expose a11y info — use screenshot + coordinates
- For canvas-based apps (GIMP): rely on menu navigation, not direct canvas clicks
- For web content in Chrome: a11y_tree reflects DOM, usually reliable

## Tips
- Always check a11y_tree AFTER an action to verify the state changed
- If an element disappears from a11y_tree, the UI updated (menu closed, dialog dismissed)
- Count elements to verify you're clicking the right one (e.g., 3rd button in toolbar)