Grounding

UIA targets

When a Windows app exposes its UI through UI Automation, the host returns sparse click targets with roles, names, and bounds. Host v0.2.14+ also includes popup menus and combo dropdown lists (not just the focused window chrome).

targets = gw.list_targets(sid)   # [] if the app exposes none
for t in targets:
    print(t.id, t.role, t.name, t.bounds)

gw.click_target(sid, target_or_id=targets[0].id, targets=targets)  # clicks bbox center

Target ids are valid for the observe / targets response that produced them. Do not persist them across sessions, and re-observe (or call list_targets again) before acting when the screen may have changed — especially lists, file browsers, and tables. Scripts should re-resolve targets each loop, not hardcode ids.

Set-of-Mark (SoM)

Draw numbered marks on a frame so a vision model can pick a target by id instead of estimating pixels.

from glasswarp import targets_from_grid, som_annotate

# From host UIA, client CV, or a grid over a board:
targets = targets_from_grid(x0=100, y0=200, x1=900, y1=800, rows=8, cols=8)

obs = gw.observe(sid, max_width=1280, mark=True, targets=targets)
# Send obs.jpeg to your model; it returns a target id →
gw.click_target(sid, "12", targets=targets)

som_annotate and the grid helpers need the grounding extra (pip install "glasswarp[grounding]", which pulls in Pillow).

The grounding ladder

Prefer host UIA targets

Most reliable — real controls with names and bounds.

Fall back to CV / grid targets

For canvases and games that expose no UIA, build targets client-side.

Use raw coordinates last

click(x, y) still works, but avoid guessing when a target exists.

Get started

Setup

Concepts

Guides

Examples

Help

UIA targets

Set-of-Mark (SoM)

The grounding ladder

​UIA targets

​Set-of-Mark (SoM)

​The grounding ladder

UIA targets

Set-of-Mark (SoM)

The grounding ladder