Grounding

The difference between a reliable agent and a flaky one is grounding: acting on structured targets instead of guessed coordinates. VNC only sees pixels; Glasswarp surfaces real UI structure when the app exposes it.

UIA targets

When a Windows app exposes its UI through UI Automation, the host returns sparse click targets with roles, names, and bounds.

targets = gw.list_targets(sid)   # [] if the app exposes none
for t in targets:
    print(t.id, t.role, t.name, t.bounds)

gw.click_target(sid, target_or_id="12", targets=targets)  # clicks bbox center

Set-of-Mark (SoM)

Draw numbered marks on a frame so a vision model can pick a target by id instead of estimating pixels.

from glasswarp import targets_from_grid, som_annotate

# From host UIA, client CV, or a grid over a board:
targets = targets_from_grid(x0=100, y0=200, x1=900, y1=800, rows=8, cols=8)

obs = gw.observe(sid, max_width=1280, mark=True, targets=targets)
# Send obs.jpeg to your model; it returns a target id →
gw.click_target(sid, "12", targets=targets)

som_annotate and the grid helpers need the grounding extra (pip install "glasswarp[grounding]", which pulls in Pillow).

The grounding ladder

Prefer host UIA targets

Most reliable — real controls with names and bounds.

Fall back to CV / grid targets

For canvases and games that expose no UIA, build targets client-side.

Use raw coordinates last

click(x, y) still works, but avoid guessing when a target exists.

Continue to Safety and consent.

Vision Safety and consent

⌘I

Get started

Setup

Concepts

Guides

Examples

Help

UIA targets

Set-of-Mark (SoM)

The grounding ladder

​UIA targets

​Set-of-Mark (SoM)

​The grounding ladder

UIA targets

Set-of-Mark (SoM)

The grounding ladder