Grounding: observe + Set-of-Mark - Glasswarp

This example shows the grounding loop end to end: build targets, get a Set-of-Mark annotated frame, let a model pick an id, and click it.

Build targets and annotate

from glasswarp import GlasswarpClient, targets_from_grid

gw = GlasswarpClient(api_key="gw_live_sk_...")
session = gw.create_session(rig_id=rig.id, mode="desktop")
sid = session.session_id

# Grid targets over a board or canvas (or use gw.list_targets(sid) for UIA).
targets = targets_from_grid(x0=100, y0=200, x1=900, y1=800, rows=8, cols=8)

obs = gw.observe(sid, max_width=1280, mark=True, targets=targets)
with open("marked.jpg", "wb") as f:
    f.write(obs.jpeg)   # numbered marks drawn on the frame

Let the model choose, then act

# Send obs.jpeg to your model; it returns a mark id, e.g. "12".
target_id = your_model_pick(obs.jpeg)
gw.click_target(sid, target_id, targets=targets)

ROI + dirty rects for tight loops

rects = gw.dirty_rects(sid)
if rects:
    r = rects[0]
    crop = gw.screenshot(sid, x=r.x, y=r.y, w=r.w, h=r.h)
    # feed only the changed region to your model

Set-of-Mark helpers (som_annotate, targets_from_grid) need the grounding extra: pip install "glasswarp[grounding]".

Reference agent loops

minesweeper_solver_demo.py — demo with CV + solver (no LLM)
gemini_agent_loop.py — SoM → Gemini → click_target → verify
claude_agent_loop.py — observe → Claude → act (skips idle frames by default)
templates/observe_think_act/ — bare skeleton

See Minesweeper and Build an agent loop.