Skip to main content
This example shows the grounding loop end to end: build targets, get a Set-of-Mark annotated frame, let a model pick an id, and click it.

Build targets and annotate

Let the model choose, then act

ROI + dirty rects for tight loops

Set-of-Mark helpers (som_annotate, targets_from_grid) need the grounding extra: pip install "glasswarp[grounding]".

Reference agent loops

  • gemini_agent_loop.py — CV grid → SoM targets → Gemini → click_target
  • claude_agent_loop.py — observe → Claude → act (VP0 on by default)
See Build an agent loop for the full pattern.