Skip to main content
Every Glasswarp integration is the same shape. Glasswarp owns observe and act; your model owns think and verify. Open-loop scripts (hardcoded coords, “always click 400,300”) break the moment scale or layout changes — fine as helpers after vision/UIA chose a region, not as the outer brain.

The shape

Verify after each meaningful step — after a batch of predictable actions, or after a single action when the screen may change unpredictably (page loads, installers, network waits, anything that can pop a modal). You do not need a fresh observe between every click in a menu path or dialog you already planned.
decide and act are yours: decide calls your model with obs.jpeg and obs.targets; act maps its output to click_target, type_text, or a batched send_input / MCP send_actions when the next few steps are known. After a meaningful step, look again — dialogs move, folders change, and “I clicked Desktop” is not the same as “Save As is on Desktop.”

Batch when you can predict

If the next 3–6 actions are predictable (menu paths, dialog fields, typing into a field you just focused), send them as one batch, then observe once to verify the whole sequence. Single-step when intermediate state is uncertain. If the verifying observe shows something unexpected, re-plan from there. MCP clients use send_actions (verification observe on by default — text and targets; a JPEG only when you set observe_image=true). SDK clients use send_input with an ordered events list — same host path.

Make it cheap: skip idle frames

The example loops skip idle frames by default. Empty dirty_rects means “don’t spend a model call.”

Ground every action

Prefer click_target over raw pixels so the loop survives layout shifts:
Target ids are valid for the observe that produced them. Do not persist them across sessions, and re-observe before acting when lists, file browsers, or tables may have changed. Scaffolded agents must re-resolve targets each loop — never hardcode ids.

Keep task logic out of the transport

Put prompts, CV, and planning in your own module. The Glasswarp client is a thin transport adapter — mixing task logic into it makes both harder to test.

Reference loops

The SDK ships runnable agent loops:
  • templates/observe_think_act/ — bare agent-loop skeleton
  • minesweeper_solver_demo.py — demo with CV + deterministic solver
  • gemini_agent_loop.py — demo with Gemini + SoM → click_target
  • claude_agent_loop.py — observe → Claude → act (skips idle frames by default)
  • paint_mona_lisa_demo.py — full demo on a real app (place → paint → Save As)
See Templates, Minesweeper, and Mona Lisa. Before you turn autonomy up, read Safe agent sessions — budgets, human-in-the-loop review, and allowlists on top of platform consent and Live View.