The shape
decide and act are yours: decide calls your model with obs.jpeg and
obs.targets; act maps its output to click_target, type_text, or a
batched send_input / MCP send_actions when the next few steps are known.
After a meaningful step, look again — dialogs move, folders change, and
“I clicked Desktop” is not the same as “Save As is on Desktop.”
Batch when you can predict
If the next 3–6 actions are predictable (menu paths, dialog fields, typing into a field you just focused), send them as one batch, then observe once to verify the whole sequence. Single-step when intermediate state is uncertain. If the verifying observe shows something unexpected, re-plan from there. MCP clients usesend_actions (verification observe on by default — text and
targets; a JPEG only when you set observe_image=true). SDK clients use
send_input with an ordered events list — same host path.
Make it cheap: skip idle frames
Ground every action
Preferclick_target over raw pixels so the loop survives layout shifts:
Keep task logic out of the transport
Put prompts, CV, and planning in your own module. The Glasswarp client is a
thin transport adapter — mixing task logic into it makes both harder to test.
Reference loops
The SDK ships runnable agent loops:templates/observe_think_act/— bare agent-loop skeletonminesweeper_solver_demo.py— demo with CV + deterministic solvergemini_agent_loop.py— demo with Gemini + SoM →click_targetclaude_agent_loop.py— observe → Claude → act (skips idle frames by default)paint_mona_lisa_demo.py— full demo on a real app (place → paint → Save As)
