Chat-driven (MCP)
Your assistant drives the PC step by step through MCP tools: it captures the screen, decides, clicks or types, and captures again. This is great for quick tasks and supervision — “what’s on my screen?”, “open Notepad and type hello”, approving a dialog while you watch. The tradeoff: each step costs a model turn. Predictable sequences — menus, dialogs, typing into a field you just clicked — can be batched into one call (send_actions), so you are not paying a turn per keystroke. MCP defaults to
text+targets observes (set image=true / observe_image=true only when you need
vision). Adaptive loops where every move depends on new information — clearing a
board, anything that repeats — still add up, and that is what the SDK path is for.
Setup: Connect via MCP.
Scaffolded SDK agents
For multi-step work, a small Python script loopsobserve → decide → act at
full speed. The decision logic lives inside the script — heuristics, computer
vision, or an LLM call — so there’s no chat round-trip per step. This is the
right tool for games, workflows, and anything that repeats.
If you’re connected through a code-capable assistant, it will typically offer to
write this script for you; you can also start from the
SDK quickstart or the agent loop guide.
Pre-built demos
Some pre-built demo agents ship with the SDK — install and run, no code to write:
MCP clients can browse the same catalog with the
list_demos / get_demo
tools, which return the install command and the exact command to run.