Minion

Mouse and keyboard. Runs the actions in order, stops at the first failure, waits for the screen to settle and reports what changed. Actions (objects with an `action` field): click | double_click | triple_click (select a line) | right_click | middle_click (target; modifiers "ctrl"), move (target), drag (from, to), type (text; \n = Enter, \t = Tab), paste (text), fill (target, text: replace a field's content), select (target, option: pick a dropdown option by its text), key (keys: "ctrl+s", "Return", or a sequence "Tab Tab Return"; repeat), scroll (direction, amount, target), wait (seconds), wait_for (text, timeout; gone=true), scroll_to (text: brings it into view wherever it is on a long page, loading more of a feed), focus (window), close (window, like its X button), copy (target: its whole text to the clipboard), goto (target: a URL in the browser's current tab), menu (item: a menu path like "File > Export as PDF"; see(menus=true) lists them), path (path: fill the open Open/Save/Export dialog with a full file path and confirm; overwrite=true replaces a file), mouse_down/mouse_up (target: press/release for custom drags), hold_key (keys, seconds). Targets: "@12" from your latest see (best), exact visible text ("Save"; fails if not on screen), a kind and a label ("button:Save", "link:Docs", "field:Email", "tab:Settings"), or "x,y". The reply says what changed: windows, a page's alert/confirm dialog, files saved or downloaded; it waits while the app shows "Loading…". A page may still fill in after a click: wait_for text you expect.

Create and manage Linux desktops (isolated computers with GUI apps). Actions: create (profile, name, size, count for several at once), list, profiles, remove, freeze (0% CPU; any later call wakes it), thaw, fork (clone disk state into a new desktop to try something risky), snapshot, restore, replay (HTML timeline of everything done), macro (the actions so far as a Python script that replays them with the script tool, no model needed), logins (sign-ins the human saved: create with login=NAME starts signed in). Profiles: browser (Firefox, blocks ads and cookie banners), office (LibreOffice + Firefox), dev (git, Python, Node.js, Chromium, IDE), research (Firefox + PDF/notes tools), data (office + pandas/matplotlib), windows (Windows .exe/.msi programs via Wine, x86-64 hosts), base (minimal).

Files in a desktop. read: any file as text: text files as they are, documents converted (docx, xlsx with values and formulas, pptx, odt/ods/odp, pdf, epub, html, images by OCR, doc/xls/ppt). convert: path -> another format (to: pdf, docx, xlsx, csv, pptx, png, md, ...; returns the new path). write: text -> path (replaces the file; creates folders). upload: local_path (on the machine running Minion) -> path. download: path -> local_path (never overwrites unless overwrite=true). local_path is limited to Minion's files folder (relative paths start there). Desktop paths may start with ~.

Open something and wait until its window is ready: an app ("writer", "calc", "firefox", "terminal", or any app by name: "gimp", "inkscape", "vscode", "chrome", "thunderbird", "blender", "dbeaver"... installed the first time, which can take a minute), a web app by name ("gmail", "figma", "google sheets"), a URL ("github.com"), a file in the desktop ("~/Documents/report.docx"), or a shell command. The reply includes short tips for the app the first time.

Run a bash command inside the desktop (as the desktop user; sudo works) and get stdout, stderr and the exit code. Best for files, data, git, conversions. Install anything with `minion-install <name or URL>` (apt packages, .deb/.AppImage/archive URLs, pip:pkg, npm:pkg). background=true for servers.

Look at the screen (PNG). Image pixels are desktop coordinates unless scaled or cropped (the reply explains the mapping). marks=true draws @ids on clickable elements. detail="low" halves each side (about 4x fewer image tokens). region=[x,y,w,h] crops; zoom=true magnifies the region.

Run Python inside the desktop to do many GUI steps in one call (loops, retries, data entry): only what it prints comes back. Preloaded: see(changes=False), elements(role, name), find(text), cells() (visible spreadsheet cells {"A1": ...}), click(target), double_click, right_click, write(text), paste(text), key(keys), scroll(dir, n), fill(target, text), drag(a, b), wait(s), wait_for(text, timeout) -> bool, menu("File > Save"), menus(), path(file), read(file) -> text, convert(file, to) -> path, launch(app_or_url), run(cmd) -> dict, windows(), focus(title), scroll_to(text), close(window), copy(target), clipboard(text=None). Targets as in act. Errors raise MinionScriptError.

Read the screen as text: windows, then the focused window's UI elements with stable @ids, roles, names, values, states and center coordinates (accessibility tree). Text that accessibility doesn't have (an app without it, slides, canvases, pictures of text, remote screens) is read with OCR in the screen's language. Exact and far cheaper than a screenshot. changes=true returns only what changed since your last look.

A live link where the human can watch the desktop and take over mouse and keyboard. With message, hand control to the human ("Please sign in to your bank") and wait until they press Done (up to timeout seconds); returns their note.