Ui.Vision MCP

Reports whether the Ui.Vision browser extension is currently connected to this bridge. Call this first if other tools fail — it tells you whether the extension side panel is open and the bridge is enabled in its settings. Several browsers can be connected at once; the status lists each one by label (chrome#1, firefox#1, ...) and marks the ACTIVE one that tool calls go to.

Returns the structure of the browser tab Ui.Vision plays in. Default mode "tree": the page as an accessibility tree — one line per node with role, accessible name and state (value, checked, expanded, disabled, offscreen), nested by container. Every interactive node carries [ref=N], and ref=N is a LOCATOR: act on it right away with uiv.page.click('ref=12'), uiv.page.fill('ref=7', text) or uiv.$('ref=12') (match object, e.g. uiv.browser.click(uiv.$('ref=12'))). Refs are numbered per page load and renumbered on navigation: call browser_snapshot again after uiv.goto or a navigating click. mode "fields": the classic list of form fields, buttons and links with id=/name=/css= locators plus refs. Pass "url" to open that page first; omit it to inspect the current tab. Cannot see into closed shadow roots; cross-origin iframes are listed separately.

EXPLORATION ONLY: click the point (x, y) read off the MOST RECENT screenshot — the extension converts the picture's pixels to the page or screen itself, so "click the thing in the picture" is one call. A browser screenshot clicks in the tab (trusted click, any frame); a desktop screenshot clicks on the screen through the XModule. Returns what changed on the page afterwards (like run_macro look: "delta"). A point is NOT a locator: for a macro that will be saved, act by ref/locator (browser_snapshot) or by a saved element image instead.

Create a NEW macro from the given Ui.Vision JSON and save it in the "AI Generated" folder under a new, unique name. PREFER the JS script form {"Name": "...", "Script": "<JavaScript using the uiv.* API>"} over the classic Commands table — scripts are modern JS with real control flow and are the recommended way to build macros. Call get_authoring_guide for the uiv.* API before writing your first script. The new macro opens in the editor — refine it afterwards with set_macro. Returns the final macro name.

Delete a macro from the "AI Generated" folder — cleanup for scratch and test macros this or an earlier session created there. Only macros inside "AI Generated" can be deleted; everything else is refused (the user deletes those in the panel). Returns a confirmation.

Reports the ACTIVE AI provider/model as the chat will resolve it, which providers have keys stored (booleans only — keys never cross the bridge), the per-provider model settings, computer-vision scope and OCR engine. Call before set_ai_settings to plan a change.

Returns the Ui.Vision macro authoring guide: the uiv.* JavaScript API for JS script macros (finders, actions, OCR, vision), the classic command table, locator rules and best practices. CALL THIS ONCE BEFORE WRITING YOUR FIRST MACRO in a session — macros written without it usually use invented APIs and fail.

Returns the in-panel AI Chat transcript and whether a run is in progress (running:true/false). Poll this after send_chat; the transcript is text-only (inline images are omitted).

Returns the macro currently loaded in the Ui.Vision editor, as Ui.Vision JSON. A JS script macro comes back with a "Script" field (its program) instead of Commands.

Alias of browser_snapshot (the older Ui.Vision name) — identical behaviour. Returns the structure of the browser tab Ui.Vision plays in. Default mode "tree": the page as an accessibility tree — one line per node with role, accessible name and state (value, checked, expanded, disabled, offscreen), nested by container. Every interactive node carries [ref=N], and ref=N is a LOCATOR: act on it right away with uiv.page.click('ref=12'), uiv.page.fill('ref=7', text) or uiv.$('ref=12') (match object, e.g. uiv.browser.click(uiv.$('ref=12'))). Refs are numbered per page load and renumbered on navigation: call browser_snapshot again after uiv.goto or a navigating click. mode "fields": the classic list of form fields, buttons and links with id=/name=/css= locators plus refs. Pass "url" to open that page first; omit it to inspect the current tab. Cannot see into closed shadow roots; cross-origin iframes are listed separately.

Lists all macros stored in Ui.Vision (name and id). Use open_macro to load one into the editor before reading or running it.

Opens the macro with the given name in the Ui.Vision editor and returns its JSON. Subsequent get_macro/set_macro/run_macro calls operate on it. Note: unsaved editor changes may be discarded.

Opens the Ui.Vision panel when the extension is NOT connected: signals the browser over a background wake channel (extension 10.0.170+) to open the Ui.Vision app in a tab. This is the ONLY way the panel opens without a human click — the extension never opens it by itself. Works when the panel or its tab was closed, and even when the browser sits in the tray with zero windows (a window is created). Call this whenever bridge_status shows no tool connection but a wake channel; the panel connects within a few seconds. Pass target ("chrome", "firefox") to wake one browser family; omit to wake every browser that is not yet connected. If it reports no wake channel, the browser process is not running (or the extension predates 10.0.170) — a human must start the browser, then retry.

Reloads the Ui.Vision extension from disk (same as the chrome://extensions reload button) — the way to load a freshly built version without a human click. The bridge connection dies with the panel; on startup the panel app reopens as a TAB and reconnects, and the new hello reports the loaded version. If no reconnect arrives within ~15s the new build failed to start and needs a human at chrome://extensions.

Run a macro against the browser tab and wait for it to finish (can take minutes). Three forms: pass "macro" (a name from list_macros) to open and run that stored macro; pass "script" (inline JavaScript) to run code directly without creating a macro; pass neither to run whatever is currently open in the editor — that last form depends on editor state, so prefer naming the target. Returns the execution log, including the error and failing line if it fails, plus a picture of the page/screen after the run with ONLY the macro's own actions and finder results marked: arrows at its clicks, numbered boxes at image/OCR matches (each number resolves to a legend line with exact coordinates), blue search-area rectangles, a magnified inset of the last click. No other overlay is painted — read what OCR recognised from the text legend, not the picture.

Runs every self-test of Settings > Desktop Automation in one go: native host connection, WASM image-search engine, desktop screenshot, real mouse input delivery, home-directory read/write, visual compatibility (colors, scaling, screenshot coordinates) and a test macro inside the desktop app - and returns ONE report that starts with "SELFTEST: ALL OK" or "SELFTEST: PROBLEMS FOUND", then one line per test with verdict and detail. Call it FIRST when a macro misbehaves (clicks land elsewhere, colors or images are not found, nothing happens on the desktop): it separates an environment problem from a macro problem in about half a minute. The Settings page opens as a tab while the tests run (the visual test shows a chart, the input test moves the mouse - hands off) and closes again unless keep_open. Pass tests to run a subset.

What the macro runner is doing RIGHT NOW: whether a run is in progress, which macro the editor holds, and the freshest log lines. Unlike every other tool, this ANSWERS WHILE run_macro is still executing (extension 10.0.172+) — poll it from a second client/session to watch a long run live instead of waiting blind. screenshot is likewise answerable mid-run for a visual peek.

Crop a rectangle from the MOST RECENT screenshot and save it as a Ui.Vision vision image, for image-based commands like "XClick | Target: <name>.png" or uiv.findImage('<name>.png') in a JS script. Coordinates are absolute pixels in the last screenshot (or in the last magnified view, which converts back itself — never scale by hand). SIZING: crop a normal control tightly with a few px of margin, but do NOT crop a SMALL or FLAT target tight — the matcher keys on structure, and a plain patch matches everywhere while one icon in a row of icons looks like all its neighbours. For those, take a WIDER box with the target EXACTLY CENTRED, including stable surroundings (for a toolbar icon: the icons to its left and right). Clicks still land on the target, because the finders return the match CENTRE. Returns the saved file name plus a picture of the SURROUNDINGS with a red box around the saved area, so you can check the crop against its neighbours, and warns when the crop is too small or too featureless to match reliably. Re-crop with THE SAME NAME to overwrite.

Create a green/pink RELATIVE vision image from the MOST RECENT screenshot, for commands like "BClickRelative | Target: <name>.png". The green box marks the anchor element that is searched on the page; the pink box marks where to click, relative to the anchor. Coordinates are absolute pixels in the last screenshot.

Returns a screenshot of the visible part of the browser tab Ui.Vision plays in. Coordinates passed to save_element_image / save_relative_image are absolute pixels in this image. scope: "desktop" captures the WHOLE SCREEN instead (needs the RealUser XModule) — the way to see browser UI outside the page (toolbar, extension popups, native dialogs) and to verify desktop automation. ZOOM: pass x/y/width/height (all four) to get a MAGNIFIED view of that region of the screenshot you already have, instead of a new capture. A full-screen capture is shrunk to fit, leaving a toolbar icon or checkbox a few pixels wide — too coarse to place a crop box on, and guessing one there picks the NEIGHBOURING element about as often as the right one. Zoom first whenever the target is under ~40 px, then read coordinates off the magnified view; it becomes the coordinate frame for the next save_element_image call. Regions nest; a call with no region returns to the full view. On the very first screenshot a region is skipped (there is nothing to magnify yet) and you get the full capture. MARKS: pass marks: "elements" to get the visible interactive elements NUMBERED on the picture (red boxes) plus a legend mapping every number to its locator and viewport rect — pick by number, act by locator, never by a pixel estimated off the picture. IMAGE CONTRACT (this tool and run_macro alike): every picture is the real capture; the only markings ever added are numbered candidate boxes (elements, image matches, OCR matches), blue search-area rectangles, arrows/rings for the macro's own input and a magnified inset of the last click — each number has a legend line with exact coordinates, nothing else is painted (no OCR word overlay), and no marking is ever page content.

Switches which connected browser the other tools talk to. Several browsers (or several instances of one browser) can hold a bridge connection at the same time; bridge_status lists their labels. Pass a full label ("chrome#2") for an exact instance, or a bare family name ("firefox") for the most recently connected instance of that family.

Sends a message to the REAL in-panel AI Chat — the configured chat model runs it with its own agent tools, exactly as if a user had typed it. Returns immediately; a chat run takes minutes, so poll get_chat until it reports running:false. For self-testing the chat agent end-to-end.

Switches AI settings for independent runs (e.g. benchmark sweeps): provider (uivision | openrouter | anthropic | local), uivision_tier (free | pro), openrouter_model / anthropic_model / local_model / local_base_url, cv_scope (browser | desktop), ocr_engine (1|2|3 OCR.Space, 90 AI provider, 98 built-in cross-platform, 99 OS reader). Only switches between providers the user already configured — API keys can NOT be set or read over the bridge (Settings > AI is the only way in). Refused while a chat run is in progress. Takes effect from the next AI call.

Apply changes to the macro in the Ui.Vision editor — pass the complete Ui.Vision JSON. PREFER the JS script form {"Name": "name", "Script": "<JavaScript using the uiv.* API>"} over the classic Commands table (call get_authoring_guide for the uiv.* API before writing your first script). When fixing or extending a classic Commands-table macro, CONVERT it to a JS script in the same call and fix it there, preserving its targeting technique (visual stays visual) — recommend the conversion in your summary; keep the classic form only if the user explicitly insists on a table macro. The user's original macro file is never overwritten: the first change to a user macro is saved as a new copy in the "AI Generated" folder, which then becomes the macro being edited. Returns the macro name that was written, or a validation error.

EXPLORATION ONLY: click the point (x, y) of the MOST RECENT screenshot and type text there (key names like ${KEY_ENTER}, ${KEY_TAB}, ${KEY_CTRL+a} are honoured). Same conversion and same caveat as click_at. Returns what changed on the page afterwards.