com.extentos/mcp-server

Upload a local audio file into a project's named-sound library. Pass the name app code will play it with and a path to an MP3/M4A/AAC/WAV file under 1 MB. After the next assistant start (app relaunch), every device running the app can glasses.audio.playSound(name) — and the sound appears in the dashboard's Agent section, where it can also be chosen as the wake sound. Code registrations (audio.registerSound) win over library sounds on name collisions. GATED by the project's MCP access grant (assistant_config.sounds — default Read+Write; a permission_denied 403 means the owner restricted it). Requires a linked Extentos account (a 401 returns account_required, run `extentos-mcp login`). USE to provision an app's UI sounds while building it — e.g. a camera app's shutter click ({ name: "shutter", filePath: "./sounds/shutter.mp3" }). DON'T USE for the assistant's VOICE (that's setAssistantConfig) or for sounds the app should bundle locally (registerSound in app code needs no upload).

Adopt the committed extentos.connection-page.json UP to the dashboard/server (file → server) — the deliberate one-time seed of an existing code/file theme into the managed config. Reads the file at projectPath, validates it, and persists it (requires a linked account — run `extentos-mcp login`). DRIFT-PROTECTED: if the app already has a managed server config that DIFFERS from the file, it returns status: needs_confirmation with both versions and does NOT clobber — re-call with confirm: true to let the file win (or run regenerateConnectionPageFile to let the server win). Android-only for now (iOS is Phase 4). USE to seed an existing committed theme into the dashboard. DON'T USE for routine edits (author via setConnectionPageConfig / dashboard, then regenerateConnectionPageFile).

**Phase 4** — wait for an `assistant.tool_called` event matching `name` (and optionally `argsMatch` partial-match). Re-scans the backend event log every ~200ms from a FIXED anchor until match or timeout, then returns the matched event payload directly (or errors with `tool_not_called` on timeout). **Pass `sinceCursor` — the `watchCursor` from the injectAssistantUtterance that triggered the tool.** That anchors the wait BEFORE the inject, so the triggered tool call (which the model fires 0.5-2s later) is always caught even if this call starts after the tool already fired. Without `sinceCursor` the call anchors at its own start ("now") and will MISS a tool that fired in the gap between your inject and this call — the dominant false-negative cause before the 2026-05-29 fix. The fixed-anchor re-scan (vs an advancing cursor) also means a late-committing event is never skipped, and the match is pure `seq`-comparison so it has zero clock-skew dependency. USE in the agent E2E loop right after `injectAssistantUtterance`. DON'T USE to inspect the full event trace (use `getEventLog`) or to check static tool registration (use `inspectIntegration`).

Poll the backend until the user finishes signup at the verificationUrl, then persist the bearer token to ~/.extentos/auth.json. USE after createSimulatorSession returns status:'auth_required' AND you set autoLink:false (or the inline poll timed out). DON'T USE preemptively. **Note (Bundle 10+):** createSimulatorSession's `autoLink` arg (default true) handles this inline for you in the typical case; you only need to call completeAuthLink directly if you opted out of autoLink or the user took longer to approve than autoLinkSeconds.

**Get-or-create**: provision a browser-based simulator session for this project, OR return the existing saved one. The simulator hosts your customer-built app running against a Meta-DAT-shaped transport stub — `glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()` and every other capability primitive flow through it exactly as they would on real hardware, so the agent can dogfood a handler in a browser before a Meta DAT is paired. Persistent-simulations model: each project + platform has at most one saved sim tied to the user's account. Calling this returns that sim's existing URL if it exists (status:'resumed') instead of creating a new one. To force a new sim ID (URL hijacked, clean-slate test), pass `resetFresh: true` — archives the existing and creates a fresh one. The dev can also reset from the dashboard at extentos.com/s. **First-time per project**: creates the sim (status:'active'); subsequent calls in any future session return the same sim. **Auto-bind** (response carries autoBind:'attached'): when the dev's running app is reachable via the MCP local bridge, attaches it to the session — no rebuild needed. Otherwise URL-bake fallback (Android: buildConfigField patch; iOS: plist write). For 'resumed' responses, the URL is unchanged from last time so no rebuild/patch is needed regardless. **Iteration model — DO NOT call this per change**: app-code edits rebuild + reinstall and reattach automatically; the simulator URL is stable. Calling this again on the same project just returns the same sim — cheap, idempotent. **First-link auto-poll**: when account-linking is required, by default polls completeAuthLink internally for `autoLinkSeconds` (default 30) and re-mints on success. `autoLink: false` for CI / non-interactive contexts.

Retire a simulator session on purpose. USE only when you genuinely need a NEW session identity — a different session id and URL. **You almost never do.** createSimulatorSession WITHOUT `resetFresh` already RESUMES the saved sim for a project: same session, same URL, your app reattaches with no rebuild. That is the persistent-sim model and it is the intended way to work. DON'T USE to 'clean up' or 'start over' when something looks wrong — deleting disconnects any attached app and browser, and an app carrying a baked EXTENTOS_SESSION_URL needs a rebuild against the new one. Deleting is the FIRST HALF of a deliberate rotation: `createSimulatorSession({ resetFresh: true })` refuses with `session_already_active` while a sim is live for the project, precisely so a rotation is two explicit steps rather than a silent replace. Sim churn is what that guard exists to stop: discarded sessions leave device registrations behind that outlive them, and a later mint can auto-bind to an app that is already gone. Idempotent — deleting a session that does not exist returns `alreadyGone: true` rather than an error.

Ensure a **connected** simulator browser tab for the session — opens one if needed and confirms the browser's WebSocket actually attached before returning. The browser tab is the hardware surrogate's viewport: camera + inject tools require it (`setSimVideo` returns `browser_not_attached` without it; the `capture_photo` / `describe_scene` flows the assistant runtime drives fail). **Idempotent** — if a browser is already connected it's a no-op (returns `alreadyOpen: true`); it never opens a redundant tab (the hub would reject a second live browser with role_conflict). **When to call:** right after createSimulatorSession before any camera-driven flow; after a backend deploy (which severs every sim WebSocket — the tab does NOT auto-reconnect); whenever the tab was closed; or any time getSimulatorStatus / a camera tool reports the browser isn't attached. **How it works:** checks the AUTHORITATIVE in-memory hub liveness (the same signal setSimVideo/inject enforce, so this can't disagree with them — unlike `getSimulatorStatus.connectedRoles.browser`, which is persisted state that can lag a dead socket ~25-50s after an unclean drop); if no browser is attached, the MCP server opens the session URL in the developer's default browser ITSELF (cross-platform — no shell command for you to run), then polls until the tab's WebSocket attaches or `timeoutMs` elapses. **Headless / remote agents:** pass `autoOpen: false` to get an immediate presence snapshot + the sessionUrl with NO spawn and NO wait — surface the URL to the developer to open on a machine with a display, then re-run with autoOpen:true to confirm. Returns `{ alreadyOpen, opened, browserConnected, browserClientId, appConnected, sessionUrl, waitedMs }`; errors with `browser_not_connected` (carrying sessionUrl) when an auto-open didn't attach in time. **DON'T USE** to check whether the device APP is attached or hardware is ready — this ensures the BROWSER viewer only; use getSimulatorStatus for app/hardware state.

One-shot project scaffold — emits the bootstrap module that wires `Extentos.create(...)` into the host app, build-script changes, dependencies, permissions, and the integration manifest. Run once per project; after this lands, the agent writes its own Handler classes against the SDK primitives surfaced by getCapabilityGuide / getCodeExample. **PATH IS DERIVED FROM `capabilities`.** An audio-only footprint emits the VOICE scaffold — SDK dependency, mic permission, bootstrap; NO connection page, NO Meta account, NO App ID or Client Token, NO credentialed repository, and NO placement round trip (single call). PLATFORM NUANCE on the vendor SDK itself: on Android the voice path genuinely does not pull it (com.extentos:glasses is vendorless; only com.extentos:glasses-meta carries the DAT artifacts, which is why the split exists — Meta's Android artifacts sit behind a credentialed repo that returns 401 without a PAT). On iOS the swift-glasses package is a single umbrella and GlassesCore links MWDATCore/MWDATCamera, so the DAT SDK IS in the graph for a voice app — but Meta's iOS repo is PUBLIC, so it resolves with no account, no token and no auth. The developer-facing promise holds on both; the dependency graph differs. Splitting iOS to mirror Android is a 3.0 item (breaking for existing consumers). A footprint including camera / video / display emits the DEVICE scaffold: adds `com.extentos:glasses-meta` plus the Meta DAT repository, and requires the vendor credential setup. Omitting `capabilities` defaults to VOICE. **The two-call flow below applies to the DEVICE scaffold only.** **Two-call flow**: call once WITHOUT `placement` to receive `status: "needs_placement"` plus the question to ask the dev (where should ExtentosConnectionPage live in the app?); after the dev answers, call again WITH `placement` set to one of the 5 ids to receive the full scaffold (files, suggestedRendering, etc.). **Existing-app detection (Android)**: pass `projectPath` so the handler reads Andro…

Read the per-project "Agent" (assistant) settings the dashboard holds for an app — the OpenAI Realtime model, the voice, the memory (compaction) model, and the within-session memory mode — plus the catalog of valid options for each, the dashboard defaults, and the SDK hard defaults. Returns { managed, config, options, dashboardDefaults, sdkHardDefaults }: managed=true means a dashboard row exists and drives these at session start; managed=false means none is set, so the SDK uses its hard defaults unless the app code sets values. Runtime precedence is code-set > dashboard config > SDK hard defaults; this tool sees the dashboard layer only. Account-scoped + read-only — requires a linked Extentos account (run `extentos-mcp login`; a 401 returns account_required). Consumed by BOTH SDKs at assistant start (iOS re-fetches on app relaunch — no rebuild needed). USE to see what models/voice a project is configured with, or to discover the valid option ids before recommending a change. DON'T USE to change them (use setAssistantConfig; the dashboard's Agent section is the human-facing editor) or for usage/cost (use getGatewayUsage).

Per-feature SDK usage guide — minimal Kotlin + Swift snippet + gotchas + which getCodeExample patterns exercise the feature. Pairs with getPlatformInfo (which lists feature names and categories) by adding the actual idiom for using each feature. Covers: lower-level primitives (capture_photo / capture_video / record_audio / transcription_incremental / speak / video_frames / audio_chunks / connection_state / toggles / voice_command); the **Phase 4 assistant runtime** (assistant_runtime as the umbrella, assistant_start, assistant_tool, assistant_provider_openai) — the canonical voice-AI surface; and the **legacy Phase 3 conversation runtime** (conversation_runtime + conversation_on_wake / conversation_listen / conversation_speak / conversation_ai_complete) — REMOVED on Android in 1.4.0, kept as historical migration reference. For voice-assistant work on new apps, start with `assistant_runtime` for the overview then drill into the individual primitives. USE when you know which feature you need but don't know the canonical call shape, or when you're hitting a confusing failure (the gotchas typically cover it). DON'T USE for a complete compositional pattern (use getCodeExample) or for capability discovery (use getPlatformInfo).

Reference library — retrieve a complete SDK code example (Kotlin + Swift) for a use case. **For voice-assistant work on any new app, start with `assistant_agent_loop`** — the Phase 4 `glasses.assistant.start { tool(name, description) { body -> ToolResult } }` API where the model owns wake detection + turn taking + intent parsing + confirmation speech and the customer only writes tool bodies that act on app state. The provider abstraction covers OpenAI Realtime, xAI Grok, and Google Gemini Live (Gemini from SDK 1.8.0+) — the model id picks the vendor. The example pairs the customer code with the agent-driven E2E loop (`injectAssistantUtterance` → `assertToolCalled`) that verifies it without humans. **For the dedicated agent-side test workflow** (explicit two-step wake, multi-tool sweep, four-channel verification: event log + adb logcat + screencap + library state; real-OpenAi capable since iter5.2): `agent_driven_e2e_full_loop`. For the glasses-display two-view navigation pattern (browse ⇄ detail on the Ray-Ban Display, Neural-Band select + assistant tools driving one state machine): `display_browse_detail`. Customer code typically drops ~60% LoC vs the legacy Phase 3 pattern. `conversation_agent_loop` is the LEGACY Phase 3 pattern (REMOVED on Android in 1.4.0) — kept in the catalog as historical migration reference. Other patterns still ship for apps that need fine-grained control: voice_qa_assistant (manual `glasses.voice.onPhrase` wake + `glasses.audio.recordDiscrete` + customer-side AnthropicClient — the pre-Phase-3 composition), barge_in_speak (manual TTS cancel on user interrupt), photo_describe_voice (wake → photo → vision LLM → speak), live_transcription_ui (transcripts into Compose/SwiftUI state), voice_notes (wake → record → persist), connection_page_setup (the minimum bootstrap wiring), byok_anthropic (an Anthropic HTTP client the CUSTOMER'S OWN handler code calls with the customer's own key — used by voice_qa_assistant / photo_describe_voice. It has noth…

Read the per-project ExtentosConnectionPage config (theming tokens + section visibility) the dashboard/server holds for an app. Returns { managed, config }: managed=true means the dashboard/server is the SINGLE source of truth (the SDK applies server.overlay(defaults) at render time) and the committed extentos.connection-page.json should mirror it; managed=false means UNMANAGED (committed file → code ExtentosTheme → built-in defaults). Read-only. USE to inspect what the dashboard currently drives, or before regenerateConnectionPageFile. DON'T USE to change it (use setConnectionPageConfig).

Return step-by-step credential setup for the Meta DAT build identity plus each BYOK AI provider the app integrates. Returns { metaCredentials, serviceCredentials, keyStorage, summary }: `metaCredentials` = { required, requiredScope: 'real_hardware_only', scopeNote, steps[], configChange: { file, keys[], example, note? } } — the `requiredScope` / `scopeNote` say Meta DAT creds are needed ONLY for builds on real Ray-Ban Meta hardware; sim/emulator dev works without them, so DON'T lead the developer through Meta's developer-portal registration until the hardware milestone. `serviceCredentials` = one entry per BYOK service { service, forHandlers[], steps[], codeExample }. `keyStorage` = platform-specific { preferredPath, fallbackPath, doNotDo[], notes[] } secure-storage guidance. Cross-links: getProductionChecklist for where these steps sit in the ship sequence, getCredentialStatus / setCredential for the account-vaulted path (secrets pasted in the dashboard, never through the agent). USE when wiring a new BYOK provider or graduating from simulator to real hardware. DON'T USE during the simulator-only dev loop (Meta DAT creds aren't needed there).

Read whether a project's Meta DAT build identity is set — plus a MASKED hint and when it was updated. Returns { credentials: { metaDat: { set, appId, clientTokenHint, teamId, urlScheme, updatedAt } } }. There is NO provider-key entry — the assistant always runs on the Extentos managed gateway and there is no key to supply or check. SECRETS ARE NEVER EXPOSED — only presence + a masked hint; the encrypted value is never read or decrypted. Account-scoped — requires a linked Extentos account (a 401 returns account_required, run `extentos-mcp login`). GATED by the project's Credentials access grant (default Read+Write; a permission_denied (403) means it was set to Off). USE to check whether a project still needs an API key before recommending setup, or to confirm a key landed after setCredential. DON'T USE to read a secret value (impossible by design) — to ADD one, use setCredential.

Read what's currently rendered on the simulated glasses DISPLAY (track 5 — the native `glasses.display.*` capability). Returns the DisplayNode tree the app most recently rendered via `glasses.display.show { column { text(); button(...) } }` (or `shown: false` after `glasses.display.clear()` / before any show), the flat list of **selectable node ids** (button ids + clickable container ids), the node count, and the root kind. This is how the agent verifies *what's on the glasses screen* without pixels, and discovers which ids it can drive with `injectInput`. Source is the live in-memory hub snapshot (the latest show frame), so it's exact + immediate — no event-log redaction. Pairs with `injectInput` to close the agent-driven E2E loop for display flows: createSimulatorSession → (app calls display.show) → **getDisplayState** (read the tree + ids) → `injectInput({ action: 'select', targetId })` → the dev's onClick runs → **getDisplayState** again to see the re-render. USE to confirm a display rendered, read its structure, or find selectable ids before injectInput. DON'T USE for the display event history (use `getEventLog(filter: 'display')`) or non-display session state (use getSimulatorStatus).

Fetch structured event trace inside a simulator session. Primary debugging tool for *why is my handler not seeing what I expect* — transcripts not arriving, photo capture failing, toggle changes not propagating, speak getting cut off, connection dropping, display not rendering. Events are grouped into six chips (errors / voice / camera / display / lifecycle / custom) — one chip per event, with `errors` absorbing every severity≥warn row regardless of modality. To see e.g. voice activity plus voice errors, fetch the chips separately and union them. USE to diagnose which capability primitive is misbehaving. DON'T USE for static configuration checking (use validateIntegration) or for live session phase (use getSimulatorStatus). **Scope:** captures the simulator's WebSocket relay — transport + SDK primitives (audio, camera, speak, toggles, voice triggers, runtime events). Customer-side direct-HTTP calls (BYOK Anthropic / OpenAI / Gemini / etc.) traverse api.<provider>.com from the customer's app, NOT the simulator relay — invisible by default. **Surface BYOK calls in the event log by wrapping them in `glasses.observability.aiCall(label) { ... }` (F-R3-11)** — the wrapper emits `ai_call_start` and `ai_call_end` frames with timing + success/error metadata. These land under the 'custom' chip — the dedicated 'ai' chip was retired 2026-07-25 after never carrying an event in production. Without the wrapper, BYOK failures show as silent gaps (e.g. capture_photo + photo_result + speak with 3 unexplained seconds between) — those failures still need logcat / OSLog. Use the wrapper for any AI call that's part of your debug story; leave it off for true fire-and-forget background calls. **Live watch:** pass `follow: true` to block until new events land instead of returning an empty snapshot, and carry the returned `cursor` between calls — loop the pair to tail the log in real time (e.g. to follow a multi-turn AI conversation the developer built into the app, reacting to each `speak`…

Read account-scoped managed-AI-gateway usage + exact cost for a project over a recent window. Returns { window, totals, byModel, creditBalanceUsd }: totals carry token counts, response count, and the exact list-price USD cost from the billing ledger; byModel breaks the same down per model (biggest first). METADATA ONLY — token counts and cost, never transcripts/audio/prompt content. Only managed-gateway + dashboard-vault-BYOK sessions are metered; a code-direct setOpenaiApiKey() bypass goes straight to OpenAI and is invisible here. Account-scoped + read-only — requires a linked Extentos account (a 401 returns account_required, run `extentos-mcp login`). USE to report spend, spot the dominant model, or ground a model-choice recommendation in real cost (pair with getAssistantConfig). DON'T USE for live per-event traces (use getEventLog) or to change config (use setAssistantConfig).

The entry point for a developer who ALREADY built their app against raw Meta DAT (the Device Access Toolkit / Wearables SDK) and wants to move onto the Extentos SDK. Extentos's production transport calls the SAME DAT underneath, so migration is a call-site swap, not a rewrite. Returns a map KEYED BY THE DAT SYMBOL you already have (`Wearables.createSession`, `photoDataPublisher`, `addStream`, `Display`, raw `CaptureError`, …) → the Extentos primitive that replaces it (`glasses.connection`, `glasses.camera.capturePhoto`, `glasses.camera.videoFrames`, `glasses.display.show`, `ExtentosResult`, …), plus a one-line 'what changes' per area and an ordered cutover plan. It re-embeds NO code — each mapping names the Extentos `feature`, so you drill into getCapabilityGuide(feature) for the Kotlin/Swift idiom and getCodeExample(pattern) for a full composition. The plan orchestrates the tools that already exist: generateConnectionModule (replaces Wearables.initialize/configure + your registration UI) → getPermissions (replaces hand-mapped manifest/plist + Meta scopes) → per-call-site swaps → validateIntegration → createSimulatorSession. **Scope: FULL CUTOVER, not coexistence** — running raw DAT and Extentos side-by-side in one process is not supported today (the DeviceSession is a single-owner handle), and the response says so. USE when a project already imports `com.meta.wearable.dat.*` (Android) / `MWDATCore`/`MWDATCamera` (iOS) and you're adopting Extentos. DON'T USE for a greenfield project with no existing DAT code (start at getPlatformInfo → generateConnectionModule).

Derive the Android permissions, iOS Info.plist keys, and Meta DAT scopes a list of SDK capabilities requires. Returns { android, ios, metaDat, summary } — **ALWAYS all three platform blocks regardless of the `platform` argument** (which only frames the one-line `summary`; it does NOT filter the response). `android` = { permissions[], manifestEntries[] (ready-to-paste <uses-permission> lines), foregroundService: { required, types[], declaration, devInstructions }, notificationListener: { required, declaration, devInstructions }, minimumSdk, compileSdk, targetSdk }. `ios` = { plistKeys: [{ key, value, reason }] }. `metaDat` = { scopes[], registrationRequired, registrationSteps[] }. Capabilities that need no platform permission (e.g. speak, connection_state, earcon) contribute nothing and are accepted silently — they never error. USE after deciding which features (capture_photo, transcription_incremental, voice_command, …) the app integrates. DON'T USE for capability discovery (use getPlatformInfo).

Return static platform metadata: library version + the list of SDK capabilities the glasses expose. **TWO INTEGRATION PATHS — decide which one BEFORE scaffolding.** Voice apps (assistant, transcriptions, recordDiscrete, speak, audio streams) reach the glasses through the phone's own Bluetooth audio routing: NO vendor SDK, NO vendor credentials, NO connection page, NO pairing flow — and the complete agent runtime (turn-taking, barge-in, tool calling, local + cloud models) works there. Only CAMERA and DISPLAY require a vendor integration. If the developer hasn't said they need camera or display, they are on the voice path — do NOT send them through Meta account setup, a Developer Center registration, or a GitHub PAT they don't need. **Phase 4 assistant runtime** (`glasses.assistant.start(provider) { tool(name, description) { body -> ToolResult } }`) is the canonical voice-assistant API — the model owns wake/turn-taking/intent/confirmation, the customer writes tool bodies that act on app state. The legacy Phase 3 conversation runtime (`glasses.conversation.onWake { listen() / speak() / ai.complete() }`) was REMOVED on Android in 1.4.0. Lower-level primitives are still surfaced for fine-grained control (audio.transcriptions, audio.recordDiscrete, audio.speak, audio.audioChunks, camera.capturePhoto, camera.videoFrames, toggles, connection.state, …). The right first call for any new task — primes the agent on what primitives are available before writing handler code. **Default response is COMPACT** — capability names and categories only — keeping the typical first-call cost ~2KB. Per-feature call shape + idiom + gotchas comes from `getCapabilityGuide(feature)`; full compositional patterns (Phase 4 `assistant_agent_loop` for the canonical glasses.assistant.start loop; legacy `conversation_agent_loop` for Phase 3; `voice_qa_assistant` for the pre-Phase-3 manual composition) from `getCodeExample(pattern)`; conceptual docs from `searchDocs(topic: 'assistant_runtime')` for th…

Return a personalized production-readiness checklist based on the SDK capabilities the app uses, the handlers it declares, and the BYOK services it integrates. Returns { ready, steps, summary }: `ready` is true when no step has status 'needed'; each `steps[]` entry is { category, description, status: 'done' | 'needed' | 'optional', details, fix?, requiredScopes?, affectedHandlers?, codeChange?: { file, hint } }. Steps are conditional — streams add a foreground-service step, voice/camera/audio add a real-hardware-verification step, and declared BYOK `services` (or the Phase 3 conversation runtime) add an API-keys step. Pass `projectPath` so the Meta-account and credential-swap steps read the project's real build state and report 'done' instead of 'needed'. Cross-links: getPermissions for the exact permission set the Permissions-Audit step verifies, getCredentialGuide for the per-provider key setup the API-Keys step needs, validateIntegration for the structural pre-test gate. USE when going to production. DON'T USE during development.

Read a project's PRODUCTION analytics — aggregate telemetry from the app's shipped (App Store / Play Store) installs over a recent window. Returns { window, totals: { events, activeInstalls }, byEvent, byDay, byVendor, byPlatform, hasData }. METADATA ONLY — event counts + active installs, NEVER transcripts/content/PII. Reads the prod-attested telemetry warehouse (the same data the dashboard Analytics page shows), so it is EMPTY until the app ships to a store and sends prod-attested events — dev/sideload telemetry is a separate tier and not included (use getEventLog for the live sim/dev event stream). Account-scoped + OWNERSHIP-CHECKED (you can only read your own projects; a 403 not_owner means your account has no data for that package) and GATED by the project's Analytics access grant (default Read; a permission_denied (403) means it was set to Off). A 401 returns account_required (run `extentos-mcp login`). USE to report a shipped app's real-world usage/adoption. DON'T USE for gateway spend (use getGatewayUsage) or live dev events (use getEventLog).

Read a live simulator session's current state — phase (active/paused/closed), hardware-ready, attached roles (app + browser), active capability streams (which `videoFrames`/`audioChunks`/`transcriptions` subscriptions are open right now), and the **testVideos** list available to drive `setSimVideo` (defaults bundled with the platform plus any MP4s uploaded for this project on the simulator page). Also returns a **freshness** advisory — each role's `connectedAt` plus the staleness rule — so you can catch the #1 silent failure in the test loop: driving a connection that went STALE after a code change (rebuilt/reinstalled the app → relaunch it so it's the fresh build; pushed a deploy → it severed the sim WebSockets, so re-ensure the browser). Session-level snapshot only. USE during testing to confirm a stream is open, the session is healthy, the app role has attached, to check a connection isn't stale after a change, or to discover which test videos you can pipe into capture_photo/capture_video. DON'T USE for event traces (use getEventLog).

Analyze proposed voice phrases for UX issues (length, homophones, digit-usage, Meta wake-word collision, ambiguity with existing phrases) before wiring them into a wake trigger. Returns { results, generalGuidance, summary }: `results` is one entry per input phrase — { phrase, issues: [{ severity: 'error' | 'warning' | 'info', rule, message }], suggestions: [], collisions: [] } — where `collisions` lists exact-duplicate matches and `issues` carries per-rule findings (rules: too_short, homophone_risk, digit_usage, punctuation, collision, disambiguation, meta_wake_word_overlap, match_mode_hint). Only `collision` and `meta_wake_word_overlap` are error-severity (fix before wiring); everything else is advisory. `generalGuidance` is cross-phrase advice; `summary` is a one-line rollup. Applies equally to Phase 3 `glasses.conversation.onWake(phrase) { ... }` registrations AND legacy `glasses.voice.onPhrase(phrase) { ... }` / direct `glasses.audio.transcriptions()` consumers — the issues this catches are the same regardless of which API surface dispatches the handler. USE before adding new phrase-match conditions to ANY voice-driven handler. DON'T USE for general voice docs (use searchDocs topic 'conversation_runtime' for Phase 3, 'voice_integration' for the legacy primitives).

**Phase 4** — drive an assistant turn from outside the live session for agent-driven E2E tests of `glasses.assistant.start { tool(...) { ... } }`. **As of iter5.2 (2026-05-27) the `text:` path works for BOTH the Mock provider and the real OpenAi Realtime provider** — same MCP call, provider-appropriate routing: (a) **OpenAi (default in production apps)** — injected as a synthetic user turn via `conversation.item.create` + `response.create` on the live Realtime WebSocket. Drives REAL model behavior with full conversation context + tool routing decisions; costs real tokens (~$0.005/turn); takes 500-2000ms for the model to respond + dispatch tool. (b) **Mock** — word-overlap-matches against registered tool descriptions + dispatches synchronously. Sub-ms, deterministic, $0. The library filters inject frames to `source: "assistant_inject"` only, so browser-mic STT transcripts (already flowing via the PCM audio path) are NOT double-injected. **Wake first if Dormant.** This handler does NOT auto-wake — if the session is Dormant (silence-timeout, fresh session, post-`end_conversation`), the library's `injectUserTurn` silently no-ops because `connectionRef` is null + the inject is dropped. Drive the wake faithfully via `injectTranscript({ text: "<your wake phrase>" })`, wait for `assistant.session_started` in `getEventLog`, THEN inject. This mirrors what a real user does on real hardware: phrase, wait, command. (The iter5.3 autoWake convenience was shipped + reverted in 0.1.4 — it collapsed the explicit two-step into one call, hiding the wake step and racing with `onWake` greeting hooks.) audioWavBase64 path is DEFERRED to v1.1 (needs Rust core changes) — handler returns not_implemented if passed. **Returns `watchCursor`** — a seq cursor captured immediately before the inject; pass it straight into `assertToolCalled({ sinceCursor })` so the assertion anchors BEFORE this inject and never misses the tool call it triggers (the model fires the tool 0.5-2s later, often after you…

Press the simulated glasses' hardware CAPTURE BUTTON (the right-temple controls) — the agent's way to exercise the wearer's hardware privacy gestures with no human and no hardware. **Tap** on a LIVE camera stream pauses it; a second tap resumes. While paused, every stream-needing capture (capturePhoto — both mechanisms —, captureVideo, videoFrames) fails with `CaptureError.StreamPaused` carrying the actionable "tap the right temple to resume" message, and a `capture_denied` row lands in getEventLog(filter: 'errors') — the same behavior a wearer's temple tap produces on real glasses, because it IS the same shared gate. **Hold** STOPS the stream (`camera_stream_closed`): on real glasses the hold stops the whole device session — the connection drops and the SDK auto-recovers over ~3-5s; the sim closes the stream and deliberately skips the connection blip (a surfaced substrate delta) — the next frame-grab photo / video / videoFrames use re-arms it. The stream-state transitions land as `camera_stream_opened` / `camera_stream_paused` / `camera_stream_resumed` / `camera_stream_closed` under filter 'camera', and the sim page's capture LED tracks them (lit = streaming, dark = paused/closed); videoFrames delivery halts while paused and dies on close. **A gesture with NO live stream is invisible to the app** (hardware-faithful — it falls through to Meta's first-party capture, which DAT apps never see): arm the stream first. Mirrors the sim page's Hardware-buttons panel — same hub path, identical downstream. Typical loop: createSimulatorSession → ensureSimulatorBrowser → setSimVideo → (app takes a photo — stream arms) → injectHardwareButton (pause) → drive a capture → getEventLog(filter: 'errors') shows capture_denied → injectHardwareButton (resume) → capture succeeds. USE to test paused/stopped-camera handling end-to-end. DON'T USE for display input (injectInput), voice (injectTranscript), or hardware alerts like thermal/hinges (those inject via the sim page).

Drive the simulated glasses' DISPLAY input — the agent's way to "click" the display with no human and no hardware (track 5). Three actions: **select** (+ `targetId`) fires that node's onClick, running the developer's handler exactly as a Neural-Band index-pinch would on real glasses — THE action that closes the display E2E loop; **navigate** (+ optional `targetId`) moves the focus highlight (to a specific id, or the next selectable node when omitted); **back** emits a back gesture. The frame is delivered to the simulator browser tab (the display surrogate that owns the rendered tree + focus), which routes the selection back to the app — so an agent-injected select and a human clicking the sim's gesture panel are identical downstream (selection flows browser → backend → app, mirroring real hardware). **Requires the simulator browser tab connected** — the display only exists while the browser renders it; call `ensureSimulatorBrowser({ sessionId })` first or this returns `browser_not_connected`. Discover `targetId` values from `getDisplayState.interactiveIds`. Typical loop: createSimulatorSession → ensureSimulatorBrowser → (app shows a display) → getDisplayState → `injectInput({ action: 'select', targetId: 'end-run' })` → `getEventLog(filter: 'display')` shows `display_select` then the re-render (`display_show`). Mirrors injectTranscript (voice) / setSimVideo (camera) for the display capability. USE to exercise display button handlers + navigation end-to-end. DON'T USE for voice flows (injectTranscript / injectAssistantUtterance) or to read display state (getDisplayState).

Inject a synthetic STT transcript into a live simulator session, the same way the simulator browser tab's click-to-fire chips do. Closes the agent-driven end-to-end test loop: after createSimulatorSession returns and your app attaches, call this with the wake phrase text to drive your voice handler — no human in the loop. Frame travels through the same hub path as a real browser click, so handler dispatch + event-log entries (visible via getEventLog) are identical. **Dual-purpose** — works for BOTH the legacy `glasses.voice.onPhrase` matcher (wake-phrase match → dispatches the handler) AND the Phase 3 `glasses.conversation` runtime (H1 wiring: the library's shell subscribes to `audio.transcriptions()` and forwards each Final to `core.injectListenText`, resolving any parked `listen()` call inside a conversation handler with `Turn::Said{text}`). Typical conversation-runtime pattern: first inject = wake phrase (fires onWake handler → listen() parks); follow-up injects = utterances the parked listen() resolves to. Wait for `conversation.turn_started` in the event log between inject and follow-up inject to guarantee the handler is parked before the next text arrives (otherwise the follow-up drops silently — no FSM state poisoning, just no effect). USE for automated voice-flow validation (createSimulatorSession → injectTranscript → getEventLog). DON'T USE for static validation (use validateIntegration) or capability discovery (use getPlatformInfo). See `getCodeExample(pattern: 'conversation_agent_loop')` for the full Phase 3 recipe.

Read-only snapshot of the current Extentos integration at a project path. Returns { found, manifest, generatedFilesStatus, dependency, drift, connectionPageConfig, gaps, summary }. **`found: false` is NOT an error** — when no extentos.manifest.json exists the call still succeeds, returning empty sub-objects plus a `gaps` remediation string that tells you to run generateConnectionModule. When `found: true`: `manifest` is the parsed extentos.manifest.json; `generatedFilesStatus` is per-file { path, exists, hasMarker, hashMatch }; `dependency` is { found, coordinate, file } for the Gradle/SPM dependency; `drift` is { fileDrift: [{ path, reason: 'missing' | 'marker_absent' | 'hash_mismatch' }], dependencyDrift: boolean }; `connectionPageConfig` is the committed connection-page snapshot (or null); `gaps` is an array of human-readable remediation strings; `summary` is a one-line rollup. USE before validateIntegration or for a 'what's wired so far?' read. DON'T USE for correctness checking / a pre-test gate (use validateIntegration — it severity-tiers the same surface and adds ~15 checks).

List a project's named-sound library — the sounds uploaded in the dashboard's Agent section or via addProjectSound. Returns { sounds: [{ id, name, url, createdAt }] }. Devices download + register the library at assistant start; app code plays any of them with glasses.audio.playSound(name), and the dashboard can pick one as the wake sound. GATED by the project's MCP access grant (assistant_config.sounds — default Read+Write; a permission_denied 403 means the owner restricted it). Requires a linked Extentos account (a 401 returns account_required, run `extentos-mcp login`). USE to see which sound names app code can play, or before adding a sound to avoid a duplicate name. DON'T USE to add sounds (use addProjectSound).

Regenerate the committed extentos.connection-page.json FROM the dashboard/server config (file ← server) — the generated mirror that makes the dashboard's tokens real in the repo and lets the app theme correctly offline. Returns file: { path, content } for you to write (it carries a generated `_comment` marker — do not hand-edit; edits belong in the dashboard). If the project is UNMANAGED (no server config) there is nothing to mirror. Pass projectPath to get a drift note when an existing committed file would be overwritten. Android-only for now (iOS file-sync is Phase 4). USE after setConnectionPageConfig / dashboard edits to sync the repo. DON'T USE to push local edits up (use adoptConnectionPageFile).

Search Extentos documentation by topic or keyword. The conceptual / narrative layer that complements the action-oriented tools — read these to understand *how* the SDK is meant to be used, not just what calls exist. Canonical post-pivot topics: `getting_started` (full topic index — read first), `custom_handlers` (the central composition doc — how to write a Handler class that subscribes to capability primitives), `conversation_runtime` (**Phase 3 — the canonical wake/listen/speak/ai.complete loop**; read first if you're building a voice assistant), `voice_integration` (legacy primitives — `glasses.voice.onPhrase` + `audio.recordDiscrete` + customer-side LLM; for fine-grained control or pre-Phase-3 apps), `connection_ui_placement`, `host_app_scaffold`, `auto_bind_session_lifecycle`, `local_bridge_discovery`, `device_code_flow`, `connection_state_model`, `permissions`, `production_checklist`, `concurrency_modes`, `multi_platform_projects`, `library_api`, `toggles`, `audio_video_coexistence`, `simulator_browser_mode`, `event_log_schema`, `file_actions`, `agent_e2e_testing` (how an AI coding agent verifies its own generated handler end-to-end — sim event log + adb-mediated emulator DB read + screencap, the dual-layer pattern that closes the agent loop without a human). Topic IDs are stable. **At least one of `topic` or `query` is required** (enforced by the handler — calling with neither returns an invalid_arguments error). Pass `topic` alone to fetch the full topic content (most common); pass `query` alone for keyword search across all topics; pass both to narrow keyword search inside a topic. **mode: "snippets" (F-R4-10)** returns only the matching paragraphs instead of full topic bodies — pair with a query when keyword-searching across topics so a single search doesn't dump ~20KB of unrelated content into context. USE to learn how primitives compose into a real flow, or to read up on a specific feature (permissions, toggles, connection lifecycle). DON'T USE for the…

Change a project's dashboard-managed "Agent" (assistant) settings via MCP — any of the OpenAI Realtime model, voice, memory (compaction) model, or within-session memory mode (PARTIAL update; pass only what you're changing). Validates each value against the catalog (call getAssistantConfig first to see valid ids) and echoes the COST IMPACT of a model change (per-1M rates, old → new) so a switch is never blind. GATED by the project's MCP access grant — the default is Read+Write so it works out of the box; the owner can restrict it per-project in the dashboard, and a permission_denied (403) means Assistant-config access was set to Read or Off. Requires a linked Extentos account (a 401 returns account_required, run `extentos-mcp login`). Applies to the next assistant session on BOTH platforms (a value set in app code still wins; Android picks it up in a fresh process, iOS on app relaunch). USE to configure a project's voice agent from the agent loop. DON'T USE to read current settings (use getAssistantConfig) or to set secrets (credentials are never written through the agent).

Persist the per-project ExtentosConnectionPage config (theming tokens + section visibility) to the dashboard/server, making the app MANAGED — the dashboard/server becomes the single source of truth and the SDK fetches it at render time (server.overlay(defaults)). Requires a linked Extentos account (run `extentos-mcp login` once; a 401 returns account_required with that hint). Validated against the connection-page schema — unknown keys are surfaced as a warning (the SDK ignores them), not blocked. After writing, run regenerateConnectionPageFile to mirror it into the committed extentos.connection-page.json. USE to author/update connection-page theming from the agent; the web dashboard's Connection section is the human-facing equivalent. DON'T USE just to read (use getConnectionPageConfig).

Start WRITE-WITHOUT-KNOWING entry of the project's Meta DAT build identity. Takes NO secret argument BY DESIGN: the value must never pass through you. Returns a browser handoff { mode: "browser", url, credentialType, currentlySet } — a link to the project's dashboard credentials page where the signed-in owner pastes the secret; it goes straight from them into the encrypted vault (you can never read it back). GATED by the project's Credentials access grant at Read+Write (default Read+Write; a permission_denied (403) means it was set to Read or Off). Account-scoped (a 401 returns account_required, run `extentos-mcp login`). USE to wire up a project's API key from the agent loop without ever handling the secret: show the developer the returned URL, ask them to paste + save, then confirm with getCredentialStatus. DON'T pass a key value (there is no field for it) and DON'T ask the developer to paste a secret into the chat.

Switch the simulated glasses DEVICE MODEL — one of the eight simulatable models: rayban_meta, oakley_meta_hstn, oakley_meta_vanguard, rayban_meta_optics, meta_glasses (camera + audio, NO display), rayban_display (display + Neural Band), and the EXPERIMENTAL Android XR pair — android_xr_audio_glasses (camera + audio, no display) and android_xr_display_glasses (adds a display, driven by the temple touchpad, panel 450x394 rather than Meta's 600x600 square). The selected device's capability profile drives `glasses.display.isAvailable` on the connected app, so this is how an agent tests BOTH branches of a capability-gated feature headless: set `rayban_display` to exercise the display path (glasses.display.show renders), set any no-display model to verify the graceful degradation (the app's isAvailable guard declines; show() no-ops). The model identity ALSO surfaces to the app as `glasses.device.type` (and in the assistant's glasses-state context), so per-model polish — e.g. an Oakley-specific default voice — is testable by switching between models that share the same capability profile. Mirrors the sim's device dropdown — the change persists on the session AND is pushed live to a connected app (the gate flips without a reconnect), and a `device_changed` event lands in getEventLog(filter:'lifecycle'). USE before driving a display flow to pin which device you're testing; pair with getDisplayState / injectInput. DON'T USE for the camera scene (setSimVideo) or voice (injectTranscript).

Pipe a Test Video into the simulator's camera input so capture_photo / capture_video / videoFrames run against a known scene instead of a blank viewport — closes the agent-driven test loop for camera-driven flows the same way injectTranscript closes it for voice flows. Discover available IDs via `getSimulatorStatus.testVideos` (bundled defaults + project uploads). Requires the simulator browser tab to be connected (browser role attached) — call `ensureSimulatorBrowser({ sessionId })` first to guarantee it (idempotent: opens the tab cross-platform + polls until it attaches). Same precondition as the simulator's photo/video capture itself. The browser loads the video as its cameraSource (same code path as a human drag-dropping a file), and a `camera_source_set` lifecycle event lands in getEventLog so the source change is visible to your verification step. USE before triggering a capture_photo / capture_video flow that needs a scene. DON'T USE to upload new videos — that's the project simulator page's job (extentos.com/projects/<id>/simulator).

Pre-test gate that runs ~15 severity-tiered checks over the whole project — manifest present + parses, generated-file markers + hashes, dependency declared, library-version freshness, Android Meta-DAT repo declaration, bootstrap wiring (ExtentosGlasses.create / Extentos.create), connection-page config, toolchain (AGP/Gradle) floors, foreground-service hints, and permissions. The permissions check DERIVES the permissions your declared `capabilities` imply and verifies they're present in the REAL app/src/main/AndroidManifest.xml <uses-permission> elements (Android) or the app target's Info.plist keys (iOS) — the runtime source of truth, not the manifest JSON's permissions array. Returns { valid, checks, summary }: each `checks[]` entry is { name, passed, severity?: 'error' | 'warn' | 'info', details?, fix? }. `valid` is true when zero ERROR-severity checks fail; warn/info are advisories that don't block. `summary` reads 'Safe to test' ONLY at zero warnings — any warning demotes it to 'review before testing'. So treat this as a graded readiness report, not a binary pass/fail. USE after making changes, before testing. DON'T USE for reading state without judging it (use inspectIntegration).