Talkthrough

Re-extract ONE frame at an exact timestamp from the ORIGINAL source video at native resolution, with an optional crop {x, y, w, h} in source pixels. Use when the stored keyframes miss the instant (they capture scene changes + a 1 fps floor) or when you need full-resolution detail. Slower than get_frames — it decodes the source file, which must still exist at its recorded path (URL jobs keep their downloaded source inside the job, no network). When NOT to use: normal browsing — get_frames serves stored keyframes instantly without touching the source. Examples: - keyframes sit at 12:31 and 12:38 but the flash happened at 12:34.5 → extract_frame(job_id, at_ms=754500) - extract_frame(job_id="...", at_ms=754500, crop={"x":800,"y":40,"w":400,"h":120}) — zoom into the toast text - tiny UI text unreadable in the 1568px keyframe → extract_frame at the same ms for native resolution - verify a one-frame glitch: extract_frame at 12300, 12400, 12500 and compare - OCR missed small text → extract_frame with a tight crop, then read the returned image - crop coordinates are SOURCE pixels (a Retina screen recording may be 2940x1912) — not keyframe scale - response JSON carries "path" (absolute) — "save this screenshot next to my docs" = copy from path yourself - source file moved or deleted → clear error; stored keyframes via get_frames still work - audio-only job → always errors: there is no video stream to decode - anti-example: "show me around 5:00" → get_frames(job_id, at_ms=300000); extract_frame is for exact instants - anti-example: scanning a range frame by frame → get_frames(start_ms, end_ms) first, refine once after

Fetch stored keyframe images (JPEG, <=1568px wide) as MCP image content: the frames nearest to at_ms, OR unique frames across [start_ms, end_ms] evenly thinned to max_frames. Serves unique frames by default (near-duplicates from static scenes are filtered); hard cap 6 images per call. When NOT to use: exact instants between keyframes or native-resolution detail (use extract_frame), or finding on-screen text (use search — OCR text is indexed). Examples: - get_frames(job_id="...", at_ms=83500) — what was on screen when the remark at 1:23.5 was spoken - get_frames(job_id="...", at_ms=83500, max_frames=2) — tighter context, fewer tokens - get_frames(job_id="...", start_ms=0, end_ms=600000, max_frames=6) — overview strip of the first 10 min - get_frames(job_id="...", start_ms=290000, end_ms=310000, include_duplicates=true) — every capture near 5:00 - transcript hit at t_ms=421500 → get_frames(job_id, at_ms=421500) for the visual evidence - walking a demo scene by scene → one ranged call per scene beats one giant range - frame files are named by video-ms (t00083500.jpg ↔ t_ms 83500) — stable refs for findings - valid_from_ms/valid_to_ms on each frame = when the screen looked like this — check the span covers your moment - keep max_frames at 2-4 unless you are truly comparing scenes; images are token-expensive - every frame entry carries "path" (absolute) — save/copy the image elsewhere with your own file tools - audio-only job → this tool errors by design; use get_transcript / get_moment instead - anti-example: need EXACTLY 12:34.500 between two keyframes → extract_frame(job_id, at_ms=754500) - anti-example: "find the screen with the red error banner" → search(job_id, "error") first, then jump

The "one remark" evidence bundle: transcript slice + up to 3 unique frames + their OCR text + the wall-clock range for [start_ms, end_ms], in a single call. This is the workhorse for triage: one call per finding gives you the quote, the screenshot, and the on-screen text. When NOT to use: broad exploration (get_transcript / get_frames) or keyword lookup (search). Examples: - get_moment(job_id="...", start_ms=83000, end_ms=97000) — full evidence for the remark at 1:23-1:37 - segment seq 12 spans t0_ms=83210, t1_ms=96800 → get_moment(job_id, 83210, 96800) - pad ±2000 ms around the spoken range — narrators react to things already on screen - triage loop: for each candidate finding, exactly one get_moment call → quote + frame + OCR - user: "what was I showing when I said 'this button is broken'?" → search first, then get_moment at the hit - opening context of a meeting: get_moment(job_id, 0, 15000) - response includes the t_wall range when known → quote it in bug reports for log correlation - diarized job → speakers_in_range + speaker on each segment: who is talking in this window, at a glance - frame entries carry "path" (absolute) — copy the screenshot elsewhere with your own file tools - "was X on screen at t?" → yes iff some frame's valid_from_ms <= t < valid_to_ms — no extra calls needed - audio-only job → returns the transcript slice plus a no-frames note (that is expected) - anti-example: whole-video summary → get_transcript(format="text"), not a chain of get_moments - anti-example: need more than 3 frames of a range → get_frames(start_ms=..., end_ms=..., max_frames=6) - keep ranges under ~30 s; a 5-min "moment" dilutes the bundle and wastes tokens

Retrieve the transcript of a processed job, lazily and paginated. Formats: "segments" (default — seq, t_ms, t_wall when known, speaker when diarized, text), "text" (plain prose; "S1:" prefixes at speaker changes), "srt" (subtitles, speaker-prefixed cues). Diarized jobs also return the roster, attribution_precision, saved speaker_name values, raw OCR name_candidates, and bounded pending-review names plus old-roster context after a relabel. Pre-0.3.1 video jobs may return name_candidates_note because their flat OCR is readable but less useful for hints. Pending names are evidence to re-check, never active identities. A stale pending label can only be removed with label_speakers(..., labels={"Sx":null}). Raw S<n> labels remain canonical. Responses are capped (~8k tokens): when truncated=true, continue from the returned next_start_ms. When NOT to use: to find one keyword (use search) or to inspect one moment with visuals (use get_moment). Examples: - get_transcript(job_id="a1b2c3d4e5f60718") — whole transcript of a short recording - get_transcript(job_id="a1b2c3d4e5f60718", start_ms=0, end_ms=120000) — just the first two minutes - get_transcript(job_id="...", format="text") — prose block for summarization - get_transcript(job_id="...", format="srt") — subtitle export the user asked for - diarized job: segments carry "speaker" + a roster header (top-12 by talk time; speakers_truncated counts the rest) - "what did S2 say?" → format="segments", collect entries with speaker=="S2" (labels are in order of first voice) - got truncated=true with next_start_ms=421500 → get_transcript(job_id="...", start_ms=421500) - user: "what was said between 5:00 and 6:30?" → start_ms=300000, end_ms=390000 - legacy video name_candidates_note → explain the limitation; safe regeneration uses force=true+diarize=true - correlate speech with logs: each segment's t_wall lines up with your log timestamps - no speaker fields on a meeting job → re-run process_media with diarize=true (adds them withou…

Persist VERIFIED human-readable names for anonymous S1/S2/… labels on one diarized job. `labels` is a patch: a 1-100 character name saves it; null or blank removes it. Optional `evidence` (max 500 characters per label) records why the mapping is trusted. Raw labels remain canonical in JSON; names appear separately and in text/SRT display. The write is atomic, locked, local, and idempotent. OCR name_candidates are raw hints only and are never saved automatically. A relabelled job can carry pending-review names with source-roster anchors. For a label still in the roster, an explicit patch confirms, replaces, or removes only that entry. For a stale pending label, only an explicit null removes its name, evidence, and context. When NOT to use: before diarization, or when a name is only a guess without human/screen evidence. Examples: - label_speakers(job_id="...", labels={"S1":"Vera"}) — save one verified mapping - label_speakers(job_id="...", labels={"S1":"Vera","S2":"Tom"}) — patch several labels - save evidence: label_speakers(job_id="...", labels={"S1":"Vera"}, evidence={"S1":"intro at 1200ms"}) - screen name plate confirms S2 → save the name and cite its frame timestamp in evidence - label_speakers(job_id="...", labels={"S2":null}) — remove S2's name and evidence together - label_speakers(job_id="...", labels={"S2":" "}) — blank also removes the mapping - named S1, new proof → label_speakers(job_id="...", labels={}, evidence={"S1":"title card at 0ms"}) - two Alexanders may map to S2 and S5 — duplicate names are allowed and search covers both - response roster carries speaker_name beside label; raw S<n> labels are never replaced - name_candidates may be UI text or another person's name → inspect frames before deciding - unknown label or a name over 100 characters → error lists the valid roster labels - fresh session: get_transcript returns saved names; do not infer the mapping again - stale pending S3 → labels={"S3":null}; never assign a name while S3 is outside t…

List processed recordings, newest first: job_id, source filename, duration, created, wall-clock start, segment/frame counts. The store is content-addressed — the same file maps to the same job even after renames or moves, and jobs persist across sessions and machines restarts. When NOT to use: as a health check or before every call — job_ids are stable, remember them. Examples: - user: "triage the recording I processed this morning" → list_jobs() → pick by filename + created - user names neither job_id nor path → list_jobs() first; only ask if still ambiguous - resume yesterday's analysis in a fresh conversation → list_jobs() → reuse its job_id directly - file was renamed after processing → match by duration/created; the content hash ignores names - wall_clock.start answers "WHEN was this session?" — pick the job from "yesterday around 15:00" - after CLI batch pre-processing (`talkthrough-mcp process big.mov`) the job shows up here — query it - two jobs with the same filename → the newer created one is usually the re-recording - diarized jobs show "speakers": N — "the 4-person meeting from Tuesday" is findable at a glance - URL jobs carry "origin" (provider, provider_id, title) — "the YouTube video from yesterday" is findable - empty list → nothing processed on this machine yet; ask the user for a file path or a public URL - job disappeared → likely `talkthrough-mcp gc` cleaned it; re-run process_media on the file (same id) - anti-example: checking whether a NEW file is processed → just call process_media, it is idempotent+instant

Ingest a LOCAL video or audio file and make it queryable: validates the file, transcribes speech locally (whisper), extracts scene-change keyframes, OCRs on-screen text, resolves the wall-clock start time, and (opt-in) labels who said what via local speaker diarization. Returns a compact summary (job_id, media info, wall_clock, transcript preview, speaker roster when diarized) — full data stays on disk and is served lazily by the other tools. Idempotent by content hash: re-calling on an already-processed file returns instantly. For MULTI-PERSON recordings (meetings, interviews, calls) diarize=true is part of a proper analysis — pass it even when the user only asks for a summary. num_speakers is a target the clusterer may not reach, not a constraint — the payload says when a re-run changed nothing (labels_changed). If an amend changes the labels, verified names become pending-review evidence rather than active identities, with old-roster anchors for re-checking. Current pending labels can be confirmed/replaced/removed; stale labels can only be removed with an explicit null patch. Full force reprocessing of a job with saved or pending identities requires diarize=true and preserves every old identity as pending review against the rebuilt roster; without diarization it refuses before changing the stored job. When NOT to use: to re-fetch data you already processed (use the retrieval tools), or for URLs — local file paths only; a public video/audio URL goes to process_url. Examples: - process_media(path="/Users/sam/Desktop/bug-repro.mov") — narrated screencast, defaults are right - meetings: model="large-v3-turbo" + vocabulary=<attendees, terms> + num_speakers=N — turbo's extra cost is trivial - process_media(path="/tmp/standup.m4a") — audio-only: transcript tools work, frame tools will error - process_media(path="/rec/panel.mov", diarize=true, num_speakers=4) — headcount known? ALWAYS pass it: best accuracy - relabel amend → names become pending with old anchors; stale…

Download ONE public video/audio URL once (this is the only tool that uses the network), then run the same LOCAL pipeline as process_media: transcript, keyframes, OCR, wall-clock, optional diarization. Supported: direct https:// links to a media file (mp4/mov/webm/mkv/ogv/m4a/mp3/wav/ogg/flac), one public YouTube video (watch, youtu.be, shorts, a completed live), and any public video PAGE yt-dlp can read — Instagram (public reels/posts), TikTok, Wikimedia Commons, pages with an HTML5/HLS player, other sites as far as their yt-dlp extractor works anonymously (Vimeo does not). Not supported: playlists, channels, active live streams, private/members-only/age-restricted/DRM videos, cookies/logins; sites that hide a video behind a login or a bot wall fail with a clear reason. The downloaded source is kept inside the job, so extract_frame works later without network; the raw URL is never stored (only a hash, the provider id/host and a bounded title). A repeat call on the same URL serves the stored job without touching the network unless refresh=true. Job ids stay content hashes: the same video from two URLs is one job. YouTube and other pages need the optional [url] extra. The provider's upload date is NOT the recording start: wall_clock stays null unless recorded_at is passed. When NOT to use: for local files (process_media), or to re-fetch data you already processed (use the retrieval tools). Examples: - process_url(url="https://youtu.be/nHfGfEiVdE8") — one public YouTube video, defaults are right - process_url(url="https://www.youtube.com/watch?v=ID&list=PL...") — the playlist part is ignored: ONE video - process_url(url="https://cdn.example.com/recordings/standup.mp4") — direct https link to a media file - process_url(url="https://www.tiktok.com/@nasa/video/7…") — a public video page; origin.provider names the site - meeting from a link: process_url(url=..., diarize=true, num_speakers=3, vocabulary="Vera, Tom, OKR") - non-English narration: process_url(url=..., model=…

Case-insensitive word search across BOTH transcript segments and frame OCR text. The default match_mode="all_words" requires EVERY query word as a substring; match_mode="any_word" requires at least one (ё and е are interchangeable). Hits carry source (transcript|ocr), t_ms, t_wall when known, the matched text, and the nearest frame position — everything needed to jump straight to evidence. Optional speaker accepts a raw label ("S2") or saved name and narrows to that voice's transcript hits. Duplicate saved names search all matching labels honestly. No embeddings. When NOT to use: fuzzy/semantic questions ("anything about performance?") — page get_transcript and read; regex is not supported. Examples: - search(job_id="...", query="login") — every spoken or on-screen mention of login - user: "what did I say about the login button?" → search(job_id, "login button") → get_moment at hits - search(job_id, "TypeError") — on-screen stack traces and error text are OCR-indexed; great for bug repros - search(job_id, "€49") — prices, IDs, and literals on screen are findable via OCR - take hit.t_wall and grep your server logs ±30 s around it to pair remark ↔ log line - broad lexical recall: search(job_id, "timeout latency", match_mode="any_word") - multi-word = ALL words as substrings, any order: "first phase" hits "the first real phase" - stems beat inflected phrases: "кнопк отправк" finds «Кнопка отправки» and «кнопку отправки» - every hit has nearest_frame_ms → get_frames(job_id, at_ms=<that>) shows the moment - diarized job: transcript hits carry "speaker" — "who mentioned the deadline?" is answered by the hit itself - search(job_id, "deadline", speaker="S2") — only S2's mentions; OCR hits are excluded (screens have no voice) - search(job_id, "deadline", speaker="Vera") — saved names are matched case-insensitively - audio-only job → transcript hits only (there is no OCR index) - anti-example: "summarize the pricing discussion" → get_transcript(format="text") and read it - a…