io.github.tlq5l/agentic-video

List non-speech sound attacks (hits, claps, beats) with millisecond timestamps and five-band energy profiles, optionally restricted to a frequency band (band_hz: sub, low_mid, mid, high, air, or [low_hz, high_hz]) and compared with another source (compare_to). min_strength 0-1 (default 0.12): raise it for only strong onsets. min_gap_sec (default 0.035 s) merges onsets within that gap of a group's first onset into one onset at the first time with the group's strongest strength; 0 keeps every raw onset. At most 500 onsets are listed; `total` is the full count and the result says how to see the rest. Needs an audio track (probe_video has_audio); audio-only files (.m4a, .mp3, .wav) work. For level over time use loudness_timeline; for speech use transcribe. To name what a sound is, use label_sounds (cloud).

Start or poll construction of the local visual frame index that search_visual needs. Call it once, then repeat the same call (or search_visual, which reports status=index_required or partial) until the status is complete; frames are processed in bounded chunks. count_occurrences, contact_sheet and the inspect tools do not need the index.

Find stretches where the picture updates less often than the file's frame rate, e.g. a 30 fps capture padded to 60 fps so every frame is doubled (judder). It decodes every frame at 160x90 grey and calls a frame a repeat when it is near-identical to the one before; a range is a stretch of at least 0.5 s where repeats fill at least a quarter of every half second while other frames still change. Each range gives start_sec, end_sec, frames, duplicate_frames and effective_fps (unique frames per second; compare with frame_rate). Runs of repeats longer than 0.5 s are holds (still shots, title cards, stalls) and are not ranges; held_frames counts them (use detect_cuts and the frozen-picture check in `python -m src.loop check` for those). It does not judge whether a range is intended: a 24 fps film in a 60 fps file is one range covering the whole file, and a slow, smooth pan can look like repeats. Look at a range with inspect_window to see what moves there. At most 500 ranges are listed; `total` is the full count. Reads the whole file: about 1 s per 700 frames of 1080p; there are no range arguments.

Get ONE grid image of evenly spaced frames, with burned timestamp badges, across the whole video or a start_sec/end_sec range: the cheapest way to see what a video contains before searching. Default 12 samples, at most 150 (larger values are clamped and the result says so). Use it to survey; use search_visual to find a specific thing, overview_chunks to survey a long video as ordered sheets, and inspect_window or inspect_windows for dense consecutive frames of a short span.

Count how many separate times a specific symbol/object or exact on-screen text appears. Use for questions like 'how many separate times does the yellow outlined circle appear (ignore other colours)', 'how many times does the exact word RELEASE appear (not RELEASED)', 'how many times does the green diamond cross the screen'. Give `description` (colour and shape words are enforced, so look-alikes in another colour or shape do not count) and/or `text` (whole-word match by offline OCR). Returns `count` plus one span per continuous appearance (start_sec, end_sec, representative_sec, region, travel, weak); gaps of min_gap_sec or more separate spans. Verify a span with inspect_window at its representative_sec. Not for periodic reps or cycles (use count_repetitions) and not for locating one moment (use search_visual, which ranks frames, is capped at top_k 20 and cannot count). No visual index needed. Repeat the same call while status=in_progress. Lists at most max_spans spans (default 20, maximum 100; larger values are clamped and the result says so); `count` includes any omitted. If 100 or more spans match, the description is too broad: `count` is null, `capped` is true and `spans_found` says how many matched, so narrow the description or window and rerun instead of reporting a number. A min_gap_sec above 0.3 can join separate appearances; then `gaps_merged` is set, and `count_without_merges` too when no `direction` is passed (with a direction, rerun with the default min_gap_sec to see the separate appearances). The count is a rough first pass: it can merge separate appearances that are split by a scene cut, split one appearance into many spans, and count appearances rather than crossings; `direction` (e.g. left_to_right) narrows the count to appearances that travel that way, plus spans too brief to show motion (travel unknown), which count under any direction: check those with inspect_window. The result also carries `contact_sheet`, one labeled image with a tile per list…

Count completed cycles of a repeated visual action (jumps, claps, bounces, swings) inside one window of at most 180 seconds. Give `region` (fractions of the frame: x, y, w, h) to isolate the moving thing. Repeat the same call while status=in_progress; verify peak badges and ambiguous edges with inspect_window. For separate appearances of a specific symbol or exact on-screen text use count_occurrences; to locate one moment use search_visual. For a longer clip call once per window, cutting at pauses between cycles so none straddles an edge. The count is a rough first pass and can be far off on real footage. Before answering, count cycles by eye with inspect_window or contact_sheet over the window; if your count differs, trust your own.

List scene changes (timestamp_sec, score, kind) in the whole video or a start_sec/end_sec range. Hard cuts and animated changes (fades, dissolves, slides, wipes, zooms) are both returned; `kind` tells them apart: "cut" or "transition". A transition's timestamp is its middle. Scores of the two kinds are not comparable. threshold 0-1 (default 0.25) applies to hard cuts only: raise it to keep only stronger cuts. A hard cut between two similar pictures is found by a second check and listed as a cut with a score below the default threshold; any threshold above 0.25 leaves these out. A change that is over within about 4 frames (a very fast wipe) is listed as a cut, and a few similar-picture cuts still come back as a transition. Film footage with heavy motion can still give false transitions; check a doubtful one with inspect_moments. At most 500 cuts are listed; `total` is the full count and the result says how to see the rest. Use it to segment a video into shots or find edit points; use audio_onsets for sounds and sync_check to compare both.

Use this when you need one exact frame at full size, or a crop to read small text; for many frames use inspect_window or contact_sheet. Returns one PNG of the frame shown at time_sec, at the source's native resolution, with no timestamp drawn on it. An optional crop (x, y, width, height in source pixels) cuts a rectangle out of that frame, and scale (1 to 4, default 1) enlarges the crop with Lanczos so small text is easier to read. frame_time_sec is the start time of the frame actually decoded: the last frame starting at or before time_sec, so on variable-frame-rate video it can be well before time_sec. A repeat call with the same source, time_sec, crop and scale returns the saved file. With read_text true it also returns the on-screen text read from that frame (Apple Vision, macOS only).

Look closely around specific instants: for each moment (timestamp_sec, optional label) returns a contact sheet of frames before, at and after it (default 4 s before and 4 s after, samples_per_moment 9). Use it to verify a hit from search_visual, search_transcript, detect_cuts, audio_onsets or track_region_changes. At most 16 moments per call and max_total_samples frames in all (default 144, maximum 400); a call over budget fails and says what to lower. For one continuous span of frames use inspect_window.

Dense consecutive frames of ONE short window [start_sec, end_sec] (both required) with timestamp overlays, as a contact sheet plus frame list. Default 4 fps, maximum 30 (larger values are clamped). At most about 150 frames and 600 seconds per call: frames = fps x duration, so a 10 s window allows at most 15 fps; a call over budget fails and says the fps that fits. Use it to verify a moment or a count. For a wide overview use contact_sheet; for several ranges use inspect_windows. With read_text true each frame also lists the on-screen text read from it (Apple Vision, macOS only).

Dense consecutive frames (default 4 fps) for several time ranges in one call, with optional context_sec padding and merging of overlapping windows. At most 16 windows and max_total_frames frames in all (default 150, maximum 500); a call over budget fails and says to lower fps or shorten the windows. For one range use inspect_window, for a coarse survey use contact_sheet, for frames around single instants use inspect_moments.

Name the sounds in a stretch of a video's audio: cuts [start_sec, end_sec] (both required) into windows of window_sec (default 5 s, at most 60 windows per call) and returns, per window, the chance of each label plus none_chance, from a cloud model (Clef-omni on Cloudflare Workers AI). THIS SENDS THE AUDIO TO CLOUDFLARE: never use it on personal recordings. The server needs CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN (or AGENTIC_VIDEO_CF_TOKEN_COMMAND, a command that prints the token) and the call fails with a message naming what is missing. `labels` (default: 20 common sounds such as dog barking, siren, glass breaking, church bells) is the closed list to choose from, so a sound outside the list is forced onto the nearest label or none: list every sound that may be present. Steady musical chords can be named church bells, so treat top_chance under 0.5 as weak and check a hit with spectrogram_window. top_label is null when none_chance beats every label. It does not search: find candidate times with audio_onsets or loudness_timeline first, then label those ranges. For speech use transcribe or search_transcript. Needs an audio track. No window starts after about 40 s and the call answers within about 50 s: a longer range returns the windows finished in time with complete=false and next_start_sec to continue from.

Measure loudness over time: RMS, peak, band energies, spectral centroid, silence gaps, and band-energy outliers classified as steps, ramps or transients, in fixed windows (window_sec 0.01-600, default 0.25) or beat-length windows (bpm 20-300; give one or the other). At most 500 windows are listed in `windows` (`total` is the full count; `outliers` and `silence_gaps` always cover the whole range); use a larger window_sec or a start_sec/end_sec range for the rest. Ranges needing over 50,000 windows are rejected with the window_sec that fits. Needs an audio track (audio-only files work too). For discrete hits use audio_onsets. `peak_dbfs` (top level, with `peak_sec`, and per window) is the loudest sample on any channel, as ffmpeg reports sample peak; `rms_dbfs` and `band_db` are measured on the mono mix (L+R)/1.414, so on centred stereo they read up to 3 dB above a per-channel meter.

Survey a long video as consecutive contact sheets, one per chunk_sec (default 60 s), so every stretch is covered in order. At most max_chunks sheets per call (default and maximum 32): for a longer video raise chunk_sec or page through it with start_sec/end_sec; a call over budget fails and says what chunk_sec would fit. samples_per_chunk defaults to 8 (maximum 150, larger values are clamped). Use contact_sheet instead when one sheet is enough.

Read a video's duration, frame rate, resolution, codecs, and whether it has an audio track and a video track. Call this first: duration_sec bounds every start_sec/end_sec you pass to other tools. Audio-only files (.m4a, .mp3, .wav) are accepted: has_video is false and width, height and fps are 0. has_audio tells you whether transcribe, search_transcript and the audio tools (audio_onsets, loudness_timeline, spectrogram_window) can work; the visual tools and sync_check need has_video true. `source` is a local file path or a video URL and is the same in every tool: an x.com / Twitter post with a video, a YouTube link, another site yt-dlp supports, or a direct .mp4 link. A URL is downloaded once (up to 1080p) and cached; the result gives source_url and local_path, and while the download runs it is status=in_progress, so call again.

Zoom into candidate time windows: splits each window into `fanout` contiguous child ranges and returns one timestamped contact sheet per child, without ranking relevance. Use it after contact_sheet or search_visual has narrowed the answer to a few windows and you want to localize inside them; use inspect_windows for dense consecutive frames instead. Budget: windows x fanout x samples_per_child must be at most max_total_samples (default 96, maximum 300); a call over budget fails and says which of the three to lower.

First look at a video in ONE call: container metadata (duration, size, fps, codecs, has_audio), a contact sheet image with burned timestamp badges (default 24 samples; `samples` up to 150, larger values are clamped and the result says so), every cut and transition from detect_cuts, every near-black stretch of at least 0.2 s (`black_ranges`: start, end and duration; at most 50 are listed and `black_total` counts them all), and a loudness summary (average and peak level, silence gaps, the strongest band-energy outliers with their times). It runs probe_video, contact_sheet, detect_cuts, the black scan and loudness_timeline for you, so start here instead of calling them one by one. A first call on a 10-minute 1080p clip takes about 35 s (the cuts scan and the black scan share one decode of the file); repeat calls on the same file return from cache. `loudness` is null when the video has no audio track. Use the times it returns to zoom in with inspect_window or spectrogram_window; for a different threshold, window or time range call detect_cuts or loudness_timeline directly. Any step failing fails the whole call with that step's error message.

Find when a word or phrase is SPOKEN. Audio-only files (.m4a, .mp3, .wav) work too. Default mode=hybrid lists exact keyword matches first (match=exact: the phrase; match=all_terms: every content word in one segment), then passages that match by meaning (match=semantic: a 20 s window, found on this Mac with EmbeddingGemma 2; nothing is sent to the cloud), so a reworded question still finds the moment. mode=lexical gives keyword matches only. When no hit is exact or all_terms, the words were not found in the speech: semantic hits are the closest passages, not proof, so read their text before answering. If semantic search is not set up or not ready, hits are keyword-only and `note` says why. Returns up to top_k timestamped hits (default 5, maximum 50; larger values are clamped and the result says so; `note` says when more keyword matches exist). For text shown on screen use count_occurrences, and for visible things use search_visual. If ASR is needed, starts or joins background transcription and returns status=in_progress with retry_after_seconds instead of waiting; repeat the same call after that delay. A hit on a flagged line carries `suspect` (see transcribe).

Find WHEN something visible happens: an object, person, action or scene (semantic match between the query and sampled frames). It matches meaning, not letters, so it cannot find on-screen words or numbers (names, scores, jersey or bib numbers): find those with count_occurrences `text`, which reads text in frames, one window of up to 2400 s per call. Returns up to top_k ranked frame hits (default 5, maximum 20; larger values are clamped and the result says so), merged intervals (the one holding rank 1 grows from its best frames only over neighbouring frames that score close to the best, up to 8 seconds, so it can be shorter; a span already longer than 8 seconds, such as a held frame or hits merged together, is kept whole, so it can be longer and can include lower-scoring hits; intervals that overlap it are merged into it), and a contact sheet to verify them. It ranks frames, so it cannot count: to count separate appearances of a symbol or exact on-screen text use count_occurrences, for a repeated action use count_repetitions, for spoken words use search_transcript. Build the local visual index first (build_visual_index), then repeat the same query to poll until complete. Use start_sec/end_sec to search one range at a time.

Render a log-frequency dB spectrogram PNG for [start_sec, end_sec] (both required, at most 600 s) with absolute timecode badges and optional event marks (time_sec + label). Use a short window (a few seconds) to look at one sound; find candidate times with audio_onsets or loudness_timeline first. Needs an audio track (audio-only files work too).

Check audio/video sync: compares visual cuts with measured audio onsets, and optionally with intended sound cues (time_sec, label, optional band_hz), within tolerance_ms (0-5000, default 80). Returns per-cut and per-cue matches and unmatched strong onsets (the cuts and unmatched_strong_onsets lists show at most 500 items each; the cues list is never cut short; total_cuts and total_cues are the full counts). Each cue reports candidate_onsets_count, the number of onsets near it; verbose true reports that cue's candidate_onsets list instead of the count. Needs an audio track. To look at the two signals separately use detect_cuts and audio_onsets.

Watch one fixed rectangle of the frame (region: x, y, width, height as fractions 0-1) from start_sec to end_sec (both required) and rank the moments its pixels change most, as candidates to verify with inspect_moments or inspect_window. Use for a scoreboard, counter, indicator light or door. Slow continuous motion ranks no candidates: read steady_change.detected for that. Returns ranked candidates, not a count: for a bounded count of state transitions use track_region_events. Budget: about (end_sec - start_sec) x sample_fps frames, at most max_total_frames (default 600, maximum 2000); a call over budget fails and says the sample_fps that fits. This tool's region uses width/height (count_repetitions uses w/h).

Count discrete state changes (transitions) inside one fixed rectangle of the frame (region: x, y, width, height as fractions 0-1) between start_sec and end_sec (both required). Returns event_count plus before/after evidence moments to verify. Use for 'how many times did the light turn on' or 'how often does the counter change'. For ranked change candidates without a count use track_region_changes; for a periodic action use count_repetitions; for a symbol appearing anywhere on screen use count_occurrences. Budget: about (end_sec - start_sec) x sample_fps frames, at most max_total_frames (default 600, maximum 2000); a call over budget fails and says the sample_fps that fits. This tool's region uses width/height (count_repetitions uses w/h).

Get the full speech transcript as timestamped segments (cached after the first run). Audio-only files (.m4a, .mp3, .wav) work too. To find where a specific word or phrase is spoken, use search_transcript instead of reading it all. If the requested model is not cached, returns another cached transcript with cached_model_size and requested_model_cached=false while one background job runs; with no transcript cached, returns status=in_progress and retry_after_seconds, so repeat the same call after that delay. model_size: tiny, base or small (more accurate, slower on the CPU), or turbo (Whisper large-v3-turbo on the GPU via whisper.cpp: the most accurate and, where installed, the fastest, about 1 minute per hour of audio). Omit model_size to get turbo where it is installed, else tiny. Long audio is decoded in 60-second chunks. Lines repeated 3 or more times in a row carry `suspect` and `notice` names them: speech models often invent such lines over music or silence, though a real chant can repeat too. A line without `suspect` is not verified.