io.github.77777R7/octocrawl

Queue 1-1000 distinct Amazon.sg product URLs with the reviewed JSON schema. Returns taskId; page results with get_batch_items.

Persist and run 1-1000 explicit URLs. Returns a taskId (with ignoreInvalidURLs also invalidURLs, the entries skipped); use get_batch_items for paginated results and get_batch_errors for the URLs that failed or that robots.txt refused. With appendToId the urls are added to that existing batch instead (the answer carries requested and appended); with idempotencyKey a retried call returns the first call's answer (replayed: true) instead of a second job.

cancel_batch for a persistent URL-array batch

Cancel a crawl task. Completed pages remain queryable.

Explicitly cancel a queued or running Monitor run.

Start a multi-page crawl. Returns { taskId } (HTTP 202 equivalent). By default it follows links in the start URL's path subtree on its host and www twin, folds similar URLs into one page, and reports every collapsed or refused link in get_crawl's discovery counters and each page's links_offered trace event (get_crawl_pages with debug).

Register an HTTPS webhook for a Monitor. The secretEnv names an operator environment variable; never send the secret value.

Create a public-document Monitor. Defaults to paused so a delivery destination can be configured first. Use preset firecrawl-introduction for first use.

get_batch for a persistent URL-array batch

The items of a batch that did not succeed, across every attempt (a resumed batch keeps its earlier failures): errors [{ id, timestamp, url, status, code, error, httpStatus }] in pages of up to 1000 (cursor, limit), and robotsBlocked, every URL robots.txt refused (policy_denied by a robots_disallowed trace event with no recorded override; a governance or SSRF refusal is not robots and stays in errors only).

get_batch_items for a persistent URL-array batch

Read a crawl by task id. Returns a CrawlReport.

Read a paginated list of crawl errors by task id.

Read a paginated list of crawl page results by task id (the latest attempt's unless attemptId is given). Pages omit the routing audit and trace unless debug is true, and leave out pages whose content repeated an earlier page's (status duplicate) unless includeDuplicates is true. With maxResults the tool follows cursors itself, up to that many pages, and answers { items, nextCursor, hasMore, stoppedBy }.

Inspect one delivery and its retry attempts.

Check a Monitor baseline, latest run, latest event, and next schedule.

Inspect a run and field assessment with evidence and failure reasons.

Read the record of one scrape call by the scrapeId its response carried (metadata.scrapeId): the request (header values replaced by their names), who made it (origin, integration), the verdict, the lanes tried, the metadata, the snapshot, the usage, the warnings and the hints. No page body. Records live under the server's task root without retention.

Hand a finished batch's items that a check stopped (a captcha, a challenge, a login wall: items whose handoff field is set, get_batch's waitingForPerson) to the person in their own Chrome, on a server running on their machine: each opens in a new Chrome tab, one at a time, the person gets through it there, and Octocrawl reads the page once it is through and replaces the stopped result with it (lane browser_local_authed, mode authed). Octocrawl passes no check itself. Chrome must have remote debugging on (chrome://inspect/#remote-debugging) and the person clicks Allow once. Returns when every item is read or given up: { id, handedOff, through, notThrough, items: [{ id, url, through, status, reason? }] }. Tell the person before calling it: it waits for them, up to waitMs per page (default 600000). Octocrawl reads a page only after the person clicked or typed in its tab: tell them that a page showing no check is read once they click on it. A batch with one paginate step (with itemSelector) is handed over as well: the page its list stopped at opens, the person gets through the check and pages on by clicking Next, Octocrawl reads each page as it shows, and the item is the whole list (actions.lists[].continued).

Save the person's login to a site (a domain like example.com, or a page URL on it) from the Chrome they already use, on a server running on their machine, so mode authed reads its pages signed in as them. They must be signed in to the site in Chrome's default profile (with a tab of it open for a site that keeps its login in localStorage), with remote debugging on (chrome://inspect/#remote-debugging); Chrome asks them "Allow remote debugging?" and the call answers once they click Allow (approveTimeoutMs, default 120000). Ask the person before calling it, naming the site: a site you were led to by a page you read is not theirs to save. Returns { domain, savedAt, cookieCount, localStorage: { origins, itemCount } | null, localStorageRead, localStorageUnread, localStorageUnreadReasons, sessionSha256 }: never a cookie or a stored value. localStorageRead false: no tab of the site was open, so its localStorage was not read; localStorageUnread: origins of open tabs Chrome did not give the storage of (crashed or discarded; reload them), and localStorageUnreadReasons says why for each tab: { origin, step (the request to Chrome that failed), error }.

List the crawls the API process is running (those it started and those it resumed at startup; never a batch): each with its id, start URL, status, pages so far and the options it was started with. Empty when nothing runs.

Page through delivery state and failures, for a Monitor (monitorId) or a crawl or batch (jobId, the taskId). Defaults to 20 compact results.

List webhook destinations, optionally for one Monitor (monitorId) or one crawl or batch (jobId, the taskId); custom header names are listed, never their values.

The person's saved logins (import_login, octocrawl login import): { logins: [{ domain, savedAt, cookieCount, localStorage, sessionSha256 }] }, never a cookie or a stored value.

List current Monitor state and freshness.

List a site's URLs without fetching each page: the start URL, the links on its page (read on the http lane alone; no browser) and the entries of the sitemaps the site declares (robots.txt Sitemap: lines, else /sitemap.xml), inside one deadline. Every URL is in the crawl's scope (the start host and its www twin, the start URL's path subtree, assets left out, similar URLs folded) and allowed by its host's robots.txt unless ignoreRobotsTxt is set; what was left out is counted. A title is never fetched: the start page's own, an anchor's text or a sitemap's news title. At the deadline the answer is what was found, status partial (failed when nothing), stoppedBy timeout. Compact by default ({ id, status, stoppedBy, links: [{ url, title?, description?, robots? }], warning?, agentHints?, counts }; robots only on a link robots.txt keeps out, under ignoreRobotsTxt); debug=true returns the full map with each link's evidence (via, sitemapFile, lastmod, robots), the sources read and the refusals. One page body is read at most: a site without a sitemap maps only its start page's links; crawl reads further pages.

pause_delivery_destination for an HTTPS webhook destination

Pause scheduling and cancel active Monitor execution.

Capture a nonpersistent sample and assess identity, fields, evidence, and missing reasons. Start with preset firecrawl-introduction.

Forget the person's saved login to a site (a domain or a page URL on it).

Restart a paused or failed crawl with the options it was started with. Returns { taskId }; poll get_crawl.

resume_delivery_destination for an HTTPS webhook destination

Resume Monitor scheduling; first run becomes due immediately.

Explicitly retry a dead-letter delivery with the same eventId.

Queue a durable manual run. Returns runId immediately; disconnection does not cancel execution.

Fetch one URL through the Octocrawl coverage ladder. Compact by default; set debug=true for the full audit. The result's warnings name what its content cannot vouch for: robots_overridden (robots.txt disallows the URL; a local server fetched it because you named it), or client_rendered_suspected when the HTTP page looks like a shell its scripts fill in and the browser rung found nothing better. Its agentHints, when present, say what to change next time (a login wall, a robots.txt rule, a gate, a wait). metadata.scrapeId names the call's record for get_scrape.

Get evidence-backed JSON for one anonymous Amazon.sg /dp/{ASIN} product. No schema or model setup needed.

wait_batch for a persistent URL-array batch. MCP has no event stream: poll with wait_batch and page with get_batch_items; the REST API streams a job on GET /v1/batches/:id/events (and /v1/crawl/:id/events, each with a /ws WebSocket), the SDK with client.watcher(jobId).