POST /api/site/stream
Crawl a whole site over Server-Sent Events - technology analysis, discovery, frontier, one event per page, and a final tally.
POST /api/site/stream crawls a site breadth-first from its root and streams an event per stage and per page. Discovery and extraction are interleaved: each extracted page's links extend the frontier, so the first result arrives in seconds and the crawl reaches everything reachable, not only what the sitemap lists. The finished graph is kept in memory and on disk for /api/site/context.
Request
{
"url": "https://docs.example.com",
"max_pages": 200,
"concurrency": 6,
"complete": true,
"crawl": { "max_depth": 4, "strict_domain": true },
"fetch": { "respect_robots": true },
"renderOptions": { "settle_ms": 900 }
}| Field | Type | Default | Meaning |
|---|---|---|---|
url | string | required | Site root. A redirect is followed and the crawl is scoped to where it lands |
max_pages | integer 0–100000 | 0 | Pages to extract. 0 means unbounded: until the frontier is exhausted |
concurrency | integer 1–12 | 6 | Pages fetched in parallel |
complete | boolean | true | true unions the plain and browser fetch per page (strategy: union); false is static-only. Neither mode alone is complete |
crawl | CrawlOptions | null | Depth, domain scope, delay, discovery limits, robots, chrome removal |
fetch | FetchOptions | null | Plain-fetch overrides, applied to every page |
renderOptions | RenderOptions | null | Browser overrides, applied to every page |
The model has no schema field, so done.site_facts and fact_sources are always empty on this route; the measured per-site strategy (CRAWL_STRATEGY = None) is reachable from the library but not from here, since complete always sets one.
Caps
The host's environment caps are applied on top of the request and the run header reports the values actually used. WEBGRAPH_MAX_PAGES clamps max_pages, and 0 from the client is the largest ask, so it clamps down to the cap. WEBGRAPH_MAX_CONCURRENCY clamps concurrency. WEBGRAPH_MAX_CONCURRENT_CRAWLS (default 3) bounds crawls in flight; a request beyond it first receives {"type": "stage", "stage": "analyze", "message": "Waiting for a crawl slot"} and then waits.
Events, in order
Framing is as for /api/text/stream. After the slot is acquired the run header arrives with mode: "site" and the applied max_pages, concurrency, strategy, complete, max_depth, strict_domain.
stage
analyze (Detecting technology stack, then Redirected to … if the root moved), enumerate (Seeding from sitemap), extract (Crawling and extracting up to N pages, with unlimited: true|false).
analysis
The root fetched both ways, measured. root, frameworks, technologies (each name, category, version, confidence, evidence - the fingerprint that matched), payload_sources, render_required, render_loses_content, static_chars, rendered_chars, union_chars, static_coverage, and strategy, the one the crawl will use.
metadata is what the root page declares in its <head>, read once here: title, description, canonical, language, charset, robots, generator, author, keywords, theme_color, viewport, icons, manifest, open_graph (every og:* and article:* property), twitter (every twitter:*), alternates (hreflang → href, capped at 24, with alternate_count whole), feeds (RSS and Atom), schema_types (the @types of its JSON-LD and microdata), and declared_elsewhere: the declarations that name a different site from the one that served the page, as "canonical -> https://old-host.example". Empty is the normal case; a site that moved domains and kept its old metadataBase is what fills it, and nothing on the page shows it. Every address is absolute; every text value is folded to one line and capped at 300 characters. The single-page stream's resolve event carries the same object for its page.
discovery
How the site wants to be found.
{
"type": "discovery",
"robots": {
"found": true, "url": "https://docs.example.com/robots.txt", "fetched_status": 200,
"group": "User-agent: *", "rules_for_us": ["Disallow: /search"], "crawl_delay": 1.0,
"text": "User-agent: *\nDisallow: /search\n", "text_truncated": false, "text_chars": 34
},
"sitemaps": {
"attempts": [
{ "url": "https://docs.example.com/sitemap.xml", "status": 200, "ok": true, "urls": 412, "index": false, "source": "conventional" }
],
"found": 1, "total_urls": 412
},
"seeds": 380,
"common_crawl": {
"status": "seen", "index": "CC-MAIN-2026-34", "index_name": "August 2026 Index",
"urls": 1952, "records": 2000, "sample": ["https://docs.example.com/", "…"], "error": null, "queued": false
}
}common_crawl is what Common Crawl's latest index last saw on this host -- asked once, in the background, with nothing asked of the site (CRAWL_COMMON_CRAWL, on by default). It is its own fact beside the site's: a monthly crawl of the whole web, months stale, chosen by someone else; urls are distinct same-site page addresses (capped at 2,000), records the index rows before deduplication, sample the first fifty. status is seen, not-seen (the host is not in that index) or unavailable (the index refused or did not answer within eight seconds -- it often does; error says which). Nothing from it is queued unless crawl.seed_from_common_crawl is set, in which case the addresses go in at depth 1 cited via: "common-crawl" and queued is true. null when not asked.
text is capped at DISCOVERY_ROBOTS_TEXT_CHARS (4000); rules_for_us is never cut. source is robots (a Sitemap: line), conventional (/sitemap.xml, /sitemap_index.xml) or index. ok means the response parsed as a sitemap and carried a <loc>. seeds is how many sitemap URLs the frontier accepted after scope and normalisation.
frontier
The queue after seeding: queued, discovered, depth_counts (URLs per depth), discovered_kinds, from_sitemap, extracted (0), new_urls (the root plus every seed - clients rebuild the discovered set from these deltas). discovered_kinds tallies addresses by kind: page, pdf, image, other_file, archive, category, tag. refused tallies the addresses the crawl met and turned away, once each, by reason: off-site (another site; a subdomain counts unless allowed), past-depth, not-a-page (mailto:, javascript:, a template's /undefined), excluded / not-included (the crawl's own patterns), queue-cap. The same tally rides on every page event, so a client can show it moving; the done event adds refused_total and refused_urls, the first 200 refused addresses with their reason, as evidence. Files are not refusals -- they are skipped, with the page that linked to each.
fetching
Emitted before each batch goes out: urls, queued, extracted, failed. It is what lets a client show work in flight rather than only work finished.
page
One per page, whether it succeeded or not.
| Field | Meaning |
|---|---|
index, url, title, depth | Position in the run, final URL, first heading, links from the root |
citation | How the address entered the crawl: via (seed, sitemap, link), found_on, anchor (the text a reader would have clicked), depth. May be null |
ok, error | error is HTTP 404, or the engine's refusal message |
chars, blocks, images, tables | Size of the document |
markdown, content_markdown, content_blocks, content_methods | As on /api/text, plus site-chrome once enough pages exist (CHROME_MIN_PAGES = 6) |
page_type, page_type_confidence, page_type_reasons, page_type_runner_up | The router's verdict with its reasons |
canvas | Set when the page draws its content in a <canvas> rather than writing it: canvases, words (readable, under 40), script_bytes (how much of its own script was read for what the canvas shows). null for every other page |
links_out | Every address the page points at beyond this site, recorded and never queued. external: real links on the page, each url and anchor, capped at 60. in_script: addresses the page's own scripts store as named values (github, live, url), each url and key -- read only for a canvas page, found in code rather than on the page, capped at 60 |
render_note | Set when the browser was shown a gate that hid the page -- one hidden element holding more of the page's own prose than the whole visible render (a country picker, a consent dialog with the body set display: none behind it) -- and the plain fetch's page was kept whole behind it. null otherwise; a hidden menu or a collapsed section is not a gate and stays hidden |
strategy, static_chars, rendered_chars | How this page was fetched, and what each fetch gave in characters of readable text. A union with rendered_chars 0 is the plain fetch's page alone: the browser produced an empty page (a WebGL scene in a headless browser does) |
queued, discovered, depth_counts, discovered_kinds, extracted, failed, newly_queued, new_urls, pages_per_minute, totals | Running counters; totals has chars, markdown, images, tables |
graph | pages, sections, entities, links, mentions in the graph so far |
warning
code: "identical-content" when IDENTICAL_CONTENT_WARNING (3) distinct URLs return byte-identical text: the site is probably serving an interstitial the engine could not open. Carries message, urls, chars.
done
{
"type": "done", "pages_ok": 198, "pages_total": 200, "failed": 2,
"discovered": 1431, "remaining_queued": 1231, "exhausted": false, "stopped": false,
"total_chars": 2105433, "total_markdown_chars": 2380911, "total_images": 640, "total_tables": 37,
"largest_identical_group": 1, "identical_content_groups": [],
"chrome_blocks": 42, "chrome_slots": 3, "entities": [], "site_facts": {}, "fact_sources": {},
"graph": { "pages": 198, "sections": 2210, "entities": 0, "links": 9120, "mentions": 0 },
"duration_seconds": 412.6
}refused, refused_total and refused_urls (see frontier) say what the crawl turned away and why -- the counterpart of what it found. A site whose sitemap and canonicals name another host shows every one of its own addresses here as off-site, which is the fact a reader needs and could not see before.
exhausted is true only when the frontier emptied on its own; stopped is true when the client disconnected and the engine's stop flag ended the run. A crawl that hit max_pages has both false and a non-zero remaining_queued.
error
{"type": "error", "message": "..."} and the stream closes. Before any page: not a crawlable URL: … or the root's own refusal (site unreachable, a wall, a refused address). Individual page failures are page events with ok: false, never error.
Politeness and robots
The crawl reads robots.txt once, skips disallowed URLs at the frontier (crawl.respect_robots) and again per page (fetch.respect_robots), and waits the larger of delay_seconds and the file's Crawl-delay between fetches per worker.
What unbounded means
max_pages: 0 runs until the frontier is exhausted. On a large site that is a long time: the first live whole-site run, vtu.ac.in on 14 September 2026, discovered 17,126 URLs and ran for roughly six hours, a third of it fetching 7,907 PDFs one by one to refuse each as not HTML. Every page event carries the full Markdown, so an unbounded run also produces a stream of tens of megabytes. Set a cap, or set WEBGRAPH_MAX_PAGES on any host you share.
A client that stops reading does stop the crawl - the engine polls a stop flag between batches - but only after the batch in flight finishes. Close the connection rather than ignoring it.
curl -N http://127.0.0.1:8000/api/site/stream \
-H 'Content-Type: application/json' \
-d '{"url": "https://docs.example.com", "max_pages": 50}' \
| grep '"type": "page"' | jq -c '{url, ok, page_type, chars}'