Request options
FetchOptions, RenderOptions and CrawlOptions - the per-request overrides, their defaults from config.py, and what each one does.
Three request fields carry per-run overrides: fetch, renderOptions (alias render_options) and crawl. Every field in them is optional and null by default; a null leaves the engine's default in place, and the defaults are the constants in packages/engine/src/webgraph/config.py quoted below. GET /api/config returns the same list under overridable, so a client can discover it at runtime. Anything not listed here is changed by editing config.py, not per request.
Overrides are applied by replacing fields on the engine's config dataclasses (FetchConfig, RenderConfig, SiteConfig). On a crawl, the host's environment caps still apply on top.
FetchOptions
Accepted by /api/text, /api/text/stream and /api/site/stream as fetch. Governs the plain HTTP fetch, which every strategy starts with.
| Field | Type | Default | Meaning |
|---|---|---|---|
timeout_seconds | number 1–120 | 20.0 (FETCH_TIMEOUT_SECONDS) | Seconds to wait for a plain HTTP response |
retries | integer 0–5 | 1 (FETCH_RETRIES) | Extra attempts after a 429/503 (honouring Retry-After up to 5.0 s) or a transport error. A definite answer such as 403 or 404 is never retried |
respect_robots | boolean | false (PAGE_RESPECT_ROBOTS) | Off by default: the engine reads every page it can reach. true asks the host's robots.txt before fetching and refuses a page it disallows for this client, quoting the rule and naming what the site offers instead. Politeness (one page a second per host, Crawl-delay) applies either way |
The fetch speaks HTTP the way a browser does - HTTP/2, brotli, the Sec-Fetch-* navigation headers - under a User-Agent that still names webgraph and carries a contact URL. Those are not overridable per request. The robots check asks for the webgraph token (ROBOTS_AGENT_TOKEN), not the browser prefix, and caches a host's file for ROBOTS_CACHE_SECONDS = 3600.
{ "fetch": { "timeout_seconds": 45, "retries": 2, "respect_robots": true } }RenderOptions
Accepted by the same three routes as renderOptions. Governs the browser, and therefore only matters when a render runs: render: true, a shell escalation on /api/text, or a crawl with complete: true.
| Field | Type | Default | Meaning |
|---|---|---|---|
timeout_ms | integer 1000–120000 | 30000 (RENDER_TIMEOUT_MS) | Milliseconds a navigation may take before the page is read as-is, or refused if it holds fewer than MIN_SALVAGED_TEXT = 200 characters |
wait_until | commit | domcontentloaded | load | networkidle | "load" (RENDER_WAIT_UNTIL) | When the navigation counts as finished. networkidle never fires on sites with analytics beacons or polling; measured, it timed out on 5 of 24 real sites |
settle_ms | integer 0–10000 | 900 (RENDER_SETTLE_MS) | Pause after load for hydration and layout to settle before geometry is measured |
dismiss_gates | boolean | true (RENDER_DISMISS_GATES) | Click through a first-run interstitial (persona picker, age gate) that blocks the page from mounting. One guarded click on a page with at most 4000 characters and one internal link, kept only if the text grows 1.5× |
reveal_collapsed | boolean | true (RENDER_REVEAL_COLLAPSED) | Open collapsed content before measuring, without clicking: <details>, ARIA disclosures, tab panels (role="tab"), and panels a control names (data-bs-target, href="#id"). Never inside nav/header/footer or a menu. Measured: recall unchanged, extra words +0.001 mean; what changes is that content behind a click is measured and ordered where it sits |
click_collapsed | boolean | true (RENDER_CLICK_COLLAPSED) | After the page is measured, click open what the reveal could not — tabs, accordions and "show more" panels wired in JavaScript alone — and measure again. Only when at least 30 words are hidden outside the site's chrome; at most 10 controls, never a link elsewhere, a form control, or a "buy / sign in / cart" label; a click that opens a popup is closed and not counted; a click that navigates or empties the page ends the step and what was measured stands. Everything opened is forced visible so a tab set ends with every panel showing |
viewport_width | integer 320–3840 | 1440 (RENDER_VIEWPORT_WIDTH) | Viewport width in CSS pixels. Width decides reading order: a narrow viewport collapses columns into one |
viewport_height | integer 320–2160 | 900 (RENDER_VIEWPORT_HEIGHT) | Viewport height in CSS pixels |
Images, media and fonts are never downloaded (RENDER_BLOCK_RESOURCES); fonts shift metrics by a pixel or two, and reading order does not care. The browser runs headless and is reused across pages on a thread; neither is overridable.
{ "renderOptions": { "wait_until": "networkidle", "settle_ms": 2000, "viewport_width": 390, "viewport_height": 844 } }A render that fails or times out does not fail the request. The result degrades to the static document and render_error on the resolve event says why.
CrawlOptions
Accepted by /api/site/stream only, as crawl. max_pages, concurrency and complete are top-level fields of the request, not options here.
| Field | Type | Default | Meaning |
|---|---|---|---|
max_depth | integer 0–50 | 12 (CRAWL_MAX_DEPTH) | Links away from the root the crawl goes. 0 is the root alone; 1 adds everything it links to or its sitemap lists. Breadth-first, so a page budget is spent near the root |
strict_domain | boolean | true (CRAWL_STRICT_DOMAIN) | Stay on the root's host (www.example.com and example.com are one host). false also follows subdomains, never other sites |
common_crawl | boolean | true (CRAWL_COMMON_CRAWL) | Ask Common Crawl's index what it last saw on the host, reported in discovery.common_crawl as its own fact. Nothing is asked of the site |
seed_from_common_crawl | boolean | false (CRAWL_SEED_FROM_COMMON_CRAWL) | Also queue what Common Crawl listed, at depth 1, cited via: common-crawl. Off, because a stale listing is full of pages that are gone and each would be fetched and recorded as a 404 |
within_path | boolean | false (CRAWL_WITHIN_PATH) | Stay under the start address's path: a crawl of example.com/docs/guide follows /docs/... and turns /blog/... away as not-included. Off, because a site is the unit this engine reads |
include_paths | string | "" (CRAWL_INCLUDE_PATHS) | Comma-separated regular expressions searched in the address's path (never the host). When set, only matching addresses are crawled; the rest are counted as not-included in the run's refused. A pattern that does not compile is a 422 |
exclude_paths | string | "" (CRAWL_EXCLUDE_PATHS) | Comma-separated regular expressions searched in the path; a match is turned away as excluded, whatever else says |
delay_seconds | number 0–10 | 0.3 (CRAWL_DELAY_SECONDS) | Pause between fetches per worker. Raised to the site's Crawl-delay when that is larger |
verify_inventory | boolean | true (CRAWL_VERIFY_INVENTORY) | Check each sitemap URL with a cheap request before spending a page on it, so a stale sitemap cannot burn the budget on 404s — accepted but not applied by /api/site/stream: it drives the CLI's webgraph site enumeration (extract_site) only; the streaming crawl discovers by links continuously and has no separate verification pass |
follow_links | boolean | true (CRAWL_FOLLOW_LINKS) | Discover pages by following links as well as reading the sitemap — accepted but not applied by /api/site/stream: it drives the CLI's webgraph site enumeration (extract_site) only; the streaming crawl discovers by links continuously and has no separate verification pass |
discovery_limit | integer 0–10000 | 400 (CRAWL_DISCOVERY_LIMIT) | URLs harvested by link-following before verification — accepted but not applied by /api/site/stream: it drives the CLI's webgraph site enumeration (extract_site) only; the streaming crawl discovers by links continuously and has no separate verification pass |
sitemap_limit | integer 0–200000 | 50000 (CRAWL_SITEMAP_LIMIT) | URLs read from sitemaps, at most; sitemap indexes are followed up to MAX_SITEMAP_DOCUMENTS = 20 files |
respect_robots | boolean | false (CRAWL_RESPECT_ROBOTS) | Off by default: the crawl explores every page it can discover. true turns away at the frontier what robots.txt disallows for this client (the file is read for its sitemaps either way). Distinct from fetch.respect_robots, which checks per page inside resolve_page. The Site Report always obeys the file, since it measures what the site declares |
remove_chrome | boolean | true (CRAWL_REMOVE_CHROME) | Produce content_markdown with landmarks, cross-page site chrome and boilerplate removed. Chrome is a block seen on CHROME_THRESHOLD = 0.9 of pages, judged once CHROME_MIN_PAGES = 6 pages exist, never removing more than half a page |
main_content | boolean | true (CRAWL_MAIN_CONTENT) | Also draw the main-content boundary when producing content_markdown. Turn off for a site whose pages are link hubs by design, where the list of links is the content |
{
"url": "https://docs.example.com",
"max_pages": 300,
"crawl": { "max_depth": 3, "strict_domain": false, "delay_seconds": 1.0, "main_content": false }
}A crawl of one section, with the rest of the site counted but not fetched:
{
"url": "https://www.gov.uk/",
"max_pages": 200,
"crawl": { "include_paths": "^/government/", "exclude_paths": "/publications/" }
}The done event then says what was turned away and why -- on gov.uk, tens of thousands of addresses as excluded and not-included, with the first two hundred named -- so the scope is a fact about the run, not an unspoken filter.
strict_domain: false turns a bounded crawl into a potentially unbounded one: blog., shop. and status. subdomains are usually separate applications. Pair it with max_pages.
Site context and graph
POST /api/site/context assembles a bounded context bundle from a crawled site; the graph routes expose what the crawl learned.
Errors and refusals
Every refusal names its reason. The error classes, the HTTP status or event each becomes, an example message, and what to do about it.