WebGraph documentation
Universal web content extraction with provenance and recovered reading order.
WebGraph takes a website — not a page, a website — and returns every public page as clean, ordered Markdown, together with an account of where each piece of information came from and how it was obtained. It detects the site's technology stack, discovers every public route, and extracts rich Markdown with reading order recovered from the rendered layout rather than guessed from the HTML. These pages cover the API, the engine's stages, crawling, watching a site for changes, the benchmarks it is measured on, and how to deploy or contribute.
Getting started
Install the engine and the API, run the web app, and extract a first site.
Python SDK
The engine as a library: every function, parameter and default, with the code to call it.
API
The HTTP endpoints, their request and response shapes, and the streamed crawl.
How it reads a page
Fetching, the union of static and rendered DOM, reading order, and Markdown.
Crawling
Technology detection, route discovery, politeness, and cross-page analysis.
Watch
Run a crawl again and get what changed, section by section, as a list or a feed.
Benchmarks
The corpora, the runners in benchmark/, and how a change is measured.
Deployment
Running the API and the web app in production.
Contributing
Branches, commits, tests, and what a pull request has to carry.