Python SDK

The engine is a plain Python package with no dependency on the API or the web UI. Everything the HTTP API does, it does by calling the functions on these pages; the API adds streaming over HTTP, per-request option validation and host-level caps, nothing else. If you are writing Python, call the library.

Install

pip install "webgraph[render] @ git+https://github.com/BeastxD7/webgraph#subdirectory=packages/engine"
playwright install chromium

Python 3.12 or newer. The render extra installs Playwright; playwright install chromium downloads the browser it drives. Without either, everything still works from the plain HTTP fetch alone and says so (reading_order_method is dom-fallback, render_error is "rendering not available"); with them, every page is also rendered and its layout measured, which is what makes the reading order a measurement rather than a guess.

Ten lines

from webgraph import resolve_page, select_content, to_markdown

page = resolve_page("https://docs.python.org/3/tutorial/introduction.html")
print(page.strategy, page.document.reading_order_method)   # union geometric-anchored

# The whole page, in reading order. This is the default output: nothing a reader sees
# is left out, and the engine says how it read the page.
print(to_markdown(page.document))

# Opt in to the content alone: navigation, footer and boilerplate taken out.
body = select_content(page.document.blocks, title=page.document.title)
print(to_markdown(page.document.model_copy(update={"blocks": tuple(body.blocks)})))

A whole site streams as it goes:

from webgraph import SiteConfig, stream_site

config = SiteConfig(max_pages=50, within_path=True)
for event in stream_site("https://docs.python.org/3/tutorial/", config=config):
    if event["type"] == "page":
        print(event["url"], len(event["markdown"].split()), "words")
    elif event["type"] == "done":
        print(event["pages_ok"], "pages in", event["duration_seconds"], "s")

Measured as written: the first snippet resolves the tutorial page in about 3 s (both fetches), 164 blocks, and select_content keeps 116 of them; the crawl reads three pages in 11 s at the default politeness of one page a second per host.

What the web UI does, as library calls

Everything on the site is a call into this package. There is no capability the UI has that the library lacks; the UI adds a screen and the API adds HTTP.

What is in the box

Every name below is importable from webgraph itself, and its signature is kept stable across minor versions. Everything else in the package is importable from its module and documented there, but may move.

webgraph.__version__ is the installed version.

Output formats

The engine produces one data structure, a Document of ordered blocks, and every format is a view of it:

How the reading is done

There is one mode today, and it is deterministic: rules and two small gradient-boosted models shipped as JSON (a page-type router and an optional per-block classifier), no language model anywhere in the path. The same page read twice gives the same output, which is what makes the benchmarks reproducible. An LLM-assisted or hybrid mode is a possible future addition and would be a separate, opt-in setting; nothing on these pages depends on one.