Intelligence

The optional model beside the parser — Intelligence, Budget, the judge slots it may enter, what every decision records, and the two ways to run the engine.

Two ways to run the engine

Pure Python is the default and the promise: no model, no key, the same output for the same page every time, every word in the Markdown a word that was on the page. Nothing in this page runs unless you ask for it.

With a model you pass llm=Intelligence(...). The model may then judge at a few named slots where the rules are unsure, and later add (image descriptions, structured extraction). It never rewrites text. Every question it is asked, the answer, the probability behind it, the backend, the model and the tokens are recorded on the page as decisions, and any failure — unreachable, rate-limited, timed out, malformed — means the rule's own answer, noted, never an error.

pip install "webgraph[llm]"     # Pydantic AI's provider layer; the pure-Python install has none of it

`Intelligence`

from webgraph.intelligence import Intelligence, Budget

llm = Intelligence(
    provider="groq",                 # groq · openrouter · openai · anthropic · gemini · mistral
                                     # vercel · ollama · deepseek · xai · together · fireworks
                                     # custom (any OpenAI-shaped endpoint, with base_url)
    model="qwen/qwen3.8-27b",
    api_key=None,                    # or the provider's usual env var: GROQ_API_KEY, OPENAI_API_KEY, …
    base_url=None,                   # for custom / self-hosted endpoints
    decisions=("router", "boundary"),  # the judge slots the model may enter; () = judges nothing
    thresholds={"boundary": 0.95},   # override a slot's default threshold
    budget=Budget(max_requests=200, max_input_tokens=2_000_000, max_output_tokens=200_000),
    timeout_seconds=8.0,             # a call inside the parse waits this long at most, then the rule answers
)

Field	Default	What it does
`provider`, `model`	required	Which model. Tested on the provider evidence board: Groq (`qwen/qwen3.8-27b`, all six checks), OpenRouter (`deepseek/deepseek-v4-flash:free`), Ollama (`qwen2.5vl:3b`, prompted output). Every other provider is supported by construction and tested when a key exists.
`api_key`	`None`	From the argument or the provider's conventional environment variable. Excluded from `repr`; `redacted()` is the only form that leaves the process, and the API's events carry `"api_key": "***"`.
`decisions`	`()`	Slot names. `router`: a second opinion on the page type, asked only when the classifier declined to commit (`unknown`). `boundary`: a yes/no per block on product, listing and collection pages in content-only mode. Empty means the model judges nothing.
`thresholds`	`{}`	Per-slot probability a model answer must clear to override the rule. Defaults: router 0.85, boundary 0.9 (in) / 0.1 (out). A language model's self-reported confidence is capped at 0.7 — it is a phrase, not a frequency — so an LLM alone can never clear the default thresholds; lower them deliberately or use a calibrated classifier backend.
`budget`	`Budget()`	Counted after every call; the next call is refused when a cap is hit and the refusal is a recorded decision. Output is capped in total, never as a small per-call `max_tokens` (a reasoning model spends that on thinking).
`timeout_seconds`	`8.0`	Hard bound on one call inside the parse.
`backend`	`None`	A backend object in place of the model: `FakeBackend` in tests; a calibrated classifier when one exists.

Where it goes

from webgraph import stream_page, stream_site, SiteConfig

for event in stream_page(url, llm=llm):
    if event["type"] == "done":
        event["decisions"]      # every decision taken for this page
        event["spent"]          # {"requests", "input_tokens", "output_tokens"}
        event["llm"]            # the Intelligence, redacted

for event in stream_site(root, config=SiteConfig(llm=llm)):
    if event["type"] == "page":
        event["decisions"]      # this page's decisions
    if event["type"] == "done":
        event["spent"]          # the crawl's total

Over the API the same object is the request's "llm" field on /api/text/stream and /api/site/stream:

{"url": "https://example.com/", "llm": {"provider": "groq", "model": "qwen/qwen3.8-27b",
 "api_key": "gsk_…", "decisions": ["router", "boundary"]}}

What a decision records

{"slot": "router", "question": "page_type", "answer": "product", "probability": 0.7,
 "applied": false, "backend": "llm", "model": "qwen/qwen3.8-27b",
 "input_tokens": 1840, "output_tokens": 31, "milliseconds": 412,
 "note": "below threshold 0.85"}

applied says whether the answer changed what the rules had decided. note says why not, or what failed: below threshold, no block cleared the threshold, timeout, HTTP 429, budget: max_requests, not installed, unknown provider.

The judge slots ship behind their thresholds on purpose. Each one is opened by a written experiment with pass criteria and evidence — see docs/research/2026-09-20-decision-models-jev.md — and a slot whose experiment fails stays closed. Until a calibrated backend exists, the layer is honest about what a language model's confidence is worth.

The evidence behind the choice

Pydantic AI was adopted after the same six checks on every provider we could reach — plain completion, schema output, a verbatim-quote rule enforced in code, image input (a bar chart read correctly), temperature / max_tokens / timeout, usage limits that stop a run — recorded in docs/research/2026-09-20-pydantic-ai-provider-evidence.md with the raw responses in docs/evidence/pydantic-ai/.