Content selection

A resolved page is the whole page: navigation, header, footer, cookie banner, related links, the article. select_content reduces it to the page's own content and reports what each step removed. It is the same function the crawl uses for content_markdown and the API for /api/text's content_markdown.

select_content

from webgraph import select_content

selection = select_content(
    page.document.blocks,
    chrome=None,          # SiteChrome | None -- a crawl's cross-page profile
    main_content=True,    # bool -- draw the boundary, or stop after the structural steps
    config=None,          # MainContentConfig | None -- the boundary's settings
    model=None,           # None (boundary), SHIPPED_MODEL, or a BlockModel
    title="",             # str -- the page's title, so its block is never cut
)

Never returns nothing for a non-empty input: each step falls open to what it was given when it would remove everything, and the boundary refuses to return a fragment (min_run_share).

The steps, in order

1. Landmarks — <nav>, <header>, <footer>, <aside> and their ARIA roles are removed, except an <aside> that carries the page's own title (a product page's buy box). Method name landmarks.
2. Comments — the comment thread under an article (Block.widget == "comments") is taken out and returned separately as selection.comments. Skipped by policy on forums, where the comments are the content, unless they hold more than half the page.
3. Scope to the article — when one <article> dominates the page, or an element declares itself the body (itemprop="articleBody", entry-content), everything outside it goes. main-landmark, article-element, article-body. Off by policy for forums, listings, collections and products, where the repeated items are <article>s themselves.
4. Site chrome — with chrome, blocks the site repeats on most pages. site-chrome.
5. Main content — the contiguous run of blocks with the best prose-density score (a maximum-subarray over per-block values), or the block model's per-block verdict. main-content or block-model.
6. Title back — if the boundary cut the title block, it is restored with the lead that sat between it and the first kept paragraph.

ContentSelection

selection = select_content(page.document.blocks, title=page.document.title)
selection.kept, selection.total     # 116, 164   (docs.python.org, the tutorial's introduction)
selection.methods                   # ('landmarks', 'main-content')
selection.landmarks_removed         # 47
selection.title_restored            # True

Page types

The boundary's best settings differ by what kind of page it is, and the engine ships a router that says which kind. Measured on WCXB dev, routing lifts listings by +0.09 F1 and services by +0.02 while leaving articles alone.

from webgraph import default_router, policy_for, select_content

router = default_router()                 # parsed once per process; None if no model ships
routing = router.route(page.document)
routing.page_type                         # PageType.DOCUMENTATION
routing.confidence                        # 0.999
routing.runner_up                         # ('service', 0.0006)

selection = select_content(page.document.blocks, config=policy_for(routing.page_type), title=page.document.title)

PageType is a string enum: article, documentation, service, forum, collection, listing, product, unknown. Below the router's confidence floor it answers unknown, and policy_for("unknown") is the default policy. router.route(document, explain=True) fills routing.reasons with the strongest features, phrased.

policy_for returns a MainContentConfig that differs from the default only where a type measured better with the change:

Cross-page chrome

A page seen alone cannot know that "Better things in a better way" appears on every page of the site. A set of pages can:

from webgraph import resolve_page, select_content
from webgraph.boilerplate import detect_site_chrome

pages = [resolve_page(u) for u in urls]              # six or more pages of one site
chrome = detect_site_chrome([p.document.blocks for p in pages])
chrome.active, chrome.page_count                     # True once there are enough pages

for p in pages:
    s = select_content(p.document.blocks, chrome=chrome, title=p.document.title)
    s.chrome_removed                                 # repeated blocks the landmarks step had not already taken

On docs.python.org the profile is active after seven pages and holds 22 repeated lines, and chrome_removed is still 0 — every one of them sits in a <nav> or <footer> and the landmarks step took it first. Chrome earns its keep on sites whose furniture is in plain <div>s.

Chrome is a block seen on 90% of pages, judged once six pages exist, never removing more than half of any page; a template slot that always holds the same value counts even when the words move. stream_site builds this profile as it goes and applies it from the point it can say anything.

The block model

from webgraph.content import SHIPPED_MODEL

selection = select_content(page.document.blocks, model=SHIPPED_MODEL, title=page.document.title)
selection.methods    # ('landmarks', 'block-model')

A gradient-boosted per-block classifier trained on WCXB dev, shipped as JSON. Out of fold it scores +0.029 word-F1 over the boundary on WCXB and holds on Zyte — and on WebMainBench, which scores Markdown by edit distance, it is worse (0.622 → 0.586): it drops most of some long documents and keeps comment furniture the boundary's contiguity excludes. So it is offered, not defaulted. benchmark/train/README.md has both sides.

MainContentConfig

The boundary step's settings. Every default was swept on a benchmark, not chosen; the docstrings in webgraph.main_content carry the sweeps. Build one with keyword arguments, or dataclasses.replace(policy_for(page_type), ...) to start from a policy.

import dataclasses
from webgraph import MainContentConfig, policy_for

config = MainContentConfig(group_repeats="main", min_run_share=0.1)
config = dataclasses.replace(policy_for("product"), product_keep_all=True)