Running them

Commands and flags for every runner, where the corpora come from, how long each takes, the runs dashboard for long boards, and the WebMainBench rule.

None of the public corpora is vendored — each is somebody else's dataset on somebody else's release cadence — so every board runner takes --corpus pointing at a local clone. The live suites need only Playwright's Chromium (uv run --package webgraph playwright install chromium). Every runner here is deterministic given its inputs; pin the source before quoting a score (git archive HEAD | tar -x -C /tmp/pinned, as the Zyte runner's docstring shows — a working-tree edit landing mid-run once moved a variant from 0.623 to 0.643).

Make targets

make bench-fidelity      # benchmark/fidelity/run.py --out fidelity.json
make bench-live          # benchmark/live/run.py --out live-suite.json
make bench               # schema extraction over benchmark/corpus-v0 (the CI gate)
make bench-content       # main content vs trafilatura / readability / jusText (needs the bench group)
make bench-routes        # route discovery vs a real-browser oracle; bench-routes-quick for 10 sites
make bench-union         # union placement (bench-union-fetch once first)
make bench-reading-order # geometric order vs a DOM walk

The four public boards have no make target; run them as below.

The board runners

git clone --depth 1 https://github.com/Murrough-Foley/web-content-extraction-benchmark   # CC-BY-4.0, 192 MB
export WCXB_ROUTER_OOF=benchmark/train/artifacts/router_oof.json   # or routed == content
uv run --package webgraph python benchmark/wcxb/run.py --corpus <clone>
uv run --package webgraph python benchmark/wcxb/run.py --corpus <clone> --split test
uv run --package webgraph python benchmark/wcxb/run.py --corpus <clone> --worst product

Flags: --split dev|test, --jobs N (CPUs minus one), --limit N for a smoke run, --out to write per-variant predictions. For the per-type diff the PR template asks for: benchmark/wcxb/per_page.py dump on main and on the branch, then diff.

Live suites (development only)

uv run python benchmark/fidelity/run.py --out /tmp/main.json          # on main
uv run python benchmark/fidelity/run.py --out /tmp/cand.json          # on the branch
uv run python benchmark/fidelity/run.py --compare /tmp/main.json /tmp/cand.json
uv run python benchmark/live/run.py     --out /tmp/main.json          # same shape
uv run python benchmark/live/run.py     --compare /tmp/main.json /tmp/cand.json

Both take --sites for another list and cache the oracle beside --out in a .pages directory. These are development diagnostics for a pull request — the diff between two runs is the signal — and their figures are not published: a number scored against a live page cannot be produced again once the page changes.

Run times

Measured on the owner's machine from runs/*.meta.json (start, end and exit code recorded by tools/run_logged.py; only runs with exit code 0 counted):

runner	pages	time
fidelity suite	31	6.9–8.1 min
live suite	23	2.9–4.3 min
WCXB dev, `per_page.py dump`	1,497	2.5–5.3 min
WebMainBench 545	545	10.4–18.6 min
Zyte article-extraction	181	12–18 s
WCEB, all eight corpora	3,985	about two hours (`run_logged.py`: "a two-hour board run")

Long runs: `run_logged.py` and the dashboard

A board run prints its progress to a terminal nobody is watching, and a laptop on battery sleeps mid-run — run_logged.py's docstring: "a two-hour board run on a laptop that sleeps on battery is a run that stops at 17:18 and looks like a hang the next morning". Two stdlib-only tools:

uv run python tools/run_logged.py wceb-main -- \
    uv run python benchmark/wceb/run.py --corpus <clone>
uv run python tools/runs_dashboard.py            # http://localhost:8765, watches ./runs

run_logged.py writes runs/<label>.log (stdout and stderr as they arrive) and runs/<label>.meta.json (command, cwd, pid, start; then end and exit code), and on macOS runs the command under caffeinate -i (--no-caffeinate to turn that off). A second run with the same label gets a numbered suffix. runs_dashboard.py serves one page showing every run — running, finished or failed — with elapsed time, a progress bar read from the runner's own N/M counter lines, and the result lines once they land; it updates live and lists the machine's engine processes (PR #70). Nothing is installed and nothing is written.

The WebMainBench rule

From CONTRIBUTING.md

"A change to table, code or Markdown rendering also re-runs WebMainBench before merge, whatever WCXB and Zyte say: they score plain text, and a link rendered into every table cell cost WebMainBench's table score for six pull requests unnoticed (#68, fixed in #78)."

The mechanism is worth knowing, because it recurs. WCXB, Zyte, WCEB and CleanEval tokenize to words; a | or a ](url) is invisible to them, so a rendering change that is wrong can measure as "=" on all four. WebMainBench scores character edit distance on Markdown and is the only board that can see structure. Its column figures have different N per column (printed per run), and the 15 September rounds each found a second-order effect where adding or removing short blocks near code moved the content boundary (SESSION-18 §5, CHANGELOG #86). Run it for anything that touches render_markdown.py, table or code handling, or short blocks around code.

What the PR template asks for

The Measurement table in .github/PULL_REQUEST_TEMPLATE.md has one row per instrument: WCXB dev routed overall; WCXB dev by the page types the change touches (per_page.py dump

diff); Zyte for anything touching articles; WCEB production path for comments, landmarks or the boundary; the reading-order benchmark for dom/reading_order.py; WebMainBench for tables, code, Markdown rendering; the live suite's pages that moved; and the fidelity suite's recall / extra / inversions on pages that moved. "Not applicable" is an answer for a docs or UI change; a blank row is not.