Site report

A site report is a check anyone can run on any website: is it readable by an agent that does not run JavaScript? What does its robots.txt say about each well-known AI and search bot? What does it declare to machines -- a Content-Signal, an llms.txt, an RSL licence, an agent card, a feed, JSON-LD, a security.txt -- and who honours each? Is anything hidden from readers that is shown to machines -- off-screen links, the shape of an SEO-spam injection? Are there walls, dead links, an old CMS? It ends with a suggested robots.txt, llms.txt and, when missing, security.txt. Run it at /report, with webgraph report <url>, or POST /api/site/report.

Everything in it comes from a measurement the engine already makes when it reads a page (packages/engine/src/webgraph/report/). Nothing is scored by opinion, and nothing is fetched under another bot's name.

What is measured

The root is probed the way every crawl starts (analyze.probe_site): fetched plainly and in a real browser, its technologies fingerprinted, its robots.txt read, its sitemaps tried. Up to REPORT_PAGES (5) pages are then sampled -- the root, then the first internal links it offers, one per path section where it links to several -- and each is resolved both ways (resolve.resolve_page). For every page:

The site-level measurements: the technology stack with versions, dated when the version's branch is in the report's release table (WordPress 4.0-7.1, Drupal 7-11, Joomla 3-6, Next.js 13-16, each date verified against the project's own release record on 16 Sep 2026; anything else is reported as a version and nothing more); robots.txt, Sitemap: lines and the sitemap walk's attempts; /llms.txt and /llms-full.txt, counted as found only when the body is plain text starting with an H1 -- a catch-all site answers /llms.txt with its HTML 404 page and status 200.

What robots.txt declares per bot

The engine never impersonates another bot. Fetching as GPTBot or Googlebot would ask a site to apply a policy it did not set for this client, and the answer would measure the disguise rather than the site. What can be measured honestly is the file: for each of fifteen well-known bots -- GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Applebot-Extended, meta-externalagent, Bytespider, Amazonbot -- the report reads the site's robots.txt the way that bot would (RFC 9309): the groups whose User-agent: equals the bot's token, case-insensitively, combined; the * group when none names it; nothing when there is no file. Googlebot-Image in the file does not govern Googlebot.

The verdict is read at the root by the same longest-match rule the crawl's own refusals use: blocked when / is disallowed, partly restricted when content paths are (the count and the paths are shown), allowed otherwise -- including when the only Disallow lines are housekeeping: /wp-admin/, /wp-login.php, /cgi-bin/, /admin/, /login, /account, /cart, /checkout, /search, a bare query-string pattern such as /*? (report.bots.ADMIN_PATHS, a small literal list). Measured on vtu.ac.in, an earlier version called all fifteen bots "restricted" for Disallow: /wp-admin/, which an owner reads as "all AI bots restricted" and is false. via says whether the bot was named or fell under *; any Crawl-delay and the lines verbatim are beside it. The report's plain fetches identify themselves as webgraph with a contact URL; its browser fetch is a real Chromium under its own User-Agent, as every render in the engine is; both obey the file like every other fetch the engine makes. The report's own requests -- the sampled pages, /llms.txt, the link checks -- are spaced REPORT_REQUEST_INTERVAL_SECONDS (1 s) apart per host; the root probe (root, robots.txt, sitemaps) is fetched as every crawl starts.

What the site declares to machines

A site in 2026 can say a great deal to machines, and almost none of it is enforced. The report reads every such signal it can find without impersonating anyone and shows each with three things: what was found, in the file's own terms; who honours it; and what it means to the owner in plain words -- "Your robots.txt tells AI systems they may index it for search and link back and use it as input to AI answers, but should not train AI models on it. Honoured voluntarily by the bots that read Content-Signal; not enforced." Twenty-four signals in five groups (packages/engine/src/webgraph/report/signals.py):

How it is read. Every probe is one streaming GET that reads the status and headers and stops there on an error, else at most 64 KB of body (512 KB for the llms files, 1 MB for the root) -- HEAD-then-GET in a single request, because a server that refuses HEAD would cost the second request anyway. Presence is never the status alone: vercel.com answers /ai.txt, /rsl.xml, /humans.txt and /manifest.json with its 2.5 MB HTML shell and status 200 (16 Sep 2026), so each signal has a shape test -- an H1 for llms.txt, the RSL namespace, a JSON array for tdmrep.json, a Contact: line, a name in an agent card -- and an HTML body fails every one of them. Probes are paced with the rest of the report, skipped (recorded as not measured, not absent) when robots.txt disallows the path for this client, and made under the engine's own User-Agent; the root is fetched once more, plainly, for the headers and the HTML a machine without JavaScript sees. Typically 12-18 requests. A 402 Payment Required on the root is noted as a pay-per-crawl wall. Web Bot Auth (RFC 9421 signatures a crawler presents) is not measured: a site cannot be tested for it without impersonating a signed bot.

What it does to the score. Nothing new: declarations to AI are choices, not virtues -- a site that says ai-train=no is not less ready -- and agent cards are rare enough that scoring their absence would penalise nearly every site. The one change is inside the existing 10-point Structured data and page metadata sub-score, whose 4 page-field points now count OpenGraph beside title, description and lang (a page with the three older fields and no og:* earns 3 of the 4). llms.txt stays at 5 points for the reason in the table below. The full research behind the list -- specs, adoption numbers, real-site examples, what could not be verified -- is in docs/research/2026-09-16-site-signals.md.

The score

An AI-readiness score out of 100 is the weighted sum of eight sub-scores, each from one measurement, each carrying its evidence and -- when short of full marks -- a recommendation in plain words ("/pricing has 41 words without JavaScript and 1,312 with it; agents that do not run JavaScript see 3% of it. Server-render or pre-render the main content."). A sub-score that could not be measured (rendering unavailable, no link checked) is left out and the total is rescaled to the weight that was measured; score.measured_weight says how much that was.

Integrity

A separate section, because an injected link block is not a matter of degree. "Likely SEO-spam injection" is said only when at least REPORT_SPAM_MIN_HOSTS distinct foreign hosts are linked from elements parked where no reader can scroll on one page; below that the links are reported as off-screen links to other hosts for the owner to read. A hidden display: none menu -- vtu.ac.in's dropdowns link to 187 affiliated colleges -- is not a verdict; its 174 links to 170 foreign hosts positioned twenty trillion pixels off the left edge are. Walls, pages that could not be read, and a CMS branch older than REPORT_STACK_OLD_YEARS (3) years are the other findings -- and three a site declares about itself without showing it:

• Canonicals name another host (canonical_elsewhere, high): sampled pages whose <link rel=canonical> is on a different site from the one that served them (www. and the bare domain are one site). Search engines fold the page into the declared address; a crawler that resolves links against it leaves the site. Usually a framework's base URL -- Next.js metadataBase, a CMS site URL -- left at a previous host after a move.
• The sitemap lists another host (sitemap_elsewhere, high): every address the sitemap lists is on another host, so to a crawler it advertises nothing of this site. The same stale base URL, the same fix.
• Pages draw content in a canvas (canvas_content, medium): a <canvas> and under forty readable words. A search engine, a screen reader or an assistant gets those words and nothing the scene draws. Put the text and the links in the markup as well.

Suggested files

The suggested robots.txt keeps the site's existing file byte for byte and appends a block in which every line is a comment: pasting it back changes nothing until the owner uncomments one of two variants -- allow every well-known bot, or allow the search and assistant bots and disallow the training crawlers. Neither is a recommendation.

The commented block also carries the Content-Signal line (contentsignals.org) in its two common forms -- search=yes, ai-input=yes, ai-train=no and all three yes -- with a note that it is a declaration a crawler chooses to read, not enforcement; a file that already has the line is shown its own values and nothing is proposed. When no feed is advertised on the root, a comment says so. A security.txt template (RFC 9116: Contact, Expires a year out, Preferred-Languages, Canonical) is offered only when the site has none.

The llms.txt draft follows llmstxt.org -- an H1 with the site's name, a blockquote summary, H2 sections of - [title](url): description -- built from the sampled pages' own titles and descriptions. It is marked optional wherever it is offered, for the reason in the score table.

Refusals

A root that is walled, disallowed for this client by robots.txt, missing or unreachable ends the report: reachable: false, the engine's own refusal in refusal, no score and no pages. The engine does not disguise itself to get past a refusal. A sampled page that cannot be read is a row with the refusal, not an abort.

Limits

• Five pages are a sample, chosen from the root's own links. A site whose problems are deeper than one click from the root will not show them here.
• The sitemap is read up to 2,000 URLs; a page not in that part is reported as unknown, not absent.
• Dead-link checks are capped per page and spaced a second apart, so a report takes one to three minutes; a link robots.txt disallows for this client is not checked.
• The stack's age is known only for the versions in the release table above.
• The bots table is what the file declares. Whether a CDN or a WAF serves those bots something else is not measured, because measuring it would mean impersonating them.
• The purpose recorded for each bot (search, assistant, training) is what its operator documents today, and will need correcting as that changes.

API and CLI

webgraph report https://example.com/ --pages 5          # a readable summary
webgraph report https://example.com/ --json             # the whole report
curl -s -X POST localhost:8000/api/site/report \
  -H 'content-type: application/json' -d '{"url":"https://example.com/","pages":5}'

The JSON carries score.subscores[] (key, weight, score, evidence, recommendation, source), findings[] (severity, kind, title, detail, page), pages[], robots.bots[], stack[], signals (signals[] with key, label, group, present -- true / false / null for not measured -- detail, meaning, who_honours, spec_url, source_url, status; groups[]; content_signals[]; root_headers; requests), suggested_robots_txt, suggested_llms_txt, suggested_security_txt, and measured -- the engine version and commit, the plain fetches' User-Agent, pages sampled, the request interval, the duration, and the statement that nothing was fetched as another bot.