The web UI

Status line and the stop button

A dot and a phrase name the phase — "Detecting the technology stack", "Seeding routes from sitemap and links", "Crawling and extracting", "Complete", "Stopped", "Failed"; a capped run that did not drain the frontier reads "Reached the page cap · N pages not crawled". Stop sets the server's stop flag; everything already streamed stays.

Progress rail

Completion against the frontier as it stands, or against the page cap when there is one. For an unbounded crawl the denominator grows, so the bar can move backwards — honest, and better than a bar implying a total nobody knows.

The stages

LivePipeline.tsx draws the engine's three stages as they happen — Look at the front door, once, Build the queue, Crawl, classify and extract — each lighting up when it starts and filling in what it found when it ends. A stage is never shown finished before its evidence arrives, and one that has not started shows nothing rather than zeros: "a zero reads as a measurement, and 'not yet' is not a measurement".

Technology detected

TechnologyPanel.tsx groups the analysis event's technologies by category, each chip carrying its evidence and confidence on hover and its version where one was read. Below: the strategy in use, static coverage, and the flags rendering required and rendering loses content when Stage 0 measured them.

How the site can be discovered

DiscoveryPanel.tsx: three collapsed rows, each a one-line verdict that opens to its evidence.

• robots.txt — "not found (HTTP 404) — everything allowed", or "found · 1 rule applies to us · crawl-delay 1s". Open: which User-agent: group was read and why, the rules (Disallow in red, Allow in green), and the file itself.
• Sitemaps — "none published — discovery is by links only", or the sitemaps found, their URLs, and how many are in scope. Open: every address tried, in order, with its status, and "· not a sitemap" against a 200 that carried no <loc>.
• URLs found by kind — a live tally of pages, PDFs, images, other files, date archives, categories and tags. When files outnumber pages the collapsed row warns: "Files outnumber pages (7,907 PDFs). Files are counted with the page that links to each, and never fetched." The run summary at the end says which limit ended the run and how many files were counted and not fetched.

Tabs

Discovered and Queued list URLs (queued includes the batch in flight); Extracted is the page list; Failed lists each refusal with its full error and its citation — the page that sent the crawl there and the link text. Three views sit beside them: Pages, Depth tree, Site graph.

The page list

PageList.tsx searches URLs, titles and page content. Each row shows character, image and table counts and the share of the page identified as chrome; open it for the Markdown, with two independent switches — content only (filtered; the whole page is the default) and preview vs source. Download N .md files (.zip) hands over one Markdown file per page, named by host and path (lakshx.in/docs/sign-in.md, the root as index.md), each opening with url and title front matter so a file read alone still says where it came from — the shape a folder of pages is expected in. single .md and Copy all hand over every successful page as one document headed by its URL. All three follow the same checkbox.

Depth tree

DepthTree.tsx draws the site as the crawl found it, one level per link away from the root. Each node sits under the page that first linked to it — "the tree draws what the frontier did rather than a prettier fiction" — and a per-depth bar shows a breadth-first crawl filling one level before the next.

Site graph and Ask

GraphPanel.tsx shows hubs (what the site links to most, with a specificity figure separating navigation from topics) and subjects (what pages call each other, from anchor text). AskPanel.tsx appears once three pages exist and returns a bounded context for a question, every section saying why it is there.

Run log

Collapsed by default, above the page lists so a 1,600-URL crawl does not bury it. It shows the tail of the event stream — the end is where a run went wrong — and Copy run log pastes the whole run, header and every event, at any moment including mid-run. This is the paste an issue report needs.