Markdown

Plain text destroys most of what a page means. A flattened table loses which column a value belonged to; a stripped link loses where it pointed; a heading becomes an indistinguishable line. render_markdown.py writes the ordered blocks as Markdown because it is the format that survives the trip — a model, an index or a document store can use it without re-inferring the structure the page already had. Blocks arrive in reading order, so the Markdown reflects how a person reads the page rather than how the HTML was authored.

What each block becomes

Consecutive list items form one list. A definition follows its term on the next line without a blank between them; two terms in a row each get their line, because a DocBook table of contents is a <dl> of terms with no definitions (catb.org).

Definition lists were chosen in this form because CommonMark has no definition lists, and under a CommonMark reader **term** then : definition still reads as one paragraph with the term marked and the association kept, where an indented paragraph after a blank line is just another paragraph. cl.cam.ac.uk's Unicode FAQ and every php.net parameter list were coming out as alternating paragraphs (PR #75).

Tables render as pipes when pipes can say what the table says, and as the table's own markup when they cannot — a <td colspan="3"> rendered as pipes drops the merge and shifts every value beneath it into the wrong column, which corrupts data rather than reformatting it. On WebMainBench's pages whose truth holds an HTML table, a pipe rendering caps at 0.445 where the table's own markup reaches 1.000. Rendering a table as pipes when its only span was a full-width title row was measured and rejected (table_edit 0.3365 → 0.3305; the corpus writes that shape as HTML six times and as pipes once).

The switches

text against markdown

Block.text stays plain; the Markdown form lives in a separate Block.rich_text. De-duplication, the content hash and reading order all key on the plain form — folding Markdown syntax into it would change every hash and make two renderings of one sentence look like different content.

Document.text is the blocks' plain text in reading order, joined by blank lines, and it is not everything the Markdown carries. Image alt text and media placeholders are left out: an image block's text is its alt, kept so the Markdown can carry it, but no reader saw it as text — ikea.com's bookcase category emitted 613 words of alt text ("A tall, white BILLY bookshelf with multiple shelves…") that no annotator marked. Rules have no text at all. Callers that relied on alt text in text should read the Markdown.

Scripts

A <sup>/<sub> becomes the exact Unicode superscript or subscript character where one exists — cm<sup>2</sup> is cm², H<sub>2</sub>O is H₂O, an isotope's mass number before its symbol (<sup>235</sup>U) is ²³⁵U. That set is small and exact (digits, `+

• = ( ), and the handful of letters — n`, and the lowercase Latin block added to Unicode in 2010 — that have a true subscript or superscript glyph): a chemical formula or an exponent then reads correctly as plain text, in a terminal or a search index, not only where the Markdown is rendered as HTML.

A character with no exact form — most letters, an exponent like x<sup>k</sup> — keeps its <sup>/<sub> tag, verbatim, as inline HTML rather than losing the distinction by falling back to plain text. This is the same choice a preserved table's own markup already makes for a cell's formula (see Tables above); ordinary prose now makes it too. A <math> element carrying MathML is a separate, earlier path — converted to LaTeX between $...$ — and unaffected by this rule.

Dollars

Every Markdown dialect a model reads delimits mathematics with $…$, so an unescaped currency amount silently becomes a formula, and a pair of them becomes a formula containing all the prose between. Measured on WebMainBench: 18 of 200 pages with no mathematics were scored as emitting formulas, because of sentences like spends $29.8 billion … a surplus of $344 million.

The rule is a dollar followed by a digit is money and is escaped; nothing else is. It was measured rather than guessed: escaping every $ fixed the false positives and broke the true ones (the formula column's page count fell from 282 to 130). In the corpus's own truth an escaped dollar is followed by a digit 953 times of 1,221, while a bare one is followed by a space, a backslash beginning a LaTeX command, or another $. Because mathematics may also begin with a digit ($0.07^{7}$), a span that contains a backslash, caret, underscore or brace is located first and left alone. This applies to the Markdown only; the plain text is not Markdown and nothing in it is a delimiter.

Hard breaks

<br> is a line break and <br><br> a paragraph break (PR #63). An address keeps its lines as Markdown hard breaks; a <br><br>-separated article — the paragraphs of a pre-CSS page, a forum post, an email pasted into a <div> — becomes paragraphs, as it would have with <p> tags; headings, captions and list items stay one line. Source newlines still collapse to spaces, as HTML says they should.