Render an article's maths instead of showing its TeX source

Reported against lcamtuf's Substack ("we should render tex",
blog.coredump.cx/p/yours-truly): reader mode showed raw
"\(p = \text{...}\)" where equations should be.

What the page actually ships is the TeX SOURCE, as text, inside a
"latex-rendered" div -- Substack renders it in the browser with its own
script, which never runs here. Readability keeps that text verbatim (it
strips the class, not the text), so nothing was being discarded during
extraction and no amount of extraction-side repair would have helped:
there was simply never any rendered markup to keep.

So render it during extraction, where JavaScript is already running for
Readability's sake. KaTeX (vendored in assets, never fetched -- this
runs against untrusted third-party pages) converts each expression to
MathML before Readability sees the document, and the reader WebView
renders that natively. The reader keeps JavaScript OFF, as it must, and
needs no stylesheet, webfont or script of KaTeX's; the only reader-side
change is containing a wide equation so it scrolls within its own box
instead of dragging the whole article sideways.

Two details worth knowing:

- KaTeX refuses {align} outside display mode, and Substack wraps display
  maths in INLINE delimiters (a block-styled span does the centring), so
  a mode that renders nothing is retried in the other one. An expression
  neither mode can render is left as source -- no worse than before.
- The reported article came with "extractedContent cached: true", so the
  raw source was already on disk and every open read it back. Recognised
  and dropped on read (same mechanism as the cached consent notice),
  which re-extracts the article the next time it is opened.

APK grows by ~90KB: the vendored katex.min.js (267KB) compresses to 91KB
in the APK. No new permissions, no schema change, no new JS-enabled
surface.

Verified on the emulator against the live article: stale raw-TeX cache
discarded, article re-extracted, all four equations rendered, and a wide
align block scrolls inside itself.

3 files changed, 194 insertions, 0 deletions

diff --git a/app/src/test/java/org/terst/feedreader/data/CachedExtractionTest.kt b/app/src/test/java/org/terst/feedreader/data/CachedExtractionTest.kt
index 72dc73f..fc9c32e 100644
--- a/app/src/test/java/org/terst/feedreader/data/CachedExtractionTest.kt
+++ b/app/src/test/java/org/terst/feedreader/data/CachedExtractionTest.kt
@@ -33,6 +33,7 @@ class CachedExtractionTest {
         val result = loadCachedExtraction(content) { cleared = true }
         assertEquals(content, result.content)
         assertFalse(result.discardedConsentNotice)
+        assertFalse(result.discarded)
         assertFalse("a good cache must not be cleared", cleared)
     }
 
@@ -54,6 +55,37 @@ class CachedExtractionTest {
         assertFalse(cleared)
     }
 
+    // Same shape of problem as the consent notice, different cause: this content was
+    // extracted before extraction knew how to render maths, so it holds the raw TeX source
+    // Substack ships. A fixed extractor never gets a look at it -- the cached copy is
+    // returned first and the article is never re-extracted.
+    @Test
+    fun `a cached extraction holding raw tex source is discarded and deleted`() = runTest {
+        var cleared = false
+        val rawTex = "<div id=\"readability-page-1\"><p>Imagine the following sentence:</p>" +
+            "<div data-component-name=\"Latex\"><p><span>" +
+            "\\(p = \\text{\u201Cif }p\\text{ is true, then pigs can fly\u201D}\\)</span></p></div>" +
+            "<p>" + "Considered as a whole, the statement cannot be true. ".repeat(20) + "</p></div>"
+        val result = loadCachedExtraction(rawTex) { cleared = true }
+        assertNull("raw TeX source must not be shown as the article", result.content)
+        assertTrue(result.discardedUnrenderedTex)
+        assertTrue("the stale row must be cleared so a re-extract can replace it", cleared)
+        assertTrue("the caller must be told to extract again", result.discarded)
+    }
+
+    @Test
+    fun `an extraction with rendered maths is kept`() = runTest {
+        var cleared = false
+        val withMathMl = "<p>A slice of the set looks like this:</p>" +
+            "<math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><semantics>" +
+            "<mrow><mi>p</mi><mo>=</mo></mrow></semantics></math>" +
+            "<p>" + "The rest of the article continues from there. ".repeat(20) + "</p>"
+        val result = loadCachedExtraction(withMathMl) { cleared = true }
+        assertEquals(withMathMl, result.content)
+        assertFalse(result.discardedUnrenderedTex)
+        assertFalse("a good cache must not be cleared", cleared)
+    }
+
     // A blank value is as good as nothing cached -- it is not a consent notice, so
     // reporting it as a discarded one would make the caller re-extract on a false
     // premise.
diff --git a/app/src/test/java/org/terst/feedreader/readability/UnrenderedTexTest.kt b/app/src/test/java/org/terst/feedreader/readability/UnrenderedTexTest.kt
new file mode 100644
index 0000000..016b148
--- /dev/null
+++ b/app/src/test/java/org/terst/feedreader/readability/UnrenderedTexTest.kt
@@ -0,0 +1,116 @@
+package org.terst.feedreader.readability
+
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+/**
+ * Recognising a cached extraction that still holds raw TeX source.
+ *
+ * Substack ships maths as TeX source text in the page (`\(p = \text{...}\)` inside a
+ * `latex-rendered` wrapper) and renders it client-side with its own script, which never
+ * runs in this app. Extraction now converts that source to MathML before Readability sees
+ * it -- but an article extracted BEFORE that existed has the raw source sitting in
+ * `articles.extractedContent`, and every subsequent open reads it straight back out. The
+ * reported article came with "extractedContent cached: true", so this is the half a fixed
+ * extractor cannot reach.
+ *
+ * Over-detecting is the real risk: a false positive makes the article re-extract on every
+ * single open, since re-extracting cannot change a verdict the content itself causes. So
+ * the bar is a complete delimiter pair around something that actually looks like TeX, and
+ * never anything already carrying rendered MathML.
+ */
+class UnrenderedTexTest {
+
+    // "$$" cannot be written inline in a Kotlin string followed by a letter -- that parses
+    // as a template expression.
+    private val dollars = "$" + "$"
+
+    @Test
+    fun `substack's inline-delimited tex source is recognised`() {
+        assertTrue(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<div data-component-name=\"Latex\"><p><span>" +
+                    "\\(p = \\text{“if }p\\text{ is true, then pigs can fly”}\\)</span></p></div>"
+            )
+        )
+    }
+
+    @Test
+    fun `bracket-delimited display tex source is recognised`() {
+        assertTrue(UnrenderedTex.looksLikeUnrenderedTex("<p>\\[\\begin{align} a &= b \\end{align}\\]</p>"))
+    }
+
+    @Test
+    fun `double-dollar display tex source is recognised`() {
+        assertTrue(UnrenderedTex.looksLikeUnrenderedTex("<p>" + dollars + "x_{1} + y^{2}" + dollars + "</p>"))
+    }
+
+    @Test
+    fun `content already carrying rendered mathml is left alone`() {
+        assertFalse(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<p><math xmlns=\"http://www.w3.org/1998/Math/MathML\"><mi>p</mi></math></p>"
+            )
+        )
+    }
+
+    // An expression KaTeX could not render stays in the output as source. If the presence of
+    // any source at all convicted the whole article, that article would re-extract on every
+    // open, for ever, and never come out any different. Rendered maths anywhere in the
+    // content is proof the extraction already had its go at it.
+    @Test
+    fun `a leftover unrenderable expression alongside rendered mathml does not convict`() {
+        assertFalse(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<p><math><mi>a</mi></math></p><p>\\(\\thisIsNotRealTex{}\\)</p>"
+            )
+        )
+    }
+
+    @Test
+    fun `ordinary prose is not tex`() {
+        assertFalse(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<p>The repair bill came to $$ in the end; the second quote was $$ as well, and (as it " +
+                    "turned out) the warranty had lapsed anyway.</p>"
+            )
+        )
+    }
+
+    // The inner span between two stray delimiters is HTML, not one expression -- markup
+    // brings its own underscores and braces along (ids, filenames, inline styles), so a tag
+    // anywhere inside is on its own enough to say these two delimiters are unrelated.
+    @Test
+    fun `delimiters spanning real markup are not one expression`() {
+        assertFalse(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<p>It cost $$ for the <a href=\"https://ex.com/part_no_4.html\">part</a>, " +
+                    "and $$ again for the labour.</p>"
+            )
+        )
+    }
+
+    @Test
+    fun `an unclosed delimiter is not tex`() {
+        assertFalse(UnrenderedTex.looksLikeUnrenderedTex("<p>Then he wrote \\(x, trailed off, and left.</p>"))
+    }
+
+    // Extraction deliberately skips <pre>/<code> when rendering maths -- an article ABOUT
+    // TeX quotes the source on purpose. Convicting on that would re-extract it on every
+    // open and still, correctly, change nothing.
+    @Test
+    fun `tex quoted inside a code block is not a stale extraction`() {
+        assertFalse(
+            UnrenderedTex.looksLikeUnrenderedTex(
+                "<p>Write it like this:</p><pre><code>\\(x^{2} + y^{2}\\)</code></pre>"
+            )
+        )
+    }
+
+    @Test
+    fun `nothing cached is not tex`() {
+        assertFalse(UnrenderedTex.looksLikeUnrenderedTex(null))
+        assertFalse(UnrenderedTex.looksLikeUnrenderedTex("   "))
+    }
+}
diff --git a/app/src/test/java/org/terst/feedreader/ui/ReaderMathTest.kt b/app/src/test/java/org/terst/feedreader/ui/ReaderMathTest.kt
new file mode 100644
index 0000000..fea5d22
--- /dev/null
+++ b/app/src/test/java/org/terst/feedreader/ui/ReaderMathTest.kt
@@ -0,0 +1,46 @@
+package org.terst.feedreader.ui
+
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+/**
+ * The reader's end of maths rendering.
+ *
+ * Extraction converts a page's TeX source to MathML (see ReaderExtractor), which the
+ * WebView renders natively -- no script, no webfont, no stylesheet of KaTeX's. Two things
+ * still have to be true of the reader HTML itself: the MathML has to reach the WebView
+ * intact (the wrapping passes rewrite iframes and images, and must not touch it), and a
+ * wide equation must not be able to drag the whole article into horizontal scrolling --
+ * an aligned multi-line derivation is routinely wider than a phone screen, and `math`
+ * does not obey the `img`/`svg` clamps already in the stylesheet.
+ */
+class ReaderMathTest {
+
+    private val displayMath =
+        "<math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><semantics>" +
+            "<mtable><mtr><mtd><mi>t</mi></mtd></mtr></mtable></semantics></math>"
+
+    @Test
+    fun `mathml reaches the webview unchanged`() {
+        val html = wrapReaderHtml("<p>A slice of the set:</p>$displayMath<p>and so on.</p>")
+        assertTrue("MathML must survive the reader's HTML assembly verbatim", html.contains(displayMath))
+    }
+
+    @Test
+    fun `a wide equation scrolls inside itself instead of scrolling the article`() {
+        val html = wrapReaderHtml("<p>x</p>$displayMath")
+        val style = html.substringAfter("<style>").substringBefore("</style>")
+        val displayMathRule = style.lines()
+            .joinToString(" ")
+            .substringAfter("math[display=\"block\"]", "")
+            .substringBefore("}")
+        assertTrue(
+            "the stylesheet needs a math[display=\"block\"] rule, found none in:\n$style",
+            displayMathRule.isNotBlank()
+        )
+        assertTrue(
+            "display maths must scroll within its own box, got: $displayMathRule",
+            displayMathRule.contains("overflow-x: auto")
+        )
+    }
+}