feedreader.git/app/src/test/java/org/terst/feedreader/data/CachedExtractionTest.kt, branch master

Fix stray promo headers, site-URL subscribing, and YouTube playback

2026-10-21T19:07:17+00:00

Three reported bugs. **"West Hawaii today articles get a bunch of stray Trending headers."** The publisher's widget platform injects promo blocks between the real paragraphs, inside the article's own container, so every structural signal tells Readability they belong. What survives is a heading labelling nothing, because the links under it are filled in by a script that never runs in the reader. Confirmed on 9 of the 12 articles in that feed, two each. Blocks marked `data-mrf-recirculation` are now removed -- the attribute says what the block is for in the widget platform's own words, which is a better signal than the word "Trending", since an article may legitimately discuss what is trending. Nothing longer than a label and a few headlines is ever removed. Also applied to already-cached articles, in place rather than by discarding them: the article was fine, only the litter has to go. **"Can't subscribe to heathercoxrichardson.substack.com."** Adding a feed handed whatever URL it was given straight to the XML parser, so a site's home page -- the only URL a reader can actually see and copy -- failed on the root element. That page advertises its feed the way sites have since 2002, in a in the head. It is now read, resolved against the page, and fetched. Only what the page declares: guessing /feed and /rss would add a burst of 404s to every failed add and eventually subscribe somebody to the wrong thing. **"YouTube play is misconfigured."** It was, and YouTube said so: the player dialog loaded the embed as its own top-level document, which answers with "Video player configuration error -- Error 153". An embed has to be embedded. It now sits in an iframe on a minimal host page. Measured in the app's own WebView: top-level, no requests for the video stream and the error on screen; framed, the stream loads and there is no error. **"Long click on YouTube placeholder should open the URL directly."** It does now. Worth recording why this is not simply the link's href: for an image inside a link, WebView's hit test reports the image's URL, so a long-press on the thumbnail arrives as the i.ytimg.com JPEG. The video id is resolved out of that. Verified on a device for all four: the Substack site URL adds "Letters from an American" and fetches 20 articles; a West Hawaii Today article extracts with no Trending headers and its body intact; the video plays; and a long-press reaches the YouTube app. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Render an article's maths instead of showing its TeX source

2026-10-16T21:04:38+00:00

Reported against lcamtuf's Substack ("we should render tex", blog.coredump.cx/p/yours-truly): reader mode showed raw "\(p = \text{...}\)" where equations should be. What the page actually ships is the TeX SOURCE, as text, inside a "latex-rendered" div -- Substack renders it in the browser with its own script, which never runs here. Readability keeps that text verbatim (it strips the class, not the text), so nothing was being discarded during extraction and no amount of extraction-side repair would have helped: there was simply never any rendered markup to keep. So render it during extraction, where JavaScript is already running for Readability's sake. KaTeX (vendored in assets, never fetched -- this runs against untrusted third-party pages) converts each expression to MathML before Readability sees the document, and the reader WebView renders that natively. The reader keeps JavaScript OFF, as it must, and needs no stylesheet, webfont or script of KaTeX's; the only reader-side change is containing a wide equation so it scrolls within its own box instead of dragging the whole article sideways. Two details worth knowing: - KaTeX refuses {align} outside display mode, and Substack wraps display maths in INLINE delimiters (a block-styled span does the centring), so a mode that renders nothing is retried in the other one. An expression neither mode can render is left as source -- no worse than before. - The reported article came with "extractedContent cached: true", so the raw source was already on disk and every open read it back. Recognised and dropped on read (same mechanism as the cached consent notice), which re-extracts the article the next time it is opened. APK grows by ~90KB: the vendored katex.min.js (267KB) compresses to 91KB in the APK. No new permissions, no schema change, no new JS-enabled surface. Verified on the emulator against the live article: stale raw-TeX cache discarded, article re-extracted, all four equations rendered, and a wide align block scrolls inside itself.

Stop reader mode passing a site's cookie notice off as the article

2026-10-12T00:41:24+00:00

Reported live: reader mode on a New Republic feature showed nothing but the cookie disclaimer, and had cached it. Reproduced outside the app by running ReaderExtractor's own injected script (Readability.js and all) against the real page. With the article body present, extraction is perfect. With the body absent from the DOM at the moment the script runs, Readability's first pass -- which discards "unlikely candidates", and the Didomi banner's class is didomi-notice-banner -- comes back under its own 500-character threshold, so it retries with that stripping switched off. The banner's single paragraph is 510 characters, i.e. just over the threshold, so it comes back as the article with the right title attached. Nothing on the Kotlin side looked at that. Any non-blank content counted as ExtractionResult.Success, was written to articles.extractedContent, and was never reconsidered -- so one bad load became a permanent cookie disclaimer, which is the "extractedContent cached: true" in the report. ConsentNotice recognises such a result: a hard plain-text length cap first (nothing long is ever rejected, however much consent boilerplate it quotes), then either a known CMP container id -- Readability strips classes but keeps ids -- or two or more distinct consent phrases. The accept/retry/give-up decision moves out of the WebView callback into decideEvalOutcome so it can be tested without a device. A consent notice is now "not ready yet", which routes it into the retry the extractor already does for an empty result; if it survives all six attempts that is what the page serves, and it fails as ExtractionFailure.ConsentInterstitial rather than being accepted. A fixed extractor cannot repair what is already on disk, so loadCachedExtraction drops a cached notice on read and re-extracts, in both ReaderScreen and LinkReaderScreen. No schema change: the repair happens when the article is next opened. Reader mode is still never turned on heuristically -- replacing a discarded notice does not flip it. The bug report now also carries the last extraction error, without which a report about bad reader content cannot say whether extraction failed. Tests: 59 -> 78. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B