From 9210bf463a8818595f7ca42486a787d4f1941d55 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Wed, 21 Oct 2026 19:07:17 +0000 Subject: Fix stray promo headers, site-URL subscribing, and YouTube playback Three reported bugs. **"West Hawaii today articles get a bunch of stray Trending headers."** The publisher's widget platform injects promo blocks between the real paragraphs, inside the article's own container, so every structural signal tells Readability they belong. What survives is a heading labelling nothing, because the links under it are filled in by a script that never runs in the reader. Confirmed on 9 of the 12 articles in that feed, two each. Blocks marked `data-mrf-recirculation` are now removed -- the attribute says what the block is for in the widget platform's own words, which is a better signal than the word "Trending", since an article may legitimately discuss what is trending. Nothing longer than a label and a few headlines is ever removed. Also applied to already-cached articles, in place rather than by discarding them: the article was fine, only the litter has to go. **"Can't subscribe to heathercoxrichardson.substack.com."** Adding a feed handed whatever URL it was given straight to the XML parser, so a site's home page -- the only URL a reader can actually see and copy -- failed on the root element. That page advertises its feed the way sites have since 2002, in a in the head. It is now read, resolved against the page, and fetched. Only what the page declares: guessing /feed and /rss would add a burst of 404s to every failed add and eventually subscribe somebody to the wrong thing. **"YouTube play is misconfigured."** It was, and YouTube said so: the player dialog loaded the embed as its own top-level document, which answers with "Video player configuration error -- Error 153". An embed has to be embedded. It now sits in an iframe on a minimal host page. Measured in the app's own WebView: top-level, no requests for the video stream and the error on screen; framed, the stream loads and there is no error. **"Long click on YouTube placeholder should open the URL directly."** It does now. Worth recording why this is not simply the link's href: for an image inside a link, WebView's hit test reports the image's URL, so a long-press on the thumbnail arrives as the i.ytimg.com JPEG. The video id is resolved out of that. Verified on a device for all four: the Substack site URL adds "Letters from an American" and fetches 20 articles; a West Hawaii Today article extracts with no Trending headers and its body intact; the video plays; and a long-press reaches the YouTube app. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- .../terst/feedreader/readability/EvalDecision.kt | 9 +- .../feedreader/readability/RecirculationWidgets.kt | 128 +++++++++++++++++++++ 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt (limited to 'app/src/main/java/org/terst/feedreader/readability') diff --git a/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt b/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt index ed0033d..dd77014 100644 --- a/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt +++ b/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt @@ -39,7 +39,14 @@ internal fun decideEvalOutcome( !isConsentNotice return when { - usable -> EvalDecision.Done(ExtractionResult.Success(article!!)) + // Cleaned on the way out rather than at the point of display: this is the one place + // every fresh extraction passes through, so the widgets never reach the screen and + // never reach `articles.extractedContent` either. + usable -> EvalDecision.Done( + ExtractionResult.Success( + article!!.copy(content = RecirculationWidgets.strip(article.content)) + ) + ) // Readability ran and objected: that verdict will not change. readabilityError != null -> diff --git a/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt b/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt new file mode 100644 index 0000000..a6515c1 --- /dev/null +++ b/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt @@ -0,0 +1,128 @@ +package org.terst.feedreader.readability + +/** + * Removes "recirculation" widgets -- the promo blocks a publisher's widget platform injects + * into the middle of the article body to send the reader off to other stories. + * + * Reported live: West Hawaii Today articles "get a bunch of stray 'Trending' headers". + * Confirmed by running the vendored Readability.js over the rendered page for the 12 articles + * in that feed: 9 carried two surviving widgets each. The three that didn't were paywalled + * stubs too short for the widgets to be reached. + * + * Readability keeps them because they are not in a sidebar -- they are inside the article's own + * container, between real paragraphs, so every structural signal says they belong. What arrives + * is a heading labelling nothing, because the list of links under it is filled in later by a + * script that never runs in the reader: + * + * ``` + *

Trending

+ * ``` + * + * The signal is the attribute, not the word "Trending": `data-mrf-recirculation` says what the + * block is for, in the widget platform's own vocabulary, and no article body is ever inside one. + * Matching on the heading text instead would be both narrower (the label varies by site and + * section) and more dangerous (an article may legitimately discuss what is trending). + * + * [MAX_WIDGET_TEXT_LENGTH] is the same safety valve [ConsentNotice] uses: however the attribute + * is applied, nothing with real writing in it is ever thrown away. + */ +object RecirculationWidgets { + + /** + * Text length past which a block is kept whatever attribute it carries. A widget is a label + * and a few headlines; anything longer is somebody's writing and is not ours to delete. + */ + private const val MAX_WIDGET_TEXT_LENGTH = 600 + + /** + * Attributes marking a block as recirculation. Only ones whose own name says so -- this list + * grows on evidence from a real article, never on a guess about what a vendor prefix means. + */ + private val WIDGET_ATTRIBUTES = listOf("data-mrf-recirculation") + + private val OPENING_DIV = Regex("]*>", RegexOption.IGNORE_CASE) + private val TAG_REGEX = Regex("<[^>]*>") + private val WHITESPACE_REGEX = Regex("\\s+") + + private const val OPEN_DIV = "` closing the div that opened before [from], or null when + * the markup never closes it. + */ + private fun matchingClose(html: String, from: Int): Int? { + var depth = 1 + var i = from + var end: Int? = null + while (end == null && i < html.length) { + val next = html.indexOf('<', i) + when { + next < 0 -> i = html.length + html.startsWith(CLOSE_DIV, next, ignoreCase = true) -> { + val close = html.indexOf('>', next) + depth-- + when { + close < 0 -> i = html.length + depth == 0 -> end = close + 1 + else -> i = close + 1 + } + } + html.startsWith(OPEN_DIV, next, ignoreCase = true) -> { + depth++ + i = next + OPEN_DIV.length + } + else -> i = next + 1 + } + } + return end + } + + private fun tooLongToBeAWidget(block: String): Boolean = + block.replace(TAG_REGEX, " ").replace(WHITESPACE_REGEX, " ").trim().length > MAX_WIDGET_TEXT_LENGTH +} -- cgit v1.2.3