From 9210bf463a8818595f7ca42486a787d4f1941d55 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Wed, 21 Oct 2026 19:07:17 +0000 Subject: Fix stray promo headers, site-URL subscribing, and YouTube playback Three reported bugs. **"West Hawaii today articles get a bunch of stray Trending headers."** The publisher's widget platform injects promo blocks between the real paragraphs, inside the article's own container, so every structural signal tells Readability they belong. What survives is a heading labelling nothing, because the links under it are filled in by a script that never runs in the reader. Confirmed on 9 of the 12 articles in that feed, two each. Blocks marked `data-mrf-recirculation` are now removed -- the attribute says what the block is for in the widget platform's own words, which is a better signal than the word "Trending", since an article may legitimately discuss what is trending. Nothing longer than a label and a few headlines is ever removed. Also applied to already-cached articles, in place rather than by discarding them: the article was fine, only the litter has to go. **"Can't subscribe to heathercoxrichardson.substack.com."** Adding a feed handed whatever URL it was given straight to the XML parser, so a site's home page -- the only URL a reader can actually see and copy -- failed on the root element. That page advertises its feed the way sites have since 2002, in a in the head. It is now read, resolved against the page, and fetched. Only what the page declares: guessing /feed and /rss would add a burst of 404s to every failed add and eventually subscribe somebody to the wrong thing. **"YouTube play is misconfigured."** It was, and YouTube said so: the player dialog loaded the embed as its own top-level document, which answers with "Video player configuration error -- Error 153". An embed has to be embedded. It now sits in an iframe on a minimal host page. Measured in the app's own WebView: top-level, no requests for the video stream and the error on screen; framed, the stream loads and there is no error. **"Long click on YouTube placeholder should open the URL directly."** It does now. Worth recording why this is not simply the link's href: for an image inside a link, WebView's hit test reports the image's URL, so a long-press on the thumbnail arrives as the i.ytimg.com JPEG. The video id is resolved out of that. Verified on a device for all four: the Substack site URL adds "Letters from an American" and fetches 20 articles; a West Hawaii Today article extracts with no Trending headers and its body intact; the video plays; and a long-press reaches the YouTube app. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- .../org/terst/feedreader/data/CachedExtraction.kt | 10 +- .../org/terst/feedreader/data/FeedDiscovery.kt | 88 ++++++++++++++ .../terst/feedreader/readability/EvalDecision.kt | 9 +- .../feedreader/readability/RecirculationWidgets.kt | 128 +++++++++++++++++++++ .../java/org/terst/feedreader/ui/AddFeedScreen.kt | 65 ++++++++--- .../org/terst/feedreader/ui/LinkReaderScreen.kt | 1 + .../java/org/terst/feedreader/ui/ReaderScreen.kt | 1 + .../java/org/terst/feedreader/ui/ReaderWebView.kt | 32 +++++- .../java/org/terst/feedreader/ui/YouTubeLinks.kt | 127 ++++++++++++++++++++ 9 files changed, 443 insertions(+), 18 deletions(-) create mode 100644 app/src/main/java/org/terst/feedreader/data/FeedDiscovery.kt create mode 100644 app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt create mode 100644 app/src/main/java/org/terst/feedreader/ui/YouTubeLinks.kt (limited to 'app/src/main/java') diff --git a/app/src/main/java/org/terst/feedreader/data/CachedExtraction.kt b/app/src/main/java/org/terst/feedreader/data/CachedExtraction.kt index 3a791cc..63614d1 100644 --- a/app/src/main/java/org/terst/feedreader/data/CachedExtraction.kt +++ b/app/src/main/java/org/terst/feedreader/data/CachedExtraction.kt @@ -1,6 +1,7 @@ package org.terst.feedreader.data import org.terst.feedreader.readability.ConsentNotice +import org.terst.feedreader.readability.RecirculationWidgets import org.terst.feedreader.readability.UnrenderedTex /** What a cached `articles.extractedContent` value turned out to be worth. */ @@ -58,7 +59,14 @@ suspend fun loadCachedExtraction( discardedUnrenderedTex = true ) else -> null - } ?: return CachedExtraction(content = cached, discardedConsentNotice = false) + } ?: return CachedExtraction( + // Recirculation widgets are deliberately not in the `discard` list above. A cached + // article carrying the publisher's promo blocks is a good article with litter in it, + // not a poisoned row: cleaning it in place shows the right thing immediately, where + // discarding it would throw away a usable body and fetch the identical one back. + content = RecirculationWidgets.strip(cached), + discardedConsentNotice = false + ) clearCache() return discard diff --git a/app/src/main/java/org/terst/feedreader/data/FeedDiscovery.kt b/app/src/main/java/org/terst/feedreader/data/FeedDiscovery.kt new file mode 100644 index 0000000..98ce5b3 --- /dev/null +++ b/app/src/main/java/org/terst/feedreader/data/FeedDiscovery.kt @@ -0,0 +1,88 @@ +package org.terst.feedreader.data + +import java.net.URI + +/** + * Finds the feed a web page advertises. + * + * Reported live: "Can't subscribe to https://heathercoxrichardson.substack.com/". Adding a feed + * fetched whatever URL it was given and handed it straight to [FeedParser], so a site's home + * page -- the only URL a reader can actually see and copy -- failed on the XML parse with a + * message about an unrecognised root element. + * + * Nothing exotic is needed to fix it. Sites have advertised their feeds in the document head + * since RSS autodiscovery was agreed in 2002, and that Substack page does: + * + * ``` + * + * ``` + * + * Only what the page itself declares is used. Guessing paths (`/feed`, `/rss`, `/index.xml`) + * would add a burst of 404s on every failed add and would eventually subscribe somebody to the + * wrong thing; a site that declares nothing is reported as having no feed, which is the truth. + */ +object FeedDiscovery { + + /** Feed content types, as `type=` gives them. */ + private val FEED_TYPES = listOf("application/rss+xml", "application/atom+xml", "application/rdf+xml", "text/xml") + + private val LINK_TAG = Regex("]*>", RegexOption.IGNORE_CASE) + + /** `name="value"`, `name='value'` or `name=value`, in any order within the tag. */ + private val ATTRIBUTE = + Regex("""(?[a-zA-Z-]+)\s*=\s*("(?[^"]*)"|'(?[^']*)'|(?[^\s">]+))""") + + /** + * The feed URL [html] advertises, absolute, or null when it advertises none. + * + * [pageUrl] is the address the HTML came from, which a relative href resolves against -- + * against the page, not the site root, since a feed link on `/blog/` routinely says + * `feed.xml` and means `/blog/feed.xml`. + */ + fun findFeedUrl(html: String, pageUrl: String): String? { + // Already a feed: there is nothing to discover, and scanning XML for tags would + // only find Atom's own self-links and send the caller round in a circle. + if (html.trimStart().startsWith(" + attrs["rel"]?.contains("alternate", ignoreCase = true) == true && + attrs["type"]?.lowercase()?.trim() in FEED_TYPES + } + ?.get("href") + ?.takeIf { it.isNotBlank() } + ?.let { resolve(it, pageUrl) } + } + + private fun attributesOf(tag: String): Map = + ATTRIBUTE.findAll(tag).associate { match -> + // Named rather than numbered: which of the three quoting alternatives matched is + // the whole question here, and "groupValues[4]" does not say that to a reader. + val value = match.groups["dq"]?.value + ?: match.groups["sq"]?.value + ?: match.groups["bare"]?.value.orEmpty() + match.groups["name"]!!.value.lowercase() to value + } + + /** + * [href] as an absolute https URL. + * + * The https upgrade is the same rule the add-feed screen already applies to what the user + * types, for the same reason: Android blocks cleartext, so an http feed would fail every + * fetch forever. A site that advertises an http feed almost always serves it over https too. + */ + private fun resolve(href: String, pageUrl: String): String? = try { + val absolute = URI(pageUrl).resolve(href.trim()).toString() + if (absolute.startsWith("http://", ignoreCase = true)) { + "https://" + absolute.removePrefix("http://").removePrefix("HTTP://") + } else { + absolute + } + } catch (ignored: IllegalArgumentException) { + // A malformed href is a page problem, not a crash: report no feed and let the caller + // say so, the same as a page that advertises none. Nothing is lost by not reporting + // it -- "no feed found at that address" is exactly as true and as useful either way. + null + } +} diff --git a/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt b/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt index ed0033d..dd77014 100644 --- a/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt +++ b/app/src/main/java/org/terst/feedreader/readability/EvalDecision.kt @@ -39,7 +39,14 @@ internal fun decideEvalOutcome( !isConsentNotice return when { - usable -> EvalDecision.Done(ExtractionResult.Success(article!!)) + // Cleaned on the way out rather than at the point of display: this is the one place + // every fresh extraction passes through, so the widgets never reach the screen and + // never reach `articles.extractedContent` either. + usable -> EvalDecision.Done( + ExtractionResult.Success( + article!!.copy(content = RecirculationWidgets.strip(article.content)) + ) + ) // Readability ran and objected: that verdict will not change. readabilityError != null -> diff --git a/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt b/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt new file mode 100644 index 0000000..a6515c1 --- /dev/null +++ b/app/src/main/java/org/terst/feedreader/readability/RecirculationWidgets.kt @@ -0,0 +1,128 @@ +package org.terst.feedreader.readability + +/** + * Removes "recirculation" widgets -- the promo blocks a publisher's widget platform injects + * into the middle of the article body to send the reader off to other stories. + * + * Reported live: West Hawaii Today articles "get a bunch of stray 'Trending' headers". + * Confirmed by running the vendored Readability.js over the rendered page for the 12 articles + * in that feed: 9 carried two surviving widgets each. The three that didn't were paywalled + * stubs too short for the widgets to be reached. + * + * Readability keeps them because they are not in a sidebar -- they are inside the article's own + * container, between real paragraphs, so every structural signal says they belong. What arrives + * is a heading labelling nothing, because the list of links under it is filled in later by a + * script that never runs in the reader: + * + * ``` + *

Trending

+ * ``` + * + * The signal is the attribute, not the word "Trending": `data-mrf-recirculation` says what the + * block is for, in the widget platform's own vocabulary, and no article body is ever inside one. + * Matching on the heading text instead would be both narrower (the label varies by site and + * section) and more dangerous (an article may legitimately discuss what is trending). + * + * [MAX_WIDGET_TEXT_LENGTH] is the same safety valve [ConsentNotice] uses: however the attribute + * is applied, nothing with real writing in it is ever thrown away. + */ +object RecirculationWidgets { + + /** + * Text length past which a block is kept whatever attribute it carries. A widget is a label + * and a few headlines; anything longer is somebody's writing and is not ours to delete. + */ + private const val MAX_WIDGET_TEXT_LENGTH = 600 + + /** + * Attributes marking a block as recirculation. Only ones whose own name says so -- this list + * grows on evidence from a real article, never on a guess about what a vendor prefix means. + */ + private val WIDGET_ATTRIBUTES = listOf("data-mrf-recirculation") + + private val OPENING_DIV = Regex("]*>", RegexOption.IGNORE_CASE) + private val TAG_REGEX = Regex("<[^>]*>") + private val WHITESPACE_REGEX = Regex("\\s+") + + private const val OPEN_DIV = "` closing the div that opened before [from], or null when + * the markup never closes it. + */ + private fun matchingClose(html: String, from: Int): Int? { + var depth = 1 + var i = from + var end: Int? = null + while (end == null && i < html.length) { + val next = html.indexOf('<', i) + when { + next < 0 -> i = html.length + html.startsWith(CLOSE_DIV, next, ignoreCase = true) -> { + val close = html.indexOf('>', next) + depth-- + when { + close < 0 -> i = html.length + depth == 0 -> end = close + 1 + else -> i = close + 1 + } + } + html.startsWith(OPEN_DIV, next, ignoreCase = true) -> { + depth++ + i = next + OPEN_DIV.length + } + else -> i = next + 1 + } + } + return end + } + + private fun tooLongToBeAWidget(block: String): Boolean = + block.replace(TAG_REGEX, " ").replace(WHITESPACE_REGEX, " ").trim().length > MAX_WIDGET_TEXT_LENGTH +} diff --git a/app/src/main/java/org/terst/feedreader/ui/AddFeedScreen.kt b/app/src/main/java/org/terst/feedreader/ui/AddFeedScreen.kt index 1e8814c..9f817ab 100644 --- a/app/src/main/java/org/terst/feedreader/ui/AddFeedScreen.kt +++ b/app/src/main/java/org/terst/feedreader/ui/AddFeedScreen.kt @@ -33,11 +33,13 @@ import kotlinx.coroutines.launch import kotlinx.coroutines.withContext import okhttp3.Request import org.terst.feedreader.data.AppDatabase +import org.terst.feedreader.data.FeedDiscovery import org.terst.feedreader.data.FeedEntity import org.terst.feedreader.data.FeedParser import org.terst.feedreader.data.Net import org.terst.feedreader.opml.Opml import org.terst.feedreader.work.FeedFetchWorker +import java.io.ByteArrayInputStream @OptIn(ExperimentalMaterial3Api::class) @Composable @@ -168,25 +170,60 @@ fun AddFeedScreen(onDone: () -> Unit) { private fun preferHttps(url: String): String = if (url.startsWith("http://", ignoreCase = true)) "https://" + url.removePrefix("http://") else url +/** + * Fetches [url] and returns its bytes, or null when the server refused. + * + * Read into memory rather than parsed from the live stream because the bytes may need looking + * at twice: once as a feed, and -- when that is not what the URL turned out to be -- again as a + * web page advertising where its feed actually lives. A response body is a one-shot stream, so + * the second look is only possible if the first did not consume it. + */ +private class Fetched(val bytes: ByteArray?, val error: String?) + +private fun fetchBytes(url: String): Fetched = + Net.client.newCall(Request.Builder().url(url).build()).execute().use { response -> + when { + !response.isSuccessful -> Fetched(null, "Couldn't fetch feed: HTTP ${response.code}") + else -> response.body?.let { Fetched(it.bytes(), null) } ?: Fetched(null, "Empty response") + } + } + private suspend fun addFeedByUrl(context: android.content.Context, db: AppDatabase, feedUrl: String): String = withContext(Dispatchers.IO) { - val url = preferHttps(feedUrl) + val typedUrl = preferHttps(feedUrl) try { - val response = Net.client.newCall(Request.Builder().url(url).build()).execute() - response.use { - if (!it.isSuccessful) return@withContext "Couldn't fetch feed: HTTP ${it.code}" - val body = it.body ?: return@withContext "Empty response" - val parsed = FeedParser.parse(body.byteStream()) - // Same REPLACE-wipes-other-fields concern as the OPML import path. - val existing = db.feedDao().getById(url) - val entity = existing?.copy( - title = parsed.title.ifBlank { url }, - siteUrl = parsed.siteUrl ?: existing.siteUrl - ) ?: FeedEntity(id = url, title = parsed.title.ifBlank { url }, siteUrl = parsed.siteUrl) - db.feedDao().upsert(entity) + val fetched = fetchBytes(typedUrl) + val bytes = fetched.bytes ?: return@withContext fetched.error ?: "Empty response" + + // What the user had to hand is usually the site, not the feed -- a site advertises + // its feed in its head, it does not print the URL anywhere a reader can copy. So a + // failed parse is not the answer yet: look for what the page says its feed is and + // try that, once. See FeedDiscovery. + var url = typedUrl + var parsed = runCatching { FeedParser.parse(ByteArrayInputStream(bytes)) }.getOrNull() + if (parsed == null) { + val discovered = FeedDiscovery.findFeedUrl(String(bytes, Charsets.UTF_8), typedUrl) + ?: return@withContext "No feed found at that address" + val feed = fetchBytes(discovered) + val feedBytes = feed.bytes ?: return@withContext feed.error ?: "Empty response" + // One hop only: whatever the page pointed at has to be a feed. Following a + // second page would be a redirect chain nobody asked for. + parsed = FeedParser.parse(ByteArrayInputStream(feedBytes)) + url = discovered } + + // Same REPLACE-wipes-other-fields concern as the OPML import path. + val existing = db.feedDao().getById(url) + val entity = existing?.copy( + title = parsed.title.ifBlank { url }, + siteUrl = parsed.siteUrl ?: existing.siteUrl + ) ?: FeedEntity(id = url, title = parsed.title.ifBlank { url }, siteUrl = parsed.siteUrl) + db.feedDao().upsert(entity) + FeedFetchWorker.fetchNow(context) - "Added" + // Says which feed was added when it is not the URL that was typed, so a site URL + // quietly becoming a different feed URL is visible rather than surprising later. + if (url == typedUrl) "Added" else "Added ${parsed.title.ifBlank { url }}" } catch (e: Exception) { "Couldn't add feed: ${e.message}" } diff --git a/app/src/main/java/org/terst/feedreader/ui/LinkReaderScreen.kt b/app/src/main/java/org/terst/feedreader/ui/LinkReaderScreen.kt index 72f6c1f..9f9f74a 100644 --- a/app/src/main/java/org/terst/feedreader/ui/LinkReaderScreen.kt +++ b/app/src/main/java/org/terst/feedreader/ui/LinkReaderScreen.kt @@ -258,6 +258,7 @@ fun LinkReaderScreen(url: String, onBack: () -> Unit, onOpenLink: (String) -> Un ReaderWebViewCallbacks( onOpenLink = onOpenLink, onLongPressLink = { longPressedLink = it }, + onOpenYouTubeExternally = { context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse(it))) }, onYouTubeVideo = { youtubeVideoId = it }, onImageTap = { zoomedImageUrl = it }, currentFontSizePx = { latestFontSizePx }, diff --git a/app/src/main/java/org/terst/feedreader/ui/ReaderScreen.kt b/app/src/main/java/org/terst/feedreader/ui/ReaderScreen.kt index 9952dc5..0d326ea 100644 --- a/app/src/main/java/org/terst/feedreader/ui/ReaderScreen.kt +++ b/app/src/main/java/org/terst/feedreader/ui/ReaderScreen.kt @@ -599,6 +599,7 @@ private fun ReaderContent( ReaderWebViewCallbacks( onOpenLink = onOpenLink, onLongPressLink = { longPressedLink = it }, + onOpenYouTubeExternally = { context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse(it))) }, onYouTubeVideo = { youtubeVideoId = it }, onImageTap = { zoomedImageUrl = it }, onScrollChanged = { scrollY -> ReaderScrollPositions.set(articleId, scrollY) }, diff --git a/app/src/main/java/org/terst/feedreader/ui/ReaderWebView.kt b/app/src/main/java/org/terst/feedreader/ui/ReaderWebView.kt index f1400d3..78020a9 100644 --- a/app/src/main/java/org/terst/feedreader/ui/ReaderWebView.kt +++ b/app/src/main/java/org/terst/feedreader/ui/ReaderWebView.kt @@ -74,6 +74,11 @@ private fun extractYouTubeVideoId(uri: Uri): String? { * WebView's own default long-press behavior (a native "open/copy link" popup) -- * callers show their own save/share/open action menu instead. * + * onOpenYouTubeExternally fires for a LONG-press on a YouTube link, with the watch URL, instead + * of the ordinary onLongPressLink actions menu -- callers hand it to the system, which is the + * YouTube app on a phone that has one. A tap plays the video in the reader; a long-press is how + * you say you wanted it somewhere else. + * * onYouTubeVideo fires for a tap on a YouTube watch link (what ReaderExtractor turns * an embedded video into a thumbnail link to) with just the video ID, instead of the * old behavior of handing off to the YouTube app/browser -- callers show an in-app @@ -124,6 +129,7 @@ private fun extractYouTubeVideoId(uri: Uri): String? { data class ReaderWebViewCallbacks( val onOpenLink: (String) -> Unit, val onLongPressLink: (String) -> Unit = {}, + val onOpenYouTubeExternally: (String) -> Unit = {}, val onYouTubeVideo: (String) -> Unit = {}, val onImageTap: (String) -> Unit = {}, val onScrollChanged: (Int) -> Unit = {}, @@ -245,7 +251,19 @@ fun newReaderWebView(context: Context, callbacks: ReaderWebViewCallbacks): Reade // Every article image now sits inside a zoom link, so a long-press on one // reports that link's internal scheme -- strip it back to the image's own URL // so the actions menu shows, and acts on, something real. - callbacks.onLongPressLink(withoutImageZoomScheme(url)) + val target = withoutImageZoomScheme(url) + // A long-press on a video reports the thumbnail's URL rather than the link's -- + // see youTubeTargetForLongPress, which resolves it back to the video. Straight + // out to whatever handles YouTube on the phone, usually the app: a tap already + // plays the video in the reader, so the only thing a long-press can usefully + // mean is "not in here, out there", and picking that off a menu would be a menu + // with one useful item. + val video = youTubeTargetForLongPress(target) + if (video != null) { + callbacks.onOpenYouTubeExternally(video) + } else { + callbacks.onLongPressLink(target) + } true } else { false @@ -380,7 +398,17 @@ fun YouTubePlayerDialog(videoId: String, onDismiss: () -> Unit) { } } webChromeClient = WebChromeClient() - loadUrl("https://www.youtube-nocookie.com/embed/$videoId?autoplay=1&playsinline=1") + // A host page with the embed in an iframe, not the embed itself as the + // top-level document. YouTube answers a top-level embed with "Video player + // configuration error -- Error 153", which is the reported "YouTube play is + // misconfigured"; framed, the same URL plays. See YouTubeLinks. + loadDataWithBaseURL( + YOUTUBE_EMBED_BASE_URL, + youTubeEmbedPageHtml(videoId), + "text/html", + "utf-8", + null + ) } }, onRelease = { it.destroy() } diff --git a/app/src/main/java/org/terst/feedreader/ui/YouTubeLinks.kt b/app/src/main/java/org/terst/feedreader/ui/YouTubeLinks.kt new file mode 100644 index 0000000..e2b3b62 --- /dev/null +++ b/app/src/main/java/org/terst/feedreader/ui/YouTubeLinks.kt @@ -0,0 +1,127 @@ +package org.terst.feedreader.ui + +/** + * Recognising a YouTube placeholder link, and building a page its embed will actually play in. + * + * Split out of ReaderWebView for the usual reason: this is string work with no WebView, no + * gestures and no Compose in it, so it can be exercised without a device -- which matters here + * because the bug it fixes is invisible until a video is tapped. + * + * **Why the embed needs a page at all.** The player dialog used to call + * `loadUrl("https://www.youtube-nocookie.com/embed/")`, loading the embed as the WebView's + * own top-level document. YouTube refuses that: the player comes up showing "Video player + * configuration error -- Error 153", which is reported as "YouTube play is misconfigured". + * Confirmed by loading the same URL three ways in a real browser on an Android user agent: + * + * | how the embed is loaded | result | + * |------------------------------------------|-----------------------------| + * | top-level, as the app did | Error 153 | + * | top-level with `&origin=` | Error 153 | + * | in an `""" + + """""" +} + +/** + * The id with anything outside YouTube's own alphabet removed. + * + * The ids reaching here are already matched against `[a-zA-Z0-9_-]{11}` upstream, so this + * changes nothing today. It exists because the value is spliced straight into an HTML + * attribute, and that should be safe on its own terms rather than because of a check in a + * different file that a later edit might move. + */ +private fun scrub(videoId: String): String = VIDEO_ID_CHARS.replace(videoId, "") + +/** The path of [url], or "/" when it has none. */ +private fun pathOf(url: String): String { + val afterScheme = url.substringAfter("://", missingDelimiterValue = "") + val afterAuthority = afterScheme.substringAfter('/', missingDelimiterValue = "") + val path = "/" + afterAuthority.substringBefore('?').substringBefore('#') + return path +} + +/** The host of [url], lowercased, or null when it is not a URL with one. */ +private fun hostOf(url: String): String? { + val afterScheme = url.substringAfter("://", missingDelimiterValue = "") + if (afterScheme.isEmpty()) return null + val authority = afterScheme.substringBefore('/').substringBefore('?').substringBefore('#') + val host = authority.substringAfterLast('@').substringBefore(':') + return host.lowercase().ifEmpty { null } +} -- cgit v1.2.3