| Age | Commit message (Collapse) | Author |
|---|
|
"Remove giant Google icon", reported on a West Hawaii Today article. It is the
"Add as a preferred source on Google" badge that sits beside the byline. Giant
because the page caps it at 120px with its own stylesheet and the reader drops
the page's CSS, so a small badge on the site becomes a full-width image above
the first paragraph.
Matched on the link's address rather than the class, and that is forced rather
than preferred: running the vendored Readability.js over the real page shows
the wrapper's preferred_source_badge class does not survive extraction at all,
so a class rule would match nothing. The anchor does survive, and its address
is Google's own and identical for every publisher in the programme -- where the
badge's image filename sits under this newspaper's theme directory and is not.
So the address is both the only signal left and the one that generalises.
A link with words in it is never touched. An article may discuss the programme
and link to it in a sentence, and that sentence is the author's.
Both cleaners now run from one entry point. There are two places a body is
cleaned -- a fresh extraction on the way out, a cached one on the way back in --
and a cached article is cleaned in place rather than discarded, so a cleaner
wired into only the fresh path would leave every already-read article carrying
litter nothing would ever remove. Peter's report says the article was cached,
which is exactly that case.
Pinned against the real thing: the reported article, put through the real
extractor, loses the badge and keeps its text and byline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three reported bugs.
**"West Hawaii today articles get a bunch of stray Trending headers."**
The publisher's widget platform injects promo blocks between the real
paragraphs, inside the article's own container, so every structural
signal tells Readability they belong. What survives is a heading
labelling nothing, because the links under it are filled in by a script
that never runs in the reader. Confirmed on 9 of the 12 articles in
that feed, two each. Blocks marked `data-mrf-recirculation` are now
removed -- the attribute says what the block is for in the widget
platform's own words, which is a better signal than the word
"Trending", since an article may legitimately discuss what is trending.
Nothing longer than a label and a few headlines is ever removed. Also
applied to already-cached articles, in place rather than by discarding
them: the article was fine, only the litter has to go.
**"Can't subscribe to heathercoxrichardson.substack.com."** Adding a
feed handed whatever URL it was given straight to the XML parser, so a
site's home page -- the only URL a reader can actually see and copy --
failed on the root element. That page advertises its feed the way sites
have since 2002, in a <link rel="alternate"> in the head. It is now
read, resolved against the page, and fetched. Only what the page
declares: guessing /feed and /rss would add a burst of 404s to every
failed add and eventually subscribe somebody to the wrong thing.
**"YouTube play is misconfigured."** It was, and YouTube said so: the
player dialog loaded the embed as its own top-level document, which
answers with "Video player configuration error -- Error 153". An embed
has to be embedded. It now sits in an iframe on a minimal host page.
Measured in the app's own WebView: top-level, no requests for the video
stream and the error on screen; framed, the stream loads and there is
no error.
**"Long click on YouTube placeholder should open the URL directly."**
It does now. Worth recording why this is not simply the link's href:
for an image inside a link, WebView's hit test reports the image's URL,
so a long-press on the thumbnail arrives as the i.ytimg.com JPEG. The
video id is resolved out of that.
Verified on a device for all four: the Substack site URL adds "Letters
from an American" and fetches 20 articles; a West Hawaii Today article
extracts with no Trending headers and its body intact; the video plays;
and a long-press reaches the YouTube app.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Reported against lcamtuf's Substack ("we should render tex",
blog.coredump.cx/p/yours-truly): reader mode showed raw
"\(p = \text{...}\)" where equations should be.
What the page actually ships is the TeX SOURCE, as text, inside a
"latex-rendered" div -- Substack renders it in the browser with its own
script, which never runs here. Readability keeps that text verbatim (it
strips the class, not the text), so nothing was being discarded during
extraction and no amount of extraction-side repair would have helped:
there was simply never any rendered markup to keep.
So render it during extraction, where JavaScript is already running for
Readability's sake. KaTeX (vendored in assets, never fetched -- this
runs against untrusted third-party pages) converts each expression to
MathML before Readability sees the document, and the reader WebView
renders that natively. The reader keeps JavaScript OFF, as it must, and
needs no stylesheet, webfont or script of KaTeX's; the only reader-side
change is containing a wide equation so it scrolls within its own box
instead of dragging the whole article sideways.
Two details worth knowing:
- KaTeX refuses {align} outside display mode, and Substack wraps display
maths in INLINE delimiters (a block-styled span does the centring), so
a mode that renders nothing is retried in the other one. An expression
neither mode can render is left as source -- no worse than before.
- The reported article came with "extractedContent cached: true", so the
raw source was already on disk and every open read it back. Recognised
and dropped on read (same mechanism as the cached consent notice),
which re-extracts the article the next time it is opened.
APK grows by ~90KB: the vendored katex.min.js (267KB) compresses to 91KB
in the APK. No new permissions, no schema change, no new JS-enabled
surface.
Verified on the emulator against the live article: stale raw-TeX cache
discarded, article re-extracted, all four equations rendered, and a wide
align block scrolls inside itself.
|
|
Reported live: reader mode on a New Republic feature showed nothing but
the cookie disclaimer, and had cached it.
Reproduced outside the app by running ReaderExtractor's own injected
script (Readability.js and all) against the real page. With the article
body present, extraction is perfect. With the body absent from the DOM at
the moment the script runs, Readability's first pass -- which discards
"unlikely candidates", and the Didomi banner's class is
didomi-notice-banner -- comes back under its own 500-character threshold,
so it retries with that stripping switched off. The banner's single
paragraph is 510 characters, i.e. just over the threshold, so it comes
back as the article with the right title attached.
Nothing on the Kotlin side looked at that. Any non-blank content counted
as ExtractionResult.Success, was written to articles.extractedContent,
and was never reconsidered -- so one bad load became a permanent cookie
disclaimer, which is the "extractedContent cached: true" in the report.
ConsentNotice recognises such a result: a hard plain-text length cap
first (nothing long is ever rejected, however much consent boilerplate it
quotes), then either a known CMP container id -- Readability strips
classes but keeps ids -- or two or more distinct consent phrases.
The accept/retry/give-up decision moves out of the WebView callback into
decideEvalOutcome so it can be tested without a device. A consent notice
is now "not ready yet", which routes it into the retry the extractor
already does for an empty result; if it survives all six attempts that is
what the page serves, and it fails as ExtractionFailure.ConsentInterstitial
rather than being accepted.
A fixed extractor cannot repair what is already on disk, so
loadCachedExtraction drops a cached notice on read and re-extracts, in
both ReaderScreen and LinkReaderScreen. No schema change: the repair
happens when the article is next opened. Reader mode is still never
turned on heuristically -- replacing a discarded notice does not flip it.
The bug report now also carries the last extraction error, without which
a report about bad reader content cannot say whether extraction failed.
Tests: 59 -> 78.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Bug report: extraction failed on one 1500days.com article while others worked.
It is not that article. Extraction was racing the page load, and the report
could say nothing more because the app discarded the reason.
Two separate defects.
1. The race. onPageFinished fires before the page has finished loading —
observed at progress=25 on this article — and the extraction script was
injected there. Sometimes the DOM is far enough along; sometimes
evaluateJavascript hands back a bare "null" because the document is still
being replaced. Measured against the reported URL: 4 successes in 6
attempts before, 10 in 10 after.
An unusable result is no longer final. A null or empty-content result is
retried (6 attempts, 500ms apart, well inside the existing 25s timeout); a
Readability error still fails immediately, since retrying will not change
what the parser thinks of the markup.
2. The silence. ReaderScreen did `extracted?.content?.let { }`, so every
failure — dead host, HTTP error, timeout, parser throw — arrived as the same
null and was dropped. The reader just kept showing raw feed content. The
only way a failure ever surfaced was the user noticing.
extractResult() now returns a typed ExtractionFailure carrying the reason;
ReaderScreen logs the diagnostic and shows the user a short explanation.
Also replaced a dead error callback. The only override was the four-argument
onReceivedError, which Android stopped calling at API 23; minSdk is 26, so it
never fired on any supported device. A failed load therefore invoked nothing
and hung for the full 25 seconds before surfacing as an indistinguishable null.
Now overrides the request/error and onReceivedHttpError forms Android actually
calls, both filtered to the main frame so a 404'd tracker cannot abort an
article that renders fine.
The instrumented test does NOT reproduce the race and says so in its own
docstring: a MockWebServer page finishes loading before onPageFinished fires,
and slow subresources do not help because WebView waits for them. It is kept as
a reliability guard over repeated back-to-back extractions. The race was
verified by hand against the live URL.
47 unit tests, 1 instrumented test, both green.
|
|
- Cache the ~90KB Readability.js asset read (identical every time)
instead of a disk read + UTF-8 decode on the main thread for every
single extraction.
- Lock down allowFileAccess/allowContentAccess/allowUniversalAccess:
this WebView runs full JS against any article link, feed URL, or
(via ShareReceiverActivity) any URL another app hands it -- no
legitimate reason to ever load file://.
- hq720 thumbnail instead of hqdefault (a letterboxed 4:3 crop).
- Validate frameSrc is actually http(s) before using it verbatim as a
fallback link -- it's attacker-controlled article content that only
had to match a substring filter, not the strict video-ID regex, so a
crafted file://, content://, or intent:// src couldn't ride along
into an ACTION_VIEW intent or a JS-enabled WebView.loadUrl.
|
|
Ars splits a longer article's body across multiple sibling
.post-content.post-content-double divs (with a mid-content ad rail
between them) instead of one continuous container. Readability scores
each as an independent candidate and keeps only the highest-scoring
one, silently dropping the rest -- confirmed live on "Reaction wheel
failures leave Swift rescue mission spinning in orbit", which lost its
first 7 paragraphs (lede through the reaction-wheel explainer).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Zwte3cyUJFUVycc3sDmjM
|
|
Stash creds before saving
- Bottom bar: Previous/Keep unread/Next are now plain text labels, no icons
- Add "Re-download and re-extract" to the overflow menu (ReaderScreen and
LinkReaderScreen) to force a fresh fetch + Readability pass on demand,
overwriting whatever's cached
- Fix a real compile bug in the previous commit's reader-mode-off handler
(passed null into a non-null setExtractedContent) via a proper
clearExtractedContent() DAO query
- YouTube embeds: fixed the broken-image-glyph bug two ways -- (1) an
11-char video-ID regex was accidentally matching a playlist embed's
literal "videoseries" placeholder path, building a thumbnail URL for a
video that doesn't exist, and (2) Readability's link-density cleanup was
discarding the replacement entirely (a bare <a><img></a> reads as an
ad/share unit). Restructured as a <figure>/<figcaption> pair -- the same
shape a normal captioned photo takes -- confirmed live this survives
extraction intact
- Stash settings: API key field is now a password field with a show/hide
toggle, and "Verify and save" makes a real authenticated request before
persisting anything, showing why it failed instead of silently saving
bad credentials
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
Ars sequel chrome
- Feed list: folders now collapse (start collapsed, with a per-folder unread
count so collapsing doesn't hide "is there anything new")
- Long-pressing a link inside an article shows an action menu (open in
browser, share, save to Stash, save for later) instead of WebView's
default text-selection popup -- confirmed live end to end, including the
Stash/save-for-later paths creating a real standalone article
- New shared ensureStandaloneArticle() helper (data/ArticleSaving.kt),
factored out of ShareReceiverActivity, now also used by the long-press menu
- YouTube iframe embeds become tap-to-open thumbnails (JS is off in the
reader WebView, so the real player never worked anyway); tapping one goes
straight to the YouTube app/browser instead of our own article-extraction
pipeline
- Fixed back navigation through a chain of links-within-links: each hop was
stacking a new backstack entry, so back stepped through the chain one link
at a time instead of returning straight to the source article. Confirmed
live: A -> link -> B -> link -> C, one back-press now goes straight to A
- Extended the <header> chrome-stripping from last round: also strip
<footer>, an author bio card, and a "Most Read" trending-stories widget --
confirmed live one of the latter showed up as a giant unrelated photo plus
an empty numbered list (its lazy-loaded items never populate in this
hidden, off-screen extraction WebView)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
reader content
- Confirmed live (Ars Technica): a section-kicker badge, the dek/subhead, and
an orphaned duplicate hero-image caption were all leaking into extracted
content because they live in a <header> Readability doesn't exclude by tag,
only deprioritizes in scoring. Strip <header> from the DOM ourselves before
handing off to Readability -- a pre-processing decision on our side, not a
change to Readability's own algorithm, per instruction not to touch it.
- Render the article's full title as a heading at the top of the reader
content itself (wrapReaderHtml), not just the top bar, which truncates to
one line. This also replaces whatever heading Readability happened to keep
in-body, now that <header> (and any <h1> inside it) is stripped pre-parse.
- Bump extraction timeout 15s -> 25s: confirmed live, several heavy
real-world pages were timing out before onPageFinished ever fired, well
short of a real parsing failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
folder-organized feed list
- Rename app to FeeDreader with an owl launcher icon and a paywall-avoidance
note in Settings
- Move Open original/Save to Stash into the reader's overflow menu; add
Mark unread there too
- Only mark an article read when navigating away via Previous/Next, not on
open or on back
- Saved list no longer crosses out read items; its count includes them
- Back from Unread (the root screen) now opens the feed list instead of
exiting the app
- Feed list: clearer separator above the per-feed list, folder grouping
(from OPML categories) with case-insensitive sort, and a virtual "Last
fetch failed" section at the top
- Reader extraction: cap unsized decorative SVG icons (fixes Ars
Technica's giant section-badge icon), force native loading="lazy"
images to load eagerly in the hidden extraction WebView, and drop a
lightbox's duplicate caption only when a real figcaption sibling exists
- Backlog notes on swipe-to-next research, extraction test coverage, and
an async-rendered-gallery race confirmed live on Ars Technica
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
Reader mode was still switching kottke articles even after the v3
opt-out and the "un-hide only if thin" fallback -- both were still
heuristic (content-length-based), and kottke's posts are routinely short
by design, not paywalled, so the heuristics kept firing on ordinary
content. Removed the auto-extract-on-thin-content logic entirely: reader
mode now NEVER turns on by itself, under any condition.
- FeedEntity.readerModeDisabled -> readerModeEnabled (opt-in, default off
for every feed -- schema v4, full table rebuild since Room's migration
validation requires an exact column match and a bare ADD COLUMN left
the old column orphaned; reproduced live as a crash, "Migration didn't
properly handle: feeds").
- The reader's toggle icon now sets reader mode for the article's WHOLE
FEED, persists it immediately, and confirms via snackbar ("Reader mode
enabled/disabled for X") -- both in the reader screen and the Feeds
list's own toggle. This is the only way reader mode ever turns on.
- ReaderExtractor: replaced the "un-hide everything if thin" paywall
fallback (which is what pulled kottke's site footer into the result --
confirmed live, twice) with one that only un-hides elements whose
class/id actually names them as paywall/premium/subscriber content
(West Hawaii Today's hidden div is literally class="paywall
paywall_<id>"). This turned out not to be the kottke bug's actual cause
either -- see below -- but it's a real, independent correctness
improvement over the previous blanket version, so it stays.
- Also added: noscript-wrapped lazy-load image fallback (a common
pattern pairs a placeholder <img> with a real one inside <noscript>,
which our JS-enabled extraction WebView never parses into DOM per the
HTML spec) and lazy data-src/srcset resolution generally.
Root cause of the kottke footer bug, found by fetching and inspecting
the actual page structure rather than guessing again: it's not a
CSS-hidden-element issue at all. kottke's real post content lives in a
plain, undecorated <div class="post"> that's a sibling (not ancestor) of
an ads container and a "Socials & More" footer widget, all under one
shared <div class="grid-container">. Because kottke's posts are so
short, Readability's generic paragraph-density scoring doesn't find
enough signal in .post alone and climbs to a broader container that
happens to include the neighboring footer. No un-hide logic is involved
at any point -- this reproduces identically on the plain, unmodified
Readability pass. This is a genuine limit of heuristic content
extraction on very short-form posts, not something to keep patching with
more heuristics. Reset kottke's setting back to off (its correct
default) after testing.
|
|
Many sites lazy-load media via JS: the real URL sits in a data-src/
data-srcset attribute and only gets copied into src/srcset once the
site's own script runs -- which never happens in the reader WebView (JS
is off by design, so articles can't run arbitrary third-party scripts).
Left alone this showed either a placeholder/category icon stretched to
fill the article width (reported live with a screenshot: a giant flask
icon in place of the real photo on an Ars Technica article), or nothing
but alt text (reported for NYT articles' images and videos).
Before either Readability pass, walk <img>/<source> elements and copy
common lazy-load attributes (data-src, data-original, data-lazy-src,
data-srcset) into src/srcset, falling back to the first srcset candidate
if src is still empty. Also strip width/height attributes on images --
they're sized for the site's real photo, not whatever was in the
unresolved src, so our own max-width:100%/height:auto sizing now takes
over based on the (correctly resolved) image's actual dimensions.
Verified live: the Ars Technica article that previously showed the
stretched flask icon now renders clean, with no image where the broken
one used to appear and the full real article text intact.
|
|
- ReaderExtractor: the un-hide-before-extracting fix (for soft paywalls
like West Hawaii Today) was unconditional -- it also un-hid normal,
legitimately-collapsed chrome (kottke's footer/membership notice), which
Readability then folded into the result alongside the real article.
Reported live with a screenshot showing the extracted content wrapped in
site footer boilerplate. Now tries the normal (hidden-respecting) pass
first and only falls back to the un-hide pass if that came back thin
(<500 chars), keeping whichever pass actually found more text. Verified
live: the same kottke article now stays clean and stable (no delayed
switch to the footer version).
- New Unread/Older split: the default Unread view is now clamped to the
last 72 hours (RECENT_WINDOW_MS); the rest of the unread backlog moved
to a new "Older" view (new FeedListScreen row, ArticleDao.observeUnread
now takes a cutoff, observeOlderUnread/observeUnreadCount/
observeOlderUnreadCount added). Undated articles are kept in the main
Unread view rather than silently dropped from both. Verified live: 150
recent / 1172 older, summing to the prior combined total.
- "Mark read from here": long-press an article row to mark it and
everything older in that same filtered list as read in one batch
(re-added ArticleDao.markReadBatch, this time as an explicit,
confirmed action -- not the implicit on-scroll version that was
reverted earlier). Verified live: confirmed a 147-article batch: the
tapped article and older were marked read, newer ones above it were not.
- BACKLOG.md: added a YouTube-embeds note (JS is off in the reader
WebView, so embeds currently show broken -- rough shape is a tap-to-play
thumbnail rather than turning JS on for third-party content).
|
|
Some "soft" paywalls (West Hawaii Today's included) deliver the full
article in the initial HTML response to every visitor, then hide it with
CSS (inline or via stylesheet) pending a client-side subscription check --
the content itself isn't withheld by the server, only its visibility.
Readability has a built-in heuristic that skips anything hidden this way,
treating it as boilerplate, which is exactly what was discarding real,
already-delivered article text and leaving only the visible teaser/
subscribe prompt.
Before parsing, walk the live document (getComputedStyle needs an attached
node) and force display/visibility back on for anything hidden via
display:none or visibility:hidden, then clone and hand that off to
Readability as before.
Verified live against a real West Hawaii Today article: extractedContent
went from 588 characters (the RSS teaser) to 3458 (the full body).
|
|
Root-caused via emulator stress test (rapid swiping through many articles):
withTimeoutOrNull's cancellation, and LaunchedEffect disposal when a page is
swiped away, both invoke invokeOnCancellation on kotlinx.coroutines.DefaultExecutor
rather than the calling coroutine's Main dispatcher. The handler called
webView.stopLoading()/destroy() directly, crashing with "A WebView method was
called on thread ..." on essentially every session once enough thin-content
articles were swiped past mid-extraction -- the reported crash loop. Now hops
back to the main looper before touching the WebView, and guards against a
double destroy() race between the JS-callback and cancellation paths.
Also fixed a real (non-crashing) leak: the visible reader WebView in
ReaderScreen was never destroy()-ed when its page left the pager, via
AndroidView's onRelease.
Reproduced pre-fix in ~80 rapid swipes on an emulator; post-fix, 320+ swipes
with real 107-feed data ran clean with stable native heap.
|
|
Feed subscriptions (RSS 2.0 + Atom, hand-rolled parser), OPML
import/export, WorkManager periodic fetch with conditional GET,
Room-backed local storage (feeds/articles/filter rules), keyword
mute/highlight filtering.
Reader mode: on-device extraction via Mozilla's Readability.js
(vendored from github.com/mozilla/readability, pinned to release
0.6.0) run in a hidden WebView, for feed articles whose provided
content is too thin to be the real article body.
Save: explicit action (from a feed article, or the Android share-sheet
target for any URL) archives to Stash via POST /artifacts?sync=true,
while a local on-device copy (Readability-extracted) is always kept so
saved articles read offline regardless of Stash's reachability.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
