| Age | Commit message (Collapse) | Author |
|---|
|
"Remove giant Google icon", reported on a West Hawaii Today article. It is the
"Add as a preferred source on Google" badge that sits beside the byline. Giant
because the page caps it at 120px with its own stylesheet and the reader drops
the page's CSS, so a small badge on the site becomes a full-width image above
the first paragraph.
Matched on the link's address rather than the class, and that is forced rather
than preferred: running the vendored Readability.js over the real page shows
the wrapper's preferred_source_badge class does not survive extraction at all,
so a class rule would match nothing. The anchor does survive, and its address
is Google's own and identical for every publisher in the programme -- where the
badge's image filename sits under this newspaper's theme directory and is not.
So the address is both the only signal left and the one that generalises.
A link with words in it is never touched. An article may discuss the programme
and link to it in a sentence, and that sentence is the author's.
Both cleaners now run from one entry point. There are two places a body is
cleaned -- a fresh extraction on the way out, a cached one on the way back in --
and a cached article is cleaned in place rather than discarded, so a cleaner
wired into only the fresh path would leave every already-read article carrying
litter nothing would ever remove. Peter's report says the article was cached,
which is exactly that case.
Pinned against the real thing: the reported article, put through the real
extractor, loses the badge and keeps its text and byline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three reported bugs.
**"West Hawaii today articles get a bunch of stray Trending headers."**
The publisher's widget platform injects promo blocks between the real
paragraphs, inside the article's own container, so every structural
signal tells Readability they belong. What survives is a heading
labelling nothing, because the links under it are filled in by a script
that never runs in the reader. Confirmed on 9 of the 12 articles in
that feed, two each. Blocks marked `data-mrf-recirculation` are now
removed -- the attribute says what the block is for in the widget
platform's own words, which is a better signal than the word
"Trending", since an article may legitimately discuss what is trending.
Nothing longer than a label and a few headlines is ever removed. Also
applied to already-cached articles, in place rather than by discarding
them: the article was fine, only the litter has to go.
**"Can't subscribe to heathercoxrichardson.substack.com."** Adding a
feed handed whatever URL it was given straight to the XML parser, so a
site's home page -- the only URL a reader can actually see and copy --
failed on the root element. That page advertises its feed the way sites
have since 2002, in a <link rel="alternate"> in the head. It is now
read, resolved against the page, and fetched. Only what the page
declares: guessing /feed and /rss would add a burst of 404s to every
failed add and eventually subscribe somebody to the wrong thing.
**"YouTube play is misconfigured."** It was, and YouTube said so: the
player dialog loaded the embed as its own top-level document, which
answers with "Video player configuration error -- Error 153". An embed
has to be embedded. It now sits in an iframe on a minimal host page.
Measured in the app's own WebView: top-level, no requests for the video
stream and the error on screen; framed, the stream loads and there is
no error.
**"Long click on YouTube placeholder should open the URL directly."**
It does now. Worth recording why this is not simply the link's href:
for an image inside a link, WebView's hit test reports the image's URL,
so a long-press on the thumbnail arrives as the i.ytimg.com JPEG. The
video id is resolved out of that.
Verified on a device for all four: the Substack site URL adds "Letters
from an American" and fetches 20 articles; a West Hawaii Today article
extracts with no Trending headers and its body intact; the video plays;
and a long-press reaches the YouTube app.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Reported against lcamtuf's Substack ("we should render tex",
blog.coredump.cx/p/yours-truly): reader mode showed raw
"\(p = \text{...}\)" where equations should be.
What the page actually ships is the TeX SOURCE, as text, inside a
"latex-rendered" div -- Substack renders it in the browser with its own
script, which never runs here. Readability keeps that text verbatim (it
strips the class, not the text), so nothing was being discarded during
extraction and no amount of extraction-side repair would have helped:
there was simply never any rendered markup to keep.
So render it during extraction, where JavaScript is already running for
Readability's sake. KaTeX (vendored in assets, never fetched -- this
runs against untrusted third-party pages) converts each expression to
MathML before Readability sees the document, and the reader WebView
renders that natively. The reader keeps JavaScript OFF, as it must, and
needs no stylesheet, webfont or script of KaTeX's; the only reader-side
change is containing a wide equation so it scrolls within its own box
instead of dragging the whole article sideways.
Two details worth knowing:
- KaTeX refuses {align} outside display mode, and Substack wraps display
maths in INLINE delimiters (a block-styled span does the centring), so
a mode that renders nothing is retried in the other one. An expression
neither mode can render is left as source -- no worse than before.
- The reported article came with "extractedContent cached: true", so the
raw source was already on disk and every open read it back. Recognised
and dropped on read (same mechanism as the cached consent notice),
which re-extracts the article the next time it is opened.
APK grows by ~90KB: the vendored katex.min.js (267KB) compresses to 91KB
in the APK. No new permissions, no schema change, no new JS-enabled
surface.
Verified on the emulator against the live article: stale raw-TeX cache
discarded, article re-extracted, all four equations rendered, and a wide
align block scrolls inside itself.
|
|
Reported live: reader mode on a New Republic feature showed nothing but
the cookie disclaimer, and had cached it.
Reproduced outside the app by running ReaderExtractor's own injected
script (Readability.js and all) against the real page. With the article
body present, extraction is perfect. With the body absent from the DOM at
the moment the script runs, Readability's first pass -- which discards
"unlikely candidates", and the Didomi banner's class is
didomi-notice-banner -- comes back under its own 500-character threshold,
so it retries with that stripping switched off. The banner's single
paragraph is 510 characters, i.e. just over the threshold, so it comes
back as the article with the right title attached.
Nothing on the Kotlin side looked at that. Any non-blank content counted
as ExtractionResult.Success, was written to articles.extractedContent,
and was never reconsidered -- so one bad load became a permanent cookie
disclaimer, which is the "extractedContent cached: true" in the report.
ConsentNotice recognises such a result: a hard plain-text length cap
first (nothing long is ever rejected, however much consent boilerplate it
quotes), then either a known CMP container id -- Readability strips
classes but keeps ids -- or two or more distinct consent phrases.
The accept/retry/give-up decision moves out of the WebView callback into
decideEvalOutcome so it can be tested without a device. A consent notice
is now "not ready yet", which routes it into the retry the extractor
already does for an empty result; if it survives all six attempts that is
what the page serves, and it fails as ExtractionFailure.ConsentInterstitial
rather than being accepted.
A fixed extractor cannot repair what is already on disk, so
loadCachedExtraction drops a cached notice on read and re-extracts, in
both ReaderScreen and LinkReaderScreen. No schema change: the repair
happens when the article is next opened. Reader mode is still never
turned on heuristically -- replacing a discarded notice does not flip it.
The bug report now also carries the last extraction error, without which
a report about bad reader content cannot say whether extraction failed.
Tests: 59 -> 78.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Bug report: extraction failed on one 1500days.com article while others worked.
It is not that article. Extraction was racing the page load, and the report
could say nothing more because the app discarded the reason.
Two separate defects.
1. The race. onPageFinished fires before the page has finished loading —
observed at progress=25 on this article — and the extraction script was
injected there. Sometimes the DOM is far enough along; sometimes
evaluateJavascript hands back a bare "null" because the document is still
being replaced. Measured against the reported URL: 4 successes in 6
attempts before, 10 in 10 after.
An unusable result is no longer final. A null or empty-content result is
retried (6 attempts, 500ms apart, well inside the existing 25s timeout); a
Readability error still fails immediately, since retrying will not change
what the parser thinks of the markup.
2. The silence. ReaderScreen did `extracted?.content?.let { }`, so every
failure — dead host, HTTP error, timeout, parser throw — arrived as the same
null and was dropped. The reader just kept showing raw feed content. The
only way a failure ever surfaced was the user noticing.
extractResult() now returns a typed ExtractionFailure carrying the reason;
ReaderScreen logs the diagnostic and shows the user a short explanation.
Also replaced a dead error callback. The only override was the four-argument
onReceivedError, which Android stopped calling at API 23; minSdk is 26, so it
never fired on any supported device. A failed load therefore invoked nothing
and hung for the full 25 seconds before surfacing as an indistinguishable null.
Now overrides the request/error and onReceivedHttpError forms Android actually
calls, both filtered to the main frame so a 404'd tracker cannot abort an
article that renders fine.
The instrumented test does NOT reproduce the race and says so in its own
docstring: a MockWebServer page finishes loading before onPageFinished fires,
and slow subresources do not help because WebView waits for them. It is kept as
a reliability guard over repeated back-to-back extractions. The race was
verified by hand against the live URL.
47 unit tests, 1 instrumented test, both green.
|
