|
Bug report: extraction failed on one 1500days.com article while others worked.
It is not that article. Extraction was racing the page load, and the report
could say nothing more because the app discarded the reason.
Two separate defects.
1. The race. onPageFinished fires before the page has finished loading —
observed at progress=25 on this article — and the extraction script was
injected there. Sometimes the DOM is far enough along; sometimes
evaluateJavascript hands back a bare "null" because the document is still
being replaced. Measured against the reported URL: 4 successes in 6
attempts before, 10 in 10 after.
An unusable result is no longer final. A null or empty-content result is
retried (6 attempts, 500ms apart, well inside the existing 25s timeout); a
Readability error still fails immediately, since retrying will not change
what the parser thinks of the markup.
2. The silence. ReaderScreen did `extracted?.content?.let { }`, so every
failure — dead host, HTTP error, timeout, parser throw — arrived as the same
null and was dropped. The reader just kept showing raw feed content. The
only way a failure ever surfaced was the user noticing.
extractResult() now returns a typed ExtractionFailure carrying the reason;
ReaderScreen logs the diagnostic and shows the user a short explanation.
Also replaced a dead error callback. The only override was the four-argument
onReceivedError, which Android stopped calling at API 23; minSdk is 26, so it
never fired on any supported device. A failed load therefore invoked nothing
and hung for the full 25 seconds before surfacing as an indistinguishable null.
Now overrides the request/error and onReceivedHttpError forms Android actually
calls, both filtered to the main frame so a 404'd tracker cannot abort an
article that renders fine.
The instrumented test does NOT reproduce the race and says so in its own
docstring: a MockWebServer page finishes loading before onPageFinished fires,
and slow subresources do not help because WebView waits for them. It is kept as
a reliability guard over repeated back-to-back extractions. The race was
verified by hand against the live URL.
47 unit tests, 1 instrumented test, both green.
|
|
BACKLOG's "Extraction test coverage for WHT / Ars Technica" item:
ReaderExtractor's site-specific preprocessing had only ever been
diagnosed by hand against single live articles across several rounds
of fixes. Four hand-authored fixtures now pin down the exact behavior:
- West Hawaii Today's paywall-div unhide (class="paywall paywall_*")
- Lazy-loaded image src resolution (data-src -> src)
- Ars Technica's duplicate PhotoSwipe caption removal
(.pswp-caption-content vs. the real <figcaption>)
- Ars's split .post-content.post-content-double block merge
Needs a real WebView to run Readability.js, so these are instrumented
tests (app/src/androidTest/), not plain JVM unit tests -- that source
set didn't exist yet (testInstrumentationRunner + androidx.test deps
added to app/build.gradle.kts). Fixtures are minimal hand-authored
markup targeting the exact selectors ReaderExtractor's own code uses,
not scraped live pages, which rot and would fail for reasons unrelated
to the extraction logic.
Fixtures are served over real http:// via MockWebServer, since
ReaderExtractor's WebView has allowFileAccess = false and won't load
file:// URLs. Added a debug-only network security config override
(app/src/debug/res/xml/network_security_config.xml) permitting
cleartext to 127.0.0.1/localhost for this -- release is untouched.
Verified the tests actually catch regressions, not just pass
trivially: temporarily broke the paywall-unhide selector, confirmed
the corresponding test failed on both connected emulators, then
reverted and confirmed green again.
Ran on 2 connected devices (feedreader_test, doot_test_api36): 4/4
passing, 0 failures. Instrumented tests aren't counted by the JVM-only
jacoco coverage metric (expected, not a gate miss -- see CLAUDE.md).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017GMEkHeqKz6FLkmizowBTK
|
