diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-11-04 10:14:52 +0000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-11-04 10:14:52 +0000 |
| commit | 9678aa3e980d1e44defb3808812291968565f869 (patch) | |
| tree | 41a2c735551253228bda3c6b6af269a340ee067d /CLAUDE.md | |
| parent | ba6a03b2bd22ba29506e16caff708383caec07ac (diff) | |
Delete on-device detection. Detection is the server's
Peter's call: server first. The audio-fingerprint subsystem ran on the phone until detection moved
to the server, and then kept being shipped: FingerprintWorker and MidrollWorker went at the time
and the twelve classes they drove stayed, unreachable, in every APK since. ReachabilityTest found
it on 2026-11-03 once it was widened to classes, and it has been sitting in that test's allowlist
with its reasoning since.
Gone: IntroOutroDetector, AudioCorrelation, MidrollScanner, MidrollDetector, MidrollUpdater,
FingerprintUpdater, MediaCodecClipSource, Decimate, Fft, MonoResampler, PcmCodec, StoredClip, and
the 67 unit tests over them. 3,092 lines.
**Recoverable by name, not by archaeology.** The tag `on-device-detection` marks the last commit
that has it, pushed to the local remote along with everything else from this session, and the tag
message says how to bring it back. The work cost weeks and the approach may be wanted again.
What stayed, because none of it is detection: SkipSummary words the readout, BreakTrack and
BreakPopup draw and explain the timeline's marks, and MidrollClipStore shrank to deleteAllFor
alone -- its write and read halves had one caller each and both were the deleted updater, but a
phone that ran the old version still has clip files on disk and unsubscribing is the only thing
that clears them. Its test now writes the old layout by hand, because that layout is a
compatibility concern now rather than this class's own choice.
IntroOutroDetector.CONFIRM_THRESHOLD and MidrollDetector.CONFIRM_THRESHOLD moved to SkipSummary,
the only thing left that reads them. Both were three and the server's DefaultConfirmThreshold is
also three, checked against each other before the move.
**The database was deliberately not touched.** show_fingerprints, show_midrolls and
episode_midrolls are the server's columns now -- ServerSkips writes its answers into exactly the
columns the on-device detector used to, which is why playback needed no changes when detection
moved. Four MidrollDao methods went because they were the discovery half (upsertCandidate,
deleteCandidate, getUncheckedDownloaded, setMidrollCheckedAtMs) and the reachability check named
them; a DAO method is not schema, so there is no migration here and nothing for a real library to
go wrong over.
Two things found while doing it, neither caused by it, both written up in CLAUDE.md:
- **The per-show "Detected skips" readout can never say anything.** It reads candidate and
confirmation rows, and nothing has written one since detection moved -- the server writes
per-episode ranges and offsets instead. Verified on the emulator, not inferred: a fresh install
with a live subscription shows "Nothing found yet / Nothing found yet / None found yet" while the
server is finding and publishing breaks for that very show. It has been lying since the
migration; deleting the detector only makes it undeniable.
- **RetentionSweeper's pending-episode guard protects nothing**, for the same reason: nothing sets
a pending basis any more, so both id sets are always empty.
The APK barely moves -- 110KB of dex, 72.61MB to 72.50MB -- because it is dominated by the
embedded Python. This was never going to be a size win and is not being sold as one; it is 3,092
fewer lines to read, search and mislead.
1004 tests pass, detekt 0, lint clean. Coverage 95.35% -> 94.58%, which is the documented
COVERAGE_EXCEPTION case: deleting dead code that happened to be well covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
Diffstat (limited to 'CLAUDE.md')
| -rw-r--r-- | CLAUDE.md | 81 |
1 files changed, 42 insertions, 39 deletions
@@ -146,45 +146,40 @@ downloaded at once, Facebook videos are supported, the library ranks shows by th and opt-in skipping of breaks the publisher marked `toc: false`. Nothing is guessed from chapter titles. -**Tier 3 skip detection — audio fingerprinting — is built, in two parts.** Both were desktop- -prototyped against real episodes before any Android code, per the design plan's own instruction. - -- **Intro/outro** (`IntroOutroDetector`, `AudioCorrelation`, `MediaCodecClipSource`, - `FingerprintUpdater`): one fixed candidate per show per end, needing 3+ independently-fetched - episodes to agree before it is trusted -- the confirmation threshold exists because two episodes - fetched close together can agree on a segment that is a dynamically-inserted ad, not the show's - real intro (found prototyping against Darknet Diaries). Triggered per download via - `FingerprintWorker`. -- **Midroll ad breaks** (`MidrollScanner`, `MidrollDetector`, `MidrollUpdater`): a show can have - several distinct midroll breaks, at a different timestamp in every episode, so this scans a - whole episode body rather than a fixed head/tail window -- `MidrollScanner.findSharedSegments` - finds every segment two episodes share, at whatever position each happens to have it, and - `MidrollUpdater` composes a coarse (decimated) pass with a native-resolution verify so that scan - is fast enough to run on a phone. Prototyping this against Darknet Diaries found a real recurring - segment (not the fixed intro) independently present across episodes nine years apart. Runs - nightly while charging (`MidrollWorker`), not per-download: a full-body scan is too expensive for - that, and unlike a download it never touches the network. No per-show toggle, same as - intro/outro -- the confirmation threshold is the safety mechanism. - -**Both halves have now run on a device**, and doing so found a blocking defect the unit tests -could never have caught. `show_fingerprints.pendingClipPcm` held a whole decoded head/tail -window -- 240 seconds at 8kHz is 3.84MB as 16-bit PCM -- and Android reads every Room row -through a CursorWindow, which refuses a row over 2MB *on the way out*. The insert succeeded; -the next read threw `SQLiteBlobTooBigException`. Intro/outro detection therefore crashed the -app the moment a show's second episode was fingerprinted, and could never have worked as -shipped. Fixed in migration 9 -> 10 by storing only `pendingEpisodeId` and re-decoding on -demand -- the pattern `midrollPendingEpisodeId` already used for the same reason. - -`StoredClip` now owns that limit, `MidrollUpdater` caps a candidate segment at 60 seconds -before persisting it (a long sponsor block would otherwise hit the same wall), and -`StoredClipSizeTest` pins both against the limit so the class of bug cannot come back quietly. - -What is proven on hardware: the MediaCodec decode loop runs, `FingerprintWorker` completes -SUCCESS per download, and the re-decode path is exercised. What is **not** yet proven is that -detection actually confirms anything -- three downloaded Up First episodes left both ends still -"Nothing found yet". That may be correct (the confirmation threshold is deliberately strict) or -it may mean the 8-second reference window is too wide for a ~6-second sting; the desktop -prototype is the thing to compare against, and nobody has. +**Tier 3 skip detection — audio fingerprinting — was built on the phone, and is gone from it.** +Deleted 2026-11-04, after `ReachabilityTest` was widened to classes and found that none of it had +been reachable since detection moved to the server: `FingerprintWorker` and `MidrollWorker` went +then, and the twelve classes they drove stayed, shipped in every APK. + +What went: `IntroOutroDetector`, `AudioCorrelation`, `MidrollScanner`, `MidrollDetector`, +`MidrollUpdater`, `FingerprintUpdater`, `MediaCodecClipSource`, `Decimate`, `Fft`, `MonoResampler`, +`PcmCodec`, `StoredClip` and 67 unit tests. **The tag `on-device-detection` is the last commit +that has it** — `git checkout on-device-detection -- app/src/main/java/org/terst/listen/fingerprint/` +brings it all back, and the detection it implemented now lives in `server/internal/skips` and +`server/internal/signature`, prototyped and measured against real audio there instead. + +What stayed, and why, because none of it is detection: `SkipSummary` words the readout, +`BreakTrack` and `BreakPopup` draw and explain the marks on the episode timeline, and +`MidrollClipStore` shrank to `deleteAllFor` alone — a phone that ran the old version still has +clip files on disk, and unsubscribing is the only thing that clears them. + +**The database was deliberately not touched.** `show_fingerprints`, `show_midrolls` and +`episode_midrolls` are the server's columns now — `ServerSkips` writes its answers into exactly the +columns the on-device detector used to, which is why playback needed no changes when detection +moved. No migration, no schema change, nothing for a real library to go wrong over. + +Two things the deletion left standing that a future session should look at rather than assume: + +- **The per-show detection readout has nothing feeding it.** `ShowSettingsViewModel.loadDetection` + reads `ShowFingerprintDao.get` and `MidrollDao.getCandidates`, and nothing has written a + *candidate* row since detection moved: the server writes per-episode ranges and offsets, not the + confirmation state the readout describes. So "2 of 3 episodes agree" can never appear, and a + phone that never ran the old version shows "nothing found yet" for every show while the server + is finding and publishing breaks. Predates this deletion; it is the same migration's doing. +- **`RetentionSweeper`'s pending-episode guard protects nothing.** It reads + `getAllPendingEpisodeIds` and `getAllMidrollPendingEpisodeIds` so retention cannot reclaim a file + mid-detection. Nothing sets a pending basis any more, so both are always empty — harmless, but a + stale row on an existing phone would pin that episode's file for ever. **ID3 `CHAP` frames are dropped, not deferred** — decided 2026-09-26 by measuring Peter's actual 61 subscriptions rather than guessing. Every feed's newest episode was fetched far enough to read @@ -213,6 +208,14 @@ by default and correct; it is simply waiting for a publisher who marks breaks. - **Auto-downloads are not retried.** A rule's download that fails is left FAILED rather than requeued: retrying forever against a 404 would burn the battery, and the failed row is visible in the episode list. Retry is the manual Download button. +> **The three bullets below are history, about code that no longer exists.** They describe the +> on-device detector deleted on 2026-11-04 (see the Tier 3 entry above and the `on-device-detection` +> tag). Kept because the measurements in them were expensive and the same questions will come back +> if audio fingerprinting is ever wanted again — but nothing here names a file in this repo any +> more, so do not go looking for `DEFAULT_REFERENCE_SECONDS`, `MidrollUpdater.discover` or +> `IntroOutroDetectorScaleTest`. The surviving instruction from all of it is the one that is not +> about the app at all: tune detection against real audio off-device, never from the phone. + - **Tier 3 had never confirmed anything, and there were two independent reasons, both now fixed.** Not the correlation algorithm -- a real production-algorithm simulation against real Up First audio confirmed cleanly at the app's actual defaults before any of this. Do not tune |
