diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-10-05 08:57:59 +0000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-10-05 08:57:59 +0000 |
| commit | 7cc88394458e9d03e3a6a6e58cdff9c8dd4ad7a3 (patch) | |
| tree | 7e9980f718b10c49895d1df673001e058f2af117 /CLAUDE.md | |
| parent | 4fb118b2db32f269c5aeb217644dc5e80640c885 (diff) | |
Find why Tier 3 never confirmed, and port detection to Go
Peter's report: the phone runs hot, uses a lot of battery, playback takes a
moment to start, and intelligent skipping has never done anything.
**The hot phone and the dead detection were one bug.** The candidate search
correlated 117 window positions against the whole of the other clip,
re-transforming that unchanged clip every time -- 199 seconds per episode end,
measured on a desktop, so minutes more on a phone. FingerprintWorker is a plain
WorkManager worker, Android stops one at ten minutes, and nothing was persisted
until a comparison returned. So it was killed mid-flight, saved nothing, and was
re-enqueued to start over, forever. IntroOutroDetectorScaleTest is the tripwire
whose absence allowed this: the unit tests ran on 300-sample clips while
production ran on 1,920,000.
Fixed coarse-to-fine, plus AudioCorrelation.prepareTarget so the target clip is
transformed once per search rather than 117 times. 10.6x, and a test asserts the
chosen reference window is byte-identical to what a full-resolution search picks.
**Midroll detection could never have run at all.** decodeFull buffered a whole
episode at its native rate -- 434MB for a 13-minute episode, no largeHeap -- to
produce 6MB of 8kHz mono. It threw OutOfMemoryError into a swallowing
runCatching and the episode was then marked checked forever. MonoResampler now
streams the decode so the peak allocation is the output, and maxDecodableMs skips
a longer episode before spending anything and without marking it checked.
**Detection is moving to a server**, Peter's call, in Go to match podkit and
doot. server/ holds the ported core, verified three ways because this is the
second port of this maths and the first shipped a normalization bug: numpy's
literals extracted mechanically from AudioCorrelationTest, the FFT against a
direct DFT, and the whole correlation path against a brute-force Pearson.
On four real Radiolab episodes it confirms the intro on 4 of 4, at 48.0s, 8.65s,
8.65s and 65.15s -- the varying pre-roll, observed in working code at last.
skips.SelfCheck verifies located segments against *each other* rather than
against the reference, which is a different claim: correlation is not transitive,
so two segments can each clear 0.6 against a reference and be unrelated. It
rejected the outro on its first real run (r=0.746), withholding offsets that
would have ended three episodes early. Offsets are published only when they
verify.
Also: the episode screen gets a working scrubber with ad-break marks and a
declared-versus-measured length readout, queue rows open the episode instead of
doing nothing, PrepareTrace logs playback-start timing against episode length
for the still-open pause question, and -PphoneOnly drops the emulator's CPython
from a sideload build (114MB -> 99MB).
Full history and numbers in docs/detection-findings.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
Diffstat (limited to 'CLAUDE.md')
| -rw-r--r-- | CLAUDE.md | 80 |
1 files changed, 68 insertions, 12 deletions
@@ -194,21 +194,77 @@ by default and correct; it is simply waiting for a publisher who marks breaks. - **Auto-downloads are not retried.** A rule's download that fails is left FAILED rather than requeued: retrying forever against a 404 would burn the battery, and the failed row is visible in the episode list. Retry is the manual Download button. -- **Tier 3 had never confirmed a real intro, outro or midroll, and root cause is now found and +- **Tier 3 had never confirmed anything, and there were two independent reasons, both now fixed.** Not the correlation algorithm -- a real production-algorithm simulation against real - Up First audio confirmed cleanly at the app's actual defaults before any of this. The real - cause: `RetentionPolicy`'s one-day grace period routinely reclaimed a show's pending episode's - file before the next download arrived to compare it against, for a fast, promptly-listened-to - show like Up First -- so the detector kept discarding its only candidate and restarting instead - of ever comparing two episodes. Confirmed live on the phone (a fresh download's - `FingerprintWorker` run replaced the stale pending episode instead of advancing toward a - comparison). Fixed by protecting a show's pending episode from reclaim, for both intro/outro - and midroll -- `RetentionSweeper` now takes a `pendingEpisodeIds` seam and - `RetentionWorker` supplies the union of both mechanisms' pending ids. **Not yet re-verified on - a device.** Full history in `docs/detection-findings.md`. Do not tune + Up First audio confirmed cleanly at the app's actual defaults before any of this. Do not tune `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE` or `CONFIRM_THRESHOLD` from the app -- they were never the problem, the app cannot show you what a change costs in false positives, and the threshold - is the whole safety mechanism. + is the whole safety mechanism. Full history, with numbers, in `docs/detection-findings.md`. + + **Reason one, retention (found 2026-09-29).** `RetentionPolicy`'s one-day grace period + routinely reclaimed a show's pending episode's file before the next download arrived to compare + it against, for a fast, promptly-listened-to show like Up First -- so the detector kept + discarding its only candidate and restarting. Confirmed live on the phone. Fixed by protecting + a show's pending episode from reclaim, for both mechanisms: `RetentionSweeper` takes a + `pendingEpisodeIds` seam and `RetentionWorker` supplies the union. + + **Reason two, cost (found 2026-10-03, and on the evidence the dominant one).** The candidate + search correlated 117 window positions against the whole of the other clip, re-transforming + that unchanged clip every time: **199 seconds per end, measured on a desktop**, so minutes more + on a phone. `FingerprintWorker` is a plain WorkManager worker, Android stops one at ~ten + minutes, and nothing is persisted until a comparison returns -- so it was killed mid-flight, + saved nothing, and was re-enqueued to start over. That is why Peter's phone ran hot *and* why + no show ever got past its first processed episode; one cause, both symptoms. Fixed by + coarse-to-fine search plus `AudioCorrelation.prepareTarget`: **10.6x faster, and + `IntroOutroDetectorScaleTest` asserts the chosen reference window is byte-identical to what a + full-resolution search picks**, so none of the speed came out of detection quality. + + **Neither fix is yet verified on a device.** + +- **`IntroOutroDetectorScaleTest` is a cost tripwire, not a benchmark -- do not delete it for + being slow.** The unit tests prove the state machine on 300-sample clips; production hands the + detector 1,920,000, and nothing measured that for two phases of work. It asserts a 60-second + budget at the real sizes with the real constants. If it starts failing, the search got + expensive again, and the consequence is a worker that burns ten minutes and persists nothing. + +- **Midroll detection could never have run on an episode of normal length**, found the same day + and independent of everything above. `decodeFull` buffered the whole episode at its native rate + (a 13-minute stereo episode: 145MB of PCM plus 289MB of mono frames live at once, no + `largeHeap`) to produce 6MB of 8kHz output. It threw `OutOfMemoryError`, `runCatching` + swallowed it, and `MidrollUpdater` then marked the episode **checked forever** -- so nothing + was ever scanned, nothing would be retried, and the readout said "None found" rather than + anything true. `MonoResampler` now consumes the decoder buffer by buffer and keeps only the + output, and `Tuning.maxDecodableMs` (20 minutes) skips a longer episode before spending + anything and without marking it checked. + + **Still outstanding:** episodes over that ceiling are not scanned at all. Lifting it means + decoding candidate ranges on demand rather than the whole body -- `MidrollUpdater.discover`, + `extractSegment` and `MidrollDetector.check` all currently want the entire native-resolution + array. Left deliberately undone rather than half-done. + +- **`PrepareTrace` logs one line per playback start**, tagged `ListenPlayback`, carrying the + episode's real length against the elapsed time to READY and to PLAYING: + `prepare ep=abc len=1774s src=local ready=812ms playing=903ms`. It exists for an open question + of Peter's -- a pause before playback that looks proportional to episode length -- which cannot + be answered from the code: nothing in the play path scales with length, so it is either Media3's + own preparation or CPU starvation, and those are indistinguishable from outside. The clock + starts in `onMediaItemTransition`, the only hook both play paths reach (a tap in the UI goes + through a MediaController and never calls the service's `play`). + + **Measured so far: ~440ms to READY for a 27-minute downloaded episode on the emulator**, with + nothing else loading the device -- so the cost is real and not purely CPU starvation. Whether + it scales with length is still open and needs two episodes of very different lengths; the log + line carries both numbers, so ordinary use on the phone answers it. + +- **Do not re-test the missing MP3 seek table.** These files carry no Xing, Info or VBRI header, + so Media3 has no seek table to read, and the obvious theory was that `Mp3Extractor` scans the + file to build one -- length-proportional, matching the report exactly. **Disproven by + measurement:** enabling `Mp3Extractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING` moved the mean not + at all (404/531/386ms against 469/450/404ms). A first single sample of each had looked like a + 38% win and was cold-start noise -- the spread here is wider than any effect, so one sample + proves nothing. It would have been the wrong trade regardless: constant-bitrate seeking makes + seeks approximate on a VBR file, and the playback position is the crown jewel. Full numbers in + `docs/detection-findings.md`. - **Android Auto is verified in a real car**, as of 2026-09-28. It appears, and connects. The notification Peter saw on first connect was Android Auto's own new-media-app announcement, |
