summaryrefslogtreecommitdiff
path: root/CLAUDE.md
diff options
context:
space:
mode:
authorPeter Stone <thepeterstone@gmail.com>2026-10-05 08:57:59 +0000
committerPeter Stone <thepeterstone@gmail.com>2026-10-05 08:57:59 +0000
commit7cc88394458e9d03e3a6a6e58cdff9c8dd4ad7a3 (patch)
tree7e9980f718b10c49895d1df673001e058f2af117 /CLAUDE.md
parent4fb118b2db32f269c5aeb217644dc5e80640c885 (diff)
Find why Tier 3 never confirmed, and port detection to Go
Peter's report: the phone runs hot, uses a lot of battery, playback takes a moment to start, and intelligent skipping has never done anything. **The hot phone and the dead detection were one bug.** The candidate search correlated 117 window positions against the whole of the other clip, re-transforming that unchanged clip every time -- 199 seconds per episode end, measured on a desktop, so minutes more on a phone. FingerprintWorker is a plain WorkManager worker, Android stops one at ten minutes, and nothing was persisted until a comparison returned. So it was killed mid-flight, saved nothing, and was re-enqueued to start over, forever. IntroOutroDetectorScaleTest is the tripwire whose absence allowed this: the unit tests ran on 300-sample clips while production ran on 1,920,000. Fixed coarse-to-fine, plus AudioCorrelation.prepareTarget so the target clip is transformed once per search rather than 117 times. 10.6x, and a test asserts the chosen reference window is byte-identical to what a full-resolution search picks. **Midroll detection could never have run at all.** decodeFull buffered a whole episode at its native rate -- 434MB for a 13-minute episode, no largeHeap -- to produce 6MB of 8kHz mono. It threw OutOfMemoryError into a swallowing runCatching and the episode was then marked checked forever. MonoResampler now streams the decode so the peak allocation is the output, and maxDecodableMs skips a longer episode before spending anything and without marking it checked. **Detection is moving to a server**, Peter's call, in Go to match podkit and doot. server/ holds the ported core, verified three ways because this is the second port of this maths and the first shipped a normalization bug: numpy's literals extracted mechanically from AudioCorrelationTest, the FFT against a direct DFT, and the whole correlation path against a brute-force Pearson. On four real Radiolab episodes it confirms the intro on 4 of 4, at 48.0s, 8.65s, 8.65s and 65.15s -- the varying pre-roll, observed in working code at last. skips.SelfCheck verifies located segments against *each other* rather than against the reference, which is a different claim: correlation is not transitive, so two segments can each clear 0.6 against a reference and be unrelated. It rejected the outro on its first real run (r=0.746), withholding offsets that would have ended three episodes early. Offsets are published only when they verify. Also: the episode screen gets a working scrubber with ad-break marks and a declared-versus-measured length readout, queue rows open the episode instead of doing nothing, PrepareTrace logs playback-start timing against episode length for the still-open pause question, and -PphoneOnly drops the emulator's CPython from a sideload build (114MB -> 99MB). Full history and numbers in docs/detection-findings.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
Diffstat (limited to 'CLAUDE.md')
-rw-r--r--CLAUDE.md80
1 files changed, 68 insertions, 12 deletions
diff --git a/CLAUDE.md b/CLAUDE.md
index 8121bfe..5a7d1ae 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -194,21 +194,77 @@ by default and correct; it is simply waiting for a publisher who marks breaks.
- **Auto-downloads are not retried.** A rule's download that fails is left FAILED rather than
requeued: retrying forever against a 404 would burn the battery, and the failed row is visible
in the episode list. Retry is the manual Download button.
-- **Tier 3 had never confirmed a real intro, outro or midroll, and root cause is now found and
+- **Tier 3 had never confirmed anything, and there were two independent reasons, both now
fixed.** Not the correlation algorithm -- a real production-algorithm simulation against real
- Up First audio confirmed cleanly at the app's actual defaults before any of this. The real
- cause: `RetentionPolicy`'s one-day grace period routinely reclaimed a show's pending episode's
- file before the next download arrived to compare it against, for a fast, promptly-listened-to
- show like Up First -- so the detector kept discarding its only candidate and restarting instead
- of ever comparing two episodes. Confirmed live on the phone (a fresh download's
- `FingerprintWorker` run replaced the stale pending episode instead of advancing toward a
- comparison). Fixed by protecting a show's pending episode from reclaim, for both intro/outro
- and midroll -- `RetentionSweeper` now takes a `pendingEpisodeIds` seam and
- `RetentionWorker` supplies the union of both mechanisms' pending ids. **Not yet re-verified on
- a device.** Full history in `docs/detection-findings.md`. Do not tune
+ Up First audio confirmed cleanly at the app's actual defaults before any of this. Do not tune
`DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE` or `CONFIRM_THRESHOLD` from the app -- they were never
the problem, the app cannot show you what a change costs in false positives, and the threshold
- is the whole safety mechanism.
+ is the whole safety mechanism. Full history, with numbers, in `docs/detection-findings.md`.
+
+ **Reason one, retention (found 2026-09-29).** `RetentionPolicy`'s one-day grace period
+ routinely reclaimed a show's pending episode's file before the next download arrived to compare
+ it against, for a fast, promptly-listened-to show like Up First -- so the detector kept
+ discarding its only candidate and restarting. Confirmed live on the phone. Fixed by protecting
+ a show's pending episode from reclaim, for both mechanisms: `RetentionSweeper` takes a
+ `pendingEpisodeIds` seam and `RetentionWorker` supplies the union.
+
+ **Reason two, cost (found 2026-10-03, and on the evidence the dominant one).** The candidate
+ search correlated 117 window positions against the whole of the other clip, re-transforming
+ that unchanged clip every time: **199 seconds per end, measured on a desktop**, so minutes more
+ on a phone. `FingerprintWorker` is a plain WorkManager worker, Android stops one at ~ten
+ minutes, and nothing is persisted until a comparison returns -- so it was killed mid-flight,
+ saved nothing, and was re-enqueued to start over. That is why Peter's phone ran hot *and* why
+ no show ever got past its first processed episode; one cause, both symptoms. Fixed by
+ coarse-to-fine search plus `AudioCorrelation.prepareTarget`: **10.6x faster, and
+ `IntroOutroDetectorScaleTest` asserts the chosen reference window is byte-identical to what a
+ full-resolution search picks**, so none of the speed came out of detection quality.
+
+ **Neither fix is yet verified on a device.**
+
+- **`IntroOutroDetectorScaleTest` is a cost tripwire, not a benchmark -- do not delete it for
+ being slow.** The unit tests prove the state machine on 300-sample clips; production hands the
+ detector 1,920,000, and nothing measured that for two phases of work. It asserts a 60-second
+ budget at the real sizes with the real constants. If it starts failing, the search got
+ expensive again, and the consequence is a worker that burns ten minutes and persists nothing.
+
+- **Midroll detection could never have run on an episode of normal length**, found the same day
+ and independent of everything above. `decodeFull` buffered the whole episode at its native rate
+ (a 13-minute stereo episode: 145MB of PCM plus 289MB of mono frames live at once, no
+ `largeHeap`) to produce 6MB of 8kHz output. It threw `OutOfMemoryError`, `runCatching`
+ swallowed it, and `MidrollUpdater` then marked the episode **checked forever** -- so nothing
+ was ever scanned, nothing would be retried, and the readout said "None found" rather than
+ anything true. `MonoResampler` now consumes the decoder buffer by buffer and keeps only the
+ output, and `Tuning.maxDecodableMs` (20 minutes) skips a longer episode before spending
+ anything and without marking it checked.
+
+ **Still outstanding:** episodes over that ceiling are not scanned at all. Lifting it means
+ decoding candidate ranges on demand rather than the whole body -- `MidrollUpdater.discover`,
+ `extractSegment` and `MidrollDetector.check` all currently want the entire native-resolution
+ array. Left deliberately undone rather than half-done.
+
+- **`PrepareTrace` logs one line per playback start**, tagged `ListenPlayback`, carrying the
+ episode's real length against the elapsed time to READY and to PLAYING:
+ `prepare ep=abc len=1774s src=local ready=812ms playing=903ms`. It exists for an open question
+ of Peter's -- a pause before playback that looks proportional to episode length -- which cannot
+ be answered from the code: nothing in the play path scales with length, so it is either Media3's
+ own preparation or CPU starvation, and those are indistinguishable from outside. The clock
+ starts in `onMediaItemTransition`, the only hook both play paths reach (a tap in the UI goes
+ through a MediaController and never calls the service's `play`).
+
+ **Measured so far: ~440ms to READY for a 27-minute downloaded episode on the emulator**, with
+ nothing else loading the device -- so the cost is real and not purely CPU starvation. Whether
+ it scales with length is still open and needs two episodes of very different lengths; the log
+ line carries both numbers, so ordinary use on the phone answers it.
+
+- **Do not re-test the missing MP3 seek table.** These files carry no Xing, Info or VBRI header,
+ so Media3 has no seek table to read, and the obvious theory was that `Mp3Extractor` scans the
+ file to build one -- length-proportional, matching the report exactly. **Disproven by
+ measurement:** enabling `Mp3Extractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING` moved the mean not
+ at all (404/531/386ms against 469/450/404ms). A first single sample of each had looked like a
+ 38% win and was cold-start noise -- the spread here is wider than any effect, so one sample
+ proves nothing. It would have been the wrong trade regardless: constant-bitrate seeking makes
+ seeks approximate on a VBR file, and the playback position is the crown jewel. Full numbers in
+ `docs/detection-findings.md`.
- **Android Auto is verified in a real car**, as of 2026-09-28. It appears, and connects. The
notification Peter saw on first connect was Android Auto's own new-media-app announcement,