# Tier 3 detection: what is known, and the root cause of why nothing confirmed Written 2026-09-29, in four passes across the same day as the picture sharpened and then got fixed. The correlation algorithm and its tuned constants were never the problem. Up First confirmed nothing because `RetentionPolicy`'s one-day grace period usually reclaimed a played episode's file before the *next* one was downloaded and processed, for a show this fast and this promptly listened to -- so the detector kept discarding its only pending candidate and restarting, rather than ever getting two episodes' files to compare at once. See "The live test, and the actual root cause" below for the mechanism, and "The fix (made)" for what actually shipped -- protecting a show's pending episode from reclaim, for both fingerprinting mechanisms. **Not yet re-verified on a device.** The rest of this document is kept in the order it was actually found, including the two hypotheses that turned out to be wrong -- both cost real time to check and are worth not re-checking blind next time. **First pass:** the algorithm was suspected. It isn't -- see "What the desktop session actually found." The production algorithm, run against real Up First audio at its real defaults, confirms cleanly on every pair tested, both ends. **Second pass, the phone reconnected the same day:** pulled the live database and WorkManager's own execution history straight off it (read-only, via `run-as`). The pipeline is not broken -- **Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) that day.** Up First specifically was stuck: its `show_fingerprints` row hadn't updated since its *first* episode was ever processed, even though a second real episode was downloaded, played and decoded successfully (by a different worker) the next day. `FingerprintUpdater` upserts on every outcome, match or not, so that frozen timestamp meant the second episode's comparison never ran at all -- not a score that failed. See "What the phone actually showed" for the detail. **Third pass, a live test on the phone:** triggered a fresh download and watched it happen in real time. That's what found the actual mechanism -- see "The live test, and the actual root cause." **Fourth pass, the fix:** Peter picked exempting a pending episode from reclaim over the other two options. Shipped for both intro/outro and midroll -- see "The fix (made)." --- ## What is settled **The pipeline runs on hardware.** `MediaCodecClipSource` decodes, `FingerprintWorker` completes SUCCESS per download, the re-decode path for the pending episode is exercised. This was never true before 2026-09-28 and two real defects were hiding behind it (below). **The confirmation threshold earns its place.** Three independently-fetched episodes must agree before a candidate is trusted. The original prototype found why: two Darknet Diaries episodes fetched minutes apart agreed on a segment that turned out to be a rotating inserted ad, not the show's intro. Everything below makes that trap *more* likely, not less. Do not lower it to make something confirm. **Storage limits are real and were learned the hard way.** Android reads every Room row through a CursorWindow that refuses anything over 2MB *on the way out* — the insert succeeds and the next read throws. A 240-second clip at 8kHz 16-bit is 3.84MB. Intro/outro pending clips are no longer stored at all (the episode id is, and the audio is re-decoded); midroll candidate clips live in files. `StoredClip` owns the limit and `StoredClipSizeTest` pins it. --- ## The finding that changes the problem **Dynamic ad insertion is not a detail; it is the environment.** Radiolab declares Grass Apes at `00:22:05`. Measured on two separate fetches of the same episode: **29:34 once, 27:39 another time.** The ads are stitched in per request, so: - the served file's length is not a property of the episode, - the ad *content* differs between two downloads of the same episode, - and anything at a fixed offset in one file is at a different offset in the next. `insertedAdMs` (declared vs measured) now exposes this per episode in the app. It is free, needs no audio analysis, and is the obvious cross-check on the detector: **whatever the midroll scan finds should add up to roughly that much.** If it finds 30 seconds where the gap says seven minutes, the scan is missing breaks; if it finds more, it is matching content. This reframes the intro/outro question. The head window is 240 seconds, which was chosen to comfortably contain an intro. With a variable-length pre-roll in front of it, a show's real intro sits at a *different offset in every file* — which the scan does handle, since it searches the window rather than a fixed position — but the window now has to contain pre-roll **plus** intro, and the pre-roll is the thing most likely to produce a false match between two close-together fetches. --- ## What the desktop session actually found Run against the four Up First episodes already cached from the original prototype (`npr/ep1..4_head.wav`, `npr/ep1..4_tail.wav`, 240s at 8kHz each), reusing the same `normalized_xcorr`/`best_match` primitive as `AudioCorrelation.kt` -- unmodified, so this is a direct check of what the app itself computes, not an approximation of it. **The leading hypothesis is disproven.** Simulating `IntroOutroDetector.findCandidate` exactly -- an 8-second window, quarter-window step, global search in the other clip -- at the app's real `DEFAULT_REFERENCE_SECONDS = 8` and `MIN_SCORE = 0.6`: | pair | intro r | intro position | outro r | outro position | | --- | --- | --- | --- | --- | | ep1→ep2 | 0.821 | a=44.0s b=44.0s | 0.999 | a=184.0s b=166.9s | | ep1→ep3 | 0.807 | a=44.0s b=46.7s | 0.994 | a=226.0s b=184.0s | | ep1→ep4 | 0.998 | a=90.0s b=93.7s | 0.995 | a=184.0s b=166.4s | | ep2→ep3 | 0.805 | a=44.0s b=46.7s | 0.997 | a=180.0s b=184.3s | | ep2→ep4 | 0.864 | a=44.0s b=49.3s | 0.997 | a=168.0s b=167.4s | | ep3→ep4 | 0.793 | a=46.0s b=48.7s | 0.998 | a=172.0s b=167.2s | Every pair clears 0.6 at both ends, several by a wide margin. The 8-second window is not too wide for the sting -- it finds it cleanly regardless of exactly how much of the window is "extra" content around it. **Pre-roll varies a lot, and the algorithm doesn't care.** The intro lands anywhere from 44s to 94s into the head window across these four episodes -- up to 50 seconds of variability -- and the global search finds it at every position tried. This is the "competing hypothesis" from the first draft of this doc, and it is real (pre-roll length does vary) but it does not defeat detection, because `findCandidate` was already built to search the whole window rather than assume a fixed position. **The resampler is not it either.** `MediaCodecClipSource.resample` is naive linear interpolation, not a proper anti-aliasing filter, and the worry was that downsampling 44.1kHz mp3 to 8kHz that way could be degrading correlation enough to matter. Decoded the same episode's native 44.1kHz stereo, downmixed and resampled with the exact linear-interpolation formula from `MediaCodecClipSource.kt`, and compared: **0.9991 correlation against a proper (ffmpeg/swresample) decode of the same audio**, and re-running the full production simulation with the naively resampled clip still confirms at r=0.821 -- indistinguishable from the ffmpeg path. Ruled out. **A same-episode re-fetch, three days apart, is byte-identical for ~50 seconds and then diverges completely.** The exact same episode URL, fetched fresh today against a copy saved 2026-09-26: zero-lag correlation is 1.000 for every 5-second block from 0s to 45s, then falls to ~0 (pure noise) for every block from 50s to 240s. Whatever comes after the intro is not stable across fetches -- consistent with, and sharper evidence for, the "dynamic ad insertion is the environment" finding above -- but the intro itself sits inside the stable prefix, so this does not threaten intro detection specifically. It is a fresh reason to be careful trusting anything found much later in a head/tail window, and matches why outro detection uses the *tail*, not a position computed from the (unreliable) declared duration. **The Darknet Diaries trap re-checked (experiment 6) without a new fetch.** The midroll prototype work already re-fetched this exact pair: a ~51-second segment at position 0 matches at r≈1.000 across `dd_179` and `dd_178` (2026), `dd_177` (2026) and `dd_008` from **2017** -- nine years apart, fetched today. A truly rotating, per-request DAI ad reproducing identically against a specific nine-year-old archived fetch, by chance, is not plausible. This still looks like Darknet Diaries' real, stable intro, not the trap the threshold exists to catch -- which doesn't make the threshold wrong (a genuinely rotating ad would still be caught the same way), just confirms this particular historical match was likely a false alarm rather than evidence of one. --- ## What the phone actually showed (2026-09-29, later) The phone reconnected mid-session. Pulled `listen.db` (via `run-as` + `adb exec-out cat`, no device modification) and `no_backup/androidx.work.workdb` -- WorkManager's own execution history, which nothing in this app writes to but which turned out to answer the question the app state alone couldn't. **The pipeline is not broken.** `show_fingerprints` for **Cruising the Waves Podcast** shows `kind=INTRO, confirmedCount=2, referenceClipPcm` populated -- a real Tentative candidate, found today, from an ordinary download. Something *does* reach Tentative on this exact build, on this exact phone. The mechanism works in general; Up First specifically is the anomaly, not Tier 3. **Up First's `show_fingerprints` row is frozen at its very first episode.** Both INTRO and OUTRO show `confirmedCount=0`, `referenceClipPcm=NULL`, `pendingEpisodeId` pointing at one single episode ("Powerful storms hit the US..."), and `updatedAtMs = 2026-09-27T02:19` -- the moment that first episode was processed. A second real episode ("Can clergy candidates help Democrats win in red states?") was downloaded, played and swept by retention the *next day*, and its `midrollCheckedAtMs` is set, proving `MidrollWorker`'s `decodeFull` read that file successfully that day. But `show_fingerprints.updatedAtMs` never moved again. That is the specific, damning fact: `FingerprintUpdater.processEnd` calls `fingerprints.upsert(...)` on **every** outcome -- a confirmed match, a failed match, all of it -- so if a comparison involving the second episode had run at all, this row would show a later timestamp regardless of whether it found anything. It doesn't. **The second episode's `FingerprintWorker` run either was never enqueued, or `decodeHead`/`decodeTail` both returned null for it** -- and the latter looks unlikely given `decodeFull` (the same underlying decode, just a different window) succeeded on the identical file later that day. `EpisodeDownloader.downloadOverHttp` was read end to end look for a race between "downloaded" and "playable" -- there isn't one, it's a fully blocking download-to-`.part`-then-atomic-rename before `onDownloaded` fires, so a half-written file is not the likely explanation either. WorkManager prunes completed `WorkSpec` rows after roughly a week, so the actual historical `FingerprintWorker` run (if any) for that second episode is already gone -- only today's runs and the next periodic `MidrollWorker` firing were still in the table. This is now a live-observation question, not an archaeology one. ## The live test, and the actual root cause Triggered a fresh Up First download on the phone and watched `androidx.work.workdb` and `show_fingerprints` immediately after, live: - A new `FingerprintWorker` `WorkSpec` row appeared within seconds and reached `SUCCEEDED`. - `show_fingerprints` for Up First **did** update -- `updatedAtMs` moved to right now, for the first time since 2026-09-27. But `pendingEpisodeId` didn't advance to a *second* episode to compare against; it just replaced the old one with this new download. `confirmedCount` stayed 0, `referenceClipPcm` stayed null. That is `checkAgainstPending`'s own designed behaviour when `pendingClipOf` returns null -- which it does whenever the *stored* pending episode's file no longer exists to re-decode. And it doesn't: the previous pending episode ("Powerful storms...") was played on 2026-09-27, and `RetentionPolicy`'s grace period is **one day** (`gracePeriodMs = TimeUnit.DAYS.toMillis(1)`, `Repositories.kt`). By the time any later Up First episode is downloaded and processed, retention has very often already reclaimed the previous one's file -- so instead of ever comparing two episodes, the detector keeps discarding its only candidate and starting over from whatever the newest download is. **This is the real mechanism, not a bug in the correlation math.** Up First is a fast, daily, ~15-30 minute news show that gets played through quickly -- close to the worst case for this race. A show downloaded weekly and left to sit would have days of margin between a play and the next download; Up First has almost none. Cruising the Waves reaching Tentative today is consistent with this: whatever gap that show's two most recent downloads happened to land at was wide enough, or its previous pending episode hadn't been played+reclaimed yet. **What's still not fully explained:** the *second* episode ever downloaded for Up First ("Can clergy candidates...", 2026-09-28) left no trace in `show_fingerprints` at all -- not even the "adopt as new pending" fallback just reproduced live, which would still have moved `updatedAtMs`. Either its `FingerprintWorker` run was deferred long enough that it never actually executed (Doze/battery-restriction deferral of a one-shot job, still unresolved days later), or something force-stopped the app in a way that drops pending `WorkSpec` rows before they run. Secondary to the retention-race finding above, and not reproduced in the live test. ## The fix (made) Peter chose option 1: exempt an episode retention would otherwise reclaim while it is a show's pending comparison basis, for both mechanisms. `RetentionSweeper` gained a `pendingEpisodeIds` seam (`suspend () -> Set`, defaulting to empty so every existing caller and test was unaffected) and filters its reclaim candidates against it before the grace-period policy ever sees them. `RetentionWorker` wires the real thing: the union of `ShowFingerprintDao.getAllPendingEpisodeIds()` (intro/outro) and `MidrollDao.getAllMidrollPendingEpisodeIds()` (midroll, reading `SubscriptionEntity.midrollPendingEpisodeId`) -- both new, narrow queries added for exactly this. Applied to both mechanisms, not just the one that got diagnosed: `MidrollUpdater`'s discovery has the identical shape (a rolling pending episode compared against whatever downloads next) and would race retention the same way for a fast show, just silently -- discovery finding nothing that round rather than a visible frozen row, since it has no fallback state to get stuck in. Not yet re-verified on a device: the next live test is downloading two Up First episodes close enough together that the first is still within its (now-irrelevant, since it's protected) grace period, and confirming `pendingEpisodeId` actually advances to a real second-episode comparison instead of restarting again. ## Experiments, in order (updated) 1. ~~Quantify pre-roll variability.~~ **Done above** -- 44s to 94s across four episodes, and it does not defeat the search. 2. ~~Locate and measure a real intro.~~ **Done above** -- the table gives exact positions; the 6-second figure from the earlier note was in the right neighbourhood but pre-roll length, not sting length, turned out to be the variable that mattered. 3. Sweep `MIN_SCORE` in a *falsely-confirms* direction, not a make-it-pass direction: now that the real default already confirms cleanly, the useful question is how much margin there is before it would start confirming something wrong, not whether a lower threshold helps. 4. **Check 240 seconds is still enough**, on a worse show than Up First. Still open. 5. **Validate midroll totals against `insertedAdMs`.** Still open; no `insertedAdMs`-bearing show was fetched fresh this session. 6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype session already had cached -- still looks like a real intro, not a trap. 7. ~~Get on-device state.~~ **Done above, and it's the one that mattered** -- see "The live test, and the actual root cause." Desktop simulation had gone as far as it could without it. --- ## Do not - **Do not change `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE` or `CONFIRM_THRESHOLD` from the app.** They are tuned numbers and the app cannot show you what a change costs in false positives. - **Do not persist a decoded clip in a database row.** See the CursorWindow note. - **Do not read `EpisodeEntity.durationMs` when you mean the real length.** It is the feed's claim, it is rewritten on every refresh, and on these shows it is minutes short. Use `playbackDurationMs`. - **Do not lower the confirmation threshold to make something confirm.** Failing to skip is a non-event; skipping real content is the failure that loses trust in the feature. --- ## Environment note YouTube extraction and resolution **cannot be exercised from tungsten** — every watch URL gets a 302 to a bot-check page. Anything touching YouTube has to be verified on the phone. Podcast feeds, Room migrations, playback and the fingerprint pipeline are all fine on the emulator. --- ## Fifth pass, 2026-10-03: the cost nobody had measured **Peter's report:** the phone runs hot and uses a lot of battery, and intelligent skipping has never done anything to any episode. Everything above is still true and the retention fix still stands. It was not the whole story, and on the evidence below it was not the dominant blocker either. ### The candidate search cost 199 seconds per end, measured `IntroOutroDetector.findCandidate` bootstrapped a show by sliding an 8-second window across a 240-second clip in quarter-window steps -- **117 positions** -- and running a *full-length* FFT cross-correlation against the whole of the other clip at every one of them, re-transforming that unchanged clip each time. 702 transforms of 2^21 points per episode. Measured, not estimated, by `IntroOutroDetectorScaleTest` at the real sizes and real constants: | path | before | after | | --- | --- | --- | | candidate search, one end | **199.1 s** | 17.8 s | | ditto, full resolution with the target transform hoisted | 136.6 s | -- | | confirming against an existing reference | 1.7 s | 1.7 s | That is on a desktop. A phone has a fraction of this machine's memory bandwidth. ### Why that meant nothing was ever persisted `FingerprintWorker` is a plain WorkManager worker, so **Android stops it at about ten minutes**. Both ends of an episode run inside one worker, and `FingerprintUpdater` writes nothing until a comparison returns -- so the worker was killed mid-correlation, saved nothing, and WorkManager re-enqueued it to start the same work again with backoff. The expensive path is specifically the *bootstrap*, the one that runs until a show has a reference clip. Escaping it requires one run to finish. None ever did. This is a complete explanation of both of Peter's reports at once: hours of full-CPU work, and a `show_fingerprints` row frozen at the first episode ever processed. It also fits the second pass's observation better than the retention theory does. That pass found Up First's row frozen even though a second episode "was downloaded, played and decoded successfully (by a different worker)", and concluded the comparison "never ran at all". On this evidence the comparison *did* run, for ten minutes, and died before its `upsert`. Both mechanisms produce a frozen row; only one of them also produces a hot phone. **One data point still does not fit:** Cruising the Waves reached `confirmedCount=2` that day, so something completed once. Worth checking whether its episodes are short enough to have come in under the limit. ### The fix Coarse-to-fine, the composition `MidrollUpdater` already used and this document already defended: decimate both clips by 8, rank candidate positions on the decimated copies, then re-slice the best three at native resolution and let a full-resolution correlation decide. Plus `AudioCorrelation.prepareTarget`, so the target clip is transformed once per search rather than 117 times. **10.6x faster, and provably the same answer.** `the coarse search picks the same segment full resolution does` asserts the chosen reference window is identical sample for sample -- both searches step the same native grid, so the positions are directly comparable. No detection threshold was touched. ### Midroll detection could never have run at all Separately and independently: `MediaCodecClipSource.decodeFull` buffered every interleaved sample at the file's native rate, then every mono frame, and only then resampled to 8kHz. | episode | native PCM buffer | mono buffer | both live at once | | --- | --- | --- | --- | | 10m | 106 MB | 212 MB | 318 MB | | 13m (Up First) | 145 MB | 289 MB | 434 MB | | 30m | 308 MB | 616 MB | 924 MB | | 60m | 611 MB | 1,221 MB | 1,832 MB | There is no `largeHeap` in the manifest. It threw `OutOfMemoryError`, the `runCatching` in `decode` swallowed it, `decodeFull` returned null, and `MidrollUpdater` then marked the episode **checked forever**. So no episode of any normal length was ever scanned, none would be retried, and the readout said "None found" rather than anything true. Two changes: `MonoResampler` consumes the decoder buffer by buffer and keeps only the 8kHz output, so the peak allocation is the result rather than the input; and `MidrollUpdater.Tuning` now has a `maxDecodableMs` (20 minutes) that skips a longer episode **before** spending anything and **without** marking it checked. **Still outstanding:** an episode over that ceiling is not scanned. Lifting it properly means decoding candidate ranges on demand (`decodeRange`) instead of holding the whole body, which is a real change to `MidrollUpdater.discover`/`extractSegment` and to `MidrollDetector.check` -- all three currently want the entire native-resolution array. Left deliberately undone rather than half-done. --- ## The playback-start pause: one hypothesis disproven, instrumentation shipped Peter's third report: "there's a longish pause when starting playback, proportional to episode length." Recorded here with the others because the next session should not re-check what has already been eliminated. **Nothing in the app's own play path scales with episode length.** Traced end to end: tap -> `PlayerViewModel.play` -> `PlaybackAdvancer.start` (a few indexed row reads) -> `PlayerConnection.play` -> `setMediaItem`/`prepare`. No file scan, no hashing, no probe. **`PrepareTrace` now measures it**, tagged `ListenPlayback`, one line per start: ``` prepare ep=9c9750ef len=1642s src=local ready=666ms playing=667ms ``` **Measured on the emulator, a 27-minute downloaded Radiolab episode: ~400-500ms to READY**, with no other load on the device. So there is a real cost here and it is not only CPU starvation from the fingerprint worker. ### Disproven: the missing MP3 seek table The downloaded file has **no Xing, Info or VBRI header** (checked directly: ID3 tag of 166 bytes, audio from byte 176, none of the three markers present). Without one, Media3 has no seek table to read, and the obvious suspicion was that `Mp3Extractor` was reading through the file to build one -- which would be length-proportional, exactly matching the report. Tested by enabling `Mp3Extractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING` on a `DefaultExtractorsFactory` and measuring the same episode: | configuration | ready (ms) | mean | | --- | --- | --- | | shipped default | 404, 531, 386 | ~440 | | constant-bitrate seeking on | 469, 450, 404 | ~441 | **No difference.** The distributions overlap entirely. A first pair of samples had looked like 666ms against 411ms, which is a 38% improvement and was cold-start noise -- the lesson being that one sample of each proves nothing here, because the spread is wider than any effect. The flag was reverted. It would also have been the wrong trade even if it had worked: constant- bitrate seeking makes seeks approximate on a genuinely VBR file, and playback position is the one piece of data in this app that cannot be reconstructed. ### What is still open Whether the ~440ms scales with episode length. It cannot be settled on the emulator, which has one episode on it. The instrumented build is on the phone, so **two plays of episodes with very different lengths will answer it from the log** -- compare `len=` against `ready=`. If it is flat, the pause is a fixed preparation cost and the "proportional" impression was the fingerprint worker saturating the CPU, which is now fixed. If it scales, the cause is inside Media3's preparation and the next place to look is what `ProgressiveMediaPeriod` reads before reporting READY. --- ## 2026-10-05: detection ported to Go and verified on real audio Peter's call: detection moves to a server, in Go, matching podkit and doot. The port is in `server/`, and this section is what it actually proves -- which is not the same as what it is. ### Verified **Against numpy, not against itself.** `AudioCorrelationTest`'s numpy-derived fixtures were extracted mechanically (not retyped) into `server/internal/detect/fixtures_test.go`, with the same pinned literals: self-match 1.0, embedded sting r=0.9998959789695058 at position 120, unrelated target r=0.3672490793439344. The Go FFT is separately checked against a direct DFT computed from the definition, and the whole FFT correlation path against a brute-force Pearson correlation. That is three independent anchors, because this is the second port of this maths and the first one shipped a normalization bug. **On real audio, end to end.** Four Radiolab episodes, fetched from the live feed: ``` intro: confirmed on 4 of 4 episodes; self-check passed on 6 pairs, worst r=0.9286 ep1 21:07 skip intro to 0:56.000 ep2 22:05 skip intro to 0:16.652 ep3 31:02 skip intro to 0:16.652 ep4 1:03:27 skip intro to 1:13.152 detection took 2m9s ``` The intro sits at a different offset in each episode -- 48.0s, 8.65s, 8.65s, 65.15s -- which is the varying pre-roll this document has described since the first pass, now observed directly in production code. Cross-checking the located segments against each other at exact sample positions gives 0.93 to 0.998 on all six pairs: it is genuinely the same recording in all four. ### What the self-check caught, immediately `skips.SelfCheck` correlates the *located segments against one another*, rather than each against the reference they were found with. Those are different claims. Correlation is not transitive: two segments can each clear 0.6 against a reference while scoring near zero against each other, and offsets derived from such a set would cut different audio out of different episodes. On the very first real run it rejected the outro: ``` outro: confirmed on 3 of 4 episodes; SELF-CHECK FAILED: ep1 vs ep2 at r=0.7460 -- offsets withheld ``` Before the check existed, that same run published outro offsets of 0:12, 0:11 and 0:11 -- which would have ended three episodes early at a point two of them do not share. **The confirmation threshold passed a false positive and the self-check stopped it.** Offsets are now only published when the segments verify against each other, and `listen-detect` exits non-zero when they do not. Outro detection on this show is therefore still unproven. It fails closed, which is the important part, but why a 0.746 pair cleared a 0.6 threshold is the next thing to look at. ### A false alarm worth not repeating The first hand check of the real run showed r=0.03 between two located segments and looked like a serious bug. It was an artifact of reading offsets the CLI had rounded to whole seconds: the true position was 8.65s, so the extracted window was 0.65s out, while two episodes that happened to share the same rounded offset lined up perfectly and scored 0.997. The CLI now prints milliseconds, and `SelfCheck` tolerates half a second of position error for the same reason. ### Not verified, and not built - **No server.** What exists is the detection core and a CLI. The feed poller, the episode mirror, the 8-episode retention, the HTTP API and the phone changes are all still to come. - **Dynamic ad insertion did not reproduce from this host.** All four episodes served *exactly* their declared length -- Grass Apes at 22:05, against the 29:34 and 27:39 measured earlier from the phone. So either insertion is conditioned on the client (user agent, geography, headers) or it is not always on. This matters in both directions: it is the entire reason the server has to serve the audio it detected against, and if the server reliably gets the clean file then mirroring strips the advertising rather than merely locating it. Four samples from one host with curl is not enough to conclude either way, and it should be measured properly before the design leans on it. - **Midroll detection is not ported.** Only intro/outro. --- ## 2026-10-05, later: the server ships, and simultaneous mirroring defeats the threshold `server/` now runs on tungsten as `listend.service`, behind Apache at https://listen.terst.org with a Let's Encrypt certificate. It polls the phone's subscriptions, mirrors the newest 8 episodes per show, detects, and serves both the offsets and the audio they belong to. The phone pushes its feed list from `SubscriptionRepository.refreshAll`, prefers the server's audio when it has it, and writes the offsets into the same `detectedIntroMs`/`detectedOutroMs` columns the on-device detector used -- so playback needed no changes at all. `FingerprintWorker` and `MidrollWorker` are deleted. Verified live, phone to server: `POST /api/subscriptions` returned 200 from the app, the server woke, mirrored 8 Radiolab episodes, and detected across all of them in 3m59s. ### Dynamic ad insertion is on every episode, and it is not subtle Every one of the 8 came back longer than declared: | episode | declared | served | inserted | | --- | --- | --- | --- | | Grass Apes | 22:05 | 28:18 | 6:13 | | The Sweetest Thing | 31:02 | 37:15 | 6:13 | | 23 Weeks 6 Days | 63:27 | 72:26 | 8:59 | | Nigel and the Real Gannet | 25:54 | 33:01 | 7:07 | | The Shoe Thrower | 77:29 | 87:02 | 9:33 | This also corrects the earlier pass in this document, which found declared == served on four fetches made with plain `curl` and doubted whether insertion was happening at all. It is, on every episode, when fetched by the server. Why `curl` got clean files and the server does not is unexplained and worth knowing -- the only obvious difference is the User-Agent. ### The confirmation threshold does not survive simultaneous fetching Detection reported the intro confirmed on 8 of 8, self-check passed. **It had locked onto an advertisement.** Proved, rather than suspected: the 8-second segment the detector chose was extracted from the server's copy of Grass Apes and searched for in the `curl` copy of the same episode fetched hours earlier with no inserted advertising. **r = 0.0621.** It is not in the programme. It is in the ad break. The mechanism is the Darknet Diaries trap at a new scale. The threshold's whole premise is that three *independently fetched* episodes are unlikely to share an inserted ad. Mirroring 8 episodes within the same minute from the same ad server makes them maximally dependent: whatever campaign is running is in all 8, identical, and the detector finds it with scores of 0.93 to 1.00. The self-check cannot catch this -- it only proves the located segments are the same audio, and they genuinely are. This is *worse* than the phone's original design, where episodes arrived days apart and so were genuinely independent. Moving to a server fixed the cost problem and broke the independence assumption the safety mechanism rests on. ### What the published offsets currently are Head-window matches only, so every offset lands inside the first four minutes, and skipping to the end of a matched segment skips pre-roll advertising. That is useful by accident, not by design, and it will move unpredictably whenever the ad campaign rotates. **It is not intro detection and must not be described as such.** ### The fix, not yet made Record when each episode's audio was fetched, and require the agreeing episodes to span more than one fetch occasion before a segment is trusted as the show's own. The mirror already rolls forward one episode at a time as feeds update, so independence arrives naturally -- it is only the initial batch that is simultaneous. Expect no offsets at all for the first day or two after a show is added, which is the correct answer rather than a confident wrong one. There is a reframing worth considering instead, which the data argues for: what this reliably detects, with very high confidence, is **the inserted advertising itself** -- because simultaneous fetches share it exactly. Skipping ads is the actual goal, and the declared-versus- served gap is a free, independent check on whether a scan has accounted for all of it. Detecting ads directly may be a better target than detecting intros and hoping ads do not get in the way. --- ## 2026-10-06: following the data -- detecting the advertising instead Peter's call, on the evidence of the previous section: stop trying to detect intros and detect what the server demonstrably finds. ### Why this is the right way round The property that broke intro detection is the property that makes ad detection work. Eight episodes mirrored within one minute are maximally dependent, because the ad server inserts whatever campaign is running into all of them identically. For intro detection that destroys the independence the confirmation threshold needs. For ad detection it *is* the signal. And unlike intro detection, this can be checked against a number the detector did not produce. The feed declares the ad-free length, the file served is longer, and the difference is the advertising that was inserted. ### The algorithm, and why the old one could not do this The sliding-window search cost one full-length correlation per window position -- 199 seconds for a 240-second clip, measured. Over a 70-minute episode it is hopeless. `detect.BestLags` instead computes the cross-correlation over *every* lag in one FFT pair, which says where two recordings align. `detect.MatchedRuns` then walks the extent of the agreement at that alignment directly, window by window, with no transform at all. Sized against measurements rather than guesses (`fft_bench_test.go`: 2^22 is 1.69s), a 70-minute episode decimated to 250Hz is three transforms of 2^22 -- about five seconds a pair, so a batch of eight is minutes. Silence is explicitly not agreement. Two recordings agree perfectly on it, and counting it would weld separate advertisements together across the pauses between them. ### Measured on the real deployment Radiolab, eight mirrored episodes: ``` Radiolab: ads on 8/8 episode(s) (5 refused as too extensive to be advertising), accounted 53:29 of 59:39 inserted, took 2m56s ``` Per episode, found against what the length gap implies: | episode | found | inserted | ratio | | --- | --- | --- | --- | | Grass Apes | 6:31 | 6:13 | 105% | | The Sweetest Thing | 6:31 | 6:13 | 105% | | 23 Weeks 6 Days | 6:40 | 8:59 | 74% | | Dolly Parton's America | 6:11 | 6:58 | 89% | | Nigel and the Real Gannet | 6:54 | 7:07 | 97% | | The Shoe Thrower | 6:47 | 9:33 | 71% | | The Holy Shiver | 6:26 | 6:59 | 92% | | This is Your Brain on Hormones | 7:26 | 7:33 | 98% | **Every episode lands between 71% and 105% of an independent measure, and the batch total is 90%.** That is the first time in this project that detection has been checked against external ground truth and agreed with it. ### What is published, and what is not Ad breaks are published, as ranges, into `episode_midrolls` -- which `MidrollSkip` and `PlayheadPolicy` have read since the on-device detector existed. Playback needed no changes and there is no migration. **Intro and outro offsets are no longer published.** They are still computed and still logged, because they will become trustworthy once the agreeing episodes are required to span more than one fetch occasion. Until then they are known to mis-identify, and the advertising they kept finding is better served by the scan that measures itself against the length gap. The previously published ad-derived offsets were cleared from the server's state. Two guards decide nothing gets published rather than something wrong: - **An episode whose matches cover more than 35% of it publishes nothing**, because two episodes of a show can legitimately share a great deal -- a re-aired episode, a two-parter with a long recap -- and skipping that would eat the programme. - **A single match longer than four minutes is dropped**, being longer than any plausible ad pod. Both record a refusal rather than silently leaving an episode empty. An episode that published nothing because its matches were implausible is a different state from one where nothing matched, and conflating those two is exactly the mistake the app's skip readout already made. ### Three data-loss bugs found while deploying this, all fixed Found by reading the journal after a restart, not by a test: 1. **The subscription list lived only in memory.** `SetFeeds` from the API did not persist, and state was saved only when a whole poll cycle finished -- which for 61 feeds is hours. A restart in between lost every subscription the phone had sent, and that is the one thing the server cannot rediscover. Now saved the moment it arrives. 2. **A poll cycle re-applied a stale subscription snapshot at the end**, which would silently undo a push that arrived during the cycle and then delete the audio of every show the stale list omitted. Now reconciled against the list as it stands. 3. **Shutdown could not complete.** Detection ignored cancellation, so systemd's default 90-second stop would SIGKILL the process mid-scan and the save-on-exit never ran. Detection now checks for cancellation and the unit allows ten minutes to stop. State is also saved after every feed now, so a restart costs one feed's progress rather than a whole cycle's. ## Detection re-ran forever, and what the first full corpus then showed The first complete pass over Peter's 42 feeds finished on 2026-10-06: 322 episodes mirrored, 286 publishing ad breaks. It then kept going over the same unchanged files, producing results identical to the digit -- Radiolab reported "53:29 of 59:39" four separate times in one day. `pollOne` called `detect` unconditionally, and the mirror skips files it already has, so nothing had changed between passes. One cycle is ~6.5 minutes a show, 4.5 hours for 42 shows, against a two-hour poll interval: the machine never finished a cycle and never idled -- 13h42m of CPU in 11h of wall clock. Roughly 60% of that was the intro/outro search whose results are deliberately not published. `Episode.DetectedAtUnix` existed but could not be used as the gate: it is only set for episodes that appear in `result.Offsets`, so an episode nothing was found in never got a timestamp and the gate would never close. The unit is the batch anyway -- every offset and every break is found by comparing an episode against the others fetched alongside it, so one arrival or one pruning legitimately changes all of them. `Show.DetectedBatch` records the set of episode ids the last completed pass examined, order-independent because feeds are not reliably sorted. Recorded only after a pass completes, so a cancelled or failed one is retried. Deployed 2026-10-07 with the field backfilled for the 42 shows already examined, which saved an immediate 4.5-hour redundant cycle. Verified in production: 31 feeds polled in five minutes with exactly one detection running -- The Ezra Klein Show, which had genuinely gained an episode. ### Filtering by partner count or run length is wrong, and the length gap proves it The obvious cleanup is to drop weakly-supported breaks. 635 of 1574 published breaks (40%) have `partners == 1`, with a median length of 5.0s against 19.0s for `partners >= 4` -- they look exactly like correlation noise, and 270 of them sit at `MinRunSamples`, the 4-second floor. Measured against the declared-versus-served length gap -- the one figure the detector does not produce -- every such filter makes agreement **worse**: | filter | median found/inserted | within 75-125% | skipping on shows with no gap | |---|---|---|---| | none (current) | 67% | 31% | 164 min | | `partners >= 2` | 43% | 17% | 141 min | | `partners >= 3` | 39% | 13% | 127 min | | length >= 8s | 58% | 24% | 141 min | | both | 41% | 13% | 130 min | Detection already **under**-reports -- a 67% median means a third of the advertising is missed -- so removing breaks moves away from the truth, and it barely dents the unverifiable skipping it was meant to clean up. The single-partner short runs are mostly real advertising. Hypothesis disproven; the thresholds stay as they are. ### What is worth fixing is fragmentation 45% of the gaps between consecutive breaks are under ten seconds, and **152 of 262** multi-break episodes contain at least one such gap -- which plays as a skip, one to nine seconds of audible advertisement, then another skip. `merge` only folds breaks that overlap or touch. Merging across a tolerance both cleans that up and moves agreement the right way, because the bridged gap was advertising too: | merge tolerance | median found/inserted | within 75-125% | total breaks | |---|---|---|---| | 0s (current) | 69% | 32% | 1240 | | 2s | 70% | 33% | 973 | | 5s | 70% | 34% | 885 | | 10s | 73% | 34% | 820 | | 15s | 73% | 35% | 786 | ### Sleepy History was not eating the programme Reported on 2026-10-06 as finding 117:42 of "advertising" in a show whose feed declares no inserted ads at all. That reading was wrong: it was a show-level sum, and almost all of it sat inside the two episodes the 35% guard **already refused** -- a 3h08m Rosa Parks compilation that literally contains other episodes (53:41 matched) and Antarctica (22:30). The six episodes that do publish skip 0:34 to 4:33 each, under 10% of their runtime, and the largest block in each is a genuine closing segment shared by every episode in the batch -- `48:20-51:36 partners=22`, `46:19-49:19 partners=26`. That is the show's own outro, which is what intro/outro detection was supposed to find in the first place. The rest is 4-second floor noise. Across the whole corpus only 4 of 279 published episodes skip more than 20% of their runtime, and all but one are corroborated by the length gap to within a minute. The exception is The Trip (3:54 of 14:51, no declared gap), which is the one show worth watching. Sleepy History also serves files consistently **shorter** than declared -- 50:59 declared against 49:23 served, about 1:36 short on every episode -- so `Accounting` reports a zero gap for it. A feed can be wrong in that direction too, and when it is there is no external check available. ## Bridging fragmented pods, and the regression it nearly caused Implemented 2026-10-08 with `AdConfig.MergeGapMs = 10s`. The guard it nearly broke is worth recording. `MaxBreakMs` refused any break longer than four minutes, applied *after* merging. Bridging makes breaks longer, so five episodes would have crossed the cap and published nothing -- including Radiolab's Grass Apes, the episode the whole chain was device-verified on, whose 3:55 pre-roll and 0:18 tag sit two seconds apart and fuse to 4:15. The cap now applies to the matches rather than to what bridging makes of them. One enormous match is evidence of shared programme; a chain of ordinary matches a beat apart is evidence of an ad pod. `MaxSkipFraction` remains the guard on the episode as a whole. ## Theme-music cues around ad breaks: real, detectable, already detected Peter named Tangle, Jon Stewart and Search Engine as shows that play a short piece of theme music at every ad break. All three do, and the scan already finds it -- it is the short, high-partner-count block that the disproven partner-count filter would have thrown away. Search Engine is the clearest: `13:08-13:16`, `35:28-35:36`, `44:51-44:59`, all 8.0s, found in 5-13 other episodes. Jon Stewart's is ~5s at four mid-episode positions per episode. Templating Search Engine's cue at 250Hz and scanning whole episodes with `NormalizedCrossCorrelation` finds it at r = 0.96, 0.92, 0.85 in episodes it was not taken from. **It marks the entry to a break, not both boundaries.** Scanning for the cue down to r = 0.6 returns one isolated hit per slot, never a pair, in ad-laden and ad-free copies alike. So the cue locates a break but says nothing about its extent, which is what a skip needs. ### Two things found while probing this **A second fetch of the same episode is not an ad-free reference.** Three consecutive fetches of one Search Engine enclosure were byte-identical to each other and 82KB different from the copy mirrored days earlier. The ad load is stable within a session and rotates over days, so the "align two fetches and read off the insertions" approach yields nothing when done promptly and yields stale offsets when done late. **Shows have an ad window.** Search Engine's three newest mirrored episodes carry 503s, 469s and 202s of insertion; the five older ones carry exactly zero -- served length equal to declared. The CDN stops inserting after some weeks. That generalises, and it is the more useful finding: **138 of 322 mirrored episodes have served == declared and therefore contain no inserted advertising at all**, and the scan publishes "ad breaks" on 111 of them, 74 minutes in total. Those are not advertisements. They are the show's own recurring furniture -- intro, break cue, outro -- which is worth skipping but is a different thing, and reporting it as inserted advertising is wrong. Every mirrored episode has a usable declared duration, so this classification is always available. ## Lowering the noise floor to 8s: measured, and badly wrong The 12-second floor was set on 2026-10-15 after Peter reported "madness" — 37 breaks in one episode, 14 of them exactly 4.0s, the old floor. It worked: his episode went 37 breaks to 1, and every sub-12s fragment in the corpus disappeared. But it also cut real detection on shows with verified advertising. LeVar Burton Reads went from 19.0 minutes found to 10.5, against the 26.6 the length gap implies. That looked like the floor being too aggressive, and an 8-second floor looked like the fix. **It is not.** Measured 2026-10-19 on an isolated copy of the live data, same episodes, same batch, production code with only `MinRunSamples` changed: | show | 12s floor (shipped) | 8s floor | length gap implies | |---|---|---|---| | LeVar Burton Reads | 10:30 | **19:43** | 27:35 | | Sleepy History | 14:11 | **174:44** | 0:48 | Sleepy History at 8 seconds finds **174 minutes** of "advertising" in a show that has 48 seconds of it — worse than the 48:03 the original 4-second floor produced, and three hours of a listener's programme proposed for deletion. **The two parameters interact, and that is the whole lesson.** `MergeGapMs` was widened to 60s in the same change that raised the floor. A high floor plus wide bridging is fine: few fragments survive, and bridging joins the ones that genuinely belong together. A low floor plus wide bridging is catastrophic: scattered noise fragments survive *and* get welded into enormous spans. Neither parameter can be tuned by looking at it alone, and the 8s number was chosen by post-hoc filtering of already-detected breaks, which cannot model a detector-level change — the same modelling mistake that produced the wrong "31% to 34%" prediction for the floor raise itself. **The floor stays at 12 seconds.** The cost is real and should be stated plainly: on shows with verified ad insertion the scan now finds roughly 38% of what the feed implies, down from about 69% at the old floor. That is the price of not proposing to skip three hours of a sleep podcast. Buying the detection back needs a mechanism that can tell a real short ad from a short coincidence — Peter's manual-skip logging idea is the obvious candidate, since it supplies labels neither the detector nor the feed can. Had only LeVar been measured, the floor would have been lowered. ## 2026-11-02: what the corpus says now, and the tier that is being acted on by accident The previous entry ended by naming what was missing: "a mechanism that can tell a real short ad from a short coincidence". Words are that mechanism, and three of them were built and compared. This entry records what the **live corpus** says, re-derived from `state.json` rather than quoted, because the figure in `publish.go`'s own comment turned out to describe a subset. ### The corpus, measured 2026-11-02 | | | |---|---| | feeds subscribed / shows tracked | 44 / 43 | | episodes known from feeds | 9,020 | | mirrored (audio on disk) | 336 | | with words (whisper or published) | 327 | | with a master to subtract against | 91 | | episodes publishing audio-signature breaks | 267 | | episodes publishing word breaks | 243 | | word break spans published | 597 `cue`, 49 `ad` | | listener labels recorded | 26 | ### Subtraction's accuracy depends entirely on where the master came from `publish.go` says subtraction "measured 92% to 104% ... across eighteen episodes". That is still true of the episodes it was measured on, and it is **not** true of the corpus. Across the 24 episodes that now publish an `ad` span and have a length gap to check it against, recovery runs from 20% to 146%, median 76%. Split by what the master actually is, the spread stops being a spread: | master source | episodes | median recovery | range | |---|---|---|---| | publisher transcript | 9 | **102%** | 94% – 104% | | refetch (a second download of the same file) | 15 | **33%** | 20% – 146% | (Heavyweight's seven carry an empty `masterSource`: the master file predates the field, and `Reclaim` faithfully restored the blank. They are publisher transcripts.) A publisher's transcript is a second *description* of the programme, written once, and subtracting it works -- 94% to 104%, every pod found, edges pulled back off the speech. A refetch is a second *copy of the same dynamically-assembled file*, carrying its own ads at its own positions, so the diff is comparing two ad loads rather than finding one. It recovers a third of the advertising, which matches the 21% and 0% measured on the refetch detector when it was first tried and rejected on exactly those grounds. ### The tier that was rejected is being published anyway Nothing in `Publishable` looks at `MasterSource`. It checks the candidate set against the length gap and nothing else, so a refetch-derived candidate publishes on the same terms as a publisher- derived one -- and refetch masters are now **15 of the 24** episodes publishing an `ad` break, the majority of a tier the design had decided not to act on. The 33% is a real measurement, not an artefact: the refetch shows it rests on have large, consistent length gaps (LeVar Burton Reads 3.39m median across 11 episodes, Darknet Diaries 3.31m, Sources & Methods 3.72m, Serial 3.22m), so there is a solid yardstick to miss two-thirds of. ### The length gap is only a guard where it is large and consistent The second finding, and the one nearly recorded wrongly. Two Reply All episodes publish `ad` spans claiming 146% and 138% of their length gap, just under `overshootLimit`: ``` #188 Into the Depths declared 42.6m served 43.6m gap 1.01m published: one ad span, 1154.2s -> 1243.0s (89s) = 146% of the gap ``` The obvious reading is that the detector over-claimed by 28 seconds and will skip programme. That reading does not survive looking at the show rather than the episode. **Reply All's length gaps across its eight mirrored episodes run from -0.48 to +1.02 minutes** -- they straddle zero. The feed's declared durations are not a baseline for this show at all; the "61 seconds of inserted advertising" that 146% is a percentage *of* is indistinguishable from encoder and feed noise. So the number being exceeded is itself noise, and which of the two is wrong on #188 cannot be settled from this data. What can be settled is the structural point: `HasInsertedAdvertising` asks only whether *this episode's* gap clears one ad unit, and 60.6 seconds clears 15. It has no notion of whether the show's gaps are consistent enough for the figure to mean anything. Compare the shows where it does mean something -- Incels 11.57-14.54m across 8 episodes, Heavyweight 0-10.06m, The Secret World of Roald Dahl 4.83-6.16m -- with the shows whose gaps sit in the noise: Reply All -0.48 to 1.02, Jon Stewart -1.48 to 0.52, Sleepy History -1.57 to 0. A per-episode threshold cannot tell those apart, and the per-show spread is sitting right there in the state file, uncomputed. ### Do not - **Do not quote `publish.go`'s 92-104% as the detector's accuracy.** It is the publisher-master figure. Say which master. - **Do not raise `overshootLimit` to admit more.** On the shows where the gap is meaningful it has never been the binding constraint; on the shows where it binds, the gap is noise. - **Do not read a single episode's gap as ground truth without checking the show's spread.** That mistake was made and caught in the course of writing this entry: the 146% was about to be recorded as 28 seconds of programme being skipped. - **Do not rebuild the ad/furniture classifier.** Measured and deleted: 33% agreement with the length gap, 263 advertisements called furniture, and the markers it keyed on appear in 5% of ads and 4% of furniture -- it was reading noise. Recurring text publishes as `cue` instead, which claims only that the audio repeats across the show's episodes, which is what was actually measured. ### The open questions, in order of how answerable they are 1. **Should a refetch master publish `ad` at all?** The data says the tiers are not comparable (102% vs 33%) and the code treats them as if they were. Answerable now; it changes what gets skipped on fifteen episodes. 2. **Should the length-gap guard consider the show's spread rather than the episode's gap?** The figures above are computable from `state.json` for every show. A show whose gaps straddle zero has no usable yardstick and arguably should not publish a subtraction-derived `ad` either. 3. Whether #188's 89-second span is actually advertising. Needs a listen, not a measurement. ## 2026-11-03: a quarter of the whisper budget was re-doing work Retention was publication date and nothing else. `Show.Partition` kept the newest N and everything else was pruned, and pruning called `words.Forget`, which deletes the transcript *file* as well as the state's record of it. So an episode outside the window was fetched on demand, recognised, served, and deleted again on the next poll -- and asking for it a second time paid the whole cost over. Measured from the live server's journal over twenty-one days: | | | |---|---| | transcription runs | 397 | | distinct episodes | 309 | | first-time work | 25.7 h | | **repeat work** | **7.7 h — 23% of all whisper time** | The mechanism, confirmed in the code rather than inferred: `serveRequest` mirrors an on-demand episode and `detect`s it, but nothing records that anybody wanted it, so the next feed poll partitions by date and prunes exactly what was just fetched. For a back catalogue -- the archive tier, the slow-drip shows, the listening this feature was for -- every episode cost a full recognition and resuming one cost another. **The window now decides what is worth fetching and being asked for decides what is worth keeping.** `Episode.RequestedAtUnix` is stamped on every `/api/episode` call, mirrored episodes included, because resuming something played last month must not let it fall out mid-listen. `Partition` spares anything asked for in the last 28 days -- the same span `refetchAfterSeconds` uses, being the same kind of judgement about how long something stays interesting. Disk makes this cheap in the right direction: transcripts are 36MB for 445 files, audio is 17GB for 336 episodes, and the volume has 365GB free. The expensive artefact is the small one. ### Still open The phone's half. A `pending` reply is retried only on a feed refresh -- the 5am alarm, launch, or the button -- so an episode the server will have in about a minute stays "waiting for server" until the next refresh. That, rather than anything on the server, is what makes "not checked yet" a state a listener sees.