# Tier 3 detection: what is known, and the root cause of why nothing confirmed Written 2026-09-29, in three passes across the same day as the picture sharpened. **Resolved as of the last pass**: the correlation algorithm and its tuned constants were never the problem. Up First specifically confirms nothing because `RetentionPolicy`'s one-day grace period usually reclaims a played episode's file before the *next* one is downloaded and processed, for a show this fast and this promptly listened to -- so the detector keeps discarding its only pending candidate and restarting, rather than ever getting two episodes' files to compare at once. See "The live test, and the actual root cause" below for the mechanism, and "What this means for a fix" for what's actually worth changing (a retention/fingerprint coupling decision, not a constant to tune). The rest of this document is kept in the order it was actually found, including the two hypotheses that turned out to be wrong -- both cost real time to check and are worth not re-checking blind next time. **First pass:** the algorithm was suspected. It isn't -- see "What the desktop session actually found." The production algorithm, run against real Up First audio at its real defaults, confirms cleanly on every pair tested, both ends. **Second pass, the phone reconnected the same day:** pulled the live database and WorkManager's own execution history straight off it (read-only, via `run-as`). The pipeline is not broken -- **Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) that day.** Up First specifically was stuck: its `show_fingerprints` row hadn't updated since its *first* episode was ever processed, even though a second real episode was downloaded, played and decoded successfully (by a different worker) the next day. `FingerprintUpdater` upserts on every outcome, match or not, so that frozen timestamp meant the second episode's comparison never ran at all -- not a score that failed. See "What the phone actually showed" for the detail. **Third pass, a live test on the phone:** triggered a fresh download and watched it happen in real time. That's what found the actual mechanism -- see "The live test, and the actual root cause." --- ## What is settled **The pipeline runs on hardware.** `MediaCodecClipSource` decodes, `FingerprintWorker` completes SUCCESS per download, the re-decode path for the pending episode is exercised. This was never true before 2026-09-28 and two real defects were hiding behind it (below). **The confirmation threshold earns its place.** Three independently-fetched episodes must agree before a candidate is trusted. The original prototype found why: two Darknet Diaries episodes fetched minutes apart agreed on a segment that turned out to be a rotating inserted ad, not the show's intro. Everything below makes that trap *more* likely, not less. Do not lower it to make something confirm. **Storage limits are real and were learned the hard way.** Android reads every Room row through a CursorWindow that refuses anything over 2MB *on the way out* — the insert succeeds and the next read throws. A 240-second clip at 8kHz 16-bit is 3.84MB. Intro/outro pending clips are no longer stored at all (the episode id is, and the audio is re-decoded); midroll candidate clips live in files. `StoredClip` owns the limit and `StoredClipSizeTest` pins it. --- ## The finding that changes the problem **Dynamic ad insertion is not a detail; it is the environment.** Radiolab declares Grass Apes at `00:22:05`. Measured on two separate fetches of the same episode: **29:34 once, 27:39 another time.** The ads are stitched in per request, so: - the served file's length is not a property of the episode, - the ad *content* differs between two downloads of the same episode, - and anything at a fixed offset in one file is at a different offset in the next. `insertedAdMs` (declared vs measured) now exposes this per episode in the app. It is free, needs no audio analysis, and is the obvious cross-check on the detector: **whatever the midroll scan finds should add up to roughly that much.** If it finds 30 seconds where the gap says seven minutes, the scan is missing breaks; if it finds more, it is matching content. This reframes the intro/outro question. The head window is 240 seconds, which was chosen to comfortably contain an intro. With a variable-length pre-roll in front of it, a show's real intro sits at a *different offset in every file* — which the scan does handle, since it searches the window rather than a fixed position — but the window now has to contain pre-roll **plus** intro, and the pre-roll is the thing most likely to produce a false match between two close-together fetches. --- ## What the desktop session actually found Run against the four Up First episodes already cached from the original prototype (`npr/ep1..4_head.wav`, `npr/ep1..4_tail.wav`, 240s at 8kHz each), reusing the same `normalized_xcorr`/`best_match` primitive as `AudioCorrelation.kt` -- unmodified, so this is a direct check of what the app itself computes, not an approximation of it. **The leading hypothesis is disproven.** Simulating `IntroOutroDetector.findCandidate` exactly -- an 8-second window, quarter-window step, global search in the other clip -- at the app's real `DEFAULT_REFERENCE_SECONDS = 8` and `MIN_SCORE = 0.6`: | pair | intro r | intro position | outro r | outro position | | --- | --- | --- | --- | --- | | ep1→ep2 | 0.821 | a=44.0s b=44.0s | 0.999 | a=184.0s b=166.9s | | ep1→ep3 | 0.807 | a=44.0s b=46.7s | 0.994 | a=226.0s b=184.0s | | ep1→ep4 | 0.998 | a=90.0s b=93.7s | 0.995 | a=184.0s b=166.4s | | ep2→ep3 | 0.805 | a=44.0s b=46.7s | 0.997 | a=180.0s b=184.3s | | ep2→ep4 | 0.864 | a=44.0s b=49.3s | 0.997 | a=168.0s b=167.4s | | ep3→ep4 | 0.793 | a=46.0s b=48.7s | 0.998 | a=172.0s b=167.2s | Every pair clears 0.6 at both ends, several by a wide margin. The 8-second window is not too wide for the sting -- it finds it cleanly regardless of exactly how much of the window is "extra" content around it. **Pre-roll varies a lot, and the algorithm doesn't care.** The intro lands anywhere from 44s to 94s into the head window across these four episodes -- up to 50 seconds of variability -- and the global search finds it at every position tried. This is the "competing hypothesis" from the first draft of this doc, and it is real (pre-roll length does vary) but it does not defeat detection, because `findCandidate` was already built to search the whole window rather than assume a fixed position. **The resampler is not it either.** `MediaCodecClipSource.resample` is naive linear interpolation, not a proper anti-aliasing filter, and the worry was that downsampling 44.1kHz mp3 to 8kHz that way could be degrading correlation enough to matter. Decoded the same episode's native 44.1kHz stereo, downmixed and resampled with the exact linear-interpolation formula from `MediaCodecClipSource.kt`, and compared: **0.9991 correlation against a proper (ffmpeg/swresample) decode of the same audio**, and re-running the full production simulation with the naively resampled clip still confirms at r=0.821 -- indistinguishable from the ffmpeg path. Ruled out. **A same-episode re-fetch, three days apart, is byte-identical for ~50 seconds and then diverges completely.** The exact same episode URL, fetched fresh today against a copy saved 2026-09-26: zero-lag correlation is 1.000 for every 5-second block from 0s to 45s, then falls to ~0 (pure noise) for every block from 50s to 240s. Whatever comes after the intro is not stable across fetches -- consistent with, and sharper evidence for, the "dynamic ad insertion is the environment" finding above -- but the intro itself sits inside the stable prefix, so this does not threaten intro detection specifically. It is a fresh reason to be careful trusting anything found much later in a head/tail window, and matches why outro detection uses the *tail*, not a position computed from the (unreliable) declared duration. **The Darknet Diaries trap re-checked (experiment 6) without a new fetch.** The midroll prototype work already re-fetched this exact pair: a ~51-second segment at position 0 matches at r≈1.000 across `dd_179` and `dd_178` (2026), `dd_177` (2026) and `dd_008` from **2017** -- nine years apart, fetched today. A truly rotating, per-request DAI ad reproducing identically against a specific nine-year-old archived fetch, by chance, is not plausible. This still looks like Darknet Diaries' real, stable intro, not the trap the threshold exists to catch -- which doesn't make the threshold wrong (a genuinely rotating ad would still be caught the same way), just confirms this particular historical match was likely a false alarm rather than evidence of one. --- ## What the phone actually showed (2026-09-29, later) The phone reconnected mid-session. Pulled `listen.db` (via `run-as` + `adb exec-out cat`, no device modification) and `no_backup/androidx.work.workdb` -- WorkManager's own execution history, which nothing in this app writes to but which turned out to answer the question the app state alone couldn't. **The pipeline is not broken.** `show_fingerprints` for **Cruising the Waves Podcast** shows `kind=INTRO, confirmedCount=2, referenceClipPcm` populated -- a real Tentative candidate, found today, from an ordinary download. Something *does* reach Tentative on this exact build, on this exact phone. The mechanism works in general; Up First specifically is the anomaly, not Tier 3. **Up First's `show_fingerprints` row is frozen at its very first episode.** Both INTRO and OUTRO show `confirmedCount=0`, `referenceClipPcm=NULL`, `pendingEpisodeId` pointing at one single episode ("Powerful storms hit the US..."), and `updatedAtMs = 2026-09-27T02:19` -- the moment that first episode was processed. A second real episode ("Can clergy candidates help Democrats win in red states?") was downloaded, played and swept by retention the *next day*, and its `midrollCheckedAtMs` is set, proving `MidrollWorker`'s `decodeFull` read that file successfully that day. But `show_fingerprints.updatedAtMs` never moved again. That is the specific, damning fact: `FingerprintUpdater.processEnd` calls `fingerprints.upsert(...)` on **every** outcome -- a confirmed match, a failed match, all of it -- so if a comparison involving the second episode had run at all, this row would show a later timestamp regardless of whether it found anything. It doesn't. **The second episode's `FingerprintWorker` run either was never enqueued, or `decodeHead`/`decodeTail` both returned null for it** -- and the latter looks unlikely given `decodeFull` (the same underlying decode, just a different window) succeeded on the identical file later that day. `EpisodeDownloader.downloadOverHttp` was read end to end look for a race between "downloaded" and "playable" -- there isn't one, it's a fully blocking download-to-`.part`-then-atomic-rename before `onDownloaded` fires, so a half-written file is not the likely explanation either. WorkManager prunes completed `WorkSpec` rows after roughly a week, so the actual historical `FingerprintWorker` run (if any) for that second episode is already gone -- only today's runs and the next periodic `MidrollWorker` firing were still in the table. This is now a live-observation question, not an archaeology one. ## The live test, and the actual root cause Triggered a fresh Up First download on the phone and watched `androidx.work.workdb` and `show_fingerprints` immediately after, live: - A new `FingerprintWorker` `WorkSpec` row appeared within seconds and reached `SUCCEEDED`. - `show_fingerprints` for Up First **did** update -- `updatedAtMs` moved to right now, for the first time since 2026-09-27. But `pendingEpisodeId` didn't advance to a *second* episode to compare against; it just replaced the old one with this new download. `confirmedCount` stayed 0, `referenceClipPcm` stayed null. That is `checkAgainstPending`'s own designed behaviour when `pendingClipOf` returns null -- which it does whenever the *stored* pending episode's file no longer exists to re-decode. And it doesn't: the previous pending episode ("Powerful storms...") was played on 2026-09-27, and `RetentionPolicy`'s grace period is **one day** (`gracePeriodMs = TimeUnit.DAYS.toMillis(1)`, `Repositories.kt`). By the time any later Up First episode is downloaded and processed, retention has very often already reclaimed the previous one's file -- so instead of ever comparing two episodes, the detector keeps discarding its only candidate and starting over from whatever the newest download is. **This is the real mechanism, not a bug in the correlation math.** Up First is a fast, daily, ~15-30 minute news show that gets played through quickly -- close to the worst case for this race. A show downloaded weekly and left to sit would have days of margin between a play and the next download; Up First has almost none. Cruising the Waves reaching Tentative today is consistent with this: whatever gap that show's two most recent downloads happened to land at was wide enough, or its previous pending episode hadn't been played+reclaimed yet. **What's still not fully explained:** the *second* episode ever downloaded for Up First ("Can clergy candidates...", 2026-09-28) left no trace in `show_fingerprints` at all -- not even the "adopt as new pending" fallback just reproduced live, which would still have moved `updatedAtMs`. Either its `FingerprintWorker` run was deferred long enough that it never actually executed (Doze/battery-restriction deferral of a one-shot job, still unresolved days later), or something force-stopped the app in a way that drops pending `WorkSpec` rows before they run. Secondary to the retention-race finding above, and not reproduced in the live test. ## What this means for a fix (not made -- needs a decision, not a desktop tweak) The retention-race finding points at the actual place to fix, and it isn't `DEFAULT_REFERENCE_SECONDS`/`MIN_SCORE`/`CONFIRM_THRESHOLD`: those were never the problem. Options, roughly in order of how invasive they are: 1. **Exempt an episode retention would otherwise reclaim while it is a show's `pendingEpisodeId`.** Keeps the file exactly as long as Tier 3 actually needs it and no longer; the natural fix, but couples `RetentionSweeper` to fingerprint state it currently knows nothing about. 2. **Extend the grace period specifically for a show still building its first candidate.** Cruder, and still racy for a show played even faster than its grace period, just less often. 3. **Accept the miss and let it self-heal.** The current behaviour already does this -- it just restarts forever for a fast show instead of ever confirming. Only really an "option" if fast shows are an acceptable gap in Tier 3's coverage. This is Peter's call, not a desktop one -- it trades file-retention precision against Tier 3 coverage for exactly the shows (short, frequent, promptly played) most likely to also carry the heaviest dynamic ad insertion, which is the case that matters most. ## Experiments, in order (updated) 1. ~~Quantify pre-roll variability.~~ **Done above** -- 44s to 94s across four episodes, and it does not defeat the search. 2. ~~Locate and measure a real intro.~~ **Done above** -- the table gives exact positions; the 6-second figure from the earlier note was in the right neighbourhood but pre-roll length, not sting length, turned out to be the variable that mattered. 3. Sweep `MIN_SCORE` in a *falsely-confirms* direction, not a make-it-pass direction: now that the real default already confirms cleanly, the useful question is how much margin there is before it would start confirming something wrong, not whether a lower threshold helps. 4. **Check 240 seconds is still enough**, on a worse show than Up First. Still open. 5. **Validate midroll totals against `insertedAdMs`.** Still open; no `insertedAdMs`-bearing show was fetched fresh this session. 6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype session already had cached -- still looks like a real intro, not a trap. 7. ~~Get on-device state.~~ **Done above, and it's the one that mattered** -- see "The live test, and the actual root cause." Desktop simulation had gone as far as it could without it. --- ## Do not - **Do not change `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE` or `CONFIRM_THRESHOLD` from the app.** They are tuned numbers and the app cannot show you what a change costs in false positives. - **Do not persist a decoded clip in a database row.** See the CursorWindow note. - **Do not read `EpisodeEntity.durationMs` when you mean the real length.** It is the feed's claim, it is rewritten on every refresh, and on these shows it is minutes short. Use `playbackDurationMs`. - **Do not lower the confirmation threshold to make something confirm.** Failing to skip is a non-event; skipping real content is the failure that loses trust in the feature. --- ## Environment note YouTube extraction and resolution **cannot be exercised from tungsten** — every watch URL gets a 302 to a bot-check page. Anything touching YouTube has to be verified on the phone. Podcast feeds, Room migrations, playback and the fingerprint pipeline are all fine on the emulator.