Disprove the leading Tier 3 detection hypothesis on real Up First audio

The brief's leading hypothesis -- an 8-second reference window too wide for
a ~6-second sting -- does not survive contact with real data. Simulating
IntroOutroDetector.findCandidate exactly (the same normalized_xcorr/best_match
math AudioCorrelation.kt uses, unmodified) against the four Up First episodes
already cached from the original prototype: every one of six pairs confirms
cleanly at both ends, at the app's real DEFAULT_REFERENCE_SECONDS=8 and
MIN_SCORE=0.6 -- intro r 0.79-0.998, outro r 0.994-0.999.

Also ruled out: the naive linear-interpolation resampler MediaCodecClipSource
actually uses. Reproduced it exactly against a fresh re-fetch's native 44.1kHz
audio and compared to a proper ffmpeg decode -- 0.9991 correlation, and the
production simulation confirms identically either way.

Two findings beyond the original brief:
- Pre-roll length varies a lot (44s-94s across four episodes) but the global
  search already handles that; it isn't the blocker the first draft worried
  it might be.
- The same episode URL, fetched fresh today against a copy from 2026-09-26,
  is byte-identical for the first ~50 seconds and then diverges completely --
  sharper evidence for "dynamic ad insertion is the environment" than the
  Radiolab duration mismatch alone, though it lands after the useful intro
  signal so doesn't threaten detection specifically.

Experiment 6 (re-check the Darknet Diaries trap) didn't need a new fetch --
the midroll prototype session's already-cached data answers it: the shared
~51s segment still matches at r=1.000 across a nine-year gap, which a truly
rotating per-request ad could not do by chance. Looks like a real intro, not
the trap the threshold exists to catch.

Net effect: the algorithm and its tuned constants are no longer plausible
suspects on real audio. What's left points at the specific on-device run
rather than the approach -- see "What to check on the phone" for the three
concrete things to look at that this sandbox has no way to check itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA

1 files changed, 107 insertions, 43 deletions

diff --git a/docs/detection-findings.md b/docs/detection-findings.md
index 1eac20a..708400f 100644
--- a/docs/detection-findings.md
+++ b/docs/detection-findings.md
@@ -8,6 +8,17 @@ the brief for answering it.
 The prototype is the right tool for this and the only one: it can hold a whole corpus in memory,
 plot correlations, and sweep parameters in seconds. Do not tune the constants from the app.
 
+**Update, later the same day:** the leading hypothesis below (the 8-second reference window
+being too wide for a ~6-second sting) is disproven. See "What the desktop session actually
+found" -- the production algorithm, run against real Up First audio at its real defaults,
+confirms cleanly on every pair tested, both ends. The mystery is now sharper, not solved: the
+identical algorithm works on desktop and reportedly found nothing on the phone. That gap is very
+unlikely to be in `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE`, `CONFIRM_THRESHOLD` or the resampler
+-- all four were checked directly -- which points at something specific to the on-device run
+itself (which episodes, in what order, whether each `FingerprintWorker` run actually completed)
+rather than at the signal-processing approach. Diagnosing that needs on-device state, which this
+sandbox cannot reach; see "What to check on the phone" at the end.
+
 ---
 
 ## What is settled
@@ -55,52 +66,105 @@ close-together fetches.
 
 ---
 
-## The specific open question
-
-Three consecutive Up First downloads left both ends at "Nothing found yet".
-
-`FingerprintUpdater.DEFAULT_REFERENCE_SECONDS` is **8**. The original prototype measured Up
-First's intro sting at about **6 seconds**. If that is right, every 8-second window being
-compared also contains ~2 seconds of differing speech, which would plausibly hold correlation
-under `MIN_SCORE` (0.6) for every pair. That is the leading hypothesis and it is untested.
-
-The competing hypothesis is that pre-roll variability is defeating it — that there is no stable
-8-second stretch anywhere in the head window because the ads differ per fetch and the intro's
-position moves with them.
-
-These are distinguishable by measurement, and the measurements are cheap.
+## What the desktop session actually found
+
+Run against the four Up First episodes already cached from the original prototype
+(`npr/ep1..4_head.wav`, `npr/ep1..4_tail.wav`, 240s at 8kHz each), reusing the same
+`normalized_xcorr`/`best_match` primitive as `AudioCorrelation.kt` -- unmodified, so this is a
+direct check of what the app itself computes, not an approximation of it.
+
+**The leading hypothesis is disproven.** Simulating `IntroOutroDetector.findCandidate` exactly --
+an 8-second window, quarter-window step, global search in the other clip -- at the app's real
+`DEFAULT_REFERENCE_SECONDS = 8` and `MIN_SCORE = 0.6`:
+
+| pair | intro r | intro position | outro r | outro position |
+| --- | --- | --- | --- | --- |
+| ep1→ep2 | 0.821 | a=44.0s b=44.0s | 0.999 | a=184.0s b=166.9s |
+| ep1→ep3 | 0.807 | a=44.0s b=46.7s | 0.994 | a=226.0s b=184.0s |
+| ep1→ep4 | 0.998 | a=90.0s b=93.7s | 0.995 | a=184.0s b=166.4s |
+| ep2→ep3 | 0.805 | a=44.0s b=46.7s | 0.997 | a=180.0s b=184.3s |
+| ep2→ep4 | 0.864 | a=44.0s b=49.3s | 0.997 | a=168.0s b=167.4s |
+| ep3→ep4 | 0.793 | a=46.0s b=48.7s | 0.998 | a=172.0s b=167.2s |
+
+Every pair clears 0.6 at both ends, several by a wide margin. The 8-second window is not too
+wide for the sting -- it finds it cleanly regardless of exactly how much of the window is
+"extra" content around it.
+
+**Pre-roll varies a lot, and the algorithm doesn't care.** The intro lands anywhere from 44s to
+94s into the head window across these four episodes -- up to 50 seconds of variability -- and
+the global search finds it at every position tried. This is the "competing hypothesis" from the
+first draft of this doc, and it is real (pre-roll length does vary) but it does not defeat
+detection, because `findCandidate` was already built to search the whole window rather than
+assume a fixed position.
+
+**The resampler is not it either.** `MediaCodecClipSource.resample` is naive linear
+interpolation, not a proper anti-aliasing filter, and the worry was that downsampling 44.1kHz
+mp3 to 8kHz that way could be degrading correlation enough to matter. Decoded the same episode's
+native 44.1kHz stereo, downmixed and resampled with the exact linear-interpolation formula from
+`MediaCodecClipSource.kt`, and compared: **0.9991 correlation against a proper (ffmpeg/swresample)
+decode of the same audio**, and re-running the full production simulation with the naively
+resampled clip still confirms at r=0.821 -- indistinguishable from the ffmpeg path. Ruled out.
+
+**A same-episode re-fetch, three days apart, is byte-identical for ~50 seconds and then diverges
+completely.** The exact same episode URL, fetched fresh today against a copy saved 2026-09-26:
+zero-lag correlation is 1.000 for every 5-second block from 0s to 45s, then falls to ~0 (pure
+noise) for every block from 50s to 240s. Whatever comes after the intro is not stable across
+fetches -- consistent with, and sharper evidence for, the "dynamic ad insertion is the
+environment" finding above -- but the intro itself sits inside the stable prefix, so this does
+not threaten intro detection specifically. It is a fresh reason to be careful trusting anything
+found much later in a head/tail window, and matches why outro detection uses the *tail*, not a
+position computed from the (unreliable) declared duration.
+
+**The Darknet Diaries trap re-checked (experiment 6) without a new fetch.** The midroll
+prototype work already re-fetched this exact pair: a ~51-second segment at position 0 matches
+at r≈1.000 across `dd_179` and `dd_178` (2026), `dd_177` (2026) and `dd_008` from **2017** --
+nine years apart, fetched today. A truly rotating, per-request DAI ad reproducing identically
+against a specific nine-year-old archived fetch, by chance, is not plausible. This still looks
+like Darknet Diaries' real, stable intro, not the trap the threshold exists to catch -- which
+doesn't make the threshold wrong (a genuinely rotating ad would still be caught the same way),
+just confirms this particular historical match was likely a false alarm rather than evidence of
+one.
 
 ---
 
-## Experiments, in order
-
-1. **Quantify pre-roll variability.** Fetch the *same* episode three times, some hours apart.
-   Diff the first 240 seconds. How much differs, and where does the common content start? This
-   also tells you whether `insertedAdMs` varies for one episode, which it did for Grass Apes.
-
-2. **Locate and measure a real intro.** Across several Up First episodes, find the sting: where
-   does it start (after how much pre-roll) and how long is it actually? The 6-second figure is
-   from an earlier note and should be confirmed, not inherited.
-
-3. **Sweep the two constants against a real corpus.** `DEFAULT_REFERENCE_SECONDS` at 4, 6, 8 and
-   `MIN_SCORE` around 0.6. Report what confirms and — more importantly — what *falsely* confirms.
-   A setting that finds Up First's intro but also learns a rotating ad is worse than the current
-   state, which at least fails safe.
-
-4. **Check 240 seconds is still enough.** Pre-roll plus intro, on the worst show you can find.
-   If it is marginal, the window is a cheaper thing to raise than the thresholds — but note it
-   costs decode time per download, and the clip must never be persisted (see the CursorWindow
-   note above).
-
-5. **Validate midroll totals against `insertedAdMs`.** Take a show with a known declared/measured
-   gap, run the midroll scan across several episodes, and compare the total detected break
-   duration against the gap. This is the best available ground truth and it costs nothing.
-
-6. **Re-check the Darknet Diaries trap under the new understanding.** The 2017 episode that
-   shared an "intro" with 2026 episodes was the original evidence for the threshold. Given ad
-   insertion varies per *request*, re-fetch that pair now and see whether the false match still
-   reproduces — it may have been a stable house promo rather than a rotating ad, which would
-   change what the threshold is actually protecting against.
+## What to check on the phone
+
+Everything above rules out the algorithm and its tuned constants as an explanation, on real Up
+First audio, at the app's real settings. That leaves the actual on-device run as the place to
+look, and none of it can be done from this sandbox:
+
+- **What did the three downloaded episodes actually decode to?** A quick temporary log of
+  `clip.samples.size` and a couple of sample values in `FingerprintUpdater.process` would show
+  whether `MediaCodecClipSource` produced a full ~240s clip each time, or something truncated --
+  a decode that silently stops early would explain "nothing found" without any correlation math
+  being wrong.
+- **Did all three `FingerprintWorker` runs actually complete and persist?** `adb shell` +
+  `sqlite3` against `show_fingerprints` after each download would show `confirmedCount` climbing
+  toward 3 (or not) and whether `pendingEpisodeId` ever got set at all -- distinguishing "never
+  got past Waiting" from "got to Tentative and stalled," which "Nothing found yet" in the UI
+  cannot.
+- **Were the three episodes actually three *different* episodes?** `IntroOutroDetector.check`'s
+  dedup guard silently no-ops on an id it has already seen (`lastConfirmedEpisodeId` or
+  `pendingEpisodeId`) -- worth ruling out that a re-queued or re-triggered download fed the same
+  episode through twice.
+
+## Experiments, in order (updated)
+
+1. ~~Quantify pre-roll variability.~~ **Done above** -- 44s to 94s across four episodes, and it
+   does not defeat the search.
+2. ~~Locate and measure a real intro.~~ **Done above** -- the table gives exact positions; the
+   6-second figure from the earlier note was in the right neighbourhood but pre-roll length, not
+   sting length, turned out to be the variable that mattered.
+3. Sweep `MIN_SCORE` in a *falsely-confirms* direction, not a make-it-pass direction: now that
+   the real default already confirms cleanly, the useful question is how much margin there is
+   before it would start confirming something wrong, not whether a lower threshold helps.
+4. **Check 240 seconds is still enough**, on a worse show than Up First. Still open.
+5. **Validate midroll totals against `insertedAdMs`.** Still open; no `insertedAdMs`-bearing show
+   was fetched fresh this session.
+6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype
+   session already had cached -- still looks like a real intro, not a trap.
+7. **New, and now the actual priority: get on-device state**, per "What to check on the phone."
+   Desktop simulation has done what it can without knowing what the phone actually saw.
 
 ---