diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/detection-findings.md | 150 |
1 files changed, 107 insertions, 43 deletions
diff --git a/docs/detection-findings.md b/docs/detection-findings.md index 1eac20a..708400f 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -8,6 +8,17 @@ the brief for answering it. The prototype is the right tool for this and the only one: it can hold a whole corpus in memory, plot correlations, and sweep parameters in seconds. Do not tune the constants from the app. +**Update, later the same day:** the leading hypothesis below (the 8-second reference window +being too wide for a ~6-second sting) is disproven. See "What the desktop session actually +found" -- the production algorithm, run against real Up First audio at its real defaults, +confirms cleanly on every pair tested, both ends. The mystery is now sharper, not solved: the +identical algorithm works on desktop and reportedly found nothing on the phone. That gap is very +unlikely to be in `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE`, `CONFIRM_THRESHOLD` or the resampler +-- all four were checked directly -- which points at something specific to the on-device run +itself (which episodes, in what order, whether each `FingerprintWorker` run actually completed) +rather than at the signal-processing approach. Diagnosing that needs on-device state, which this +sandbox cannot reach; see "What to check on the phone" at the end. + --- ## What is settled @@ -55,52 +66,105 @@ close-together fetches. --- -## The specific open question - -Three consecutive Up First downloads left both ends at "Nothing found yet". - -`FingerprintUpdater.DEFAULT_REFERENCE_SECONDS` is **8**. The original prototype measured Up -First's intro sting at about **6 seconds**. If that is right, every 8-second window being -compared also contains ~2 seconds of differing speech, which would plausibly hold correlation -under `MIN_SCORE` (0.6) for every pair. That is the leading hypothesis and it is untested. - -The competing hypothesis is that pre-roll variability is defeating it — that there is no stable -8-second stretch anywhere in the head window because the ads differ per fetch and the intro's -position moves with them. - -These are distinguishable by measurement, and the measurements are cheap. +## What the desktop session actually found + +Run against the four Up First episodes already cached from the original prototype +(`npr/ep1..4_head.wav`, `npr/ep1..4_tail.wav`, 240s at 8kHz each), reusing the same +`normalized_xcorr`/`best_match` primitive as `AudioCorrelation.kt` -- unmodified, so this is a +direct check of what the app itself computes, not an approximation of it. + +**The leading hypothesis is disproven.** Simulating `IntroOutroDetector.findCandidate` exactly -- +an 8-second window, quarter-window step, global search in the other clip -- at the app's real +`DEFAULT_REFERENCE_SECONDS = 8` and `MIN_SCORE = 0.6`: + +| pair | intro r | intro position | outro r | outro position | +| --- | --- | --- | --- | --- | +| ep1→ep2 | 0.821 | a=44.0s b=44.0s | 0.999 | a=184.0s b=166.9s | +| ep1→ep3 | 0.807 | a=44.0s b=46.7s | 0.994 | a=226.0s b=184.0s | +| ep1→ep4 | 0.998 | a=90.0s b=93.7s | 0.995 | a=184.0s b=166.4s | +| ep2→ep3 | 0.805 | a=44.0s b=46.7s | 0.997 | a=180.0s b=184.3s | +| ep2→ep4 | 0.864 | a=44.0s b=49.3s | 0.997 | a=168.0s b=167.4s | +| ep3→ep4 | 0.793 | a=46.0s b=48.7s | 0.998 | a=172.0s b=167.2s | + +Every pair clears 0.6 at both ends, several by a wide margin. The 8-second window is not too +wide for the sting -- it finds it cleanly regardless of exactly how much of the window is +"extra" content around it. + +**Pre-roll varies a lot, and the algorithm doesn't care.** The intro lands anywhere from 44s to +94s into the head window across these four episodes -- up to 50 seconds of variability -- and +the global search finds it at every position tried. This is the "competing hypothesis" from the +first draft of this doc, and it is real (pre-roll length does vary) but it does not defeat +detection, because `findCandidate` was already built to search the whole window rather than +assume a fixed position. + +**The resampler is not it either.** `MediaCodecClipSource.resample` is naive linear +interpolation, not a proper anti-aliasing filter, and the worry was that downsampling 44.1kHz +mp3 to 8kHz that way could be degrading correlation enough to matter. Decoded the same episode's +native 44.1kHz stereo, downmixed and resampled with the exact linear-interpolation formula from +`MediaCodecClipSource.kt`, and compared: **0.9991 correlation against a proper (ffmpeg/swresample) +decode of the same audio**, and re-running the full production simulation with the naively +resampled clip still confirms at r=0.821 -- indistinguishable from the ffmpeg path. Ruled out. + +**A same-episode re-fetch, three days apart, is byte-identical for ~50 seconds and then diverges +completely.** The exact same episode URL, fetched fresh today against a copy saved 2026-09-26: +zero-lag correlation is 1.000 for every 5-second block from 0s to 45s, then falls to ~0 (pure +noise) for every block from 50s to 240s. Whatever comes after the intro is not stable across +fetches -- consistent with, and sharper evidence for, the "dynamic ad insertion is the +environment" finding above -- but the intro itself sits inside the stable prefix, so this does +not threaten intro detection specifically. It is a fresh reason to be careful trusting anything +found much later in a head/tail window, and matches why outro detection uses the *tail*, not a +position computed from the (unreliable) declared duration. + +**The Darknet Diaries trap re-checked (experiment 6) without a new fetch.** The midroll +prototype work already re-fetched this exact pair: a ~51-second segment at position 0 matches +at r≈1.000 across `dd_179` and `dd_178` (2026), `dd_177` (2026) and `dd_008` from **2017** -- +nine years apart, fetched today. A truly rotating, per-request DAI ad reproducing identically +against a specific nine-year-old archived fetch, by chance, is not plausible. This still looks +like Darknet Diaries' real, stable intro, not the trap the threshold exists to catch -- which +doesn't make the threshold wrong (a genuinely rotating ad would still be caught the same way), +just confirms this particular historical match was likely a false alarm rather than evidence of +one. --- -## Experiments, in order - -1. **Quantify pre-roll variability.** Fetch the *same* episode three times, some hours apart. - Diff the first 240 seconds. How much differs, and where does the common content start? This - also tells you whether `insertedAdMs` varies for one episode, which it did for Grass Apes. - -2. **Locate and measure a real intro.** Across several Up First episodes, find the sting: where - does it start (after how much pre-roll) and how long is it actually? The 6-second figure is - from an earlier note and should be confirmed, not inherited. - -3. **Sweep the two constants against a real corpus.** `DEFAULT_REFERENCE_SECONDS` at 4, 6, 8 and - `MIN_SCORE` around 0.6. Report what confirms and — more importantly — what *falsely* confirms. - A setting that finds Up First's intro but also learns a rotating ad is worse than the current - state, which at least fails safe. - -4. **Check 240 seconds is still enough.** Pre-roll plus intro, on the worst show you can find. - If it is marginal, the window is a cheaper thing to raise than the thresholds — but note it - costs decode time per download, and the clip must never be persisted (see the CursorWindow - note above). - -5. **Validate midroll totals against `insertedAdMs`.** Take a show with a known declared/measured - gap, run the midroll scan across several episodes, and compare the total detected break - duration against the gap. This is the best available ground truth and it costs nothing. - -6. **Re-check the Darknet Diaries trap under the new understanding.** The 2017 episode that - shared an "intro" with 2026 episodes was the original evidence for the threshold. Given ad - insertion varies per *request*, re-fetch that pair now and see whether the false match still - reproduces — it may have been a stable house promo rather than a rotating ad, which would - change what the threshold is actually protecting against. +## What to check on the phone + +Everything above rules out the algorithm and its tuned constants as an explanation, on real Up +First audio, at the app's real settings. That leaves the actual on-device run as the place to +look, and none of it can be done from this sandbox: + +- **What did the three downloaded episodes actually decode to?** A quick temporary log of + `clip.samples.size` and a couple of sample values in `FingerprintUpdater.process` would show + whether `MediaCodecClipSource` produced a full ~240s clip each time, or something truncated -- + a decode that silently stops early would explain "nothing found" without any correlation math + being wrong. +- **Did all three `FingerprintWorker` runs actually complete and persist?** `adb shell` + + `sqlite3` against `show_fingerprints` after each download would show `confirmedCount` climbing + toward 3 (or not) and whether `pendingEpisodeId` ever got set at all -- distinguishing "never + got past Waiting" from "got to Tentative and stalled," which "Nothing found yet" in the UI + cannot. +- **Were the three episodes actually three *different* episodes?** `IntroOutroDetector.check`'s + dedup guard silently no-ops on an id it has already seen (`lastConfirmedEpisodeId` or + `pendingEpisodeId`) -- worth ruling out that a re-queued or re-triggered download fed the same + episode through twice. + +## Experiments, in order (updated) + +1. ~~Quantify pre-roll variability.~~ **Done above** -- 44s to 94s across four episodes, and it + does not defeat the search. +2. ~~Locate and measure a real intro.~~ **Done above** -- the table gives exact positions; the + 6-second figure from the earlier note was in the right neighbourhood but pre-roll length, not + sting length, turned out to be the variable that mattered. +3. Sweep `MIN_SCORE` in a *falsely-confirms* direction, not a make-it-pass direction: now that + the real default already confirms cleanly, the useful question is how much margin there is + before it would start confirming something wrong, not whether a lower threshold helps. +4. **Check 240 seconds is still enough**, on a worse show than Up First. Still open. +5. **Validate midroll totals against `insertedAdMs`.** Still open; no `insertedAdMs`-bearing show + was fetched fresh this session. +6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype + session already had cached -- still looks like a real intro, not a trap. +7. **New, and now the actual priority: get on-device state**, per "What to check on the phone." + Desktop simulation has done what it can without knowing what the phone actually saw. --- |