1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
|
# Tier 3 detection: what is known, and what the next desktop session should measure
Written 2026-09-29, after getting both halves of the fingerprinting onto a device for the first
time. The code is built, tested and running. **It has not yet confirmed an intro, an outro or a
midroll for a real show**, and the reason is an open question rather than a known bug. This is
the brief for answering it.
The prototype is the right tool for this and the only one: it can hold a whole corpus in memory,
plot correlations, and sweep parameters in seconds. Do not tune the constants from the app.
**Update, later the same day:** the leading hypothesis below (the 8-second reference window
being too wide for a ~6-second sting) is disproven. See "What the desktop session actually
found" -- the production algorithm, run against real Up First audio at its real defaults,
confirms cleanly on every pair tested, both ends. The mystery is now sharper, not solved: the
identical algorithm works on desktop and reportedly found nothing on the phone. That gap is very
unlikely to be in `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE`, `CONFIRM_THRESHOLD` or the resampler
-- all four were checked directly -- which points at something specific to the on-device run
itself (which episodes, in what order, whether each `FingerprintWorker` run actually completed)
rather than at the signal-processing approach. Diagnosing that needs on-device state, which this
sandbox cannot reach; see "What to check on the phone" at the end.
---
## What is settled
**The pipeline runs on hardware.** `MediaCodecClipSource` decodes, `FingerprintWorker` completes
SUCCESS per download, the re-decode path for the pending episode is exercised. This was never
true before 2026-09-28 and two real defects were hiding behind it (below).
**The confirmation threshold earns its place.** Three independently-fetched episodes must agree
before a candidate is trusted. The original prototype found why: two Darknet Diaries episodes
fetched minutes apart agreed on a segment that turned out to be a rotating inserted ad, not the
show's intro. Everything below makes that trap *more* likely, not less. Do not lower it to make
something confirm.
**Storage limits are real and were learned the hard way.** Android reads every Room row through
a CursorWindow that refuses anything over 2MB *on the way out* — the insert succeeds and the
next read throws. A 240-second clip at 8kHz 16-bit is 3.84MB. Intro/outro pending clips are no
longer stored at all (the episode id is, and the audio is re-decoded); midroll candidate clips
live in files. `StoredClip` owns the limit and `StoredClipSizeTest` pins it.
---
## The finding that changes the problem
**Dynamic ad insertion is not a detail; it is the environment.**
Radiolab declares Grass Apes at `00:22:05`. Measured on two separate fetches of the same
episode: **29:34 once, 27:39 another time.** The ads are stitched in per request, so:
- the served file's length is not a property of the episode,
- the ad *content* differs between two downloads of the same episode,
- and anything at a fixed offset in one file is at a different offset in the next.
`insertedAdMs` (declared vs measured) now exposes this per episode in the app. It is free, needs
no audio analysis, and is the obvious cross-check on the detector: **whatever the midroll scan
finds should add up to roughly that much.** If it finds 30 seconds where the gap says seven
minutes, the scan is missing breaks; if it finds more, it is matching content.
This reframes the intro/outro question. The head window is 240 seconds, which was chosen to
comfortably contain an intro. With a variable-length pre-roll in front of it, a show's real
intro sits at a *different offset in every file* — which the scan does handle, since it searches
the window rather than a fixed position — but the window now has to contain pre-roll **plus**
intro, and the pre-roll is the thing most likely to produce a false match between two
close-together fetches.
---
## What the desktop session actually found
Run against the four Up First episodes already cached from the original prototype
(`npr/ep1..4_head.wav`, `npr/ep1..4_tail.wav`, 240s at 8kHz each), reusing the same
`normalized_xcorr`/`best_match` primitive as `AudioCorrelation.kt` -- unmodified, so this is a
direct check of what the app itself computes, not an approximation of it.
**The leading hypothesis is disproven.** Simulating `IntroOutroDetector.findCandidate` exactly --
an 8-second window, quarter-window step, global search in the other clip -- at the app's real
`DEFAULT_REFERENCE_SECONDS = 8` and `MIN_SCORE = 0.6`:
| pair | intro r | intro position | outro r | outro position |
| --- | --- | --- | --- | --- |
| ep1→ep2 | 0.821 | a=44.0s b=44.0s | 0.999 | a=184.0s b=166.9s |
| ep1→ep3 | 0.807 | a=44.0s b=46.7s | 0.994 | a=226.0s b=184.0s |
| ep1→ep4 | 0.998 | a=90.0s b=93.7s | 0.995 | a=184.0s b=166.4s |
| ep2→ep3 | 0.805 | a=44.0s b=46.7s | 0.997 | a=180.0s b=184.3s |
| ep2→ep4 | 0.864 | a=44.0s b=49.3s | 0.997 | a=168.0s b=167.4s |
| ep3→ep4 | 0.793 | a=46.0s b=48.7s | 0.998 | a=172.0s b=167.2s |
Every pair clears 0.6 at both ends, several by a wide margin. The 8-second window is not too
wide for the sting -- it finds it cleanly regardless of exactly how much of the window is
"extra" content around it.
**Pre-roll varies a lot, and the algorithm doesn't care.** The intro lands anywhere from 44s to
94s into the head window across these four episodes -- up to 50 seconds of variability -- and
the global search finds it at every position tried. This is the "competing hypothesis" from the
first draft of this doc, and it is real (pre-roll length does vary) but it does not defeat
detection, because `findCandidate` was already built to search the whole window rather than
assume a fixed position.
**The resampler is not it either.** `MediaCodecClipSource.resample` is naive linear
interpolation, not a proper anti-aliasing filter, and the worry was that downsampling 44.1kHz
mp3 to 8kHz that way could be degrading correlation enough to matter. Decoded the same episode's
native 44.1kHz stereo, downmixed and resampled with the exact linear-interpolation formula from
`MediaCodecClipSource.kt`, and compared: **0.9991 correlation against a proper (ffmpeg/swresample)
decode of the same audio**, and re-running the full production simulation with the naively
resampled clip still confirms at r=0.821 -- indistinguishable from the ffmpeg path. Ruled out.
**A same-episode re-fetch, three days apart, is byte-identical for ~50 seconds and then diverges
completely.** The exact same episode URL, fetched fresh today against a copy saved 2026-09-26:
zero-lag correlation is 1.000 for every 5-second block from 0s to 45s, then falls to ~0 (pure
noise) for every block from 50s to 240s. Whatever comes after the intro is not stable across
fetches -- consistent with, and sharper evidence for, the "dynamic ad insertion is the
environment" finding above -- but the intro itself sits inside the stable prefix, so this does
not threaten intro detection specifically. It is a fresh reason to be careful trusting anything
found much later in a head/tail window, and matches why outro detection uses the *tail*, not a
position computed from the (unreliable) declared duration.
**The Darknet Diaries trap re-checked (experiment 6) without a new fetch.** The midroll
prototype work already re-fetched this exact pair: a ~51-second segment at position 0 matches
at r≈1.000 across `dd_179` and `dd_178` (2026), `dd_177` (2026) and `dd_008` from **2017** --
nine years apart, fetched today. A truly rotating, per-request DAI ad reproducing identically
against a specific nine-year-old archived fetch, by chance, is not plausible. This still looks
like Darknet Diaries' real, stable intro, not the trap the threshold exists to catch -- which
doesn't make the threshold wrong (a genuinely rotating ad would still be caught the same way),
just confirms this particular historical match was likely a false alarm rather than evidence of
one.
---
## What to check on the phone
Everything above rules out the algorithm and its tuned constants as an explanation, on real Up
First audio, at the app's real settings. That leaves the actual on-device run as the place to
look, and none of it can be done from this sandbox:
- **What did the three downloaded episodes actually decode to?** A quick temporary log of
`clip.samples.size` and a couple of sample values in `FingerprintUpdater.process` would show
whether `MediaCodecClipSource` produced a full ~240s clip each time, or something truncated --
a decode that silently stops early would explain "nothing found" without any correlation math
being wrong.
- **Did all three `FingerprintWorker` runs actually complete and persist?** `adb shell` +
`sqlite3` against `show_fingerprints` after each download would show `confirmedCount` climbing
toward 3 (or not) and whether `pendingEpisodeId` ever got set at all -- distinguishing "never
got past Waiting" from "got to Tentative and stalled," which "Nothing found yet" in the UI
cannot.
- **Were the three episodes actually three *different* episodes?** `IntroOutroDetector.check`'s
dedup guard silently no-ops on an id it has already seen (`lastConfirmedEpisodeId` or
`pendingEpisodeId`) -- worth ruling out that a re-queued or re-triggered download fed the same
episode through twice.
## Experiments, in order (updated)
1. ~~Quantify pre-roll variability.~~ **Done above** -- 44s to 94s across four episodes, and it
does not defeat the search.
2. ~~Locate and measure a real intro.~~ **Done above** -- the table gives exact positions; the
6-second figure from the earlier note was in the right neighbourhood but pre-roll length, not
sting length, turned out to be the variable that mattered.
3. Sweep `MIN_SCORE` in a *falsely-confirms* direction, not a make-it-pass direction: now that
the real default already confirms cleanly, the useful question is how much margin there is
before it would start confirming something wrong, not whether a lower threshold helps.
4. **Check 240 seconds is still enough**, on a worse show than Up First. Still open.
5. **Validate midroll totals against `insertedAdMs`.** Still open; no `insertedAdMs`-bearing show
was fetched fresh this session.
6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype
session already had cached -- still looks like a real intro, not a trap.
7. **New, and now the actual priority: get on-device state**, per "What to check on the phone."
Desktop simulation has done what it can without knowing what the phone actually saw.
---
## Do not
- **Do not change `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE` or `CONFIRM_THRESHOLD` from the app.**
They are tuned numbers and the app cannot show you what a change costs in false positives.
- **Do not persist a decoded clip in a database row.** See the CursorWindow note.
- **Do not read `EpisodeEntity.durationMs` when you mean the real length.** It is the feed's
claim, it is rewritten on every refresh, and on these shows it is minutes short. Use
`playbackDurationMs`.
- **Do not lower the confirmation threshold to make something confirm.** Failing to skip is a
non-event; skipping real content is the failure that loses trust in the feature.
---
## Environment note
YouTube extraction and resolution **cannot be exercised from tungsten** — every watch URL gets a
302 to a bot-check page. Anything touching YouTube has to be verified on the phone. Podcast
feeds, Room migrations, playback and the fingerprint pipeline are all fine on the emulator.
|
