Find the actual root cause: retention races the pending-episode file

Triggered a fresh Up First download on the phone and watched
androidx.work.workdb and show_fingerprints immediately after, live. A new
FingerprintWorker job appeared and succeeded within seconds, and
show_fingerprints updated for the first time since 2026-09-27 -- but instead
of advancing toward a second comparison, it just replaced the stale pending
episode with the new download. confirmedCount stayed 0.

That's checkAgainstPending's own designed fallback when pendingClipOf can't
re-decode the stored pending episode because its file is gone. And it is
gone: RetentionPolicy's grace period is one day, and Up First is a fast,
daily, ~15-30 minute news show that gets played through quickly. By the time
a second episode is downloaded and processed, retention has very often
already reclaimed the first one's file -- so the detector never gets two
episodes' audio to compare at once, and keeps discarding its only candidate
and restarting from whatever the newest download is. Cruising the Waves
reaching a real Tentative candidate the same day is consistent with this: a
show whose downloads happen to land further apart, or whose previous pending
episode hadn't been played+reclaimed yet, doesn't hit the race.

This was never a signal-processing problem -- DEFAULT_REFERENCE_SECONDS,
MIN_SCORE and CONFIRM_THRESHOLD were the wrong place to look the whole time.
Wrote up three options (exempt a pending episode from reclaim, extend the
grace period for a show still building its first candidate, or accept the
gap for fast shows) as a decision for Peter, not something to just change.

Separately unresolved and now secondary: Up First's actual second download
(2026-09-28) left zero trace in show_fingerprints, which the live test's
fallback path would have if it had run at all -- most likely a deferred or
dropped one-shot WorkManager job, not reproduced live this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA

1 files changed, 81 insertions, 33 deletions

diff --git a/docs/detection-findings.md b/docs/detection-findings.md
index cacb2f0..4e8e721 100644
--- a/docs/detection-findings.md
+++ b/docs/detection-findings.md
@@ -1,27 +1,35 @@
-# Tier 3 detection: what is known, and what the next desktop session should measure
-
-Written 2026-09-29, after getting both halves of the fingerprinting onto a device for the first
-time. The code is built, tested and running. **It has not yet confirmed an intro, an outro or a
-midroll for a real show**, and the reason is an open question rather than a known bug. This is
-the brief for answering it.
-
-The prototype is the right tool for this and the only one: it can hold a whole corpus in memory,
-plot correlations, and sweep parameters in seconds. Do not tune the constants from the app.
-
-**Update, later the same day:** the leading hypothesis below (the 8-second reference window
-being too wide for a ~6-second sting) is disproven -- see "What the desktop session actually
+# Tier 3 detection: what is known, and the root cause of why nothing confirmed
+
+Written 2026-09-29, in three passes across the same day as the picture sharpened. **Resolved as
+of the last pass**: the correlation algorithm and its tuned constants were never the problem.
+Up First specifically confirms nothing because `RetentionPolicy`'s one-day grace period usually
+reclaims a played episode's file before the *next* one is downloaded and processed, for a show
+this fast and this promptly listened to -- so the detector keeps discarding its only pending
+candidate and restarting, rather than ever getting two episodes' files to compare at once. See
+"The live test, and the actual root cause" below for the mechanism, and "What this means for a
+fix" for what's actually worth changing (a retention/fingerprint coupling decision, not a
+constant to tune).
+
+The rest of this document is kept in the order it was actually found, including the two
+hypotheses that turned out to be wrong -- both cost real time to check and are worth not
+re-checking blind next time.
+
+**First pass:** the algorithm was suspected. It isn't -- see "What the desktop session actually
 found." The production algorithm, run against real Up First audio at its real defaults, confirms
 cleanly on every pair tested, both ends.
 
-**Second update, the phone reconnected the same day:** pulled the live database and WorkManager's
+**Second pass, the phone reconnected the same day:** pulled the live database and WorkManager's
 own execution history straight off it (read-only, via `run-as`). The pipeline is not broken --
-**Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) today.** Up
-First specifically is stuck: its `show_fingerprints` row hasn't updated since its *first* episode
+**Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) that day.** Up
+First specifically was stuck: its `show_fingerprints` row hadn't updated since its *first* episode
 was ever processed, even though a second real episode was downloaded, played and decoded
 successfully (by a different worker) the next day. `FingerprintUpdater` upserts on every outcome,
-match or not, so that frozen timestamp means the second episode's comparison never ran at all --
-not a score that failed. See "What the phone actually showed" below for the detail and "What to
-check on the phone" for the one live test that would settle it.
+match or not, so that frozen timestamp meant the second episode's comparison never ran at all --
+not a score that failed. See "What the phone actually showed" for the detail.
+
+**Third pass, a live test on the phone:** triggered a fresh download and watched it happen in
+real time. That's what found the actual mechanism -- see "The live test, and the actual root
+cause."
 
 ---
 
@@ -167,19 +175,59 @@ WorkManager prunes completed `WorkSpec` rows after roughly a week, so the actual
 the next periodic `MidrollWorker` firing were still in the table. This is now a live-observation
 question, not an archaeology one.
 
-## What to check on the phone (updated -- narrower than the original three)
-
-- **Trigger a fresh Up First download and watch `androidx.work.workdb` and `show_fingerprints`
-  immediately after**, the same way this session just did. If a `FingerprintWorker` `WorkSpec`
-  row appears for the new episode's id and reaches `SUCCEEDED`, but `show_fingerprints.updatedAtMs`
-  still doesn't move, that is a real bug in `processEnd`/the upsert path, reproduced live. If no
-  `WorkSpec` row appears at all, `onDownloaded` isn't firing for this show specifically -- check
-  whether Up First episodes are reaching playback through a route other than
-  `EpisodeDownloader.download()`.
-- **If a `FingerprintWorker` row does appear and reaches SUCCEEDED with no DB change**, add a
-  temporary log of `clip.samples.size` in `FingerprintUpdater.process` for exactly this case --
-  the one thing still not directly observed is whether `decodeHead`/`decodeTail` returned null or
-  an empty/garbage clip for that specific file.
+## The live test, and the actual root cause
+
+Triggered a fresh Up First download on the phone and watched `androidx.work.workdb` and
+`show_fingerprints` immediately after, live:
+
+- A new `FingerprintWorker` `WorkSpec` row appeared within seconds and reached `SUCCEEDED`.
+- `show_fingerprints` for Up First **did** update -- `updatedAtMs` moved to right now, for the
+  first time since 2026-09-27. But `pendingEpisodeId` didn't advance to a *second* episode to
+  compare against; it just replaced the old one with this new download. `confirmedCount` stayed
+  0, `referenceClipPcm` stayed null.
+
+That is `checkAgainstPending`'s own designed behaviour when `pendingClipOf` returns null --
+which it does whenever the *stored* pending episode's file no longer exists to re-decode. And it
+doesn't: the previous pending episode ("Powerful storms...") was played on 2026-09-27, and
+`RetentionPolicy`'s grace period is **one day** (`gracePeriodMs = TimeUnit.DAYS.toMillis(1)`,
+`Repositories.kt`). By the time any later Up First episode is downloaded and processed, retention
+has very often already reclaimed the previous one's file -- so instead of ever comparing two
+episodes, the detector keeps discarding its only candidate and starting over from whatever the
+newest download is.
+
+**This is the real mechanism, not a bug in the correlation math.** Up First is a fast, daily,
+~15-30 minute news show that gets played through quickly -- close to the worst case for this
+race. A show downloaded weekly and left to sit would have days of margin between a play and the
+next download; Up First has almost none. Cruising the Waves reaching Tentative today is
+consistent with this: whatever gap that show's two most recent downloads happened to land at was
+wide enough, or its previous pending episode hadn't been played+reclaimed yet.
+
+**What's still not fully explained:** the *second* episode ever downloaded for Up First
+("Can clergy candidates...", 2026-09-28) left no trace in `show_fingerprints` at all -- not even
+the "adopt as new pending" fallback just reproduced live, which would still have moved
+`updatedAtMs`. Either its `FingerprintWorker` run was deferred long enough that it never actually
+executed (Doze/battery-restriction deferral of a one-shot job, still unresolved days later), or
+something force-stopped the app in a way that drops pending `WorkSpec` rows before they run.
+Secondary to the retention-race finding above, and not reproduced in the live test.
+
+## What this means for a fix (not made -- needs a decision, not a desktop tweak)
+
+The retention-race finding points at the actual place to fix, and it isn't
+`DEFAULT_REFERENCE_SECONDS`/`MIN_SCORE`/`CONFIRM_THRESHOLD`: those were never the problem.
+Options, roughly in order of how invasive they are:
+
+1. **Exempt an episode retention would otherwise reclaim while it is a show's `pendingEpisodeId`.**
+   Keeps the file exactly as long as Tier 3 actually needs it and no longer; the natural fix, but
+   couples `RetentionSweeper` to fingerprint state it currently knows nothing about.
+2. **Extend the grace period specifically for a show still building its first candidate.** Cruder,
+   and still racy for a show played even faster than its grace period, just less often.
+3. **Accept the miss and let it self-heal.** The current behaviour already does this -- it just
+   restarts forever for a fast show instead of ever confirming. Only really an "option" if fast
+   shows are an acceptable gap in Tier 3's coverage.
+
+This is Peter's call, not a desktop one -- it trades file-retention precision against Tier 3
+coverage for exactly the shows (short, frequent, promptly played) most likely to also carry the
+heaviest dynamic ad insertion, which is the case that matters most.
 
 ## Experiments, in order (updated)
 
@@ -196,8 +244,8 @@ question, not an archaeology one.
    was fetched fresh this session.
 6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype
    session already had cached -- still looks like a real intro, not a trap.
-7. **New, and now the actual priority: get on-device state**, per "What to check on the phone."
-   Desktop simulation has done what it can without knowing what the phone actually saw.
+7. ~~Get on-device state.~~ **Done above, and it's the one that mattered** -- see "The live test,
+   and the actual root cause." Desktop simulation had gone as far as it could without it.
 
 ---