From 9512dd2d38dd1da747ea71e6cb622c2f917583eb Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Tue, 29 Sep 2026 10:30:48 -1000 Subject: Find the actual root cause: retention races the pending-episode file Triggered a fresh Up First download on the phone and watched androidx.work.workdb and show_fingerprints immediately after, live. A new FingerprintWorker job appeared and succeeded within seconds, and show_fingerprints updated for the first time since 2026-09-27 -- but instead of advancing toward a second comparison, it just replaced the stale pending episode with the new download. confirmedCount stayed 0. That's checkAgainstPending's own designed fallback when pendingClipOf can't re-decode the stored pending episode because its file is gone. And it is gone: RetentionPolicy's grace period is one day, and Up First is a fast, daily, ~15-30 minute news show that gets played through quickly. By the time a second episode is downloaded and processed, retention has very often already reclaimed the first one's file -- so the detector never gets two episodes' audio to compare at once, and keeps discarding its only candidate and restarting from whatever the newest download is. Cruising the Waves reaching a real Tentative candidate the same day is consistent with this: a show whose downloads happen to land further apart, or whose previous pending episode hadn't been played+reclaimed yet, doesn't hit the race. This was never a signal-processing problem -- DEFAULT_REFERENCE_SECONDS, MIN_SCORE and CONFIRM_THRESHOLD were the wrong place to look the whole time. Wrote up three options (exempt a pending episode from reclaim, extend the grace period for a show still building its first candidate, or accept the gap for fast shows) as a decision for Peter, not something to just change. Separately unresolved and now secondary: Up First's actual second download (2026-09-28) left zero trace in show_fingerprints, which the live test's fallback path would have if it had run at all -- most likely a deferred or dropped one-shot WorkManager job, not reproduced live this session. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA --- docs/detection-findings.md | 114 ++++++++++++++++++++++++++++++++------------- 1 file changed, 81 insertions(+), 33 deletions(-) diff --git a/docs/detection-findings.md b/docs/detection-findings.md index cacb2f0..4e8e721 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -1,27 +1,35 @@ -# Tier 3 detection: what is known, and what the next desktop session should measure - -Written 2026-09-29, after getting both halves of the fingerprinting onto a device for the first -time. The code is built, tested and running. **It has not yet confirmed an intro, an outro or a -midroll for a real show**, and the reason is an open question rather than a known bug. This is -the brief for answering it. - -The prototype is the right tool for this and the only one: it can hold a whole corpus in memory, -plot correlations, and sweep parameters in seconds. Do not tune the constants from the app. - -**Update, later the same day:** the leading hypothesis below (the 8-second reference window -being too wide for a ~6-second sting) is disproven -- see "What the desktop session actually +# Tier 3 detection: what is known, and the root cause of why nothing confirmed + +Written 2026-09-29, in three passes across the same day as the picture sharpened. **Resolved as +of the last pass**: the correlation algorithm and its tuned constants were never the problem. +Up First specifically confirms nothing because `RetentionPolicy`'s one-day grace period usually +reclaims a played episode's file before the *next* one is downloaded and processed, for a show +this fast and this promptly listened to -- so the detector keeps discarding its only pending +candidate and restarting, rather than ever getting two episodes' files to compare at once. See +"The live test, and the actual root cause" below for the mechanism, and "What this means for a +fix" for what's actually worth changing (a retention/fingerprint coupling decision, not a +constant to tune). + +The rest of this document is kept in the order it was actually found, including the two +hypotheses that turned out to be wrong -- both cost real time to check and are worth not +re-checking blind next time. + +**First pass:** the algorithm was suspected. It isn't -- see "What the desktop session actually found." The production algorithm, run against real Up First audio at its real defaults, confirms cleanly on every pair tested, both ends. -**Second update, the phone reconnected the same day:** pulled the live database and WorkManager's +**Second pass, the phone reconnected the same day:** pulled the live database and WorkManager's own execution history straight off it (read-only, via `run-as`). The pipeline is not broken -- -**Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) today.** Up -First specifically is stuck: its `show_fingerprints` row hasn't updated since its *first* episode +**Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) that day.** Up +First specifically was stuck: its `show_fingerprints` row hadn't updated since its *first* episode was ever processed, even though a second real episode was downloaded, played and decoded successfully (by a different worker) the next day. `FingerprintUpdater` upserts on every outcome, -match or not, so that frozen timestamp means the second episode's comparison never ran at all -- -not a score that failed. See "What the phone actually showed" below for the detail and "What to -check on the phone" for the one live test that would settle it. +match or not, so that frozen timestamp meant the second episode's comparison never ran at all -- +not a score that failed. See "What the phone actually showed" for the detail. + +**Third pass, a live test on the phone:** triggered a fresh download and watched it happen in +real time. That's what found the actual mechanism -- see "The live test, and the actual root +cause." --- @@ -167,19 +175,59 @@ WorkManager prunes completed `WorkSpec` rows after roughly a week, so the actual the next periodic `MidrollWorker` firing were still in the table. This is now a live-observation question, not an archaeology one. -## What to check on the phone (updated -- narrower than the original three) - -- **Trigger a fresh Up First download and watch `androidx.work.workdb` and `show_fingerprints` - immediately after**, the same way this session just did. If a `FingerprintWorker` `WorkSpec` - row appears for the new episode's id and reaches `SUCCEEDED`, but `show_fingerprints.updatedAtMs` - still doesn't move, that is a real bug in `processEnd`/the upsert path, reproduced live. If no - `WorkSpec` row appears at all, `onDownloaded` isn't firing for this show specifically -- check - whether Up First episodes are reaching playback through a route other than - `EpisodeDownloader.download()`. -- **If a `FingerprintWorker` row does appear and reaches SUCCEEDED with no DB change**, add a - temporary log of `clip.samples.size` in `FingerprintUpdater.process` for exactly this case -- - the one thing still not directly observed is whether `decodeHead`/`decodeTail` returned null or - an empty/garbage clip for that specific file. +## The live test, and the actual root cause + +Triggered a fresh Up First download on the phone and watched `androidx.work.workdb` and +`show_fingerprints` immediately after, live: + +- A new `FingerprintWorker` `WorkSpec` row appeared within seconds and reached `SUCCEEDED`. +- `show_fingerprints` for Up First **did** update -- `updatedAtMs` moved to right now, for the + first time since 2026-09-27. But `pendingEpisodeId` didn't advance to a *second* episode to + compare against; it just replaced the old one with this new download. `confirmedCount` stayed + 0, `referenceClipPcm` stayed null. + +That is `checkAgainstPending`'s own designed behaviour when `pendingClipOf` returns null -- +which it does whenever the *stored* pending episode's file no longer exists to re-decode. And it +doesn't: the previous pending episode ("Powerful storms...") was played on 2026-09-27, and +`RetentionPolicy`'s grace period is **one day** (`gracePeriodMs = TimeUnit.DAYS.toMillis(1)`, +`Repositories.kt`). By the time any later Up First episode is downloaded and processed, retention +has very often already reclaimed the previous one's file -- so instead of ever comparing two +episodes, the detector keeps discarding its only candidate and starting over from whatever the +newest download is. + +**This is the real mechanism, not a bug in the correlation math.** Up First is a fast, daily, +~15-30 minute news show that gets played through quickly -- close to the worst case for this +race. A show downloaded weekly and left to sit would have days of margin between a play and the +next download; Up First has almost none. Cruising the Waves reaching Tentative today is +consistent with this: whatever gap that show's two most recent downloads happened to land at was +wide enough, or its previous pending episode hadn't been played+reclaimed yet. + +**What's still not fully explained:** the *second* episode ever downloaded for Up First +("Can clergy candidates...", 2026-09-28) left no trace in `show_fingerprints` at all -- not even +the "adopt as new pending" fallback just reproduced live, which would still have moved +`updatedAtMs`. Either its `FingerprintWorker` run was deferred long enough that it never actually +executed (Doze/battery-restriction deferral of a one-shot job, still unresolved days later), or +something force-stopped the app in a way that drops pending `WorkSpec` rows before they run. +Secondary to the retention-race finding above, and not reproduced in the live test. + +## What this means for a fix (not made -- needs a decision, not a desktop tweak) + +The retention-race finding points at the actual place to fix, and it isn't +`DEFAULT_REFERENCE_SECONDS`/`MIN_SCORE`/`CONFIRM_THRESHOLD`: those were never the problem. +Options, roughly in order of how invasive they are: + +1. **Exempt an episode retention would otherwise reclaim while it is a show's `pendingEpisodeId`.** + Keeps the file exactly as long as Tier 3 actually needs it and no longer; the natural fix, but + couples `RetentionSweeper` to fingerprint state it currently knows nothing about. +2. **Extend the grace period specifically for a show still building its first candidate.** Cruder, + and still racy for a show played even faster than its grace period, just less often. +3. **Accept the miss and let it self-heal.** The current behaviour already does this -- it just + restarts forever for a fast show instead of ever confirming. Only really an "option" if fast + shows are an acceptable gap in Tier 3's coverage. + +This is Peter's call, not a desktop one -- it trades file-retention precision against Tier 3 +coverage for exactly the shows (short, frequent, promptly played) most likely to also carry the +heaviest dynamic ad insertion, which is the case that matters most. ## Experiments, in order (updated) @@ -196,8 +244,8 @@ question, not an archaeology one. was fetched fresh this session. 6. ~~Re-check the Darknet Diaries trap.~~ **Done above**, using data the midroll prototype session already had cached -- still looks like a real intro, not a trap. -7. **New, and now the actual priority: get on-device state**, per "What to check on the phone." - Desktop simulation has done what it can without knowing what the phone actually saw. +7. ~~Get on-device state.~~ **Done above, and it's the one that mattered** -- see "The live test, + and the actual root cause." Desktop simulation had gone as far as it could without it. --- -- cgit v1.2.3