From 7018b03d874930a44764131e160b3e3738ae2f53 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Tue, 29 Sep 2026 10:19:54 -1000 Subject: Find the real Tier 3 blocker: Up First's second episode never got compared The phone reconnected mid-session, so this moved from desktop simulation to live device state -- pulled listen.db and WorkManager's own execution history straight off it (read-only, via run-as + adb exec-out cat, nothing modified). The pipeline is not broken: show_fingerprints for Cruising the Waves Podcast shows a real Tentative INTRO candidate (confirmedCount=2, a populated referenceClipPcm) from an ordinary download today. Something does reach Tentative on this exact build. Up First specifically is stuck, and the evidence is more precise than "nothing found yet" suggested. Its show_fingerprints row for both INTRO and OUTRO hasn't updated since 2026-09-27T02:19 -- the moment its very first episode was ever processed. A second real episode was downloaded, played, and swept by retention the next day, and its midrollCheckedAtMs is set, proving MidrollWorker's decodeFull read that exact file successfully that day. But FingerprintUpdater.processEnd calls fingerprints.upsert() on every outcome, confirmed, tentative, or discarded -- so if any comparison involving that second episode had run at all, the row's updatedAtMs would have moved. It didn't. The second episode's FingerprintWorker run either was never enqueued, or decodeHead/decodeTail both returned null for it, and the latter looks unlikely given decodeFull succeeded on the identical file the same day. Read EpisodeDownloader.downloadOverHttp end to end looking for a download-vs-playable race; there isn't one -- it's a blocking download to .part then an atomic rename before onDownloaded fires. WorkManager prunes completed WorkSpec rows after about a week, so the actual historical FingerprintWorker run (if any) for that episode is already gone. Narrows "What to check on the phone" to one live test: trigger a fresh Up First download and watch androidx.work.workdb and show_fingerprints immediately after, the same way this session just did. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA --- docs/detection-findings.md | 91 +++++++++++++++++++++++++++++++--------------- 1 file changed, 62 insertions(+), 29 deletions(-) (limited to 'docs') diff --git a/docs/detection-findings.md b/docs/detection-findings.md index 708400f..cacb2f0 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -9,15 +9,19 @@ The prototype is the right tool for this and the only one: it can hold a whole c plot correlations, and sweep parameters in seconds. Do not tune the constants from the app. **Update, later the same day:** the leading hypothesis below (the 8-second reference window -being too wide for a ~6-second sting) is disproven. See "What the desktop session actually -found" -- the production algorithm, run against real Up First audio at its real defaults, -confirms cleanly on every pair tested, both ends. The mystery is now sharper, not solved: the -identical algorithm works on desktop and reportedly found nothing on the phone. That gap is very -unlikely to be in `DEFAULT_REFERENCE_SECONDS`, `MIN_SCORE`, `CONFIRM_THRESHOLD` or the resampler --- all four were checked directly -- which points at something specific to the on-device run -itself (which episodes, in what order, whether each `FingerprintWorker` run actually completed) -rather than at the signal-processing approach. Diagnosing that needs on-device state, which this -sandbox cannot reach; see "What to check on the phone" at the end. +being too wide for a ~6-second sting) is disproven -- see "What the desktop session actually +found." The production algorithm, run against real Up First audio at its real defaults, confirms +cleanly on every pair tested, both ends. + +**Second update, the phone reconnected the same day:** pulled the live database and WorkManager's +own execution history straight off it (read-only, via `run-as`). The pipeline is not broken -- +**Cruising the Waves Podcast reached a real Tentative candidate (confirmedCount=2) today.** Up +First specifically is stuck: its `show_fingerprints` row hasn't updated since its *first* episode +was ever processed, even though a second real episode was downloaded, played and decoded +successfully (by a different worker) the next day. `FingerprintUpdater` upserts on every outcome, +match or not, so that frozen timestamp means the second episode's comparison never ran at all -- +not a score that failed. See "What the phone actually showed" below for the detail and "What to +check on the phone" for the one live test that would settle it. --- @@ -127,26 +131,55 @@ one. --- -## What to check on the phone - -Everything above rules out the algorithm and its tuned constants as an explanation, on real Up -First audio, at the app's real settings. That leaves the actual on-device run as the place to -look, and none of it can be done from this sandbox: - -- **What did the three downloaded episodes actually decode to?** A quick temporary log of - `clip.samples.size` and a couple of sample values in `FingerprintUpdater.process` would show - whether `MediaCodecClipSource` produced a full ~240s clip each time, or something truncated -- - a decode that silently stops early would explain "nothing found" without any correlation math - being wrong. -- **Did all three `FingerprintWorker` runs actually complete and persist?** `adb shell` + - `sqlite3` against `show_fingerprints` after each download would show `confirmedCount` climbing - toward 3 (or not) and whether `pendingEpisodeId` ever got set at all -- distinguishing "never - got past Waiting" from "got to Tentative and stalled," which "Nothing found yet" in the UI - cannot. -- **Were the three episodes actually three *different* episodes?** `IntroOutroDetector.check`'s - dedup guard silently no-ops on an id it has already seen (`lastConfirmedEpisodeId` or - `pendingEpisodeId`) -- worth ruling out that a re-queued or re-triggered download fed the same - episode through twice. +## What the phone actually showed (2026-09-29, later) + +The phone reconnected mid-session. Pulled `listen.db` (via `run-as` + `adb exec-out cat`, no +device modification) and `no_backup/androidx.work.workdb` -- WorkManager's own execution +history, which nothing in this app writes to but which turned out to answer the question the app +state alone couldn't. + +**The pipeline is not broken.** `show_fingerprints` for **Cruising the Waves Podcast** shows +`kind=INTRO, confirmedCount=2, referenceClipPcm` populated -- a real Tentative candidate, found +today, from an ordinary download. Something *does* reach Tentative on this exact build, on this +exact phone. The mechanism works in general; Up First specifically is the anomaly, not Tier 3. + +**Up First's `show_fingerprints` row is frozen at its very first episode.** Both INTRO and OUTRO +show `confirmedCount=0`, `referenceClipPcm=NULL`, `pendingEpisodeId` pointing at one single +episode ("Powerful storms hit the US..."), and `updatedAtMs = 2026-09-27T02:19` -- the moment +that first episode was processed. A second real episode ("Can clergy candidates help Democrats +win in red states?") was downloaded, played and swept by retention the *next day*, and its +`midrollCheckedAtMs` is set, proving `MidrollWorker`'s `decodeFull` read that file successfully +that day. But `show_fingerprints.updatedAtMs` never moved again. + +That is the specific, damning fact: `FingerprintUpdater.processEnd` calls `fingerprints.upsert(...)` +on **every** outcome -- a confirmed match, a failed match, all of it -- so if a comparison +involving the second episode had run at all, this row would show a later timestamp regardless of +whether it found anything. It doesn't. **The second episode's `FingerprintWorker` run either was +never enqueued, or `decodeHead`/`decodeTail` both returned null for it** -- and the latter looks +unlikely given `decodeFull` (the same underlying decode, just a different window) succeeded on +the identical file later that day. `EpisodeDownloader.downloadOverHttp` was read end to end look +for a race between "downloaded" and "playable" -- there isn't one, it's a fully blocking +download-to-`.part`-then-atomic-rename before `onDownloaded` fires, so a half-written file is not +the likely explanation either. + +WorkManager prunes completed `WorkSpec` rows after roughly a week, so the actual historical +`FingerprintWorker` run (if any) for that second episode is already gone -- only today's runs and +the next periodic `MidrollWorker` firing were still in the table. This is now a live-observation +question, not an archaeology one. + +## What to check on the phone (updated -- narrower than the original three) + +- **Trigger a fresh Up First download and watch `androidx.work.workdb` and `show_fingerprints` + immediately after**, the same way this session just did. If a `FingerprintWorker` `WorkSpec` + row appears for the new episode's id and reaches `SUCCEEDED`, but `show_fingerprints.updatedAtMs` + still doesn't move, that is a real bug in `processEnd`/the upsert path, reproduced live. If no + `WorkSpec` row appears at all, `onDownloaded` isn't firing for this show specifically -- check + whether Up First episodes are reaching playback through a route other than + `EpisodeDownloader.download()`. +- **If a `FingerprintWorker` row does appear and reaches SUCCEEDED with no DB change**, add a + temporary log of `clip.samples.size` in `FingerprintUpdater.process` for exactly this case -- + the one thing still not directly observed is whether `decodeHead`/`decodeTail` returned null or + an empty/garbage clip for that specific file. ## Experiments, in order (updated) -- cgit v1.2.3