listen.git/docs/detection-findings.md, branch main

Keep what the listener asked for, and publish ads only from a publisher's master

2026-11-03T08:22:11+00:00

Two changes to the server, both measured against the live data rather than reasoned about. **Subtraction publishes only from a publisher's transcript.** Subtracting a publisher's transcript recovers 94-104% of the advertising the file lengths imply; subtracting a refetch -- a second download of the same assembled file, carrying its own ads in its own places -- recovers a median of 33%. That tier had already been tried and rejected at 21% and 0%, but nothing in the publish path was told which master it had, so it reached the phone anyway on seventeen ad spans. Written as a denylist first, and measuring it against a copy of the live data showed why that was the wrong shape: sixteen episodes carry a master whose label is blank -- the field postdates the first masters written, and Reclaim restored those files after the library was destroyed with the blank intact -- and they published on the strength of not being the one named bad kind. They happened to be publisher transcripts, so the answer was right by luck. It is an allowlist now, and the blank labels are repaired from the master files, which have said "published" all along. Measured on a copy, not predicted: before 49 ad spans (17 refetch, 24 unlabelled, 8 published) + 604 cue, 245 episodes after 62 ad spans (all publisher-master) + 611 cue, 245 episodes No episode loses its skips; the refetch ones trade advertising nobody could measure for the show's own recurring audio, which is a preference rather than an unconditional skip. Measuring also caught the recurring pass blanking the refusal subtraction had just written -- both passes answer "why is nothing published" through one field and recurring runs second, so all thirty-two episodes that now carry a reason were losing it a moment later. **Being asked for keeps an episode mirrored.** Retention was publication date alone, and pruning calls words.Forget, which deletes the transcript *file*. So an episode outside the newest eight was fetched on demand, recognised, served, and deleted on the next poll -- and asking again paid the whole cost over. From the server's own journal over twenty-one days: 397 transcription runs covering 309 distinct episodes, with 7.7 of the 33.4 hours of whisper time spent on episodes it had already recognised and thrown away. 23% of the budget, re-doing work it had discarded. Working through a back catalogue -- the archive tier, the slow-drip shows, the listening this was built for -- therefore paid full recognition for every episode and again to resume one. The window still decides what is worth fetching; RequestedAtUnix now decides what is worth keeping, stamped on every /api/episode call including for episodes already mirrored, because resuming something played last month must not let it fall out mid-listen. Kept for 28 days after the last ask, the same span refetchAfterSeconds uses, being the same kind of judgement about how long something stays interesting. Disk points the same way: transcripts are 36MB for 445 files, audio is 17GB for 336 episodes, and the volume has 365GB free. The expensive artefact is the small one. 374 server tests pass. Not yet deployed; the phone's half of this -- a pending reply is retried only on a feed refresh -- is still to come. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Write down what the corpus actually says, and correct three stale claims

2026-11-02T07:51:10+00:00

The findings doc's last entry was 2026-10-19 and ended by naming what was missing: "a mechanism that can tell a real short ad from a short coincidence". Words are that mechanism, it was built, and thirty-two server commits later the doc never recorded what it measured. A future session would have re-attempted the ad/furniture classifier that was already measured and deleted. The new entry is derived from the live state file rather than quoted, which is how it caught that publish.go's own figure describes a subset. "92% to 104% across eighteen episodes" is the publisher-transcript number. Across the 24 episodes now publishing an ad span, recovery runs 20% to 146%. Split by where the master came from, the spread stops being a spread: a publisher's transcript gives 94-104%, a refetch gives a median of 33%. Nothing in Publishable looks at which it has, and refetch masters are now fifteen of those twenty-four -- the majority of a tier the design had decided not to act on. It also caught me getting a conclusion wrong. Two Reply All episodes publish ad spans at 146% and 138% of their length gap, which reads as the detector over-claiming by half a minute of programme. Looking at the show instead of the episode: Reply All's gaps run from -0.48 to +1.02 minutes across eight episodes. They straddle zero, so the feed's declared durations are not a baseline for that show and the thing being exceeded is noise. The structural finding is the useful one -- HasInsertedAdvertising asks only whether this episode's gap clears one ad unit, and has no notion of whether the show's gaps are consistent enough to mean anything. Both versions are in the entry, because the wrong one is the one a reader would reach for next time. Nothing here changes behaviour. Whether a refetch should publish an ad at all is Peter's call and is written up as such. CLAUDE.md, separately: - It still said the episode description shows about two lines. That was changed to ten yesterday, for a reason that is recorded: the limit existed to stop long notes pushing the transport off-screen, and the merge had already put the notes below the transport. Left as it was, the doc was an instruction to reintroduce the bug. - "207 unit tests" has been wrong for most of the project's life; the suite is past nine hundred. Replaced with a note to ask the wrappers, since any number written here goes stale the same way. - The TDD rule banned backfilling tests onto existing code without qualification, which also forbids the one case where it is right -- pinning behaviour before moving it, so a refactor is provably behaviour-preserving. Narrowed to say that, and to say the order matters. The sixteen formatter tests in 2cc252a were written that way. - Added the rule those formatter files were moved to satisfy: anything the app says out loud lives in a named file of its own and is tested, never inside the composable that renders it. It was already followed by eight files and stated only in WaitingForServerText's own comment. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Record that lowering the noise floor to 8s is wrong

2026-10-19T09:15:03+00:00

The 12s floor cut real detection on shows with verified ads -- LeVar Burton Reads fell from 19.0 to 10.5 minutes found against a 26.6 minute gap -- so 8s looked like the fix. Measured on an isolated copy of the live data it is not: Sleepy History at 8s finds 174 minutes of "advertising" in a show that has 48 seconds of it, worse than the 4s floor that produced Peter's original complaint. MinRunSamples and MergeGapMs interact and cannot be tuned separately. A low floor lets noise fragments survive and the 60s bridging then welds them into enormous spans. Had only LeVar been measured the floor would have been lowered. Also re-pins .coverage-baseline, which the hook left modified after d96c876 deleted covered readout code. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Bridge ad pods fragmented by brief lapses in agreement

2026-10-08T04:02:39+00:00

A pod of several advertisements is found as several matches, because agreement lapses between them. Published unbridged, 45% of the gaps between consecutive breaks came in under ten seconds and 152 of 262 multi-break episodes contained one -- which plays as a skip, a one-to- nine-second burst of advertisement, then another skip. Bridging also moves agreement with the declared length gap from 69% to 73% median, because what was bridged was advertising too. MaxBreakMs now applies to the matches rather than to what bridging makes of them. Applied after bridging it refused five episodes that bridging had pushed over the four-minute cap, Radiolab's Grass Apes among them -- a 3:55 pre-roll and an 0:18 tag two seconds apart fuse to 4:15, and the episode the whole chain was device-verified on would have published nothing. One enormous match is evidence of shared programme; a chain of ordinary matches a beat apart is evidence of an ad pod. MaxSkipFraction still guards the episode as a whole. Also pins the ad scan's production values, which were measured rather than picked, and records what probing Peter's theme-music observation turned up: the cue is real and already detected, it marks a break's entry but not its extent, a second fetch is not an ad-free reference, and 138 of 322 mirrored episodes contain no inserted advertising at all while the scan reports breaks on 111 of them. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Examine each batch once instead of every poll

2026-10-07T09:14:11+00:00

The first full pass over all 42 feeds finished, then the server kept re-deriving the same answers: Radiolab reported "53:29 of 59:39" four times in one day. pollOne called detect unconditionally and the mirror skips files it already has, so nothing had changed between passes. At ~6.5 minutes a show that is 4.5 hours a cycle against a two-hour poll interval -- the machine never finished a cycle and never idled, burning 13h42m of CPU in 11h of wall clock, roughly 60% of it on the intro search whose results are deliberately not published. Show.DetectedBatch records the episode ids a completed pass examined. The batch is the unit rather than the episode because every offset and every break comes from comparing an episode against the others fetched alongside it, so one arrival or one pruning legitimately changes all of them. Order-independent, since feeds are not reliably sorted. Recorded only on completion, so a cancelled or failed pass is retried. Episode.DetectedAtUnix could not serve as the gate: it is only set for episodes appearing in result.Offsets, so an episode nothing was found in never got a timestamp and the gate would never close. Also records what the first full corpus showed, including a disproven hypothesis worth not retrying -- filtering breaks by partner count or run length looks obviously right and makes agreement with the declared length gap substantially worse -- and a correction: Sleepy History's 117 minutes of "advertising" was a show-level sum sitting almost entirely inside the two episodes its own guard already refused. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Follow the data: detect the advertising, not the intro

2026-10-06T07:31:53+00:00

The property that broke intro detection is the property that makes this work. Eight episodes mirrored within one minute are maximally dependent, because the ad server inserts whatever campaign is running into all of them identically. For intro detection that destroys the independence the confirmation threshold needs -- which is how it came to confirm an advertisement on 8 of 8 episodes. For ad detection it is the signal. And unlike intro detection it is checkable against a number the detector does not produce: the feed declares the ad-free length, the file served is longer, and the difference is the advertising. **Measured on the live deployment, Radiolab's eight mirrored episodes: the scan accounted for 53:29 of the 59:39 the length gap implies, with every individual episode between 71% and 105%.** That is the first time detection in this project has been checked against external ground truth and agreed with it. The old sliding search could never have done this -- one full-length correlation per window position, 199 seconds for a 240-second clip. BestLags computes the cross-correlation over every lag in a single FFT pair, which says where two recordings align; MatchedRuns then walks the extent of the agreement directly, window by window, with no transform. Sized against measurements rather than guesses: 2^22 is 1.69s, so a 70-minute episode decimated to 250Hz is about five seconds a pair. Silence is explicitly not agreement, or separate advertisements weld together across the pauses between them. Breaks publish as ranges into episode_midrolls, which MidrollSkip and PlayheadPolicy already read -- no playback changes and no migration. Intro and outro offsets are still computed and logged but no longer published, and the ad-derived ones already in the server's state were cleared. Two guards publish nothing rather than something wrong: an episode whose matches cover over 35% of it, and any single match over four minutes. A re-aired episode or a two-parter would otherwise be eaten. Both record the refusal, because "found nothing" and "refused what it found" are different states and conflating them is the readout mistake over again. Also three data-loss bugs, found by reading the journal after a restart rather than by any test. The subscription list -- the one thing the server cannot rediscover -- was persisted only when a whole poll cycle finished, which for 61 feeds is hours. A cycle re-applied its stale snapshot at the end, which would undo a push that arrived during it and then delete the audio of every show the snapshot omitted. And detection ignored cancellation, so systemd's 90-second stop SIGKILLed the process before the save-on-exit could run. All three fixed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Move skip detection to a server, and find that it detects ads

2026-10-05T09:54:59+00:00

The server is Peter's design and it works: listend polls the phone's feed list, mirrors the newest 8 episodes per show, detects across all of them at once, and serves the offsets *and the audio they belong to*. Deployed on tungsten behind Apache at https://listen.terst.org. FingerprintWorker and MidrollWorker are deleted -- the battery drain is gone because the work is. Serving the audio is load-bearing. Dynamic ad insertion turned out to be on every episode: Grass Apes declares 22:05 and the server was served 28:18. An offset measured on one copy is minutes wrong for another, so the phone has to play the bytes the detector examined. The offsets land in the same columns the on-device detector wrote, so playback needed no changes and there is no migration. **The headline finding is a failure, and it is proved rather than suspected.** Detection reported the intro confirmed on 8 of 8 Radiolab episodes and passed its self-check. It had locked onto an advertisement. The chosen segment was searched for in an ad-free copy of the same episode fetched hours earlier: r = 0.0621. It is in the ad break, not the programme. The confirmation threshold assumes three *independently fetched* episodes. Mirroring 8 within one minute from the same ad server makes them maximally dependent -- whatever campaign is running is in all of them, identical -- and SelfCheck cannot catch it, because the segments genuinely are the same audio. Moving to a server fixed the cost problem and broke the independence the safety mechanism rests on. The published offsets are head-window matches, so they happen to skip pre-roll advertising, which is useful by accident and will move whenever the campaign rotates. It is not intro detection and is not described as such anywhere. The fix, not made: record when each episode was fetched and require the agreeing episodes to span more than one fetch occasion. Reasoning, and an argument for detecting the ads directly instead, in docs/detection-findings.md. Also recorded there: two tooling traps that each cost real time. `android-ui tap` can land on the soft keyboard, because the dump reports layout coordinates the IME does not move -- a Save button and a text field both appeared tapped and neither fired. And the server's token is alphanumeric now because `adb shell input text` ate the leading "-" of a base64url one and produced a silent 401 that looked like broken wiring. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Find why Tier 3 never confirmed, and port detection to Go

2026-10-05T08:57:59+00:00

Peter's report: the phone runs hot, uses a lot of battery, playback takes a moment to start, and intelligent skipping has never done anything. **The hot phone and the dead detection were one bug.** The candidate search correlated 117 window positions against the whole of the other clip, re-transforming that unchanged clip every time -- 199 seconds per episode end, measured on a desktop, so minutes more on a phone. FingerprintWorker is a plain WorkManager worker, Android stops one at ten minutes, and nothing was persisted until a comparison returned. So it was killed mid-flight, saved nothing, and was re-enqueued to start over, forever. IntroOutroDetectorScaleTest is the tripwire whose absence allowed this: the unit tests ran on 300-sample clips while production ran on 1,920,000. Fixed coarse-to-fine, plus AudioCorrelation.prepareTarget so the target clip is transformed once per search rather than 117 times. 10.6x, and a test asserts the chosen reference window is byte-identical to what a full-resolution search picks. **Midroll detection could never have run at all.** decodeFull buffered a whole episode at its native rate -- 434MB for a 13-minute episode, no largeHeap -- to produce 6MB of 8kHz mono. It threw OutOfMemoryError into a swallowing runCatching and the episode was then marked checked forever. MonoResampler now streams the decode so the peak allocation is the output, and maxDecodableMs skips a longer episode before spending anything and without marking it checked. **Detection is moving to a server**, Peter's call, in Go to match podkit and doot. server/ holds the ported core, verified three ways because this is the second port of this maths and the first shipped a normalization bug: numpy's literals extracted mechanically from AudioCorrelationTest, the FFT against a direct DFT, and the whole correlation path against a brute-force Pearson. On four real Radiolab episodes it confirms the intro on 4 of 4, at 48.0s, 8.65s, 8.65s and 65.15s -- the varying pre-roll, observed in working code at last. skips.SelfCheck verifies located segments against *each other* rather than against the reference, which is a different claim: correlation is not transitive, so two segments can each clear 0.6 against a reference and be unrelated. It rejected the outro on its first real run (r=0.746), withholding offsets that would have ended three episodes early. Offsets are published only when they verify. Also: the episode screen gets a working scrubber with ad-break marks and a declared-versus-measured length readout, queue rows open the episode instead of doing nothing, PrepareTrace logs playback-start timing against episode length for the still-open pause question, and -PphoneOnly drops the emulator's CPython from a sideload build (114MB -> 99MB). Full history and numbers in docs/detection-findings.md. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

Protect a show's Tier 3 pending episode from retention reclaim

2026-09-29T20:41:33+00:00

Peter's call (option 1 of the three in docs/detection-findings.md): exempt an episode retention would otherwise reclaim while it is a show's pending comparison basis, rather than extending the grace period or accepting the gap. RetentionSweeper gains a pendingEpisodeIds seam (suspend () -> Set, defaulting to empty) and filters its reclaim candidates against it before the grace-period policy ever sees them -- every existing caller and test was unaffected by the default. RetentionWorker wires the real thing: the union of two new, narrow DAO queries -- ShowFingerprintDao.getAllPendingEpisodeIds() (intro/outro) and MidrollDao.getAllMidrollPendingEpisodeIds() (midroll, reading SubscriptionEntity.midrollPendingEpisodeId). Applied to both fingerprinting mechanisms, not just the one that got diagnosed: MidrollUpdater's discovery has the identical rolling-pending- episode shape and would race retention the same way for a fast show, just silently, with no frozen row to reveal it. Not yet re-verified on a device -- the next live test is two Up First downloads close enough together to actually compare, confirming pendingEpisodeId advances to a real second episode instead of restarting. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA

Find the actual root cause: retention races the pending-episode file

2026-09-29T20:30:48+00:00

Triggered a fresh Up First download on the phone and watched androidx.work.workdb and show_fingerprints immediately after, live. A new FingerprintWorker job appeared and succeeded within seconds, and show_fingerprints updated for the first time since 2026-09-27 -- but instead of advancing toward a second comparison, it just replaced the stale pending episode with the new download. confirmedCount stayed 0. That's checkAgainstPending's own designed fallback when pendingClipOf can't re-decode the stored pending episode because its file is gone. And it is gone: RetentionPolicy's grace period is one day, and Up First is a fast, daily, ~15-30 minute news show that gets played through quickly. By the time a second episode is downloaded and processed, retention has very often already reclaimed the first one's file -- so the detector never gets two episodes' audio to compare at once, and keeps discarding its only candidate and restarting from whatever the newest download is. Cruising the Waves reaching a real Tentative candidate the same day is consistent with this: a show whose downloads happen to land further apart, or whose previous pending episode hadn't been played+reclaimed yet, doesn't hit the race. This was never a signal-processing problem -- DEFAULT_REFERENCE_SECONDS, MIN_SCORE and CONFIRM_THRESHOLD were the wrong place to look the whole time. Wrote up three options (exempt a pending episode from reclaim, extend the grace period for a show still building its first candidate, or accept the gap for fast shows) as a decision for Peter, not something to just change. Separately unresolved and now secondary: Up First's actual second download (2026-09-28) left zero trace in show_fingerprints, which the live test's fallback path would have if it had run at all -- most likely a deferred or dropped one-shot WorkManager job, not reproduced live this session. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA