From 7847e13c3fab23edfda096396e4dbe01df3471c3 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Tue, 6 Oct 2026 07:31:53 +0000 Subject: Follow the data: detect the advertising, not the intro The property that broke intro detection is the property that makes this work. Eight episodes mirrored within one minute are maximally dependent, because the ad server inserts whatever campaign is running into all of them identically. For intro detection that destroys the independence the confirmation threshold needs -- which is how it came to confirm an advertisement on 8 of 8 episodes. For ad detection it is the signal. And unlike intro detection it is checkable against a number the detector does not produce: the feed declares the ad-free length, the file served is longer, and the difference is the advertising. **Measured on the live deployment, Radiolab's eight mirrored episodes: the scan accounted for 53:29 of the 59:39 the length gap implies, with every individual episode between 71% and 105%.** That is the first time detection in this project has been checked against external ground truth and agreed with it. The old sliding search could never have done this -- one full-length correlation per window position, 199 seconds for a 240-second clip. BestLags computes the cross-correlation over every lag in a single FFT pair, which says where two recordings align; MatchedRuns then walks the extent of the agreement directly, window by window, with no transform. Sized against measurements rather than guesses: 2^22 is 1.69s, so a 70-minute episode decimated to 250Hz is about five seconds a pair. Silence is explicitly not agreement, or separate advertisements weld together across the pauses between them. Breaks publish as ranges into episode_midrolls, which MidrollSkip and PlayheadPolicy already read -- no playback changes and no migration. Intro and outro offsets are still computed and logged but no longer published, and the ad-derived ones already in the server's state were cleared. Two guards publish nothing rather than something wrong: an episode whose matches cover over 35% of it, and any single match over four minutes. A re-aired episode or a two-parter would otherwise be eaten. Both record the refusal, because "found nothing" and "refused what it found" are different states and conflating them is the readout mistake over again. Also three data-loss bugs, found by reading the journal after a restart rather than by any test. The subscription list -- the one thing the server cannot rediscover -- was persisted only when a whole poll cycle finished, which for 61 feeds is hours. A cycle re-applied its stale snapshot at the end, which would undo a push that arrived during it and then delete the audio of every show the snapshot omitted. And detection ignored cancellation, so systemd's 90-second stop SIGKILLed the process before the save-on-exit could run. All three fixed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- docs/detection-findings.md | 99 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) (limited to 'docs') diff --git a/docs/detection-findings.md b/docs/detection-findings.md index 26b31b3..35c4525 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -569,3 +569,102 @@ detects, with very high confidence, is **the inserted advertising itself** -- be simultaneous fetches share it exactly. Skipping ads is the actual goal, and the declared-versus- served gap is a free, independent check on whether a scan has accounted for all of it. Detecting ads directly may be a better target than detecting intros and hoping ads do not get in the way. + +--- + +## 2026-10-06: following the data -- detecting the advertising instead + +Peter's call, on the evidence of the previous section: stop trying to detect intros and detect +what the server demonstrably finds. + +### Why this is the right way round + +The property that broke intro detection is the property that makes ad detection work. Eight +episodes mirrored within one minute are maximally dependent, because the ad server inserts +whatever campaign is running into all of them identically. For intro detection that destroys the +independence the confirmation threshold needs. For ad detection it *is* the signal. + +And unlike intro detection, this can be checked against a number the detector did not produce. +The feed declares the ad-free length, the file served is longer, and the difference is the +advertising that was inserted. + +### The algorithm, and why the old one could not do this + +The sliding-window search cost one full-length correlation per window position -- 199 seconds for +a 240-second clip, measured. Over a 70-minute episode it is hopeless. + +`detect.BestLags` instead computes the cross-correlation over *every* lag in one FFT pair, which +says where two recordings align. `detect.MatchedRuns` then walks the extent of the agreement at +that alignment directly, window by window, with no transform at all. Sized against measurements +rather than guesses (`fft_bench_test.go`: 2^22 is 1.69s), a 70-minute episode decimated to 250Hz +is three transforms of 2^22 -- about five seconds a pair, so a batch of eight is minutes. + +Silence is explicitly not agreement. Two recordings agree perfectly on it, and counting it would +weld separate advertisements together across the pauses between them. + +### Measured on the real deployment + +Radiolab, eight mirrored episodes: + +``` +Radiolab: ads on 8/8 episode(s) (5 refused as too extensive to be advertising), + accounted 53:29 of 59:39 inserted, took 2m56s +``` + +Per episode, found against what the length gap implies: + +| episode | found | inserted | ratio | +| --- | --- | --- | --- | +| Grass Apes | 6:31 | 6:13 | 105% | +| The Sweetest Thing | 6:31 | 6:13 | 105% | +| 23 Weeks 6 Days | 6:40 | 8:59 | 74% | +| Dolly Parton's America | 6:11 | 6:58 | 89% | +| Nigel and the Real Gannet | 6:54 | 7:07 | 97% | +| The Shoe Thrower | 6:47 | 9:33 | 71% | +| The Holy Shiver | 6:26 | 6:59 | 92% | +| This is Your Brain on Hormones | 7:26 | 7:33 | 98% | + +**Every episode lands between 71% and 105% of an independent measure, and the batch total is +90%.** That is the first time in this project that detection has been checked against external +ground truth and agreed with it. + +### What is published, and what is not + +Ad breaks are published, as ranges, into `episode_midrolls` -- which `MidrollSkip` and +`PlayheadPolicy` have read since the on-device detector existed. Playback needed no changes and +there is no migration. + +**Intro and outro offsets are no longer published.** They are still computed and still logged, +because they will become trustworthy once the agreeing episodes are required to span more than +one fetch occasion. Until then they are known to mis-identify, and the advertising they kept +finding is better served by the scan that measures itself against the length gap. The previously +published ad-derived offsets were cleared from the server's state. + +Two guards decide nothing gets published rather than something wrong: + +- **An episode whose matches cover more than 35% of it publishes nothing**, because two episodes + of a show can legitimately share a great deal -- a re-aired episode, a two-parter with a long + recap -- and skipping that would eat the programme. +- **A single match longer than four minutes is dropped**, being longer than any plausible ad pod. + +Both record a refusal rather than silently leaving an episode empty. An episode that published +nothing because its matches were implausible is a different state from one where nothing +matched, and conflating those two is exactly the mistake the app's skip readout already made. + +### Three data-loss bugs found while deploying this, all fixed + +Found by reading the journal after a restart, not by a test: + +1. **The subscription list lived only in memory.** `SetFeeds` from the API did not persist, and + state was saved only when a whole poll cycle finished -- which for 61 feeds is hours. A + restart in between lost every subscription the phone had sent, and that is the one thing the + server cannot rediscover. Now saved the moment it arrives. +2. **A poll cycle re-applied a stale subscription snapshot at the end**, which would silently + undo a push that arrived during the cycle and then delete the audio of every show the stale + list omitted. Now reconciled against the list as it stands. +3. **Shutdown could not complete.** Detection ignored cancellation, so systemd's default + 90-second stop would SIGKILL the process mid-scan and the save-on-exit never ran. Detection + now checks for cancellation and the unit allows ten minutes to stop. + +State is also saved after every feed now, so a restart costs one feed's progress rather than a +whole cycle's. -- cgit v1.2.3