From 9ba30c95c6bc9a6405ab21f616e933b09fa0358f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 09:14:11 +0000 Subject: Examine each batch once instead of every poll The first full pass over all 42 feeds finished, then the server kept re-deriving the same answers: Radiolab reported "53:29 of 59:39" four times in one day. pollOne called detect unconditionally and the mirror skips files it already has, so nothing had changed between passes. At ~6.5 minutes a show that is 4.5 hours a cycle against a two-hour poll interval -- the machine never finished a cycle and never idled, burning 13h42m of CPU in 11h of wall clock, roughly 60% of it on the intro search whose results are deliberately not published. Show.DetectedBatch records the episode ids a completed pass examined. The batch is the unit rather than the episode because every offset and every break comes from comparing an episode against the others fetched alongside it, so one arrival or one pruning legitimately changes all of them. Order-independent, since feeds are not reliably sorted. Recorded only on completion, so a cancelled or failed pass is retried. Episode.DetectedAtUnix could not serve as the gate: it is only set for episodes appearing in result.Offsets, so an episode nothing was found in never got a timestamp and the gate would never close. Also records what the first full corpus showed, including a disproven hypothesis worth not retrying -- filtering breaks by partner count or run length looks obviously right and makes agreement with the declared length gap substantially worse -- and a correction: Sleepy History's 117 minutes of "advertising" was a show-level sum sitting almost entirely inside the two episodes its own guard already refused. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- docs/detection-findings.md | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) (limited to 'docs') diff --git a/docs/detection-findings.md b/docs/detection-findings.md index 35c4525..c821062 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -668,3 +668,86 @@ Found by reading the journal after a restart, not by a test: State is also saved after every feed now, so a restart costs one feed's progress rather than a whole cycle's. + +## Detection re-ran forever, and what the first full corpus then showed + +The first complete pass over Peter's 42 feeds finished on 2026-10-06: 322 episodes mirrored, +286 publishing ad breaks. It then kept going over the same unchanged files, producing results +identical to the digit -- Radiolab reported "53:29 of 59:39" four separate times in one day. + +`pollOne` called `detect` unconditionally, and the mirror skips files it already has, so nothing +had changed between passes. One cycle is ~6.5 minutes a show, 4.5 hours for 42 shows, against a +two-hour poll interval: the machine never finished a cycle and never idled -- 13h42m of CPU in +11h of wall clock. Roughly 60% of that was the intro/outro search whose results are deliberately +not published. + +`Episode.DetectedAtUnix` existed but could not be used as the gate: it is only set for episodes +that appear in `result.Offsets`, so an episode nothing was found in never got a timestamp and +the gate would never close. The unit is the batch anyway -- every offset and every break is +found by comparing an episode against the others fetched alongside it, so one arrival or one +pruning legitimately changes all of them. `Show.DetectedBatch` records the set of episode ids +the last completed pass examined, order-independent because feeds are not reliably sorted. +Recorded only after a pass completes, so a cancelled or failed one is retried. + +Deployed 2026-10-07 with the field backfilled for the 42 shows already examined, which saved an +immediate 4.5-hour redundant cycle. Verified in production: 31 feeds polled in five minutes with +exactly one detection running -- The Ezra Klein Show, which had genuinely gained an episode. + +### Filtering by partner count or run length is wrong, and the length gap proves it + +The obvious cleanup is to drop weakly-supported breaks. 635 of 1574 published breaks (40%) have +`partners == 1`, with a median length of 5.0s against 19.0s for `partners >= 4` -- they look +exactly like correlation noise, and 270 of them sit at `MinRunSamples`, the 4-second floor. + +Measured against the declared-versus-served length gap -- the one figure the detector does not +produce -- every such filter makes agreement **worse**: + +| filter | median found/inserted | within 75-125% | skipping on shows with no gap | +|---|---|---|---| +| none (current) | 67% | 31% | 164 min | +| `partners >= 2` | 43% | 17% | 141 min | +| `partners >= 3` | 39% | 13% | 127 min | +| length >= 8s | 58% | 24% | 141 min | +| both | 41% | 13% | 130 min | + +Detection already **under**-reports -- a 67% median means a third of the advertising is missed -- +so removing breaks moves away from the truth, and it barely dents the unverifiable skipping it +was meant to clean up. The single-partner short runs are mostly real advertising. Hypothesis +disproven; the thresholds stay as they are. + +### What is worth fixing is fragmentation + +45% of the gaps between consecutive breaks are under ten seconds, and **152 of 262** multi-break +episodes contain at least one such gap -- which plays as a skip, one to nine seconds of audible +advertisement, then another skip. `merge` only folds breaks that overlap or touch. + +Merging across a tolerance both cleans that up and moves agreement the right way, because the +bridged gap was advertising too: + +| merge tolerance | median found/inserted | within 75-125% | total breaks | +|---|---|---|---| +| 0s (current) | 69% | 32% | 1240 | +| 2s | 70% | 33% | 973 | +| 5s | 70% | 34% | 885 | +| 10s | 73% | 34% | 820 | +| 15s | 73% | 35% | 786 | + +### Sleepy History was not eating the programme + +Reported on 2026-10-06 as finding 117:42 of "advertising" in a show whose feed declares no +inserted ads at all. That reading was wrong: it was a show-level sum, and almost all of it sat +inside the two episodes the 35% guard **already refused** -- a 3h08m Rosa Parks compilation that +literally contains other episodes (53:41 matched) and Antarctica (22:30). + +The six episodes that do publish skip 0:34 to 4:33 each, under 10% of their runtime, and the +largest block in each is a genuine closing segment shared by every episode in the batch -- +`48:20-51:36 partners=22`, `46:19-49:19 partners=26`. That is the show's own outro, which is what +intro/outro detection was supposed to find in the first place. The rest is 4-second floor noise. + +Across the whole corpus only 4 of 279 published episodes skip more than 20% of their runtime, and +all but one are corroborated by the length gap to within a minute. The exception is The Trip +(3:54 of 14:51, no declared gap), which is the one show worth watching. + +Sleepy History also serves files consistently **shorter** than declared -- 50:59 declared against +49:23 served, about 1:36 short on every episode -- so `Accounting` reports a zero gap for it. A +feed can be wrong in that direction too, and when it is there is no external check available. -- cgit v1.2.3