diff options
Diffstat (limited to 'docs/detection-findings.md')
| -rw-r--r-- | docs/detection-findings.md | 107 |
1 files changed, 107 insertions, 0 deletions
diff --git a/docs/detection-findings.md b/docs/detection-findings.md index f3eb30f..9898dfd 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -838,3 +838,110 @@ Peter's manual-skip logging idea is the obvious candidate, since it supplies lab detector nor the feed can. Had only LeVar been measured, the floor would have been lowered. + +## 2026-11-02: what the corpus says now, and the tier that is being acted on by accident + +The previous entry ended by naming what was missing: "a mechanism that can tell a real short ad +from a short coincidence". Words are that mechanism, and three of them were built and compared. +This entry records what the **live corpus** says, re-derived from `state.json` rather than quoted, +because the figure in `publish.go`'s own comment turned out to describe a subset. + +### The corpus, measured 2026-11-02 + +| | | +|---|---| +| feeds subscribed / shows tracked | 44 / 43 | +| episodes known from feeds | 9,020 | +| mirrored (audio on disk) | 336 | +| with words (whisper or published) | 327 | +| with a master to subtract against | 91 | +| episodes publishing audio-signature breaks | 267 | +| episodes publishing word breaks | 243 | +| word break spans published | 597 `cue`, 49 `ad` | +| listener labels recorded | 26 | + +### Subtraction's accuracy depends entirely on where the master came from + +`publish.go` says subtraction "measured 92% to 104% ... across eighteen episodes". That is still +true of the episodes it was measured on, and it is **not** true of the corpus. Across the 24 +episodes that now publish an `ad` span and have a length gap to check it against, recovery runs +from 20% to 146%, median 76%. + +Split by what the master actually is, the spread stops being a spread: + +| master source | episodes | median recovery | range | +|---|---|---|---| +| publisher transcript | 9 | **102%** | 94% – 104% | +| refetch (a second download of the same file) | 15 | **33%** | 20% – 146% | + +(Heavyweight's seven carry an empty `masterSource`: the master file predates the field, and +`Reclaim` faithfully restored the blank. They are publisher transcripts.) + +A publisher's transcript is a second *description* of the programme, written once, and +subtracting it works -- 94% to 104%, every pod found, edges pulled back off the speech. A refetch +is a second *copy of the same dynamically-assembled file*, carrying its own ads at its own +positions, so the diff is comparing two ad loads rather than finding one. It recovers a third of +the advertising, which matches the 21% and 0% measured on the refetch detector when it was first +tried and rejected on exactly those grounds. + +### The tier that was rejected is being published anyway + +Nothing in `Publishable` looks at `MasterSource`. It checks the candidate set against the length +gap and nothing else, so a refetch-derived candidate publishes on the same terms as a publisher- +derived one -- and refetch masters are now **15 of the 24** episodes publishing an `ad` break, +the majority of a tier the design had decided not to act on. + +The 33% is a real measurement, not an artefact: the refetch shows it rests on have large, +consistent length gaps (LeVar Burton Reads 3.39m median across 11 episodes, Darknet Diaries 3.31m, +Sources & Methods 3.72m, Serial 3.22m), so there is a solid yardstick to miss two-thirds of. + +### The length gap is only a guard where it is large and consistent + +The second finding, and the one nearly recorded wrongly. Two Reply All episodes publish `ad` spans +claiming 146% and 138% of their length gap, just under `overshootLimit`: + +``` +#188 Into the Depths declared 42.6m served 43.6m gap 1.01m + published: one ad span, 1154.2s -> 1243.0s (89s) = 146% of the gap +``` + +The obvious reading is that the detector over-claimed by 28 seconds and will skip programme. That +reading does not survive looking at the show rather than the episode. **Reply All's length gaps +across its eight mirrored episodes run from -0.48 to +1.02 minutes** -- they straddle zero. The +feed's declared durations are not a baseline for this show at all; the "61 seconds of inserted +advertising" that 146% is a percentage *of* is indistinguishable from encoder and feed noise. + +So the number being exceeded is itself noise, and which of the two is wrong on #188 cannot be +settled from this data. What can be settled is the structural point: `HasInsertedAdvertising` asks +only whether *this episode's* gap clears one ad unit, and 60.6 seconds clears 15. It has no notion +of whether the show's gaps are consistent enough for the figure to mean anything. Compare the +shows where it does mean something -- Incels 11.57-14.54m across 8 episodes, Heavyweight +0-10.06m, The Secret World of Roald Dahl 4.83-6.16m -- with the shows whose gaps sit in the +noise: Reply All -0.48 to 1.02, Jon Stewart -1.48 to 0.52, Sleepy History -1.57 to 0. +A per-episode threshold cannot tell those apart, and the per-show spread is sitting right there in +the state file, uncomputed. + +### Do not + +- **Do not quote `publish.go`'s 92-104% as the detector's accuracy.** It is the publisher-master + figure. Say which master. +- **Do not raise `overshootLimit` to admit more.** On the shows where the gap is meaningful it has + never been the binding constraint; on the shows where it binds, the gap is noise. +- **Do not read a single episode's gap as ground truth without checking the show's spread.** That + mistake was made and caught in the course of writing this entry: the 146% was about to be + recorded as 28 seconds of programme being skipped. +- **Do not rebuild the ad/furniture classifier.** Measured and deleted: 33% agreement with the + length gap, 263 advertisements called furniture, and the markers it keyed on appear in 5% of ads + and 4% of furniture -- it was reading noise. Recurring text publishes as `cue` instead, which + claims only that the audio repeats across the show's episodes, which is what was actually + measured. + +### The open questions, in order of how answerable they are + +1. **Should a refetch master publish `ad` at all?** The data says the tiers are not comparable + (102% vs 33%) and the code treats them as if they were. Answerable now; it changes what gets + skipped on fifteen episodes. +2. **Should the length-gap guard consider the show's spread rather than the episode's gap?** The + figures above are computable from `state.json` for every show. A show whose gaps straddle zero + has no usable yardstick and arguably should not publish a subtraction-derived `ad` either. +3. Whether #188's 89-second span is actually advertising. Needs a listen, not a measurement. |