From 9e1dfe2989dfce61bb96007cf52faf519cd7b654 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 04:02:39 +0000 Subject: Bridge ad pods fragmented by brief lapses in agreement A pod of several advertisements is found as several matches, because agreement lapses between them. Published unbridged, 45% of the gaps between consecutive breaks came in under ten seconds and 152 of 262 multi-break episodes contained one -- which plays as a skip, a one-to- nine-second burst of advertisement, then another skip. Bridging also moves agreement with the declared length gap from 69% to 73% median, because what was bridged was advertising too. MaxBreakMs now applies to the matches rather than to what bridging makes of them. Applied after bridging it refused five episodes that bridging had pushed over the four-minute cap, Radiolab's Grass Apes among them -- a 3:55 pre-roll and an 0:18 tag two seconds apart fuse to 4:15, and the episode the whole chain was device-verified on would have published nothing. One enormous match is evidence of shared programme; a chain of ordinary matches a beat apart is evidence of an ad pod. MaxSkipFraction still guards the episode as a whole. Also pins the ad scan's production values, which were measured rather than picked, and records what probing Peter's theme-music observation turned up: the cue is real and already detected, it marks a break's entry but not its extent, a second fetch is not an ad-free reference, and 138 of 322 mirrored episodes contain no inserted advertising at all while the scan reports breaks on 111 of them. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- docs/detection-findings.md | 48 ++++++++++++++++ server/internal/skips/ads.go | 61 +++++++++++++------- server/internal/skips/ads_test.go | 112 ++++++++++++++++++++++++++++++++++++ server/internal/skips/skips_test.go | 23 ++++++++ 4 files changed, 224 insertions(+), 20 deletions(-) diff --git a/docs/detection-findings.md b/docs/detection-findings.md index c821062..6f48b72 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -751,3 +751,51 @@ all but one are corroborated by the length gap to within a minute. The exception Sleepy History also serves files consistently **shorter** than declared -- 50:59 declared against 49:23 served, about 1:36 short on every episode -- so `Accounting` reports a zero gap for it. A feed can be wrong in that direction too, and when it is there is no external check available. + +## Bridging fragmented pods, and the regression it nearly caused + +Implemented 2026-10-08 with `AdConfig.MergeGapMs = 10s`. + +The guard it nearly broke is worth recording. `MaxBreakMs` refused any break longer than four +minutes, applied *after* merging. Bridging makes breaks longer, so five episodes would have +crossed the cap and published nothing -- including Radiolab's Grass Apes, the episode the whole +chain was device-verified on, whose 3:55 pre-roll and 0:18 tag sit two seconds apart and fuse to +4:15. + +The cap now applies to the matches rather than to what bridging makes of them. One enormous match +is evidence of shared programme; a chain of ordinary matches a beat apart is evidence of an ad +pod. `MaxSkipFraction` remains the guard on the episode as a whole. + +## Theme-music cues around ad breaks: real, detectable, already detected + +Peter named Tangle, Jon Stewart and Search Engine as shows that play a short piece of theme music +at every ad break. All three do, and the scan already finds it -- it is the short, +high-partner-count block that the disproven partner-count filter would have thrown away. + +Search Engine is the clearest: `13:08-13:16`, `35:28-35:36`, `44:51-44:59`, all 8.0s, found in +5-13 other episodes. Jon Stewart's is ~5s at four mid-episode positions per episode. Templating +Search Engine's cue at 250Hz and scanning whole episodes with `NormalizedCrossCorrelation` finds +it at r = 0.96, 0.92, 0.85 in episodes it was not taken from. + +**It marks the entry to a break, not both boundaries.** Scanning for the cue down to r = 0.6 +returns one isolated hit per slot, never a pair, in ad-laden and ad-free copies alike. So the cue +locates a break but says nothing about its extent, which is what a skip needs. + +### Two things found while probing this + +**A second fetch of the same episode is not an ad-free reference.** Three consecutive fetches of +one Search Engine enclosure were byte-identical to each other and 82KB different from the copy +mirrored days earlier. The ad load is stable within a session and rotates over days, so the +"align two fetches and read off the insertions" approach yields nothing when done promptly and +yields stale offsets when done late. + +**Shows have an ad window.** Search Engine's three newest mirrored episodes carry 503s, 469s and +202s of insertion; the five older ones carry exactly zero -- served length equal to declared. +The CDN stops inserting after some weeks. + +That generalises, and it is the more useful finding: **138 of 322 mirrored episodes have served +== declared and therefore contain no inserted advertising at all**, and the scan publishes "ad +breaks" on 111 of them, 74 minutes in total. Those are not advertisements. They are the show's +own recurring furniture -- intro, break cue, outro -- which is worth skipping but is a different +thing, and reporting it as inserted advertising is wrong. Every mirrored episode has a usable +declared duration, so this classification is always available. diff --git a/server/internal/skips/ads.go b/server/internal/skips/ads.go index b3c1f3b..05259c3 100644 --- a/server/internal/skips/ads.go +++ b/server/internal/skips/ads.go @@ -83,8 +83,23 @@ type AdConfig struct { // re-aired episode, a two-parter with a long recap -- and skipping all of it would eat the // programme rather than the advertising. MaxSkipFraction float64 - // MaxBreakMs caps a single break. Longer than any plausible ad pod means shared programme. + // MaxBreakMs caps a single *match*. Longer than any plausible ad pod means the episodes + // share programme -- a re-aired episode, a two-parter's recap -- rather than advertising. + // + // Deliberately applied before bridging, not after. One enormous match is evidence of shared + // programme; a chain of ordinary matches a beat apart is evidence of an ad pod, and fusing + // them must not manufacture a refusal. Radiolab's Grass Apes carries a 3:55 pre-roll and an + // 0:18 tag two seconds later, and capping the bridged 4:15 would publish nothing for the + // best-validated episode in the corpus. MaxBreakMs int64 + // MergeGapMs bridges matches separated by less than this. + // + // A pod of several advertisements is found as several matches, because agreement lapses + // briefly between them. Published unbridged, 45% of the gaps between consecutive breaks came + // in under ten seconds and 152 of 262 multi-break episodes had one -- which plays as a skip, + // a short burst of advertisement, then another skip. Bridging also improves agreement with + // the declared length gap, because what was bridged was advertising too. + MergeGapMs int64 // Runs is what counts as agreement within an alignment. Runs detect.RunOptions } @@ -96,6 +111,7 @@ func DefaultAdConfig() AdConfig { LagsPerPair: 8, MaxSkipFraction: 0.35, MaxBreakMs: 240_000, + MergeGapMs: 10_000, Runs: detect.DefaultRunOptions(), } } @@ -161,7 +177,19 @@ func DetectAds(ctx context.Context, src AudioSource, episodes []Episode, cfg AdC // episode that publishes no breaks because its matches were implausible is a different // state from one where nothing matched, and the two looking identical is the mistake // the app's own skip readout already made once. - matched := merge(found[i]) + // A single match longer than any real ad pod is shared programme -- a recap, or a + // repeat -- and cutting it would remove the episode rather than the advertising. Judged + // on the matches, before bridging fuses neighbours into something longer. + kept := make([]AdBreak, 0, len(found[i])) + for _, br := range found[i] { + if cfg.MaxBreakMs > 0 && br.DurationMs() > cfg.MaxBreakMs { + result.Refused[clip.episode.ID] = true + continue + } + kept = append(kept, br) + } + + matched := merge(kept, cfg.MergeGapMs) total := int64(0) for _, br := range matched { total += br.DurationMs() @@ -179,18 +207,8 @@ func DetectAds(ctx context.Context, src AudioSource, episodes []Episode, cfg AdC continue } - breaks := make([]AdBreak, 0, len(matched)) - for _, br := range matched { - // A single match longer than any real ad pod is shared programme -- a recap, or a - // repeat -- and cutting it would remove the episode rather than the advertising. - if cfg.MaxBreakMs > 0 && br.DurationMs() > cfg.MaxBreakMs { - result.Refused[clip.episode.ID] = true - continue - } - breaks = append(breaks, br) - } - if len(breaks) > 0 { - result.Breaks[clip.episode.ID] = breaks + if len(matched) > 0 { + result.Breaks[clip.episode.ID] = matched } } return result, nil @@ -205,12 +223,15 @@ func breakOf(startSamples, endSamples int, score float64, rate int) AdBreak { } } -// merge folds overlapping and touching matches into single breaks, chronologically. +// merge folds matches into single breaks, chronologically, bridging gaps up to gapMs. +// +// Two things at once. The same advertisement is found once per partner that carries it, so +// without merging an ad in three episodes becomes three identical breaks. And a pod of several +// advertisements is found as several matches with brief lapses between them, which bridging +// turns back into the one stretch a listener experiences. // -// The same advertisement is found once per partner that carries it, so without merging an ad in -// three episodes becomes three identical breaks. The partner count is what survives: it says how -// much agreement a break rests on. -func merge(raw []AdBreak) []AdBreak { +// The partner count is what survives: it says how much agreement a break rests on. +func merge(raw []AdBreak, gapMs int64) []AdBreak { if len(raw) == 0 { return nil } @@ -219,7 +240,7 @@ func merge(raw []AdBreak) []AdBreak { out := make([]AdBreak, 0, len(raw)) current := raw[0] for _, next := range raw[1:] { - if next.StartMs <= current.EndMs { + if next.StartMs <= current.EndMs+gapMs { if next.EndMs > current.EndMs { current.EndMs = next.EndMs } diff --git a/server/internal/skips/ads_test.go b/server/internal/skips/ads_test.go index 476b819..baaec43 100644 --- a/server/internal/skips/ads_test.go +++ b/server/internal/skips/ads_test.go @@ -314,3 +314,115 @@ func TestOneEpisodeHasNothingToCompareAgainst(t *testing.T) { t.Errorf("found breaks with a single episode: %+v", got.Breaks) } } + +/* +Fragmentation, and why bridging it is not cosmetic. + +A pod of several advertisements is found as several separate matches, because agreement lapses +briefly between them -- a beat of near-silence, a jingle one episode's ad server trimmed +differently. Published as-is, 45% of the gaps between consecutive breaks measured under ten +seconds, and 152 of 262 multi-break episodes contained at least one. That plays as a skip, a +one-to-nine-second burst of advertisement, then another skip. + +Bridging also moves agreement with the declared length gap the right way -- 69% to 73% median -- +because the bridged gap was advertising too. +*/ + +func TestAdsSeparatedByABriefGapBecomeOneBreak(t *testing.T) { + first := adNoise(910, 12*adRate) + second := adNoise(911, 15*adRate) + // Two different advertisements six seconds apart, carried by both episodes. + withPod := func(seed int64, at int) []float64 { + out := put(adNoise(seed, 600*adRate), first, at*adRate) + return put(out, second, (at+18)*adRate) + } + stub := &adStub{ + whole: map[string][]float64{"a": withPod(1, 40), "b": withPod(2, 200)}, + durations: map[string]int64{"a": 600_000, "b": 600_000}, + } + eps := []Episode{{ID: "e1", Path: "a"}, {ID: "e2", Path: "b"}} + + got, err := DetectAds(context.Background(), stub, eps, adConfig()) + if err != nil { + t.Fatal(err) + } + + if len(got.Breaks["e1"]) != 1 { + t.Fatalf("e1 breaks %+v, want the pod bridged into 1", got.Breaks["e1"]) + } + br := got.Breaks["e1"][0] + if br.StartMs > 41_000 || br.EndMs < 72_000 { + t.Errorf("bridged break %d-%dms, want roughly 40000-73000", br.StartMs, br.EndMs) + } +} + +// The counterpart: a real stretch of programme between two pods must survive as programme. +func TestAdsSeparatedByProgrammeStaySeparate(t *testing.T) { + first := adNoise(920, 12*adRate) + second := adNoise(921, 12*adRate) + withTwo := func(seed int64, at int) []float64 { + out := put(adNoise(seed, 900*adRate), first, at*adRate) + return put(out, second, (at+300)*adRate) + } + stub := &adStub{ + whole: map[string][]float64{"a": withTwo(3, 60), "b": withTwo(4, 400)}, + durations: map[string]int64{"a": 900_000, "b": 900_000}, + } + eps := []Episode{{ID: "e1", Path: "a"}, {ID: "e2", Path: "b"}} + + got, err := DetectAds(context.Background(), stub, eps, adConfig()) + if err != nil { + t.Fatal(err) + } + + if len(got.Breaks["e1"]) != 2 { + t.Errorf("e1 breaks %+v, want 2 -- five minutes of programme is not a gap", got.Breaks["e1"]) + } +} + +/* +The regression this nearly shipped. + +MaxBreakMs refuses a single match longer than any plausible ad pod, which is how a re-aired +episode or a long shared recap is kept from eating the programme. Applied to breaks *after* +bridging, it also refuses legitimate pods that merging made long: Radiolab's Grass Apes carries +a 3:55 pre-roll and an 0:18 tag two seconds later, and fusing them to 4:15 would have put the +best-validated episode in the corpus over the cap and published nothing for it. Five episodes +regressed that way. + +So the cap belongs on the matches, not on what bridging makes of them: one enormous match is +evidence of shared programme, whereas a chain of ordinary matches a beat apart is evidence of an +ad pod. MaxSkipFraction remains the guard against an episode that matches too much overall. +*/ +func TestAPodBridgedPastTheSingleMatchCapIsStillPublished(t *testing.T) { + preroll := adNoise(930, 200*adRate) + tag := adNoise(931, 60*adRate) + withPreroll := func(seed int64, at int) []float64 { + out := put(adNoise(seed, 1800*adRate), preroll, at*adRate) + return put(out, tag, (at+202)*adRate) + } + stub := &adStub{ + whole: map[string][]float64{"a": withPreroll(5, 10), "b": withPreroll(6, 900)}, + durations: map[string]int64{"a": 1_800_000, "b": 1_800_000}, + } + eps := []Episode{{ID: "e1", Path: "a"}, {ID: "e2", Path: "b"}} + + cfg := adConfig() + cfg.MaxBreakMs = 240_000 // production's cap; the bridged pod exceeds it, each match does not + got, err := DetectAds(context.Background(), stub, eps, cfg) + if err != nil { + t.Fatal(err) + } + + if got.Refused["e1"] { + t.Error("refused an episode whose only long break is a bridged pod of short matches") + } + if len(got.Breaks["e1"]) != 1 { + t.Fatalf("e1 breaks %+v, want the pod published as 1", got.Breaks["e1"]) + } + // 200s + 2s gap + 60s is 262s of continuous advertising: over the cap as one break, while + // neither match that composes it comes close. + if d := got.Breaks["e1"][0].DurationMs(); d < 255_000 { + t.Errorf("bridged pod is %dms, want the whole ~262000 of it", d) + } +} diff --git a/server/internal/skips/skips_test.go b/server/internal/skips/skips_test.go index bfddcd3..936723f 100644 --- a/server/internal/skips/skips_test.go +++ b/server/internal/skips/skips_test.go @@ -329,6 +329,29 @@ func TestDefaultConfigUsesTheValidatedValues(t *testing.T) { } } +// Likewise for the ad scan, whose numbers were each chosen against measured corpus behaviour +// rather than picked: 250Hz is what makes a whole-episode correlation affordable, 0.35 and +// 240s are the two refusals, and 10s is the bridging tolerance that removed the +// skip-burst-skip stutter from 152 of 262 multi-break episodes. +func TestDefaultAdConfigUsesTheValidatedValues(t *testing.T) { + cfg := DefaultAdConfig() + if cfg.DiscoveryRate != 250 { + t.Errorf("DiscoveryRate = %d, want 250", cfg.DiscoveryRate) + } + if cfg.LagsPerPair != 8 { + t.Errorf("LagsPerPair = %d, want 8", cfg.LagsPerPair) + } + if cfg.MaxSkipFraction != 0.35 { + t.Errorf("MaxSkipFraction = %v, want 0.35", cfg.MaxSkipFraction) + } + if cfg.MaxBreakMs != 240_000 { + t.Errorf("MaxBreakMs = %d, want 240000", cfg.MaxBreakMs) + } + if cfg.MergeGapMs != 10_000 { + t.Errorf("MergeGapMs = %d, want 10000", cfg.MergeGapMs) + } +} + // The wiring that makes the self-check matter: a confirmation that cannot verify itself must // not produce skip offsets. Forced here by demanding near-perfect agreement, which a genuine // match between two separately-encoded episodes does not reach -- so this exercises the -- cgit v1.2.3