From 87b8cfa942b995d1c013827ec7014cf6ef4138ca Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Tue, 3 Nov 2026 08:22:11 +0000 Subject: Keep what the listener asked for, and publish ads only from a publisher's master Two changes to the server, both measured against the live data rather than reasoned about. **Subtraction publishes only from a publisher's transcript.** Subtracting a publisher's transcript recovers 94-104% of the advertising the file lengths imply; subtracting a refetch -- a second download of the same assembled file, carrying its own ads in its own places -- recovers a median of 33%. That tier had already been tried and rejected at 21% and 0%, but nothing in the publish path was told which master it had, so it reached the phone anyway on seventeen ad spans. Written as a denylist first, and measuring it against a copy of the live data showed why that was the wrong shape: sixteen episodes carry a master whose label is blank -- the field postdates the first masters written, and Reclaim restored those files after the library was destroyed with the blank intact -- and they published on the strength of not being the one named bad kind. They happened to be publisher transcripts, so the answer was right by luck. It is an allowlist now, and the blank labels are repaired from the master files, which have said "published" all along. Measured on a copy, not predicted: before 49 ad spans (17 refetch, 24 unlabelled, 8 published) + 604 cue, 245 episodes after 62 ad spans (all publisher-master) + 611 cue, 245 episodes No episode loses its skips; the refetch ones trade advertising nobody could measure for the show's own recurring audio, which is a preference rather than an unconditional skip. Measuring also caught the recurring pass blanking the refusal subtraction had just written -- both passes answer "why is nothing published" through one field and recurring runs second, so all thirty-two episodes that now carry a reason were losing it a moment later. **Being asked for keeps an episode mirrored.** Retention was publication date alone, and pruning calls words.Forget, which deletes the transcript *file*. So an episode outside the newest eight was fetched on demand, recognised, served, and deleted on the next poll -- and asking again paid the whole cost over. From the server's own journal over twenty-one days: 397 transcription runs covering 309 distinct episodes, with 7.7 of the 33.4 hours of whisper time spent on episodes it had already recognised and thrown away. 23% of the budget, re-doing work it had discarded. Working through a back catalogue -- the archive tier, the slow-drip shows, the listening this was built for -- therefore paid full recognition for every episode and again to resume one. The window still decides what is worth fetching; RequestedAtUnix now decides what is worth keeping, stamped on every /api/episode call including for episodes already mirrored, because resuming something played last month must not let it fall out mid-listen. Kept for 28 days after the last ask, the same span refetchAfterSeconds uses, being the same kind of judgement about how long something stays interesting. Disk points the same way: transcripts are 36MB for 445 files, audio is 17GB for 336 episodes, and the volume has 365GB free. The expensive artefact is the small one. 374 server tests pass. Not yet deployed; the phone's half of this -- a pending reply is retried only on a feed refresh -- is still to come. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- docs/detection-findings.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) (limited to 'docs') diff --git a/docs/detection-findings.md b/docs/detection-findings.md index 9898dfd..8f5cea7 100644 --- a/docs/detection-findings.md +++ b/docs/detection-findings.md @@ -945,3 +945,42 @@ the state file, uncomputed. figures above are computable from `state.json` for every show. A show whose gaps straddle zero has no usable yardstick and arguably should not publish a subtraction-derived `ad` either. 3. Whether #188's 89-second span is actually advertising. Needs a listen, not a measurement. + +## 2026-11-03: a quarter of the whisper budget was re-doing work + +Retention was publication date and nothing else. `Show.Partition` kept the newest N and everything +else was pruned, and pruning called `words.Forget`, which deletes the transcript *file* as well as +the state's record of it. So an episode outside the window was fetched on demand, recognised, +served, and deleted again on the next poll -- and asking for it a second time paid the whole cost +over. + +Measured from the live server's journal over twenty-one days: + +| | | +|---|---| +| transcription runs | 397 | +| distinct episodes | 309 | +| first-time work | 25.7 h | +| **repeat work** | **7.7 h — 23% of all whisper time** | + +The mechanism, confirmed in the code rather than inferred: `serveRequest` mirrors an on-demand +episode and `detect`s it, but nothing records that anybody wanted it, so the next feed poll +partitions by date and prunes exactly what was just fetched. For a back catalogue -- the archive +tier, the slow-drip shows, the listening this feature was for -- every episode cost a full +recognition and resuming one cost another. + +**The window now decides what is worth fetching and being asked for decides what is worth +keeping.** `Episode.RequestedAtUnix` is stamped on every `/api/episode` call, mirrored episodes +included, because resuming something played last month must not let it fall out mid-listen. +`Partition` spares anything asked for in the last 28 days -- the same span `refetchAfterSeconds` +uses, being the same kind of judgement about how long something stays interesting. + +Disk makes this cheap in the right direction: transcripts are 36MB for 445 files, audio is 17GB +for 336 episodes, and the volume has 365GB free. The expensive artefact is the small one. + +### Still open + +The phone's half. A `pending` reply is retried only on a feed refresh -- the 5am alarm, launch, or +the button -- so an episode the server will have in about a minute stays "waiting for server" +until the next refresh. That, rather than anything on the server, is what makes "not checked yet" +a state a listener sees. -- cgit v1.2.3