| Age | Commit message (Collapse) | Author |
|---|
|
Apache logs the request line, so a token in the query string was a copy of the
secret in every line of a file logrotate keeps fourteen of. That is why the
vhosts log /audio/ without query strings at all; this removes the reason they
had to.
Safe now because both halves of the phone send the header: the downloader
always did, and the player does since it got a ResolvingDataSource. Verified
against this server before changing it -- the player reaches READY on
listen.terst.org audio with a header and no token in the URL.
Query tokens are still *accepted*. Withdrawing that as well would lock out an
older build with no way back in, and accepting one costs nothing now that
nothing emits it.
The transcript URL made this move already, for the same reason. This was the
last one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Tier said "how important" and the shows needed "what kind". Seven ordered
categories replace it: news, analysis, curiosity, sailing, true stories,
fiction, deep dives. The order is still the server's processing priority, so
one field carries both axes -- which is the idiosyncrasy docs/show-kinds.md
is written to watch, with four signs that a better abstraction has arrived.
All 42 known feeds are refiled by migration 20 -> 21, which rebuilds the
subscriptions table rather than renaming the column: RENAME COLUMN keeps the
old 'ordinary' DEFAULT and fails schema validation, and DROP COLUMN needs
SQLite 3.35 against minSdk 29's 3.28. The wire value for true stories is
"truestory", not "true", because SQLite reads an unquoted true in a DEFAULT
as the boolean and stores 1.
Schema 21 is a new exported JSON; migrate20To21RenamesTierAndRefilesTheKnownShows
covers it, and all 22 device migration tests pass.
The coverage exclusion for the settings composable follows the file's rename;
left pointing at TierSettingKt it read as a 0.6% drop. The remaining 0.08% is
the deletion of the covered tier-seeding code, taken as an exception.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
**Android Auto was the one caller still streaming from publishers.** `onAddMediaItems` resolved a
browse id straight to a media item without asking whether the audio existed, so an undownloaded
episode was handed to the player as its publisher's URL. That is the worst place for it: the car
is where most of this library is actually heard, and misplaced skips are least obvious there.
It goes through `PlaybackAdvancer.startFetchingIfNeeded` now -- start, and fetch first if there is
no file -- the same decision the in-app paths make, in the one class tested for it. A fetch that
fails stays failed rather than falling back: the publisher's copy carries different advertising,
so the server's measured breaks land in the wrong place, and silence that can be retried beats
skipping into the programme.
With that, nothing reached the fallback, so it is gone. `setUri(localPath ?: mediaUrl)` is now
`setUri(localPath)`, `MediaUrl` is deleted, and OutboundRequestsTest asserts zero remote media
items rather than pinning one. `start` also reports Gone for an episode whose show *and* file have
both gone, which is what made the fallback removable -- an orphan that still has its file plays.
**Two findings from the security review of the last commit, both mine, both real:**
- `Request.Builder().url()` throws on a URL it cannot parse and only IOException was caught, so a
feed with a junk enclosure threw straight past the Result its caller folds. Survivable when this
ran from a worker; not now that every play goes through a fetch, where one bad enclosure in one
feed would stop any episode starting. Caught in the downloader, and `fetchAndStart` turns a
thrown fetch into FetchFailed -- rethrowing CancellationException, which a test now pins,
because swallowing it would report a cancelled start as a failed one and leave a false error on
screen.
- `FeedFetcher` logged whole feed URLs. Private feeds carry a token in the query and some carry
credentials in the userinfo. `FeedFailureText.redact` cuts them to scheme and host, and is
tested as the security control it is rather than as tidiness. The unparsable branch logs neither
the address nor the exception, and says why: that exception's message *is* the address, so
logging the cause would put back exactly what redacting removes.
**The transcript URL no longer carries the token.** Apache logs the request line, so every
/api/transcript fetch wrote the server's token into its access log in plaintext, across fourteen
rotations. It was redundant there: the phone sends Authorization: Bearer anyway and tokenOK takes
either. The audio URL still carries one -- not redundant, the downloader has no credentials of its
own -- so removing that needs an app change installed before the server stops emitting it.
The gate refused this twice and was right both times. The coverage drop was not all deletion: the
redaction and the cancellation contract were genuinely untested, and have tests now. Then detekt
caught an import left behind by moving the redaction into FeedFailureText -- a move I made after
the last time I ran it.
1031 app tests, 376 server tests, detekt 0, lint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Two changes to the server, both measured against the live data rather than reasoned about.
**Subtraction publishes only from a publisher's transcript.** Subtracting a publisher's transcript
recovers 94-104% of the advertising the file lengths imply; subtracting a refetch -- a second
download of the same assembled file, carrying its own ads in its own places -- recovers a median of
33%. That tier had already been tried and rejected at 21% and 0%, but nothing in the publish path
was told which master it had, so it reached the phone anyway on seventeen ad spans.
Written as a denylist first, and measuring it against a copy of the live data showed why that was
the wrong shape: sixteen episodes carry a master whose label is blank -- the field postdates the
first masters written, and Reclaim restored those files after the library was destroyed with the
blank intact -- and they published on the strength of not being the one named bad kind. They
happened to be publisher transcripts, so the answer was right by luck. It is an allowlist now, and
the blank labels are repaired from the master files, which have said "published" all along.
Measured on a copy, not predicted:
before 49 ad spans (17 refetch, 24 unlabelled, 8 published) + 604 cue, 245 episodes
after 62 ad spans (all publisher-master) + 611 cue, 245 episodes
No episode loses its skips; the refetch ones trade advertising nobody could measure for the show's
own recurring audio, which is a preference rather than an unconditional skip. Measuring also caught
the recurring pass blanking the refusal subtraction had just written -- both passes answer "why is
nothing published" through one field and recurring runs second, so all thirty-two episodes that now
carry a reason were losing it a moment later.
**Being asked for keeps an episode mirrored.** Retention was publication date alone, and pruning
calls words.Forget, which deletes the transcript *file*. So an episode outside the newest eight was
fetched on demand, recognised, served, and deleted on the next poll -- and asking again paid the
whole cost over. From the server's own journal over twenty-one days: 397 transcription runs
covering 309 distinct episodes, with 7.7 of the 33.4 hours of whisper time spent on episodes it had
already recognised and thrown away. 23% of the budget, re-doing work it had discarded.
Working through a back catalogue -- the archive tier, the slow-drip shows, the listening this was
built for -- therefore paid full recognition for every episode and again to resume one. The window
still decides what is worth fetching; RequestedAtUnix now decides what is worth keeping, stamped on
every /api/episode call including for episodes already mirrored, because resuming something played
last month must not let it fall out mid-listen. Kept for 28 days after the last ask, the same span
refetchAfterSeconds uses, being the same kind of judgement about how long something stays
interesting.
Disk points the same way: transcripts are 36MB for 445 files, audio is 17GB for 336 episodes, and
the volume has 365GB free. The expensive artefact is the small one.
374 server tests pass. Not yet deployed; the phone's half of this -- a pending reply is retried
only on a feed refresh -- is still to come.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The findings doc's last entry was 2026-10-19 and ended by naming what was missing: "a mechanism
that can tell a real short ad from a short coincidence". Words are that mechanism, it was built,
and thirty-two server commits later the doc never recorded what it measured. A future session
would have re-attempted the ad/furniture classifier that was already measured and deleted.
The new entry is derived from the live state file rather than quoted, which is how it caught that
publish.go's own figure describes a subset. "92% to 104% across eighteen episodes" is the
publisher-transcript number. Across the 24 episodes now publishing an ad span, recovery runs 20%
to 146%. Split by where the master came from, the spread stops being a spread: a publisher's
transcript gives 94-104%, a refetch gives a median of 33%. Nothing in Publishable looks at which
it has, and refetch masters are now fifteen of those twenty-four -- the majority of a tier the
design had decided not to act on.
It also caught me getting a conclusion wrong. Two Reply All episodes publish ad spans at 146% and
138% of their length gap, which reads as the detector over-claiming by half a minute of
programme. Looking at the show instead of the episode: Reply All's gaps run from -0.48 to +1.02
minutes across eight episodes. They straddle zero, so the feed's declared durations are not a
baseline for that show and the thing being exceeded is noise. The structural finding is the useful
one -- HasInsertedAdvertising asks only whether this episode's gap clears one ad unit, and has no
notion of whether the show's gaps are consistent enough to mean anything. Both versions are in the
entry, because the wrong one is the one a reader would reach for next time.
Nothing here changes behaviour. Whether a refetch should publish an ad at all is Peter's call and
is written up as such.
CLAUDE.md, separately:
- It still said the episode description shows about two lines. That was changed to ten yesterday,
for a reason that is recorded: the limit existed to stop long notes pushing the transport
off-screen, and the merge had already put the notes below the transport. Left as it was, the
doc was an instruction to reintroduce the bug.
- "207 unit tests" has been wrong for most of the project's life; the suite is past nine hundred.
Replaced with a note to ask the wrappers, since any number written here goes stale the same way.
- The TDD rule banned backfilling tests onto existing code without qualification, which also
forbids the one case where it is right -- pinning behaviour before moving it, so a refactor is
provably behaviour-preserving. Narrowed to say that, and to say the order matters. The sixteen
formatter tests in 2cc252a were written that way.
- Added the rule those formatter files were moved to satisfy: anything the app says out loud lives
in a named file of its own and is tested, never inside the composable that renders it. It was
already followed by eight files and stated only in WaitingForServerText's own comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Four packages each wrote 15 seconds for the length of an advertising slot -- skips, transcript
twice, and signature -- and every copy carried a comment saying it agreed with the others: "the
same unit skips.Classify uses", "the same figure the length-gap rule uses", "one ad unit from the
start". The agreement was real and deliberate, and recorded only in prose, so a corpus that
revised the figure would have moved one copy and left three disagreeing silently.
internal/advertising holds it once. Each site keeps its own name and its own explanation of what
the unit means there, because those genuinely differ; only the number is shared.
The consolidation on its own fixes the four copies that existed. The audit test refuses the
fifth: it walks the server's production source and fails on the number written as a literal,
in either of Go's spellings. Watched failing on both before being trusted. Same arrangement as
safefetch/audit_test.go, and for the same reason -- the instance that was noticed was never the
whole of it.
No behaviour change: 366 server tests pass, same as before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
I pointed the emulator at the live server to verify that published breaks
actually skip. It worked. It also pushed the emulator's own four-show
subscription list, which replaced the real one, and at 07:07:13 the server
reclaimed every show that device had never heard of: 26 shows and 290 mirrored
episodes, deleted in a second.
Three fixes, in the order they matter.
**The list can no longer be emptied by one push.** It is still replaced
wholesale -- the phone owns it, and reconciling two editable copies of something
changed this rarely costs more in sync bugs than it saves -- but a push that
keeps fewer than half of ten or more feeds is refused with a 409 and logged.
Losing most of a library in one push is a second device or a bug; it is never
somebody unsubscribing. Below ten feeds there is nothing to protect: a new
phone's first push legitimately replaces nothing with everything.
**Recognition is recovered rather than redone.** A transcript is keyed by the
episode id, which comes from the feed and the guid, so it outlives the record
and matches again when the episode is mirrored afresh. 332 transcript files
survived on disk. Reclaim restores them -- 18 immediately, the rest as
re-mirroring reaches them -- instead of spending twenty hours of CPU recognising
text the server is already holding. It refuses a transcript whose last cue runs
past the end of the audio, which is the same check made of a publisher's.
**The pipeline's Save was never wired**, which the security review caught before
it could panic in anger: the extraction moved the call and not the field. It is
now optional, because every test wants it nil, and assumed-non-nil is what
shipped a server that would have died on its first recognised episode.
Reclaim runs before each show is polled as well as at startup, or a re-mirrored
episode would wait for a restart to find the words already sitting next to it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
cmd/listend/main.go had reached 1,249 lines carrying nine passes, and cmd has no
test file -- so every judgement in the transcript pipeline lived where nothing
could reach it. That is not hypothetical: a refetch master blocked the recurring
detector for a fortnight because the condition was wrong in a place no test
could see.
internal/words now holds the five passes with judgement in them -- Harvest,
Recognise, Rederive, Recurring, SecondCopy -- behind a Pipeline that names its
dependencies. main.go is 631 lines and does what a command should: read flags,
wire things together, serve.
**Moving it found a latent bug.** Pruning an episode's audio cleared its
transcript fields but not its master ones, so a re-mirrored episode would claim
a master whose file had been deleted -- a second description it could not
produce, and subtraction finding nothing for it for ever. Exactly the shape of
the bug that had just cost a fortnight. Pipeline.Forget now clears all of it,
including the attempt timestamps, so a re-mirrored episode is done again rather
than skipped for ever. Pinned by a test that asserts it needs recognising
afterwards.
Also in this part: Gradle's file-system watching is off. Chaquopy extracts a
3,200-file Python environment into app/build mid-build and the watcher tripped
over re-registering it, printing "Caught exception: Already watching path" in
front of every build error this project produced all week. It costs nothing to
turn off -- measured twice each on a no-op compileDebugKotlin, 1.50s and 1.59s
with, 1.46s and 1.46s without -- because that extraction invalidates the VFS
anyway. A watcher holding stale state about a tree another tool rewrites causes
the "clean and try again" class of problem rather than curing it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Peter's framing: a few shows are news and stop being worth hearing within a day,
a tier he keeps up with, and a long tail that is not topical at all. It matters
because recognising one episode costs the server about five minutes, so a
five-hour back-catalogue episode arriving the same morning as the news takes
most of a cycle before the news is looked at.
Three tiers and no more. A number would invite fiddling and would not mean
anything; these are the three distinctions that were described.
The tier rides beside the subscription list rather than inside it, so a server
that knows nothing about tiers reads the list exactly as before and a phone that
knows nothing keeps pushing what it always did. Only the shows placed somewhere
other than the default are sent -- "ordinary" for sixty shows is the same
information as nothing, in sixty times the space.
It orders both the poll and the recognition queue. Recognition now sorts on
three things in the order they matter: a topical show first, then an episode
whose answer checks itself against the length gap, then newest.
**A tier only exists for a subscribed feed**, which a test caught: the list and
the tiers arrive in one push, and setting the tiers after the list put back a
placement for a feed that same push had just removed. A resubscription months
later would have inherited it.
Unrecognised tiers are ignored at both ends and read as the default, which is
the one that starves nothing -- the same rule the label store follows for a
verdict it does not know. Schema 20, migrated and checked on a device.
The control says what it does not do as well as what it does: nothing about
playback changes, and a show placed anywhere still plays, downloads and skips
identically. A setting that looks like it changes playback and does not is worse
than no setting.
The coverage gate caught the push body having no test, which was fair -- it is
hand-built JSON going over the wire, where a missing brace is a library that
silently stops being mirrored. Its exact output is now pinned, including a URL
with a quote in it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three detectors now cut audio out of episodes and the app said nothing about
which. The listener judging a skip wants to know what judged it, and they are
not equally sure of themselves: one compares waveforms across a batch of
episodes, the other reads what was said and checks itself against the declared
length. A break both found is marked as both, because agreement by entirely
different means is the strongest thing either can offer.
"Found by the audio and the words", under the numbers and quieter than them --
it explains them rather than competing with them.
Schema 19. Existing rows default to the audio scan because that is where every
break in the database came from; the words could not produce one until this
week. "Unknown" would have been honest about the column and useless about the
rows. A server that says nothing gets the same default, for the same reason.
The coverage gate caught the new field's default never being exercised, which
was fair: every call site passes it, and the case it exists for is an episode
with nothing cut and nothing to account for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Every other number in this project comes from the declared-versus-served length
gap, which says how much advertising a copy carries and nothing about where. A
detector could find the right total in the wrong places and score 100%. The
position check I wrote earlier used the same signal that builds the candidates,
so it was partly circular and structurally blind to the failure it most needed
to see.
The 19 labels are the only statement about *where* that no detector produced:
each is Peter pressing a button at a moment in an episode. listen-score is
read-only, takes the state file, and prints how the published breaks did.
**Its first answer was wrong, and checking rather than reporting it caught
that.** It said 2 of 18 advertisements were covered, which would have meant the
detectors were far worse at position than at totals. They are not: breaks at
829-999 and 1360-1519 against labels at 853-1000 and 1489-1540 -- correct
detections, scored as complete misses because the break stopped a second short
of the label's end. Twelve of the nineteen labels have an end the store itself
documents as a guess, "where a jump happened to land". Scoring against a guessed
end is scoring against noise. A label's start is solid; its end counts only when
measured. 5 of 18 after that.
**A refetch master was blocking the better detector.** Tangle's "The Israeli
elections" has a 229-second gap, a master from a second fetch that carried the
same campaign, and therefore no subtraction candidates at all -- and
findRecurring skipped it because it had a master. Treating "has a master" as
"has a better answer" is wrong when the master is useless. It now skips only
episodes that actually published something.
**The one programme cut is the audio scan's, not the words'.** Its break at
696-708 sits exactly on the stretch Peter said was programme. Worth knowing
before anything else gets blamed for it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The text analogue of the audio scan, and it costs nothing: the server already
holds transcripts for eight episodes of each show, mirrored within a minute of
each other, carrying one campaign and eight different programmes. Text several
of them share is not the programme.
It reaches where subtraction cannot. Subtraction needs a second description of
one episode -- a publisher's transcript or a second fetch -- and 18 of 337
mirrored episodes have one. This needs only the episodes already held. Measured
against the length gap on the live library: Heavyweight 111%, Incels 105%, Ezra
Klein 92%, Tangle 83%, against refetching's 21% on the same show.
**It is published as the show's own audio, not as advertising, and that is a
measured limit rather than caution.** What recurs is a show's sponsor reads and
its furniture together -- its welcome, its break cue, its credits -- and I tried
to separate them and could not. Scored against the 404 recurring stretches in
episodes that also hold a publisher's master, which says which is which:
ad copy markers 5% of the advertising, 4% of the furniture
length mean 51s against 29s, overlapping throughout
position ads 62 open / 176 mid / 41 close
furniture 12 open / 61 mid / 52 close
A first classifier built on ad copy agreed with the master 33% of the time and
called 263 advertisements furniture. None of those signals separates the two.
So the detector claims only what it measured: this is text the show says every
week. That is what KindCue means, and it leaves the listener's setting in charge
-- a show's own audio is skipped by preference, advertising unconditionally, and
claiming the stronger of the two on a coin flip would be the worse mistake.
Refused when more than 35% of an episode recurs, the same share the ad scan
refuses at: a re-aired episode or a formulaic segment would otherwise be eaten
whole. Guarded by share rather than by the length gap, because the episodes this
is worth most on are the ones with no gap at all -- a show that bakes its reads
into what it publishes declares the length it serves, so the gap is blind to
every advertisement in it.
Skipped entirely for an episode that already has measured advertising. The
better answer must not be overwritten by the vaguer one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Subtraction has earned it. Eighteen episodes, 92% to 104% of what the
declared-versus-served length gap implies, every pod found, edges pulled back
off the programme, and the one class of mistake that mattered -- a break opening
on the host's own sentence -- traced and fixed.
**Only subtraction is published.** Reading the ad copy measured 0%, 0%, 15% and
29% on the same kind of episode, and on a show that bakes its reads in it found
2:13 where the lengths can see nothing at all. That stays a hint beside the
words, dimmed on the transcript screen, and never becomes a seek.
**The length gap is the guard as well as the measure.** It comes from the feed
and the file rather than from any detector, so it is the one number able to
contradict one. Nothing is published for an episode with no gap -- there would
be nothing to check it against -- and a candidate set claiming half again as
much as was inserted is refused with its reason, because it has found something
other than advertising.
They land in their own field rather than beside the audio scan's, which rewrites
its list from scratch on every detection pass and would wipe them. The API
serves the union, merging ranges that overlap or touch: playback seeks to a
break's end, so two overlapping breaks would seek out of the first and straight
into the second, and two touching ones would resume for no time at all before
skipping again. Advertising wins where merged ranges disagree about what they
are -- the phone skips advertising unconditionally and a show's own audio only
by preference, and being wrong the other way means sitting through an
advertisement the server had already found.
Nothing about playback changes. They arrive in the field the phone has read
since the on-device detector existed.
**Refetching is now gated on age**, which is Peter's steer and what the numbers
say. A second fetch made minutes after the first came back twenty seconds
different and found 21% of the advertising; another came back the same length to
the second and found none. Insertion varies the amount per request, not the
campaign. So it is for shows whose episodes are still worth hearing four weeks
later -- a slow-drip archive, a leisure show -- and not for daily news, which is
pruned long before a campaign rotates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three numbers on the line that already says an episode has a transcript: how
many stretches of advertising were found, how long they come to, and how.
The method travels with them because subtraction and ad-copy reading are not
close in quality -- 92-104% of the length gap against 0-29% -- and a figure on
its own would invite reading the two alike. An episode that turned up nothing
says nothing rather than zero, which is the same three-state care the rest of
this takes.
Still nothing published as a skip. This is so the evidence can be looked at from
the app rather than from my logs, which is what the decision to publish should
rest on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Subtraction measures itself at 92-104% against the length gap and finds every
pod, but it needs a publisher's transcript, and only 18 of 336 mirrored episodes
have a usable one. So the best detector in the project covers about five per
cent of what Peter listens to. This is Peter's idea for the other ninety-five.
Advertising is inserted per request, which this project measured the hard way:
the same Radiolab episode came back 7:29 of advertising on one fetch and 5:34 on
another. So two fetches of an episode agree on the programme and differ on the
advertising, and subtracting one from the other says where the advertising in
the other one is. Inserted needs no change at all -- it already takes two cue
lists and reports what the second has that the first does not.
Confirmed before building the pass: a second fetch of Tangle's "The corruption
is still the story" came back at 986 seconds against the mirrored copy's 1056,
with both declaring 690. Seventy seconds of different advertising, from a
request made minutes apart.
What it cannot see is a campaign both copies happened to carry. That is the
exact opposite blind spot to the audio scan, whose entire method is finding what
a batch of episodes shares -- so between them the two cover each other, which is
worth more than either being better.
The second copy's audio is deleted the moment it is recognised, and it is
fetched into its own directory under its own id, so nothing can mistake it for
the bytes the phone plays even for the moment it exists. Only episodes whose
served length exceeds what the feed declares are fetched at all: without a gap
nothing was inserted, and the second copy would be the same recording.
Bounded to twenty minutes a cycle, abandoned after three failures in a row with
the attempts given back, and ordered newest first -- the same shape as
recognition, for the same reasons.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Printing the first line of all 65 pods in the corpus found the one failure that
actually matters -- a break opening on the programme -- in about eight of them.
Three were identical: "I'm Jack Resider.", which is the Darknet Diaries host
saying his own name, misheard. Rhysider is not a word whisper knows.
Four words out of four means one misheard word anywhere in a short line hides
the whole line from the master, so it is never given back. Three of "i m jack
resider" still finds "i m jack" where the publisher wrote "I'm Jack Rhysider".
Short lines are now judged on three; long ones are unchanged, where a full run
is affordable and a looser one would match by accident.
The annotation pattern also missed a bare "♪", which is a music note without
brackets round it.
Measured across the eighteen episodes holding a master:
before 99% - 106%, mean 101%
after 92% - 104%, mean 100%
The three that fell furthest are the three Darknet Diaries episodes, and they
fell because the host's sentence is no longer inside the break. Falling below
the length gap is the right answer there: the gap counts advertising, and his
name was never advertising.
Eight suspect pod openings are down to four, and all four are now either a
one-word line too short to judge at all ("Yay!", "- Pessimism.") or a line that
is mostly advertisement with a second of handoff in front of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
whisper writes "[music]", "(upbeat music)", "(audience laughing)" -- 312 of the
56,306 recognised lines in the library. They are not speech, so no publisher's
written transcript contains them, so they can never anchor, so they sit at the
front of a break and widen it.
Earlier I said these could not be told from an advertisement and left them
alone. That conflated two different things. A title card -- "Today's episode,
Tamara." -- genuinely cannot be told apart, and is still left alone. A non-speech
note says what it is by its shape.
Measured across the eighteen episodes holding a master:
before 99% - 110%, mean 103%
after 99% - 106%, mean 101%
#69 Tamara 110% -> 103%
#70 Cayden 109% -> 106%
Regrets 108% -> 104%
Gobi 104% -> 100%
Fourteen of the eighteen now sit within one point of the length gap. Nothing
regressed.
They are trimmed from break edges but still stored and still shown: a reader
following along is better off seeing the laughter than a hole where it was. This
is a judgement about where an advertisement starts, not about what was said.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three episodes were missing a whole advertisement, which was the largest error
left in subtraction. Diagnosed rather than guessed at: Incels "From Victim to
Violence" has a three-minute pre-roll in which the served copy runs 463 words
against the master's 84, and it was refused because the master advanced more
than the ten words of drift the rule allowed.
That guard was there for a real reason -- a stretch where both texts advanced is
one where you cannot say which part of it is the advertisement -- but it answered
the question at the wrong resolution. The gap as a whole need not be all
advertising; its individual lines are each either in the master or not, and that
is answerable. So the drift guard is gone and every gap is partitioned line by
line against the master instead, which is the same machinery the edge trim
already used. The trim is no longer a separate step a caller has to remember,
which is just as well: a caller who forgot it would publish a skip over the last
sentence of the programme.
Measured across all eighteen episodes holding a master:
before 73% - 110%, mean 99%, three episodes missing a pod
after 99% - 110%, mean 103%, every episode finding every pod
#67 Jenna 81% -> 106% 3 pods -> 4
Behind the Scenes 73% -> 101% 3 pods -> 4
From Victim to Violence 78% -> 99% 3 pods -> 4
Nothing got worse. The fifteen episodes that were already right are unchanged to
the percentage point.
The fixtures spoke at 360 words a minute, which hid the fact that the word floor
and the duration floor disagree about what counts as an advertisement. They now
run at about 180, which is roughly how fast people talk, and the two floors
agree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Subtraction answers "not in the master", which is a superset of "advertising".
Across the eighteen episodes holding a master the candidates come to a mean of
exactly 100% of what the length gap implies, and individually overshoot by up to
10%. Reading their edges showed two causes.
The first is the programme leaking in. Darknet Diaries' "NSL" opened a break on
"I'm Jack Resider." and closed one on "So Nick and ACLU, we're going to
challenge"; Incels opened two on "let's stop here for a...". Those lines failed
to anchor only because recognition dented a word near them, and skipping them is
the harm this whole thing exists to prevent. Their words are in the master, so
they can be found there and given back.
The second is whisper's own "[MUSIC PLAYING]" and "(upbeat music)" and a title
card. Those are genuinely absent from a publisher's written transcript, so
nothing here can tell them from an advertisement. Left alone: they sit against
the break anyway, and a rule invented to catch them would be a guess.
**Measured in both directions**, which is the lesson MinRunSamples taught:
Worship at the Feet of a Coward 108% -> 100%
the 764 Network 104% -> 100%
The Manosphere 104% -> 101%
I hit you because I love you 104% -> 101%
Gobi 107% -> 104%
From Victim to Violence 86% -> 78%
Behind the Scenes 75% -> 73%
The three that got worse were already the worst, and not because of this: each
finds three pods where its show's others find four, so it is missing a whole
break. The trim removes edge text from the pods they did find, before and after
alike. Mean 100% -> 98%, and the top of the range is where it should be.
Four words, five and six all measured. Five and six are identical and four
differs by one point on one episode, which is noise at this sample size. Four is
kept because it is the only one that still recognises a short line with a
mistake in the middle -- an eight-word line with a dented fifth word has no
clean five-word run, and "So Nick and ACLU, we're going to challenge" is eight
words. Trimming more also errs towards smaller candidates, which is the safe
direction here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Peter's idea, and it is the best detector in this project because it is a
measurement rather than a guess.
A publisher's transcript describes the ad-free master. Recognition over the
mirrored file describes the copy the phone will play. The copy is the master
with advertising stitched into it, so the text present in the served copy and
absent from the master *is* the advertising, with its own boundaries. No phrase
list to be wrong about, and no show whose hosts happen to talk like an advert.
Measured on Darknet Diaries' "Conti" before it was wired in. The feed declares
59:17, the file served runs 63:33, so 4:15 was inserted. Subtraction found
**4:20 in three pods -- 102%** against a figure no detector produced. Reading the
ad copy on the same words found 0:16, or 6%. Every pod starts on the first word
of an ad read and ends on the last: a NetSuite pre-roll, a 2:20 midroll, and a
Doppel read at 32:08. Nothing of the programme is clipped.
The alignment has to survive recognition being approximate -- whisper writes
"Bombus" for "Bombas" and spells numbers differently -- so it anchors on runs of
five words occurring exactly once in each text. A window containing a mistake
simply fails to anchor and its neighbours carry the alignment instead.
Uniqueness is what makes an anchor safe: a phrase appearing twice in the master
cannot say which occurrence it matches, and a wrong anchor drags the alignment
across a break. Inside an advertisement nothing matches at all, which is exactly
what makes a pod show up as a gap.
An anchor names the *start* of a matching run, so the content between two
anchors begins after the first run ends. Getting that wrong put every break one
anchor-length early, which on a real episode is a skip over the last sentence of
the programme. The tests pinned it.
**This is why a declined transcript is now kept.** Declined means "this copy
carries inserted advertising the master does not describe" -- precisely the
episode subtraction works on. One declined for describing a different recording
is still thrown away, since it is no use for anything, and Fits now carries a
reason rather than making callers match on its wording. The nineteen episodes
declined before this existed fetch their masters once on the next poll.
Nothing is published as a skip yet. The candidates are stored with the words and
dimmed on the transcript screen, and every episode now logs what was found
against what the lengths imply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
whisper.cpp, installed on tungsten and shelled out to per episode. The other
304 of 342 mirrored episodes -- the ones whose publisher's transcript was
declined because this copy carries inserted advertising the master does not
describe, which is both why it is unusable and why having words would be worth
most.
**Measured, and an order of magnitude cheaper than I told Peter.** base.en on
six threads does 10.3x realtime on this box: a 63-minute Darknet Diaries episode
took 6m11s at 793MB peak. The earlier "roughly realtime, days of backfill" was
wrong. The real backlog is 298 episodes, 266 hours of audio, about 26 hours of
CPU -- three days at 45 minutes per two-hour cycle, which is the default budget.
Bounded per cycle rather than exhaustive, and only on the timer: a wakeup is the
phone pushing a subscription list and wanting that show polled now, and
following it with three quarters of an hour of recognition would make a new
subscription feel broken. Newest first, because an episode about to be played is
worth having words for before one from the back of a catalogue.
**The first real run found two bugs the tests could not have.** The working
directory was never created, so ffmpeg could not write and every episode failed
-- and each failure was recorded as "do not try this one again". A misconfigured
deploy would have marked the whole backlog attempted in seconds and nothing
would ever have been transcribed, which is precisely the shape of the midroll
scan marking episodes checked forever after an OutOfMemoryError nobody saw.
transcribe.Breaker now abandons a pass after three consecutive failures and
gives those attempts back, since a run of failures is the engine rather than the
episodes -- while one bad file among good ones is still marked, because the same
bytes will fail the same way next time.
NeedsRecognising is deliberately not the negation of TranscriptExamined. A
declined transcript means examined, no words, and exactly the episode
recognition exists for.
**First validation against ground truth the detector did not produce.** Up First
"Democrats Lead In NPR Poll": the feed declares 788s, the file served 890s, so
102s of advertising was inserted -- and reading the recognised words found 100s
of it, in two pods, every line unmistakable ad copy. The sibling episode found
21s of 131s: a whole pod at 54-130s was missed because its copy carries only one
kind of marker. Reported rather than tuned around, on two episodes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
A second ad detector, and it finds a class the first structurally cannot.
The audio scan compares episodes mirrored within a minute of each other, so
what it finds is advertising a CDN stitched into all of them -- validated
against the declared-versus-served length gap. A show that inserts nothing has
no gap, which by that measure means it carries no advertising, and the host-read
sponsor block in the middle of it is invisible. Every published transcript the
server can use belongs to exactly such a show, because having no inserted
advertising is the condition for using one. The two detectors do not merely fail
differently; they look at disjoint sets of episodes.
**Density, not phrases.** One marker is a mention. Measured against the 38
transcripts harvested from the live library: a single "brought to you by" is a
one-second tag inside play-by-play and skipping it would cut the game, and a
guest describing how a sponsor came forward for his yacht event says "sponsor"
four times while being the programme -- the same trap the toc=false chapter rule
already documents. A real read stacks different *kinds* of marker inside a few
seconds: who it is for, where to go, what the offer is, what the terms are. Two
kinds and fifteen seconds, the same ad unit the length-gap rule uses.
Every threshold comes from a measurement rather than a guess. The 25-second
bridge is the 17.7-second gap between "presented by Delta" and "12% off" in the
Seahawks Rewind read, with the product description in between. "Slash" needs a
domain in front of it because the corpus contains "a children's movie slash book
for adults".
Across all 38 live transcripts it reports **one** candidate: that Delta block,
41:07 to 41:33, with no programme clipped at either end. That is low recall by
design -- these are the shows with the least advertising, because they are the
only ones whose transcripts can be used at all. The shows with real ad breaks
need speech recognition, which is not built.
**Nothing is published as a skip.** The candidates are stored with the words and
the lines are dimmed on the transcript screen, so the reader can see what the
server thinks is an advertisement and disagree with it. Dimmed rather than
hidden, because a wrong call has to stay visible. Nothing in the playback path
reads them -- same posture as the signature library's shadow run, and the same
reason: a detector acting on its own unchecked first guesses makes skips into
the middle of the programme, which is worse than no skips.
Schema 18 adds the mark to the cue, defaulting to not: a line nobody judged is
not an advertisement. Migrated and checked on a device.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Two follow-up findings on the same file, both real.
**A proxy would have made the whole check pointless.** http.DefaultTransport
carries ProxyFromEnvironment, so with HTTP_PROXY set the connection goes to the
proxy and the proxy connects to whatever the URL named -- the address being
checked would be the proxy's, and the one that matters would never be looked at.
The safe client now sets no proxy, and a test asserts it.
**"Not RFC 1918" is not the same as "public".** The check is now an allowlist:
an ordinary global unicast address that is not in an IANA special-purpose range.
The ranges that were being let through include 100.64.0.0/10, which is
carrier-grade NAT and routes to a real host on a real network, and 64:ff9b::/96
and 2002::/16, which are NAT64 and 6to4 and carry an IPv4 address inside them --
127.0.0.1 included. None is private by Go's definition and none is somewhere a
podcast is served from.
Checked against the live library before committing: 130 real URLs -- all 44
feeds and two enclosures from every show -- resolve to addresses this permits,
so nothing Peter subscribes to stops mirroring. That check is kept as a test
which skips where the state file is not present, because the failure mode of
being too strict is a feed that silently stops updating.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
A security review flagged the transcript download as an SSRF: the URL comes from
a <podcast:transcript> tag, so a publisher chooses it. It was the third place
with that shape, not the first -- the feed fetch takes a URL the phone sent, and
the enclosure download takes one from inside the feed, and both had been doing
it for longer. Fixing the flagged instance alone would have fixed nothing.
The exposure is specific rather than theoretical. listend listens on
127.0.0.1:38090 behind Apache on a host running other services, so a feed naming
a loopback, private or link-local address -- 169.254.169.254 among them -- would
have the server fetch something only it can reach. In the transcript case the
result would then be stored and handed to the phone as an episode's words, which
turns an internal reachability problem into a readable one.
The check runs when the connection is dialled, not when the URL is read, so it
also covers a redirect into an internal address and a hostname that resolves to
one. The address connected to is the address that was checked.
internal/safefetch now has a structural test that walks this server's own source
and refuses any production file that builds its own HTTP client or uses the
shared default. It was watched failing against a deliberately added violation
before being kept. Tests still inject a plain client through the seam each type
already had, because a httptest server is on loopback by definition -- which is
exactly what production is now refusing.
Checked against the live subscription list before committing: all 44 feeds use
https and none resolves to a private, loopback or link-local address, so nothing
Peter subscribes to is affected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Eleven of the 44 subscribed feeds carry <podcast:transcript>, in SRT or WebVTT.
Those are the publisher's exact words with the times they were said, which is
free and better than anything recognition would produce.
**They are only usable when the show does not insert advertising**, and that
turns out to be exact rather than approximate. A published transcript describes
the publisher's master. Measured against the live library on 2026-10-28:
Darknet Diaries "Conti" declared 3557s served 3813s last cue 3544s
Wards Way "A Family Bar" declared 1216s served 1217s last cue 1213s
LOW "Afterword" declared 610s served 610s last cue 603s
LOW "Part 5: Bane" declared 2822s served 2823s last cue 2807s
The transcript tracks the *declared* length every time. So it aligns precisely
when served and declared agree, and is minutes out of step precisely when they
do not -- which is the case where a transcript would be worth most. Reading
along with one of those would put every line in the wrong place, and a break
placed from it would seek into the programme.
transcript.Fits makes that call from the two lengths rather than from the words,
because the lengths are independent of the transcript: the feed declares the
ad-free programme and the file measures what was served, so their difference is
this copy's advertising. It is the same figure and the same fifteen-second unit
the ad scan already validates itself against.
A refusal is recorded, not retried. It is a judgement about this copy, so
re-fetching every poll would be work done for ever to reach the same answer --
and the refusal is the record of which episodes speech recognition will have to
do. Refused, taken and never-looked-at are three states, kept apart here and in
the API, because conflating the first and third is the readout mistake this
project has already made twice.
Run against the live library before committing: 342 mirrored episodes, 57
offered a timestamped transcript, 38 taken, 19 declined -- and every decline is
a show with dynamic insertion (Heavyweight 10:03, Incels 11:54, Darknet Diaries
4:23).
Transcripts live one file per episode beside the audio, not in state.json: that
file is the only record of where mirrored audio is and is rewritten whole on
every label and every poll, and Darknet Diaries' "Conti" alone is 1784 cues.
They are pruned with the audio they describe.
Both spellings of the Podcasting 2.0 namespace are read. Every feed here
declares https:// today, but the specification was published under http:// and
Go matches a namespace by exact string -- so the wrong one would mean episodes
silently never getting words, with nothing failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The labels have been arriving and sitting there. Now the shadow run
acts on all three kinds.
A report is advertising pointed at where detection found nothing, so
it becomes a signature in its own right -- that being the whole gap
the library exists to close, since the scan accounts for about 38% of
what the feeds imply is there.
A mark saying a break is right adds no audio, because the break is
harvested anyway. What it adds is that a person vouched for it.
A mark saying a break is wrong does two things. It stops that audio
being harvested, and it takes back any signature already held that
matches it. That second half is the one that matters: a signature
built from programme would cut the programme in every episode carrying
it, until somebody noticed. So his "no" removes it outright rather
than lowering a score.
Being pointed at outranks partners and occasions both. Those are ways
of inferring what a stretch of audio is; he was there. A confirmed
signature is trusted at once and stays confirmed -- he does not
un-hear it next week.
A report whose end was never corrected is trimmed to its first fifteen
seconds. An uncorrected end is wherever the 25-second jump landed
rather than where the advertising stopped, so its tail may be
programme -- and one ad unit is already far more audio than a landmark
fingerprint needs to be distinctive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The first real shadow run died on its first batch. Two faults, and
the second is the one worth keeping.
store.Episode.Path is relative to the audio directory. Detection joins
it before handing it to the decoder and this did not, so ffmpeg went
looking in the working directory and found nothing.
Then that single failure aborted the whole show's learning. Every
other per-item loop in this server isolates one item's failure --
the refresh sweep, the poll loop, the detection pass all say so in
their own comments -- and this one propagated instead. An unreadable
episode is now counted and stepped over.
Counted once per episode rather than once per failure: a file that
cannot be read fails both the matching pass and the harvesting pass,
and reporting it twice would overstate the damage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The library now learns from every detection pass and says what it
would have done, without doing any of it. The log is the deliverable:
a few weeks of real batches answers whether it finds advertising or
accumulates junk, before one extra second is skipped from something
somebody is listening to.
It matches before it harvests, deliberately. Harvesting first would
let a batch's own signatures match the batch they came from, and every
run would report itself a triumph.
A detection pass is one occasion however many of its episodes carried
the audio -- they were mirrored within minutes and all ran whatever
campaign was running then. What a break's partner count does buy is
different: an advertisement is published only because the audio
recurred across episodes and correlation proved it, so a copy that
arrives with partners is already a confirmed repeat rather than a
single unsupported sighting, and is worth matching at once. Furniture
gets no such shortcut -- that is exactly the independence the withheld
intro and outro offsets are waiting for, and a sting shared by all
eight episodes turned out to be an advertisement once already.
Proposals are marked novel when detection did not already publish a
break covering them. Those are the only interesting ones: the library
earning its place rather than agreeing with what is already known.
They are logged individually, with their votes and how many occasions
and partners back them, so they can be listened to and judged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The spike proved landmark hashing separates a real repeat from a
coincidence; this is the library it unlocks. Rebuilt test-first rather
than lifted from the throwaway harness, against a fixture whose
partials shift several times a second -- a steady tone has only a
handful of peaks repeated endlessly and collides with itself far more
than any recording does, so it would have flattered the design.
Two kinds, which behave nothing alike. A cue is a show's own
furniture, belongs to one feed, and never expires: a break stinger is
not a campaign, and a show that goes quiet for a year comes back using
the same one. It goes when the subscription does. A copy is one
advertisement, belongs to nobody, and is forgotten six months after it
was last heard.
Confidence is a count of separate occasions, and a whole batch of
episodes mirrored in one minute is one occasion however many of them
carried the audio. That is the same independence problem that made
eight agreeing episodes prove nothing about an intro -- they all ran
whatever campaign was running that minute. Nothing is matched against
until it has been seen twice.
Signatures merge by matching rather than by identity, because the same
advertisement re-encoded by another CDN is not byte-identical and
would otherwise pile up as a new signature every week.
Stored as packed base64 rather than JSON arrays: three thousand hashes
per signature is 30KB as numbers and a quarter of that packed, and
this file gets parsed on every start. The index is deliberately not
stored, and is rebuilt on load -- skip that and every signature
silently stops matching after a restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The phone has had two kinds of label and nowhere to put them: a mark,
which is a judgement on a break detection found, and a report, which
is the notification button pointing at advertising it missed. They
catch opposite errors, and the signature library needs both.
POST /api/labels now takes them and state.json keeps them. Episodes
are named by feed and guid rather than by either side's own id -- the
phone never sees the server's, and the server's belongs to a mirror
that forgets an episode after eight. The labels sit beside the shows
rather than on them for the same reason: the corpus has to outlive the
audio it was given on.
A report's end is sent as guessed or measured. It is only measured
once the listener came back and trimmed it; otherwise it is wherever
the jump landed, and a signature cut to that would carry programme.
The phone re-sends everything each sweep and keeps no record of what
arrived. The server replaces rather than accumulating, so a re-send is
free, there is no sync state to get wrong, and a label lost to a crash
comes back on the next refresh. Gated on audioFromServer, like every
other server re-query: a label names a position in a particular file,
and dynamic ad insertion means the publisher's copy is a different one.
Accepting a label writes the state file immediately. Everything else
is written at the end of a poll cycle, which for 61 feeds is hours,
and a tap made once should not wait that long to become durable.
Deployed and checked against the live server: a label is accepted, a
re-send leaves one, an unauthenticated POST gets 401, and the label is
on disk within seconds. The probe label used for that check was
removed afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The server has computed and stored Partners since ad detection shipped --
state.json carries it on every break -- but breakRange never put it on the
wire, so the phone's new tap-a-break popup could only ever say "the server
did not say how many other episodes carried this audio". That count is the
one piece of evidence a listener can weigh before judging a break, and the
popup exists to show it.
My brief for the popup asserted the server already sent it. It did not.
The implementing agent checked rather than taking my word, which is why
this is a two-line fix now instead of a wrong "why" line forever.
Also lands the handover plan for the correction channel and the signature
library, written before a context compaction, including the gesture design
Peter specified for the notification button and the jump-size argument he
correctly rejected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Measured against the owner's 334 mirrored episodes, the symmetric
durationSlopMs rule from the previous round costs far more than it
protects: 31% of episodes are served shorter than their declared
length (feeds are sloppy by tens of seconds -- one show ran -47s,
-33s, +88s, -95s across episodes), and every break in every one of
them fell to the unjudgeable default and got labelled "Ad break" --
including closing credits at 37:41 of a 38:25 episode. The owner's
furniture toggle did nothing on nearly a third of his library.
You cannot insert advertising into a file and have it come out
shorter, so a negative gap is evidence that nothing was inserted,
not evidence the feed can't be trusted. Classify's insertion test is
one-sided again: inserted := servedMs<=0 || declaredMs<=0 ||
(servedMs-declaredMs) > adUnitMs. durationSlopMs (2000) is replaced
by adUnitMs (15000) -- advertising is sold in ~15s units, so an
overrun under one unit is the feed being approximate, not an
insertion. Both unjudgeable cases (servedMs<=0, declaredMs<=0) are
unchanged; furnitureKindAt's position logic is unchanged.
This reverses the FIX-1 decision in this same file that made the
test symmetric, on the theory that a feed wrong in the short
direction couldn't be trusted to judge the other direction either.
Rewrote the two pre-existing tests that encoded that decision
(TestAnEpisodeShorterThanDeclaredIsAlsoAllFurniture, folded as a
duplicate, and the renamed
TestAServedLengthShorterThanDeclaredByFarIsStillFurnitureByPosition)
plus one comment on an unchanged assertion, and added coverage for
the new threshold's band (a 10s overrun is now furniture, not ad).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Owner reported a 38-minute episode with 37 breaks, 14 of them exactly
4.0 seconds and scattered roughly one per minute through the whole
thing: "This is madness. Ads are usually multiples of 15 seconds,
grouped together in a few spots." He was right. DefaultRunOptions's
MinRunSamples: 1000 is exactly 4.0s at the 250Hz discovery rate, so
anything that long was taken seriously -- that floor, not a real ad,
is why 14 breaks landed on precisely 4.0s: correlation noise pinned
at the floor.
Measured across 333 mirrored episodes: episodes the feed proves carry
advertising (served >=60s longer than declared) have a median of 3
breaks, max 7. All six badly-fragmented episodes in the corpus belong
to one show (Sleepy History), which reuses long ambient music beds --
the detector was matching the programme's own texture against itself.
Fix, validated against the declared-vs-served length gap (a measure
the detector does not produce and so can't flatter itself with):
- MinRunSamples 1000 -> 3000 (4s -> 12s). 12 rather than 15 so a real
15s ad whose correlation lapses at one end still survives; both
floors were measured and 12 scored better.
- AdConfig.MergeGapMs 10_000 -> 60_000, so fragments of one real ad
pod separated by more than 10s bridge into the single break a
listener experiences, rather than staying fragmented.
Corpus-wide: 1091 -> 539 breaks, agreement with the length gap 31% ->
34%, unverifiable skipping on no-insertion shows 73min -> 64min. Half
the breaks removed and accuracy up is the signal they were noise.
Both constants' doc comments rewritten to state the new value and why,
not just the mechanics -- the comments are the documentation here.
TDD: added TestDefaultRunOptionsFloorIsTwelveSeconds (no pin existed
for MinRunSamples) and updated TestDefaultAdConfigUsesTheValidatedValues's
MergeGapMs pin; both confirmed red against the old constants before the
change. Three existing ads_test.go fixtures built at or under 10-12s
were lengthened to clear the new floor (not weakened). Added
TestScatteredSubFloorFragmentsProduceNoBreaksButARealPodStillDoes,
confirmed it fails against the old constants and passes against the
new ones.
gofmt, go vet, go test ./... all clean (server/ only -- no app/ changes).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Six fixes surfaced only by a whole-branch view of the kind/furniture work:
symmetric served-vs-declared comparison in Classify so a short-served
episode (e.g. 30:00 declared, 29:30 served) can no longer read a real
pre-roll as the show's own intro; a runCatching guard around the DataStore
read backing skipFurniture() so a corrupted preferences file can no longer
kill the media service at episode start; a proper isAdvertising predicate
on SkipSummary so the episode screen's ad/furniture colour split no longer
depends on comparing against a renamable display string; a stale comment
in PlayheadPolicy describing the pre-toggle behaviour; the already-unreachable
servedMs<=0 guard in furnitureKindAt now fails safe to KindAd instead of
KindCue; and an ORDER BY on getRanges so kind-filtered lookups are
deterministic. Throughout: anything unknown, empty or unjudgeable behaves
as advertising, never as furniture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The phone filters on it: advertising is always skipped, the show's own
audio by preference. Without the kind on the wire every break arrives
looking like an advertisement.
|
|
Finding 1: Treat zero or invalid servedMs as unjudgeable, falling back to
advertising classification to match the constraint that anything unjudgeable
must be conservative (ffprobe duration probe returning 0 is plausible upstream
defect).
Finding 2: Use proportional edge windows for short episodes (< 240s) to avoid
misclassifying interior and closing breaks as intro when opening and closing
windows overlap. For a short episode with breaks at 0s, 50s, 90s in a 100s
episode, classify them correctly as intro, cue, outro instead of all intro.
|
|
An episode no longer than its declared length had nothing inserted into
it, so whatever recurs in it is the show's furniture however strongly it
matched. 138 of 322 mirrored episodes are in that position and the scan
published ad breaks on 111 of them.
|
|
A pod of several advertisements is found as several matches, because
agreement lapses between them. Published unbridged, 45% of the gaps
between consecutive breaks came in under ten seconds and 152 of 262
multi-break episodes contained one -- which plays as a skip, a one-to-
nine-second burst of advertisement, then another skip. Bridging also
moves agreement with the declared length gap from 69% to 73% median,
because what was bridged was advertising too.
MaxBreakMs now applies to the matches rather than to what bridging makes
of them. Applied after bridging it refused five episodes that bridging
had pushed over the four-minute cap, Radiolab's Grass Apes among them --
a 3:55 pre-roll and an 0:18 tag two seconds apart fuse to 4:15, and the
episode the whole chain was device-verified on would have published
nothing. One enormous match is evidence of shared programme; a chain of
ordinary matches a beat apart is evidence of an ad pod. MaxSkipFraction
still guards the episode as a whole.
Also pins the ad scan's production values, which were measured rather
than picked, and records what probing Peter's theme-music observation
turned up: the cue is real and already detected, it marks a break's entry
but not its extent, a second fetch is not an ad-free reference, and 138
of 322 mirrored episodes contain no inserted advertising at all while the
scan reports breaks on 111 of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The first full pass over all 42 feeds finished, then the server kept
re-deriving the same answers: Radiolab reported "53:29 of 59:39" four
times in one day. pollOne called detect unconditionally and the mirror
skips files it already has, so nothing had changed between passes. At
~6.5 minutes a show that is 4.5 hours a cycle against a two-hour poll
interval -- the machine never finished a cycle and never idled, burning
13h42m of CPU in 11h of wall clock, roughly 60% of it on the intro search
whose results are deliberately not published.
Show.DetectedBatch records the episode ids a completed pass examined.
The batch is the unit rather than the episode because every offset and
every break comes from comparing an episode against the others fetched
alongside it, so one arrival or one pruning legitimately changes all of
them. Order-independent, since feeds are not reliably sorted. Recorded
only on completion, so a cancelled or failed pass is retried.
Episode.DetectedAtUnix could not serve as the gate: it is only set for
episodes appearing in result.Offsets, so an episode nothing was found in
never got a timestamp and the gate would never close.
Also records what the first full corpus showed, including a disproven
hypothesis worth not retrying -- filtering breaks by partner count or run
length looks obviously right and makes agreement with the declared length
gap substantially worse -- and a correction: Sleepy History's 117 minutes
of "advertising" was a show-level sum sitting almost entirely inside the
two episodes its own guard already refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The property that broke intro detection is the property that makes this work.
Eight episodes mirrored within one minute are maximally dependent, because the
ad server inserts whatever campaign is running into all of them identically.
For intro detection that destroys the independence the confirmation threshold
needs -- which is how it came to confirm an advertisement on 8 of 8 episodes.
For ad detection it is the signal.
And unlike intro detection it is checkable against a number the detector does
not produce: the feed declares the ad-free length, the file served is longer,
and the difference is the advertising.
**Measured on the live deployment, Radiolab's eight mirrored episodes: the scan
accounted for 53:29 of the 59:39 the length gap implies, with every individual
episode between 71% and 105%.** That is the first time detection in this
project has been checked against external ground truth and agreed with it.
The old sliding search could never have done this -- one full-length
correlation per window position, 199 seconds for a 240-second clip. BestLags
computes the cross-correlation over every lag in a single FFT pair, which says
where two recordings align; MatchedRuns then walks the extent of the agreement
directly, window by window, with no transform. Sized against measurements
rather than guesses: 2^22 is 1.69s, so a 70-minute episode decimated to 250Hz
is about five seconds a pair. Silence is explicitly not agreement, or separate
advertisements weld together across the pauses between them.
Breaks publish as ranges into episode_midrolls, which MidrollSkip and
PlayheadPolicy already read -- no playback changes and no migration. Intro and
outro offsets are still computed and logged but no longer published, and the
ad-derived ones already in the server's state were cleared.
Two guards publish nothing rather than something wrong: an episode whose
matches cover over 35% of it, and any single match over four minutes. A
re-aired episode or a two-parter would otherwise be eaten. Both record the
refusal, because "found nothing" and "refused what it found" are different
states and conflating them is the readout mistake over again.
Also three data-loss bugs, found by reading the journal after a restart rather
than by any test. The subscription list -- the one thing the server cannot
rediscover -- was persisted only when a whole poll cycle finished, which for 61
feeds is hours. A cycle re-applied its stale snapshot at the end, which would
undo a push that arrived during it and then delete the audio of every show the
snapshot omitted. And detection ignored cancellation, so systemd's 90-second
stop SIGKILLed the process before the save-on-exit could run. All three fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The server is Peter's design and it works: listend polls the phone's feed
list, mirrors the newest 8 episodes per show, detects across all of them at
once, and serves the offsets *and the audio they belong to*. Deployed on
tungsten behind Apache at https://listen.terst.org. FingerprintWorker and
MidrollWorker are deleted -- the battery drain is gone because the work is.
Serving the audio is load-bearing. Dynamic ad insertion turned out to be on
every episode: Grass Apes declares 22:05 and the server was served 28:18. An
offset measured on one copy is minutes wrong for another, so the phone has to
play the bytes the detector examined. The offsets land in the same columns the
on-device detector wrote, so playback needed no changes and there is no
migration.
**The headline finding is a failure, and it is proved rather than suspected.**
Detection reported the intro confirmed on 8 of 8 Radiolab episodes and passed
its self-check. It had locked onto an advertisement. The chosen segment was
searched for in an ad-free copy of the same episode fetched hours earlier:
r = 0.0621. It is in the ad break, not the programme.
The confirmation threshold assumes three *independently fetched* episodes.
Mirroring 8 within one minute from the same ad server makes them maximally
dependent -- whatever campaign is running is in all of them, identical -- and
SelfCheck cannot catch it, because the segments genuinely are the same audio.
Moving to a server fixed the cost problem and broke the independence the
safety mechanism rests on. The published offsets are head-window matches, so
they happen to skip pre-roll advertising, which is useful by accident and will
move whenever the campaign rotates. It is not intro detection and is not
described as such anywhere.
The fix, not made: record when each episode was fetched and require the
agreeing episodes to span more than one fetch occasion. Reasoning, and an
argument for detecting the ads directly instead, in docs/detection-findings.md.
Also recorded there: two tooling traps that each cost real time. `android-ui
tap` can land on the soft keyboard, because the dump reports layout
coordinates the IME does not move -- a Save button and a text field both
appeared tapped and neither fired. And the server's token is alphanumeric now
because `adb shell input text` ate the leading "-" of a base64url one and
produced a silent 401 that looked like broken wiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Peter's report: the phone runs hot, uses a lot of battery, playback takes a
moment to start, and intelligent skipping has never done anything.
**The hot phone and the dead detection were one bug.** The candidate search
correlated 117 window positions against the whole of the other clip,
re-transforming that unchanged clip every time -- 199 seconds per episode end,
measured on a desktop, so minutes more on a phone. FingerprintWorker is a plain
WorkManager worker, Android stops one at ten minutes, and nothing was persisted
until a comparison returned. So it was killed mid-flight, saved nothing, and was
re-enqueued to start over, forever. IntroOutroDetectorScaleTest is the tripwire
whose absence allowed this: the unit tests ran on 300-sample clips while
production ran on 1,920,000.
Fixed coarse-to-fine, plus AudioCorrelation.prepareTarget so the target clip is
transformed once per search rather than 117 times. 10.6x, and a test asserts the
chosen reference window is byte-identical to what a full-resolution search picks.
**Midroll detection could never have run at all.** decodeFull buffered a whole
episode at its native rate -- 434MB for a 13-minute episode, no largeHeap -- to
produce 6MB of 8kHz mono. It threw OutOfMemoryError into a swallowing
runCatching and the episode was then marked checked forever. MonoResampler now
streams the decode so the peak allocation is the output, and maxDecodableMs skips
a longer episode before spending anything and without marking it checked.
**Detection is moving to a server**, Peter's call, in Go to match podkit and
doot. server/ holds the ported core, verified three ways because this is the
second port of this maths and the first shipped a normalization bug: numpy's
literals extracted mechanically from AudioCorrelationTest, the FFT against a
direct DFT, and the whole correlation path against a brute-force Pearson.
On four real Radiolab episodes it confirms the intro on 4 of 4, at 48.0s, 8.65s,
8.65s and 65.15s -- the varying pre-roll, observed in working code at last.
skips.SelfCheck verifies located segments against *each other* rather than
against the reference, which is a different claim: correlation is not transitive,
so two segments can each clear 0.6 against a reference and be unrelated. It
rejected the outro on its first real run (r=0.746), withholding offsets that
would have ended three episodes early. Offsets are published only when they
verify.
Also: the episode screen gets a working scrubber with ad-break marks and a
declared-versus-measured length readout, queue rows open the episode instead of
doing nothing, PrepareTrace logs playback-start timing against episode length
for the still-open pause question, and -PphoneOnly drops the emulator's CPython
from a sideload build (114MB -> 99MB).
Full history and numbers in docs/detection-findings.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
