| Age | Commit message (Collapse) | Author |
|---|
|
Tier said "how important" and the shows needed "what kind". Seven ordered
categories replace it: news, analysis, curiosity, sailing, true stories,
fiction, deep dives. The order is still the server's processing priority, so
one field carries both axes -- which is the idiosyncrasy docs/show-kinds.md
is written to watch, with four signs that a better abstraction has arrived.
All 42 known feeds are refiled by migration 20 -> 21, which rebuilds the
subscriptions table rather than renaming the column: RENAME COLUMN keeps the
old 'ordinary' DEFAULT and fails schema validation, and DROP COLUMN needs
SQLite 3.35 against minSdk 29's 3.28. The wire value for true stories is
"truestory", not "true", because SQLite reads an unquoted true in a DEFAULT
as the boolean and stores 1.
Schema 21 is a new exported JSON; migrate20To21RenamesTierAndRefilesTheKnownShows
covers it, and all 22 device migration tests pass.
The coverage exclusion for the settings composable follows the file's rename;
left pointing at TierSettingKt it read as a 0.6% drop. The remaining 0.08% is
the deletion of the covered tier-seeding code, taken as an exception.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Two changes to the server, both measured against the live data rather than reasoned about.
**Subtraction publishes only from a publisher's transcript.** Subtracting a publisher's transcript
recovers 94-104% of the advertising the file lengths imply; subtracting a refetch -- a second
download of the same assembled file, carrying its own ads in its own places -- recovers a median of
33%. That tier had already been tried and rejected at 21% and 0%, but nothing in the publish path
was told which master it had, so it reached the phone anyway on seventeen ad spans.
Written as a denylist first, and measuring it against a copy of the live data showed why that was
the wrong shape: sixteen episodes carry a master whose label is blank -- the field postdates the
first masters written, and Reclaim restored those files after the library was destroyed with the
blank intact -- and they published on the strength of not being the one named bad kind. They
happened to be publisher transcripts, so the answer was right by luck. It is an allowlist now, and
the blank labels are repaired from the master files, which have said "published" all along.
Measured on a copy, not predicted:
before 49 ad spans (17 refetch, 24 unlabelled, 8 published) + 604 cue, 245 episodes
after 62 ad spans (all publisher-master) + 611 cue, 245 episodes
No episode loses its skips; the refetch ones trade advertising nobody could measure for the show's
own recurring audio, which is a preference rather than an unconditional skip. Measuring also caught
the recurring pass blanking the refusal subtraction had just written -- both passes answer "why is
nothing published" through one field and recurring runs second, so all thirty-two episodes that now
carry a reason were losing it a moment later.
**Being asked for keeps an episode mirrored.** Retention was publication date alone, and pruning
calls words.Forget, which deletes the transcript *file*. So an episode outside the newest eight was
fetched on demand, recognised, served, and deleted on the next poll -- and asking again paid the
whole cost over. From the server's own journal over twenty-one days: 397 transcription runs
covering 309 distinct episodes, with 7.7 of the 33.4 hours of whisper time spent on episodes it had
already recognised and thrown away. 23% of the budget, re-doing work it had discarded.
Working through a back catalogue -- the archive tier, the slow-drip shows, the listening this was
built for -- therefore paid full recognition for every episode and again to resume one. The window
still decides what is worth fetching; RequestedAtUnix now decides what is worth keeping, stamped on
every /api/episode call including for episodes already mirrored, because resuming something played
last month must not let it fall out mid-listen. Kept for 28 days after the last ask, the same span
refetchAfterSeconds uses, being the same kind of judgement about how long something stays
interesting.
Disk points the same way: transcripts are 36MB for 445 files, audio is 17GB for 336 episodes, and
the volume has 365GB free. The expensive artefact is the small one.
374 server tests pass. Not yet deployed; the phone's half of this -- a pending reply is retried
only on a feed refresh -- is still to come.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The findings doc's last entry was 2026-10-19 and ended by naming what was missing: "a mechanism
that can tell a real short ad from a short coincidence". Words are that mechanism, it was built,
and thirty-two server commits later the doc never recorded what it measured. A future session
would have re-attempted the ad/furniture classifier that was already measured and deleted.
The new entry is derived from the live state file rather than quoted, which is how it caught that
publish.go's own figure describes a subset. "92% to 104% across eighteen episodes" is the
publisher-transcript number. Across the 24 episodes now publishing an ad span, recovery runs 20%
to 146%. Split by where the master came from, the spread stops being a spread: a publisher's
transcript gives 94-104%, a refetch gives a median of 33%. Nothing in Publishable looks at which
it has, and refetch masters are now fifteen of those twenty-four -- the majority of a tier the
design had decided not to act on.
It also caught me getting a conclusion wrong. Two Reply All episodes publish ad spans at 146% and
138% of their length gap, which reads as the detector over-claiming by half a minute of
programme. Looking at the show instead of the episode: Reply All's gaps run from -0.48 to +1.02
minutes across eight episodes. They straddle zero, so the feed's declared durations are not a
baseline for that show and the thing being exceeded is noise. The structural finding is the useful
one -- HasInsertedAdvertising asks only whether this episode's gap clears one ad unit, and has no
notion of whether the show's gaps are consistent enough to mean anything. Both versions are in the
entry, because the wrong one is the one a reader would reach for next time.
Nothing here changes behaviour. Whether a refetch should publish an ad at all is Peter's call and
is written up as such.
CLAUDE.md, separately:
- It still said the episode description shows about two lines. That was changed to ten yesterday,
for a reason that is recorded: the limit existed to stop long notes pushing the transport
off-screen, and the merge had already put the notes below the transport. Left as it was, the
doc was an instruction to reintroduce the bug.
- "207 unit tests" has been wrong for most of the project's life; the suite is past nine hundred.
Replaced with a note to ask the wrappers, since any number written here goes stale the same way.
- The TDD rule banned backfilling tests onto existing code without qualification, which also
forbids the one case where it is right -- pinning behaviour before moving it, so a refactor is
provably behaviour-preserving. Narrowed to say that, and to say the order matters. The sixteen
formatter tests in 2cc252a were written that way.
- Added the rule those formatter files were moved to satisfy: anything the app says out loud lives
in a named file of its own and is tested, never inside the composable that renders it. It was
already followed by eight files and stated only in WaitingForServerText's own comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Verified on the live server: a mark vouching for a real detected break
reaches the signature that break produced and sets confirmed on it.
The probe and the flag it set were removed afterwards; that corpus
holds the listener's statements, not test data.
Also records the property that cost two runs to notice -- a label is
only actionable while its episode is still mirrored. They outlive the
audio deliberately, so the server holds plenty naming episodes long
rotated out, and those are simply not applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The labels have been arriving and sitting there. Now the shadow run
acts on all three kinds.
A report is advertising pointed at where detection found nothing, so
it becomes a signature in its own right -- that being the whole gap
the library exists to close, since the scan accounts for about 38% of
what the feeds imply is there.
A mark saying a break is right adds no audio, because the break is
harvested anyway. What it adds is that a person vouched for it.
A mark saying a break is wrong does two things. It stops that audio
being harvested, and it takes back any signature already held that
matches it. That second half is the one that matters: a signature
built from programme would cut the programme in every episode carrying
it, until somebody noticed. So his "no" removes it outright rather
than lowering a score.
Being pointed at outranks partners and occasions both. Those are ways
of inferring what a stretch of audio is; he was there. A confirmed
signature is trusted at once and stays confirmed -- he does not
un-hear it next week.
A report whose end was never corrected is trimmed to its first fifteen
seconds. An uncorrected end is wherever the 25-second jump landed
rather than where the advertising stopped, so its tail may be
programme -- and one ad unit is already far more audio than a landmark
fingerprint needs to be distinctive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The dependencies for this were already in the build -- ui-test-junit4
on androidTest, ui-test-manifest on debug -- and had never been used.
So the spike was one test, and it passed first run. Eighteen now.
What they cover is the data-driven composables, chosen because a
regression in them looks completely normal: a readout that quietly
stops printing one of its three numbers, a control that stops
reporting whether it will play or pause, a bar that stops naming what
is playing while a screen shows something else.
The episode transport: the ad button is offered only while something
is playing, because with nothing playing there is no advertising to be
in the middle of and an enabled button would record a claim about a
position nobody is at. The primary control says what pressing it will
do. The skip buttons move by the amount they advertise.
The skip readout: all three numbers on one line, the status when there
is nothing to cut -- "nothing found" and "not checked yet" being
different answers is the mistake the old readout made -- and a legend
that names only what is actually on the track.
The now-playing bar: names the episode and the show, opens what is
playing when tapped, and reports play against pause.
Screens that need view models are still verified by eye. This is the
half that did not need them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
401 and 403 are permanent by decision -- a refused token cannot fix
itself -- and the notice saying so was held in memory. So it vanished
with the process and only came back once the next refresh had gone and
been refused all over again, which for 61 feeds is hours of the app
quietly downloading from publishers with nothing on screen to say why.
It now lives in the preferences file and is there at launch.
Saving different credentials clears it at once rather than waiting for
a request to prove it, because that is the one act that answers the
complaint. Saving the same ones again proves nothing and leaves the
notice standing.
Behind a small interface so SkipServer does not have to know whether
it is writing to a preferences file or to nothing: every test wants it
in memory, without a Context to hang a DataStore off.
The stored one takes the reading and the writing as two narrow seams
rather than a Preferences. Passing Preferences dragged a Context in
and made it the one piece of the server client that could only be
exercised on a device -- the coverage gate caught that, which is what
it is for. What it actually decides, when *not* to write, is now plain
Kotlin with five tests on it: clearing runs after every successful
reply, so two hundred successes in a sweep must write nothing.
Verified on a device: a refusal written by one Preferences instance is
read back by another, and four cases around saving credentials behave
as described.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Records what moved and why, including the two controls that left
Settings -- speed because the episode screen already had it, the sleep
timer because it is an action rather than a setting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Four and five are done and six is deliberately not. Everything six
needs exists; what is missing is weeks of real batches, and building
it now would throw away the only reason the shadow run exists.
Records what to watch for in the log, and the one decision the tests
forced: a copy that arrives with partners is trusted at once, because
holding it back would mean only ever proposing advertising detection
had already found twice. A cue gets no shortcut, which is the
independence intro and outro are waiting for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The phone has had two kinds of label and nowhere to put them: a mark,
which is a judgement on a break detection found, and a report, which
is the notification button pointing at advertising it missed. They
catch opposite errors, and the signature library needs both.
POST /api/labels now takes them and state.json keeps them. Episodes
are named by feed and guid rather than by either side's own id -- the
phone never sees the server's, and the server's belongs to a mirror
that forgets an episode after eight. The labels sit beside the shows
rather than on them for the same reason: the corpus has to outlive the
audio it was given on.
A report's end is sent as guessed or measured. It is only measured
once the listener came back and trimmed it; otherwise it is wherever
the jump landed, and a signature cut to that would carry programme.
The phone re-sends everything each sweep and keeps no record of what
arrived. The server replaces rather than accumulating, so a re-send is
free, there is no sync state to get wrong, and a label lost to a crash
comes back on the next refresh. Gated on audioFromServer, like every
other server re-query: a label names a position in a particular file,
and dynamic ad insertion means the publisher's copy is a different one.
Accepting a label writes the state file immediately. Everything else
is written at the end of a poll cycle, which for 61 feeds is hours,
and a tap made once should not wait that long to become durable.
Deployed and checked against the live server: a label is accepted, a
re-send leaves one, an unauthenticated POST gets 401, and the label is
on disk within seconds. The probe label used for that check was
removed afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The plan made this the gate: a linear scan of N stored signatures
against each episode does not survive, so the library is only possible
if an index can shortlist candidates cheaply. It can.
Built as a throwaway harness against the live corpus. The episodes the
plan named had rotated out of the server's 8-episode window, so the
test set was rebuilt from current audio and came out stronger than the
one it replaces -- 6 true repeats and 16 non-repeats, against 2 and 3.
Six repeats scored 813-1564 votes; sixteen non-repeats scored 27-94.
The narrowest separation is 8.6x, where the correlator it replaces
manages 4.9x on the same audio. Nothing sits in the gap.
Offsets land within 1.3s and every proposal is cheap to confirm by
correlating against a short window at the claimed position, so the
shape is: index proposes, correlator disposes. That is not a
formality. In one episode the advertisement occurs twice and the index
picked the weaker instance, which only a confirming pass would catch.
Measured: 9.8ms to hash a signature when it is stored, 196ms to look
one up against 23 episodes, ~8s per episode to index, 353MB for 23.
Five hundred signatures cost 98 seconds where linear correlation costs
about 63 minutes.
The harness is not committed -- it read absolute server paths and had
no tests, and its output was an answer rather than code to keep. The
parameters it found are written into the plan so it can be rebuilt
test-first, along with the one weakness worth knowing: the hashes
collide heavily, and that density is what makes a lookup cost
milliseconds rather than microseconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The "this is an ad" press jumped 45 seconds, a number argued from
ad-unit arithmetic rather than from the controls that already exist.
Peter: "the jump should stay 25 seconds. Don't go inventing new
timing."
Skipping forward is already what you do to an advertisement, so the
button goes the same distance and additionally says why. One number to
know, and a second press goes the same distance again.
The amount had three definitions already -- the player's increments,
the episode screen's transport buttons, and the notification caption's
hardcoded "Forward 25 seconds" -- and this nearly became a fourth with
a different value. They now all read one, and the captions are derived
from it so a caption cannot name a number its button does not do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The notification carried four buttons doing two jobs: Media3's default
previous/next (which PodcastPlayer redefines as seek) plus our own
back-15/forward-25 pair. Media button preferences with explicit slots
let the custom pair claim the two seek slots, so the duplicates go and
the freed space carries "this is an ad". No player command is removed
to achieve it, which is the route that once silently killed the
headphone skip button.
One press means four things, told apart by where the playhead lands
rather than by guessing: press claims advertising from here and jumps
past it, pressing again extends the same claim, coming back inside the
jump moves its end to where you landed, and coming back past the press
withdraws it. Outside a 20-second window a seek is ordinary navigation
and says nothing. The window is stored with the label so changing it
later cannot reinterpret old data.
Every backwards seek feeds the gesture, whatever caused it --
notification, headphone remote, in-app bar, dragged slider -- because
you correct with whatever is to hand.
This is the opposite evidence to the tap-a-break popup. That one can
only correct breaks detection already found, and detection accounts for
about 38% of the advertising the feeds imply.
Verified on the emulator: the notification now shows exactly back-15,
pause, forward-25 and the new button; the lock screen shows it too and
pressing it there records a claim without unlocking; all four gesture
arms behave as specified; and the media transport keys still seek by
25s and 15s. Android Auto still needs the car.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Keeps partners on episode_midrolls (schema 15), stores verdicts in their own
break_verdicts table so a server refresh cannot wipe them, and drops invalidated
breaks from skipping on the next tick. Verdicts are local only.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The 12s floor cut real detection on shows with verified ads -- LeVar
Burton Reads fell from 19.0 to 10.5 minutes found against a 26.6 minute
gap -- so 8s looked like the fix. Measured on an isolated copy of the
live data it is not: Sleepy History at 8s finds 174 minutes of
"advertising" in a show that has 48 seconds of it, worse than the 4s
floor that produced Peter's original complaint.
MinRunSamples and MergeGapMs interact and cannot be tuned separately. A
low floor lets noise fragments survive and the 60s bridging then welds
them into enormous spans.
Had only LeVar been measured the floor would have been lowered.
Also re-pins .coverage-baseline, which the hook left modified after
d96c876 deleted covered readout code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Eight tasks covering the spec decisions end to end: the server decides a
kind for every break it publishes, the phone stores it, advertising is
always skipped and a show own intro, cue and credits follow one
preference, and the readout names which is which.
The cue prior and the furniture library are deliberately not planned
here. Each is independently shippable, and the cue signature thresholds
should be chosen against kinds visible in real data rather than guessed
at before any exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Approach 2 chosen, built in the phase order already proposed -- it is a
superset of approach 1, so the ad-detection work still lands first and the
library last.
Advertising is always skipped; the show own intro, cue and credits obey a
single preference defaulting to on, because that is what happens today.
The toggle settles a bootstrapping question the design had left open. A
furniture clip cannot be promoted until a second fetch occasion has
passed, so until then there is nothing to label with. The zero-gap
classification fills that gap: served length equal to declared means
nothing was inserted, so every break is furniture and position decides
which kind; otherwise a break is advertising unless a promoted clip says
otherwise. Day one is correct on the 138 zero-gap episodes and identical
to current behaviour everywhere else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three defects turn out to be one missing distinction. Ad detection finds
a median 67% of the advertising the length gap implies, because it can
only see what two simultaneously-mirrored episodes share. It reports a
show's own intro, break cue and credits as advertisements on 111 of the
138 episodes that had nothing inserted into them. And intro/outro
offsets stay withheld for want of independent evidence.
Peter's observation about theme music around ad breaks is the clue, and
probing it showed both what it gives and what it does not: the cue is
real, already detected, and marks a break's entry but not its extent --
scanning to r=0.6 finds one isolated hit per slot in ad-laden and ad-free
copies alike, so there is no closing cue to bracket against.
Writing this up changed the recommendation. The furniture library is only
needed for the intro/outro problem; the mislabelling falls out exactly
and for free from served length versus declared, and cues are
identifiable from their signature within a single batch. So the proposal
is now the cheap approach first, with the library kept for later and only
if the withheld offsets are wanted.
Nothing here is built. Review document, not an implementation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
A pod of several advertisements is found as several matches, because
agreement lapses between them. Published unbridged, 45% of the gaps
between consecutive breaks came in under ten seconds and 152 of 262
multi-break episodes contained one -- which plays as a skip, a one-to-
nine-second burst of advertisement, then another skip. Bridging also
moves agreement with the declared length gap from 69% to 73% median,
because what was bridged was advertising too.
MaxBreakMs now applies to the matches rather than to what bridging makes
of them. Applied after bridging it refused five episodes that bridging
had pushed over the four-minute cap, Radiolab's Grass Apes among them --
a 3:55 pre-roll and an 0:18 tag two seconds apart fuse to 4:15, and the
episode the whole chain was device-verified on would have published
nothing. One enormous match is evidence of shared programme; a chain of
ordinary matches a beat apart is evidence of an ad pod. MaxSkipFraction
still guards the episode as a whole.
Also pins the ad scan's production values, which were measured rather
than picked, and records what probing Peter's theme-music observation
turned up: the cue is real and already detected, it marks a break's entry
but not its extent, a second fetch is not an ad-free reference, and 138
of 322 mirrored episodes contain no inserted advertising at all while the
scan reports breaks on 111 of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The first full pass over all 42 feeds finished, then the server kept
re-deriving the same answers: Radiolab reported "53:29 of 59:39" four
times in one day. pollOne called detect unconditionally and the mirror
skips files it already has, so nothing had changed between passes. At
~6.5 minutes a show that is 4.5 hours a cycle against a two-hour poll
interval -- the machine never finished a cycle and never idled, burning
13h42m of CPU in 11h of wall clock, roughly 60% of it on the intro search
whose results are deliberately not published.
Show.DetectedBatch records the episode ids a completed pass examined.
The batch is the unit rather than the episode because every offset and
every break comes from comparing an episode against the others fetched
alongside it, so one arrival or one pruning legitimately changes all of
them. Order-independent, since feeds are not reliably sorted. Recorded
only on completion, so a cancelled or failed pass is retried.
Episode.DetectedAtUnix could not serve as the gate: it is only set for
episodes appearing in result.Offsets, so an episode nothing was found in
never got a timestamp and the gate would never close.
Also records what the first full corpus showed, including a disproven
hypothesis worth not retrying -- filtering breaks by partner count or run
length looks obviously right and makes agreement with the declared length
gap substantially worse -- and a correction: Sleepy History's 117 minutes
of "advertising" was a show-level sum sitting almost entirely inside the
two episodes its own guard already refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The property that broke intro detection is the property that makes this work.
Eight episodes mirrored within one minute are maximally dependent, because the
ad server inserts whatever campaign is running into all of them identically.
For intro detection that destroys the independence the confirmation threshold
needs -- which is how it came to confirm an advertisement on 8 of 8 episodes.
For ad detection it is the signal.
And unlike intro detection it is checkable against a number the detector does
not produce: the feed declares the ad-free length, the file served is longer,
and the difference is the advertising.
**Measured on the live deployment, Radiolab's eight mirrored episodes: the scan
accounted for 53:29 of the 59:39 the length gap implies, with every individual
episode between 71% and 105%.** That is the first time detection in this
project has been checked against external ground truth and agreed with it.
The old sliding search could never have done this -- one full-length
correlation per window position, 199 seconds for a 240-second clip. BestLags
computes the cross-correlation over every lag in a single FFT pair, which says
where two recordings align; MatchedRuns then walks the extent of the agreement
directly, window by window, with no transform. Sized against measurements
rather than guesses: 2^22 is 1.69s, so a 70-minute episode decimated to 250Hz
is about five seconds a pair. Silence is explicitly not agreement, or separate
advertisements weld together across the pauses between them.
Breaks publish as ranges into episode_midrolls, which MidrollSkip and
PlayheadPolicy already read -- no playback changes and no migration. Intro and
outro offsets are still computed and logged but no longer published, and the
ad-derived ones already in the server's state were cleared.
Two guards publish nothing rather than something wrong: an episode whose
matches cover over 35% of it, and any single match over four minutes. A
re-aired episode or a two-parter would otherwise be eaten. Both record the
refusal, because "found nothing" and "refused what it found" are different
states and conflating them is the readout mistake over again.
Also three data-loss bugs, found by reading the journal after a restart rather
than by any test. The subscription list -- the one thing the server cannot
rediscover -- was persisted only when a whole poll cycle finished, which for 61
feeds is hours. A cycle re-applied its stale snapshot at the end, which would
undo a push that arrived during it and then delete the audio of every show the
snapshot omitted. And detection ignored cancellation, so systemd's 90-second
stop SIGKILLed the process before the save-on-exit could run. All three fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The server is Peter's design and it works: listend polls the phone's feed
list, mirrors the newest 8 episodes per show, detects across all of them at
once, and serves the offsets *and the audio they belong to*. Deployed on
tungsten behind Apache at https://listen.terst.org. FingerprintWorker and
MidrollWorker are deleted -- the battery drain is gone because the work is.
Serving the audio is load-bearing. Dynamic ad insertion turned out to be on
every episode: Grass Apes declares 22:05 and the server was served 28:18. An
offset measured on one copy is minutes wrong for another, so the phone has to
play the bytes the detector examined. The offsets land in the same columns the
on-device detector wrote, so playback needed no changes and there is no
migration.
**The headline finding is a failure, and it is proved rather than suspected.**
Detection reported the intro confirmed on 8 of 8 Radiolab episodes and passed
its self-check. It had locked onto an advertisement. The chosen segment was
searched for in an ad-free copy of the same episode fetched hours earlier:
r = 0.0621. It is in the ad break, not the programme.
The confirmation threshold assumes three *independently fetched* episodes.
Mirroring 8 within one minute from the same ad server makes them maximally
dependent -- whatever campaign is running is in all of them, identical -- and
SelfCheck cannot catch it, because the segments genuinely are the same audio.
Moving to a server fixed the cost problem and broke the independence the
safety mechanism rests on. The published offsets are head-window matches, so
they happen to skip pre-roll advertising, which is useful by accident and will
move whenever the campaign rotates. It is not intro detection and is not
described as such anywhere.
The fix, not made: record when each episode was fetched and require the
agreeing episodes to span more than one fetch occasion. Reasoning, and an
argument for detecting the ads directly instead, in docs/detection-findings.md.
Also recorded there: two tooling traps that each cost real time. `android-ui
tap` can land on the soft keyboard, because the dump reports layout
coordinates the IME does not move -- a Save button and a text field both
appeared tapped and neither fired. And the server's token is alphanumeric now
because `adb shell input text` ate the leading "-" of a base64url one and
produced a silent 401 that looked like broken wiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Peter's report: the phone runs hot, uses a lot of battery, playback takes a
moment to start, and intelligent skipping has never done anything.
**The hot phone and the dead detection were one bug.** The candidate search
correlated 117 window positions against the whole of the other clip,
re-transforming that unchanged clip every time -- 199 seconds per episode end,
measured on a desktop, so minutes more on a phone. FingerprintWorker is a plain
WorkManager worker, Android stops one at ten minutes, and nothing was persisted
until a comparison returned. So it was killed mid-flight, saved nothing, and was
re-enqueued to start over, forever. IntroOutroDetectorScaleTest is the tripwire
whose absence allowed this: the unit tests ran on 300-sample clips while
production ran on 1,920,000.
Fixed coarse-to-fine, plus AudioCorrelation.prepareTarget so the target clip is
transformed once per search rather than 117 times. 10.6x, and a test asserts the
chosen reference window is byte-identical to what a full-resolution search picks.
**Midroll detection could never have run at all.** decodeFull buffered a whole
episode at its native rate -- 434MB for a 13-minute episode, no largeHeap -- to
produce 6MB of 8kHz mono. It threw OutOfMemoryError into a swallowing
runCatching and the episode was then marked checked forever. MonoResampler now
streams the decode so the peak allocation is the output, and maxDecodableMs skips
a longer episode before spending anything and without marking it checked.
**Detection is moving to a server**, Peter's call, in Go to match podkit and
doot. server/ holds the ported core, verified three ways because this is the
second port of this maths and the first shipped a normalization bug: numpy's
literals extracted mechanically from AudioCorrelationTest, the FFT against a
direct DFT, and the whole correlation path against a brute-force Pearson.
On four real Radiolab episodes it confirms the intro on 4 of 4, at 48.0s, 8.65s,
8.65s and 65.15s -- the varying pre-roll, observed in working code at last.
skips.SelfCheck verifies located segments against *each other* rather than
against the reference, which is a different claim: correlation is not transitive,
so two segments can each clear 0.6 against a reference and be unrelated. It
rejected the outro on its first real run (r=0.746), withholding offsets that
would have ended three episodes early. Offsets are published only when they
verify.
Also: the episode screen gets a working scrubber with ad-break marks and a
declared-versus-measured length readout, queue rows open the episode instead of
doing nothing, PrepareTrace logs playback-start timing against episode length
for the still-open pause question, and -PphoneOnly drops the emulator's CPython
from a sideload build (114MB -> 99MB).
Full history and numbers in docs/detection-findings.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Peter's call (option 1 of the three in docs/detection-findings.md): exempt
an episode retention would otherwise reclaim while it is a show's pending
comparison basis, rather than extending the grace period or accepting the
gap.
RetentionSweeper gains a pendingEpisodeIds seam (suspend () -> Set<String>,
defaulting to empty) and filters its reclaim candidates against it before
the grace-period policy ever sees them -- every existing caller and test
was unaffected by the default. RetentionWorker wires the real thing: the
union of two new, narrow DAO queries --
ShowFingerprintDao.getAllPendingEpisodeIds() (intro/outro) and
MidrollDao.getAllMidrollPendingEpisodeIds() (midroll, reading
SubscriptionEntity.midrollPendingEpisodeId).
Applied to both fingerprinting mechanisms, not just the one that got
diagnosed: MidrollUpdater's discovery has the identical rolling-pending-
episode shape and would race retention the same way for a fast show, just
silently, with no frozen row to reveal it.
Not yet re-verified on a device -- the next live test is two Up First
downloads close enough together to actually compare, confirming
pendingEpisodeId advances to a real second episode instead of restarting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA
|
|
Triggered a fresh Up First download on the phone and watched
androidx.work.workdb and show_fingerprints immediately after, live. A new
FingerprintWorker job appeared and succeeded within seconds, and
show_fingerprints updated for the first time since 2026-09-27 -- but instead
of advancing toward a second comparison, it just replaced the stale pending
episode with the new download. confirmedCount stayed 0.
That's checkAgainstPending's own designed fallback when pendingClipOf can't
re-decode the stored pending episode because its file is gone. And it is
gone: RetentionPolicy's grace period is one day, and Up First is a fast,
daily, ~15-30 minute news show that gets played through quickly. By the time
a second episode is downloaded and processed, retention has very often
already reclaimed the first one's file -- so the detector never gets two
episodes' audio to compare at once, and keeps discarding its only candidate
and restarting from whatever the newest download is. Cruising the Waves
reaching a real Tentative candidate the same day is consistent with this: a
show whose downloads happen to land further apart, or whose previous pending
episode hadn't been played+reclaimed yet, doesn't hit the race.
This was never a signal-processing problem -- DEFAULT_REFERENCE_SECONDS,
MIN_SCORE and CONFIRM_THRESHOLD were the wrong place to look the whole time.
Wrote up three options (exempt a pending episode from reclaim, extend the
grace period for a show still building its first candidate, or accept the
gap for fast shows) as a decision for Peter, not something to just change.
Separately unresolved and now secondary: Up First's actual second download
(2026-09-28) left zero trace in show_fingerprints, which the live test's
fallback path would have if it had run at all -- most likely a deferred or
dropped one-shot WorkManager job, not reproduced live this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA
|
|
The phone reconnected mid-session, so this moved from desktop simulation to
live device state -- pulled listen.db and WorkManager's own execution
history straight off it (read-only, via run-as + adb exec-out cat, nothing
modified).
The pipeline is not broken: show_fingerprints for Cruising the Waves Podcast
shows a real Tentative INTRO candidate (confirmedCount=2, a populated
referenceClipPcm) from an ordinary download today. Something does reach
Tentative on this exact build.
Up First specifically is stuck, and the evidence is more precise than
"nothing found yet" suggested. Its show_fingerprints row for both INTRO and
OUTRO hasn't updated since 2026-09-27T02:19 -- the moment its very first
episode was ever processed. A second real episode was downloaded, played,
and swept by retention the next day, and its midrollCheckedAtMs is set,
proving MidrollWorker's decodeFull read that exact file successfully that
day. But FingerprintUpdater.processEnd calls fingerprints.upsert() on every
outcome, confirmed, tentative, or discarded -- so if any comparison
involving that second episode had run at all, the row's updatedAtMs would
have moved. It didn't. The second episode's FingerprintWorker run either was
never enqueued, or decodeHead/decodeTail both returned null for it, and the
latter looks unlikely given decodeFull succeeded on the identical file the
same day. Read EpisodeDownloader.downloadOverHttp end to end looking for a
download-vs-playable race; there isn't one -- it's a blocking download to
.part then an atomic rename before onDownloaded fires.
WorkManager prunes completed WorkSpec rows after about a week, so the actual
historical FingerprintWorker run (if any) for that episode is already gone.
Narrows "What to check on the phone" to one live test: trigger a fresh Up
First download and watch androidx.work.workdb and show_fingerprints
immediately after, the same way this session just did.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA
|
|
The brief's leading hypothesis -- an 8-second reference window too wide for
a ~6-second sting -- does not survive contact with real data. Simulating
IntroOutroDetector.findCandidate exactly (the same normalized_xcorr/best_match
math AudioCorrelation.kt uses, unmodified) against the four Up First episodes
already cached from the original prototype: every one of six pairs confirms
cleanly at both ends, at the app's real DEFAULT_REFERENCE_SECONDS=8 and
MIN_SCORE=0.6 -- intro r 0.79-0.998, outro r 0.994-0.999.
Also ruled out: the naive linear-interpolation resampler MediaCodecClipSource
actually uses. Reproduced it exactly against a fresh re-fetch's native 44.1kHz
audio and compared to a proper ffmpeg decode -- 0.9991 correlation, and the
production simulation confirms identically either way.
Two findings beyond the original brief:
- Pre-roll length varies a lot (44s-94s across four episodes) but the global
search already handles that; it isn't the blocker the first draft worried
it might be.
- The same episode URL, fetched fresh today against a copy from 2026-09-26,
is byte-identical for the first ~50 seconds and then diverges completely --
sharper evidence for "dynamic ad insertion is the environment" than the
Radiolab duration mismatch alone, though it lands after the useful intro
signal so doesn't threaten detection specifically.
Experiment 6 (re-check the Darknet Diaries trap) didn't need a new fetch --
the midroll prototype session's already-cached data answers it: the shared
~51s segment still matches at r=1.000 across a nine-year gap, which a truly
rotating per-request ad could not do by chance. Looks like a real intro, not
the trap the threshold exists to catch.
Net effect: the algorithm and its tuned constants are no longer plausible
suspects on real audio. What's left points at the specific on-device run
rather than the approach -- see "What to check on the phone" for the three
concrete things to look at that this sandbox has no way to check itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sj3CoWEedM56bebByKgNxA
|
|
The fingerprinting runs on a device now and has still confirmed nothing for
a real show. That is an open question, not a known bug, and answering it
needs the prototype rather than the app -- only the prototype can show what
a parameter change costs in false positives, which is the number that
matters here.
docs/detection-findings.md collects what this round established: that the
pipeline works, that the confirmation threshold earns its place, and the
finding that reframes the problem -- dynamic ad insertion means the same
Grass Apes came back 29:34 on one fetch and 27:39 on another, so ad content
and the offset of everything behind it differ per request. It lists the
experiments in order, including using insertedAdMs as ground truth for the
midroll scan, and what not to touch.
Linked from the known-gaps list, which is where a future session looks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
