From bc33a9bdee12e41ed8b017c3f751a5dcdf6d1c97 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Sat, 24 Oct 2026 08:23:02 +0000 Subject: Spike: landmark hashing passes the gate for the signature library The plan made this the gate: a linear scan of N stored signatures against each episode does not survive, so the library is only possible if an index can shortlist candidates cheaply. It can. Built as a throwaway harness against the live corpus. The episodes the plan named had rotated out of the server's 8-episode window, so the test set was rebuilt from current audio and came out stronger than the one it replaces -- 6 true repeats and 16 non-repeats, against 2 and 3. Six repeats scored 813-1564 votes; sixteen non-repeats scored 27-94. The narrowest separation is 8.6x, where the correlator it replaces manages 4.9x on the same audio. Nothing sits in the gap. Offsets land within 1.3s and every proposal is cheap to confirm by correlating against a short window at the claimed position, so the shape is: index proposes, correlator disposes. That is not a formality. In one episode the advertisement occurs twice and the index picked the weaker instance, which only a confirming pass would catch. Measured: 9.8ms to hash a signature when it is stored, 196ms to look one up against 23 episodes, ~8s per episode to index, 353MB for 23. Five hundred signatures cost 98 seconds where linear correlation costs about 63 minutes. The harness is not committed -- it read absolute server paths and had no tests, and its output was an answer rather than code to keep. The parameters it found are written into the plan so it can be rebuilt test-first, along with the one weakness worth knowing: the hashes collide heavily, and that density is what makes a lookup cost milliseconds rather than microseconds. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- ...2026-10-20-corrections-and-signature-library.md | 57 +++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-10-20-corrections-and-signature-library.md b/docs/superpowers/plans/2026-10-20-corrections-and-signature-library.md index 7ec6c52..2572cc3 100644 --- a/docs/superpowers/plans/2026-10-20-corrections-and-signature-library.md +++ b/docs/superpowers/plans/2026-10-20-corrections-and-signature-library.md @@ -180,7 +180,7 @@ Phone to server. Needs an endpoint and server-side storage. Design note: a verdi *range of a specific episode's audio*, so it is only meaningful for episodes whose bytes the server itself served — gate on `audioFromServer`, the same invariant the refresh sweep uses. -### 3. The index spike — NOT STARTED, and it is the gate +### 3. The index spike — DONE 2026-10-24. **It passes.** **Do this before anything else in the library, because it decides whether the library is possible.** A linear scan of N signatures against each episode does not survive: at ~0.33s per @@ -201,6 +201,61 @@ true match produces many hashes agreeing on one offset; noise scatters. If it cannot do both, fast, the library stops there and nothing else is wasted. +**Result: it does both, with room to spare.** Landmark hashing was built as a throwaway harness +against the live corpus and measured on 6 true repeats and 16 non-repeats. + +**The original named files had rotated out of the 8-episode window**, so the test set was rebuilt +from current audio and is stronger than the one it replaces: 6 positives and 16 negatives rather +than 2 and 3. Template: `e86df60c3d8ae9cfc67b0193` at 785-822s, a 37s advertising break. Scored +first with the existing correlator to establish ground truth, then with the index. + +| | correlator `r` | index votes | +|---|---|---| +| 6 repeats, within Up First | 0.590 - 0.910 | 813 - 1564 | +| 16 non-repeats, Serial + Heavyweight + one Up First | 0.077 - 0.120 | 27 - 94 | + +The separation is **8.6x** at its narrowest (813 against 94), against 4.9x for the correlator it +replaces. Any threshold between them works; nothing sits in the gap. + +**Offsets are good enough to act on, and every one is cheap to confirm.** Three agreed with the +correlator to the frame, two were 1.3s out, and all were re-scored by correlating the template +against a 45-second window at the claimed position -- which costs almost nothing next to scanning +a whole episode. The top-scoring negative confirmed at r=0.071, exactly as it should. + +So the architecture is **index proposes, correlator disposes**. That matters because the index +is not always right about *which* occurrence: in one episode the advertisement appears twice and +the index picked the weaker instance (r=0.357 where the stronger was 0.590). Confirming is what +turns a shortlist into a published break. + +**Costs, measured.** + +| what | cost | +|---|---| +| hashing a 37s signature | 9.8ms, paid once when stored | +| lookup, 1 signature against 23 indexed episodes | 196ms, paid per scan | +| building the index | ~8s per episode | +| peak memory, 23 episodes | 353MB | +| **500 signatures against 23 episodes** | **98s** | +| the same by linear correlation | ~63 minutes | + +About **forty times** better, which is the answer the gate was asking for. + +**Parameters that worked**, so this can be rebuilt test-first rather than rediscovered: 2000Hz +(Nyquist 1kHz is enough for speech), 512-sample frames, 256 hop, 6 frequency bands with one peak +each per frame, an energy floor of 2x the frame's mean bin, fan-out 3, pairs 1-20 frames apart, +hash = (binA, binB, frameGap) packed into 24 bits. + +**The one weakness, and the lever if it is ever needed.** 3.49M hashes collapsed into only 86k +distinct values -- about 40 postings each -- and that density is what makes a lookup cost 196ms +rather than microseconds. The peaks cluster because a band maximum repeats. More bands, finer +time resolution or a larger transform would spread them out and cut the lookup cost +substantially. Not needed to pass the gate; worth knowing before anyone optimises the wrong +thing. + +**The harness is deliberately not committed.** It read absolute `/site/listen.terst.org` paths +and had no tests; its output was an answer, not code to keep. Everything needed to rebuild it +properly is above. + ### 4. Store and age signatures — NOT STARTED Two kinds, which behave nothing alike: -- cgit v1.2.3