From e3ecae12d7a309eb5a6c8e62910257096cb99375 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 21 Sep 2026 07:05:24 +0000 Subject: Split download from hosting; carry the token into enclosure URLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The machine that hosts a feed and the machine that downloads the audio do not have to be the same, and usually shouldn't be: a datacentre IP is treated very differently from a home connection by the larger video sites. Downloading where the person actually is sidesteps that entirely, and means the server needs no credentials for or route to the source. - push/ingest: `podkit push` uploads a local library to a remote `serve --ingest`, asking for a manifest first so only the difference is sent. Uploads land via a temp file and rename, so an interrupted transfer cannot be picked up as a short episode. Ingest is off by default and refuses to start without a token. - get: downloads a direct audio URL or every enclosure in an RSS feed, with no extractor involved. Numbers RSS items in reverse so a mirrored series plays in order rather than backwards. Two bugs found by running it rather than by testing it: - A tokened server produced enclosure URLs with no token. The feed loaded, listed every episode, and all of them 404'd in a real client — the whole point of the tool, broken in the default configuration. Feeds now carry the token; regression test asserts enclosures from a tokened feed actually resolve against a tokened server. - Ingest ran filepath.Base before validating, so "../evil.mp3" was stored as "evil.mp3" — contained, but under a name the client never asked for, leaving push and the server disagreeing about the library. Names are now validated raw and rejected. Also: identify with a real user-agent (libsyn 403s Go's default), write feed.xml via rename so a polling client never reads a truncated feed, and match ingested file permissions to downloaded ones. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- README.md | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) (limited to 'README.md') diff --git a/README.md b/README.md index 6157744..3cc8265 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,64 @@ post-processing, after the bytes are already on disk. | Command | What it does | | --- | --- | -| `podkit fetch ` | Download and extract audio | +| `podkit fetch ` | Download and extract audio via yt-dlp | +| `podkit get ` | Download a direct audio URL, or every enclosure in an RSS feed — no yt-dlp, no ffmpeg | | `podkit tag` | Write ID3 tags derived from filenames | | `podkit feed` | Generate `feed.xml` | | `podkit serve` | Host the directory and feed over HTTP | +| `podkit push` | Upload a local library to a remote `podkit serve --ingest` | | `podkit all ` | fetch → tag → feed | Each stage runs independently on purpose: the download is the slow part and shouldn't be repeated while you're adjusting titles or the base URL. +## Downloading somewhere other than where you host + +The download and the hosting do not have to happen on the same machine, and +usually shouldn't. A server on a datacentre IP is treated very differently from +a home connection by the larger video sites — YouTube in particular will refuse +a datacentre request outright with *"Sign in to confirm you're not a bot"* while +the identical command works fine from a laptop. + +So run the download where you actually are, and push the result to the host: + +```bash +# on your own machine +podkit get "" --dir ./audio +podkit tag --dir ./audio --title "…" --artist "…" +podkit push --dir ./audio --server https://your-host/podkit --token "$TOKEN" +``` + +```bash +# on the server, once +podkit serve --dir /srv/podkit/audio --addr 127.0.0.1:8099 \ + --ingest --token "$TOKEN" \ + --title "…" --artist "…" \ + --base-url "https://your-host/podkit/audio/" +``` + +`push` asks the server what it already has and uploads only the difference, so +re-running after adding three episodes to a 200-episode series sends three +files. `--ingest` regenerates the feed after each upload. + +This also means the server never needs credentials for, or a working route to, +wherever the audio came from. + +**`--ingest` is off by default.** A host that accepts writes is a different +security proposition from one that only reads, and it refuses to start without +a token. + +### Behind a reverse proxy + +`serve` expects to be mounted with the path prefix stripped, which is what +doot's `NewServiceProxy` already does. Give `--base-url` the public path and +point `push --server` at the same place: + +``` +--base-url https://doot.terst.org/podkit/audio/ +--server https://doot.terst.org/podkit +``` + ## Notes on the things that usually go wrong These are the parts a hand-rolled version of this pipeline tends to get wrong, @@ -74,6 +123,15 @@ preserves artwork yt-dlp embedded rather than replacing the tag wholesale. **ID3 only on MP3.** ID3v2 is an MP3 convention; writing it into `.opus` or `.m4a` produces something no player reads, so other formats are left alone. +**The token has to be in the enclosure URLs.** A podcast client fetches the +feed, then fetches each episode as a plain GET carrying nothing from the first +request. If the server wants a token and the enclosure URLs don't have one, the +feed loads, lists every episode, and not one of them will ever download. +`--token` on `feed` puts it in the URLs; `serve --ingest` does it for you. + +**Identifying itself.** Go's default user-agent is rejected outright by several +podcast CDNs — libsyn returns 403 for it. `get` sends a real one. + ## "Private" `--token` means unlisted-and-unguessable, not authenticated. Podcast clients @@ -90,6 +148,11 @@ For anything beyond a home network, put it behind a reverse proxy with TLS. may download a given item is between you and whoever published it — check the source's terms. +There is deliberately no cookie-jar or sign-in option. Feeding a site your +logged-in session to get past a check that exists to stop automated downloading +is a different act from downloading, and this tool doesn't do it. `get` covers +the sources that don't ask: RSS feeds, archive.org, direct links. + ## Tests ``` -- cgit v1.2.3