Lectures on Physics

From e3ecae12d7a309eb5a6c8e62910257096cb99375 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 21 Sep 2026 07:05:24 +0000 Subject: Split download from hosting; carry the token into enclosure URLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The machine that hosts a feed and the machine that downloads the audio do not have to be the same, and usually shouldn't be: a datacentre IP is treated very differently from a home connection by the larger video sites. Downloading where the person actually is sidesteps that entirely, and means the server needs no credentials for or route to the source. - push/ingest: `podkit push` uploads a local library to a remote `serve --ingest`, asking for a manifest first so only the difference is sent. Uploads land via a temp file and rename, so an interrupted transfer cannot be picked up as a short episode. Ingest is off by default and refuses to start without a token. - get: downloads a direct audio URL or every enclosure in an RSS feed, with no extractor involved. Numbers RSS items in reverse so a mirrored series plays in order rather than backwards. Two bugs found by running it rather than by testing it: - A tokened server produced enclosure URLs with no token. The feed loaded, listed every episode, and all of them 404'd in a real client — the whole point of the tool, broken in the default configuration. Feeds now carry the token; regression test asserts enclosures from a tokened feed actually resolve against a tokened server. - Ingest ran filepath.Base before validating, so "../evil.mp3" was stored as "evil.mp3" — contained, but under a name the client never asked for, leaving push and the server disagreeing about the library. Names are now validated raw and rejected. Also: identify with a real user-agent (libsyn 403s Go's default), write feed.xml via rename so a polling client never reads a truncated feed, and match ingested file permissions to downloaded ones. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- README.md | 65 +++++++++- internal/feed/feed.go | 40 +++++- internal/feed/feed_test.go | 6 +- internal/fetch/direct.go | 247 ++++++++++++++++++++++++++++++++++++ internal/fetch/direct_test.go | 242 ++++++++++++++++++++++++++++++++++++ internal/push/push.go | 216 ++++++++++++++++++++++++++++++++ internal/push/push_test.go | 262 +++++++++++++++++++++++++++++++++++++++ internal/serve/ingest.go | 162 ++++++++++++++++++++++++ internal/serve/ingest_test.go | 210 +++++++++++++++++++++++++++++++ internal/serve/roundtrip_test.go | 102 +++++++++++++++ internal/serve/serve.go | 15 +++ main.go | 149 +++++++++++++++++++++- 12 files changed, 1701 insertions(+), 15 deletions(-) create mode 100644 internal/fetch/direct.go create mode 100644 internal/fetch/direct_test.go create mode 100644 internal/push/push.go create mode 100644 internal/push/push_test.go create mode 100644 internal/serve/ingest.go create mode 100644 internal/serve/ingest_test.go create mode 100644 internal/serve/roundtrip_test.go diff --git a/README.md b/README.md index 6157744..3cc8265 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,64 @@ post-processing, after the bytes are already on disk. | Command | What it does | | --- | --- | -| `podkit fetch ` | Download and extract audio | +| `podkit fetch ` | Download and extract audio via yt-dlp | +| `podkit get ` | Download a direct audio URL, or every enclosure in an RSS feed — no yt-dlp, no ffmpeg | | `podkit tag` | Write ID3 tags derived from filenames | | `podkit feed` | Generate `feed.xml` | | `podkit serve` | Host the directory and feed over HTTP | +| `podkit push` | Upload a local library to a remote `podkit serve --ingest` | | `podkit all ` | fetch → tag → feed | Each stage runs independently on purpose: the download is the slow part and shouldn't be repeated while you're adjusting titles or the base URL. +## Downloading somewhere other than where you host + +The download and the hosting do not have to happen on the same machine, and +usually shouldn't. A server on a datacentre IP is treated very differently from +a home connection by the larger video sites — YouTube in particular will refuse +a datacentre request outright with *"Sign in to confirm you're not a bot"* while +the identical command works fine from a laptop. + +So run the download where you actually are, and push the result to the host: + +```bash +# on your own machine +podkit get "" --dir ./audio +podkit tag --dir ./audio --title "…" --artist "…" +podkit push --dir ./audio --server https://your-host/podkit --token "$TOKEN" +``` + +```bash +# on the server, once +podkit serve --dir /srv/podkit/audio --addr 127.0.0.1:8099 \ + --ingest --token "$TOKEN" \ + --title "…" --artist "…" \ + --base-url "https://your-host/podkit/audio/" +``` + +`push` asks the server what it already has and uploads only the difference, so +re-running after adding three episodes to a 200-episode series sends three +files. `--ingest` regenerates the feed after each upload. + +This also means the server never needs credentials for, or a working route to, +wherever the audio came from. + +**`--ingest` is off by default.** A host that accepts writes is a different +security proposition from one that only reads, and it refuses to start without +a token. + +### Behind a reverse proxy + +`serve` expects to be mounted with the path prefix stripped, which is what +doot's `NewServiceProxy` already does. Give `--base-url` the public path and +point `push --server` at the same place: + +``` +--base-url https://doot.terst.org/podkit/audio/ +--server https://doot.terst.org/podkit +``` + ## Notes on the things that usually go wrong These are the parts a hand-rolled version of this pipeline tends to get wrong, @@ -74,6 +123,15 @@ preserves artwork yt-dlp embedded rather than replacing the tag wholesale. **ID3 only on MP3.** ID3v2 is an MP3 convention; writing it into `.opus` or `.m4a` produces something no player reads, so other formats are left alone. +**The token has to be in the enclosure URLs.** A podcast client fetches the +feed, then fetches each episode as a plain GET carrying nothing from the first +request. If the server wants a token and the enclosure URLs don't have one, the +feed loads, lists every episode, and not one of them will ever download. +`--token` on `feed` puts it in the URLs; `serve --ingest` does it for you. + +**Identifying itself.** Go's default user-agent is rejected outright by several +podcast CDNs — libsyn returns 403 for it. `get` sends a real one. + ## "Private" `--token` means unlisted-and-unguessable, not authenticated. Podcast clients @@ -90,6 +148,11 @@ For anything beyond a home network, put it behind a reverse proxy with TLS. may download a given item is between you and whoever published it — check the source's terms. +There is deliberately no cookie-jar or sign-in option. Feeding a site your +logged-in session to get past a check that exists to stop automated downloading +is a different act from downloading, and this tool doesn't do it. `get` covers +the sources that don't ask: RSS feeds, archive.org, direct links. + ## Tests ``` diff --git a/internal/feed/feed.go b/internal/feed/feed.go index 02a016e..642d8f3 100644 --- a/internal/feed/feed.go +++ b/internal/feed/feed.go @@ -28,6 +28,13 @@ type Options struct { // successively later dates so that episode order is unambiguous; see // pubDate handling in Build. Epoch time.Time + // Token, if set, is appended to every enclosure URL. + // + // This is not optional bookkeeping: a podcast client fetches an enclosure as + // a plain GET with nothing carried over from the feed request. If the server + // requires a token and the enclosure URLs lack one, the feed loads, shows + // every episode, and no episode will ever download. + Token string } // Item is one episode. @@ -106,6 +113,22 @@ var audioTypes = map[string]string{ ".wav": "audio/wav", } +// IsAudio reports whether a filename is one podkit treats as an episode. +// +// Exported because the server and the uploader need exactly the same answer as +// the feed generator: a file the server accepts but the generator ignores would +// upload successfully and then never appear in the feed. +func IsAudio(filename string) bool { + _, ok := audioTypes[strings.ToLower(filepath.Ext(filename))] + return ok +} + +// MIMEType returns the enclosure content type for a filename, or "" if it is +// not audio podkit recognises. +func MIMEType(filename string) string { + return audioTypes[strings.ToLower(filepath.Ext(filename))] +} + // ParseName splits "03 - Great Conservation Principles.mp3" into its track // number and title. Files without a numeric prefix keep their whole stem as the // title and report track 0. @@ -126,7 +149,9 @@ func ParseName(filename string) (track int, title string) { // contain spaces and ampersands, and pasting them raw produces both an invalid // URL and invalid XML. Only the final path segment is escaped — escaping the // whole base would mangle its scheme and host. -func EnclosureURL(baseURL, filename string) (string, error) { +// A token, when supplied, is carried on the URL, because that is the only thing +// a podcast client will send back. +func EnclosureURL(baseURL, filename, token string) (string, error) { base, err := url.Parse(baseURL) if err != nil { return "", fmt.Errorf("parsing base URL %q: %w", baseURL, err) @@ -135,6 +160,11 @@ func EnclosureURL(baseURL, filename string) (string, error) { return "", fmt.Errorf("base URL %q needs a scheme and host", baseURL) } base.Path = path.Join(base.Path, filename) + if token != "" { + q := base.Query() + q.Set("k", token) + base.RawQuery = q.Encode() + } // url.URL.String() escapes the path for us; RawPath stays unset so the // escaping is derived from Path. return base.String(), nil @@ -167,11 +197,9 @@ func Build(opts Options, readTags TagReader) (*RSS, error) { if e.IsDir() { continue } - mime, isAudio := audioTypes[strings.ToLower(filepath.Ext(e.Name()))] - if !isAudio { + if !IsAudio(e.Name()) { continue } - _ = mime info, err := e.Info() if err != nil { // A file that vanished mid-scan is not a reason to abandon the @@ -209,7 +237,7 @@ func Build(opts Options, readTags TagReader) (*RSS, error) { items := make([]Item, 0, len(found)) for i, c := range found { - encURL, err := EnclosureURL(opts.BaseURL, c.name) + encURL, err := EnclosureURL(opts.BaseURL, c.name, opts.Token) if err != nil { return nil, err } @@ -229,7 +257,7 @@ func Build(opts Options, readTags TagReader) (*RSS, error) { PubDate: pub.Format(time.RFC1123Z), Enclosure: Enclosure{ URL: encURL, - Type: audioTypes[strings.ToLower(filepath.Ext(c.name))], + Type: MIMEType(c.name), Length: c.size, }, Episode: episode, diff --git a/internal/feed/feed_test.go b/internal/feed/feed_test.go index a529e6f..588c652 100644 --- a/internal/feed/feed_test.go +++ b/internal/feed/feed_test.go @@ -66,7 +66,7 @@ func TestParseName(t *testing.T) { // enclosure URL unescaped produces both an invalid URL and invalid XML — the // single most likely way a hand-rolled feed fails to load at all. func TestEnclosureURL_EscapesFilenames(t *testing.T) { - got, err := EnclosureURL("https://example.com/audio/", "10 - Probability & Uncertainty.mp3") + got, err := EnclosureURL("https://example.com/audio/", "10 - Probability & Uncertainty.mp3", "") if err != nil { t.Fatalf("EnclosureURL: %v", err) } @@ -85,7 +85,7 @@ func TestEnclosureURL_EscapesFilenames(t *testing.T) { } func TestEnclosureURL_HandlesMissingTrailingSlash(t *testing.T) { - got, err := EnclosureURL("https://example.com/audio", "01 - One.mp3") + got, err := EnclosureURL("https://example.com/audio", "01 - One.mp3", "") if err != nil { t.Fatalf("EnclosureURL: %v", err) } @@ -96,7 +96,7 @@ func TestEnclosureURL_HandlesMissingTrailingSlash(t *testing.T) { func TestEnclosureURL_RejectsBadBase(t *testing.T) { for _, bad := range []string{"", "not-a-url", "/just/a/path"} { - if _, err := EnclosureURL(bad, "a.mp3"); err == nil { + if _, err := EnclosureURL(bad, "a.mp3", ""); err == nil { t.Errorf("EnclosureURL(%q) should have failed", bad) } } diff --git a/internal/fetch/direct.go b/internal/fetch/direct.go new file mode 100644 index 0000000..9bd7116 --- /dev/null +++ b/internal/fetch/direct.go @@ -0,0 +1,247 @@ +package fetch + +import ( + "context" + "encoding/xml" + "fmt" + "io" + "net/http" + "net/url" + "os" + "path" + "path/filepath" + "strings" + + "podkit/internal/feed" +) + +// userAgent identifies podkit to feed hosts. +// +// Go's default ("Go-http-client/…") is rejected outright by several podcast +// CDNs — libsyn returns 403 for it. Identifying honestly as what this is gets a +// normal response and leaves an accurate trace in the host's logs. +const userAgent = "podkit/1.0 (+https://github.com/; podcast feed mirror)" + +func get(ctx context.Context, c *http.Client, rawURL string) (*http.Response, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil) + if err != nil { + return nil, err + } + req.Header.Set("User-Agent", userAgent) + return c.Do(req) +} + +// Source is one downloadable audio file discovered from a URL. +type Source struct { + URL string + Name string + Track int +} + +// rssEnvelope is the minimum needed to pull enclosures out of a podcast feed. +type rssEnvelope struct { + Channel struct { + Title string `xml:"title"` + Items []struct { + Title string `xml:"title"` + Enclosure struct { + URL string `xml:"url,attr"` + Type string `xml:"type,attr"` + } `xml:"enclosure"` + } `xml:"item"` + } `xml:"channel"` +} + +// Discover resolves a URL into the audio files behind it. +// +// Handles two cases that need no extractor at all: a direct link to an audio +// file, and an RSS feed whose items carry enclosures. Between them these cover +// archive.org, university lecture pages, and every existing podcast — none of +// which require yt-dlp, and none of which are subject to YouTube's bot checks. +func Discover(ctx context.Context, rawURL string, client *http.Client) ([]Source, string, error) { + if client == nil { + client = http.DefaultClient + } + u, err := url.Parse(rawURL) + if err != nil || u.Scheme == "" || u.Host == "" { + return nil, "", fmt.Errorf("%q is not a usable URL", rawURL) + } + + // A direct audio link needs no fetch to classify. + if feed.IsAudio(path.Base(u.Path)) { + return []Source{{URL: rawURL, Name: nameFromURL(u)}}, "", nil + } + + resp, err := get(ctx, client, rawURL) + if err != nil { + return nil, "", fmt.Errorf("fetching %s: %w", rawURL, err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, "", fmt.Errorf("fetching %s: %s", rawURL, resp.Status) + } + + // Capped: a feed is text, and an unbounded read here would let a wrong URL + // pull an arbitrary amount into memory. + body, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) + if err != nil { + return nil, "", fmt.Errorf("reading %s: %w", rawURL, err) + } + + var rss rssEnvelope + if err := xml.Unmarshal(body, &rss); err != nil { + return nil, "", fmt.Errorf("%s is neither a direct audio file nor an RSS feed", rawURL) + } + if len(rss.Channel.Items) == 0 { + return nil, "", fmt.Errorf("%s parsed as XML but has no items with enclosures", rawURL) + } + + var sources []Source + // RSS is newest-first by convention, so the last item is episode 1. Numbering + // in reverse keeps a series in listening order once it reaches the feed. + total := len(rss.Channel.Items) + for i, it := range rss.Channel.Items { + if it.Enclosure.URL == "" { + continue + } + eu, err := url.Parse(it.Enclosure.URL) + if err != nil { + continue + } + track := total - i + title := strings.TrimSpace(it.Title) + if title == "" { + title = nameFromURL(eu) + } + sources = append(sources, Source{ + URL: it.Enclosure.URL, + Name: fmt.Sprintf("%02d - %s%s", track, sanitise(title), extOf(eu, it.Enclosure.Type)), + Track: track, + }) + } + if len(sources) == 0 { + return nil, "", fmt.Errorf("%s has items but none carry an audio enclosure", rawURL) + } + return sources, strings.TrimSpace(rss.Channel.Title), nil +} + +// Download writes sources into dir, skipping ones already present. +func Download(ctx context.Context, sources []Source, dir string, client *http.Client, progress io.Writer) error { + if client == nil { + client = http.DefaultClient + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return fmt.Errorf("creating %s: %w", dir, err) + } + + for _, s := range sources { + target := filepath.Join(dir, s.Name) + if _, err := os.Stat(target); err == nil { + if progress != nil { + fmt.Fprintf(progress, " have %s\n", s.Name) + } + continue + } + if progress != nil { + fmt.Fprintf(progress, " get %s\n", s.Name) + } + if err := downloadOne(ctx, client, s.URL, target); err != nil { + return err + } + } + return nil +} + +func downloadOne(ctx context.Context, c *http.Client, src, target string) error { + resp, err := get(ctx, c, src) + if err != nil { + return fmt.Errorf("downloading %s: %w", src, err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("downloading %s: %s", src, resp.Status) + } + + // Downloaded to a partial name and renamed on success, so an interrupted + // run leaves nothing that looks like a complete episode. + tmp := target + ".part" + f, err := os.Create(tmp) + if err != nil { + return err + } + _, copyErr := io.Copy(f, resp.Body) + closeErr := f.Close() + if copyErr != nil { + os.Remove(tmp) + return fmt.Errorf("downloading %s: %w", src, copyErr) + } + if closeErr != nil { + os.Remove(tmp) + return closeErr + } + return os.Rename(tmp, target) +} + +func nameFromURL(u *url.URL) string { + name := path.Base(u.Path) + if name == "" || name == "." || name == "/" { + return "audio.mp3" + } + if decoded, err := url.PathUnescape(name); err == nil { + name = decoded + } + return sanitise(name) +} + +// extOf picks a file extension from the enclosure URL, falling back to its +// declared MIME type, then to .mp3. +func extOf(u *url.URL, mimeType string) string { + if ext := path.Ext(u.Path); feed.IsAudio("x" + ext) { + return ext + } + switch strings.ToLower(strings.TrimSpace(mimeType)) { + case "audio/mp4", "audio/x-m4a": + return ".m4a" + case "audio/aac": + return ".aac" + case "audio/ogg": + return ".ogg" + case "audio/opus": + return ".opus" + case "audio/flac": + return ".flac" + case "audio/wav", "audio/x-wav": + return ".wav" + default: + return ".mp3" + } +} + +// sanitise strips the characters that make a filename awkward in a shell or a +// URL. Spaces are kept — the feed generator escapes them properly. +func sanitise(s string) string { + s = strings.Map(func(r rune) rune { + switch r { + case '/', '\\', ':', '*', '?', '"', '<', '>', '|', '\n', '\r', '\t': + return -1 + } + if r < 0x20 { + return -1 + } + return r + }, s) + for strings.Contains(s, " ") { + s = strings.ReplaceAll(s, " ", " ") + } + s = strings.TrimSpace(s) + s = strings.TrimLeft(s, ".") + if s == "" { + return "episode" + } + // Keep well clear of the 255-byte limit most filesystems impose, leaving + // room for the extension and the ".part" suffix used while downloading. + if len(s) > 180 { + s = strings.TrimSpace(s[:180]) + } + return s +} diff --git a/internal/fetch/direct_test.go b/internal/fetch/direct_test.go new file mode 100644 index 0000000..86b9e31 --- /dev/null +++ b/internal/fetch/direct_test.go @@ -0,0 +1,242 @@ +package fetch + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +const sampleRSS = ` + + + Lectures on Physics + + Three - The Great Conservation Principles + + + + Two - The Relation of Mathematics & Physics + + + + One - The Law of Gravitation + + + +` + +func rssServer(t *testing.T) *httptest.Server { + t.Helper() + var srv *httptest.Server + srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, ".mp3") { + fmt.Fprintf(w, "audio for %s", r.URL.Path) + return + } + w.Header().Set("Content-Type", "application/rss+xml") + fmt.Fprintf(w, sampleRSS, srv.URL, srv.URL, srv.URL) + })) + t.Cleanup(srv.Close) + return srv +} + +func TestDiscoverDirectAudioURL(t *testing.T) { + sources, _, err := Discover(context.Background(), "https://example.org/talks/01%20-%20Intro.mp3", nil) + if err != nil { + t.Fatal(err) + } + if len(sources) != 1 { + t.Fatalf("got %d sources", len(sources)) + } + if sources[0].Name != "01 - Intro.mp3" { + t.Errorf("name %q — the URL escape should be decoded", sources[0].Name) + } +} + +func TestDiscoverRSSEnclosures(t *testing.T) { + srv := rssServer(t) + sources, title, err := Discover(context.Background(), srv.URL+"/feed.xml", nil) + if err != nil { + t.Fatal(err) + } + if title != "Lectures on Physics" { + t.Errorf("title %q", title) + } + if len(sources) != 3 { + t.Fatalf("got %d sources", len(sources)) + } +} + +// RSS is newest-first by convention, so the last item is episode 1. Numbering +// has to run in reverse or a mirrored course plays backwards. +func TestDiscoverNumbersRSSInListeningOrder(t *testing.T) { + srv := rssServer(t) + sources, _, err := Discover(context.Background(), srv.URL+"/feed.xml", nil) + if err != nil { + t.Fatal(err) + } + + byTrack := map[int]string{} + for _, s := range sources { + byTrack[s.Track] = s.Name + } + if !strings.Contains(byTrack[1], "One - The Law of Gravitation") { + t.Errorf("track 1 is %q, want the feed's last item", byTrack[1]) + } + if !strings.Contains(byTrack[3], "Three - The Great Conservation") { + t.Errorf("track 3 is %q, want the feed's first item", byTrack[3]) + } + if !strings.HasPrefix(byTrack[1], "01 - ") { + t.Errorf("track 1 name %q is not zero-padded", byTrack[1]) + } +} + +func TestDiscoverRejectsNonFeed(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + fmt.Fprint(w, "not a feed") + })) + defer srv.Close() + + if _, _, err := Discover(context.Background(), srv.URL, nil); err == nil { + t.Error("expected an error for a page that is not a feed") + } +} + +func TestDiscoverRejectsBadURL(t *testing.T) { + for _, u := range []string{"", "not-a-url", "/just/a/path"} { + if _, _, err := Discover(context.Background(), u, nil); err == nil { + t.Errorf("%q was accepted", u) + } + } +} + +func TestDownloadWritesFiles(t *testing.T) { + srv := rssServer(t) + dir := t.TempDir() + + sources, _, err := Discover(context.Background(), srv.URL+"/feed.xml", nil) + if err != nil { + t.Fatal(err) + } + if err := Download(context.Background(), sources, dir, nil, nil); err != nil { + t.Fatal(err) + } + + entries, _ := os.ReadDir(dir) + if len(entries) != 3 { + t.Fatalf("got %d files", len(entries)) + } + for _, e := range entries { + if strings.HasSuffix(e.Name(), ".part") { + t.Errorf("partial file %q left behind", e.Name()) + } + } +} + +// Re-running must not re-download; this is what makes a long series resumable. +func TestDownloadSkipsExisting(t *testing.T) { + var hits int + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits++ + fmt.Fprint(w, "audio") + })) + defer srv.Close() + + dir := t.TempDir() + sources := []Source{{URL: srv.URL + "/a.mp3", Name: "01 - A.mp3", Track: 1}} + + for i := 0; i < 3; i++ { + if err := Download(context.Background(), sources, dir, nil, nil); err != nil { + t.Fatal(err) + } + } + if hits != 1 { + t.Errorf("downloaded %d times, want 1", hits) + } +} + +// A failed download must not leave something the feed generator would treat as +// a complete episode. +func TestDownloadLeavesNothingOnFailure(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Error(w, "gone", http.StatusNotFound) + })) + defer srv.Close() + + dir := t.TempDir() + err := Download(context.Background(), []Source{{URL: srv.URL + "/a.mp3", Name: "01 - A.mp3"}}, dir, nil, nil) + if err == nil { + t.Fatal("expected an error") + } + if entries, _ := os.ReadDir(dir); len(entries) != 0 { + t.Errorf("left %d file(s) behind after a failed download", len(entries)) + } +} + +func TestSanitiseRemovesPathCharacters(t *testing.T) { + cases := []struct{ in, want string }{ + {"A/B: The Sequel", "AB The Sequel"}, + {"What? Why!", "What Why!"}, + {" spaced out ", "spaced out"}, + // Separators are dropped and the leading dots trimmed, so a name that + // tried to traverse comes out as an ordinary filename. + {"../../etc/passwd", "etcpasswd"}, + {"", "episode"}, + {"...", "episode"}, + } + for _, c := range cases { + if got := sanitise(c.in); got != c.want { + t.Errorf("sanitise(%q) = %q, want %q", c.in, got, c.want) + } + } +} + +// A sanitised name must never be able to escape the download directory. +func TestSanitisedNamesStayInDirectory(t *testing.T) { + dir := t.TempDir() + for _, in := range []string{"../escape", "../../escape", "/absolute", "a/b/c"} { + joined := filepath.Join(dir, sanitise(in)+".mp3") + if !strings.HasPrefix(joined, dir+string(filepath.Separator)) { + t.Errorf("sanitise(%q) produced a path outside the directory: %s", in, joined) + } + } +} + +func TestExtOfPrefersURLThenMIME(t *testing.T) { + sources, _, err := Discover(context.Background(), "https://example.org/a/file.opus", nil) + if err != nil { + t.Fatal(err) + } + if filepath.Ext(sources[0].Name) != ".opus" { + t.Errorf("name %q", sources[0].Name) + } +} + +// An enclosure URL with no useful extension has to fall back to the declared +// MIME type, or the file lands as .mp3 and no player reads its tags. +func TestExtOfFallsBackToMIMEType(t *testing.T) { + var srv *httptest.Server + srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasPrefix(r.URL.Path, "/stream") { + fmt.Fprint(w, "audio") + return + } + fmt.Fprintf(w, `T + Only + `, srv.URL) + })) + defer srv.Close() + + sources, _, err := Discover(context.Background(), srv.URL+"/feed.xml", nil) + if err != nil { + t.Fatal(err) + } + if filepath.Ext(sources[0].Name) != ".m4a" { + t.Errorf("name %q, want a .m4a extension from the audio/mp4 type", sources[0].Name) + } +} diff --git a/internal/push/push.go b/internal/push/push.go new file mode 100644 index 0000000..b640127 --- /dev/null +++ b/internal/push/push.go @@ -0,0 +1,216 @@ +// Package push uploads locally-downloaded audio to a remote podkit server. +// +// This exists because the download and the hosting do not have to happen on the +// same machine, and usually shouldn't. Downloading is something a person does +// from their own connection; hosting is something a server does. Splitting them +// means the server never needs credentials for, or a working route to, whatever +// the audio came from. +package push + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "os" + "path" + "path/filepath" + "sort" + "strings" + + "podkit/internal/feed" +) + +// Options configure an upload run. +type Options struct { + Dir string // local directory holding the audio + Server string // base URL of the remote podkit server + Token string // the remote server's token + Force bool // upload even when the server already has a matching file + Client *http.Client +} + +// Result describes what happened to one file. +type Result struct { + Name string + Bytes int64 + Skipped bool + Err error +} + +// Run uploads every audio file in Dir that the server does not already have. +func Run(ctx context.Context, opts Options) ([]Result, error) { + if opts.Server == "" { + return nil, fmt.Errorf("a server URL is required") + } + base, err := url.Parse(opts.Server) + if err != nil || base.Scheme == "" || base.Host == "" { + return nil, fmt.Errorf("server %q needs a scheme and host, e.g. https://host/podkit", opts.Server) + } + + client := opts.Client + if client == nil { + // No overall timeout: a large episode on a slow uplink is a legitimate + // long request. Cancellation comes from ctx instead. + client = &http.Client{Timeout: 0} + } + + local, err := localFiles(opts.Dir) + if err != nil { + return nil, err + } + if len(local) == 0 { + return nil, fmt.Errorf("no audio files in %s", opts.Dir) + } + + remote := map[string]int64{} + if !opts.Force { + remote, err = fetchManifest(ctx, client, base, opts.Token) + if err != nil { + return nil, err + } + } + + var results []Result + for _, f := range local { + if size, ok := remote[f.name]; ok && size == f.size { + results = append(results, Result{Name: f.name, Bytes: f.size, Skipped: true}) + continue + } + err := upload(ctx, client, base, opts.Token, opts.Dir, f.name, f.size) + results = append(results, Result{Name: f.name, Bytes: f.size, Err: err}) + if err != nil && ctx.Err() != nil { + // Interrupted rather than rejected; stop instead of hammering + // through the rest of the list with a dead context. + break + } + } + return results, nil +} + +type localFile struct { + name string + size int64 +} + +func localFiles(dir string) ([]localFile, error) { + entries, err := os.ReadDir(dir) + if err != nil { + return nil, fmt.Errorf("reading %s: %w", dir, err) + } + var out []localFile + for _, e := range entries { + if e.IsDir() || !feed.IsAudio(e.Name()) { + continue + } + info, err := e.Info() + if err != nil { + return nil, fmt.Errorf("stat %s: %w", e.Name(), err) + } + out = append(out, localFile{e.Name(), info.Size()}) + } + sort.Slice(out, func(i, j int) bool { return out[i].name < out[j].name }) + return out, nil +} + +func fetchManifest(ctx context.Context, c *http.Client, base *url.URL, token string) (map[string]int64, error) { + u := *base + u.Path = path.Join(u.Path, "manifest") + req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil) + if err != nil { + return nil, err + } + auth(req, token) + + resp, err := c.Do(req) + if err != nil { + return nil, fmt.Errorf("contacting %s: %w", u.String(), err) + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + // The server 404s both a wrong token and a server without --ingest, on + // purpose. Say so, rather than reporting a bare 404 the user then has + // to guess the meaning of. + return nil, fmt.Errorf("manifest request returned %s — check the token, and that the server was started with --ingest", + resp.Status) + } + + var m struct { + Files map[string]int64 `json:"files"` + } + if err := json.NewDecoder(resp.Body).Decode(&m); err != nil { + return nil, fmt.Errorf("parsing manifest from %s: %w", u.String(), err) + } + if m.Files == nil { + m.Files = map[string]int64{} + } + return m.Files, nil +} + +func upload(ctx context.Context, c *http.Client, base *url.URL, token, dir, name string, size int64) error { + f, err := os.Open(filepath.Join(dir, name)) + if err != nil { + return err + } + defer f.Close() + + u := *base + u.Path = path.Join(u.Path, "ingest") + q := u.Query() + q.Set("name", name) + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, http.MethodPost, u.String(), f) + if err != nil { + return err + } + auth(req, token) + req.Header.Set("Content-Type", feed.MIMEType(name)) + // Set explicitly so the server sees a length rather than a chunked body, + // and so MaxBytesReader can reject an oversized file before the transfer. + req.ContentLength = size + + resp, err := c.Do(req) + if err != nil { + return fmt.Errorf("uploading %s: %w", name, err) + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(io.LimitReader(resp.Body, 512)) + msg := strings.TrimSpace(string(body)) + if msg == "" { + msg = resp.Status + } + return fmt.Errorf("uploading %s: %s", name, msg) + } + _, _ = io.Copy(io.Discard, resp.Body) + return nil +} + +func auth(req *http.Request, token string) { + if token == "" { + return + } + // Header rather than query string: it keeps the secret out of the server's + // access log, and push is not a podcast client so it has the choice. + req.Header.Set("Authorization", "Bearer "+token) +} + +// Summarise renders results for the terminal. +func Summarise(results []Result) (uploaded, skipped, failed int) { + for _, r := range results { + switch { + case r.Err != nil: + failed++ + case r.Skipped: + skipped++ + default: + uploaded++ + } + } + return +} diff --git a/internal/push/push_test.go b/internal/push/push_test.go new file mode 100644 index 0000000..8e79739 --- /dev/null +++ b/internal/push/push_test.go @@ -0,0 +1,262 @@ +package push + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync" + "testing" +) + +// fakeServer stands in for a remote podkit server with ingest enabled. +type fakeServer struct { + mu sync.Mutex + existing map[string]int64 + received map[string]string + srv *httptest.Server +} + +func newFakeServer(t *testing.T, existing map[string]int64) *fakeServer { + t.Helper() + f := &fakeServer{existing: existing, received: map[string]string{}} + if f.existing == nil { + f.existing = map[string]int64{} + } + + mux := http.NewServeMux() + mux.HandleFunc("/manifest", func(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{"files": f.existing}) + }) + mux.HandleFunc("/ingest", func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + f.mu.Lock() + defer f.mu.Unlock() + f.received[r.URL.Query().Get("name")] = string(body) + w.WriteHeader(http.StatusOK) + }) + + f.srv = httptest.NewServer(mux) + t.Cleanup(f.srv.Close) + return f +} + +func libraryDir(t *testing.T, files map[string]string) string { + t.Helper() + dir := t.TempDir() + for name, body := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +func TestPushUploadsAudio(t *testing.T) { + dir := libraryDir(t, map[string]string{ + "01 - April.mp3": "aaa", + "02 - May.mp3": "bbbb", + }) + f := newFakeServer(t, nil) + + results, err := Run(context.Background(), Options{Dir: dir, Server: f.srv.URL, Token: "t"}) + if err != nil { + t.Fatal(err) + } + + uploaded, skipped, failed := Summarise(results) + if uploaded != 2 || skipped != 0 || failed != 0 { + t.Fatalf("uploaded=%d skipped=%d failed=%d", uploaded, skipped, failed) + } + if f.received["01 - April.mp3"] != "aaa" || f.received["02 - May.mp3"] != "bbbb" { + t.Errorf("server received %v", f.received) + } +} + +// Re-running a push must not re-send what is already there; a lecture series is +// gigabytes and the second run should be nearly free. +func TestPushSkipsFilesTheServerHas(t *testing.T) { + dir := libraryDir(t, map[string]string{ + "01 - April.mp3": "aaa", + "02 - May.mp3": "bbbb", + }) + f := newFakeServer(t, map[string]int64{"01 - April.mp3": 3}) + + results, err := Run(context.Background(), Options{Dir: dir, Server: f.srv.URL}) + if err != nil { + t.Fatal(err) + } + + uploaded, skipped, _ := Summarise(results) + if uploaded != 1 || skipped != 1 { + t.Fatalf("uploaded=%d skipped=%d", uploaded, skipped) + } + if _, sent := f.received["01 - April.mp3"]; sent { + t.Error("a file the server already had was re-uploaded") + } +} + +// A same-named file of a different size is a different file — most likely a +// truncated earlier upload — and must be re-sent rather than assumed good. +func TestPushResendsOnSizeMismatch(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + f := newFakeServer(t, map[string]int64{"01 - April.mp3": 99}) + + results, err := Run(context.Background(), Options{Dir: dir, Server: f.srv.URL}) + if err != nil { + t.Fatal(err) + } + uploaded, skipped, _ := Summarise(results) + if uploaded != 1 || skipped != 0 { + t.Fatalf("uploaded=%d skipped=%d", uploaded, skipped) + } +} + +func TestPushForceIgnoresManifest(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + f := newFakeServer(t, map[string]int64{"01 - April.mp3": 3}) + + results, err := Run(context.Background(), Options{Dir: dir, Server: f.srv.URL, Force: true}) + if err != nil { + t.Fatal(err) + } + if uploaded, _, _ := Summarise(results); uploaded != 1 { + t.Fatalf("uploaded=%d, want 1 under --force", uploaded) + } +} + +func TestPushIgnoresNonAudio(t *testing.T) { + dir := libraryDir(t, map[string]string{ + "01 - April.mp3": "aaa", + "feed.xml": "", + "cover.jpg": "img", + "notes.txt": "hello", + }) + f := newFakeServer(t, nil) + + if _, err := Run(context.Background(), Options{Dir: dir, Server: f.srv.URL}); err != nil { + t.Fatal(err) + } + if len(f.received) != 1 { + t.Errorf("server received %v, want only the mp3", f.received) + } +} + +func TestPushSendsToken(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + + var gotAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/manifest" { + gotAuth = r.Header.Get("Authorization") + _ = json.NewEncoder(w).Encode(map[string]any{"files": map[string]int64{}}) + return + } + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + if _, err := Run(context.Background(), Options{Dir: dir, Server: srv.URL, Token: "sekrit"}); err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer sekrit" { + t.Errorf("Authorization header was %q", gotAuth) + } +} + +// A 404 from the manifest means either a bad token or a server without +// --ingest. The error has to say so, since a bare "404" is unactionable. +func TestPushExplainsManifest404(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.NotFound(w, r) + })) + defer srv.Close() + + _, err := Run(context.Background(), Options{Dir: dir, Server: srv.URL}) + if err == nil { + t.Fatal("expected an error") + } + for _, want := range []string{"token", "--ingest"} { + if !contains(err.Error(), want) { + t.Errorf("error should mention %q; got %q", want, err) + } + } +} + +func TestPushReportsUploadFailure(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/manifest" { + _ = json.NewEncoder(w).Encode(map[string]any{"files": map[string]int64{}}) + return + } + http.Error(w, "disk full", http.StatusInsufficientStorage) + })) + defer srv.Close() + + results, err := Run(context.Background(), Options{Dir: dir, Server: srv.URL}) + if err != nil { + t.Fatal(err) + } + if _, _, failed := Summarise(results); failed != 1 { + t.Fatalf("failed=%d, want 1", failed) + } + if !contains(results[0].Err.Error(), "disk full") { + t.Errorf("server's reason was lost: %v", results[0].Err) + } +} + +func TestPushRejectsBadServerURL(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + for _, s := range []string{"", "not-a-url", "/relative/path"} { + if _, err := Run(context.Background(), Options{Dir: dir, Server: s}); err == nil { + t.Errorf("server %q was accepted", s) + } + } +} + +func TestPushRejectsEmptyLibrary(t *testing.T) { + f := newFakeServer(t, nil) + if _, err := Run(context.Background(), Options{Dir: t.TempDir(), Server: f.srv.URL}); err == nil { + t.Error("expected an error for a directory with no audio") + } +} + +// The base URL may include a path, since the server will usually sit behind a +// reverse proxy at something like /podkit. +func TestPushHonoursServerPathPrefix(t *testing.T) { + dir := libraryDir(t, map[string]string{"01 - April.mp3": "aaa"}) + + var paths []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + paths = append(paths, r.URL.Path) + if r.URL.Path == "/podkit/manifest" { + _ = json.NewEncoder(w).Encode(map[string]any{"files": map[string]int64{}}) + return + } + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + if _, err := Run(context.Background(), Options{Dir: dir, Server: srv.URL + "/podkit"}); err != nil { + t.Fatal(err) + } + if len(paths) != 2 || paths[0] != "/podkit/manifest" || paths[1] != "/podkit/ingest" { + t.Errorf("requested %v", paths) + } +} + +func contains(h, n string) bool { + for i := 0; i+len(n) <= len(h); i++ { + if h[i:i+len(n)] == n { + return true + } + } + return false +} diff --git a/internal/serve/ingest.go b/internal/serve/ingest.go new file mode 100644 index 0000000..a949e1e --- /dev/null +++ b/internal/serve/ingest.go @@ -0,0 +1,162 @@ +package serve + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + + "podkit/internal/feed" +) + +// Manifest is what the server already holds, used by `podkit push` to decide +// what still needs uploading. Keyed by filename. +type Manifest struct { + Files map[string]int64 `json:"files"` +} + +// maxUpload caps a single uploaded file. Generous enough for a long lecture at +// a high bitrate; present so a runaway or hostile client cannot fill the disk +// in one request. +const maxUpload = 500 << 20 // 500 MiB + +// readManifest lists the audio already present in dir. +func readManifest(dir string) (Manifest, error) { + m := Manifest{Files: map[string]int64{}} + entries, err := os.ReadDir(dir) + if err != nil { + return m, err + } + for _, e := range entries { + if e.IsDir() || !feed.IsAudio(e.Name()) { + continue + } + info, err := e.Info() + if err != nil { + // A file that vanished mid-scan just isn't in the manifest; push + // will then offer it, which is the safe direction to be wrong in. + continue + } + m.Files[e.Name()] = info.Size() + } + return m, nil +} + +func handleManifest(dir string) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + m, err := readManifest(dir) + if err != nil { + http.Error(w, "cannot read library", http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(m) + } +} + +// handleIngest accepts an uploaded audio file and rebuilds the feed. +// +// This is the half of podkit that runs on the server. The download half runs +// wherever the person doing it actually is — which is the point: the machine +// hosting the feed never needs to talk to the source at all. +func handleIngest(dir string, rebuild func() error) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.Header().Set("Allow", http.MethodPost) + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + + // Validated raw, not via filepath.Base: Base would quietly turn + // "../evil.mp3" into "evil.mp3" and store it under a name the client + // never asked for. That is safe from traversal but leaves push and the + // server disagreeing about what the library contains, which breaks the + // manifest comparison on the next run. A name with a separator in it is + // a bug or an attack either way, so reject it. + name := r.URL.Query().Get("name") + if !validUploadName(name) { + http.Error(w, "invalid or missing name parameter", http.StatusBadRequest) + return + } + if !feed.IsAudio(name) { + http.Error(w, "not an audio file", http.StatusUnsupportedMediaType) + return + } + + if err := os.MkdirAll(dir, 0o755); err != nil { + http.Error(w, "cannot write to library", http.StatusInternalServerError) + return + } + + // Written to a temporary name and renamed into place, so an upload cut + // off halfway can never be picked up as a half-length episode. + tmp, err := os.CreateTemp(dir, ".upload-*") + if err != nil { + http.Error(w, "cannot write to library", http.StatusInternalServerError) + return + } + tmpName := tmp.Name() + defer os.Remove(tmpName) // no-op once the rename below succeeds + + written, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload)) + closeErr := tmp.Close() + if err != nil { + http.Error(w, "upload failed or too large", http.StatusBadRequest) + return + } + if closeErr != nil { + http.Error(w, "cannot write to library", http.StatusInternalServerError) + return + } + if written == 0 { + http.Error(w, "empty upload", http.StatusBadRequest) + return + } + + // CreateTemp makes the file 0600; match the 0644 that downloaded files + // get, so a library's permissions don't depend on how each episode + // happened to arrive. + if err := os.Chmod(tmpName, 0o644); err != nil { + http.Error(w, "cannot write to library", http.StatusInternalServerError) + return + } + + if err := os.Rename(tmpName, filepath.Join(dir, name)); err != nil { + http.Error(w, "cannot write to library", http.StatusInternalServerError) + return + } + + // A new file that isn't in the feed is invisible to the podcast app, so + // a rebuild failure has to be reported rather than logged and forgotten. + if rebuild != nil { + if err := rebuild(); err != nil { + http.Error(w, fmt.Sprintf("stored %s but the feed rebuild failed: %v", name, err), + http.StatusInternalServerError) + return + } + } + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "stored": name, + "bytes": written, + }) + } +} + +// validUploadName rejects names that would escape the directory or collide with +// podkit's own files. +func validUploadName(name string) bool { + switch name { + case "", ".", "..", "/", "feed.xml": + return false + } + if strings.ContainsAny(name, `/\`) { + return false + } + // Dotfiles are podkit's own bookkeeping (.podkit-archive, .upload-*). + return !strings.HasPrefix(name, ".") +} diff --git a/internal/serve/ingest_test.go b/internal/serve/ingest_test.go new file mode 100644 index 0000000..38f7a86 --- /dev/null +++ b/internal/serve/ingest_test.go @@ -0,0 +1,210 @@ +package serve + +import ( + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +func ingestServer(t *testing.T, dir string, rebuild func() error) http.Handler { + t.Helper() + return Handler(Options{Dir: dir, Token: "secret", AllowIngest: true, Rebuild: rebuild}) +} + +func post(t *testing.T, h http.Handler, target, body string) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(http.MethodPost, target, strings.NewReader(body)) + req.Header.Set("Authorization", "Bearer secret") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + return rec +} + +func TestIngestStoresFileAndRebuilds(t *testing.T) { + dir := t.TempDir() + rebuilt := 0 + h := ingestServer(t, dir, func() error { rebuilt++; return nil }) + + rec := post(t, h, "/ingest?name=01+-+April.mp3", "audio-bytes") + if rec.Code != http.StatusOK { + t.Fatalf("got %d: %s", rec.Code, rec.Body.String()) + } + + body, err := os.ReadFile(filepath.Join(dir, "01 - April.mp3")) + if err != nil { + t.Fatalf("file not stored: %v", err) + } + if string(body) != "audio-bytes" { + t.Errorf("stored %q", body) + } + if rebuilt != 1 { + t.Errorf("rebuild ran %d times, want 1", rebuilt) + } +} + +// A stored file that never reaches the feed is invisible to the podcast app, +// so a rebuild failure must surface as an error rather than a quiet success. +func TestIngestReportsRebuildFailure(t *testing.T) { + dir := t.TempDir() + h := ingestServer(t, dir, func() error { return errors.New("no title configured") }) + + rec := post(t, h, "/ingest?name=01+-+April.mp3", "audio-bytes") + if rec.Code != http.StatusInternalServerError { + t.Fatalf("got %d, want 500", rec.Code) + } + if !strings.Contains(rec.Body.String(), "no title configured") { + t.Errorf("underlying cause not reported: %s", rec.Body.String()) + } +} + +func TestIngestRejectsPathTraversal(t *testing.T) { + dir := t.TempDir() + h := ingestServer(t, dir, nil) + + for _, name := range []string{ + "../escaped.mp3", + "..%2Fescaped.mp3", + "/etc/cron.d/evil.mp3", + "sub/dir.mp3", + } { + rec := post(t, h, "/ingest?name="+name, "x") + if rec.Code == http.StatusOK { + t.Errorf("name %q was accepted", name) + } + } + + // Nothing may have been created outside the library directory. + if _, err := os.Stat(filepath.Join(filepath.Dir(dir), "escaped.mp3")); err == nil { + t.Error("a file was written outside the library directory") + } +} + +func TestIngestRejectsNonAudio(t *testing.T) { + h := ingestServer(t, t.TempDir(), nil) + for _, name := range []string{"notes.txt", "cover.jpg", "script.sh", "feed.xml"} { + if rec := post(t, h, "/ingest?name="+name, "x"); rec.Code == http.StatusOK { + t.Errorf("name %q was accepted", name) + } + } +} + +// Dotfiles are podkit's own bookkeeping; an upload must not be able to replace +// the download archive or masquerade as a temporary upload. +func TestIngestRejectsDotfiles(t *testing.T) { + h := ingestServer(t, t.TempDir(), nil) + for _, name := range []string{".podkit-archive", ".upload-x.mp3", ".mp3"} { + if rec := post(t, h, "/ingest?name="+name, "x"); rec.Code == http.StatusOK { + t.Errorf("name %q was accepted", name) + } + } +} + +func TestIngestRejectsEmptyUpload(t *testing.T) { + dir := t.TempDir() + h := ingestServer(t, dir, nil) + + rec := post(t, h, "/ingest?name=01+-+April.mp3", "") + if rec.Code == http.StatusOK { + t.Fatal("an empty upload was accepted") + } + if _, err := os.Stat(filepath.Join(dir, "01 - April.mp3")); err == nil { + t.Error("an empty file was left in the library") + } +} + +// A half-written upload must never be left behind under its real name, where +// the feed generator would pick it up as a complete episode. +func TestIngestLeavesNoPartialFile(t *testing.T) { + dir := t.TempDir() + h := ingestServer(t, dir, nil) + + post(t, h, "/ingest?name=01+-+April.mp3", "") + + entries, _ := os.ReadDir(dir) + for _, e := range entries { + if strings.HasPrefix(e.Name(), ".upload-") { + t.Errorf("temporary upload file %q was left behind", e.Name()) + } + } +} + +func TestIngestRequiresPost(t *testing.T) { + h := ingestServer(t, t.TempDir(), nil) + req := httptest.NewRequest(http.MethodGet, "/ingest?name=a.mp3", nil) + req.Header.Set("Authorization", "Bearer secret") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + if rec.Code != http.StatusMethodNotAllowed { + t.Errorf("got %d, want 405", rec.Code) + } +} + +// Without the token, the write endpoints must be as invisible as the feed. +func TestIngestRequiresToken(t *testing.T) { + dir := t.TempDir() + h := ingestServer(t, dir, nil) + + for _, target := range []string{"/ingest?name=01+-+April.mp3", "/manifest"} { + req := httptest.NewRequest(http.MethodPost, target, strings.NewReader("x")) + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + if rec.Code != http.StatusNotFound { + t.Errorf("%s without a token returned %d, want 404", target, rec.Code) + } + } + if _, err := os.Stat(filepath.Join(dir, "01 - April.mp3")); err == nil { + t.Error("an untokened upload was stored") + } +} + +// A server started without --ingest must not expose the write endpoints at all. +func TestIngestDisabledByDefault(t *testing.T) { + dir := t.TempDir() + h := Handler(Options{Dir: dir, Token: "secret"}) + + rec := post(t, h, "/ingest?name=01+-+April.mp3", "x") + if rec.Code == http.StatusOK { + t.Error("ingest was reachable on a server started without it") + } + if _, err := os.Stat(filepath.Join(dir, "01 - April.mp3")); err == nil { + t.Error("a file was stored by a server with ingest disabled") + } +} + +func TestManifestListsAudioOnly(t *testing.T) { + dir := t.TempDir() + write := func(name, body string) { + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + write("01 - April.mp3", "aaa") + write("02 - May.mp3", "bbbb") + write("feed.xml", "") + write("cover.jpg", "img") + + h := ingestServer(t, dir, nil) + req := httptest.NewRequest(http.MethodGet, "/manifest", nil) + req.Header.Set("Authorization", "Bearer secret") + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + + if rec.Code != http.StatusOK { + t.Fatalf("got %d", rec.Code) + } + var m Manifest + if err := json.Unmarshal(rec.Body.Bytes(), &m); err != nil { + t.Fatal(err) + } + if len(m.Files) != 2 { + t.Fatalf("got %v, want 2 audio files", m.Files) + } + if m.Files["01 - April.mp3"] != 3 || m.Files["02 - May.mp3"] != 4 { + t.Errorf("sizes wrong: %v", m.Files) + } +} diff --git a/internal/serve/roundtrip_test.go b/internal/serve/roundtrip_test.go new file mode 100644 index 0000000..c18de79 --- /dev/null +++ b/internal/serve/roundtrip_test.go @@ -0,0 +1,102 @@ +package serve + +import ( + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" + + "podkit/internal/feed" +) + +// A podcast client fetches the feed, then fetches each enclosure as a bare GET +// carrying nothing from the first request. If the server wants a token and the +// enclosure URLs do not have one, the feed loads, lists every episode, and not +// one of them will ever download — a failure that looks like success right up +// until playback. +// +// Found live: a tokened server produced enclosure URLs without the token, and +// every episode 404'd. +func TestTokenedFeedEnclosuresAreFetchable(t *testing.T) { + const token = "s3cret" + + dir := t.TempDir() + for _, name := range []string{"01 - April.mp3", "10 - Probability & Uncertainty.mp3"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte("audio"), 0o644); err != nil { + t.Fatal(err) + } + } + + srv := httptest.NewServer(Handler(Options{Dir: dir, Token: token})) + defer srv.Close() + + // Build the feed exactly as the server would, pointed at itself. + rss, err := feed.Build(feed.Options{ + Dir: dir, + BaseURL: srv.URL + "/audio/", + Title: "Test", + Token: token, + Epoch: time.Date(2000, time.January, 1, 12, 0, 0, 0, time.UTC), + }, nil) + if err != nil { + t.Fatal(err) + } + if len(rss.Channel.Items) != 2 { + t.Fatalf("got %d items", len(rss.Channel.Items)) + } + + for _, item := range rss.Channel.Items { + resp, err := http.Get(item.Enclosure.URL) + if err != nil { + t.Fatalf("GET %s: %v", item.Enclosure.URL, err) + } + resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Errorf("enclosure %s returned %d — a podcast client could not download this episode", + item.Enclosure.URL, resp.StatusCode) + } + } +} + +// The converse: without a token configured anywhere, enclosures must still +// resolve, and must not carry a stray empty k= parameter. +func TestUntokenedFeedEnclosuresAreFetchable(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "01 - April.mp3"), []byte("audio"), 0o644); err != nil { + t.Fatal(err) + } + + srv := httptest.NewServer(Handler(Options{Dir: dir})) + defer srv.Close() + + rss, err := feed.Build(feed.Options{ + Dir: dir, BaseURL: srv.URL + "/audio/", Title: "Test", + }, nil) + if err != nil { + t.Fatal(err) + } + + url := rss.Channel.Items[0].Enclosure.URL + if got := rss.Channel.Items[0].Enclosure.URL; len(got) > 0 && containsQ(got, "k=") { + t.Errorf("enclosure %s carries a token parameter when none was set", got) + } + resp, err := http.Get(url) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Errorf("enclosure %s returned %d", url, resp.StatusCode) + } +} + +func containsQ(h, n string) bool { + for i := 0; i+len(n) <= len(h); i++ { + if h[i:i+len(n)] == n { + return true + } + } + return false +} diff --git a/internal/serve/serve.go b/internal/serve/serve.go index 0b3b340..396babe 100644 --- a/internal/serve/serve.go +++ b/internal/serve/serve.go @@ -23,6 +23,16 @@ type Options struct { // secret in the URL is the mechanism that actually works with them. Anyone // holding the URL holds the access. Token string + + // AllowIngest enables /manifest and /ingest, letting `podkit push` upload + // episodes from elsewhere. Off by default: a feed host that also accepts + // writes is a different security proposition from one that only reads. + AllowIngest bool + + // Rebuild regenerates feed.xml after an upload. Injected rather than built + // here so the server does not need to know the feed's metadata, and so the + // ingest path is testable without a real feed. + Rebuild func() error } // NewToken returns a random token suitable for Options.Token. @@ -45,6 +55,11 @@ func Handler(opts Options) http.Handler { http.ServeFile(w, r, path.Join(opts.Dir, "feed.xml")) }) + if opts.AllowIngest { + mux.HandleFunc("/manifest", handleManifest(opts.Dir)) + mux.HandleFunc("/ingest", handleIngest(opts.Dir, opts.Rebuild)) + } + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/" { http.NotFound(w, r) diff --git a/main.go b/main.go index 2bb3a2c..fb8ddde 100644 --- a/main.go +++ b/main.go @@ -17,11 +17,13 @@ import ( "os" "os/signal" "path/filepath" + "strings" "syscall" "time" "podkit/internal/feed" "podkit/internal/fetch" + "podkit/internal/push" "podkit/internal/serve" "podkit/internal/tag" ) @@ -30,9 +32,12 @@ const usage = `podkit — build a private podcast feed from a playlist or video Usage: podkit fetch [flags] download and extract audio (needs yt-dlp + ffmpeg) + podkit get [flags] download a direct audio URL or an RSS feed's + enclosures (no yt-dlp, no ffmpeg) podkit tag [flags] write ID3 tags from filenames podkit feed [flags] generate feed.xml podkit serve [flags] host the directory and feed over HTTP + podkit push [flags] upload local audio to a remote podkit server podkit all [flags] fetch, tag and feed in one go Run "podkit -h" for that command's flags. @@ -59,6 +64,10 @@ func main() { switch os.Args[1] { case "fetch": err = cmdFetch(ctx, os.Args[2:]) + case "get": + err = cmdGet(ctx, os.Args[2:]) + case "push": + err = cmdPush(ctx, os.Args[2:]) case "tag": err = cmdTag(os.Args[2:]) case "feed": @@ -91,6 +100,7 @@ type metaFlags struct { baseURL string lang string year string + token string } func (m *metaFlags) register(fs *flag.FlagSet) { @@ -102,6 +112,9 @@ func (m *metaFlags) register(fs *flag.FlagSet) { fs.StringVar(&m.baseURL, "base-url", "", "public URL the audio directory is served from") fs.StringVar(&m.lang, "lang", "en", "feed language") fs.StringVar(&m.year, "year", "", "year tag") + // Must match the serving token. Podcast clients send nothing but the URL, + // so without this the enclosures are unreachable behind a tokened server. + fs.StringVar(&m.token, "token", "", "server token, embedded in every enclosure URL") } func (m *metaFlags) albumName() string { @@ -143,6 +156,81 @@ func cmdFetch(ctx context.Context, args []string) error { return nil } +// cmdGet downloads without an extractor: a direct audio URL, or every +// enclosure in an RSS feed. +func cmdGet(ctx context.Context, args []string) error { + fs := flag.NewFlagSet("get", flag.ExitOnError) + dir := fs.String("dir", "./audio", "directory to download into") + if err := fs.Parse(args); err != nil { + return err + } + if fs.NArg() < 1 { + return errors.New("a URL is required: podkit get ") + } + + sources, title, err := fetch.Discover(ctx, fs.Arg(0), nil) + if err != nil { + return err + } + if title != "" { + fmt.Printf("==> %s\n", title) + } + fmt.Printf("==> %d file(s) into %s\n", len(sources), *dir) + if err := fetch.Download(ctx, sources, *dir, nil, os.Stdout); err != nil { + return err + } + if title != "" { + fmt.Printf("\n Suggested: --title %q\n", title) + } + return nil +} + +// cmdPush uploads a local library to a remote podkit server, so the download +// can happen on one machine and the hosting on another. +func cmdPush(ctx context.Context, args []string) error { + fs := flag.NewFlagSet("push", flag.ExitOnError) + dir := fs.String("dir", "./audio", "local directory to upload from") + server := fs.String("server", "", "remote podkit base URL, e.g. https://host/podkit") + token := fs.String("token", "", "the remote server's token") + force := fs.Bool("force", false, "re-upload files the server already has") + if err := fs.Parse(args); err != nil { + return err + } + if *server == "" { + return errors.New("--server is required: the base URL of the remote podkit server") + } + + fmt.Printf("==> Pushing %s to %s\n", *dir, *server) + results, err := push.Run(ctx, push.Options{ + Dir: *dir, Server: *server, Token: *token, Force: *force, + }) + if err != nil { + return err + } + + for _, r := range results { + switch { + case r.Err != nil: + fmt.Fprintf(os.Stderr, " FAILED %s: %v\n", r.Name, r.Err) + case r.Skipped: + fmt.Printf(" have %s\n", r.Name) + default: + fmt.Printf(" sent %s\n", r.Name) + } + } + + uploaded, skipped, failed := push.Summarise(results) + fmt.Printf("==> %d uploaded, %d already present", uploaded, skipped) + if failed > 0 { + fmt.Printf(", %d failed", failed) + } + fmt.Println() + if failed > 0 { + return fmt.Errorf("%d file(s) failed to upload", failed) + } + return nil +} + func cmdTag(args []string) error { fs := flag.NewFlagSet("tag", flag.ExitOnError) var m metaFlags @@ -195,7 +283,13 @@ func cmdFeed(args []string) error { if m.baseURL == "" { return errors.New("--base-url is required: it is the public URL the audio is served from") } + return writeFeed(m, *out) +} +// writeFeed builds and writes feed.xml. Shared by `podkit feed` and the rebuild +// that runs after an ingest, so an uploaded episode goes through exactly the +// same generator as a locally-built one. +func writeFeed(m metaFlags, out string) error { r, err := feed.Build(feed.Options{ Dir: m.dir, BaseURL: m.baseURL, @@ -203,6 +297,7 @@ func cmdFeed(args []string) error { Description: m.desc, Author: m.artist, Language: m.lang, + Token: m.token, Epoch: time.Date(2000, time.January, 1, 12, 0, 0, 0, time.UTC), }, tag.Read) if err != nil { @@ -214,7 +309,7 @@ func cmdFeed(args []string) error { return err } - target := *out + target := out if target == "" { target = filepath.Join(m.dir, "feed.xml") } @@ -222,7 +317,14 @@ func cmdFeed(args []string) error { _, err := os.Stdout.Write(data) return err } - if err := os.WriteFile(target, data, 0o644); err != nil { + // Written via a temporary file and renamed, so a podcast client polling + // mid-write never reads a truncated feed. + tmp := target + ".tmp" + if err := os.WriteFile(tmp, data, 0o644); err != nil { + return fmt.Errorf("writing %s: %w", tmp, err) + } + if err := os.Rename(tmp, target); err != nil { + os.Remove(tmp) return fmt.Errorf("writing %s: %w", target, err) } fmt.Printf("==> Wrote %s (%d episode(s))\n", target, len(r.Channel.Items)) @@ -235,6 +337,12 @@ func cmdServe(ctx context.Context, args []string) error { addr := fs.String("addr", ":8080", "listen address") token := fs.String("token", "", "shared secret required on every request (recommended)") newToken := fs.Bool("new-token", false, "generate a token and use it") + ingest := fs.Bool("ingest", false, "accept uploads from `podkit push` and rebuild the feed") + title := fs.String("title", "", "feed title (required with --ingest)") + artist := fs.String("artist", "", "artist / author (used with --ingest)") + desc := fs.String("description", "", "feed description (used with --ingest)") + baseURL := fs.String("base-url", "", "public URL of the audio directory (required with --ingest)") + lang := fs.String("lang", "en", "feed language") if err := fs.Parse(args); err != nil { return err } @@ -247,20 +355,41 @@ func cmdServe(ctx context.Context, args []string) error { } } + opts := serve.Options{Dir: *dir, Token: tok, AllowIngest: *ingest} + if *ingest { + // An upload that cannot be turned into a feed entry is useless, so the + // metadata is required up front rather than discovered at upload time. + if *title == "" || *baseURL == "" { + return errors.New("--ingest needs --title and --base-url: an uploaded file has to be turned into a feed entry") + } + if tok == "" { + return errors.New("--ingest without a token would let anyone write to the library; pass --token or --new-token") + } + opts.Rebuild = func() error { + return writeFeed(metaFlags{ + dir: *dir, title: *title, artist: *artist, desc: *desc, + baseURL: *baseURL, lang: *lang, token: tok, + }, "") + } + } + srv := &http.Server{ Addr: *addr, - Handler: serve.Handler(serve.Options{Dir: *dir, Token: tok}), + Handler: serve.Handler(opts), ReadHeaderTimeout: 10 * time.Second, } fmt.Printf("==> Serving %s on %s\n", *dir, *addr) if tok != "" { - fmt.Printf(" Feed URL: http://%s/feed.xml?k=%s\n", *addr, tok) + fmt.Printf(" Feed URL: %s/feed.xml?k=%s\n", displayHost(*addr), tok) fmt.Println(" Anyone with that URL has access; it is a shared secret, not a login.") } else { - fmt.Printf(" Feed URL: http://%s/feed.xml\n", *addr) + fmt.Printf(" Feed URL: %s/feed.xml\n", displayHost(*addr)) fmt.Println(" No token set — anyone who can reach this port can read the feed.") } + if *ingest { + fmt.Println(" Accepting uploads from `podkit push`.") + } errCh := make(chan error, 1) go func() { @@ -280,6 +409,15 @@ func cmdServe(ctx context.Context, args []string) error { } } +// displayHost turns a listen address into something printable. A bare ":8080" +// has no host to show, so it gets a placeholder; "127.0.0.1:8080" does not. +func displayHost(addr string) string { + if strings.HasPrefix(addr, ":") { + return "http://" + addr + } + return "http://" + addr +} + func cmdAll(ctx context.Context, args []string) error { fs := flag.NewFlagSet("all", flag.ExitOnError) var m metaFlags @@ -307,6 +445,7 @@ func cmdAll(ctx context.Context, args []string) error { "--dir", m.dir, "--title", m.title, "--artist", m.artist, "--album", m.album, "--description", m.desc, "--base-url", m.baseURL, "--lang", m.lang, "--year", m.year, + "--token", m.token, } if err := cmdFetch(ctx, []string{"--dir", m.dir, fs.Arg(0)}); err != nil { -- cgit v1.2.3