summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPeter Stone <thepeterstone@gmail.com>2026-09-26 09:17:14 +0000
committerPeter Stone <thepeterstone@gmail.com>2026-09-26 09:17:14 +0000
commit750f4a54cbce570c3d555bceadab68d1b25b47e7 (patch)
treefde3b8f76f7f9804473132ffa35758929b5d0b6b
parentd4b00e59086e1afc69464164a907bcfddd560a50 (diff)
Facebook videos, and a muxed fallback so they can be fetched at all
Individual videos, which is the whole of what is reliably possible: Facebook pages publish no RSS, so there is no equivalent of subscribing to a YouTube channel. Flagged that before building rather than quietly delivering half. A shared link is recognised across the shapes Facebook actually produces -- watch, page videos with or without a slug, reels, fb.watch and /share/v codes, on www, m and web hosts -- and lands in its own "Saved Facebook videos" container. Its own, because the container carries the kind that decides how its episodes are fetched, and one row cannot be honest about two sources. The guid is prefixed per source too, so a Facebook and a YouTube video that happen to share a numeric id stay separate episodes. The og: tag parsing YouTube already did privately is now OpenGraph, used by both, per the standing instruction about two solutions to the same problem. It also learned to tolerate either attribute order and single quotes, which real pages use. The download seam stops claiming to be YouTube's: yt-dlp takes a URL and works out the site itself, which is exactly what made this cheap. YouTubeDownloadSource is ExtractedAudioSource and ChaquopyYouTubeSource is ChaquopyAudioSource. One real find, and the reason the first attempt failed on a device: Facebook publishes no audio-only stream at all, only pre-muxed sd/hd mp4, so "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio" resolved to nothing and yt-dlp reported "Requested format is not available". _FORMAT now falls back to a muxed mp4. That is still a single already-complete file, so the no-ffmpeg rule holds -- what must never appear there is bestvideo+bestaudio. Media3 plays an mp4's audio track without complaint, and the data model has been media-agnostic since phase 1 for this sort of reason. Verified end to end on a device against a real public Facebook video: resolved to its own container with the right kind and guid, fetched by yt-dlp, and played. This also stands as the first proof of the previous commit's fetch-then-play path against a genuine extractor rather than a direct URL, since YouTube itself is bot-blocked from this machine. Known gap, not fixed here: Facebook serves no og: tags to a non-browser at all -- checked with the app's own user agent -- so a shared video shows as "Video <id>". yt-dlp knows the real title, and YouTubeAudio.probe is still sitting unused in the reachability allowlist waiting for exactly this. Small follow-up. No permission changes, no schema change. 425 unit tests green, detekt and lint clean, coverage 64.90 -> 65.18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
-rw-r--r--.coverage-baseline2
-rw-r--r--CLAUDE.md33
-rw-r--r--app/src/main/java/org/terst/listen/data/Entities.kt15
-rw-r--r--app/src/main/java/org/terst/listen/data/Repositories.kt4
-rw-r--r--app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt29
-rw-r--r--app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt (renamed from app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt)10
-rw-r--r--app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt19
-rw-r--r--app/src/main/java/org/terst/listen/feed/FacebookUrls.kt67
-rw-r--r--app/src/main/java/org/terst/listen/feed/FeedResolver.kt31
-rw-r--r--app/src/main/java/org/terst/listen/feed/OpenGraph.kt37
-rw-r--r--app/src/main/python/ytaudio.py19
-rw-r--r--app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt84
-rw-r--r--app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt125
-rw-r--r--app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt67
14 files changed, 495 insertions, 47 deletions
diff --git a/.coverage-baseline b/.coverage-baseline
index 92ef7f2..e44f50b 100644
--- a/.coverage-baseline
+++ b/.coverage-baseline
@@ -1 +1 @@
-64.90
+65.18
diff --git a/CLAUDE.md b/CLAUDE.md
index 3e41672..623ba1b 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -104,6 +104,11 @@ claims three separate times across these repos.
queue, play, variable speed, sleep timer, retention, unsubscribe, Android Auto with a browsable
library and voice search. 207 unit tests plus instrumented tests covering the media browse tree.
+**Since Phase 2**, in response to a round of Peter's feedback: yt-dlp sources play (they are
+fetched first), playback errors are surfaced instead of being swallowed, anything queued is
+downloaded at once, Facebook videos are supported, the library ranks shows by their newest
+*unqueued* episode, and trailer/preview filter rules ship as defaults.
+
**Phase 2 is complete**, in three tracks. Peter deferred the audio-fingerprint skip detection
(the design plan's Tier 3), so the simple skip work came last.
@@ -149,6 +154,18 @@ out of a possibly-partial MP3, and the JSON path covers the shows that mark chap
title+date. An episode that loses its identity loses its playback position.
- **Retention is auto-delete-once-played** with a grace period, so a mis-tap doesn't destroy an
episode you were still listening to.
+- **A yt-dlp source has no playable URL until it is fetched.** A YouTube or Facebook episode's
+ `mediaUrl` is its watch page; the audio URL only exists once yt-dlp has resolved it.
+ `SubscriptionKind.needsExtraction` says so, `PlaybackAdvancer.start` reports `NeedsFetch`
+ rather than a `NextUp` that cannot work, and `fetchAndStart` does the fetch. Both the service
+ and the view model go through it -- the queue advances without the UI.
+
+- **Never `bestvideo+bestaudio` in `ytaudio.py`.** That is the form that needs ffmpeg, which is
+ not on Android. Every selector in `_FORMAT` picks one already-complete file. Audio-only comes
+ first, then a muxed mp4: **Facebook publishes no audio-only stream at all**, only pre-muxed
+ sd/hd, so the original audio-only-or-nothing selector failed with "Requested format is not
+ available". A muxed mp4 is still a single file and Media3 plays its audio track.
+
- **YouTube audio is yt-dlp, embedded.** Chaquopy runs an in-process CPython with real yt-dlp;
see `youtube/YouTubeAudio.kt` and `src/main/python/ytaudio.py`. No Termux, no podkit, no
server round trip.
@@ -191,6 +208,22 @@ out of a possibly-partial MP3, and the JSON path covers the shows that mark chap
alias and password, because that is the key already on the phone. Never swap it for a fresh
one — that costs another uninstall. Never use it for a release.
+- **Facebook is individual videos only.** Facebook pages publish no RSS, so there is no
+ equivalent of subscribing to a YouTube channel. Shared videos land in their own "Saved
+ Facebook videos" container, because the container carries the kind that decides how its
+ episodes are fetched and one row cannot be honest about two sources.
+
+- **Facebook tells a non-browser nothing.** It serves no `og:` tags at all -- verified with the
+ app's own user agent -- so a shared video gets a fallback title of "Video <id>". yt-dlp knows
+ the real title (`probe` in `ytaudio.py` returns it, and `YouTubeAudio.probe` is still sitting
+ in the reachability allowlist waiting for a caller). Backfilling from it is the obvious fix
+ and is not done yet.
+
+- **Anything entering the queue is downloaded immediately, on any network.** The hook is
+ `QueueRepository`'s `onQueued`, fired with newly-present ids only, because the queue is the
+ one choke point every route goes through. The one-off job deliberately has no network
+ constraint; only the periodic sweep is wifi-only.
+
- **Feeds are big.** A real subscription in testing produced 2753 episodes in one refresh. Any
per-episode work — a query in a loop, an O(n²) match — needs to be considered at that scale.
diff --git a/app/src/main/java/org/terst/listen/data/Entities.kt b/app/src/main/java/org/terst/listen/data/Entities.kt
index 2d0298a..ed80c4b 100644
--- a/app/src/main/java/org/terst/listen/data/Entities.kt
+++ b/app/src/main/java/org/terst/listen/data/Entities.kt
@@ -10,14 +10,23 @@ import androidx.room.PrimaryKey
enum class SubscriptionKind {
PODCAST,
YOUTUBE_CHANNEL,
+
+ /**
+ * Individual Facebook videos, shared one at a time.
+ *
+ * Not a subscription to anything: Facebook pages publish no RSS, so there is no equivalent
+ * of following a YouTube channel.
+ */
+ FACEBOOK_VIDEO,
;
/**
* Whether an episode of this kind needs yt-dlp before it can be played at all.
*
- * A podcast enclosure is a real audio URL and streams as-is. A YouTube episode's mediaUrl is
- * its watch page -- the feed carries no enclosure, and the audio URL only exists once yt-dlp
- * has resolved it. Handing that page to ExoPlayer is what made these episodes fail to play.
+ * A podcast enclosure is a real audio URL and streams as-is. A YouTube or Facebook
+ * episode's mediaUrl is its watch page -- there is no enclosure, and the audio URL only
+ * exists once yt-dlp has resolved it. Handing that page to ExoPlayer is what made these
+ * episodes fail to play.
*/
val needsExtraction: Boolean get() = this != PODCAST
}
diff --git a/app/src/main/java/org/terst/listen/data/Repositories.kt b/app/src/main/java/org/terst/listen/data/Repositories.kt
index da3c8d6..6604a10 100644
--- a/app/src/main/java/org/terst/listen/data/Repositories.kt
+++ b/app/src/main/java/org/terst/listen/data/Repositories.kt
@@ -4,7 +4,7 @@ import android.content.Context
import coil3.ImageLoader
import coil3.network.okhttp.OkHttpNetworkFetcherFactory
import okhttp3.OkHttpClient
-import org.terst.listen.download.ChaquopyYouTubeSource
+import org.terst.listen.download.ChaquopyAudioSource
import org.terst.listen.download.EpisodeDownloader
import org.terst.listen.feed.ChapterFetcher
import org.terst.listen.feed.FeedFetcher
@@ -83,7 +83,7 @@ class AppGraph(private val context: Context) {
}
val downloader: EpisodeDownloader by lazy {
- EpisodeDownloader(client, episodes, libraryDir, ChaquopyYouTubeSource(context))
+ EpisodeDownloader(client, episodes, libraryDir, ChaquopyAudioSource(context))
}
val retention: RetentionPolicy by lazy {
diff --git a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt
index 5a341ff..5cdc9d4 100644
--- a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt
+++ b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt
@@ -24,6 +24,8 @@ sealed interface ArchiveOutcome {
private const val SAVED_FEED_URL = "listen:saved"
private const val SAVED_TITLE = "Saved videos"
+private const val SAVED_FACEBOOK_FEED_URL = "listen:saved:facebook"
+private const val SAVED_FACEBOOK_TITLE = "Saved Facebook videos"
class SubscriptionRepository(
private val subscriptions: SubscriptionDao,
@@ -149,24 +151,31 @@ class SubscriptionRepository(
/**
* Adds one video as a standalone episode.
*
- * Everything lands in a single container subscription rather than one per video: adding
- * five would otherwise put five entries in the library that will never publish again. The
- * container is marked YOUTUBE_CHANNEL so downloads route through yt-dlp, and its feed URL
- * is a local sentinel because there is no feed to refresh.
+ * Videos land in a container subscription rather than one each: adding five would
+ * otherwise put five entries in the library that will never publish again. The container
+ * carries the extracted kind so downloads route through yt-dlp, and its feed URL is a local
+ * sentinel because there is no feed to refresh. There is one container per source, because
+ * the kind lives on the container and has to be honest about where its episodes come from.
*/
suspend fun addSingleEpisode(video: FeedResolver.Resolution.SingleEpisode): SubscriptionEntity {
- val container = subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: run {
+ // One container per source rather than one shared one: a container carries the kind
+ // that decides how its episodes are fetched, and a single row cannot be honest about
+ // two sources at once.
+ val facebook = video.kind == SubscriptionKind.FACEBOOK_VIDEO
+ val feedUrl = if (facebook) SAVED_FACEBOOK_FEED_URL else SAVED_FEED_URL
+
+ val container = subscriptions.getByFeedUrl(feedUrl) ?: run {
val entity = SubscriptionEntity(
id = UUID.randomUUID().toString(),
- feedUrl = SAVED_FEED_URL,
- kind = SubscriptionKind.YOUTUBE_CHANNEL,
- title = SAVED_TITLE,
+ feedUrl = feedUrl,
+ kind = video.kind,
+ title = if (facebook) SAVED_FACEBOOK_TITLE else SAVED_TITLE,
description = "Individual videos added by sharing a link",
author = video.author,
addedAtMs = now(),
)
subscriptions.insert(entity)
- subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: entity
+ subscriptions.getByFeedUrl(feedUrl) ?: entity
}
// insertAll ignores a clashing (subscriptionId, guid), so re-adding the same video is a
@@ -176,7 +185,7 @@ class SubscriptionRepository(
EpisodeEntity(
id = UUID.randomUUID().toString(),
subscriptionId = container.id,
- guid = "yt:video:${video.videoId}",
+ guid = if (facebook) "fb:video:${video.videoId}" else "yt:video:${video.videoId}",
title = video.title?.takeIf { it.isNotBlank() } ?: "Video ${video.videoId}",
mediaUrl = video.watchUrl,
// yt-dlp is asked for bestaudio, which comes back as m4a.
diff --git a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt
index 267cc3e..88038f6 100644
--- a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt
+++ b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt
@@ -7,12 +7,14 @@ import org.terst.listen.youtube.YouTubeAudio
import java.io.File
/**
- * The real YouTube source, backed by yt-dlp on the embedded interpreter.
+ * The real extractor, backed by yt-dlp on the embedded interpreter.
*
- * Isolated behind [YouTubeDownloadSource] so the downloader's own logic stays testable off
- * device -- nothing here can run without an Android Python runtime.
+ * Not YouTube-specific: yt-dlp takes a URL and works out the site itself, which is what makes
+ * Facebook videos nearly free to support. Isolated behind [ExtractedAudioSource] so the
+ * downloader's own logic stays testable off device -- nothing here runs without an Android
+ * Python runtime.
*/
-class ChaquopyYouTubeSource(private val context: Context) : YouTubeDownloadSource {
+class ChaquopyAudioSource(private val context: Context) : ExtractedAudioSource {
override suspend fun download(url: String, targetDir: File): File = withContext(Dispatchers.IO) {
// Idempotent, and cheap after the first call.
diff --git a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt
index 3a87359..e671aa8 100644
--- a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt
+++ b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt
@@ -14,7 +14,7 @@ import java.io.IOException
/**
* Fetches an episode's audio for offline playback.
*
- * Two sources, because a YouTube subscription's media URL is a watch page rather than audio.
+ * Two routes, because a YouTube or Facebook media URL is a watch page rather than audio.
* Fetching one over plain HTTP would store an HTML document under an .mp3 name that fails to
* play with no indication why -- so the kind decides the route, and there is no fallback
* between them.
@@ -23,7 +23,7 @@ class EpisodeDownloader(
private val client: OkHttpClient,
private val episodes: EpisodeDao,
private val libraryDir: File,
- private val youtube: YouTubeDownloadSource? = null,
+ private val extractor: ExtractedAudioSource? = null,
private val now: () -> Long = System::currentTimeMillis,
) {
@@ -33,7 +33,12 @@ class EpisodeDownloader(
episodes.updateDownload(episode.id, DownloadState.DOWNLOADING, null, null)
val result = when (kind) {
- SubscriptionKind.YOUTUBE_CHANNEL -> downloadViaYouTube(episode)
+ // Both extracted kinds take the same route: the source is yt-dlp, and which site
+ // the URL points at is yt-dlp's problem rather than this app's.
+ SubscriptionKind.YOUTUBE_CHANNEL,
+ SubscriptionKind.FACEBOOK_VIDEO,
+ -> downloadViaExtractor(episode)
+
SubscriptionKind.PODCAST -> downloadOverHttp(episode)
}
@@ -56,9 +61,9 @@ class EpisodeDownloader(
return File(path).takeIf { it.isFile && it.length() > 0 }
}
- private suspend fun downloadViaYouTube(episode: EpisodeEntity): Result<File> {
- val source = youtube
- ?: return Result.failure(IllegalStateException("No YouTube downloader configured"))
+ private suspend fun downloadViaExtractor(episode: EpisodeEntity): Result<File> {
+ val source = extractor
+ ?: return Result.failure(IllegalStateException("No audio extractor configured"))
return runCatching { source.download(episode.mediaUrl, libraryDir) }
}
@@ -141,6 +146,6 @@ class EpisodeDownloader(
}
/** Indirection so the Chaquopy-backed downloader can be swapped out in tests. */
-interface YouTubeDownloadSource {
+interface ExtractedAudioSource {
suspend fun download(url: String, targetDir: File): File
}
diff --git a/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt
new file mode 100644
index 0000000..ed4a818
--- /dev/null
+++ b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt
@@ -0,0 +1,67 @@
+package org.terst.listen.feed
+
+import java.net.URI
+
+/**
+ * Facebook video URL shapes, and what can be read off a video page.
+ *
+ * Individual videos only. Facebook pages publish no RSS, so there is no equivalent of
+ * subscribing to a YouTube channel -- the closest thing would be scraping a page's video list,
+ * which is login-gated and changes without notice.
+ *
+ * Pure and network-free like [YouTubeUrls]: Facebook produces a lot of link shapes, and each
+ * one is a chance to silently not recognise a video that was shared.
+ */
+object FacebookUrls {
+
+ private val HOSTS = setOf("facebook.com", "fb.watch", "fb.com")
+
+ /** `/watch?v=<id>`, the desktop share shape. */
+ private val WATCH_PARAM = Regex("""[?&]v=(\d+)""")
+
+ /** `/<page>/videos/<id>` or `/<page>/videos/<slug>/<id>`. */
+ private val PAGE_VIDEO = Regex("""/videos/(?:[^/?#]+/)?(\d+)""")
+
+ private val REEL = Regex("""/reel/(\d+)""")
+
+ /** `fb.watch/<code>` and `/share/v/<code>` carry a code rather than a numeric id. */
+ private val SHARE_CODE = Regex("""/share/v/([\w-]+)""")
+
+ private val PATTERNS = listOf(WATCH_PARAM, PAGE_VIDEO, REEL, SHARE_CODE)
+
+ fun isFacebook(url: String): Boolean {
+ val host = hostOf(url)?.removePrefix("www.")?.removePrefix("m.")?.removePrefix("web.")
+ ?: return false
+ return host in HOSTS
+ }
+
+ /**
+ * A stable handle for the video, or null when the link is not a video at all.
+ *
+ * Used for the episode's guid, so it only has to be stable per video rather than be
+ * Facebook's canonical id -- a short link's code qualifies.
+ */
+ fun videoId(url: String): String? {
+ if (!isFacebook(url)) return null
+ return PATTERNS.firstNotNullOfOrNull { it.find(url)?.groupValues?.get(1) }
+ ?: shortLinkCode(url)
+ }
+
+ /** fb.watch exists only to point at a video, so its path is the handle. */
+ private fun shortLinkCode(url: String): String? {
+ if (hostOf(url)?.removePrefix("www.") != "fb.watch") return null
+ return runCatching { URI(url).path }.getOrNull()
+ ?.trim('/')
+ ?.takeIf { it.isNotEmpty() && !it.contains('/') }
+ }
+
+ /** What a video page says about itself. Facebook gates a lot, so every field is optional. */
+ fun videoMetadataFrom(html: String) = YouTubeUrls.VideoMetadata(
+ title = OpenGraph.title(html),
+ imageUrl = OpenGraph.imageUrl(html),
+ durationMs = null,
+ author = null,
+ )
+
+ private fun hostOf(url: String): String? = runCatching { URI(url).host?.lowercase() }.getOrNull()
+}
diff --git a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt
index 2cc164b..b7dafea 100644
--- a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt
+++ b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt
@@ -5,6 +5,7 @@ import kotlinx.coroutines.withContext
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
+import org.terst.listen.data.SubscriptionKind
import okhttp3.OkHttpClient
import okhttp3.Request
import java.io.IOException
@@ -40,6 +41,8 @@ class FeedResolver(
val imageUrl: String?,
val durationMs: Long?,
val author: String?,
+ /** Which extractor route the audio comes down; decides where the episode lands. */
+ val kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL,
) : Resolution
data class Failed(val message: String) : Resolution
@@ -51,6 +54,7 @@ class FeedResolver(
?: return Resolution.Failed("No link found in that text.")
if (YouTubeUrls.isYouTube(url)) return resolveYouTube(url)
+ if (FacebookUrls.isFacebook(url)) return resolveFacebook(url)
SubscriptionResolver.applePodcastId(url)?.let { return resolveApple(it) }
return resolveWeb(url)
}
@@ -96,6 +100,33 @@ class FeedResolver(
Resolution.Feed("https://www.youtube.com/feeds/videos.xml?channel_id=$channelId")
}
+ /**
+ * A Facebook video, added on its own.
+ *
+ * Only individual videos: Facebook pages publish no RSS, so there is nothing to subscribe
+ * to. The page is fetched for a title and thumbnail, but Facebook gates a great deal behind
+ * a login, so failing to read either is normal and not an error -- yt-dlp still knows what
+ * to do with the URL.
+ */
+ private suspend fun resolveFacebook(url: String): Resolution = withContext(Dispatchers.IO) {
+ val videoId = FacebookUrls.videoId(url)
+ ?: return@withContext Resolution.Failed(
+ "That Facebook link doesn't point at a video. Share a video, a reel, or an " +
+ "fb.watch link."
+ )
+
+ val meta = runCatching { fetch(url) }.getOrNull()?.let(FacebookUrls::videoMetadataFrom)
+ Resolution.SingleEpisode(
+ watchUrl = url,
+ videoId = videoId,
+ title = meta?.title,
+ imageUrl = meta?.imageUrl,
+ durationMs = null,
+ author = null,
+ kind = SubscriptionKind.FACEBOOK_VIDEO,
+ )
+ }
+
private suspend fun resolveApple(id: String): Resolution = withContext(Dispatchers.IO) {
val response = runCatching { fetch("$lookupBase?id=$id&entity=podcast") }
.getOrElse { return@withContext Resolution.Failed("Could not reach Apple: ${it.message}") }
diff --git a/app/src/main/java/org/terst/listen/feed/OpenGraph.kt b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt
new file mode 100644
index 0000000..54aed6d
--- /dev/null
+++ b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt
@@ -0,0 +1,37 @@
+package org.terst.listen.feed
+
+/**
+ * The two Open Graph tags a video page carries: what it is called and its thumbnail.
+ *
+ * Extracted from [YouTubeUrls] when Facebook needed exactly the same thing. Per the standing
+ * instruction in CLAUDE.md, two solutions to the same problem should not quietly diverge.
+ */
+object OpenGraph {
+
+ fun title(html: String): String? = content(html, "og:title")
+
+ fun imageUrl(html: String): String? = content(html, "og:image")
+
+ private fun content(html: String, property: String): String? =
+ // Both attribute orders, because real pages use both.
+ (tag(html, """property=["']$property["'][^>]*?content=["']([^"']*)["']""")
+ ?: tag(html, """content=["']([^"']*)["'][^>]*?property=["']$property["']"""))
+ ?.takeIf { it.isNotBlank() }
+ ?.let(::decodeEntities)
+
+ private fun tag(html: String, pattern: String): String? =
+ Regex(pattern, RegexOption.IGNORE_CASE).find(html)?.groupValues?.get(1)
+
+ /**
+ * The handful of entities that actually appear in a meta attribute.
+ *
+ * Matters most for the image: a thumbnail URL carries query parameters and the raw
+ * attribute has them as `&amp;` -- left encoded, the image simply fails to load.
+ */
+ private fun decodeEntities(raw: String) = raw
+ .replace("&amp;", "&")
+ .replace("&quot;", "\"")
+ .replace("&#39;", "'")
+ .replace("&lt;", "<")
+ .replace("&gt;", ">")
+}
diff --git a/app/src/main/python/ytaudio.py b/app/src/main/python/ytaudio.py
index 0e5993a..95a2ada 100644
--- a/app/src/main/python/ytaudio.py
+++ b/app/src/main/python/ytaudio.py
@@ -1,15 +1,22 @@
-"""Audio-only resolution and download via yt-dlp.
-
-Audio-only on purpose. The documented pain point with yt-dlp on Android is that separate audio
-and video streams need ffmpeg to merge; requesting a single pre-muxed audio stream means there
-is nothing to merge and no ffmpeg dependency at all.
+"""Single-file audio resolution and download via yt-dlp.
+
+Never a merge, on purpose. The documented pain point with yt-dlp on Android is that separate
+audio and video streams need ffmpeg to combine; every selector here picks one already-complete
+file, so there is nothing to merge and no ffmpeg dependency at all.
+
+Audio-only where it exists, which on YouTube it does. Facebook is the reason for the fallback:
+it publishes no audio-only stream at all, only pre-muxed sd/hd mp4, so demanding bestaudio
+failed outright with "Requested format is not available". A muxed mp4 is still a single file --
+Media3 plays its audio track without complaint, and the model has been media-agnostic since
+phase 1 for exactly this sort of reason. What must never appear here is `bestvideo+bestaudio`,
+which is the form that needs ffmpeg.
"""
import json
import yt_dlp
-_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio"
+_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio/best[ext=mp4]/best"
def probe(url):
diff --git a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt
index 93942ed..a5544e7 100644
--- a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt
+++ b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt
@@ -153,21 +153,28 @@ class SubscriptionRepositoryTest {
assertEquals("Renamed Show", subs.rows.single().title)
}
- private fun video(
- id: String,
- title: String?,
- imageUrl: String? = null,
- durationMs: Long? = null,
- author: String? = null,
- ) = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode(
- watchUrl = "https://www.youtube.com/watch?v=$id",
- videoId = id,
- title = title,
- imageUrl = imageUrl,
- durationMs = durationMs,
- author = author,
+ /** A resolved video; each test copies it with the one field it is about. */
+ private val aVideo = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode(
+ watchUrl = "https://www.youtube.com/watch?v=aaa",
+ videoId = "aaa",
+ title = "First",
+ imageUrl = null,
+ durationMs = null,
+ author = null,
)
+ private fun video(id: String, title: String?, kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL) =
+ aVideo.copy(
+ watchUrl = if (kind == SubscriptionKind.FACEBOOK_VIDEO) {
+ "https://www.facebook.com/watch?v=$id"
+ } else {
+ "https://www.youtube.com/watch?v=$id"
+ },
+ videoId = id,
+ title = title,
+ kind = kind,
+ )
+
// --- single episodes ------------------------------------------------------------------
/**
@@ -186,7 +193,12 @@ class SubscriptionRepositoryTest {
@Test
fun `a single video keeps its metadata`() = runTest {
repo.addSingleEpisode(
- video("aaa", "Feynman on gravitation", "https://img/1.jpg", 3337_000, "Der Atlantik Report")
+ aVideo.copy(
+ title = "Feynman on gravitation",
+ imageUrl = "https://img/1.jpg",
+ durationMs = 3337_000,
+ author = "Der Atlantik Report",
+ )
)
val episode = eps.rows.single()
assertEquals("Feynman on gravitation", episode.title)
@@ -202,6 +214,50 @@ class SubscriptionRepositoryTest {
assertEquals(SubscriptionKind.YOUTUBE_CHANNEL, subs.rows.single().kind)
}
+ /**
+ * Facebook videos get their own container, because the kind lives on the container and
+ * decides how its episodes are fetched -- one row cannot be honest about two sources.
+ */
+ @Test
+ fun `a facebook video lands in its own container`() = runTest {
+ repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO))
+
+ val container = subs.rows.single()
+ assertEquals(SubscriptionKind.FACEBOOK_VIDEO, container.kind)
+ assertTrue(container.title.contains("Facebook"))
+ }
+
+ @Test
+ fun `facebook and youtube videos do not share a container`() = runTest {
+ repo.addSingleEpisode(video("aaa", "A video"))
+ repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO))
+
+ assertEquals(2, subs.rows.size)
+ assertEquals(2, eps.rows.size)
+ assertEquals(
+ setOf(SubscriptionKind.YOUTUBE_CHANNEL, SubscriptionKind.FACEBOOK_VIDEO),
+ subs.rows.map { it.kind }.toSet(),
+ )
+ }
+
+ /** Two sources could hand out the same numeric id; the guid has to keep them apart. */
+ @Test
+ fun `a facebook video and a youtube video with the same id are both kept`() = runTest {
+ repo.addSingleEpisode(video("123", "From YouTube"))
+ repo.addSingleEpisode(video("123", "From Facebook", kind = SubscriptionKind.FACEBOOK_VIDEO))
+
+ assertEquals(2, eps.rows.size)
+ assertEquals(setOf("yt:video:123", "fb:video:123"), eps.rows.map { it.guid }.toSet())
+ }
+
+ @Test
+ fun `adding the same facebook video twice does not duplicate it`() = runTest {
+ repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO))
+ repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO))
+
+ assertEquals(1, eps.rows.size)
+ }
+
@Test
fun `adding the same video twice does not duplicate it`() = runTest {
repo.addSingleEpisode(video("aaa", "First"))
diff --git a/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt
new file mode 100644
index 0000000..01576e1
--- /dev/null
+++ b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt
@@ -0,0 +1,125 @@
+package org.terst.listen.feed
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertNull
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+/**
+ * Facebook video URL shapes.
+ *
+ * Pure and network-free like [YouTubeUrls], because Facebook produces a lot of link shapes and
+ * each is a chance to silently not recognise a video Peter shared.
+ *
+ * Only individual videos: Facebook pages publish no RSS, so there is no equivalent of
+ * subscribing to a YouTube channel.
+ */
+class FacebookUrlsTest {
+
+ @Test
+ fun `facebook hosts are recognised, including the mobile and short ones`() {
+ assertTrue(FacebookUrls.isFacebook("https://www.facebook.com/watch?v=123"))
+ assertTrue(FacebookUrls.isFacebook("https://facebook.com/watch?v=123"))
+ assertTrue(FacebookUrls.isFacebook("https://m.facebook.com/watch?v=123"))
+ assertTrue(FacebookUrls.isFacebook("https://web.facebook.com/watch?v=123"))
+ assertTrue(FacebookUrls.isFacebook("https://fb.watch/aBcD1234/"))
+ }
+
+ @Test
+ fun `other hosts are not facebook`() {
+ assertFalse(FacebookUrls.isFacebook("https://www.youtube.com/watch?v=123"))
+ assertFalse(FacebookUrls.isFacebook("https://example.com/facebook.com/watch"))
+ assertFalse(FacebookUrls.isFacebook("not a url"))
+ }
+
+ @Test
+ fun `a watch link names its video`() {
+ assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789"))
+ assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch/?v=123456789"))
+ }
+
+ @Test
+ fun `a page video link names its video`() {
+ assertEquals(
+ "987654321",
+ FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/"),
+ )
+ }
+
+ /** Facebook often puts a human-readable slug before the id. */
+ @Test
+ fun `a page video link with a slug still names its video`() {
+ assertEquals(
+ "987654321",
+ FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/a-nice-title/987654321/"),
+ )
+ }
+
+ @Test
+ fun `a reel names its video`() {
+ assertEquals("55512345", FacebookUrls.videoId("https://www.facebook.com/reel/55512345"))
+ }
+
+ /** The short link the share sheet produces; its code is the only stable handle. */
+ @Test
+ fun `a short link uses its code`() {
+ assertEquals("aBcD1234", FacebookUrls.videoId("https://fb.watch/aBcD1234/"))
+ }
+
+ /** The newer share URL, also just a code. */
+ @Test
+ fun `a share link uses its code`() {
+ assertEquals("1A2b3C4d5", FacebookUrls.videoId("https://www.facebook.com/share/v/1A2b3C4d5/"))
+ }
+
+ @Test
+ fun `query parameters after the id do not confuse it`() {
+ assertEquals(
+ "123456789",
+ FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789&ref=sharing"),
+ )
+ assertEquals(
+ "987654321",
+ FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/?mibextid=x"),
+ )
+ }
+
+ /** Anything that is not a video gets no id, so the resolver can say so rather than guess. */
+ @Test
+ fun `a non-video facebook link has no video id`() {
+ assertNull(FacebookUrls.videoId("https://www.facebook.com/SomePage"))
+ assertNull(FacebookUrls.videoId("https://www.facebook.com/groups/12345"))
+ assertNull(FacebookUrls.videoId("https://www.youtube.com/watch?v=123"))
+ }
+
+ /**
+ * What a video page gives up.
+ *
+ * Facebook gates a great deal behind a login, so a title and thumbnail are the most that
+ * can be expected -- and often not even those. Both are optional by design, because yt-dlp
+ * still knows what to do with the URL when the page says nothing.
+ */
+ @Test
+ fun `a video page yields its title and thumbnail`() {
+ val html = """
+ <meta property="og:title" content="Sheep dog at work">
+ <meta property="og:image" content="https://scontent.xx/t.jpg?a=1&amp;b=2">
+ """.trimIndent()
+
+ val meta = FacebookUrls.videoMetadataFrom(html)
+
+ assertEquals("Sheep dog at work", meta.title)
+ assertEquals("https://scontent.xx/t.jpg?a=1&b=2", meta.imageUrl)
+ }
+
+ /** A login wall parses to nothing, which is not an error. */
+ @Test
+ fun `a page that says nothing yields empty metadata`() {
+ val meta = FacebookUrls.videoMetadataFrom("<html><body>Log in to continue</body></html>")
+
+ assertNull(meta.title)
+ assertNull(meta.imageUrl)
+ assertNull(meta.durationMs)
+ }
+}
diff --git a/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt
new file mode 100644
index 0000000..08c2930
--- /dev/null
+++ b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt
@@ -0,0 +1,67 @@
+package org.terst.listen.feed
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertNull
+import org.junit.Test
+
+/**
+ * The two Open Graph tags every video page carries.
+ *
+ * Extracted from YouTubeUrls when Facebook needed the same thing. Per the standing instruction
+ * in CLAUDE.md: two solutions to the same problem should not quietly diverge.
+ */
+class OpenGraphTest {
+
+ @Test
+ fun `title and image are read from the meta tags`() {
+ val html = """
+ <meta property="og:title" content="A video about things">
+ <meta property="og:image" content="https://cdn/thumb.jpg">
+ """.trimIndent()
+
+ assertEquals("A video about things", OpenGraph.title(html))
+ assertEquals("https://cdn/thumb.jpg", OpenGraph.imageUrl(html))
+ }
+
+ /**
+ * Entities are decoded, which matters most for the image.
+ *
+ * A thumbnail URL carries query parameters and the raw attribute has them as `&amp;` --
+ * left encoded, the image simply fails to load.
+ */
+ @Test
+ fun `entities are decoded`() {
+ val html = """<meta property="og:image" content="https://cdn/t.jpg?a=1&amp;b=2">""" +
+ """<meta property="og:title" content="Bob&#39;s &quot;big&quot; day">"""
+
+ assertEquals("https://cdn/t.jpg?a=1&b=2", OpenGraph.imageUrl(html))
+ assertEquals("""Bob's "big" day""", OpenGraph.title(html))
+ }
+
+ /** Real pages put the attributes in either order; both are the same tag. */
+ @Test
+ fun `attribute order does not matter`() {
+ val html = """<meta content="Reversed" property="og:title">"""
+
+ assertEquals("Reversed", OpenGraph.title(html))
+ }
+
+ @Test
+ fun `single quotes are accepted`() {
+ val html = "<meta property='og:title' content='Quoted differently'>"
+
+ assertEquals("Quoted differently", OpenGraph.title(html))
+ }
+
+ @Test
+ fun `a page without the tags yields nothing`() {
+ assertNull(OpenGraph.title("<html><body>nothing here</body></html>"))
+ assertNull(OpenGraph.imageUrl("<html><body>nothing here</body></html>"))
+ }
+
+ /** A login wall is still a page; it must not read as a title. */
+ @Test
+ fun `an empty content attribute counts as absent`() {
+ assertNull(OpenGraph.title("""<meta property="og:title" content="">"""))
+ }
+}