diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-09-26 09:17:14 +0000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-09-26 09:17:14 +0000 |
| commit | 750f4a54cbce570c3d555bceadab68d1b25b47e7 (patch) | |
| tree | fde3b8f76f7f9804473132ffa35758929b5d0b6b /app/src | |
| parent | d4b00e59086e1afc69464164a907bcfddd560a50 (diff) | |
Facebook videos, and a muxed fallback so they can be fetched at all
Individual videos, which is the whole of what is reliably possible: Facebook
pages publish no RSS, so there is no equivalent of subscribing to a YouTube
channel. Flagged that before building rather than quietly delivering half.
A shared link is recognised across the shapes Facebook actually produces --
watch, page videos with or without a slug, reels, fb.watch and /share/v codes,
on www, m and web hosts -- and lands in its own "Saved Facebook videos"
container. Its own, because the container carries the kind that decides how its
episodes are fetched, and one row cannot be honest about two sources. The guid
is prefixed per source too, so a Facebook and a YouTube video that happen to
share a numeric id stay separate episodes.
The og: tag parsing YouTube already did privately is now OpenGraph, used by
both, per the standing instruction about two solutions to the same problem.
It also learned to tolerate either attribute order and single quotes, which
real pages use.
The download seam stops claiming to be YouTube's: yt-dlp takes a URL and works
out the site itself, which is exactly what made this cheap. YouTubeDownloadSource
is ExtractedAudioSource and ChaquopyYouTubeSource is ChaquopyAudioSource.
One real find, and the reason the first attempt failed on a device: Facebook
publishes no audio-only stream at all, only pre-muxed sd/hd mp4, so
"bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio" resolved to nothing and
yt-dlp reported "Requested format is not available". _FORMAT now falls back to
a muxed mp4. That is still a single already-complete file, so the no-ffmpeg
rule holds -- what must never appear there is bestvideo+bestaudio. Media3 plays
an mp4's audio track without complaint, and the data model has been
media-agnostic since phase 1 for this sort of reason.
Verified end to end on a device against a real public Facebook video: resolved
to its own container with the right kind and guid, fetched by yt-dlp, and
played. This also stands as the first proof of the previous commit's
fetch-then-play path against a genuine extractor rather than a direct URL,
since YouTube itself is bot-blocked from this machine.
Known gap, not fixed here: Facebook serves no og: tags to a non-browser at all
-- checked with the app's own user agent -- so a shared video shows as "Video
<id>". yt-dlp knows the real title, and YouTubeAudio.probe is still sitting
unused in the reachability allowlist waiting for exactly this. Small follow-up.
No permission changes, no schema change. 425 unit tests green, detekt and lint
clean, coverage 64.90 -> 65.18.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
Diffstat (limited to 'app/src')
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/Entities.kt | 15 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/Repositories.kt | 4 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt | 29 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt (renamed from app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt) | 10 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt | 19 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/FacebookUrls.kt | 67 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/FeedResolver.kt | 31 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/OpenGraph.kt | 37 | ||||
| -rw-r--r-- | app/src/main/python/ytaudio.py | 19 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt | 84 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt | 125 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt | 67 |
12 files changed, 461 insertions, 46 deletions
diff --git a/app/src/main/java/org/terst/listen/data/Entities.kt b/app/src/main/java/org/terst/listen/data/Entities.kt index 2d0298a..ed80c4b 100644 --- a/app/src/main/java/org/terst/listen/data/Entities.kt +++ b/app/src/main/java/org/terst/listen/data/Entities.kt @@ -10,14 +10,23 @@ import androidx.room.PrimaryKey enum class SubscriptionKind { PODCAST, YOUTUBE_CHANNEL, + + /** + * Individual Facebook videos, shared one at a time. + * + * Not a subscription to anything: Facebook pages publish no RSS, so there is no equivalent + * of following a YouTube channel. + */ + FACEBOOK_VIDEO, ; /** * Whether an episode of this kind needs yt-dlp before it can be played at all. * - * A podcast enclosure is a real audio URL and streams as-is. A YouTube episode's mediaUrl is - * its watch page -- the feed carries no enclosure, and the audio URL only exists once yt-dlp - * has resolved it. Handing that page to ExoPlayer is what made these episodes fail to play. + * A podcast enclosure is a real audio URL and streams as-is. A YouTube or Facebook + * episode's mediaUrl is its watch page -- there is no enclosure, and the audio URL only + * exists once yt-dlp has resolved it. Handing that page to ExoPlayer is what made these + * episodes fail to play. */ val needsExtraction: Boolean get() = this != PODCAST } diff --git a/app/src/main/java/org/terst/listen/data/Repositories.kt b/app/src/main/java/org/terst/listen/data/Repositories.kt index da3c8d6..6604a10 100644 --- a/app/src/main/java/org/terst/listen/data/Repositories.kt +++ b/app/src/main/java/org/terst/listen/data/Repositories.kt @@ -4,7 +4,7 @@ import android.content.Context import coil3.ImageLoader import coil3.network.okhttp.OkHttpNetworkFetcherFactory import okhttp3.OkHttpClient -import org.terst.listen.download.ChaquopyYouTubeSource +import org.terst.listen.download.ChaquopyAudioSource import org.terst.listen.download.EpisodeDownloader import org.terst.listen.feed.ChapterFetcher import org.terst.listen.feed.FeedFetcher @@ -83,7 +83,7 @@ class AppGraph(private val context: Context) { } val downloader: EpisodeDownloader by lazy { - EpisodeDownloader(client, episodes, libraryDir, ChaquopyYouTubeSource(context)) + EpisodeDownloader(client, episodes, libraryDir, ChaquopyAudioSource(context)) } val retention: RetentionPolicy by lazy { diff --git a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt index 5a341ff..5cdc9d4 100644 --- a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt +++ b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt @@ -24,6 +24,8 @@ sealed interface ArchiveOutcome { private const val SAVED_FEED_URL = "listen:saved" private const val SAVED_TITLE = "Saved videos" +private const val SAVED_FACEBOOK_FEED_URL = "listen:saved:facebook" +private const val SAVED_FACEBOOK_TITLE = "Saved Facebook videos" class SubscriptionRepository( private val subscriptions: SubscriptionDao, @@ -149,24 +151,31 @@ class SubscriptionRepository( /** * Adds one video as a standalone episode. * - * Everything lands in a single container subscription rather than one per video: adding - * five would otherwise put five entries in the library that will never publish again. The - * container is marked YOUTUBE_CHANNEL so downloads route through yt-dlp, and its feed URL - * is a local sentinel because there is no feed to refresh. + * Videos land in a container subscription rather than one each: adding five would + * otherwise put five entries in the library that will never publish again. The container + * carries the extracted kind so downloads route through yt-dlp, and its feed URL is a local + * sentinel because there is no feed to refresh. There is one container per source, because + * the kind lives on the container and has to be honest about where its episodes come from. */ suspend fun addSingleEpisode(video: FeedResolver.Resolution.SingleEpisode): SubscriptionEntity { - val container = subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: run { + // One container per source rather than one shared one: a container carries the kind + // that decides how its episodes are fetched, and a single row cannot be honest about + // two sources at once. + val facebook = video.kind == SubscriptionKind.FACEBOOK_VIDEO + val feedUrl = if (facebook) SAVED_FACEBOOK_FEED_URL else SAVED_FEED_URL + + val container = subscriptions.getByFeedUrl(feedUrl) ?: run { val entity = SubscriptionEntity( id = UUID.randomUUID().toString(), - feedUrl = SAVED_FEED_URL, - kind = SubscriptionKind.YOUTUBE_CHANNEL, - title = SAVED_TITLE, + feedUrl = feedUrl, + kind = video.kind, + title = if (facebook) SAVED_FACEBOOK_TITLE else SAVED_TITLE, description = "Individual videos added by sharing a link", author = video.author, addedAtMs = now(), ) subscriptions.insert(entity) - subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: entity + subscriptions.getByFeedUrl(feedUrl) ?: entity } // insertAll ignores a clashing (subscriptionId, guid), so re-adding the same video is a @@ -176,7 +185,7 @@ class SubscriptionRepository( EpisodeEntity( id = UUID.randomUUID().toString(), subscriptionId = container.id, - guid = "yt:video:${video.videoId}", + guid = if (facebook) "fb:video:${video.videoId}" else "yt:video:${video.videoId}", title = video.title?.takeIf { it.isNotBlank() } ?: "Video ${video.videoId}", mediaUrl = video.watchUrl, // yt-dlp is asked for bestaudio, which comes back as m4a. diff --git a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt index 267cc3e..88038f6 100644 --- a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt +++ b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt @@ -7,12 +7,14 @@ import org.terst.listen.youtube.YouTubeAudio import java.io.File /** - * The real YouTube source, backed by yt-dlp on the embedded interpreter. + * The real extractor, backed by yt-dlp on the embedded interpreter. * - * Isolated behind [YouTubeDownloadSource] so the downloader's own logic stays testable off - * device -- nothing here can run without an Android Python runtime. + * Not YouTube-specific: yt-dlp takes a URL and works out the site itself, which is what makes + * Facebook videos nearly free to support. Isolated behind [ExtractedAudioSource] so the + * downloader's own logic stays testable off device -- nothing here runs without an Android + * Python runtime. */ -class ChaquopyYouTubeSource(private val context: Context) : YouTubeDownloadSource { +class ChaquopyAudioSource(private val context: Context) : ExtractedAudioSource { override suspend fun download(url: String, targetDir: File): File = withContext(Dispatchers.IO) { // Idempotent, and cheap after the first call. diff --git a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt index 3a87359..e671aa8 100644 --- a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt +++ b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt @@ -14,7 +14,7 @@ import java.io.IOException /** * Fetches an episode's audio for offline playback. * - * Two sources, because a YouTube subscription's media URL is a watch page rather than audio. + * Two routes, because a YouTube or Facebook media URL is a watch page rather than audio. * Fetching one over plain HTTP would store an HTML document under an .mp3 name that fails to * play with no indication why -- so the kind decides the route, and there is no fallback * between them. @@ -23,7 +23,7 @@ class EpisodeDownloader( private val client: OkHttpClient, private val episodes: EpisodeDao, private val libraryDir: File, - private val youtube: YouTubeDownloadSource? = null, + private val extractor: ExtractedAudioSource? = null, private val now: () -> Long = System::currentTimeMillis, ) { @@ -33,7 +33,12 @@ class EpisodeDownloader( episodes.updateDownload(episode.id, DownloadState.DOWNLOADING, null, null) val result = when (kind) { - SubscriptionKind.YOUTUBE_CHANNEL -> downloadViaYouTube(episode) + // Both extracted kinds take the same route: the source is yt-dlp, and which site + // the URL points at is yt-dlp's problem rather than this app's. + SubscriptionKind.YOUTUBE_CHANNEL, + SubscriptionKind.FACEBOOK_VIDEO, + -> downloadViaExtractor(episode) + SubscriptionKind.PODCAST -> downloadOverHttp(episode) } @@ -56,9 +61,9 @@ class EpisodeDownloader( return File(path).takeIf { it.isFile && it.length() > 0 } } - private suspend fun downloadViaYouTube(episode: EpisodeEntity): Result<File> { - val source = youtube - ?: return Result.failure(IllegalStateException("No YouTube downloader configured")) + private suspend fun downloadViaExtractor(episode: EpisodeEntity): Result<File> { + val source = extractor + ?: return Result.failure(IllegalStateException("No audio extractor configured")) return runCatching { source.download(episode.mediaUrl, libraryDir) } } @@ -141,6 +146,6 @@ class EpisodeDownloader( } /** Indirection so the Chaquopy-backed downloader can be swapped out in tests. */ -interface YouTubeDownloadSource { +interface ExtractedAudioSource { suspend fun download(url: String, targetDir: File): File } diff --git a/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt new file mode 100644 index 0000000..ed4a818 --- /dev/null +++ b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt @@ -0,0 +1,67 @@ +package org.terst.listen.feed + +import java.net.URI + +/** + * Facebook video URL shapes, and what can be read off a video page. + * + * Individual videos only. Facebook pages publish no RSS, so there is no equivalent of + * subscribing to a YouTube channel -- the closest thing would be scraping a page's video list, + * which is login-gated and changes without notice. + * + * Pure and network-free like [YouTubeUrls]: Facebook produces a lot of link shapes, and each + * one is a chance to silently not recognise a video that was shared. + */ +object FacebookUrls { + + private val HOSTS = setOf("facebook.com", "fb.watch", "fb.com") + + /** `/watch?v=<id>`, the desktop share shape. */ + private val WATCH_PARAM = Regex("""[?&]v=(\d+)""") + + /** `/<page>/videos/<id>` or `/<page>/videos/<slug>/<id>`. */ + private val PAGE_VIDEO = Regex("""/videos/(?:[^/?#]+/)?(\d+)""") + + private val REEL = Regex("""/reel/(\d+)""") + + /** `fb.watch/<code>` and `/share/v/<code>` carry a code rather than a numeric id. */ + private val SHARE_CODE = Regex("""/share/v/([\w-]+)""") + + private val PATTERNS = listOf(WATCH_PARAM, PAGE_VIDEO, REEL, SHARE_CODE) + + fun isFacebook(url: String): Boolean { + val host = hostOf(url)?.removePrefix("www.")?.removePrefix("m.")?.removePrefix("web.") + ?: return false + return host in HOSTS + } + + /** + * A stable handle for the video, or null when the link is not a video at all. + * + * Used for the episode's guid, so it only has to be stable per video rather than be + * Facebook's canonical id -- a short link's code qualifies. + */ + fun videoId(url: String): String? { + if (!isFacebook(url)) return null + return PATTERNS.firstNotNullOfOrNull { it.find(url)?.groupValues?.get(1) } + ?: shortLinkCode(url) + } + + /** fb.watch exists only to point at a video, so its path is the handle. */ + private fun shortLinkCode(url: String): String? { + if (hostOf(url)?.removePrefix("www.") != "fb.watch") return null + return runCatching { URI(url).path }.getOrNull() + ?.trim('/') + ?.takeIf { it.isNotEmpty() && !it.contains('/') } + } + + /** What a video page says about itself. Facebook gates a lot, so every field is optional. */ + fun videoMetadataFrom(html: String) = YouTubeUrls.VideoMetadata( + title = OpenGraph.title(html), + imageUrl = OpenGraph.imageUrl(html), + durationMs = null, + author = null, + ) + + private fun hostOf(url: String): String? = runCatching { URI(url).host?.lowercase() }.getOrNull() +} diff --git a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt index 2cc164b..b7dafea 100644 --- a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt +++ b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt @@ -5,6 +5,7 @@ import kotlinx.coroutines.withContext import kotlinx.serialization.SerialName import kotlinx.serialization.Serializable import kotlinx.serialization.json.Json +import org.terst.listen.data.SubscriptionKind import okhttp3.OkHttpClient import okhttp3.Request import java.io.IOException @@ -40,6 +41,8 @@ class FeedResolver( val imageUrl: String?, val durationMs: Long?, val author: String?, + /** Which extractor route the audio comes down; decides where the episode lands. */ + val kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL, ) : Resolution data class Failed(val message: String) : Resolution @@ -51,6 +54,7 @@ class FeedResolver( ?: return Resolution.Failed("No link found in that text.") if (YouTubeUrls.isYouTube(url)) return resolveYouTube(url) + if (FacebookUrls.isFacebook(url)) return resolveFacebook(url) SubscriptionResolver.applePodcastId(url)?.let { return resolveApple(it) } return resolveWeb(url) } @@ -96,6 +100,33 @@ class FeedResolver( Resolution.Feed("https://www.youtube.com/feeds/videos.xml?channel_id=$channelId") } + /** + * A Facebook video, added on its own. + * + * Only individual videos: Facebook pages publish no RSS, so there is nothing to subscribe + * to. The page is fetched for a title and thumbnail, but Facebook gates a great deal behind + * a login, so failing to read either is normal and not an error -- yt-dlp still knows what + * to do with the URL. + */ + private suspend fun resolveFacebook(url: String): Resolution = withContext(Dispatchers.IO) { + val videoId = FacebookUrls.videoId(url) + ?: return@withContext Resolution.Failed( + "That Facebook link doesn't point at a video. Share a video, a reel, or an " + + "fb.watch link." + ) + + val meta = runCatching { fetch(url) }.getOrNull()?.let(FacebookUrls::videoMetadataFrom) + Resolution.SingleEpisode( + watchUrl = url, + videoId = videoId, + title = meta?.title, + imageUrl = meta?.imageUrl, + durationMs = null, + author = null, + kind = SubscriptionKind.FACEBOOK_VIDEO, + ) + } + private suspend fun resolveApple(id: String): Resolution = withContext(Dispatchers.IO) { val response = runCatching { fetch("$lookupBase?id=$id&entity=podcast") } .getOrElse { return@withContext Resolution.Failed("Could not reach Apple: ${it.message}") } diff --git a/app/src/main/java/org/terst/listen/feed/OpenGraph.kt b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt new file mode 100644 index 0000000..54aed6d --- /dev/null +++ b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt @@ -0,0 +1,37 @@ +package org.terst.listen.feed + +/** + * The two Open Graph tags a video page carries: what it is called and its thumbnail. + * + * Extracted from [YouTubeUrls] when Facebook needed exactly the same thing. Per the standing + * instruction in CLAUDE.md, two solutions to the same problem should not quietly diverge. + */ +object OpenGraph { + + fun title(html: String): String? = content(html, "og:title") + + fun imageUrl(html: String): String? = content(html, "og:image") + + private fun content(html: String, property: String): String? = + // Both attribute orders, because real pages use both. + (tag(html, """property=["']$property["'][^>]*?content=["']([^"']*)["']""") + ?: tag(html, """content=["']([^"']*)["'][^>]*?property=["']$property["']""")) + ?.takeIf { it.isNotBlank() } + ?.let(::decodeEntities) + + private fun tag(html: String, pattern: String): String? = + Regex(pattern, RegexOption.IGNORE_CASE).find(html)?.groupValues?.get(1) + + /** + * The handful of entities that actually appear in a meta attribute. + * + * Matters most for the image: a thumbnail URL carries query parameters and the raw + * attribute has them as `&` -- left encoded, the image simply fails to load. + */ + private fun decodeEntities(raw: String) = raw + .replace("&", "&") + .replace(""", "\"") + .replace("'", "'") + .replace("<", "<") + .replace(">", ">") +} diff --git a/app/src/main/python/ytaudio.py b/app/src/main/python/ytaudio.py index 0e5993a..95a2ada 100644 --- a/app/src/main/python/ytaudio.py +++ b/app/src/main/python/ytaudio.py @@ -1,15 +1,22 @@ -"""Audio-only resolution and download via yt-dlp. - -Audio-only on purpose. The documented pain point with yt-dlp on Android is that separate audio -and video streams need ffmpeg to merge; requesting a single pre-muxed audio stream means there -is nothing to merge and no ffmpeg dependency at all. +"""Single-file audio resolution and download via yt-dlp. + +Never a merge, on purpose. The documented pain point with yt-dlp on Android is that separate +audio and video streams need ffmpeg to combine; every selector here picks one already-complete +file, so there is nothing to merge and no ffmpeg dependency at all. + +Audio-only where it exists, which on YouTube it does. Facebook is the reason for the fallback: +it publishes no audio-only stream at all, only pre-muxed sd/hd mp4, so demanding bestaudio +failed outright with "Requested format is not available". A muxed mp4 is still a single file -- +Media3 plays its audio track without complaint, and the model has been media-agnostic since +phase 1 for exactly this sort of reason. What must never appear here is `bestvideo+bestaudio`, +which is the form that needs ffmpeg. """ import json import yt_dlp -_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio" +_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio/best[ext=mp4]/best" def probe(url): diff --git a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt index 93942ed..a5544e7 100644 --- a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt +++ b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt @@ -153,21 +153,28 @@ class SubscriptionRepositoryTest { assertEquals("Renamed Show", subs.rows.single().title) } - private fun video( - id: String, - title: String?, - imageUrl: String? = null, - durationMs: Long? = null, - author: String? = null, - ) = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode( - watchUrl = "https://www.youtube.com/watch?v=$id", - videoId = id, - title = title, - imageUrl = imageUrl, - durationMs = durationMs, - author = author, + /** A resolved video; each test copies it with the one field it is about. */ + private val aVideo = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode( + watchUrl = "https://www.youtube.com/watch?v=aaa", + videoId = "aaa", + title = "First", + imageUrl = null, + durationMs = null, + author = null, ) + private fun video(id: String, title: String?, kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL) = + aVideo.copy( + watchUrl = if (kind == SubscriptionKind.FACEBOOK_VIDEO) { + "https://www.facebook.com/watch?v=$id" + } else { + "https://www.youtube.com/watch?v=$id" + }, + videoId = id, + title = title, + kind = kind, + ) + // --- single episodes ------------------------------------------------------------------ /** @@ -186,7 +193,12 @@ class SubscriptionRepositoryTest { @Test fun `a single video keeps its metadata`() = runTest { repo.addSingleEpisode( - video("aaa", "Feynman on gravitation", "https://img/1.jpg", 3337_000, "Der Atlantik Report") + aVideo.copy( + title = "Feynman on gravitation", + imageUrl = "https://img/1.jpg", + durationMs = 3337_000, + author = "Der Atlantik Report", + ) ) val episode = eps.rows.single() assertEquals("Feynman on gravitation", episode.title) @@ -202,6 +214,50 @@ class SubscriptionRepositoryTest { assertEquals(SubscriptionKind.YOUTUBE_CHANNEL, subs.rows.single().kind) } + /** + * Facebook videos get their own container, because the kind lives on the container and + * decides how its episodes are fetched -- one row cannot be honest about two sources. + */ + @Test + fun `a facebook video lands in its own container`() = runTest { + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + val container = subs.rows.single() + assertEquals(SubscriptionKind.FACEBOOK_VIDEO, container.kind) + assertTrue(container.title.contains("Facebook")) + } + + @Test + fun `facebook and youtube videos do not share a container`() = runTest { + repo.addSingleEpisode(video("aaa", "A video")) + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(2, subs.rows.size) + assertEquals(2, eps.rows.size) + assertEquals( + setOf(SubscriptionKind.YOUTUBE_CHANNEL, SubscriptionKind.FACEBOOK_VIDEO), + subs.rows.map { it.kind }.toSet(), + ) + } + + /** Two sources could hand out the same numeric id; the guid has to keep them apart. */ + @Test + fun `a facebook video and a youtube video with the same id are both kept`() = runTest { + repo.addSingleEpisode(video("123", "From YouTube")) + repo.addSingleEpisode(video("123", "From Facebook", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(2, eps.rows.size) + assertEquals(setOf("yt:video:123", "fb:video:123"), eps.rows.map { it.guid }.toSet()) + } + + @Test + fun `adding the same facebook video twice does not duplicate it`() = runTest { + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(1, eps.rows.size) + } + @Test fun `adding the same video twice does not duplicate it`() = runTest { repo.addSingleEpisode(video("aaa", "First")) diff --git a/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt new file mode 100644 index 0000000..01576e1 --- /dev/null +++ b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt @@ -0,0 +1,125 @@ +package org.terst.listen.feed + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Facebook video URL shapes. + * + * Pure and network-free like [YouTubeUrls], because Facebook produces a lot of link shapes and + * each is a chance to silently not recognise a video Peter shared. + * + * Only individual videos: Facebook pages publish no RSS, so there is no equivalent of + * subscribing to a YouTube channel. + */ +class FacebookUrlsTest { + + @Test + fun `facebook hosts are recognised, including the mobile and short ones`() { + assertTrue(FacebookUrls.isFacebook("https://www.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://m.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://web.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://fb.watch/aBcD1234/")) + } + + @Test + fun `other hosts are not facebook`() { + assertFalse(FacebookUrls.isFacebook("https://www.youtube.com/watch?v=123")) + assertFalse(FacebookUrls.isFacebook("https://example.com/facebook.com/watch")) + assertFalse(FacebookUrls.isFacebook("not a url")) + } + + @Test + fun `a watch link names its video`() { + assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789")) + assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch/?v=123456789")) + } + + @Test + fun `a page video link names its video`() { + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/"), + ) + } + + /** Facebook often puts a human-readable slug before the id. */ + @Test + fun `a page video link with a slug still names its video`() { + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/a-nice-title/987654321/"), + ) + } + + @Test + fun `a reel names its video`() { + assertEquals("55512345", FacebookUrls.videoId("https://www.facebook.com/reel/55512345")) + } + + /** The short link the share sheet produces; its code is the only stable handle. */ + @Test + fun `a short link uses its code`() { + assertEquals("aBcD1234", FacebookUrls.videoId("https://fb.watch/aBcD1234/")) + } + + /** The newer share URL, also just a code. */ + @Test + fun `a share link uses its code`() { + assertEquals("1A2b3C4d5", FacebookUrls.videoId("https://www.facebook.com/share/v/1A2b3C4d5/")) + } + + @Test + fun `query parameters after the id do not confuse it`() { + assertEquals( + "123456789", + FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789&ref=sharing"), + ) + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/?mibextid=x"), + ) + } + + /** Anything that is not a video gets no id, so the resolver can say so rather than guess. */ + @Test + fun `a non-video facebook link has no video id`() { + assertNull(FacebookUrls.videoId("https://www.facebook.com/SomePage")) + assertNull(FacebookUrls.videoId("https://www.facebook.com/groups/12345")) + assertNull(FacebookUrls.videoId("https://www.youtube.com/watch?v=123")) + } + + /** + * What a video page gives up. + * + * Facebook gates a great deal behind a login, so a title and thumbnail are the most that + * can be expected -- and often not even those. Both are optional by design, because yt-dlp + * still knows what to do with the URL when the page says nothing. + */ + @Test + fun `a video page yields its title and thumbnail`() { + val html = """ + <meta property="og:title" content="Sheep dog at work"> + <meta property="og:image" content="https://scontent.xx/t.jpg?a=1&b=2"> + """.trimIndent() + + val meta = FacebookUrls.videoMetadataFrom(html) + + assertEquals("Sheep dog at work", meta.title) + assertEquals("https://scontent.xx/t.jpg?a=1&b=2", meta.imageUrl) + } + + /** A login wall parses to nothing, which is not an error. */ + @Test + fun `a page that says nothing yields empty metadata`() { + val meta = FacebookUrls.videoMetadataFrom("<html><body>Log in to continue</body></html>") + + assertNull(meta.title) + assertNull(meta.imageUrl) + assertNull(meta.durationMs) + } +} diff --git a/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt new file mode 100644 index 0000000..08c2930 --- /dev/null +++ b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt @@ -0,0 +1,67 @@ +package org.terst.listen.feed + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Test + +/** + * The two Open Graph tags every video page carries. + * + * Extracted from YouTubeUrls when Facebook needed the same thing. Per the standing instruction + * in CLAUDE.md: two solutions to the same problem should not quietly diverge. + */ +class OpenGraphTest { + + @Test + fun `title and image are read from the meta tags`() { + val html = """ + <meta property="og:title" content="A video about things"> + <meta property="og:image" content="https://cdn/thumb.jpg"> + """.trimIndent() + + assertEquals("A video about things", OpenGraph.title(html)) + assertEquals("https://cdn/thumb.jpg", OpenGraph.imageUrl(html)) + } + + /** + * Entities are decoded, which matters most for the image. + * + * A thumbnail URL carries query parameters and the raw attribute has them as `&` -- + * left encoded, the image simply fails to load. + */ + @Test + fun `entities are decoded`() { + val html = """<meta property="og:image" content="https://cdn/t.jpg?a=1&b=2">""" + + """<meta property="og:title" content="Bob's "big" day">""" + + assertEquals("https://cdn/t.jpg?a=1&b=2", OpenGraph.imageUrl(html)) + assertEquals("""Bob's "big" day""", OpenGraph.title(html)) + } + + /** Real pages put the attributes in either order; both are the same tag. */ + @Test + fun `attribute order does not matter`() { + val html = """<meta content="Reversed" property="og:title">""" + + assertEquals("Reversed", OpenGraph.title(html)) + } + + @Test + fun `single quotes are accepted`() { + val html = "<meta property='og:title' content='Quoted differently'>" + + assertEquals("Quoted differently", OpenGraph.title(html)) + } + + @Test + fun `a page without the tags yields nothing`() { + assertNull(OpenGraph.title("<html><body>nothing here</body></html>")) + assertNull(OpenGraph.imageUrl("<html><body>nothing here</body></html>")) + } + + /** A login wall is still a page; it must not read as a title. */ + @Test + fun `an empty content attribute counts as absent`() { + assertNull(OpenGraph.title("""<meta property="og:title" content="">""")) + } +} |
