diff options
| -rw-r--r-- | .coverage-baseline | 2 | ||||
| -rw-r--r-- | CLAUDE.md | 33 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/Entities.kt | 15 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/Repositories.kt | 4 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt | 29 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt (renamed from app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt) | 10 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt | 19 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/FacebookUrls.kt | 67 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/FeedResolver.kt | 31 | ||||
| -rw-r--r-- | app/src/main/java/org/terst/listen/feed/OpenGraph.kt | 37 | ||||
| -rw-r--r-- | app/src/main/python/ytaudio.py | 19 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt | 84 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt | 125 | ||||
| -rw-r--r-- | app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt | 67 |
14 files changed, 495 insertions, 47 deletions
diff --git a/.coverage-baseline b/.coverage-baseline index 92ef7f2..e44f50b 100644 --- a/.coverage-baseline +++ b/.coverage-baseline @@ -1 +1 @@ -64.90 +65.18 @@ -104,6 +104,11 @@ claims three separate times across these repos. queue, play, variable speed, sleep timer, retention, unsubscribe, Android Auto with a browsable library and voice search. 207 unit tests plus instrumented tests covering the media browse tree. +**Since Phase 2**, in response to a round of Peter's feedback: yt-dlp sources play (they are +fetched first), playback errors are surfaced instead of being swallowed, anything queued is +downloaded at once, Facebook videos are supported, the library ranks shows by their newest +*unqueued* episode, and trailer/preview filter rules ship as defaults. + **Phase 2 is complete**, in three tracks. Peter deferred the audio-fingerprint skip detection (the design plan's Tier 3), so the simple skip work came last. @@ -149,6 +154,18 @@ out of a possibly-partial MP3, and the JSON path covers the shows that mark chap title+date. An episode that loses its identity loses its playback position. - **Retention is auto-delete-once-played** with a grace period, so a mis-tap doesn't destroy an episode you were still listening to. +- **A yt-dlp source has no playable URL until it is fetched.** A YouTube or Facebook episode's + `mediaUrl` is its watch page; the audio URL only exists once yt-dlp has resolved it. + `SubscriptionKind.needsExtraction` says so, `PlaybackAdvancer.start` reports `NeedsFetch` + rather than a `NextUp` that cannot work, and `fetchAndStart` does the fetch. Both the service + and the view model go through it -- the queue advances without the UI. + +- **Never `bestvideo+bestaudio` in `ytaudio.py`.** That is the form that needs ffmpeg, which is + not on Android. Every selector in `_FORMAT` picks one already-complete file. Audio-only comes + first, then a muxed mp4: **Facebook publishes no audio-only stream at all**, only pre-muxed + sd/hd, so the original audio-only-or-nothing selector failed with "Requested format is not + available". A muxed mp4 is still a single file and Media3 plays its audio track. + - **YouTube audio is yt-dlp, embedded.** Chaquopy runs an in-process CPython with real yt-dlp; see `youtube/YouTubeAudio.kt` and `src/main/python/ytaudio.py`. No Termux, no podkit, no server round trip. @@ -191,6 +208,22 @@ out of a possibly-partial MP3, and the JSON path covers the shows that mark chap alias and password, because that is the key already on the phone. Never swap it for a fresh one — that costs another uninstall. Never use it for a release. +- **Facebook is individual videos only.** Facebook pages publish no RSS, so there is no + equivalent of subscribing to a YouTube channel. Shared videos land in their own "Saved + Facebook videos" container, because the container carries the kind that decides how its + episodes are fetched and one row cannot be honest about two sources. + +- **Facebook tells a non-browser nothing.** It serves no `og:` tags at all -- verified with the + app's own user agent -- so a shared video gets a fallback title of "Video <id>". yt-dlp knows + the real title (`probe` in `ytaudio.py` returns it, and `YouTubeAudio.probe` is still sitting + in the reachability allowlist waiting for a caller). Backfilling from it is the obvious fix + and is not done yet. + +- **Anything entering the queue is downloaded immediately, on any network.** The hook is + `QueueRepository`'s `onQueued`, fired with newly-present ids only, because the queue is the + one choke point every route goes through. The one-off job deliberately has no network + constraint; only the periodic sweep is wifi-only. + - **Feeds are big.** A real subscription in testing produced 2753 episodes in one refresh. Any per-episode work — a query in a loop, an O(n²) match — needs to be considered at that scale. diff --git a/app/src/main/java/org/terst/listen/data/Entities.kt b/app/src/main/java/org/terst/listen/data/Entities.kt index 2d0298a..ed80c4b 100644 --- a/app/src/main/java/org/terst/listen/data/Entities.kt +++ b/app/src/main/java/org/terst/listen/data/Entities.kt @@ -10,14 +10,23 @@ import androidx.room.PrimaryKey enum class SubscriptionKind { PODCAST, YOUTUBE_CHANNEL, + + /** + * Individual Facebook videos, shared one at a time. + * + * Not a subscription to anything: Facebook pages publish no RSS, so there is no equivalent + * of following a YouTube channel. + */ + FACEBOOK_VIDEO, ; /** * Whether an episode of this kind needs yt-dlp before it can be played at all. * - * A podcast enclosure is a real audio URL and streams as-is. A YouTube episode's mediaUrl is - * its watch page -- the feed carries no enclosure, and the audio URL only exists once yt-dlp - * has resolved it. Handing that page to ExoPlayer is what made these episodes fail to play. + * A podcast enclosure is a real audio URL and streams as-is. A YouTube or Facebook + * episode's mediaUrl is its watch page -- there is no enclosure, and the audio URL only + * exists once yt-dlp has resolved it. Handing that page to ExoPlayer is what made these + * episodes fail to play. */ val needsExtraction: Boolean get() = this != PODCAST } diff --git a/app/src/main/java/org/terst/listen/data/Repositories.kt b/app/src/main/java/org/terst/listen/data/Repositories.kt index da3c8d6..6604a10 100644 --- a/app/src/main/java/org/terst/listen/data/Repositories.kt +++ b/app/src/main/java/org/terst/listen/data/Repositories.kt @@ -4,7 +4,7 @@ import android.content.Context import coil3.ImageLoader import coil3.network.okhttp.OkHttpNetworkFetcherFactory import okhttp3.OkHttpClient -import org.terst.listen.download.ChaquopyYouTubeSource +import org.terst.listen.download.ChaquopyAudioSource import org.terst.listen.download.EpisodeDownloader import org.terst.listen.feed.ChapterFetcher import org.terst.listen.feed.FeedFetcher @@ -83,7 +83,7 @@ class AppGraph(private val context: Context) { } val downloader: EpisodeDownloader by lazy { - EpisodeDownloader(client, episodes, libraryDir, ChaquopyYouTubeSource(context)) + EpisodeDownloader(client, episodes, libraryDir, ChaquopyAudioSource(context)) } val retention: RetentionPolicy by lazy { diff --git a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt index 5a341ff..5cdc9d4 100644 --- a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt +++ b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt @@ -24,6 +24,8 @@ sealed interface ArchiveOutcome { private const val SAVED_FEED_URL = "listen:saved" private const val SAVED_TITLE = "Saved videos" +private const val SAVED_FACEBOOK_FEED_URL = "listen:saved:facebook" +private const val SAVED_FACEBOOK_TITLE = "Saved Facebook videos" class SubscriptionRepository( private val subscriptions: SubscriptionDao, @@ -149,24 +151,31 @@ class SubscriptionRepository( /** * Adds one video as a standalone episode. * - * Everything lands in a single container subscription rather than one per video: adding - * five would otherwise put five entries in the library that will never publish again. The - * container is marked YOUTUBE_CHANNEL so downloads route through yt-dlp, and its feed URL - * is a local sentinel because there is no feed to refresh. + * Videos land in a container subscription rather than one each: adding five would + * otherwise put five entries in the library that will never publish again. The container + * carries the extracted kind so downloads route through yt-dlp, and its feed URL is a local + * sentinel because there is no feed to refresh. There is one container per source, because + * the kind lives on the container and has to be honest about where its episodes come from. */ suspend fun addSingleEpisode(video: FeedResolver.Resolution.SingleEpisode): SubscriptionEntity { - val container = subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: run { + // One container per source rather than one shared one: a container carries the kind + // that decides how its episodes are fetched, and a single row cannot be honest about + // two sources at once. + val facebook = video.kind == SubscriptionKind.FACEBOOK_VIDEO + val feedUrl = if (facebook) SAVED_FACEBOOK_FEED_URL else SAVED_FEED_URL + + val container = subscriptions.getByFeedUrl(feedUrl) ?: run { val entity = SubscriptionEntity( id = UUID.randomUUID().toString(), - feedUrl = SAVED_FEED_URL, - kind = SubscriptionKind.YOUTUBE_CHANNEL, - title = SAVED_TITLE, + feedUrl = feedUrl, + kind = video.kind, + title = if (facebook) SAVED_FACEBOOK_TITLE else SAVED_TITLE, description = "Individual videos added by sharing a link", author = video.author, addedAtMs = now(), ) subscriptions.insert(entity) - subscriptions.getByFeedUrl(SAVED_FEED_URL) ?: entity + subscriptions.getByFeedUrl(feedUrl) ?: entity } // insertAll ignores a clashing (subscriptionId, guid), so re-adding the same video is a @@ -176,7 +185,7 @@ class SubscriptionRepository( EpisodeEntity( id = UUID.randomUUID().toString(), subscriptionId = container.id, - guid = "yt:video:${video.videoId}", + guid = if (facebook) "fb:video:${video.videoId}" else "yt:video:${video.videoId}", title = video.title?.takeIf { it.isNotBlank() } ?: "Video ${video.videoId}", mediaUrl = video.watchUrl, // yt-dlp is asked for bestaudio, which comes back as m4a. diff --git a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt index 267cc3e..88038f6 100644 --- a/app/src/main/java/org/terst/listen/download/ChaquopyYouTubeSource.kt +++ b/app/src/main/java/org/terst/listen/download/ChaquopyAudioSource.kt @@ -7,12 +7,14 @@ import org.terst.listen.youtube.YouTubeAudio import java.io.File /** - * The real YouTube source, backed by yt-dlp on the embedded interpreter. + * The real extractor, backed by yt-dlp on the embedded interpreter. * - * Isolated behind [YouTubeDownloadSource] so the downloader's own logic stays testable off - * device -- nothing here can run without an Android Python runtime. + * Not YouTube-specific: yt-dlp takes a URL and works out the site itself, which is what makes + * Facebook videos nearly free to support. Isolated behind [ExtractedAudioSource] so the + * downloader's own logic stays testable off device -- nothing here runs without an Android + * Python runtime. */ -class ChaquopyYouTubeSource(private val context: Context) : YouTubeDownloadSource { +class ChaquopyAudioSource(private val context: Context) : ExtractedAudioSource { override suspend fun download(url: String, targetDir: File): File = withContext(Dispatchers.IO) { // Idempotent, and cheap after the first call. diff --git a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt index 3a87359..e671aa8 100644 --- a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt +++ b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt @@ -14,7 +14,7 @@ import java.io.IOException /** * Fetches an episode's audio for offline playback. * - * Two sources, because a YouTube subscription's media URL is a watch page rather than audio. + * Two routes, because a YouTube or Facebook media URL is a watch page rather than audio. * Fetching one over plain HTTP would store an HTML document under an .mp3 name that fails to * play with no indication why -- so the kind decides the route, and there is no fallback * between them. @@ -23,7 +23,7 @@ class EpisodeDownloader( private val client: OkHttpClient, private val episodes: EpisodeDao, private val libraryDir: File, - private val youtube: YouTubeDownloadSource? = null, + private val extractor: ExtractedAudioSource? = null, private val now: () -> Long = System::currentTimeMillis, ) { @@ -33,7 +33,12 @@ class EpisodeDownloader( episodes.updateDownload(episode.id, DownloadState.DOWNLOADING, null, null) val result = when (kind) { - SubscriptionKind.YOUTUBE_CHANNEL -> downloadViaYouTube(episode) + // Both extracted kinds take the same route: the source is yt-dlp, and which site + // the URL points at is yt-dlp's problem rather than this app's. + SubscriptionKind.YOUTUBE_CHANNEL, + SubscriptionKind.FACEBOOK_VIDEO, + -> downloadViaExtractor(episode) + SubscriptionKind.PODCAST -> downloadOverHttp(episode) } @@ -56,9 +61,9 @@ class EpisodeDownloader( return File(path).takeIf { it.isFile && it.length() > 0 } } - private suspend fun downloadViaYouTube(episode: EpisodeEntity): Result<File> { - val source = youtube - ?: return Result.failure(IllegalStateException("No YouTube downloader configured")) + private suspend fun downloadViaExtractor(episode: EpisodeEntity): Result<File> { + val source = extractor + ?: return Result.failure(IllegalStateException("No audio extractor configured")) return runCatching { source.download(episode.mediaUrl, libraryDir) } } @@ -141,6 +146,6 @@ class EpisodeDownloader( } /** Indirection so the Chaquopy-backed downloader can be swapped out in tests. */ -interface YouTubeDownloadSource { +interface ExtractedAudioSource { suspend fun download(url: String, targetDir: File): File } diff --git a/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt new file mode 100644 index 0000000..ed4a818 --- /dev/null +++ b/app/src/main/java/org/terst/listen/feed/FacebookUrls.kt @@ -0,0 +1,67 @@ +package org.terst.listen.feed + +import java.net.URI + +/** + * Facebook video URL shapes, and what can be read off a video page. + * + * Individual videos only. Facebook pages publish no RSS, so there is no equivalent of + * subscribing to a YouTube channel -- the closest thing would be scraping a page's video list, + * which is login-gated and changes without notice. + * + * Pure and network-free like [YouTubeUrls]: Facebook produces a lot of link shapes, and each + * one is a chance to silently not recognise a video that was shared. + */ +object FacebookUrls { + + private val HOSTS = setOf("facebook.com", "fb.watch", "fb.com") + + /** `/watch?v=<id>`, the desktop share shape. */ + private val WATCH_PARAM = Regex("""[?&]v=(\d+)""") + + /** `/<page>/videos/<id>` or `/<page>/videos/<slug>/<id>`. */ + private val PAGE_VIDEO = Regex("""/videos/(?:[^/?#]+/)?(\d+)""") + + private val REEL = Regex("""/reel/(\d+)""") + + /** `fb.watch/<code>` and `/share/v/<code>` carry a code rather than a numeric id. */ + private val SHARE_CODE = Regex("""/share/v/([\w-]+)""") + + private val PATTERNS = listOf(WATCH_PARAM, PAGE_VIDEO, REEL, SHARE_CODE) + + fun isFacebook(url: String): Boolean { + val host = hostOf(url)?.removePrefix("www.")?.removePrefix("m.")?.removePrefix("web.") + ?: return false + return host in HOSTS + } + + /** + * A stable handle for the video, or null when the link is not a video at all. + * + * Used for the episode's guid, so it only has to be stable per video rather than be + * Facebook's canonical id -- a short link's code qualifies. + */ + fun videoId(url: String): String? { + if (!isFacebook(url)) return null + return PATTERNS.firstNotNullOfOrNull { it.find(url)?.groupValues?.get(1) } + ?: shortLinkCode(url) + } + + /** fb.watch exists only to point at a video, so its path is the handle. */ + private fun shortLinkCode(url: String): String? { + if (hostOf(url)?.removePrefix("www.") != "fb.watch") return null + return runCatching { URI(url).path }.getOrNull() + ?.trim('/') + ?.takeIf { it.isNotEmpty() && !it.contains('/') } + } + + /** What a video page says about itself. Facebook gates a lot, so every field is optional. */ + fun videoMetadataFrom(html: String) = YouTubeUrls.VideoMetadata( + title = OpenGraph.title(html), + imageUrl = OpenGraph.imageUrl(html), + durationMs = null, + author = null, + ) + + private fun hostOf(url: String): String? = runCatching { URI(url).host?.lowercase() }.getOrNull() +} diff --git a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt index 2cc164b..b7dafea 100644 --- a/app/src/main/java/org/terst/listen/feed/FeedResolver.kt +++ b/app/src/main/java/org/terst/listen/feed/FeedResolver.kt @@ -5,6 +5,7 @@ import kotlinx.coroutines.withContext import kotlinx.serialization.SerialName import kotlinx.serialization.Serializable import kotlinx.serialization.json.Json +import org.terst.listen.data.SubscriptionKind import okhttp3.OkHttpClient import okhttp3.Request import java.io.IOException @@ -40,6 +41,8 @@ class FeedResolver( val imageUrl: String?, val durationMs: Long?, val author: String?, + /** Which extractor route the audio comes down; decides where the episode lands. */ + val kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL, ) : Resolution data class Failed(val message: String) : Resolution @@ -51,6 +54,7 @@ class FeedResolver( ?: return Resolution.Failed("No link found in that text.") if (YouTubeUrls.isYouTube(url)) return resolveYouTube(url) + if (FacebookUrls.isFacebook(url)) return resolveFacebook(url) SubscriptionResolver.applePodcastId(url)?.let { return resolveApple(it) } return resolveWeb(url) } @@ -96,6 +100,33 @@ class FeedResolver( Resolution.Feed("https://www.youtube.com/feeds/videos.xml?channel_id=$channelId") } + /** + * A Facebook video, added on its own. + * + * Only individual videos: Facebook pages publish no RSS, so there is nothing to subscribe + * to. The page is fetched for a title and thumbnail, but Facebook gates a great deal behind + * a login, so failing to read either is normal and not an error -- yt-dlp still knows what + * to do with the URL. + */ + private suspend fun resolveFacebook(url: String): Resolution = withContext(Dispatchers.IO) { + val videoId = FacebookUrls.videoId(url) + ?: return@withContext Resolution.Failed( + "That Facebook link doesn't point at a video. Share a video, a reel, or an " + + "fb.watch link." + ) + + val meta = runCatching { fetch(url) }.getOrNull()?.let(FacebookUrls::videoMetadataFrom) + Resolution.SingleEpisode( + watchUrl = url, + videoId = videoId, + title = meta?.title, + imageUrl = meta?.imageUrl, + durationMs = null, + author = null, + kind = SubscriptionKind.FACEBOOK_VIDEO, + ) + } + private suspend fun resolveApple(id: String): Resolution = withContext(Dispatchers.IO) { val response = runCatching { fetch("$lookupBase?id=$id&entity=podcast") } .getOrElse { return@withContext Resolution.Failed("Could not reach Apple: ${it.message}") } diff --git a/app/src/main/java/org/terst/listen/feed/OpenGraph.kt b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt new file mode 100644 index 0000000..54aed6d --- /dev/null +++ b/app/src/main/java/org/terst/listen/feed/OpenGraph.kt @@ -0,0 +1,37 @@ +package org.terst.listen.feed + +/** + * The two Open Graph tags a video page carries: what it is called and its thumbnail. + * + * Extracted from [YouTubeUrls] when Facebook needed exactly the same thing. Per the standing + * instruction in CLAUDE.md, two solutions to the same problem should not quietly diverge. + */ +object OpenGraph { + + fun title(html: String): String? = content(html, "og:title") + + fun imageUrl(html: String): String? = content(html, "og:image") + + private fun content(html: String, property: String): String? = + // Both attribute orders, because real pages use both. + (tag(html, """property=["']$property["'][^>]*?content=["']([^"']*)["']""") + ?: tag(html, """content=["']([^"']*)["'][^>]*?property=["']$property["']""")) + ?.takeIf { it.isNotBlank() } + ?.let(::decodeEntities) + + private fun tag(html: String, pattern: String): String? = + Regex(pattern, RegexOption.IGNORE_CASE).find(html)?.groupValues?.get(1) + + /** + * The handful of entities that actually appear in a meta attribute. + * + * Matters most for the image: a thumbnail URL carries query parameters and the raw + * attribute has them as `&` -- left encoded, the image simply fails to load. + */ + private fun decodeEntities(raw: String) = raw + .replace("&", "&") + .replace(""", "\"") + .replace("'", "'") + .replace("<", "<") + .replace(">", ">") +} diff --git a/app/src/main/python/ytaudio.py b/app/src/main/python/ytaudio.py index 0e5993a..95a2ada 100644 --- a/app/src/main/python/ytaudio.py +++ b/app/src/main/python/ytaudio.py @@ -1,15 +1,22 @@ -"""Audio-only resolution and download via yt-dlp. - -Audio-only on purpose. The documented pain point with yt-dlp on Android is that separate audio -and video streams need ffmpeg to merge; requesting a single pre-muxed audio stream means there -is nothing to merge and no ffmpeg dependency at all. +"""Single-file audio resolution and download via yt-dlp. + +Never a merge, on purpose. The documented pain point with yt-dlp on Android is that separate +audio and video streams need ffmpeg to combine; every selector here picks one already-complete +file, so there is nothing to merge and no ffmpeg dependency at all. + +Audio-only where it exists, which on YouTube it does. Facebook is the reason for the fallback: +it publishes no audio-only stream at all, only pre-muxed sd/hd mp4, so demanding bestaudio +failed outright with "Requested format is not available". A muxed mp4 is still a single file -- +Media3 plays its audio track without complaint, and the model has been media-agnostic since +phase 1 for exactly this sort of reason. What must never appear here is `bestvideo+bestaudio`, +which is the form that needs ffmpeg. """ import json import yt_dlp -_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio" +_FORMAT = "bestaudio[ext=m4a]/bestaudio[ext=webm]/bestaudio/best[ext=mp4]/best" def probe(url): diff --git a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt index 93942ed..a5544e7 100644 --- a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt +++ b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt @@ -153,21 +153,28 @@ class SubscriptionRepositoryTest { assertEquals("Renamed Show", subs.rows.single().title) } - private fun video( - id: String, - title: String?, - imageUrl: String? = null, - durationMs: Long? = null, - author: String? = null, - ) = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode( - watchUrl = "https://www.youtube.com/watch?v=$id", - videoId = id, - title = title, - imageUrl = imageUrl, - durationMs = durationMs, - author = author, + /** A resolved video; each test copies it with the one field it is about. */ + private val aVideo = org.terst.listen.feed.FeedResolver.Resolution.SingleEpisode( + watchUrl = "https://www.youtube.com/watch?v=aaa", + videoId = "aaa", + title = "First", + imageUrl = null, + durationMs = null, + author = null, ) + private fun video(id: String, title: String?, kind: SubscriptionKind = SubscriptionKind.YOUTUBE_CHANNEL) = + aVideo.copy( + watchUrl = if (kind == SubscriptionKind.FACEBOOK_VIDEO) { + "https://www.facebook.com/watch?v=$id" + } else { + "https://www.youtube.com/watch?v=$id" + }, + videoId = id, + title = title, + kind = kind, + ) + // --- single episodes ------------------------------------------------------------------ /** @@ -186,7 +193,12 @@ class SubscriptionRepositoryTest { @Test fun `a single video keeps its metadata`() = runTest { repo.addSingleEpisode( - video("aaa", "Feynman on gravitation", "https://img/1.jpg", 3337_000, "Der Atlantik Report") + aVideo.copy( + title = "Feynman on gravitation", + imageUrl = "https://img/1.jpg", + durationMs = 3337_000, + author = "Der Atlantik Report", + ) ) val episode = eps.rows.single() assertEquals("Feynman on gravitation", episode.title) @@ -202,6 +214,50 @@ class SubscriptionRepositoryTest { assertEquals(SubscriptionKind.YOUTUBE_CHANNEL, subs.rows.single().kind) } + /** + * Facebook videos get their own container, because the kind lives on the container and + * decides how its episodes are fetched -- one row cannot be honest about two sources. + */ + @Test + fun `a facebook video lands in its own container`() = runTest { + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + val container = subs.rows.single() + assertEquals(SubscriptionKind.FACEBOOK_VIDEO, container.kind) + assertTrue(container.title.contains("Facebook")) + } + + @Test + fun `facebook and youtube videos do not share a container`() = runTest { + repo.addSingleEpisode(video("aaa", "A video")) + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(2, subs.rows.size) + assertEquals(2, eps.rows.size) + assertEquals( + setOf(SubscriptionKind.YOUTUBE_CHANNEL, SubscriptionKind.FACEBOOK_VIDEO), + subs.rows.map { it.kind }.toSet(), + ) + } + + /** Two sources could hand out the same numeric id; the guid has to keep them apart. */ + @Test + fun `a facebook video and a youtube video with the same id are both kept`() = runTest { + repo.addSingleEpisode(video("123", "From YouTube")) + repo.addSingleEpisode(video("123", "From Facebook", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(2, eps.rows.size) + assertEquals(setOf("yt:video:123", "fb:video:123"), eps.rows.map { it.guid }.toSet()) + } + + @Test + fun `adding the same facebook video twice does not duplicate it`() = runTest { + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + repo.addSingleEpisode(video("111", "A reel", kind = SubscriptionKind.FACEBOOK_VIDEO)) + + assertEquals(1, eps.rows.size) + } + @Test fun `adding the same video twice does not duplicate it`() = runTest { repo.addSingleEpisode(video("aaa", "First")) diff --git a/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt new file mode 100644 index 0000000..01576e1 --- /dev/null +++ b/app/src/test/java/org/terst/listen/feed/FacebookUrlsTest.kt @@ -0,0 +1,125 @@ +package org.terst.listen.feed + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Facebook video URL shapes. + * + * Pure and network-free like [YouTubeUrls], because Facebook produces a lot of link shapes and + * each is a chance to silently not recognise a video Peter shared. + * + * Only individual videos: Facebook pages publish no RSS, so there is no equivalent of + * subscribing to a YouTube channel. + */ +class FacebookUrlsTest { + + @Test + fun `facebook hosts are recognised, including the mobile and short ones`() { + assertTrue(FacebookUrls.isFacebook("https://www.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://m.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://web.facebook.com/watch?v=123")) + assertTrue(FacebookUrls.isFacebook("https://fb.watch/aBcD1234/")) + } + + @Test + fun `other hosts are not facebook`() { + assertFalse(FacebookUrls.isFacebook("https://www.youtube.com/watch?v=123")) + assertFalse(FacebookUrls.isFacebook("https://example.com/facebook.com/watch")) + assertFalse(FacebookUrls.isFacebook("not a url")) + } + + @Test + fun `a watch link names its video`() { + assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789")) + assertEquals("123456789", FacebookUrls.videoId("https://www.facebook.com/watch/?v=123456789")) + } + + @Test + fun `a page video link names its video`() { + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/"), + ) + } + + /** Facebook often puts a human-readable slug before the id. */ + @Test + fun `a page video link with a slug still names its video`() { + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/a-nice-title/987654321/"), + ) + } + + @Test + fun `a reel names its video`() { + assertEquals("55512345", FacebookUrls.videoId("https://www.facebook.com/reel/55512345")) + } + + /** The short link the share sheet produces; its code is the only stable handle. */ + @Test + fun `a short link uses its code`() { + assertEquals("aBcD1234", FacebookUrls.videoId("https://fb.watch/aBcD1234/")) + } + + /** The newer share URL, also just a code. */ + @Test + fun `a share link uses its code`() { + assertEquals("1A2b3C4d5", FacebookUrls.videoId("https://www.facebook.com/share/v/1A2b3C4d5/")) + } + + @Test + fun `query parameters after the id do not confuse it`() { + assertEquals( + "123456789", + FacebookUrls.videoId("https://www.facebook.com/watch?v=123456789&ref=sharing"), + ) + assertEquals( + "987654321", + FacebookUrls.videoId("https://www.facebook.com/SomePage/videos/987654321/?mibextid=x"), + ) + } + + /** Anything that is not a video gets no id, so the resolver can say so rather than guess. */ + @Test + fun `a non-video facebook link has no video id`() { + assertNull(FacebookUrls.videoId("https://www.facebook.com/SomePage")) + assertNull(FacebookUrls.videoId("https://www.facebook.com/groups/12345")) + assertNull(FacebookUrls.videoId("https://www.youtube.com/watch?v=123")) + } + + /** + * What a video page gives up. + * + * Facebook gates a great deal behind a login, so a title and thumbnail are the most that + * can be expected -- and often not even those. Both are optional by design, because yt-dlp + * still knows what to do with the URL when the page says nothing. + */ + @Test + fun `a video page yields its title and thumbnail`() { + val html = """ + <meta property="og:title" content="Sheep dog at work"> + <meta property="og:image" content="https://scontent.xx/t.jpg?a=1&b=2"> + """.trimIndent() + + val meta = FacebookUrls.videoMetadataFrom(html) + + assertEquals("Sheep dog at work", meta.title) + assertEquals("https://scontent.xx/t.jpg?a=1&b=2", meta.imageUrl) + } + + /** A login wall parses to nothing, which is not an error. */ + @Test + fun `a page that says nothing yields empty metadata`() { + val meta = FacebookUrls.videoMetadataFrom("<html><body>Log in to continue</body></html>") + + assertNull(meta.title) + assertNull(meta.imageUrl) + assertNull(meta.durationMs) + } +} diff --git a/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt new file mode 100644 index 0000000..08c2930 --- /dev/null +++ b/app/src/test/java/org/terst/listen/feed/OpenGraphTest.kt @@ -0,0 +1,67 @@ +package org.terst.listen.feed + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Test + +/** + * The two Open Graph tags every video page carries. + * + * Extracted from YouTubeUrls when Facebook needed the same thing. Per the standing instruction + * in CLAUDE.md: two solutions to the same problem should not quietly diverge. + */ +class OpenGraphTest { + + @Test + fun `title and image are read from the meta tags`() { + val html = """ + <meta property="og:title" content="A video about things"> + <meta property="og:image" content="https://cdn/thumb.jpg"> + """.trimIndent() + + assertEquals("A video about things", OpenGraph.title(html)) + assertEquals("https://cdn/thumb.jpg", OpenGraph.imageUrl(html)) + } + + /** + * Entities are decoded, which matters most for the image. + * + * A thumbnail URL carries query parameters and the raw attribute has them as `&` -- + * left encoded, the image simply fails to load. + */ + @Test + fun `entities are decoded`() { + val html = """<meta property="og:image" content="https://cdn/t.jpg?a=1&b=2">""" + + """<meta property="og:title" content="Bob's "big" day">""" + + assertEquals("https://cdn/t.jpg?a=1&b=2", OpenGraph.imageUrl(html)) + assertEquals("""Bob's "big" day""", OpenGraph.title(html)) + } + + /** Real pages put the attributes in either order; both are the same tag. */ + @Test + fun `attribute order does not matter`() { + val html = """<meta content="Reversed" property="og:title">""" + + assertEquals("Reversed", OpenGraph.title(html)) + } + + @Test + fun `single quotes are accepted`() { + val html = "<meta property='og:title' content='Quoted differently'>" + + assertEquals("Quoted differently", OpenGraph.title(html)) + } + + @Test + fun `a page without the tags yields nothing`() { + assertNull(OpenGraph.title("<html><body>nothing here</body></html>")) + assertNull(OpenGraph.imageUrl("<html><body>nothing here</body></html>")) + } + + /** A login wall is still a page; it must not read as a title. */ + @Test + fun `an empty content attribute counts as absent`() { + assertNull(OpenGraph.title("""<meta property="og:title" content="">""")) + } +} |