diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-10-05 09:54:59 +0000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-10-05 09:54:59 +0000 |
| commit | 217c982403a105e986fb7ce13b922d9f08583632 (patch) | |
| tree | 38f2d3a9556b3dfa1b835b9b2c1f7efcb547e719 /app/src | |
| parent | 7cc88394458e9d03e3a6a6e58cdff9c8dd4ad7a3 (diff) | |
Move skip detection to a server, and find that it detects ads
The server is Peter's design and it works: listend polls the phone's feed
list, mirrors the newest 8 episodes per show, detects across all of them at
once, and serves the offsets *and the audio they belong to*. Deployed on
tungsten behind Apache at https://listen.terst.org. FingerprintWorker and
MidrollWorker are deleted -- the battery drain is gone because the work is.
Serving the audio is load-bearing. Dynamic ad insertion turned out to be on
every episode: Grass Apes declares 22:05 and the server was served 28:18. An
offset measured on one copy is minutes wrong for another, so the phone has to
play the bytes the detector examined. The offsets land in the same columns the
on-device detector wrote, so playback needed no changes and there is no
migration.
**The headline finding is a failure, and it is proved rather than suspected.**
Detection reported the intro confirmed on 8 of 8 Radiolab episodes and passed
its self-check. It had locked onto an advertisement. The chosen segment was
searched for in an ad-free copy of the same episode fetched hours earlier:
r = 0.0621. It is in the ad break, not the programme.
The confirmation threshold assumes three *independently fetched* episodes.
Mirroring 8 within one minute from the same ad server makes them maximally
dependent -- whatever campaign is running is in all of them, identical -- and
SelfCheck cannot catch it, because the segments genuinely are the same audio.
Moving to a server fixed the cost problem and broke the independence the
safety mechanism rests on. The published offsets are head-window matches, so
they happen to skip pre-roll advertising, which is useful by accident and will
move whenever the campaign rotates. It is not intro detection and is not
described as such anywhere.
The fix, not made: record when each episode was fetched and require the
agreeing episodes to span more than one fetch occasion. Reasoning, and an
argument for detecting the ads directly instead, in docs/detection-findings.md.
Also recorded there: two tooling traps that each cost real time. `android-ui
tap` can land on the soft keyboard, because the dump reports layout
coordinates the IME does not move -- a Save button and a text field both
appeared tapped and neither fired. And the server's token is alphanumeric now
because `adb shell input text` ate the leading "-" of a base64url one and
produced a silent 401 that looked like broken wiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
Diffstat (limited to 'app/src')
16 files changed, 1096 insertions, 115 deletions
diff --git a/app/src/main/java/org/terst/listen/data/Preferences.kt b/app/src/main/java/org/terst/listen/data/Preferences.kt index 1f7e14f..0c170a1 100644 --- a/app/src/main/java/org/terst/listen/data/Preferences.kt +++ b/app/src/main/java/org/terst/listen/data/Preferences.kt @@ -2,6 +2,7 @@ package org.terst.listen.data import android.content.Context import androidx.datastore.preferences.core.booleanPreferencesKey +import androidx.datastore.preferences.core.stringPreferencesKey import androidx.datastore.preferences.core.edit import androidx.datastore.preferences.preferencesDataStore import kotlinx.coroutines.flow.Flow @@ -31,7 +32,24 @@ class Preferences(private val context: Context) { context.store.edit { it[SHOW_PLAYED] = value } } + /** + * Where the skip server is, and the token for it. Blank means not configured, which is a + * working state: the app downloads from publishers and plays without detected skips, + * exactly as it did before the server existed. + */ + val serverUrl: Flow<String> = context.store.data.map { it[SERVER_URL] ?: "" } + val serverToken: Flow<String> = context.store.data.map { it[SERVER_TOKEN] ?: "" } + + suspend fun setServer(url: String, token: String) { + context.store.edit { + it[SERVER_URL] = url.trim() + it[SERVER_TOKEN] = token.trim() + } + } + private companion object { val SHOW_PLAYED = booleanPreferencesKey("show_played") + val SERVER_URL = stringPreferencesKey("server_url") + val SERVER_TOKEN = stringPreferencesKey("server_token") } } diff --git a/app/src/main/java/org/terst/listen/data/Repositories.kt b/app/src/main/java/org/terst/listen/data/Repositories.kt index 1b174cf..e425ccf 100644 --- a/app/src/main/java/org/terst/listen/data/Repositories.kt +++ b/app/src/main/java/org/terst/listen/data/Repositories.kt @@ -3,7 +3,12 @@ package org.terst.listen.data import android.content.Context import coil3.ImageLoader import coil3.network.okhttp.OkHttpNetworkFetcherFactory +import kotlinx.coroutines.flow.first import okhttp3.OkHttpClient +import org.terst.listen.server.OkHttpSkipTransport +import org.terst.listen.server.ServerSkips +import org.terst.listen.server.SkipServer +import org.terst.listen.server.SkipServerConfig import org.terst.listen.download.ChaquopyAudioSource import org.terst.listen.download.EpisodeDownloader import org.terst.listen.feed.ChapterFetcher @@ -96,8 +101,30 @@ class AppGraph(private val context: Context) { subscriptions, episodes, FeedFetcher(client), - onDiscovered = { ruleApplier.apply(it) }, - probe = { url -> ChaquopyAudioSource(context).probe(url) }, + SubscriptionCollaborators( + onDiscovered = { ruleApplier.apply(it) }, + probe = { url -> ChaquopyAudioSource(context).probe(url) }, + onRefreshed = { feedUrls -> skipServer.pushSubscriptions(feedUrls) }, + ), + ) + } + + /** + * The skip server, which is where detection happens now. + * + * It moved off the phone because it could never work here: comparing several episodes' + * audio at once needs more memory than an Android process gets and more time than + * WorkManager's ten-minute ceiling allows, and the attempt burned the battery for weeks + * while confirming nothing. + */ + val skipServer: SkipServer by lazy { + SkipServer( + config = { + val url = preferences.serverUrl.first() + val token = preferences.serverToken.first() + if (url.isBlank() || token.isBlank()) null else SkipServerConfig(url, token) + }, + transport = OkHttpSkipTransport(client), ) } @@ -107,10 +134,7 @@ class AppGraph(private val context: Context) { episodes, libraryDir, ChaquopyAudioSource(context), - // Fingerprinting needs the file on disk to decode, so it can only start once a - // download actually lands -- one job per episode rather than a periodic sweep, - // since the file is right there and there is nothing to gain by waiting. - onDownloaded = { episode -> Scheduling.fingerprintNow(context, episode.id) }, + serverAudio = ServerSkips(skipServer, subscriptions, fingerprints, playbackState), ) } diff --git a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt index 9b94c75..a66441a 100644 --- a/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt +++ b/app/src/main/java/org/terst/listen/data/SubscriptionRepository.kt @@ -30,26 +30,43 @@ private const val SAVED_TITLE = "Saved videos" private const val SAVED_FACEBOOK_FEED_URL = "listen:saved:facebook" private const val SAVED_FACEBOOK_TITLE = "Saved Facebook videos" -class SubscriptionRepository( - private val subscriptions: SubscriptionDao, - private val episodes: EpisodeDao, - private val fetcher: FeedFetcher, +/** + * Everything [SubscriptionRepository] tells the rest of the app about, bundled. + * + * Grouped rather than passed one by one so the constructor stays a readable, fixed size as more + * of the app comes to care what a refresh found -- the same reasoning as [RuleEngine]'s and + * [org.terst.listen.fingerprint.MidrollUpdater]'s own tuning values. + */ +data class SubscriptionCollaborators( /** * Told about episodes a refresh has just discovered, so the auto-queue rules can act on them. * - * A collaborator rather than a call into the queue, so this class never learns what a queue - * is. Deliberately not invoked by [subscribe] or [addSingleEpisode]: see their comments. + * Deliberately not invoked by `subscribe` or `addSingleEpisode`: see their comments. */ - private val onDiscovered: suspend (List<EpisodeEntity>) -> Unit = {}, + val onDiscovered: suspend (List<EpisodeEntity>) -> Unit = {}, /** * Asks the extractor what a video is, for the cases where its page will not say. * * Facebook serves no `og:` tags to a non-browser at all, so a shared video would otherwise * be filed as "Video 10153231379946729". yt-dlp knows, and is going to be run against the - * URL anyway. A collaborator rather than a direct call because it reaches the embedded - * Python, which cannot run in a unit test. + * URL anyway. + */ + val probe: suspend (String) -> ProbedVideo? = { null }, + /** + * Told the subscription list on every refresh, so the skip server knows which feeds to + * mirror and analyse. + * + * Fired from `refreshAll`, the choke point every refresh route passes through -- including + * the library's own refresh button, which does not go via a worker. */ - private val probe: suspend (String) -> ProbedVideo? = { null }, + val onRefreshed: suspend (List<String>) -> Unit = {}, +) + +class SubscriptionRepository( + private val subscriptions: SubscriptionDao, + private val episodes: EpisodeDao, + private val fetcher: FeedFetcher, + private val collaborators: SubscriptionCollaborators = SubscriptionCollaborators(), private val now: () -> Long = System::currentTimeMillis, ) { @@ -108,8 +125,15 @@ class SubscriptionRepository( } } - suspend fun refreshAll(): Map<String, RefreshOutcome> = - subscriptions.getAll().associate { it.id to refresh(it) } + suspend fun refreshAll(): Map<String, RefreshOutcome> { + val shows = subscriptions.getAll() + // The one place every refresh passes through, which is why the server is told here + // rather than in a worker: the refresh button in the library calls this directly and + // would otherwise never announce a new subscription. Failure is ignored on purpose -- + // the skip server improves playback rather than being a precondition for it. + collaborators.onRefreshed(shows.map { it.feedUrl }) + return shows.associate { it.id to refresh(it) } + } /** * Pulls a show's back catalogue by walking its RFC 5005 `prev-archive` chain. @@ -156,7 +180,7 @@ class SubscriptionRepository( val result = EpisodeMatcher.match(subscriptionId, stored, feed.episodes, now()) if (result.inserts.isNotEmpty()) episodes.insertAll(result.inserts) if (result.updates.isNotEmpty()) episodes.updateAll(result.updates) - if (report && result.inserts.isNotEmpty()) onDiscovered(result.inserts) + if (report && result.inserts.isNotEmpty()) collaborators.onDiscovered(result.inserts) return RefreshOutcome.Updated(result.inserts.size, result.updates.size) } @@ -192,7 +216,7 @@ class SubscriptionRepository( // Only when the page gave nothing: a title that was shared is the one that was meant, // and this costs a network round trip. - val probed = if (video.title.isNullOrBlank()) probe(video.watchUrl) else null + val probed = if (video.title.isNullOrBlank()) collaborators.probe(video.watchUrl) else null val guid = if (facebook) "fb:video:${video.videoId}" else "yt:video:${video.videoId}" val existing = episodes.getByGuid(container.id, guid) diff --git a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt index ed8bf70..32180c4 100644 --- a/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt +++ b/app/src/main/java/org/terst/listen/download/EpisodeDownloader.kt @@ -9,6 +9,7 @@ import org.terst.listen.data.EpisodeDao import org.terst.listen.data.EpisodeEntity import org.terst.listen.data.ProbedVideo import org.terst.listen.data.SubscriptionKind +import org.terst.listen.server.ServerEpisode import java.io.File import java.io.IOException @@ -20,18 +21,35 @@ import java.io.IOException * play with no indication why -- so the kind decides the route, and there is no fallback * between them. */ +/** + * The skip server's part in a download: where the audio should come from, and what to write + * down once it has arrived. + * + * Behind an interface because the decision belongs to the download path while the HTTP and the + * database writes do not. + */ +interface ServerAudio { + /** What the server has for this episode, or null to download from the publisher. */ + suspend fun resolve(episode: EpisodeEntity): ServerEpisode? + + /** Records the server's offsets, once its audio is actually the file on disk. */ + suspend fun record(episode: EpisodeEntity, info: ServerEpisode) +} + class EpisodeDownloader( private val client: OkHttpClient, private val episodes: EpisodeDao, private val libraryDir: File, private val extractor: ExtractedAudioSource? = null, /** - * Fired once a download actually lands on disk -- the trigger for Tier 3 fingerprint - * matching, which needs the file present to decode. Every download route ends up here - * regardless of whether it was a manual tap or a queued/rule-triggered batch, so this is - * the one place that guarantees every newly downloaded episode gets checked. + * The skip server, preferred over the publisher when it has the episode. + * + * Not an optimisation. A show using dynamic ad insertion serves a different file on every + * request -- one real episode measured 22:05 on one fetch and 28:18 on another -- so an + * offset computed on the server's copy points at the wrong moment in any other copy. + * Playing the bytes the detector examined is the only thing that makes the offsets true. */ - private val onDownloaded: suspend (EpisodeEntity) -> Unit = {}, + private val serverAudio: ServerAudio? = null, private val now: () -> Long = System::currentTimeMillis, ) { @@ -47,13 +65,15 @@ class EpisodeDownloader( SubscriptionKind.FACEBOOK_VIDEO, -> downloadViaExtractor(episode) - SubscriptionKind.PODCAST -> downloadOverHttp(episode) + SubscriptionKind.PODCAST -> downloadPodcast(episode) } result.fold( + // Nothing is triggered by a completed download any more. It used to start Tier 3 + // fingerprinting, which has moved to the server -- where comparing several + // episodes at once is possible at all. onSuccess = { episodes.updateDownload(episode.id, DownloadState.DOWNLOADED, it.absolutePath, now()) - onDownloaded(episode.copy(downloadState = DownloadState.DOWNLOADED, localPath = it.absolutePath)) }, // localPath is cleared: a FAILED row pointing at a path invites something downstream // to try playing a file that is not there. @@ -72,13 +92,35 @@ class EpisodeDownloader( return File(path).takeIf { it.isFile && it.length() > 0 } } + /** + * Prefers the skip server, falling back to the publisher. + * + * The fallback is deliberate and matters: skips improve playback rather than being a + * precondition for it, so a server that is down or mid-fetch must not turn a reachable + * episode into a failed download. Offsets are recorded only when the server's audio is + * actually what landed on disk. + */ + private suspend fun downloadPodcast(episode: EpisodeEntity): Result<File> { + val fromServer = serverAudio?.resolve(episode) + val serverUrl = fromServer?.audioUrl?.takeIf { fromServer.ready } + + if (serverUrl != null) { + val result = downloadOverHttp(episode, serverUrl) + if (result.isSuccess) { + serverAudio?.record(episode, fromServer) + return result + } + } + return downloadOverHttp(episode, episode.mediaUrl) + } + private suspend fun downloadViaExtractor(episode: EpisodeEntity): Result<File> { val source = extractor ?: return Result.failure(IllegalStateException("No audio extractor configured")) return runCatching { source.download(episode.mediaUrl, libraryDir) } } - private suspend fun downloadOverHttp(episode: EpisodeEntity): Result<File> = + private suspend fun downloadOverHttp(episode: EpisodeEntity, url: String): Result<File> = withContext(Dispatchers.IO) { val target = targetFile(episode) // Written to a partial name and renamed on success. A truncated file recorded as @@ -87,7 +129,7 @@ class EpisodeDownloader( partial.parentFile?.mkdirs() try { - val request = Request.Builder().url(episode.mediaUrl).header("User-Agent", USER_AGENT).build() + val request = Request.Builder().url(url).header("User-Agent", USER_AGENT).build() client.newCall(request).execute().use { response -> if (!response.isSuccessful) { partial.delete() diff --git a/app/src/main/java/org/terst/listen/server/OkHttpSkipTransport.kt b/app/src/main/java/org/terst/listen/server/OkHttpSkipTransport.kt new file mode 100644 index 0000000..1541dfb --- /dev/null +++ b/app/src/main/java/org/terst/listen/server/OkHttpSkipTransport.kt @@ -0,0 +1,50 @@ +package org.terst.listen.server + +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody + +/** + * [SkipTransport] over the app's own OkHttp client. + * + * On the framework-adapter exclusion list: there is no decision in here, only the HTTP that + * [SkipServer]'s logic is deliberately kept free of. + * + * Every failure is a null or a false rather than an exception. The server is an improvement the + * app prefers, not something it depends on -- so unreachable, slow and broken all have to mean + * the same thing to the caller as not configured: carry on without it. + */ +class OkHttpSkipTransport(private val client: OkHttpClient) : SkipTransport { + + override suspend fun get(url: String, token: String): String? = withContext(Dispatchers.IO) { + runCatching { + val request = Request.Builder() + .url(url) + .header("Authorization", "Bearer $token") + .build() + client.newCall(request).execute().use { response -> + if (!response.isSuccessful) return@use null + response.body?.string() + } + }.getOrNull() + } + + override suspend fun post(url: String, token: String, body: String): Boolean = + withContext(Dispatchers.IO) { + runCatching { + val request = Request.Builder() + .url(url) + .header("Authorization", "Bearer $token") + .post(body.toRequestBody(JSON)) + .build() + client.newCall(request).execute().use { it.isSuccessful } + }.getOrDefault(false) + } + + private companion object { + val JSON = "application/json; charset=utf-8".toMediaType() + } +} diff --git a/app/src/main/java/org/terst/listen/server/ServerSkips.kt b/app/src/main/java/org/terst/listen/server/ServerSkips.kt new file mode 100644 index 0000000..edff77b --- /dev/null +++ b/app/src/main/java/org/terst/listen/server/ServerSkips.kt @@ -0,0 +1,56 @@ +package org.terst.listen.server + +import org.terst.listen.data.EpisodeEntity +import org.terst.listen.data.PlaybackStateDao +import org.terst.listen.data.ShowFingerprintDao +import org.terst.listen.data.SubscriptionDao +import org.terst.listen.download.ServerAudio + +/** + * Connects the skip server to the download path. + * + * What the server returns goes into exactly the columns the on-device detector used to write, + * which is why playback needed no changes at all: `effectiveIntroMs`, `effectiveOutroMs` and + * `PlayheadPolicy` already read them. Only the origin of the numbers has moved. + */ +class ServerSkips( + private val server: SkipServer, + private val subscriptions: SubscriptionDao, + private val fingerprints: ShowFingerprintDao, + private val playbackState: PlaybackStateDao, +) : ServerAudio { + + /** + * Asks the server about an episode, identifying it the way the server keys them. + * + * Three identifiers are sent, not one, because publishers rewrite feeds: a GUID that has + * moved still matches on its enclosure URL, and failing that on its title. An episode that + * loses its identity loses its offsets. + */ + override suspend fun resolve(episode: EpisodeEntity): ServerEpisode? { + val feedUrl = subscriptions.getById(episode.subscriptionId)?.feedUrl ?: return null + return server.episode( + feedUrl = feedUrl, + guid = episode.guid, + enclosureUrl = episode.mediaUrl, + title = episode.title, + ) + } + + /** + * Writes down what the server knew, once its audio is the file on disk. + * + * Each value is written only if the server actually reported it. A missing offset is not a + * zero: a detected zero means "this show has no intro", which playback treats differently + * from having nothing detected, and collapsing the two would make a real finding + * indistinguishable from an absent one. + */ + override suspend fun record(episode: EpisodeEntity, info: ServerEpisode) { + info.introMs?.let { fingerprints.setDetectedIntroMs(episode.id, it) } + info.outroMs?.let { fingerprints.setDetectedOutroMs(episode.id, it) } + // The length the file really is. The feed's figure is minutes short on any show using + // dynamic ad insertion, and this is the number that places the outro offset and decides + // when an episode counts as finished. + info.durationMs?.let { playbackState.updateDuration(episode.id, it) } + } +} diff --git a/app/src/main/java/org/terst/listen/server/SkipServer.kt b/app/src/main/java/org/terst/listen/server/SkipServer.kt new file mode 100644 index 0000000..a480fb5 --- /dev/null +++ b/app/src/main/java/org/terst/listen/server/SkipServer.kt @@ -0,0 +1,144 @@ +package org.terst.listen.server + +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.JsonPrimitive + +/** + * Where the skip server is and how to authenticate to it. + * + * Null everywhere it appears means "not configured", which is a working state: the app falls + * back to downloading from publishers with no detected skips, exactly as it did before the + * server existed. + */ +data class SkipServerConfig(val baseUrl: String, val token: String) + +/** The HTTP the client needs, behind an interface so its logic is testable off-device. */ +interface SkipTransport { + /** Returns the response body, or null for any failure. */ + suspend fun get(url: String, token: String): String? + + /** Returns whether the post succeeded. */ + suspend fun post(url: String, token: String, body: String): Boolean +} + +/** What the server knows about one episode. */ +data class ServerEpisode( + /** Whether the audio can be played from the server now. */ + val ready: Boolean, + /** An absolute, token-carrying URL, usable by the player as given. */ + val audioUrl: String?, + /** The file's real length, which is routinely not what the feed declared. */ + val durationMs: Long?, + val introMs: Long?, + val outroMs: Long?, + /** + * Whether detection has run over this episode's show. Distinct from having offsets: a show + * can be examined and genuinely have no recurring intro, and reporting that as "nothing + * found yet" is the mistake the app's own readout used to make. + */ + val detected: Boolean, +) + +/** + * The phone's half of the skip server. + * + * Detection itself moved off the phone. The intro search compares several episodes' audio at + * once, which needs more memory than an Android process is given and more time than + * WorkManager's ten-minute ceiling allows -- the attempt burned the battery for weeks and + * confirmed nothing. What is left here is asking. + * + * The server serves the audio as well as the offsets, and that is load-bearing rather than + * convenient: a show using dynamic ad insertion returns a different file on every request, so + * an offset measured on the server's copy points at the wrong moment in a copy the phone + * fetched for itself. Measured on one real episode: 22:05 on one fetch, 28:18 on another. + * Playing the server's bytes is what makes its numbers true. + */ +class SkipServer( + private val config: suspend () -> SkipServerConfig?, + private val transport: SkipTransport, +) { + + /** + * Tells the server what the phone subscribes to. + * + * The phone owns this list: everything already built around it -- the share sheet, OPML + * import, the queue rules, Android Auto's browse tree -- keeps working untouched, and the + * server stays something the app prefers rather than depends on. + */ + suspend fun pushSubscriptions(feedUrls: List<String>): Boolean { + val target = config() ?: return false + val body = buildString { + append("""{"feeds":[""") + feedUrls.forEachIndexed { index, url -> + if (index > 0) append(",") + append(JsonPrimitive(url)) + } + append("]}") + } + return transport.post(base(target) + "/api/subscriptions", target.token, body) + } + + /** + * Asks what to skip in one episode, and where its audio is. + * + * Null whenever there is no answer -- not configured, unreachable, or a response that is not + * what this expects. Every one of those means the same thing to the caller: carry on the way + * the app worked before the server existed. + */ + suspend fun episode( + feedUrl: String, + guid: String, + enclosureUrl: String, + title: String, + ): ServerEpisode? { + val target = config() ?: return null + val query = buildString { + append("?feed=").append(encode(feedUrl)) + append("&guid=").append(encode(guid)) + append("&enclosure=").append(encode(enclosureUrl)) + append("&title=").append(encode(title)) + } + + val body = transport.get(base(target) + "/api/episode" + query, target.token) ?: return null + val parsed = runCatching { lenient.decodeFromString<EpisodeResponse>(body) }.getOrNull() ?: return null + + return ServerEpisode( + ready = parsed.status == "ready" && parsed.audioUrl != null, + audioUrl = parsed.audioUrl?.let { absolute(base(target), it) }, + durationMs = parsed.durationMs?.takeIf { it > 0 }, + introMs = parsed.introMs, + outroMs = parsed.outroMs, + detected = parsed.detected, + ) + } + + private fun base(target: SkipServerConfig) = target.baseUrl.trimEnd('/') + + /** + * The server returns a root-relative audio path; a player handed that unresolved would try + * to open a local file. + */ + private fun absolute(base: String, path: String): String = + if (path.startsWith("http://") || path.startsWith("https://")) path else base + path + + @Serializable + private data class EpisodeResponse( + val status: String = "", + @SerialName("audioUrl") val audioUrl: String? = null, + @SerialName("durationMs") val durationMs: Long? = null, + @SerialName("introMs") val introMs: Long? = null, + @SerialName("outroMs") val outroMs: Long? = null, + val detected: Boolean = false, + ) + + private companion object { + /** Tolerant of fields this app does not know about, so the server can grow. */ + val lenient = Json { ignoreUnknownKeys = true; isLenient = true } + } +} + +/** Percent-encoding for a query value. A feed URL and a GUID both contain characters that + * would otherwise end the parameter early. */ +private fun encode(value: String): String = java.net.URLEncoder.encode(value, "UTF-8") diff --git a/app/src/main/java/org/terst/listen/ui/MainActivity.kt b/app/src/main/java/org/terst/listen/ui/MainActivity.kt index a4cc048..771e844 100644 --- a/app/src/main/java/org/terst/listen/ui/MainActivity.kt +++ b/app/src/main/java/org/terst/listen/ui/MainActivity.kt @@ -16,6 +16,8 @@ import androidx.compose.runtime.collectAsState import androidx.compose.runtime.getValue import androidx.compose.ui.Modifier import androidx.navigation.compose.NavHost +import androidx.navigation.NavGraphBuilder +import androidx.navigation.NavHostController import androidx.navigation.compose.composable import androidx.navigation.compose.rememberNavController @@ -108,10 +110,7 @@ private fun ListenApp( PlayerScreen(player = player, onBack = { nav.popBackStack() }) } episodeRoute(nav, viewModel, player) - composable("settings") { - SettingsScreen(opml = opmlViewModel, player = player, - onOpenRules = { nav.navigate("rules") }, onBack = { nav.popBackStack() }) - } + settingsRoute(nav, opmlViewModel, player) rulesRoutes(nav, rulesViewModel) composable("queue") { QueueScreen( @@ -216,3 +215,23 @@ private fun androidx.navigation.NavGraphBuilder.rulesRoutes( } } } + +/** + * The settings route, extracted so the app's own navigation graph stays readable as screens are + * added to it. + */ +private fun NavGraphBuilder.settingsRoute( + nav: NavHostController, + opml: OpmlViewModel, + player: PlayerViewModel, +) { + composable("settings") { + SettingsScreen( + opml = opml, + player = player, + server = rememberSkipServerSettings(), + onOpenRules = { nav.navigate("rules") }, + onBack = { nav.popBackStack() }, + ) + } +} diff --git a/app/src/main/java/org/terst/listen/ui/SettingsScreen.kt b/app/src/main/java/org/terst/listen/ui/SettingsScreen.kt index 1d8da1b..9afa3e9 100644 --- a/app/src/main/java/org/terst/listen/ui/SettingsScreen.kt +++ b/app/src/main/java/org/terst/listen/ui/SettingsScreen.kt @@ -15,11 +15,12 @@ import androidx.compose.material.icons.Icons import androidx.compose.material.icons.automirrored.filled.ArrowBack import androidx.compose.material3.Button import androidx.compose.material3.ExperimentalMaterial3Api +import androidx.compose.material3.HorizontalDivider import androidx.compose.material3.Icon import androidx.compose.material3.IconButton -import androidx.compose.material3.HorizontalDivider import androidx.compose.material3.MaterialTheme import androidx.compose.material3.OutlinedButton +import androidx.compose.material3.OutlinedTextField import androidx.compose.material3.Scaffold import androidx.compose.material3.Snackbar import androidx.compose.material3.SnackbarHost @@ -30,7 +31,9 @@ import androidx.compose.runtime.Composable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.collectAsState import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier import androidx.compose.ui.unit.dp @@ -47,6 +50,7 @@ private val OPML_MIME_TYPES = arrayOf("text/x-opml", "text/xml", "application/xm fun SettingsScreen( opml: OpmlViewModel, player: PlayerViewModel, + server: SkipServerSettings, onOpenRules: () -> Unit, onBack: () -> Unit, ) { @@ -103,10 +107,64 @@ fun SettingsScreen( onImport = { importLauncher.launch(OPML_MIME_TYPES) }, onExport = { exportLauncher.launch("listen-subscriptions.opml") }, ) + HorizontalDivider() + SkipServerSection(server) } } } +/** The skip server's address and token, as the settings screen reads and writes them. */ +data class SkipServerSettings( + val url: String, + val token: String, + val onSave: (url: String, token: String) -> Unit, +) + +/** + * Where the skip detection comes from. + * + * Detection used to run on the phone and could not work there: comparing several episodes' + * audio at once needs more memory than an Android process gets and more time than WorkManager + * allows, so it burned the battery for weeks and confirmed nothing. The server does that work + * and serves the audio it examined -- which is the part that matters, because a show using + * dynamic ad insertion returns a different file on every request and an offset measured on one + * copy is wrong for another. + * + * Left blank the app behaves as it did before: downloads from publishers, no detected skips. + */ +@Composable +private fun SkipServerSection(settings: SkipServerSettings) { + var url by remember(settings.url) { mutableStateOf(settings.url) } + var token by remember(settings.token) { mutableStateOf(settings.token) } + + Section("Skip server") { + OutlinedTextField( + value = url, + onValueChange = { url = it }, + label = { Text("Address") }, + placeholder = { Text("https://listen.terst.org") }, + singleLine = true, + modifier = Modifier.fillMaxWidth(), + ) + OutlinedTextField( + value = token, + onValueChange = { token = it }, + label = { Text("Token") }, + singleLine = true, + modifier = Modifier.fillMaxWidth(), + ) + Button(onClick = { settings.onSave(url, token) }) { Text("Save") } + Text( + if (settings.url.isBlank()) { + "Not set. Episodes download from publishers and play without detected skips." + } else { + "Episodes download from the server, which is also what analysed them." + }, + style = MaterialTheme.typography.bodySmall, + ) + } +} + @Composable private fun SpeedSection(current: Float, onSelect: (Float) -> Unit) { Section("Playback speed") { diff --git a/app/src/main/java/org/terst/listen/ui/SkipServerSettingsState.kt b/app/src/main/java/org/terst/listen/ui/SkipServerSettingsState.kt new file mode 100644 index 0000000..fc0e49d --- /dev/null +++ b/app/src/main/java/org/terst/listen/ui/SkipServerSettingsState.kt @@ -0,0 +1,35 @@ +package org.terst.listen.ui + +import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember +import androidx.compose.runtime.rememberCoroutineScope +import androidx.compose.ui.platform.LocalContext +import kotlinx.coroutines.launch +import org.terst.listen.data.Graph + +/** + * Reads and writes the skip server's address and token. + * + * Trivial glue, kept out of the composable that renders them so the screen stays a screen. In + * DataStore rather than Room, like the other settings that describe how the app is being used + * rather than what is in the library. + */ +@Composable +internal fun rememberSkipServerSettings(): SkipServerSettings { + val context = LocalContext.current + val preferences = remember(context) { Graph.get(context).preferences } + val scope = rememberCoroutineScope() + + val url by preferences.serverUrl.collectAsState(initial = "") + val token by preferences.serverToken.collectAsState(initial = "") + + return SkipServerSettings( + url = url, + token = token, + onSave = { newUrl, newToken -> + scope.launch { preferences.setServer(newUrl, newToken) } + }, + ) +} diff --git a/app/src/main/java/org/terst/listen/work/Workers.kt b/app/src/main/java/org/terst/listen/work/Workers.kt index 9c48db0..9809e9d 100644 --- a/app/src/main/java/org/terst/listen/work/Workers.kt +++ b/app/src/main/java/org/terst/listen/work/Workers.kt @@ -12,9 +12,6 @@ import androidx.work.WorkManager import androidx.work.WorkerParameters import org.terst.listen.data.Graph import org.terst.listen.data.RefreshOutcome -import org.terst.listen.fingerprint.FingerprintUpdater -import org.terst.listen.fingerprint.MediaCodecClipSource -import org.terst.listen.fingerprint.MidrollUpdater import java.util.concurrent.TimeUnit /** @@ -28,6 +25,9 @@ class RefreshWorker(context: Context, params: WorkerParameters) : CoroutineWorke override suspend fun doWork(): Result { val graph = Graph.get(applicationContext) + // The subscription list reaches the skip server from SubscriptionRepository.refreshAll, + // which every refresh route goes through -- including the library's refresh button, + // which does not come via this worker. val outcomes = graph.repository.refreshAll() val failed = outcomes.values.count { it is RefreshOutcome.Failed } // After the refresh, so a drip release can pick up anything that just arrived. @@ -77,62 +77,15 @@ class RetentionWorker(context: Context, params: WorkerParameters) : CoroutineWor } } -/** - * Checks one newly downloaded episode against its show's Tier 3 fingerprint state. - * - * The decode is [org.terst.listen.fingerprint.MediaCodecClipSource] (no JVM equivalent); the - * decision is [org.terst.listen.fingerprint.FingerprintUpdater], which is tested. - */ -class FingerprintWorker(context: Context, params: WorkerParameters) : CoroutineWorker(context, params) { - - override suspend fun doWork(): Result { - val episodeId = inputData.getString(KEY_EPISODE_ID) ?: return Result.failure() - val graph = Graph.get(applicationContext) - val episode = graph.episodes.getById(episodeId) ?: return Result.success() - - FingerprintUpdater(graph.fingerprints, graph.episodes, MediaCodecClipSource()).process(episode) - - return Result.success() - } - - companion object { - const val KEY_EPISODE_ID = "episodeId" - } -} - -/** - * Runs [MidrollUpdater] for every show, nightly rather than per-download like [FingerprintWorker]: - * a full-body scan is expensive enough (see the desktop prototype findings) that it is worth - * batching, and unlike a download it never touches the network, so there is nothing per-episode - * to react to promptly. - */ -class MidrollWorker(context: Context, params: WorkerParameters) : CoroutineWorker(context, params) { - - override suspend fun doWork(): Result { - val graph = Graph.get(applicationContext) - val updater = MidrollUpdater( - graph.subscriptions, - graph.episodes, - graph.midrolls, - MediaCodecClipSource(), - graph.midrollClips, - ) - for (show in graph.subscriptions.getAll()) updater.processShow(show.id) - return Result.success() - } -} - object Scheduling { private const val REFRESH_WORK = "listen-refresh" private const val DOWNLOAD_WORK = "listen-download" private const val DOWNLOAD_NOW_WORK = "listen-download-now" private const val RETENTION_WORK = "listen-retention" - private const val MIDROLL_WORK = "listen-midroll" private const val REFRESH_HOURS = 6L private const val DOWNLOAD_HOURS = 2L private const val RETENTION_HOURS = 12L - private const val MIDROLL_HOURS = 24L /** * Fetches what is in the queue now, rather than at the next periodic sweep. @@ -150,20 +103,6 @@ object Scheduling { } /** - * Checks one freshly downloaded episode against its show's fingerprint state. - * - * Not a unique work name: unlike a burst of queueing, a run of downloads is one episode - * each, and there is no "latest one wins" -- every episode needs its own check. - */ - fun fingerprintNow(context: Context, episodeId: String) { - WorkManager.getInstance(context).enqueue( - OneTimeWorkRequestBuilder<FingerprintWorker>() - .setInputData(androidx.work.Data.Builder().putString(FingerprintWorker.KEY_EPISODE_ID, episodeId).build()) - .build(), - ) - } - - /** * The periodic jobs are KEEP rather than UPDATE, so the schedule survives an app restart * instead of resetting its interval every launch -- which on a frequently-opened app means * the periodic work never actually fires. @@ -208,18 +147,5 @@ object Scheduling { .build(), ) - manager.enqueueUniquePeriodicWork( - MIDROLL_WORK, - ExistingPeriodicWorkPolicy.KEEP, - PeriodicWorkRequestBuilder<MidrollWorker>(MIDROLL_HOURS, TimeUnit.HOURS) - .setConstraints( - Constraints.Builder() - // Charging, not wifi: everything it touches is already on disk, so - // unlike DownloadWorker there is no data cost to gate on network for. - .setRequiresCharging(true) - .build() - ) - .build(), - ) } } diff --git a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt index ccede65..2474f8a 100644 --- a/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt +++ b/app/src/test/java/org/terst/listen/data/SubscriptionRepositoryTest.kt @@ -349,7 +349,7 @@ class SubscriptionRepositoryTest { subs, eps, FeedFetcher(OkHttpClient()), - probe = { ProbedVideo(title = "Sheep dog at work", durationMs = 92_000) }, + SubscriptionCollaborators(probe = { ProbedVideo(title = "Sheep dog at work", durationMs = 92_000) }), ) { clock } probing.addSingleEpisode(video("111", null, kind = SubscriptionKind.FACEBOOK_VIDEO)) @@ -366,7 +366,7 @@ class SubscriptionRepositoryTest { subs, eps, FeedFetcher(OkHttpClient()), - probe = { ProbedVideo(title = "Something else", durationMs = 1) }, + SubscriptionCollaborators(probe = { ProbedVideo(title = "Something else", durationMs = 1) }), ) { clock } probing.addSingleEpisode(video("aaa", "From the page")) @@ -381,7 +381,7 @@ class SubscriptionRepositoryTest { subs, eps, FeedFetcher(OkHttpClient()), - probe = { null }, + SubscriptionCollaborators(probe = { null }), ) { clock } probing.addSingleEpisode(video("111", null, kind = SubscriptionKind.FACEBOOK_VIDEO)) @@ -403,7 +403,7 @@ class SubscriptionRepositoryTest { subs, eps, FeedFetcher(OkHttpClient()), - onDiscovered = { discovered += it }, + SubscriptionCollaborators(onDiscovered = { discovered += it }), ) { clock } @Test diff --git a/app/src/test/java/org/terst/listen/download/EpisodeDownloaderTest.kt b/app/src/test/java/org/terst/listen/download/EpisodeDownloaderTest.kt index d5e971b..b35983a 100644 --- a/app/src/test/java/org/terst/listen/download/EpisodeDownloaderTest.kt +++ b/app/src/test/java/org/terst/listen/download/EpisodeDownloaderTest.kt @@ -17,6 +17,7 @@ import org.terst.listen.data.DownloadState import org.terst.listen.data.EpisodeEntity import org.terst.listen.data.FakeEpisodeDao import org.terst.listen.data.SubscriptionKind +import org.terst.listen.server.ServerEpisode import java.io.File class EpisodeDownloaderTest { @@ -161,4 +162,126 @@ class EpisodeDownloaderTest { downloader.download(ep, SubscriptionKind.YOUTUBE_CHANNEL) assertEquals("no plain HTTP request should have been made", 0, server.requestCount) } + + // ---- the skip server as the preferred source ---- + // + // The server has to serve the audio, not just the offsets. A show using dynamic ad + // insertion returns a different file on every request -- one real episode measured 22:05 on + // one fetch and 28:18 on another -- so an offset computed on the server's copy is simply + // wrong for a copy the phone fetched itself. Downloading from the server is what makes its + // numbers apply to the bytes being played. + + private class FakeServerAudio(private val info: ServerEpisode?) : ServerAudio { + var recorded: Pair<String, ServerEpisode>? = null + override suspend fun resolve(episode: EpisodeEntity) = info + override suspend fun record(episode: EpisodeEntity, info: ServerEpisode) { + recorded = episode.id to info + } + } + + private fun withServer(audio: ServerAudio?) = + EpisodeDownloader(OkHttpClient(), dao, library, null, audio) { 1_000L } + + @Test + fun `downloads from the skip server when it has the episode`() = runTest { + server.enqueue(MockResponse().setBody("server-bytes")) + val audio = FakeServerAudio( + ServerEpisode( + ready = true, + audioUrl = server.url("/audio/abc?token=t").toString(), + durationMs = 1_698_717, + introMs = 142_000, + outroMs = 11_000, + detected = true, + ) + ) + val ep = episode(url = "https://publisher.example/original.mp3").also { dao.rows += it } + + val result = withServer(audio).download(ep, SubscriptionKind.PODCAST) + + assertTrue(result.isSuccess) + assertEquals("server-bytes", result.getOrThrow().readText()) + assertEquals("/audio/abc?token=t", server.takeRequest().path) + } + + @Test + fun `records what the server knew once the file is on disk`() = runTest { + server.enqueue(MockResponse().setBody("server-bytes")) + val info = ServerEpisode( + ready = true, + audioUrl = server.url("/audio/abc").toString(), + durationMs = 1_698_717, + introMs = 142_000, + outroMs = 11_000, + detected = true, + ) + val audio = FakeServerAudio(info) + val ep = episode(url = "https://publisher.example/original.mp3").also { dao.rows += it } + + withServer(audio).download(ep, SubscriptionKind.PODCAST) + + assertEquals(ep.id to info, audio.recorded) + } + + @Test + fun `downloads from the publisher when the server has nothing yet`() = runTest { + server.enqueue(MockResponse().setBody("publisher-bytes")) + val ep = episode(url = server.url("/original.mp3").toString()).also { dao.rows += it } + + val result = withServer(FakeServerAudio(null)).download(ep, SubscriptionKind.PODCAST) + + assertTrue(result.isSuccess) + assertEquals("/original.mp3", server.takeRequest().path) + } + + @Test + fun `records nothing when the server had nothing`() = runTest { + server.enqueue(MockResponse().setBody("publisher-bytes")) + val audio = FakeServerAudio(null) + val ep = episode(url = server.url("/original.mp3").toString()).also { dao.rows += it } + + withServer(audio).download(ep, SubscriptionKind.PODCAST) + + assertNull(audio.recorded) + } + + /** + * The server going down must not cost Peter the episode. Skips are an improvement on + * playback, not a precondition for it -- and a dependency that turns a reachable publisher + * into a failed download would make the app worse than before the server existed. + */ + @Test + fun `falls back to the publisher when the server's own audio fails`() = runTest { + server.enqueue(MockResponse().setResponseCode(502)) + server.enqueue(MockResponse().setBody("publisher-bytes")) + val audio = FakeServerAudio( + ServerEpisode(ready = true, audioUrl = server.url("/audio/broken").toString(), + durationMs = null, introMs = 1, outroMs = 2, detected = true) + ) + val ep = episode(url = server.url("/original.mp3").toString()).also { dao.rows += it } + + val result = withServer(audio).download(ep, SubscriptionKind.PODCAST) + + assertTrue("expected the publisher fallback to succeed", result.isSuccess) + assertEquals("publisher-bytes", result.getOrThrow().readText()) + assertEquals("/audio/broken", server.takeRequest().path) + assertEquals("/original.mp3", server.takeRequest().path) + } + + /** Offsets describe the server's copy, so they must not be recorded against a file that + * came from the publisher instead. */ + @Test + fun `records nothing when the audio came from the publisher after all`() = runTest { + server.enqueue(MockResponse().setResponseCode(502)) + server.enqueue(MockResponse().setBody("publisher-bytes")) + val audio = FakeServerAudio( + ServerEpisode(ready = true, audioUrl = server.url("/audio/broken").toString(), + durationMs = null, introMs = 1, outroMs = 2, detected = true) + ) + val ep = episode(url = server.url("/original.mp3").toString()).also { dao.rows += it } + + withServer(audio).download(ep, SubscriptionKind.PODCAST) + + assertNull("recorded offsets for audio it did not download", audio.recorded) + } } diff --git a/app/src/test/java/org/terst/listen/server/OkHttpSkipTransportTest.kt b/app/src/test/java/org/terst/listen/server/OkHttpSkipTransportTest.kt new file mode 100644 index 0000000..335b68d --- /dev/null +++ b/app/src/test/java/org/terst/listen/server/OkHttpSkipTransportTest.kt @@ -0,0 +1,113 @@ +package org.terst.listen.server + +import kotlinx.coroutines.test.runTest +import okhttp3.OkHttpClient +import okhttp3.mockwebserver.MockResponse +import okhttp3.mockwebserver.MockWebServer +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Test + +/** + * The transport's whole contract is that nothing it does can break the app. + * + * The skip server is an improvement on playback, not a precondition for it: unreachable, slow, + * misconfigured and returning nonsense all have to mean the same thing to the caller as "not + * configured" -- carry on downloading from publishers. An exception escaping from here would + * turn a reachable episode into a failed download, which is worse than never having had a + * server at all. + */ +class OkHttpSkipTransportTest { + + private lateinit var server: MockWebServer + private lateinit var transport: OkHttpSkipTransport + + @Before + fun setUp() { + server = MockWebServer() + server.start() + transport = OkHttpSkipTransport(OkHttpClient()) + } + + @After + fun tearDown() = server.shutdown() + + @Test + fun `returns the body of a successful get`() = runTest { + server.enqueue(MockResponse().setBody("""{"status":"ready"}""")) + + val got = transport.get(server.url("/api/episode").toString(), "tok") + + assertEquals("""{"status":"ready"}""", got) + } + + @Test + fun `sends the token as a bearer header`() = runTest { + server.enqueue(MockResponse().setBody("{}")) + + transport.get(server.url("/api/episode").toString(), "hunter2") + + assertEquals("Bearer hunter2", server.takeRequest().getHeader("Authorization")) + } + + @Test + fun `an unauthorised response is nothing rather than an error`() = runTest { + server.enqueue(MockResponse().setResponseCode(401).setBody("""{"error":"unauthorized"}""")) + + assertNull(transport.get(server.url("/api/episode").toString(), "wrong")) + } + + @Test + fun `a server error is nothing rather than an error`() = runTest { + server.enqueue(MockResponse().setResponseCode(502)) + + assertNull(transport.get(server.url("/api/episode").toString(), "tok")) + } + + /** A misconfigured address must not throw out of the download path. */ + @Test + fun `an unreachable host is nothing rather than an error`() = runTest { + assertNull(transport.get("http://127.0.0.1:1/api/episode", "tok")) + } + + @Test + fun `a malformed url is nothing rather than an error`() = runTest { + assertNull(transport.get("not a url at all", "tok")) + } + + @Test + fun `posts the body as json`() = runTest { + server.enqueue(MockResponse().setResponseCode(200)) + + val ok = transport.post( + server.url("/api/subscriptions").toString(), + "tok", + """{"feeds":["https://a"]}""", + ) + + assertTrue(ok) + val request = server.takeRequest() + assertEquals("POST", request.method) + assertEquals("""{"feeds":["https://a"]}""", request.body.readUtf8()) + assertTrue( + "content type was ${request.getHeader("Content-Type")}", + request.getHeader("Content-Type")?.startsWith("application/json") == true, + ) + } + + @Test + fun `a rejected post reports false`() = runTest { + server.enqueue(MockResponse().setResponseCode(400)) + + assertFalse(transport.post(server.url("/api/subscriptions").toString(), "tok", "{}")) + } + + @Test + fun `a post to an unreachable host reports false`() = runTest { + assertFalse(transport.post("http://127.0.0.1:1/api/subscriptions", "tok", "{}")) + } +} diff --git a/app/src/test/java/org/terst/listen/server/ServerSkipsTest.kt b/app/src/test/java/org/terst/listen/server/ServerSkipsTest.kt new file mode 100644 index 0000000..57c03f3 --- /dev/null +++ b/app/src/test/java/org/terst/listen/server/ServerSkipsTest.kt @@ -0,0 +1,155 @@ +package org.terst.listen.server + +import kotlinx.coroutines.test.runTest +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test +import org.terst.listen.data.EpisodeEntity +import org.terst.listen.data.FakeEpisodeDao +import org.terst.listen.data.FakePlaybackStateDao +import org.terst.listen.data.FakeShowFingerprintDao +import org.terst.listen.data.FakeSubscriptionDao +import org.terst.listen.data.SubscriptionEntity + +/** + * What the phone does with the server's answer. + * + * The offsets land in the same columns the on-device detector used to write, which is why + * playback needed no changes at all: `effectiveIntroMs`, `MidrollSkip` and `PlayheadPolicy` + * already read exactly these. Only where the numbers come from has changed. + */ +class ServerSkipsTest { + + private class StubTransport(private val body: String?) : SkipTransport { + val gets = mutableListOf<String>() + override suspend fun get(url: String, token: String): String? { + gets += url + return body + } + + override suspend fun post(url: String, token: String, body: String) = true + } + + private val show = SubscriptionEntity( + id = "s1", + feedUrl = "https://feeds.simplecast.com/EmVW7VGp", + title = "Radiolab", + addedAtMs = 0, + ) + + private val episode = EpisodeEntity( + id = "e1", + subscriptionId = "s1", + guid = "guid-1", + title = "Grass Apes", + mediaUrl = "https://cdn.example/grass.mp3", + mediaType = "audio/mpeg", + discoveredAtMs = 0, + ) + + private fun subject( + transport: SkipTransport, + subscriptions: FakeSubscriptionDao = FakeSubscriptionDao().apply { rows += show }, + episodes: FakeEpisodeDao = FakeEpisodeDao().apply { rows += episode }, + fingerprints: FakeShowFingerprintDao = FakeShowFingerprintDao(), + ): ServerSkips { + // Both fakes write through to the same episode rows, the way the real foreign table + // does, so a test sees the value where production would put it. + fingerprints.episodeRows = episodes.rows + return ServerSkips( + SkipServer({ SkipServerConfig("https://listen.terst.org", "tok") }, transport), + subscriptions, + fingerprints, + FakePlaybackStateDao(episodes.rows), + ) + } + + /** The server keys episodes by feed, so the show's feed URL has to be looked up and sent. */ + @Test + fun `asks the server using the show's feed url`() = runTest { + val transport = StubTransport("""{"status":"pending"}""") + + subject(transport).resolve(episode) + + val url = transport.gets.single() + assertTrue("feed url not sent in $url", url.contains("EmVW7VGp")) + assertTrue("guid not sent in $url", url.contains("guid-1")) + } + + /** An episode whose show has been unsubscribed has no feed to ask about. */ + @Test + fun `does not ask when the show is gone`() = runTest { + val transport = StubTransport("""{"status":"pending"}""") + + val got = subject(transport, subscriptions = FakeSubscriptionDao()).resolve(episode) + + assertNull(got) + assertTrue("asked about a show that is not subscribed", transport.gets.isEmpty()) + } + + @Test + fun `records the detected offsets against the episode`() = runTest { + val episodes = FakeEpisodeDao().apply { rows += episode } + + subject(StubTransport(null), episodes = episodes).record( + episode, + ServerEpisode(ready = true, audioUrl = "x", durationMs = 1_698_717, + introMs = 142_000, outroMs = 11_000, detected = true), + ) + + val stored = episodes.rows.single { it.id == "e1" } + assertEquals(142_000L, stored.detectedIntroMs) + assertEquals(11_000L, stored.detectedOutroMs) + } + + /** + * The measured length too: the feed's claim is minutes short on any show using dynamic ad + * insertion, and this length is the one the file actually has. + */ + @Test + fun `records the measured duration`() = runTest { + val episodes = FakeEpisodeDao().apply { rows += episode } + + subject(StubTransport(null), episodes = episodes).record( + episode, + ServerEpisode(ready = true, audioUrl = "x", durationMs = 1_698_717, + introMs = null, outroMs = null, detected = true), + ) + + assertEquals(1_698_717L, episodes.rows.single { it.id == "e1" }.actualDurationMs) + } + + /** + * An absent offset must not be written as zero. Zero is a real detected value meaning "this + * show has no intro", and playback distinguishes the two. + */ + @Test + fun `leaves an offset the server did not report alone`() = runTest { + val episodes = FakeEpisodeDao().apply { rows += episode } + + subject(StubTransport(null), episodes = episodes).record( + episode, + ServerEpisode(ready = true, audioUrl = "x", durationMs = null, + introMs = 142_000, outroMs = null, detected = true), + ) + + val stored = episodes.rows.single { it.id == "e1" } + assertEquals(142_000L, stored.detectedIntroMs) + assertNull("wrote an outro the server never reported", stored.detectedOutroMs) + } + + /** A detected zero is a finding, not an absence, and has to be stored as one. */ + @Test + fun `records a detected zero offset`() = runTest { + val episodes = FakeEpisodeDao().apply { rows += episode } + + subject(StubTransport(null), episodes = episodes).record( + episode, + ServerEpisode(ready = true, audioUrl = "x", durationMs = null, + introMs = 0, outroMs = null, detected = true), + ) + + assertEquals(0L, episodes.rows.single { it.id == "e1" }.detectedIntroMs) + } +} diff --git a/app/src/test/java/org/terst/listen/server/SkipServerTest.kt b/app/src/test/java/org/terst/listen/server/SkipServerTest.kt new file mode 100644 index 0000000..bdf9d6a --- /dev/null +++ b/app/src/test/java/org/terst/listen/server/SkipServerTest.kt @@ -0,0 +1,194 @@ +package org.terst.listen.server + +import kotlinx.coroutines.test.runTest +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The phone's half of the skip server. + * + * Detection does not happen here any more, and for a reason worth keeping written down: the + * intro search needs several episodes' audio compared at once, which needs more memory than an + * Android process gets and more time than WorkManager allows. The phone's job is now to say + * what it subscribes to, ask what to skip, and play the audio the answer belongs to. + * + * That last part is not optional. A show using dynamic ad insertion serves a different file on + * every request, so an offset measured on the server's copy is wrong for a copy the phone + * fetched itself -- Grass Apes came back at 22:05 on one fetch and 28:18 on another. Playing + * the server's bytes is what makes the numbers apply. + */ +class SkipServerTest { + + private class FakeTransport : SkipTransport { + var response: String? = null + var failPost = false + val gets = mutableListOf<String>() + val posts = mutableListOf<Pair<String, String>>() + var seenToken: String? = null + + override suspend fun get(url: String, token: String): String? { + gets += url + seenToken = token + return response + } + + override suspend fun post(url: String, token: String, body: String): Boolean { + posts += url to body + seenToken = token + return !failPost + } + } + + private fun server( + transport: SkipTransport, + baseUrl: String? = "https://listen.terst.org", + token: String? = "tok", + ) = SkipServer({ if (baseUrl == null || token == null) null else SkipServerConfig(baseUrl, token) }, transport) + + private val ready = """ + {"status":"ready","audioUrl":"/audio/abc123?token=tok","durationMs":1698717, + "introMs":142000,"outroMs":11000,"detected":true} + """.trimIndent() + + @Test + fun `reads what the server says about an episode`() = runTest { + val transport = FakeTransport().apply { response = ready } + + val got = server(transport).episode("https://f", "g1", "https://cdn/1.mp3", "One") + + assertTrue("expected a ready episode, got $got", got?.ready == true) + assertEquals("https://listen.terst.org/audio/abc123?token=tok", got?.audioUrl) + assertEquals(1698717L, got?.durationMs) + assertEquals(142000L, got?.introMs) + assertEquals(11000L, got?.outroMs) + assertTrue(got?.detected == true) + } + + /** + * The server returns a root-relative audio path. Handing that to a player unresolved would + * have it try to open a local file. + */ + @Test + fun `resolves a relative audio url against the server`() = runTest { + val transport = FakeTransport().apply { + response = """{"status":"ready","audioUrl":"/audio/abc?token=tok"}""" + } + + val got = server(transport, baseUrl = "https://listen.terst.org/").episode("https://f", "g1", "", "") + + assertEquals("https://listen.terst.org/audio/abc?token=tok", got?.audioUrl) + } + + @Test + fun `an absolute audio url is left alone`() = runTest { + val transport = FakeTransport().apply { + response = """{"status":"ready","audioUrl":"https://cdn.example/abc.mp3"}""" + } + + val got = server(transport).episode("https://f", "g1", "", "") + + assertEquals("https://cdn.example/abc.mp3", got?.audioUrl) + } + + /** Pending is a real answer: the server is fetching it, so the phone should not treat it + * as a failure, but it has nothing to play yet either. */ + @Test + fun `a pending episode is reported as not ready`() = runTest { + val transport = FakeTransport().apply { response = """{"status":"pending","detected":false}""" } + + val got = server(transport).episode("https://f", "g1", "", "") + + assertEquals(false, got?.ready) + assertNull(got?.audioUrl) + } + + /** Missing offsets must come back as absent, not as zero: zero is a real detected value + * meaning "this show has no intro", and playback treats the two differently. */ + @Test + fun `absent offsets are null rather than zero`() = runTest { + val transport = FakeTransport().apply { response = """{"status":"ready","audioUrl":"/audio/a"}""" } + + val got = server(transport).episode("https://f", "g1", "", "") + + assertNull(got?.introMs) + assertNull(got?.outroMs) + } + + /** A feed URL and a GUID both contain characters that would otherwise break the query. */ + @Test + fun `encodes the feed url and guid into the query`() = runTest { + val transport = FakeTransport() + + server(transport).episode( + feedUrl = "https://feeds.simplecast.com/EmVW7VGp?x=1&y=2", + guid = "tag:example,2026:1 odd", + enclosureUrl = "https://cdn/a b.mp3", + title = "Grass Apes & Co", + ) + + val url = transport.gets.single() + assertTrue("feed not encoded in $url", url.contains("feed=https%3A%2F%2Ffeeds.simplecast.com%2FEmVW7VGp%3Fx%3D1%26y%3D2")) + assertTrue("guid not encoded in $url", url.contains("guid=tag%3Aexample%2C2026%3A1+odd")) + assertTrue("ampersand in the title not encoded in $url", url.contains("%26+Co")) + } + + @Test + fun `sends the token`() = runTest { + val transport = FakeTransport().apply { response = ready } + server(transport, token = "hunter2").episode("https://f", "g1", "", "") + assertEquals("hunter2", transport.seenToken) + } + + /** With nothing configured the server is simply not used, and nothing throws. */ + @Test + fun `an unconfigured server is a no-op rather than an error`() = runTest { + val transport = FakeTransport() + + assertNull(server(transport, baseUrl = null).episode("https://f", "g1", "", "")) + assertEquals(false, server(transport, token = null).pushSubscriptions(listOf("https://f"))) + assertTrue("talked to a server that is not configured", transport.gets.isEmpty() && transport.posts.isEmpty()) + } + + @Test + fun `rubbish from the server is reported as nothing rather than crashing`() = runTest { + val transport = FakeTransport().apply { response = "<html>502 Bad Gateway</html>" } + + assertNull(server(transport).episode("https://f", "g1", "", "")) + } + + @Test + fun `no response at all is reported as nothing`() = runTest { + assertNull(server(FakeTransport()).episode("https://f", "g1", "", "")) + } + + @Test + fun `pushes the subscription list as json`() = runTest { + val transport = FakeTransport() + + val ok = server(transport).pushSubscriptions(listOf("https://a", "https://b")) + + assertTrue(ok) + val (url, body) = transport.posts.single() + assertEquals("https://listen.terst.org/api/subscriptions", url) + assertEquals("""{"feeds":["https://a","https://b"]}""", body) + } + + @Test + fun `a failed push is reported rather than thrown`() = runTest { + val transport = FakeTransport().apply { failPost = true } + + assertEquals(false, server(transport).pushSubscriptions(listOf("https://a"))) + } + + /** A trailing slash on the configured URL must not produce a double slash in the path. */ + @Test + fun `tolerates a trailing slash on the base url`() = runTest { + val transport = FakeTransport() + + server(transport, baseUrl = "https://listen.terst.org/").pushSubscriptions(listOf("https://a")) + + assertEquals("https://listen.terst.org/api/subscriptions", transport.posts.single().first) + } +} |
