--- name: feedback-verify-before-asserting-root-cause description: "Don't assert a root cause for a production bug report without checking server-side evidence first; a plausible-sounding theory is not a diagnosis" metadata: node_type: memory type: feedback originSessionId: 2607e018-d4ad-4ec6-bb68-5f1d0f9a3906 --- **Never assert a root cause for a live/production bug report without first checking the actual evidence available on the affected system.** For a server-backed bug, that means: server logs, a live reachability test (curl the endpoint directly), and tracing the actual code path the report implicates -- BEFORE proposing any explanation, and especially before shipping a "fix." **Why:** On 2026-08-04, reported symptom was "doot widget stuck showing last Saturday, add silently does nothing." First pass: found one real-but-secondary bug (missing `.onFailure` handler in the Android quick-add flow -- a legitimate fix, but not the cause), then asserted the "today frozen" symptom was Android battery optimization / Doze killing the background sync job -- a theory that was never checked against anything on the actual server, just pattern-matched from "widgets sometimes have this problem." The user correctly called this "voodoo bullshit": a device-side theory asserted with confidence, resting on zero server-side verification, when the server was one `curl` away from being checked. It turned out to be a genuinely dead-simple, checkable fact: the production server itself was completely wedged (Apache proxy_http timeouts against the Go app, confirmed live), and had been since the exact minute the symptom started. The user's own words -- "I've used lots of widgets and not had this problem" -- was a legitimate signal the theory didn't fit and should have prompted more digging, not a shipped build. **How to apply:** - For any "X is broken / stopped working" report involving a client talking to a server I control: check the server first. `curl` the live endpoint (with and without auth if relevant), read the reverse-proxy error log (not just the app's own request log -- the app's log only shows requests that *completed*; a hung request never appears there, which is itself a clue easy to miss), and check `systemctl status` / process health before reaching for client-side or environmental explanations. - Treat "the client stopped making requests entirely" and "the client's requests are failing/hanging" as distinct hypotheses to check separately -- they have very different causes (client gave up vs. server not answering) and the log evidence to distinguish them (are there recent proxy-timeout errors, or truly zero recent traffic of any kind) is usually sitting right there. - If a "fix" is shipped before root cause is nailed down, say so explicitly ("this addresses a real bug I found but I haven't confirmed it's *the* reported bug") rather than presenting it as *the* fix. Don't let finding *a* bug substitute for finding *the* bug. - When the user pushes back with skepticism grounded in their own experience ("I've used lots of X and never seen this"), that's a strong signal to re-open the investigation with fresh server-side evidence, not to defend the original theory. - Related: [[project-doot-prod]] for this project's actual server topology (corrected 2026-08-04 after this incident: production reverse proxy is Apache, not Caddy -- see that memory for the correction).