summaryrefslogtreecommitdiff
path: root/bin/gradle-test
diff options
context:
space:
mode:
authorPeter Stone <thepeterstone@gmail.com>2026-09-18 21:04:06 +0000
committerPeter Stone <thepeterstone@gmail.com>2026-09-18 21:04:06 +0000
commitb51ba30f1c77bcadfbf96b2eb7b4917c20faa7b0 (patch)
tree5d4d3f5c258de177d12882ffa17803e8df323ca8 /bin/gradle-test
Add devtools: build/test wrappers that can't be misread
Three times in one session I reported a build or test result that was about something other than what I thought I was testing: - A failed `cd` meant Gradle ran in the previous directory; the "BUILD SUCCESSFUL" I reported was for the wrong tree. - `adb shell monkey` + a single `pidof` sample reported a healthy app as crashed and a crashed app as healthy, on the same emulator minutes apart. - `adb install` refused a filename and printed so to stdout, which was redirected to /dev/null; the old build stayed installed and got tested. All three share a cause: concluding from console text rather than from exit codes and structured artifacts. These wrappers make that mistake hard. gradle-verify pins the project with -p from an absolutely-resolved argument, treats a missing dir as fatal rather than falling back to cwd, and reports from the exit code. gradle-test parses JUnit XML for counts, and fails on zero tests executed, on no XML at all, and on results whose mtime predates the run (Gradle reporting UP-TO-DATE and skipping the task). android-launch-check resolves the launcher activity from the package manager rather than guessing, starts logcat capture before launching, uses `am start -W`, samples the pid across a settle window so a process that starts and then dies is a failure, and prints the fatal stack. Optionally installs an APK first and verifies the install actually reported Success. Two bugs of exactly this class were found and fixed in the scripts themselves while testing them, both of which broke the crash-detection path specifically: `... | grep -q` under pipefail SIGPIPEs its upstream and reports failure on a successful match, and `pid=$(pidof ...)` under `set -e` aborts the script at the moment the process is gone. Verified against the real projects: nav/android-app 241 tests, nav/test-runner 122, feedreader 39, and both the pass and crash paths of android-launch-check on a live emulator. gradle-test immediately caught nav reporting UP-TO-DATE with 33 stale result files, which naively read as "241 tests pass" without running one. 12 self-tests, mostly negative cases.
Diffstat (limited to 'bin/gradle-test')
-rwxr-xr-xbin/gradle-test140
1 files changed, 140 insertions, 0 deletions
diff --git a/bin/gradle-test b/bin/gradle-test
new file mode 100755
index 0000000..5b059d7
--- /dev/null
+++ b/bin/gradle-test
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+#
+# Run a Gradle test task and report counts parsed from the JUnit XML it wrote,
+# not from console text.
+#
+# Catches three failure modes that all look like success from the console:
+#
+# 1. Zero tests ran. A unit-test source set that fails to compile, or a filter
+# that matches nothing, can leave the build green with nothing executed.
+# "BUILD SUCCESSFUL" and "0 tests" are the same line of output.
+# 2. Stale results. If Gradle decides the test task is UP-TO-DATE it does not
+# re-run anything, and the XML on disk describes an older version of the
+# code. Reading those numbers as current is how a passing suite gets
+# reported for code that was never tested.
+# 3. Results from the wrong project, via cwd drift — handled by gradle-verify.
+#
+# Usage:
+# gradle-test <project-dir> [test-task] [--allow-stale] [-- <gradle args>]
+#
+# Default task: testDebugUnitTest for Android projects, test otherwise.
+
+set -euo pipefail
+
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+die() { printf '%s: %s\n' "$(basename "$0")" "$*" >&2; exit 2; }
+
+[ $# -ge 1 ] || die "usage: $(basename "$0") <project-dir> [test-task] [--allow-stale] [-- <gradle args>]"
+
+raw_dir=$1; shift
+project_dir=$(cd "$raw_dir" 2>/dev/null && pwd) || die "no such project dir: $raw_dir"
+
+task=""
+allow_stale=0
+extra=()
+seen_sep=0
+for arg in "$@"; do
+ if [ "$arg" = "--" ]; then seen_sep=1; continue; fi
+ if [ $seen_sep -eq 1 ]; then extra+=("$arg"); continue; fi
+ case "$arg" in
+ --allow-stale) allow_stale=1 ;;
+ -*) extra+=("$arg") ;;
+ *) task=$arg ;;
+ esac
+done
+
+if [ -z "$task" ]; then
+ if grep -rqs "com.android.application\|com.android.library" \
+ "$project_dir"/build.gradle "$project_dir"/build.gradle.kts \
+ "$project_dir"/*/build.gradle "$project_dir"/*/build.gradle.kts 2>/dev/null; then
+ task=testDebugUnitTest
+ else
+ task=test
+ fi
+fi
+
+# Marker file: anything written at or after this is from the run we just did.
+# Using a file rather than a clock reading avoids any filesystem/shell timestamp
+# granularity mismatch.
+marker=$(mktemp)
+trap 'rm -f "$marker"' EXIT
+sleep 1 # ensure a strictly-earlier mtime than anything the build writes
+
+set +e
+"$here/gradle-verify" "$project_dir" "$task" ${extra[@]+-- "${extra[@]}"}
+gradle_status=$?
+set -e
+
+printf '\n=== gradle-test results ===\n'
+
+mapfile -t all_xml < <(find "$project_dir" -path '*/test-results/*' -name 'TEST-*.xml' 2>/dev/null | sort)
+if [ ${#all_xml[@]} -eq 0 ]; then
+ printf 'result : FAIL — no JUnit XML found anywhere under %s\n' "$project_dir"
+ printf 'hint : the test task may not have run, or writes results elsewhere\n'
+ exit 1
+fi
+
+fresh=(); stale=()
+for f in "${all_xml[@]}"; do
+ if [ "$f" -nt "$marker" ]; then fresh+=("$f"); else stale+=("$f"); fi
+done
+
+printf 'task : %s\n' "$task"
+printf 'xml : %d file(s) total, %d fresh, %d stale\n' \
+ "${#all_xml[@]}" "${#fresh[@]}" "${#stale[@]}"
+
+if [ ${#fresh[@]} -eq 0 ]; then
+ if [ $allow_stale -eq 1 ]; then
+ printf 'note : no fresh results; counting stale ones because --allow-stale\n'
+ fresh=("${all_xml[@]}")
+ else
+ printf 'result : FAIL — every result file predates this run\n'
+ printf 'hint : Gradle treated %s as UP-TO-DATE, so these numbers describe\n' "$task"
+ printf ' older code. Re-run with: -- --rerun-tasks\n'
+ exit 1
+ fi
+fi
+
+counts=$(python3 - "${fresh[@]}" <<'PY'
+import sys, xml.etree.ElementTree as ET
+tests = failures = errors = skipped = 0
+bad = []
+for p in sys.argv[1:]:
+ try:
+ r = ET.parse(p).getroot()
+ except Exception as e:
+ bad.append(f"{p}: unparseable ({e})")
+ continue
+ tests += int(r.get('tests', 0))
+ failures += int(r.get('failures', 0))
+ errors += int(r.get('errors', 0))
+ skipped += int(r.get('skipped', 0))
+print(f"{tests} {failures} {errors} {skipped}")
+for b in bad:
+ print("UNPARSEABLE " + b, file=sys.stderr)
+PY
+)
+read -r tests failures errors skipped <<<"$counts"
+
+printf 'tests : %s\n' "$tests"
+printf 'failures: %s\n' "$failures"
+printf 'errors : %s\n' "$errors"
+printf 'skipped : %s\n' "$skipped"
+
+rc=0
+[ "$gradle_status" -ne 0 ] && rc=1
+[ "$failures" -gt 0 ] && rc=1
+[ "$errors" -gt 0 ] && rc=1
+if [ "$tests" -eq 0 ]; then
+ printf 'result : FAIL — zero tests executed\n'
+ printf 'hint : a green build with no tests is not a passing suite\n'
+ exit 1
+fi
+
+if [ $rc -eq 0 ]; then
+ printf 'result : PASS (%s tests, gradle exit %d)\n' "$tests" "$gradle_status"
+else
+ printf 'result : FAIL (gradle exit %d, %s failures, %s errors)\n' \
+ "$gradle_status" "$failures" "$errors"
+fi
+exit $rc