summaryrefslogtreecommitdiff
path: root/docs/adr/004-concurrent-data-fetching.md
diff options
context:
space:
mode:
authorPeter Stone <thepeterstone@gmail.com>2026-02-05 10:37:09 -1000
committerPeter Stone <thepeterstone@gmail.com>2026-02-05 10:37:09 -1000
commit223c94f52ebaa878f6951ebf7d08754e413bdca7 (patch)
treefc02a201fb83ec49fd5978d2ec9bf79f83e5d57c /docs/adr/004-concurrent-data-fetching.md
parent4ac78382343e14c00ba21ee11ee4642255509eef (diff)
Document multi-agent workflow, ADRs, and Agent API
- Add Development Workflow section to DESIGN.md documenting three-role system (Architect, Implementor, Reviewer) with handoff documents - Update CLAUDE.md with Key Documents section pointing to DESIGN.md, role definitions, and ADRs - Add ADR-first documentation policy across all role definitions - Update REVIEWER_ROLE.md with comprehensive test quality checklist - Document Agent API and completed tasks log in DESIGN.md - Update database schema table with 5 missing tables - Update endpoint reference with 10 missing routes - Create ADRs 002-005 capturing key architectural decisions: - 002: Timeline aggregation architecture - 003: HTMX write operations pattern - 004: Concurrent data fetching with graceful degradation - 005: Agent API notification-based authentication - Add migrations/README.md documenting schema history and 007 gap Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Diffstat (limited to 'docs/adr/004-concurrent-data-fetching.md')
-rw-r--r--docs/adr/004-concurrent-data-fetching.md104
1 files changed, 104 insertions, 0 deletions
diff --git a/docs/adr/004-concurrent-data-fetching.md b/docs/adr/004-concurrent-data-fetching.md
new file mode 100644
index 0000000..c7238f8
--- /dev/null
+++ b/docs/adr/004-concurrent-data-fetching.md
@@ -0,0 +1,104 @@
+# ADR 004: Concurrent Data Fetching with Graceful Degradation
+
+## Status
+Accepted
+
+## Context
+The dashboard fetches data from 5+ external APIs on each page load:
+- Todoist (tasks)
+- Trello (boards, cards)
+- PlanToEat (meals, shopping)
+- Google Calendar (events)
+- Google Tasks (tasks)
+
+Sequential fetching would be unacceptably slow (5+ seconds). However, concurrent fetching introduces:
+- Rate limiting concerns (especially Trello)
+- Partial failure scenarios
+- Data race risks
+
+## Decision
+Implement **parallel fetching with semaphore-limited concurrency** and **cache-first fallback** for graceful degradation.
+
+### Technical Details:
+
+**Parallel Aggregation (`internal/handlers/handlers.go`):**
+```go
+func (h *Handler) aggregateData(ctx context.Context) (*DashboardData, error) {
+ var wg sync.WaitGroup
+ var mu sync.Mutex
+ data := &DashboardData{}
+
+ // Launch goroutines for each data source
+ wg.Add(4)
+ go func() { defer wg.Done(); fetchTasks(ctx, &mu, data) }()
+ go func() { defer wg.Done(); fetchBoards(ctx, &mu, data) }()
+ go func() { defer wg.Done(); fetchMeals(ctx, &mu, data) }()
+ go func() { defer wg.Done(); fetchCalendar(ctx, &mu, data) }()
+ wg.Wait()
+
+ return data, nil
+}
+```
+
+**Semaphore for Trello (`internal/api/trello.go`):**
+```go
+const maxConcurrentRequests = 5
+
+func (c *TrelloClient) GetBoardsWithCards(ctx context.Context) ([]Board, error) {
+ sem := make(chan struct{}, maxConcurrentRequests)
+ // ... limit concurrent board fetches
+}
+```
+
+**Cache-First Fallback:**
+```go
+func (h *Handler) fetchWithFallback(ctx context.Context, fetch func() error, getCache func() (T, error)) T {
+ if err := fetch(); err != nil {
+ log.Printf("Warning: API fetch failed, using cache: %v", err)
+ cached, _ := getCache()
+ return cached
+ }
+ return freshData
+}
+```
+
+**Error Collection:**
+- Errors are collected in `data.Errors` slice
+- UI displays warning banner if any source failed
+- Page still renders with available data
+
+## Consequences
+
+**Pros:**
+- Fast page loads (parallel fetching)
+- Resilient to individual API failures
+- Respects rate limits (semaphore pattern)
+- Users see cached data rather than error page
+
+**Cons:**
+- Complexity in error handling
+- Cache fallback may show stale data without clear indication
+- Debugging requires checking multiple error sources
+- Mutex needed for shared data structure
+
+## Alternatives Considered
+
+**Option A: Sequential fetching**
+- Rejected: 5+ second page loads unacceptable
+
+**Option B: Fail-fast on any error**
+- Rejected: One flaky API shouldn't break entire dashboard
+
+**Option C: Background sync with eventual consistency**
+- Rejected: Adds complexity, users expect fresh data on load
+
+## Implementation Notes
+
+**Race Condition Prevention:**
+- All goroutines write to different fields OR use mutex
+- Trello client uses mutex for `boards[i]` slice element writes
+- Run `go test -race ./...` to verify
+
+**Monitoring:**
+- Log all API failures with `Warning:` prefix
+- Consider adding metrics for cache hit/miss rates