summaryrefslogtreecommitdiff
path: root/issues/phase4_step2_research_sync.md
diff options
context:
space:
mode:
authorPeter Stone <thepeterstone@gmail.com>2026-01-20 15:25:00 -1000
committerPeter Stone <thepeterstone@gmail.com>2026-01-20 15:25:00 -1000
commitd799d4d04cc18654de5864a458668ff073e26284 (patch)
treec47ce10b1eb72bc6886768afaae201a4a33151b8 /issues/phase4_step2_research_sync.md
parent991cce6e7ed77cc164df04457942a14da1e50039 (diff)
Add task tracking issues
Document completed and planned tasks: - Obsidian removal (completed) - Authentication implementation (completed) - VPS deployment preparation (planned) - API sync optimization research Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Diffstat (limited to 'issues/phase4_step2_research_sync.md')
-rw-r--r--issues/phase4_step2_research_sync.md124
1 files changed, 124 insertions, 0 deletions
diff --git a/issues/phase4_step2_research_sync.md b/issues/phase4_step2_research_sync.md
new file mode 100644
index 0000000..d690e6f
--- /dev/null
+++ b/issues/phase4_step2_research_sync.md
@@ -0,0 +1,124 @@
+# Phase 4 Step 2: Research Efficient Sync
+
+## Research Findings
+
+### Todoist Sync API
+
+**Test Results (from `research_test.go`):**
+- Full sync response: **17,502 bytes**
+- Incremental sync response: **179 bytes** (99% reduction!)
+- `sync_token` successfully enables incremental updates
+
+**Response Structure:**
+```json
+{
+ "sync_token": "...",
+ "full_sync": true/false,
+ "items": [...] // tasks
+}
+```
+
+**Item (Task) Fields Available:**
+- `id`, `content`, `description`, `project_id`
+- `checked` (completion status)
+- `is_deleted` (for handling deletions)
+- `updated_at` (for tracking changes)
+- `due` (with `date`, `is_recurring`, `string`)
+
+**Implementation Approach:**
+1. Store `sync_token` in database (new table or cache_metadata)
+2. On refresh, call Sync API with stored token
+3. Apply incremental changes to local database:
+ - Update modified tasks
+ - Delete tasks where `is_deleted=true`
+ - Mark tasks where `checked=true` as completed
+4. If `full_sync=true` in response, clear and rebuild local state
+
+### Trello API Optimization
+
+**Test Results:**
+- Full boards response: **48,228 bytes** (all fields)
+- Limited fields (id,name): **462 bytes** (99% reduction!)
+- Single board with embedded cards/lists: **10,452 bytes**
+
+**Optimization Strategies:**
+
+1. **Field Filtering** (immediate win):
+ ```
+ /boards?fields=id,name,dateLastActivity
+ /boards/{id}/cards?fields=id,name,idList,due,url
+ ```
+
+2. **Batch Requests** (reduce API calls):
+ ```
+ /boards/{id}?cards=visible&card_fields=id,name,idList,due,url&lists=open&list_fields=id,name
+ ```
+ - Fetches board + cards + lists in single request
+
+3. **`dateLastActivity` Caching**:
+ - Store `dateLastActivity` for each board
+ - On refresh, fetch boards with minimal fields
+ - Only fetch cards for boards where `dateLastActivity` changed
+ - Note: `since` parameter exists but returns full data (not truly incremental)
+
+### Architecture Implications
+
+**Current Architecture:**
+- Cache-aside pattern: Check cache -> Miss -> Fetch API -> Store in cache
+- Problem: Full data fetch on every cache miss
+
+**Proposed Architecture:**
+- **Sync Engine Pattern**:
+ - SQLite is the source of truth
+ - Sync process updates SQLite incrementally
+ - Read operations always hit SQLite (fast)
+
+**Migration Path:**
+1. Phase A: Add field filtering to Trello (immediate benefit, no architecture change)
+2. Phase B: Implement Todoist Sync API with sync_token storage
+3. Phase C: Add dateLastActivity tracking for Trello boards
+
+## Implementation Plan
+
+### Phase A: Trello Field Optimization (Low effort, High impact)
+
+1. Update `GetBoards` to use `fields=id,name,dateLastActivity`
+2. Update `GetCards` to use `fields=id,name,idList,due,url,desc`
+3. Consider batch endpoint for single-board-with-cards
+
+### Phase B: Todoist Incremental Sync (Medium effort, High impact)
+
+1. Add `sync_tokens` table:
+ ```sql
+ CREATE TABLE sync_tokens (
+ service TEXT PRIMARY KEY,
+ token TEXT NOT NULL,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
+ );
+ ```
+
+2. Create `SyncTasks` method in `TodoistClient`:
+ - Accept previous sync_token
+ - Return new sync_token + changed items
+
+3. Create `ApplyTaskChanges` method in `Store`:
+ - Handle updates, deletions, completions
+ - Update sync_token after successful apply
+
+4. Update `fetchTasks` to use sync when token exists
+
+### Phase C: Trello Smart Refresh (Medium effort, Medium impact)
+
+1. Store `dateLastActivity` per board in database
+2. On refresh:
+ - Fetch boards with minimal fields
+ - Compare `dateLastActivity` with stored values
+ - Only fetch cards for changed boards
+
+## Recommendation
+
+**Start with Phase A** - simple field filtering provides immediate 99% reduction in Trello response size with minimal code changes.
+
+**Then Phase B** - Todoist Sync API provides the best efficiency gains (99% reduction) and is well-documented.
+
+**Phase C is optional** - Trello doesn't have true incremental sync, so the benefit is limited to skipping unchanged boards.