diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-01-20 15:25:00 -1000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-01-20 15:25:00 -1000 |
| commit | d799d4d04cc18654de5864a458668ff073e26284 (patch) | |
| tree | c47ce10b1eb72bc6886768afaae201a4a33151b8 /issues/phase4_step2_research_sync.md | |
| parent | 991cce6e7ed77cc164df04457942a14da1e50039 (diff) | |
Add task tracking issues
Document completed and planned tasks:
- Obsidian removal (completed)
- Authentication implementation (completed)
- VPS deployment preparation (planned)
- API sync optimization research
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Diffstat (limited to 'issues/phase4_step2_research_sync.md')
| -rw-r--r-- | issues/phase4_step2_research_sync.md | 124 |
1 files changed, 124 insertions, 0 deletions
diff --git a/issues/phase4_step2_research_sync.md b/issues/phase4_step2_research_sync.md new file mode 100644 index 0000000..d690e6f --- /dev/null +++ b/issues/phase4_step2_research_sync.md @@ -0,0 +1,124 @@ +# Phase 4 Step 2: Research Efficient Sync + +## Research Findings + +### Todoist Sync API + +**Test Results (from `research_test.go`):** +- Full sync response: **17,502 bytes** +- Incremental sync response: **179 bytes** (99% reduction!) +- `sync_token` successfully enables incremental updates + +**Response Structure:** +```json +{ + "sync_token": "...", + "full_sync": true/false, + "items": [...] // tasks +} +``` + +**Item (Task) Fields Available:** +- `id`, `content`, `description`, `project_id` +- `checked` (completion status) +- `is_deleted` (for handling deletions) +- `updated_at` (for tracking changes) +- `due` (with `date`, `is_recurring`, `string`) + +**Implementation Approach:** +1. Store `sync_token` in database (new table or cache_metadata) +2. On refresh, call Sync API with stored token +3. Apply incremental changes to local database: + - Update modified tasks + - Delete tasks where `is_deleted=true` + - Mark tasks where `checked=true` as completed +4. If `full_sync=true` in response, clear and rebuild local state + +### Trello API Optimization + +**Test Results:** +- Full boards response: **48,228 bytes** (all fields) +- Limited fields (id,name): **462 bytes** (99% reduction!) +- Single board with embedded cards/lists: **10,452 bytes** + +**Optimization Strategies:** + +1. **Field Filtering** (immediate win): + ``` + /boards?fields=id,name,dateLastActivity + /boards/{id}/cards?fields=id,name,idList,due,url + ``` + +2. **Batch Requests** (reduce API calls): + ``` + /boards/{id}?cards=visible&card_fields=id,name,idList,due,url&lists=open&list_fields=id,name + ``` + - Fetches board + cards + lists in single request + +3. **`dateLastActivity` Caching**: + - Store `dateLastActivity` for each board + - On refresh, fetch boards with minimal fields + - Only fetch cards for boards where `dateLastActivity` changed + - Note: `since` parameter exists but returns full data (not truly incremental) + +### Architecture Implications + +**Current Architecture:** +- Cache-aside pattern: Check cache -> Miss -> Fetch API -> Store in cache +- Problem: Full data fetch on every cache miss + +**Proposed Architecture:** +- **Sync Engine Pattern**: + - SQLite is the source of truth + - Sync process updates SQLite incrementally + - Read operations always hit SQLite (fast) + +**Migration Path:** +1. Phase A: Add field filtering to Trello (immediate benefit, no architecture change) +2. Phase B: Implement Todoist Sync API with sync_token storage +3. Phase C: Add dateLastActivity tracking for Trello boards + +## Implementation Plan + +### Phase A: Trello Field Optimization (Low effort, High impact) + +1. Update `GetBoards` to use `fields=id,name,dateLastActivity` +2. Update `GetCards` to use `fields=id,name,idList,due,url,desc` +3. Consider batch endpoint for single-board-with-cards + +### Phase B: Todoist Incremental Sync (Medium effort, High impact) + +1. Add `sync_tokens` table: + ```sql + CREATE TABLE sync_tokens ( + service TEXT PRIMARY KEY, + token TEXT NOT NULL, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP + ); + ``` + +2. Create `SyncTasks` method in `TodoistClient`: + - Accept previous sync_token + - Return new sync_token + changed items + +3. Create `ApplyTaskChanges` method in `Store`: + - Handle updates, deletions, completions + - Update sync_token after successful apply + +4. Update `fetchTasks` to use sync when token exists + +### Phase C: Trello Smart Refresh (Medium effort, Medium impact) + +1. Store `dateLastActivity` per board in database +2. On refresh: + - Fetch boards with minimal fields + - Compare `dateLastActivity` with stored values + - Only fetch cards for changed boards + +## Recommendation + +**Start with Phase A** - simple field filtering provides immediate 99% reduction in Trello response size with minimal code changes. + +**Then Phase B** - Todoist Sync API provides the best efficiency gains (99% reduction) and is well-documented. + +**Phase C is optional** - Trello doesn't have true incremental sync, so the benefit is limited to skipping unchanged boards. |
