# Phase 4 Step 2: Research Efficient Sync

## Research Findings

### Todoist Sync API

**Test Results (from `research_test.go`):**
- Full sync response: **17,502 bytes**
- Incremental sync response: **179 bytes** (99% reduction!)
- `sync_token` successfully enables incremental updates

**Response Structure:**
```json
{
  "sync_token": "...",
  "full_sync": true/false,
  "items": [...]  // tasks
}
```

**Item (Task) Fields Available:**
- `id`, `content`, `description`, `project_id`
- `checked` (completion status)
- `is_deleted` (for handling deletions)
- `updated_at` (for tracking changes)
- `due` (with `date`, `is_recurring`, `string`)

**Implementation Approach:**
1. Store `sync_token` in database (new table or cache_metadata)
2. On refresh, call Sync API with stored token
3. Apply incremental changes to local database:
   - Update modified tasks
   - Delete tasks where `is_deleted=true`
   - Mark tasks where `checked=true` as completed
4. If `full_sync=true` in response, clear and rebuild local state

### Trello API Optimization

**Test Results:**
- Full boards response: **48,228 bytes** (all fields)
- Limited fields (id,name): **462 bytes** (99% reduction!)
- Single board with embedded cards/lists: **10,452 bytes**

**Optimization Strategies:**

1. **Field Filtering** (immediate win):
   ```
   /boards?fields=id,name,dateLastActivity
   /boards/{id}/cards?fields=id,name,idList,due,url
   ```

2. **Batch Requests** (reduce API calls):
   ```
   /boards/{id}?cards=visible&card_fields=id,name,idList,due,url&lists=open&list_fields=id,name
   ```
   - Fetches board + cards + lists in single request

3. **`dateLastActivity` Caching**:
   - Store `dateLastActivity` for each board
   - On refresh, fetch boards with minimal fields
   - Only fetch cards for boards where `dateLastActivity` changed
   - Note: `since` parameter exists but returns full data (not truly incremental)

### Architecture Implications

**Current Architecture:**
- Cache-aside pattern: Check cache -> Miss -> Fetch API -> Store in cache
- Problem: Full data fetch on every cache miss

**Proposed Architecture:**
- **Sync Engine Pattern**:
  - SQLite is the source of truth
  - Sync process updates SQLite incrementally
  - Read operations always hit SQLite (fast)

**Migration Path:**
1. Phase A: Add field filtering to Trello (immediate benefit, no architecture change)
2. Phase B: Implement Todoist Sync API with sync_token storage
3. Phase C: Add dateLastActivity tracking for Trello boards

## Implementation Plan

### Phase A: Trello Field Optimization (Low effort, High impact)

1. Update `GetBoards` to use `fields=id,name,dateLastActivity`
2. Update `GetCards` to use `fields=id,name,idList,due,url,desc`
3. Consider batch endpoint for single-board-with-cards

### Phase B: Todoist Incremental Sync (Medium effort, High impact)

1. Add `sync_tokens` table:
   ```sql
   CREATE TABLE sync_tokens (
     service TEXT PRIMARY KEY,
     token TEXT NOT NULL,
     updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
   );
   ```

2. Create `SyncTasks` method in `TodoistClient`:
   - Accept previous sync_token
   - Return new sync_token + changed items

3. Create `ApplyTaskChanges` method in `Store`:
   - Handle updates, deletions, completions
   - Update sync_token after successful apply

4. Update `fetchTasks` to use sync when token exists

### Phase C: Trello Smart Refresh (Medium effort, Medium impact)

1. Store `dateLastActivity` per board in database
2. On refresh:
   - Fetch boards with minimal fields
   - Compare `dateLastActivity` with stored values
   - Only fetch cards for changed boards

## Recommendation

**Start with Phase A** - simple field filtering provides immediate 99% reduction in Trello response size with minimal code changes.

**Then Phase B** - Todoist Sync API provides the best efficiency gains (99% reduction) and is well-documented.

**Phase C is optional** - Trello doesn't have true incremental sync, so the benefit is limited to skipping unchanged boards.