Async Drive sync task + surface sync errors to clients #57

New Issue

2026-08-02T03:59:04-07:00

westfarn commented

2026-08-02 03:59:04 -07:00

Summary

POST /api/drive/connections/<id>/sync/ currently runs sync_connection() inline in the request. Large Drive libraries block the HTTP request; Google/Microsoft API failures (e.g. Drive API disabled → 403) still often return HTTP 200 with last_sync_status=error, and the FE only console.logs — users see a green “Drive connected” banner from OAuth and no clear sync failure.

Make Sync now (and ideally webhook/cron triggers) enqueue a background Django task, return quickly with pending status, and surface sync errors to the client.

Parent: #42
Related: #52 (webhooks + periodic workers), #55

Companion FE: create / link chat_web_app ticket for toast/alert UX (see FE acceptance below).

Current behavior

DriveConnectionSyncView.post → sync_connection(connection) → blocking
On provider errors, connection is updated (last_sync_status=error, last_sync_error=...) but response is still 200 { result, connection }
FE handleSync does not check last_sync_status / last_sync_error
No Celery / RQ / Huey in repo today — only manage.py sync_drive_connections for cron (#52)

Desired behavior

Backend

Introduce a background task runner suitable for this deploy (prefer Celery + Redis/Rabbit if ops-ready; document choice in PR). Reuse same worker for #52 delta/cron if practical.
POST .../sync/:
- Auth / RAG gate / ownership checks unchanged
- Set last_sync_status=pending, clear or keep prior error per product choice
- Enqueue sync_connection(connection_id) task
- Return 202 Accepted (or 200) with serialized connection in pending state — do not wait for Drive list/download/ingest
Task runs sync_connection; on success/error update last_sync_status, last_sync_at, last_sync_error as today
Idempotency: avoid stacking duplicate concurrent syncs for the same connection (skip/enqueue-once while pending)
Webhook path (_maybe_sync_from_webhook) and management command should enqueue the same task (or call shared entrypoint) so all triggers are async-consistent
Optional: GET .../connections/<id>/ already exposes status — ensure FE can poll; document poll interval

Error surfacing (API contract for FE)

DriveConnectionSerializer already has last_sync_status, last_sync_error, last_sync_at — keep accurate after async job
Sync enqueue response must include those fields
Do not rely on HTTP 500 for provider/config errors (Drive API disabled, token refresh fail) — those are connection-level errors
Consider a short machine-readable last_sync_error_code later; for this ticket, human-readable last_sync_error is enough if FE shows it

Frontend (companion work — may be separate `chat_web_app` issue)

After Sync now: treat enqueue success ≠ sync success
While pending: show “Syncing…” / disable button; poll connection list or single connection until status leaves pending (timeout + message)
On error: error toast/alert with last_sync_error (or truncated + “see details”)
On ok: optional success toast; refresh documents list
Green “Drive connected” banner stays OAuth-only (drive_connected=1); never imply sync succeeded

Acceptance

Sync now returns quickly without waiting for Drive API / ingest
Worker updates last_sync_status / last_sync_error / last_sync_at
Concurrent duplicate syncs for one connection are coalesced or rejected cleanly
Cron/webhook use same async path (or documented exception)
Tests: enqueue mocked; task updates status on success/failure
FE (or linked ticket): error alert/toast when sync ends in error; pending UX while running
README / .env*.example: worker process + broker settings

Out of scope

Enabling Google Drive API in GCP (ops/config — separate from this bug)
Full webhook subscription setup beyond enqueue hook (#52)
Multi-workspace personal+company retrieval (#46)

Repro (today)

Connect Google Drive (OAuth succeeds → green banner)
Click Sync now
Request blocks / returns 200 with last_sync_status=error and Drive API disabled message in last_sync_error
UI shows no error toast

## Summary `POST /api/drive/connections/<id>/sync/` currently runs `sync_connection()` **inline** in the request. Large Drive libraries block the HTTP request; Google/Microsoft API failures (e.g. Drive API disabled → 403) still often return **HTTP 200** with `last_sync_status=error`, and the FE only `console.log`s — users see a green “Drive connected” banner from OAuth and no clear sync failure. Make **Sync now** (and ideally webhook/cron triggers) enqueue a **background Django task**, return quickly with `pending` status, and **surface sync errors** to the client. Parent: [#42](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/42) Related: [#52](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/52) (webhooks + periodic workers), [#55](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/55) Companion FE: create / link `chat_web_app` ticket for toast/alert UX (see FE acceptance below). ## Current behavior - `DriveConnectionSyncView.post` → `sync_connection(connection)` → blocking - On provider errors, connection is updated (`last_sync_status=error`, `last_sync_error=...`) but response is still 200 `{ result, connection }` - FE `handleSync` does not check `last_sync_status` / `last_sync_error` - No Celery / RQ / Huey in repo today — only `manage.py sync_drive_connections` for cron (#52) ## Desired behavior ### Backend 1. Introduce a background task runner suitable for this deploy (prefer **Celery** + Redis/Rabbit if ops-ready; document choice in PR). Reuse same worker for #52 delta/cron if practical. 2. `POST .../sync/`: - Auth / RAG gate / ownership checks unchanged - Set `last_sync_status=pending`, clear or keep prior error per product choice - Enqueue `sync_connection(connection_id)` task - Return **202 Accepted** (or 200) with serialized connection in `pending` state — **do not** wait for Drive list/download/ingest 3. Task runs `sync_connection`; on success/error update `last_sync_status`, `last_sync_at`, `last_sync_error` as today 4. Idempotency: avoid stacking duplicate concurrent syncs for the same connection (skip/enqueue-once while `pending`) 5. Webhook path (`_maybe_sync_from_webhook`) and management command should enqueue the same task (or call shared entrypoint) so all triggers are async-consistent 6. Optional: `GET .../connections/<id>/` already exposes status — ensure FE can poll; document poll interval ### Error surfacing (API contract for FE) - `DriveConnectionSerializer` already has `last_sync_status`, `last_sync_error`, `last_sync_at` — keep accurate after async job - Sync enqueue response must include those fields - Do **not** rely on HTTP 500 for provider/config errors (Drive API disabled, token refresh fail) — those are connection-level errors - Consider a short machine-readable `last_sync_error_code` later; for this ticket, human-readable `last_sync_error` is enough if FE shows it ### Frontend (companion work — may be separate `chat_web_app` issue) - After Sync now: treat enqueue success ≠ sync success - While `pending`: show “Syncing…” / disable button; poll connection list or single connection until status leaves `pending` (timeout + message) - On `error`: **error toast/alert** with `last_sync_error` (or truncated + “see details”) - On `ok`: optional success toast; refresh documents list - Green “Drive connected” banner stays **OAuth-only** (`drive_connected=1`); never imply sync succeeded ## Acceptance - [ ] Sync now returns quickly without waiting for Drive API / ingest - [ ] Worker updates `last_sync_status` / `last_sync_error` / `last_sync_at` - [ ] Concurrent duplicate syncs for one connection are coalesced or rejected cleanly - [ ] Cron/webhook use same async path (or documented exception) - [ ] Tests: enqueue mocked; task updates status on success/failure - [ ] FE (or linked ticket): error alert/toast when sync ends in `error`; pending UX while running - [ ] README / `.env*.example`: worker process + broker settings ## Out of scope - Enabling Google Drive API in GCP (ops/config — separate from this bug) - Full webhook subscription setup beyond enqueue hook (#52) - Multi-workspace personal+company retrieval (#46) ## Repro (today) 1. Connect Google Drive (OAuth succeeds → green banner) 2. Click Sync now 3. Request blocks / returns 200 with `last_sync_status=error` and Drive API disabled message in `last_sync_error` 4. UI shows no error toast

westfarn commented

2026-08-02 03:59:07 -07:00