Add tier-gated RAG and Drive document sources (#42)
CI / test (pull_request) Successful in 10s
Unit Tests / test (pull_request) Successful in 10s

Ship allows_rag entitlement (founders/backer/pro/business), enforce it on
document APIs and RAG chat, harden ingest/delete/active lifecycle, and add
Google/Microsoft Drive connect + sync for personal and company knowledge bases.

Closes #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53
Parent epic: #42
Related: #11
This commit is contained in:
2026-08-01 15:33:55 -05:00
parent 2e9e95e16c
commit c5efe60e0b
38 changed files with 3166 additions and 98 deletions
+44
View File
@@ -93,6 +93,8 @@ with `COMPOSE_DATABASE_URL` if needed.
| `ENABLE_ACCOUNT_REGISTRATION` | `false` | optional | Self-serve sign-up; keep false until ready |
| `STRIPE_SECRET_KEY` / `STRIPE_PUBLISHABLE_KEY` / `STRIPE_WEBHOOK_SECRET` | empty | yes for billing | Stripe API + webhook |
| `STRIPE_PRICE_ID` | empty | optional | Pre-created Price; else `$10/mo` from settings |
| `GOOGLE_OAUTH_CLIENT_ID` / `..._SECRET` | empty | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `MICROSOFT_OAUTH_CLIENT_ID` / `..._SECRET` / `..._TENANT` | empty / `common` | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `FRONTEND_BASE_URL` | `http://localhost:3000` | set in prod/beta | Checkout success/cancel, portal return, OAuth return |
| `STRIPE_PORTAL_RETURN_URL` | `{FRONTEND}/account/` | optional | Stripe Customer Portal return URL |
| `CORS_ALLOWED_ORIGINS` | local + chat FE (+ beta FE default) | set in prod/beta | Frontend origin(s) |
@@ -209,6 +211,48 @@ Subscription audit (`UserAuthEvent` on the user admin):
- `subscription_started` — first active plan (Checkout, Backer redeem, admin assign)
- `subscription_updated` — plan/status/cancel-at-period-end changes (portal + webhooks)
### Drive / RAG sync ([#47](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/47)-[#53](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/53))
Personal Google Drive / OneDrive and company Google Shared Drive / SharePoint
sync into the existing RAG `Document` pipeline. Every endpoint below is gated
by `assert_feature_allowed(user, "rag")` (`SubscriptionPlan.allows_rag`
true for Founders/Pro/Business/Backer, false for Standard by default).
**Connect (OAuth, reuses `#24` SSO app registrations with incremental scopes):**
| | |
|--|--|
| Personal | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_drive` (authenticated) |
| Company | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_company_drive` (company manager only) |
| Callback | Same `/api/auth/oauth/<provider>/callback/` as SSO; the signed OAuth `state` carries the linking `user_id` since the browser has no session on the IdP redirect. Upserts a `DriveConnection` and redirects to `{FRONTEND_BASE_URL}/account/?drive_connected=1&provider=<provider>&kind=<personal\|company>` (or `?error=<code>`) |
**Manage:**
| Method / path | Notes |
|--|--|
| `GET /api/drive/connections/` | Caller's personal connections + their company's company connections |
| `DELETE /api/drive/connections/<id>/` | Disconnect (owner for personal, company manager for company) — deactivates + clears tokens, keeps history |
| `POST /api/drive/connections/<id>/resources/` | `{ "resource_ids": [...], "resource_labels": [...] }` — folder/shared-drive/site ids to sync; empty = provider root |
| `POST /api/drive/connections/<id>/sync/` | Sync now (`chat_backend/services/drive_sync.py::sync_connection`) |
**Provider scope differences:**
- Google: same `drive.readonly` scope for personal and company; company sync
reads Shared Drives via `corpora=drive` + `supportsAllDrives`.
- Microsoft: personal uses `Files.Read`; company uses `Files.Read.All
Sites.Read.All` and syncs SharePoint sites (`selected_resource_ids` = site ids).
**Workers / webhooks (#52):**
- `python manage.py sync_drive_connections [--connection-id N]` — cron/worker entry point.
- `POST /api/drive/webhooks/google/` / `POST /api/drive/webhooks/microsoft/` —
provider push-notification stubs (`AllowAny`); acknowledge `200` and call
`sync_connection` when the notification's `connection_id` is resolvable,
else just `200` (no-op). Microsoft's subscription-creation `validationToken`
handshake is echoed back as `text/plain`.
Google-native Docs/Sheets/Slides are exported to `.docx`/`.xlsx`/`.pdf` before
ingest (Chroma/RAG loaders don't read the native formats). Documents whose
remote file was deleted upstream are removed on the next sync.
## Security note
Secrets previously hardcoded in `settings.py` (email password, captcha, Django