Tier-gated RAG + Drive document sources (#42) (#54)
Unit Tests / test (push) Successful in 10s
Deploy Beta / unit-tests (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 40s

## Summary

Implements epic [#42](#42) (children #43–#53) and advances [#11](#11).

- **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag`
- **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")`
- **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE
- **Workspaces:** auto-create default company workspace; fail-closed scoping
- **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command
- **Docs/env:** README + `.env*.example` updated

Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85).

## Test plan

- [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK)
- [ ] Migrate finance `0004` + chat_backend `0028` on beta
- [ ] Verify Standard user: Documents API 403 + no RAG retrieval
- [ ] Verify Founders/Pro: upload + list + active toggle
- [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync
- [ ] Company manager: `link_company_drive`; non-manager 403
- [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54
This commit was merged in pull request #54.
This commit is contained in:
2026-08-01 14:02:36 -07:00
parent 2e9e95e16c
commit d54094f5e0
38 changed files with 3166 additions and 98 deletions
+44
View File
@@ -93,6 +93,8 @@ with `COMPOSE_DATABASE_URL` if needed.
| `ENABLE_ACCOUNT_REGISTRATION` | `false` | optional | Self-serve sign-up; keep false until ready |
| `STRIPE_SECRET_KEY` / `STRIPE_PUBLISHABLE_KEY` / `STRIPE_WEBHOOK_SECRET` | empty | yes for billing | Stripe API + webhook |
| `STRIPE_PRICE_ID` | empty | optional | Pre-created Price; else `$10/mo` from settings |
| `GOOGLE_OAUTH_CLIENT_ID` / `..._SECRET` | empty | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `MICROSOFT_OAUTH_CLIENT_ID` / `..._SECRET` / `..._TENANT` | empty / `common` | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `FRONTEND_BASE_URL` | `http://localhost:3000` | set in prod/beta | Checkout success/cancel, portal return, OAuth return |
| `STRIPE_PORTAL_RETURN_URL` | `{FRONTEND}/account/` | optional | Stripe Customer Portal return URL |
| `CORS_ALLOWED_ORIGINS` | local + chat FE (+ beta FE default) | set in prod/beta | Frontend origin(s) |
@@ -209,6 +211,48 @@ Subscription audit (`UserAuthEvent` on the user admin):
- `subscription_started` — first active plan (Checkout, Backer redeem, admin assign)
- `subscription_updated` — plan/status/cancel-at-period-end changes (portal + webhooks)
### Drive / RAG sync ([#47](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/47)-[#53](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/53))
Personal Google Drive / OneDrive and company Google Shared Drive / SharePoint
sync into the existing RAG `Document` pipeline. Every endpoint below is gated
by `assert_feature_allowed(user, "rag")` (`SubscriptionPlan.allows_rag`
true for Founders/Pro/Business/Backer, false for Standard by default).
**Connect (OAuth, reuses `#24` SSO app registrations with incremental scopes):**
| | |
|--|--|
| Personal | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_drive` (authenticated) |
| Company | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_company_drive` (company manager only) |
| Callback | Same `/api/auth/oauth/<provider>/callback/` as SSO; the signed OAuth `state` carries the linking `user_id` since the browser has no session on the IdP redirect. Upserts a `DriveConnection` and redirects to `{FRONTEND_BASE_URL}/account/?drive_connected=1&provider=<provider>&kind=<personal\|company>` (or `?error=<code>`) |
**Manage:**
| Method / path | Notes |
|--|--|
| `GET /api/drive/connections/` | Caller's personal connections + their company's company connections |
| `DELETE /api/drive/connections/<id>/` | Disconnect (owner for personal, company manager for company) — deactivates + clears tokens, keeps history |
| `POST /api/drive/connections/<id>/resources/` | `{ "resource_ids": [...], "resource_labels": [...] }` — folder/shared-drive/site ids to sync; empty = provider root |
| `POST /api/drive/connections/<id>/sync/` | Sync now (`chat_backend/services/drive_sync.py::sync_connection`) |
**Provider scope differences:**
- Google: same `drive.readonly` scope for personal and company; company sync
reads Shared Drives via `corpora=drive` + `supportsAllDrives`.
- Microsoft: personal uses `Files.Read`; company uses `Files.Read.All
Sites.Read.All` and syncs SharePoint sites (`selected_resource_ids` = site ids).
**Workers / webhooks (#52):**
- `python manage.py sync_drive_connections [--connection-id N]` — cron/worker entry point.
- `POST /api/drive/webhooks/google/` / `POST /api/drive/webhooks/microsoft/` —
provider push-notification stubs (`AllowAny`); acknowledge `200` and call
`sync_connection` when the notification's `connection_id` is resolvable,
else just `200` (no-op). Microsoft's subscription-creation `validationToken`
handshake is echoed back as `text/plain`.
Google-native Docs/Sheets/Slides are exported to `.docx`/`.xlsx`/`.pdf` before
ingest (Chroma/RAG loaders don't read the native formats). Documents whose
remote file was deleted upstream are removed on the next sync.
## Security note
Secrets previously hardcoded in `settings.py` (email password, captcha, Django