ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

Connect OneDrive / Google Drive as RAG document sources #11

New Issue
Open
opened 2026-07-26 04:10:07 -07:00 by westfarn · 2 comments
westfarn commented 2026-07-26 04:10:07 -07:00
Owner
Copy Link
Copy Source

Summary

Let a user connect their OneDrive or Google Drive account so files from those drives can be ingested into a RAG document workspace, instead of only manual uploads.

Motivation

Today RAG ingestion only accepts files uploaded directly (DocumentUploadView -> Document -> RAGService.ingest_documents). Users already keep docs in cloud drives; letting them link a drive removes manual re-uploads and keeps the knowledge base fresh.

Proposed behavior

  1. OAuth connect – user authorizes OneDrive (Microsoft Graph) and/or Google Drive (Drive API) per account/company; store tokens securely (encrypted / refresh-token aware).
  2. Source selection – user picks folders/files (or a whole drive) to attach to a DocumentWorkspace.
  3. Ingestion – pull selected files, run through existing loaders (PyPDFLoader, Docx2txtLoader, TextLoader, UnstructuredFileLoader), chunk + embed into the workspace vector store.
  4. Sync – re-ingest on demand and/or on a schedule; detect changed/deleted files and update the vector store accordingly.

Likely touch points

  • Models: new DriveConnection (provider, account, tokens, workspace FK) + link to Document (source provider + remote file id/etag).
  • services/rag_services.py – add cloud-drive source loaders alongside upload path.
  • views.py / serializers.py – OAuth start/callback endpoints, list-remote-files, attach-to-workspace.
  • Settings/secrets – Google + Microsoft OAuth client IDs/secrets, redirect URIs.
  • Background task/scheduler for sync (out of scope if none exists — call it out).

Open questions

  • Per-user vs per-company drive connections?
  • Token storage: reuse DB storage + encryption, or a dedicated secrets store?
  • Sync trigger: manual button, webhook (Graph/Drive change notifications), or periodic poll?
  • File-size / type limits and how to handle Google-native docs (export to pdf/docx)?

Acceptance criteria

  • User can connect a OneDrive account via OAuth
  • User can connect a Google Drive account via OAuth
  • User can select drive folders/files and attach them to a DocumentWorkspace
  • Selected files are ingested into the workspace vector store using existing loaders
  • Re-sync updates the vector store for added/changed/removed files
  • Tokens stored securely and refreshed automatically
  • Tests for OAuth callback handling and ingestion of a mocked drive file
## Summary Let a user connect their **OneDrive** or **Google Drive** account so files from those drives can be ingested into a RAG document workspace, instead of only manual uploads. ## Motivation Today RAG ingestion only accepts files uploaded directly (`DocumentUploadView` -> `Document` -> `RAGService.ingest_documents`). Users already keep docs in cloud drives; letting them link a drive removes manual re-uploads and keeps the knowledge base fresh. ## Proposed behavior 1. **OAuth connect** – user authorizes OneDrive (Microsoft Graph) and/or Google Drive (Drive API) per account/company; store tokens securely (encrypted / refresh-token aware). 2. **Source selection** – user picks folders/files (or a whole drive) to attach to a `DocumentWorkspace`. 3. **Ingestion** – pull selected files, run through existing loaders (`PyPDFLoader`, `Docx2txtLoader`, `TextLoader`, `UnstructuredFileLoader`), chunk + embed into the workspace vector store. 4. **Sync** – re-ingest on demand and/or on a schedule; detect changed/deleted files and update the vector store accordingly. ## Likely touch points - Models: new `DriveConnection` (provider, account, tokens, workspace FK) + link to `Document` (source provider + remote file id/etag). - `services/rag_services.py` – add cloud-drive source loaders alongside upload path. - `views.py` / `serializers.py` – OAuth start/callback endpoints, list-remote-files, attach-to-workspace. - Settings/secrets – Google + Microsoft OAuth client IDs/secrets, redirect URIs. - Background task/scheduler for sync (out of scope if none exists — call it out). ## Open questions - Per-user vs per-company drive connections? - Token storage: reuse DB storage + encryption, or a dedicated secrets store? - Sync trigger: manual button, webhook (Graph/Drive change notifications), or periodic poll? - File-size / type limits and how to handle Google-native docs (export to pdf/docx)? ## Acceptance criteria - [ ] User can connect a OneDrive account via OAuth - [ ] User can connect a Google Drive account via OAuth - [ ] User can select drive folders/files and attach them to a `DocumentWorkspace` - [ ] Selected files are ingested into the workspace vector store using existing loaders - [ ] Re-sync updates the vector store for added/changed/removed files - [ ] Tokens stored securely and refreshed automatically - [ ] Tests for OAuth callback handling and ingestion of a mocked drive file
westfarn commented 2026-07-27 04:12:03 -07:00
Author
Owner
Copy Link
Copy Source

Prerequisite / blocks this work: ai_ml_operations/chat_backend#24 (SSO with Google / Microsoft for registration & sign-in).

#24

Do #24 first so Drive connect can reuse the same Google/Microsoft OAuth app setup and token patterns.

Prerequisite / blocks this work: ai_ml_operations/chat_backend#24 (SSO with Google / Microsoft for registration & sign-in). https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/24 Do #24 first so Drive connect can reuse the same Google/Microsoft OAuth app setup and token patterns.
westfarn commented 2026-08-01 13:05:51 -07:00
Author
Owner
Copy Link
Copy Source

Parented under epic #42.

Scope split:

  • #47 — OAuth Drive connect + incremental scopes
  • #48 — Google Drive personal sync
  • #49 — OneDrive personal sync
  • #50 / #51 — Business M365 + Google Shared Drives (separate from personal)
  • #52 — webhooks/delta workers

Keep this issue as the original Drive-source tracker; prefer closing via the split children when done.

Parented under epic [#42](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/42). Scope split: - [#47](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/47) — OAuth Drive connect + incremental scopes - [#48](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/48) — Google Drive personal sync - [#49](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/49) — OneDrive personal sync - [#50](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/50) / [#51](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/51) — Business M365 + Google Shared Drives (separate from personal) - [#52](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/52) — webhooks/delta workers Keep this issue as the original Drive-source tracker; prefer closing via the split children when done.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Depends on
#24 Add SSO (Google / Microsoft) for account registration and sign-in
ai_ml_operations/chat_backend
Reference: ai_ml_operations/chat_backend#11