chat_backend/README.md

# Chat Backend

Django + Channels API for AIML Operations chat (`chatbackend.aimloperations.com`).
Packaging via `uv`; production deploy via `server-infra`.

Companion frontend: [`chat_web_app`](https://git.aimloperations.com/ai_ml_operations/chat_web_app)
(node-static, not Docker).

Ticket: [chat_backend#6](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/6)

## Layout

```text
chat_backend/                 ← repo root (Dockerfile, compose, pyproject, .gitea)
├── llm_be/                   ← Django project root (manage.py)
│   ├── manage.py
│   ├── llm_be/               ← settings, urls, asgi/wsgi
│   └── chat_backend/         ← app (models, consumers, services, storage)
├── scripts/
│   ├── docker-entrypoint.sh
│   └── validate-env.sh
├── docker-compose.yml        ← local/CI (bundled Postgres)
└── docker-compose.prod.yml   ← server-infra (external DATABASE_URL)
```

## Local development

### Prerequisites

- Python 3.12+
- [uv](https://docs.astral.sh/uv/)
- Docker + Docker Compose (optional, recommended)
- Ollama reachable at `OLLAMA_BASE_URL` for LLM features

### uv (host)

```bash
cp .env.example .env
uv sync
cd llm_be
uv run python manage.py migrate
uv run python manage.py runserver 0.0.0.0:8003
```

Without `DATABASE_URL` / `DB_HOST`, settings fall back to SQLite (`llm_be/db.sqlite3`).

Tests:

```bash
cd llm_be
uv run python manage.py test
```

The suite is offline by default — the custom test runner
(`llm_be/test_runner.py`) sets `SKIP_RAG_INIT=1` and a cheap password hasher, and
LLM chains are faked, so no Ollama, Chroma, SMTP or network access is needed.
Tests live in `llm_be/chat_backend/tests/` (models, storage, serializers, views,
services, signals, consumers).

Non-deterministic checks against a real model server are opt-in:

```bash
cd llm_be
RUN_LIVE_OLLAMA_TESTS=1 uv run python manage.py test chat_backend.tests.test_live_ollama
```

### Docker (dev, bundled Postgres)

```bash
docker compose up --build
```

App: http://localhost:8003 — Postgres via bundled `db`
(`postgres://chat_backend:chat_backend@db:5432/chat_backend`).
Compose does **not** read host `DATABASE_URL` (avoids CI/prod leaks); override
with `COMPOSE_DATABASE_URL` if needed.

## Environment variables

| Variable | Dev default | Prod required | Notes |
|----------|-------------|---------------|-------|
| `DJANGO_ENV` | `dev` | `prod` / `beta` | |
| `DJANGO_SECRET_KEY` | insecure default | yes | Must be real in prod/beta |
| `DJANGO_DEBUG` | true when `dev` | `false` | |
| `DJANGO_ALLOWED_HOSTS` | localhost + chat hosts | yes | Comma-separated |
| `DJANGO_CSRF_TRUSTED_ORIGINS` | derived from hosts | optional | Full origins |
| `DATABASE_URL` | SQLite fallback | yes | Shared Postgres in prod |
| `WEB_PORT` | n/a (compose maps 8003) | `8003` | Host port for prod compose |
| `OLLAMA_BASE_URL` | `http://127.0.0.1:11434` | yes | GPU host in prod: `http://10.0.0.128:11434` |
| `OLLAMA_MODEL` | `gpt-oss:20b` | optional | Legacy fallback for THINKING |
| `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` | see defaults | optional | Role-scoped chat models (#62) |
| `OLLAMA_EMBED_MODEL` | `nomic-embed-text` | optional | Never falls back to a chat model |
| `OLLAMA_NUM_CTX_THINKING` / `_FAST` | `16384` / `8192` | optional | Context window per role |
| `ALLOW_INTERNET_ACCESS` | `true` | optional | Gate for live web retrieval |
| `SEARCH_PROVIDER` | `searxng` | optional | Primary search provider (#62) |
| `SEARCH_FAILOVER_PROVIDER` | `ddgs` | optional | Automatic failover |
| `SEARXNG_BASE_URL` | `http://127.0.0.1:8088` | yes if using SearxNG | Self-hosted SearxNG JSON API |
| `EMAIL_HOST_*` | empty | yes (prod/beta) | SMTP2GO |
| `CAPTCHA_SECRET_KEY` | empty | recommended | |
| `ENABLE_ACCOUNT_REGISTRATION` | `false` | optional | Self-serve sign-up; keep false until ready |
| `STRIPE_SECRET_KEY` / `STRIPE_PUBLISHABLE_KEY` / `STRIPE_WEBHOOK_SECRET` | empty | yes for billing | Stripe API + webhook |
| `STRIPE_PRICE_ID` | empty | optional | Pre-created Price; else `$10/mo` from settings |
| `GOOGLE_OAUTH_CLIENT_ID` / `..._SECRET` | empty | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `MICROSOFT_OAUTH_CLIENT_ID` / `..._SECRET` / `..._TENANT` | empty / `common` | for SSO/Drive | Also used for Drive linking (#47), incremental scopes |
| `FRONTEND_BASE_URL` | `http://localhost:3000` | set in prod/beta | Checkout success/cancel, portal return, OAuth return |
| `STRIPE_PORTAL_RETURN_URL` | `{FRONTEND}/account/` | optional | Stripe Customer Portal return URL |
| `CORS_ALLOWED_ORIGINS` | local + chat FE (+ beta FE default) | set in prod/beta | Frontend origin(s) |
| `USE_TLS_PROXY` | false (dev) | true behind NPM | Sets `SECURE_PROXY_SSL_HEADER` |
| `GUNICORN_WORKERS` / `GUNICORN_BIND` | 2 / `0.0.0.0:8000` | optional | Entrypoint |
| `SKIP_RAG_INIT` | unset | CI/migrate often `1` | Skip Chroma/Ollama boot work |

Assistant identity (`Hesychia`) lives in code:
`llm_be/chat_backend/services/assistant_identity.py` — prepended to user-facing
generation prompts (chat, RAG, data analysis). Not env-configurable.

Templates: `.env.example` (local), `.env.prod.example` (control-node secret).

Control-node secret paths (server-infra on ai-server-4080):

```text
~/Documents/secrets/chat_backend/chat_backend_prod.env
~/Documents/secrets/chat_backend/chat_backend_beta.env
```

Validate with:

```bash
./scripts/validate-env.sh ~/Documents/secrets/chat_backend/chat_backend_prod.env
./scripts/validate-env.sh ~/Documents/secrets/chat_backend/chat_backend_beta.env
```

If `DATABASE_URL` password contains `$`, escape each as `$$` for Compose.

## Ollama

All clients (`ollama.Client`, `OllamaLLM`, `OllamaEmbeddings`, `ChatOllama`) use
`OLLAMA_BASE_URL` — never hardcoded localhost in deployed code.

| Env | Typical URL |
|-----|-------------|
| Local (Ollama on same machine) | `http://127.0.0.1:11434` |
| prod / beta (containers on adama/roslin/ai-server) | `http://10.0.0.128:11434` |

Firewall / Ollama listen on ai-server-4080 must allow `10.0.0.0/24` → `:11434`.

### Role-scoped models (#62)

| Role | Setting | Default | Used for |
|------|---------|---------|----------|
| THINKING | `OLLAMA_MODEL_THINKING` | `gpt-oss:20b` | Default chat / grounded answers |
| FAST | `OLLAMA_MODEL_FAST` | `gemma4:latest` | FE `modelName=FAST` (smaller/faster — still grounded) |
| UTILITY | `OLLAMA_MODEL_UTILITY` | `llama3.2` | Classify / moderate / title / grounding decision |
| EMBED | `OLLAMA_EMBED_MODEL` | `nomic-embed-text` | Chroma embeddings |

After changing `OLLAMA_EMBED_MODEL`, rebuild the vector store (dimension change):

```bash
SKIP_RAG_INIT=1 uv run python manage.py reindex_embeddings
```

### SearxNG (web search)

Grounded chat uses a self-hosted **SearxNG** instance as the primary search
provider (`SEARCH_PROVIDER=searxng`), with DuckDuckGo (`ddgs`) as automatic
failover. Point `SEARXNG_BASE_URL` at the JSON API (no trailing path).

**Recommended: run SearxNG on the GPU/infra host next to Ollama**
(`10.0.0.128`), reachable from the chat_backend containers on the LAN.

Minimal compose snippet (add to `server-infra` or run on ai-server-4080):

```yaml
services:
  searxng:
    image: searxng/searxng:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng:rw
    environment:
      - SEARXNG_BASE_URL=http://10.0.0.128:8088/
```

In `searxng/settings.yml` (created on first start), enable the JSON format:

```yaml
search:
  formats:
    - html
    - json
```

Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:

```text
ALLOW_INTERNET_ACCESS=true
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```

Firewall: allow `10.0.0.0/24` → `:8080` on the SearxNG host (same pattern as
Ollama `:11434`). Verify from a backend container:

```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```

If SearxNG is down, chat still works for non-factual turns; factual turns that
require retrieval return an explicit "couldn't reach live sources" message
instead of hallucinating from parametric memory.

## File storage

Prompt attachments and workspace documents use **`DatabaseStorage`**
(`chat_backend.StoredFile` BinaryField in Postgres). Blobs are **not** written
to the container filesystem under `media/`.

RAG loaders that need a path materialize a short-lived temp file, then delete it.
Chroma’s vector index may still use a volume (`chroma_db`); that is embeddings
metadata, not the original upload.

## Production / beta (docker-compose.prod.yml)

- Single `web` service; **no** bundled DB — `DATABASE_URL` → shared Postgres (`10.0.0.230`).
- Host port from `WEB_PORT` (prod **8003**; beta **8013**).
- Entrypoint: wait DB → migrate → collectstatic → `gunicorn` + `UvicornWorker`
  (ASGI for HTTP **and** WebSockets).
- Active/active on **adama + roslin + ai-server-4080**; NPM balances upstreams.
- Manual / local deploy:

```bash
# beta (day-to-day)
~/Documents/repos/server-infra/scripts/deploy.sh \
  --app chat_backend --env beta --ref <sha>

# prod (intentional)
~/Documents/repos/server-infra/scripts/deploy.sh \
  --app chat_backend --env prod --ref <sha>
```

Beta hosts / CORS: `beta.chatbackend.aimloperations.com` API +
`https://beta.chat.aimloperations.com` SPA (see `.env.prod.example` beta block).
DB: `chat_backend_beta`. Pair with [server-infra#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7)
and frontend [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35).

## CI / CD (Gitea Actions)

| Workflow | Trigger | Action |
|----------|---------|--------|
| `unittests.yml` | push + PR → `master` | `uv sync` + `manage.py test` |
| `ci.yml` | PR → `master` | same unit tests |
| `deploy-beta.yml` | **push** to `master` | unit tests → docker compose tests → `deploy.sh --env beta` |
| `deploy-prod.yml` | **manual** `workflow_dispatch` only | unit tests → docker compose tests → `deploy.sh --env prod` |

Push/merge to `master` auto-deploys **beta** only. Prod requires the Gitea
**Run workflow** button on **Deploy Prod**. Deploy never runs on PRs.

## Frontend API notes

### Self-delete account ([#34](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/34))

| | |
|--|--|
| Method / path | `DELETE /api/user/` |
| Auth | JWT (authenticated user only; always deletes `request.user`) |
| Optional body | `{ "refresh_token": "<current refresh>" }` |
| Success | `200` `{ "detail": "Account deleted.", "deleted": true }` |
| Effects | Sets `deleted=True`, `is_active=False`; soft-deletes conversations; blacklists outstanding refresh tokens; logs `UserAuthEvent` `account_deleted` |
| Staff | Staff/superuser self-delete rejected (`400`, `code=staff_forbidden`) |
| Privacy v1 | Soft-delete only (no anonymization / hard purge) |

Post-delete UX: clear local tokens → redirect to sign-in. Subsequent
`/token/obtain/` fails. Do **not** send another user's id/email — ignored.

### Subscription change / cancel (portal + webhooks)

Plan change and cancel stay on Stripe Customer Portal
(`POST /api/finance/portal/`). Local state syncs via
`customer.subscription.updated` / `deleted` webhooks.
`GET /api/finance/subscription/` includes `cancel_at_period_end` and
`current_period_end` for Account UI messaging.

Subscription audit (`UserAuthEvent` on the user admin):
- `subscription_started` — first active plan (Checkout, Backer redeem, admin assign)
- `subscription_updated` — plan/status/cancel-at-period-end changes (portal + webhooks)

### Drive / RAG sync ([#47](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/47)-[#53](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/53))

Personal Google Drive / OneDrive and company Google Shared Drive / SharePoint
sync into the existing RAG `Document` pipeline. Personal connections work for
users **without** a company (personal workspace); company connections require a
company manager. Every endpoint below is gated
by `assert_feature_allowed(user, "rag")` (`SubscriptionPlan.allows_rag` —
true for Founders/Pro/Business/Backer, false for Standard by default).

**Connect (OAuth, reuses `#24` SSO app registrations with incremental scopes):**

| | |
|--|--|
| Personal | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_drive` (authenticated) |
| Company | `GET /api/auth/oauth/<google\|microsoft>/start/?intent=link_company_drive` (company manager only) |
| Callback | Same `/api/auth/oauth/<provider>/callback/` as SSO; the signed OAuth `state` carries the linking `user_id` since the browser has no session on the IdP redirect. Upserts a `DriveConnection` and redirects to `{FRONTEND_BASE_URL}/account/?drive_connected=1&provider=<provider>&kind=<personal\|company>` (or `?error=<code>`) |

**Manage:**

| Method / path | Notes |
|--|--|
| `GET /api/drive/connections/` | Caller's personal connections + their company's company connections |
| `DELETE /api/drive/connections/<id>/` | Disconnect (owner for personal, company manager for company) — deactivates + clears tokens, keeps history |
| `POST /api/drive/connections/<id>/resources/` | `{ "resource_ids": [...], "resource_labels": [...] }` — folder/shared-drive/site ids to sync; empty = provider root |
| `POST /api/drive/connections/<id>/sync/` | Enqueue sync now (`chat_backend/drive_tasks.py`) — returns **202** with `connection.last_sync_status=pending`; poll `GET /api/drive/connections/` for `ok` / `error` + `last_sync_error` |

**Provider scope differences:**
- Google: same `drive.readonly` scope for personal and company; company sync
  reads Shared Drives via `corpora=drive` + `supportsAllDrives`.
- Microsoft: personal uses `Files.Read`; company uses `Files.Read.All
  Sites.Read.All` and syncs SharePoint sites (`selected_resource_ids` = site ids).

**Workers / webhooks (#52, #57):**
- `python manage.py sync_drive_connections [--connection-id N]` — enqueue sync tasks (default).
- `python manage.py sync_drive_connections --sync-now` — run sync inline in this process (cron/debug).
- Django 6 `TASKS` (see `settings.py`): default `ImmediateBackend` runs tasks in-process; Sync now still returns 202 by dispatching on a background thread. Swap `TASKS` to a durable queue + worker for production scale.
- `POST /api/drive/webhooks/google/` / `POST /api/drive/webhooks/microsoft/` —
  provider push-notification stubs (`AllowAny`); acknowledge `200` and enqueue
  `sync_connection` when the notification's `connection_id` is resolvable,
  else just `200` (no-op). Microsoft's subscription-creation `validationToken`
  handshake is echoed back as `text/plain`.

Google-native Docs/Sheets/Slides are exported to `.docx`/`.xlsx`/`.pdf` before
ingest (Chroma/RAG loaders don't read the native formats). Documents whose
remote file was deleted upstream are removed on the next sync.

## Security note

Secrets previously hardcoded in `settings.py` (email password, captcha, Django
secret) must live only in the control-node env file. Rotate anything that was
ever committed; never commit `.env` or `~/Documents/secrets/`.