Always-on grounded retrieval + role-scoped Ollama models (#62 Phases 1–3) (#65)

## Summary
- Closes Phases 1–3 of [#62](#62) (Phase 4 eval harness left for a follow-up).
- **Accuracy:** Retrieval is decided every turn (`GroundingDecider`, fails open). `FAST` no longer skips search — it only selects `OLLAMA_MODEL_FAST`. Search failures surface an explicit error instead of hallucinating from parametric memory.
- **Search:** Pluggable `services/search/` with **SearxNG primary** + DDGS failover, ranking/dedupe/rumour filtering, numbered dated source blocks, citations persisted on `Prompt.citations` and emitted as `{"v":1,"type":"citations",...}` after stream end.
- **Models:** Role-scoped `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` / `OLLAMA_EMBED_MODEL=nomic-embed-text`, configurable `num_ctx`, real model name on `PromptMetric`, `reindex_embeddings` management command + loud embedding-dimension mismatch.

## SearxNG (ops)
See README **SearxNG** section. Short version: run `searxng/searxng` on the GPU host, enable `json` in `settings.yml`, set `SEARXNG_BASE_URL=http://10.0.0.128:8080` in prod/beta secrets, open `:8080` on the LAN firewall like Ollama.

## Test plan
- [x] `SKIP_RAG_INIT=1 python manage.py test chat_backend.tests` — 442 OK (6 skipped)
- [ ] Deploy beta with updated secrets (`OLLAMA_MODEL_*`, `OLLAMA_EMBED_MODEL=nomic-embed-text`, `SEARXNG_BASE_URL`)
- [ ] After embed change: `python manage.py reindex_embeddings`
- [ ] Verify `did Taylor Swift get married` in FAST and THINKING returns grounded answer with citations frame
- [ ] Kill SearxNG and confirm factual turns return search_unavailable (not Joe Alwyn hallucination); non-factual chat still worksReviewed-on: #65

README.md

@@ -87,7 +87,14 @@ with `COMPOSE_DATABASE_URL` if needed.
 | `DATABASE_URL` | SQLite fallback | yes | Shared Postgres in prod |
 | `WEB_PORT` | n/a (compose maps 8003) | `8003` | Host port for prod compose |
 | `OLLAMA_BASE_URL` | `http://127.0.0.1:11434` | yes | GPU host in prod: `http://10.0.0.128:11434` |
-| `OLLAMA_MODEL` / `OLLAMA_EMBED_MODEL` | from `DEBUG` | optional | Override model names |
+| `OLLAMA_MODEL` | `gpt-oss:20b` | optional | Legacy fallback for THINKING |
+| `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` | see defaults | optional | Role-scoped chat models (#62) |
+| `OLLAMA_EMBED_MODEL` | `nomic-embed-text` | optional | Never falls back to a chat model |
+| `OLLAMA_NUM_CTX_THINKING` / `_FAST` | `16384` / `8192` | optional | Context window per role |
+| `ALLOW_INTERNET_ACCESS` | `true` | optional | Gate for live web retrieval |
+| `SEARCH_PROVIDER` | `searxng` | optional | Primary search provider (#62) |
+| `SEARCH_FAILOVER_PROVIDER` | `ddgs` | optional | Automatic failover |
+| `SEARXNG_BASE_URL` | `http://127.0.0.1:8088` | yes if using SearxNG | Self-hosted SearxNG JSON API |
 | `EMAIL_HOST_*` | empty | yes (prod/beta) | SMTP2GO |
 | `CAPTCHA_SECRET_KEY` | empty | recommended | |
 | `ENABLE_ACCOUNT_REGISTRATION` | `false` | optional | Self-serve sign-up; keep false until ready |
@@ -136,6 +143,74 @@ All clients (`ollama.Client`, `OllamaLLM`, `OllamaEmbeddings`, `ChatOllama`) use
 
 Firewall / Ollama listen on ai-server-4080 must allow `10.0.0.0/24` → `:11434`.
 
+### Role-scoped models (#62)
+
+| Role | Setting | Default | Used for |
+|------|---------|---------|----------|
+| THINKING | `OLLAMA_MODEL_THINKING` | `gpt-oss:20b` | Default chat / grounded answers |
+| FAST | `OLLAMA_MODEL_FAST` | `gemma4:latest` | FE `modelName=FAST` (smaller/faster — still grounded) |
+| UTILITY | `OLLAMA_MODEL_UTILITY` | `llama3.2` | Classify / moderate / title / grounding decision |
+| EMBED | `OLLAMA_EMBED_MODEL` | `nomic-embed-text` | Chroma embeddings |
+
+After changing `OLLAMA_EMBED_MODEL`, rebuild the vector store (dimension change):
+
+```bash
+SKIP_RAG_INIT=1 uv run python manage.py reindex_embeddings
+```
+
+### SearxNG (web search)
+
+Grounded chat uses a self-hosted **SearxNG** instance as the primary search
+provider (`SEARCH_PROVIDER=searxng`), with DuckDuckGo (`ddgs`) as automatic
+failover. Point `SEARXNG_BASE_URL` at the JSON API (no trailing path).
+
+**Recommended: run SearxNG on the GPU/infra host next to Ollama**
+(`10.0.0.128`), reachable from the chat_backend containers on the LAN.
+
+Minimal compose snippet (add to `server-infra` or run on ai-server-4080):
+
+```yaml
+services:
+  searxng:
+    image: searxng/searxng:latest
+    restart: unless-stopped
+    ports:
+      - "8080:8080"
+    volumes:
+      - ./searxng:/etc/searxng:rw
+    environment:
+      - SEARXNG_BASE_URL=http://10.0.0.128:8088/
+```
+
+In `searxng/settings.yml` (created on first start), enable the JSON format:
+
+```yaml
+search:
+  formats:
+    - html
+    - json
+```
+
+Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:
+
+```text
+ALLOW_INTERNET_ACCESS=true
+SEARCH_PROVIDER=searxng
+SEARCH_FAILOVER_PROVIDER=ddgs
+SEARXNG_BASE_URL=http://10.0.0.128:8088
+```
+
+Firewall: allow `10.0.0.0/24` → `:8080` on the SearxNG host (same pattern as
+Ollama `:11434`). Verify from a backend container:
+
+```bash
+curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
+```
+
+If SearxNG is down, chat still works for non-factual turns; factual turns that
+require retrieval return an explicit "couldn't reach live sources" message
+instead of hallucinating from parametric memory.
+
 ## File storage
 
 Prompt attachments and workspace documents use **`DatabaseStorage`**