Deploy SearxNG on ai-server-4080 for chat_backend grounded search (#10) (#11)

## Summary
- Closes [#10](#10).
- Deploys **SearxNG** on **ai-server-4080** (`10.0.0.128`) for [chat_backend#62](ai_ml_operations/chat_backend#62) / [PR #65](ai_ml_operations/chat_backend#65) grounded search.
- New `roles/searxng/` (compose + JSON-enabled `settings.yml`), gated by `searxng_stack: true`, wired into `site.yml`.
- **Host port 8088** (not 8080 — that is `dta_webapp` on this host). UFW allows `10.0.0.0/24` → `8088/tcp` only.

## Ops after merge

```bash
./scripts/provision.sh ai-server-4080
# or targeted:
ansible-playbook playbooks/site.yml --limit ai-server-4080 --tags never  # full site play includes searxng when searxng_stack
```

Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:

```text
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```

Smoke test from any app host:

```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```

## Test plan
- [ ] Provision ai-server-4080; confirm `docker ps` shows `searxng`
- [ ] Confirm `:8088` responds with JSON; `:8080` still serves dta_webapp
- [ ] Confirm UFW rule is LAN-only
- [ ] From adama/roslin container network, curl SearxNG succeeds
- [ ] Update chat_backend secrets to `:8088` and redeploy betaReviewed-on: #11

IMPLEMENTATION.md

@@ -35,7 +35,7 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
 |------|-----|------|
 | adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
 | roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
-| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner (no app workloads) |
+| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
 
 Hostname on this machine: `ryan-development-1`
 
@@ -54,7 +54,7 @@ server-infra/
 │   └── host_vars/
 │       ├── adama.yml       # host_apps (django + dta_webapp)
 │       ├── roslin.yml      # host_apps (mirrors adama)
-│       └── ai-server-4080.yml  # control node / act runner, no workloads
+│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
 ├── playbooks/
 │   ├── site.yml            # Phase 1: provision
 │   └── deploy-apps.yml     # Phase 2: CI deploy
@@ -65,6 +65,9 @@ server-infra/
 │   ├── nodejs/             # Node.js + npm + npx (NodeSource)
 │   ├── gitea-key/          # per-server SSH key + Gitea access probe
 │   ├── tianji/             # Monitoring reporter
+│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
+│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
+│   ├── alloy/              # log/metrics shipper
 │   ├── app-deploy/         # django (docker) + node-static deploy
 │   └── web-static/         # nginx container serving /var/www builds
 └── scripts/
@@ -194,7 +197,7 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 | `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.app.ditchtheagent/html`, served by web-static nginx |
 | `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
 | `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
-| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama via `OLLAMA_BASE_URL=http://10.0.0.128:11434` |
+| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
 
 Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
 env file) so active/active replicas share one database. Beta and prod never share
@@ -221,6 +224,20 @@ future beta replica.
 | chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 |
 | dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
 | chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 |
+| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
+
+Host-local services on ai-server-4080 (not balanced by NPM):
+
+| Service | Port | Notes |
+|---------|------|-------|
+| Ollama | 11434 | Not Ansible-managed today; GPU host |
+| SearxNG | 8088 | `roles/searxng` (#10); JSON API for chat_backend grounded search |
+| Loki | 3100 | `roles/observability` |
+| Prometheus | 9090 | `roles/observability` |
+| Grafana | 3000 | `roles/observability` |
+
+**Port clash warning:** do **not** bind SearxNG to `8080` — that is `dta_webapp` prod.
+chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.
 
 ### Flow