Deploy SearxNG on ai-server-4080 for chat_backend grounded search (#10) (#11)

## Summary
- Closes [#10](#10).
- Deploys **SearxNG** on **ai-server-4080** (`10.0.0.128`) for [chat_backend#62](ai_ml_operations/chat_backend#62) / [PR #65](ai_ml_operations/chat_backend#65) grounded search.
- New `roles/searxng/` (compose + JSON-enabled `settings.yml`), gated by `searxng_stack: true`, wired into `site.yml`.
- **Host port 8088** (not 8080 — that is `dta_webapp` on this host). UFW allows `10.0.0.0/24` → `8088/tcp` only.

## Ops after merge

```bash
./scripts/provision.sh ai-server-4080
# or targeted:
ansible-playbook playbooks/site.yml --limit ai-server-4080 --tags never  # full site play includes searxng when searxng_stack
```

Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:

```text
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```

Smoke test from any app host:

```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```

## Test plan
- [ ] Provision ai-server-4080; confirm `docker ps` shows `searxng`
- [ ] Confirm `:8088` responds with JSON; `:8080` still serves dta_webapp
- [ ] Confirm UFW rule is LAN-only
- [ ] From adama/roslin container network, curl SearxNG succeeds
- [ ] Update chat_backend secrets to `:8088` and redeploy betaReviewed-on: #11

IMPLEMENTATION.md

@@ -35,7 +35,7 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
 |------|-----|------|
 | adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
 | roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
-| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner (no app workloads) |
+| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
 
 Hostname on this machine: `ryan-development-1`
 
@@ -54,7 +54,7 @@ server-infra/
 │   └── host_vars/
 │       ├── adama.yml       # host_apps (django + dta_webapp)
 │       ├── roslin.yml      # host_apps (mirrors adama)
-│       └── ai-server-4080.yml  # control node / act runner, no workloads
+│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
 ├── playbooks/
 │   ├── site.yml            # Phase 1: provision
 │   └── deploy-apps.yml     # Phase 2: CI deploy
@@ -65,6 +65,9 @@ server-infra/
 │   ├── nodejs/             # Node.js + npm + npx (NodeSource)
 │   ├── gitea-key/          # per-server SSH key + Gitea access probe
 │   ├── tianji/             # Monitoring reporter
+│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
+│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
+│   ├── alloy/              # log/metrics shipper
 │   ├── app-deploy/         # django (docker) + node-static deploy
 │   └── web-static/         # nginx container serving /var/www builds
 └── scripts/
@@ -191,8 +194,10 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 |-----|------|-------|------|-------|
 | `company_site` | django (docker) | adama + roslin (+ ai-server-4080) | prod | active/active behind NPM; beta port reserved |
 | `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM |
-| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>_dta_webapp`, served by web-static nginx |
+| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.app.ditchtheagent/html`, served by web-static nginx |
 | `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
+| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
+| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
 
 Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
 env file) so active/active replicas share one database. Beta and prod never share
@@ -216,7 +221,23 @@ future beta replica.
 | company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 |
 | dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 |
 | scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 |
+| chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 |
 | dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
+| chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 |
+| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
+
+Host-local services on ai-server-4080 (not balanced by NPM):
+
+| Service | Port | Notes |
+|---------|------|-------|
+| Ollama | 11434 | Not Ansible-managed today; GPU host |
+| SearxNG | 8088 | `roles/searxng` (#10); JSON API for chat_backend grounded search |
+| Loki | 3100 | `roles/observability` |
+| Prometheus | 9090 | `roles/observability` |
+| Grafana | 3000 | `roles/observability` |
+
+**Port clash warning:** do **not** bind SearxNG to `8080` — that is `dta_webapp` prod.
+chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.
 
 ### Flow
 
@@ -235,9 +256,10 @@ future beta replica.
   to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
   `docker compose build` → `up -d` → migrate (run once, shared DB).
 - **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
-  (writes to `/var/www/<env>_dta_webapp`).
+  (writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.app.ditchtheagent/html`
-- **web-static** role: one nginx container per app host (adama + roslin) serving
+  or `/var/www/{env}.chat.aimloperations/html`).
-  the static roots on their ports; NPM balances across both.
+- **web-static** role: one nginx container per app host serving the static roots
+  on their ports (from `host_apps`); NPM balances across hosts.
 
 ### Reverse proxy / load balancing (NPM at 10.0.0.230)
 
@@ -255,6 +277,8 @@ point each domain at the backend(s):
 | dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 |
 | dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin |
 | scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) |
+| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / ai-server-4080 |
+| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / ai-server-4080 |
 
 ### Required changes IN each app repo (owned separately)
 
@@ -265,7 +289,11 @@ point each domain at the backend(s):
       with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
       (keep the test/docker jobs).
 - [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
-      `/var/www/beta_dta_webapp` / `/var/www/prod_dta_webapp`.
+      `/var/www/beta.app.ditchtheagent/html` / `/var/www/prod.app.ditchtheagent/html`.
+- [ ] `chat_web_app`: `npm run build:beta` / `build:prod` output to
+      `/var/www/beta.chat.aimloperations/html` / `/var/www/prod.chat.aimloperations/html`.
+
+Companion `chat_web_app` frontend is already registered in this infrastructure repo.
 
 ### Shared Postgres (10.0.0.230, same box as NPM)
 
@@ -281,6 +309,8 @@ do not).
 | dta_service | beta | `dta_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta` |
 | scha | prod | `scha` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha` |
 | scha | beta | `scha_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha_beta` |
+| chat_backend | prod | `chat_backend` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend` |
+| chat_backend | beta | `chat_backend_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend_beta` |
 
 Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 `listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
@@ -356,6 +386,8 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
 | 9 | Stub `deploy-apps.yml` + update `company_site` workflow | Future |
 | 10 | Dockerize `company_site` | Future (separate ticket) |
 | 10a | Register + deploy `scha` (all webservers, port 8002) | In progress ([scha#19](https://git.aimloperations.com/ai_ml_operations/scha/issues/19)) |
+| 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
+| 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
 | 11 | Gitea container registry (optional) | Future |
 
 ## Open Decisions

inventory/group_vars/all.yml

@@ -87,6 +87,22 @@ app_catalog:
     compose_file: docker-compose.prod.yml
     web_service: web
     migrate_cmd: "uv run python manage.py migrate --noinput"
+  chat_web_app:
+    type: node-static
+    repo: "{{ git_base_url }}/ai_ml_operations/chat_web_app.git"
+    default_branch: master
+    # package.json lives in this subdir of the checkout (npm runs here).
+    subdir: llm-fe
+    # Document root served by nginx and written by `npm run build:<env>`.
+    # {env} is replaced with the entry's env (beta/prod).
+    webroot_pattern: "/var/www/{env}.chat.aimloperations/html"
+  chat_backend:
+    type: django
+    repo: "{{ git_base_url }}/ai_ml_operations/chat_backend.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
 
 # Deploy filter vars. CI passes these; manual runs may leave them undefined
 # to (re)deploy every app listed in the host's host_apps.

inventory/host_vars/adama.yml

@@ -12,3 +12,7 @@ host_apps:
   - { name: dta_webapp, env: beta, port: 8081 }
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
+  - { name: chat_web_app, env: prod, port: 8082 }
+  - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_backend, env: prod, port: 8003 }
+  - { name: chat_backend, env: beta, port: 8013 }

inventory/host_vars/ai-server-4080.yml

@@ -11,6 +11,11 @@ act_runner_enabled: true
 # Central Loki + Prometheus + Grafana (roles/observability).
 observability_stack: true
 
+# SearxNG JSON search API for chat_backend grounded retrieval (#10).
+# Host port 8088 — 8080 is dta_webapp on this host. chat_backend secret:
+#   SEARXNG_BASE_URL=http://10.0.0.128:8088
+searxng_stack: true
+
 # This host's pre-existing ~/.ssh/id_ed25519 is a personal key WITH a passphrase,
 # which hangs the (non-BatchMode) gitea access probe. Use a dedicated,
 # passphrase-less deploy key here instead.
@@ -25,3 +30,7 @@ host_apps:
   - { name: dta_service, env: prod, port: 8001 }
   - { name: dta_service, env: beta, port: 8011 }
   - { name: scha, env: prod, port: 8002 }
+  - { name: chat_web_app, env: prod, port: 8082 }
+  - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_backend, env: prod, port: 8003 }
+  - { name: chat_backend, env: beta, port: 8013 }

inventory/host_vars/roslin.yml

@@ -9,3 +9,7 @@ host_apps:
   - { name: dta_webapp, env: beta, port: 8081 }
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
+  - { name: chat_web_app, env: prod, port: 8082 }
+  - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_backend, env: prod, port: 8003 }
+  - { name: chat_backend, env: beta, port: 8013 }

playbooks/site.yml

@@ -21,6 +21,13 @@
     - role: observability
       when: observability_stack | default(false) | bool
 
+- name: Provision SearxNG (chat_backend grounded search)
+  hosts: ai-server-4080
+  become: true
+  roles:
+    - role: searxng
+      when: searxng_stack | default(false) | bool
+
 - name: Provision Alloy agents
   hosts: webservers
   become: true

roles/app-deploy/tasks/main.yml

@@ -24,6 +24,29 @@
   ansible.builtin.debug:
     msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
 
+# Explicit --app/--env with no host_apps match used to silently no-op Django
+# and still refresh web-static (looked like a "static-only" deploy). Fail loud.
+- name: Fail when requested app+env is not on this host
+  ansible.builtin.fail:
+    msg: >-
+      No host_apps entry for app={{ app }} env={{ app_env }} on
+      {{ inventory_hostname }}. Check inventory/host_vars and app_catalog
+      (django apps need type: django + compose_file). Catalog keys:
+      {{ app_catalog.keys() | list }}.
+  when:
+    - app is defined
+    - app_env is defined
+    - deploy_targets | length == 0
+
+- name: Fail when requested app missing from app_catalog
+  ansible.builtin.fail:
+    msg: >-
+      app={{ app }} is not in app_catalog. Add it with type: django (docker)
+      or type: node-static. Known: {{ app_catalog.keys() | list }}.
+  when:
+    - app is defined
+    - app not in app_catalog
+
 - name: Deploy Django (docker compose) apps
   ansible.builtin.include_tasks: django.yml
   loop: "{{ deploy_targets | selectattr('name', 'in', django_names) | list }}"

roles/searxng/defaults/main.yml

@@ -0,0 +1,19 @@
+---
+# SearxNG for chat_backend grounded web search (#10).
+# Hosted on ai-server-4080 only (next to Ollama). LAN-only — not NPM public.
+
+searxng_dir: "{{ apps_base_dir }}/searxng"
+searxng_image: "searxng/searxng:latest"
+
+# Host port 8088 — 8080 is already dta_webapp prod on ai-server-4080.
+searxng_host_port: 8088
+searxng_container_port: 8080
+
+# Public base URL as seen by chat_backend containers on the LAN.
+searxng_base_url: "http://{{ ansible_host }}:{{ searxng_host_port }}/"
+
+# Override via host_vars or vault; must be stable across restarts.
+searxng_secret_key: "CHANGE_ME_SEARXNG_SECRET"
+
+# LAN CIDR allowed to hit the JSON API (same pattern as observability).
+searxng_ufw_from: "{{ ufw_ssh_allowed_network }}"

roles/searxng/tasks/main.yml

@@ -0,0 +1,64 @@
+---
+# SearxNG JSON search API for chat_backend grounded retrieval.
+# Enabled on ai-server-4080 via searxng_stack: true (issue #10).
+
+- name: searxng | ensure project directory
+  ansible.builtin.file:
+    path: "{{ searxng_dir }}"
+    state: directory
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0750"
+
+- name: searxng | settings.yml
+  ansible.builtin.template:
+    src: settings.yml.j2
+    dest: "{{ searxng_dir }}/settings.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0640"
+  register: _searxng_settings
+
+- name: searxng | compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ searxng_dir }}/docker-compose.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0644"
+  register: _searxng_compose
+
+- name: searxng | allow JSON API from LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ searxng_host_port }}"
+    proto: tcp
+    from_ip: "{{ searxng_ufw_from }}"
+    comment: SearxNG for chat_backend grounded search
+
+- name: searxng | start stack
+  ansible.builtin.command:
+    cmd: >-
+      docker compose up -d --remove-orphans
+      {{ '--force-recreate' if (
+        _searxng_compose is changed
+        or _searxng_settings is changed
+      ) else '' }}
+    chdir: "{{ searxng_dir }}"
+  become: true
+  become_user: "{{ admin_user }}"
+  register: _searxng_up
+  changed_when: >-
+    _searxng_up.rc == 0 and (
+      'Started' in (_searxng_up.stdout | default(''))
+      or 'Recreated' in (_searxng_up.stdout | default(''))
+      or 'Created' in (_searxng_up.stdout | default(''))
+      or _searxng_compose is changed
+      or _searxng_settings is changed
+    )
+  failed_when: _searxng_up.rc != 0
+
+- name: searxng | show compose failure output
+  ansible.builtin.debug:
+    msg: "{{ _searxng_up.stderr_lines | default(_searxng_up.stdout_lines) }}"
+  when: _searxng_up is failed

roles/searxng/templates/docker-compose.yml.j2

@@ -0,0 +1,23 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+services:
+  searxng:
+    image: {{ searxng_image }}
+    container_name: searxng
+    restart: unless-stopped
+    ports:
+      - "{{ searxng_host_port }}:{{ searxng_container_port }}"
+    volumes:
+      - ./settings.yml:/etc/searxng/settings.yml:rw
+    environment:
+      - SEARXNG_BASE_URL={{ searxng_base_url }}
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      - SETGID
+      - SETUID
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"

roles/searxng/templates/settings.yml.j2

@@ -0,0 +1,20 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+# Minimal overlay on SearxNG defaults — JSON format required by chat_backend (#62).
+
+use_default_settings: true
+
+server:
+  secret_key: "{{ searxng_secret_key }}"
+  limiter: false
+  image_proxy: false
+  port: {{ searxng_container_port }}
+  bind_address: "0.0.0.0"
+  base_url: "{{ searxng_base_url }}"
+
+search:
+  safe_search: 0
+  autocomplete: ""
+  default_lang: "en"
+  formats:
+    - html
+    - json

scripts/deploy.sh

@@ -17,7 +17,7 @@ Deploy applications with deploy-apps.yml.
   HOST              Optional. Limit to one host: adama, roslin, or ai-server-4080.
 
 Options:
-  --app NAME        App to deploy (company_site, dta_service, dta_webapp, scha)
+  --app NAME        App to deploy (company_site, dta_service, dta_webapp, scha, chat_web_app, chat_backend)
   --env ENV         Environment: beta or prod
   --ref REF         Git ref/sha to deploy (default: master)
   --check           Dry run

scripts/provision.sh

@@ -15,7 +15,8 @@ Provision server(s) with site.yml.
 
 Applies: common, ufw, docker, nodejs, gitea-key, tianji, alloy (every host).
 On ai-server-4080 also: observability (Loki + Prometheus + Grafana) when
-observability_stack is true in host_vars.
+observability_stack is true, and SearxNG when searxng_stack is true
+(chat_backend grounded search on :8088).
 
   HOST              Optional. Limit to one host: adama, roslin, or ai-server-4080.
                     Omit to run against all webservers.
@@ -31,7 +32,7 @@ Examples:
   $(basename "$0") adama --check    # dry run on adama only
   $(basename "$0") adama            # provision adama (includes Alloy)
   $(basename "$0") adama --ask-pass  # first SSH login before ssh-copy-id
-  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana
+  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana + SearxNG
   $(basename "$0")                  # provision all hosts
 EOF
 }