Fix SearxNG crash loop: make settings.yml readable inside the container.

Container runs as non-root; mode 0640 owned by westfarn caused
PermissionError on /etc/searxng/settings.yml and a restart loop.
Also lengthen the default secret_key to >= 32 chars.

IMPLEMENTATION.md

@@ -35,7 +35,7 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
 |------|-----|------|
 | adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
 | roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
-| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner (no app workloads) |
+| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
 
 Hostname on this machine: `ryan-development-1`
 
@@ -54,7 +54,7 @@ server-infra/
 │   └── host_vars/
 │       ├── adama.yml       # host_apps (django + dta_webapp)
 │       ├── roslin.yml      # host_apps (mirrors adama)
-│       └── ai-server-4080.yml  # control node / act runner, no workloads
+│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
 ├── playbooks/
 │   ├── site.yml            # Phase 1: provision
 │   └── deploy-apps.yml     # Phase 2: CI deploy
@@ -65,6 +65,9 @@ server-infra/
 │   ├── nodejs/             # Node.js + npm + npx (NodeSource)
 │   ├── gitea-key/          # per-server SSH key + Gitea access probe
 │   ├── tianji/             # Monitoring reporter
+│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
+│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
+│   ├── alloy/              # log/metrics shipper
 │   ├── app-deploy/         # django (docker) + node-static deploy
 │   └── web-static/         # nginx container serving /var/www builds
 └── scripts/
@@ -193,8 +196,8 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 | `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM |
 | `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.app.ditchtheagent/html`, served by web-static nginx |
 | `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
-| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx; beta port reserved |
+| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
-| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; Ollama via `OLLAMA_BASE_URL=http://10.0.0.128:11434`; beta port reserved |
+| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
 
 Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
 env file) so active/active replicas share one database. Beta and prod never share
@@ -218,9 +221,23 @@ future beta replica.
 | company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 |
 | dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 |
 | scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 |
-| chat_backend | 8013 (*not deployed*) | 8003 | adama, roslin, ai-server-4080 |
+| chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 |
 | dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
-| chat_web_app (nginx) | 8083 (*not deployed*) | 8082 | adama, roslin, ai-server-4080 |
+| chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 |
+| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
+
+Host-local services on ai-server-4080 (not balanced by NPM):
+
+| Service | Port | Notes |
+|---------|------|-------|
+| Ollama | 11434 | Not Ansible-managed today; GPU host |
+| SearxNG | 8088 | `roles/searxng` (#10); JSON API for chat_backend grounded search |
+| Loki | 3100 | `roles/observability` |
+| Prometheus | 9090 | `roles/observability` |
+| Grafana | 3000 | `roles/observability` |
+
+**Port clash warning:** do **not** bind SearxNG to `8080` — that is `dta_webapp` prod.
+chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.
 
 ### Flow
 
@@ -260,8 +277,8 @@ point each domain at the backend(s):
 | dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 |
 | dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin |
 | scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) |
-| chat_web_app | `chat.aimloperations.com` (+ www) | `adama:8082` + `roslin:8082` (+ `ai-server-4080:8082`) |
+| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / ai-server-4080 |
-| chat_backend | `chatbackend.aimloperations.com` | `adama:8003` + `roslin:8003` + `ai-server-4080:8003` |
+| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / ai-server-4080 |
 
 ### Required changes IN each app repo (owned separately)
 
@@ -369,8 +386,8 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
 | 9 | Stub `deploy-apps.yml` + update `company_site` workflow | Future |
 | 10 | Dockerize `company_site` | Future (separate ticket) |
 | 10a | Register + deploy `scha` (all webservers, port 8002) | In progress ([scha#19](https://git.aimloperations.com/ai_ml_operations/scha/issues/19)) |
-| 10b | Register + deploy `chat_web_app` (node-static, port 8082) | In progress ([chat_web_app#13](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/13)) |
+| 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
-| 10c | Register + deploy `chat_backend` (django, port 8003) | In progress ([chat_backend#6](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/6)) |
+| 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
 | 11 | Gitea container registry (optional) | Future |
 
 ## Open Decisions

inventory/host_vars/adama.yml

@@ -13,6 +13,6 @@ host_apps:
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
   - { name: chat_web_app, env: prod, port: 8082 }
-  # optional: - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
-  # optional: - { name: chat_backend, env: beta, port: 8013 }
+  - { name: chat_backend, env: beta, port: 8013 }

inventory/host_vars/ai-server-4080.yml

@@ -11,6 +11,11 @@ act_runner_enabled: true
 # Central Loki + Prometheus + Grafana (roles/observability).
 observability_stack: true
 
+# SearxNG JSON search API for chat_backend grounded retrieval (#10).
+# Host port 8088 — 8080 is dta_webapp on this host. chat_backend secret:
+#   SEARXNG_BASE_URL=http://10.0.0.128:8088
+searxng_stack: true
+
 # This host's pre-existing ~/.ssh/id_ed25519 is a personal key WITH a passphrase,
 # which hangs the (non-BatchMode) gitea access probe. Use a dedicated,
 # passphrase-less deploy key here instead.
@@ -26,6 +31,6 @@ host_apps:
   - { name: dta_service, env: beta, port: 8011 }
   - { name: scha, env: prod, port: 8002 }
   - { name: chat_web_app, env: prod, port: 8082 }
-  # optional: - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
-  # optional: - { name: chat_backend, env: beta, port: 8013 }
+  - { name: chat_backend, env: beta, port: 8013 }

inventory/host_vars/roslin.yml

@@ -10,6 +10,6 @@ host_apps:
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
   - { name: chat_web_app, env: prod, port: 8082 }
-  # optional: - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
-  # optional: - { name: chat_backend, env: beta, port: 8013 }
+  - { name: chat_backend, env: beta, port: 8013 }

playbooks/site.yml

@@ -21,6 +21,13 @@
     - role: observability
       when: observability_stack | default(false) | bool
 
+- name: Provision SearxNG (chat_backend grounded search)
+  hosts: ai-server-4080
+  become: true
+  roles:
+    - role: searxng
+      when: searxng_stack | default(false) | bool
+
 - name: Provision Alloy agents
   hosts: webservers
   become: true

roles/app-deploy/tasks/main.yml

@@ -24,6 +24,29 @@
   ansible.builtin.debug:
     msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
 
+# Explicit --app/--env with no host_apps match used to silently no-op Django
+# and still refresh web-static (looked like a "static-only" deploy). Fail loud.
+- name: Fail when requested app+env is not on this host
+  ansible.builtin.fail:
+    msg: >-
+      No host_apps entry for app={{ app }} env={{ app_env }} on
+      {{ inventory_hostname }}. Check inventory/host_vars and app_catalog
+      (django apps need type: django + compose_file). Catalog keys:
+      {{ app_catalog.keys() | list }}.
+  when:
+    - app is defined
+    - app_env is defined
+    - deploy_targets | length == 0
+
+- name: Fail when requested app missing from app_catalog
+  ansible.builtin.fail:
+    msg: >-
+      app={{ app }} is not in app_catalog. Add it with type: django (docker)
+      or type: node-static. Known: {{ app_catalog.keys() | list }}.
+  when:
+    - app is defined
+    - app not in app_catalog
+
 - name: Deploy Django (docker compose) apps
   ansible.builtin.include_tasks: django.yml
   loop: "{{ deploy_targets | selectattr('name', 'in', django_names) | list }}"

roles/searxng/defaults/main.yml

@@ -0,0 +1,20 @@
+---
+# SearxNG for chat_backend grounded web search (#10).
+# Hosted on ai-server-4080 only (next to Ollama). LAN-only — not NPM public.
+
+searxng_dir: "{{ apps_base_dir }}/searxng"
+searxng_image: "searxng/searxng:latest"
+
+# Host port 8088 — 8080 is already dta_webapp prod on ai-server-4080.
+searxng_host_port: 8088
+searxng_container_port: 8080
+
+# Public base URL as seen by chat_backend containers on the LAN.
+searxng_base_url: "http://{{ ansible_host }}:{{ searxng_host_port }}/"
+
+# Override via host_vars or vault; must be stable across restarts.
+# SearxNG expects a long random string (use >= 32 chars).
+searxng_secret_key: "change-me-searxng-secret-key-32chars"
+
+# LAN CIDR allowed to hit the JSON API (same pattern as observability).
+searxng_ufw_from: "{{ ufw_ssh_allowed_network }}"

roles/searxng/tasks/main.yml

@@ -0,0 +1,65 @@
+---
+# SearxNG JSON search API for chat_backend grounded retrieval.
+# Enabled on ai-server-4080 via searxng_stack: true (issue #10).
+
+- name: searxng | ensure project directory
+  ansible.builtin.file:
+    path: "{{ searxng_dir }}"
+    state: directory
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0755"
+
+- name: searxng | settings.yml
+  ansible.builtin.template:
+    src: settings.yml.j2
+    dest: "{{ searxng_dir }}/settings.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    # Container runs as non-root searxng UID — must be world-readable (#10).
+    mode: "0644"
+  register: _searxng_settings
+
+- name: searxng | compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ searxng_dir }}/docker-compose.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0644"
+  register: _searxng_compose
+
+- name: searxng | allow JSON API from LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ searxng_host_port }}"
+    proto: tcp
+    from_ip: "{{ searxng_ufw_from }}"
+    comment: SearxNG for chat_backend grounded search
+
+- name: searxng | start stack
+  ansible.builtin.command:
+    cmd: >-
+      docker compose up -d --remove-orphans
+      {{ '--force-recreate' if (
+        _searxng_compose is changed
+        or _searxng_settings is changed
+      ) else '' }}
+    chdir: "{{ searxng_dir }}"
+  become: true
+  become_user: "{{ admin_user }}"
+  register: _searxng_up
+  changed_when: >-
+    _searxng_up.rc == 0 and (
+      'Started' in (_searxng_up.stdout | default(''))
+      or 'Recreated' in (_searxng_up.stdout | default(''))
+      or 'Created' in (_searxng_up.stdout | default(''))
+      or _searxng_compose is changed
+      or _searxng_settings is changed
+    )
+  failed_when: _searxng_up.rc != 0
+
+- name: searxng | show compose failure output
+  ansible.builtin.debug:
+    msg: "{{ _searxng_up.stderr_lines | default(_searxng_up.stdout_lines) }}"
+  when: _searxng_up is failed

roles/searxng/templates/docker-compose.yml.j2

@@ -0,0 +1,23 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+services:
+  searxng:
+    image: {{ searxng_image }}
+    container_name: searxng
+    restart: unless-stopped
+    ports:
+      - "{{ searxng_host_port }}:{{ searxng_container_port }}"
+    volumes:
+      - ./settings.yml:/etc/searxng/settings.yml:rw
+    environment:
+      - SEARXNG_BASE_URL={{ searxng_base_url }}
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      - SETGID
+      - SETUID
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"

roles/searxng/templates/settings.yml.j2

@@ -0,0 +1,20 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+# Minimal overlay on SearxNG defaults — JSON format required by chat_backend (#62).
+
+use_default_settings: true
+
+server:
+  secret_key: "{{ searxng_secret_key }}"
+  limiter: false
+  image_proxy: false
+  port: {{ searxng_container_port }}
+  bind_address: "0.0.0.0"
+  base_url: "{{ searxng_base_url }}"
+
+search:
+  safe_search: 0
+  autocomplete: ""
+  default_lang: "en"
+  formats:
+    - html
+    - json

scripts/provision.sh

@@ -15,7 +15,8 @@ Provision server(s) with site.yml.
 
 Applies: common, ufw, docker, nodejs, gitea-key, tianji, alloy (every host).
 On ai-server-4080 also: observability (Loki + Prometheus + Grafana) when
-observability_stack is true in host_vars.
+observability_stack is true, and SearxNG when searxng_stack is true
+(chat_backend grounded search on :8088).
 
   HOST              Optional. Limit to one host: adama, roslin, or ai-server-4080.
                     Omit to run against all webservers.
@@ -31,7 +32,7 @@ Examples:
   $(basename "$0") adama --check    # dry run on adama only
   $(basename "$0") adama            # provision adama (includes Alloy)
   $(basename "$0") adama --ask-pass  # first SSH login before ssh-copy-id
-  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana
+  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana + SearxNG
   $(basename "$0")                  # provision all hosts
 EOF
 }