Treat omitted host_apps.enabled as true (#36)

Closes #35.

## Summary
- `#33` used Jinja `rejectattr('enabled', 'equalto', false)` to skip stopped apps. That filter **requires** the key, so every `host_apps` row that omits `enabled` (the documented default) crashed deploy on all webservers before git/compose.
- Filter with `item.enabled | default(true)` instead, in `app-deploy` and `web-static`. `print_forge` (`enabled: false`) still skipped. `url_shortening_service` and every other omitted-key row deploy again.

## Test plan
- [ ] `ansible-playbook playbooks/deploy-apps.yml --syntax-check`
- [ ] Redeploy `url_shortening_service` beta: `deploy.sh --app url_shortening_service --env beta --ref <sha>`
- [ ] Confirm `print_forge` is still not started (no compose up on 8007/8019)

Reviewed-on: #36

IMPLEMENTATION.md

@@ -10,6 +10,8 @@ flowchart TB
         Control1["ai-server-4080\n(control node)"]
         Control1 -->|ansible-playbook site.yml| Adama
         Control1 -->|ansible-playbook site.yml| Roslin
+        Control1 -->|ansible-playbook site.yml| Starbuck
+        Control1 -->|ansible-playbook site.yml| Apollo
     end
 
     subgraph cicd ["CI/CD (every merge to master)"]
@@ -19,6 +21,8 @@ flowchart TB
         Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
         AnsibleDeploy --> Adama2["adama"]
         AnsibleDeploy --> Roslin2["roslin"]
+        AnsibleDeploy --> Starbuck2["starbuck"]
+        AnsibleDeploy --> Apollo2["apollo"]
     end
 ```
 
@@ -26,6 +30,7 @@ flowchart TB
 |----------|------|----------|---------------|
 | **Provision** | New VM, OS change, firewall, Docker install | `site.yml` | ai-server-4080 — run manually |
 | **Deploy** | Green unit tests on `master` | `deploy-apps.yml` | Gitea Act runner on ai-server-4080 |
+| **Stop** | Take a compose app down | `stop-apps.yml` (`scripts/stop.sh`) | ai-server-4080 — run manually |
 
 Both pipelines share the same inventory (`inventory/hosts.yml`).
 
@@ -35,7 +40,9 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
 |------|-----|------|
 | adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
 | roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
-| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner (no app workloads) |
+| starbuck | 10.0.0.44 | Ubuntu Server VM (Proxmox) — app host |
+| apollo | 10.0.0.7 | Ubuntu Server VM (Proxmox) — app host |
+| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
 
 Hostname on this machine: `ryan-development-1`
 
@@ -52,12 +59,15 @@ server-infra/
 │   ├── group_vars/
 │   │   └── all.yml         # vars + app_catalog
 │   └── host_vars/
-│       ├── adama.yml       # host_apps (django + dta_webapp)
+│       ├── adama.yml       # host_apps (django + static; monica worker)
-│       ├── roslin.yml      # host_apps (mirrors adama)
+│       ├── roslin.yml      # host_apps (mirrors adama, no worker)
-│       └── ai-server-4080.yml  # control node / act runner, no workloads
+│       ├── starbuck.yml    # host_apps (mirrors roslin)
+│       ├── apollo.yml      # host_apps (mirrors roslin)
+│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
 ├── playbooks/
 │   ├── site.yml            # Phase 1: provision
-│   └── deploy-apps.yml     # Phase 2: CI deploy
+│   ├── deploy-apps.yml     # Phase 2: CI deploy
+│   └── stop-apps.yml       # compose down one app+env
 ├── roles/
 │   ├── common/             # Base packages
 │   ├── ufw/                # Firewall
@@ -65,11 +75,15 @@ server-infra/
 │   ├── nodejs/             # Node.js + npm + npx (NodeSource)
 │   ├── gitea-key/          # per-server SSH key + Gitea access probe
 │   ├── tianji/             # Monitoring reporter
+│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
+│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
+│   ├── alloy/              # log/metrics shipper
 │   ├── app-deploy/         # django (docker) + node-static deploy
 │   └── web-static/         # nginx container serving /var/www builds
 └── scripts/
     ├── provision.sh        # Wrapper with --limit support
-    └── deploy.sh           # Wrapper for deploy playbook
+    ├── deploy.sh           # Wrapper for deploy playbook
+    └── stop.sh             # Wrapper for stop playbook (compose down)
 ```
 
 ## Prerequisites (One-Time Bootstrap)
@@ -81,11 +95,15 @@ Ansible needs SSH + sudo on each target before playbooks work.
    ```bash
    ssh-copy-id westfarn@10.0.0.77
    ssh-copy-id westfarn@10.0.0.176
+   ssh-copy-id westfarn@10.0.0.44
+   ssh-copy-id westfarn@10.0.0.7
    ```
 3. Confirm passwordless SSH:
    ```bash
    ssh westfarn@10.0.0.77
    ssh westfarn@10.0.0.176
+   ssh westfarn@10.0.0.44
+   ssh westfarn@10.0.0.7
    ```
 4. **First-time only** — grant passwordless sudo on each new host before the first
    `provision.sh` run. Ubuntu 26.04 ships `sudo-rs` by default; Ansible's
@@ -139,6 +157,8 @@ New hosts need the one-time passwordless sudo bootstrap in
 
 # Same for other hosts
 ./scripts/provision.sh roslin
+./scripts/provision.sh starbuck
+./scripts/provision.sh apollo
 ./scripts/provision.sh ai-server-4080
 ```
 
@@ -153,8 +173,22 @@ New hosts need the one-time passwordless sudo bootstrap in
 ```bash
 ./scripts/deploy.sh adama
 ./scripts/deploy.sh --check roslin
+./scripts/deploy.sh starbuck
+./scripts/deploy.sh apollo
 ```
 
+### Stop a compose app
+
+```bash
+./scripts/stop.sh --app print_forge --env prod
+./scripts/stop.sh --app print_forge --env beta
+./scripts/stop.sh adama --app chat_backend --env beta --check
+```
+
+`stop.sh` is `docker compose down` for one `--app` + `--env` (no `-v`). Checkouts and
+secrets stay. Next `deploy.sh` of that app+env starts it again unless `host_apps`
+sets `enabled: false`. Node-static apps have no compose project; stop.sh skips them.
+
 Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`.
 
 ## Phase 1: Provision (`site.yml`)
@@ -189,12 +223,21 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 
 | App | Type | Hosts | Envs | Notes |
 |-----|------|-------|------|-------|
-| `company_site` | django (docker) | adama + roslin (+ ai-server-4080) | prod | active/active behind NPM; beta port reserved |
+| `company_site` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
-| `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM |
+| `dta_service` | django (docker) | all webservers | beta + prod | active/active behind NPM |
-| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.app.ditchtheagent/html`, served by web-static nginx |
+| `dta_webapp` | node/vite static | all webservers | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx |
-| `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
+| `dta_blog` | node-static (Python SSG) | all webservers | beta + prod | active/active; built to `/var/www/<env>.blog.realpath.app/html`; **own hosts** (`blog.realpath.app` / `beta.blog.realpath.app`), not `realpath.app/blog`; Tianji ids baked at build |
-| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
+| `scha` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
-| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama via `OLLAMA_BASE_URL=http://10.0.0.128:11434` |
+| `chat_web_app` | node-static (CRA) | all webservers | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
+| `chat_backend` | django (docker) | all webservers | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
+| `monica_site` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
+| `url_shortening_service` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres; **no worker**. Two public hosts, same container: short domain (`GET /`, `GET /<code>` 302) and API host (`/api/links/`, Bearer required). |
+| `college_craft` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha` / `monica_site`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434`; Nominatim `http://10.0.0.128:8089`; prod `SITE_UNDER_CONSTRUCTION=true` until launch |
+| `print_forge` | django (docker) | all webservers | beta + prod | **stopped** (`enabled: false`, [#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)). Ports reserved. Re-enable + `deploy.sh` to bring back. Worker was adama-only. |
+| `abc_be` | django (docker) | all webservers | **beta only** | AI Benefits Coach API; no bundled Postgres; Ollama `http://10.0.0.128:11434`; shares DB `abc_be_beta` with `abc_worker` |
+| `abc_worker` | django-type compose (FastAPI) | all webservers | **beta only** | same Postgres as `abc_be`; `migrate_cmd: true`; WS on host port 8018 |
+| `abc_fe` | node-static (Vite) | all webservers | **beta only** | built to `/var/www/<env>.abc.aimloperations/html`; no prod `host_apps` row |
+| `livekit` | django-type compose (LiveKit SFU) | all webservers | **beta only** | Docker like `abc_worker`. Host network `:7880`. **Do not NPM-balance** until Redis — pick one upstream (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that host. Secrets `~/Documents/secrets/livekit/livekit_beta.env` |
 
 Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
 env file) so active/active replicas share one database. Beta and prod never share
@@ -204,8 +247,13 @@ a DB.
 
 - `app_catalog` (`group_vars/all.yml`) — how each app is built (repo, type, compose file, migrate cmd).
 - `host_apps` (`host_vars/<host>.yml`) — which app+env+port runs on that host.
+  Optional `compose_profiles: [worker]` activates docker compose profiles on that
+  host only (used for `monica_site` / `college_craft` / `print_forge` dj-queue singleton on adama).
+  Optional `enabled: false` skips deploy (CI included) but keeps the row for ports
+  and `stop.sh`. Omit the key (or `true`) to deploy as before. Do not use Jinja
+  `rejectattr('enabled')` — missing key is an error, not "enabled".
 - Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
-  Ports match across adama/roslin so NPM can balance `adama:PORT` + `roslin:PORT`.
+  Ports match across app hosts so NPM can balance `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT`.
 
 ### Ports
 
@@ -215,12 +263,36 @@ future beta replica.
 
 | App | beta | prod | Deployed on |
 |-----|------|------|-------------|
-| company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 |
+| company_site | 8010 (*not deployed*) | 8000 | all webservers |
-| dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 |
+| dta_service | 8011 | 8001 | all webservers |
-| scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 |
+| scha | 8012 (*not deployed*) | 8002 | all webservers |
-| chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 |
+| chat_backend | 8013 | 8003 | all webservers |
-| dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
+| monica_site | 8014 | 8004 | all webservers |
-| chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 |
+| url_shortening_service | 8015 | 8005 | all webservers |
+| college_craft | 8016 | 8006 | all webservers |
+| print_forge | **8019** (*stopped*) | **8007** (*stopped*) | all webservers (`enabled: false`) |
+| abc_be | **8017** | 8009 (*not deployed*) | all webservers |
+| abc_worker | **8018** | 8008 (*not deployed*) | all webservers |
+| dta_webapp (nginx) | 8081 | 8080 | all webservers |
+| chat_web_app (nginx) | 8083 | 8082 | all webservers |
+| abc_fe (nginx) | **8085** | 8084 (*not deployed*) | all webservers |
+| dta_blog (nginx) | **8087** | **8086** | all webservers |
+| livekit | **7880** | — | all webservers (signaling). UDP 3478 + 50000–60000 + TCP 7881 via router DNAT to the NPM upstream host |
+| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
+
+Host-local services on ai-server-4080 (not balanced by NPM):
+
+| Service | Port | Notes |
+|---------|------|-------|
+| Ollama | 11434 | Not Ansible-managed today; GPU host (`monica_site` / `college_craft` social drafting) |
+| Nominatim | 8089 | Not Ansible-managed today; LAN address autocomplete for `college_craft` / `print_forge` |
+| SearxNG | 8088 | `roles/searxng` (#10); JSON API for chat_backend grounded search |
+| Loki | 3100 | `roles/observability` |
+| Prometheus | 9090 | `roles/observability` |
+| Grafana | 3000 | `roles/observability` |
+
+**Port clash warning:** do **not** bind SearxNG to `8080` — that is `dta_webapp` prod.
+chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.
 
 ### Flow
 
@@ -231,18 +303,35 @@ future beta replica.
      --app company_site --env prod --ref "${{ gitea.sha }}"
    ```
 3. `deploy-apps.yml` runs against `webservers`; each host deploys only the
-   matching app+env from its `host_apps`.
+   matching **enabled** app+env from its `host_apps` (`enabled: false` is skipped).
+
+To stop a compose app without deleting inventory:
+
+```bash
+./scripts/stop.sh --app print_forge --env prod
+```
+
+Set `enabled: false` on that `host_apps` row so the next CI deploy cannot start it
+again. `stop.sh` still matches disabled rows.
 
 ### `app-deploy` role behavior
 
 - **django**: push per-app secret from control node `{{ secrets_dir }}/<app>/<app>_<env>.env`
   to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
-  `docker compose build` → `up -d` → migrate (run once, shared DB).
+  `docker compose build` → `up -d` (with `COMPOSE_PROFILES` from optional
+  `host_apps.compose_profiles`) → migrate (run once, shared DB).
+  **`monica_site` / `college_craft` / `print_forge` worker:** adama `host_apps` sets
+  `compose_profiles: [worker]` so deploy starts dj-queue with web. Other hosts
+  omit profiles (web only).
 - **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
-  (writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.app.ditchtheagent/html`
+  (writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.realpath.app/html`,
-  or `/var/www/{env}.chat.aimloperations/html`).
+  `/var/www/{env}.blog.realpath.app/html`, or `/var/www/{env}.chat.aimloperations/html`).
+  Optional catalog `error_page_404` (used by `dta_blog`) serves that file instead
+  of the SPA `/index.html` fallback.
 - **web-static** role: one nginx container per app host serving the static roots
-  on their ports (from `host_apps`); NPM balances across hosts.
+  on their ports (from `host_apps`); NPM balances across hosts. Before `compose up`,
+  removes any container currently publishing those host ports (`docker ps --filter
+  publish=<port>`) so leftovers cannot block the bind, then recreates web-static.
 
 ### Reverse proxy / load balancing (NPM at 10.0.0.230)
 
@@ -251,17 +340,27 @@ point each domain at the backend(s):
 
 - Single host: standard Proxy Host → `adama:PORT`.
 - Active/active: jc21 NPM's UI Proxy Host is single-target. To balance
-  adama+roslin you need the **Advanced** tab with a custom `upstream {}` block
+  app hosts you need the **Advanced** tab with a custom `upstream {}` block
   (or a real LB). Confirm this before relying on active/active.
 
 | App | Domains | Backends |
 |-----|---------|----------|
-| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` |
+| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` + `starbuck:8000` + `apollo:8000` |
-| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 |
+| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / starbuck / apollo / ai-server-4080 |
-| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin |
+| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin / starbuck / apollo |
-| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) |
+| dta_blog | `blog.realpath.app` (prod); `beta.blog.realpath.app` (beta). **Do not** reverse-proxy onto `realpath.app/blog` | `adama:8086` / `8087` + same on roslin / starbuck / apollo / ai-server-4080 |
-| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / ai-server-4080 |
+| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` + `starbuck:8002` + `apollo:8002` (+ `ai-server-4080:8002`) |
-| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / ai-server-4080 |
+| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / starbuck / apollo / ai-server-4080 |
+| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / starbuck / apollo / ai-server-4080 |
+| monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / starbuck / apollo / ai-server-4080 |
+| url_shortening_service (short) | `aiml.pw` and/or `cidinn.li` (pick when DNS is ready); `short-beta.aimloperations.com` (beta). Proxy `/` + `/[a-z0-9]{4,8}` only — 404 `/api/`, `/admin/`, `/debug/` | `adama:8005` / `8015` + same on roslin / starbuck / apollo / ai-server-4080 |
+| url_shortening_service (API) | `shortener.aimloperations.com`; `shortener-beta.aimloperations.com` (beta). Proxy `/api/` only — 404 `/admin/` | same ports as short host (one container) |
+| college_craft | `collegecraft.com` (+ www); `college-craft-preview.aimloperations.com` (beta) | `adama:8006` / `8016` + same on roslin / starbuck / apollo / ai-server-4080 |
+| print_forge | `printforgeprints.com` (+ www); `print-forge-preview.aimloperations.com` (beta) — **app stopped** (`enabled: false`). Disable NPM hosts or expect 502. | `:8007` / `:8019` reserved |
+| abc_fe | `beta.abc.aimloperations.com` (beta only) | `adama:8085` + same on roslin / starbuck / apollo / ai-server-4080 |
+| abc_be | `beta.abc.be.aimloperations.com` (beta only) | `adama:8017` + same on roslin / starbuck / apollo / ai-server-4080 |
+| abc_worker | `beta.abc.worker.aimloperations.com` (beta only; HTTP + WebSocket upgrade) | `adama:8018` + same on roslin / starbuck / apollo / ai-server-4080 |
+| livekit | `LIVEKIT_DOMAIN` (e.g. `livekit.aimloperations.com`) — NPM Proxy Host, WebSocket, **single upstream** | `adama:7880` (do not balance until Redis) |
 
 ### Required changes IN each app repo (owned separately)
 
@@ -272,7 +371,10 @@ point each domain at the backend(s):
       with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
       (keep the test/docker jobs).
 - [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
-      `/var/www/beta.app.ditchtheagent/html` / `/var/www/prod.app.ditchtheagent/html`.
+      `/var/www/beta.realpath.app/html` / `/var/www/prod.realpath.app/html`.
+- [ ] `dta_blog`: `npm run build:beta` / `build:prod` output to
+      `/var/www/beta.blog.realpath.app/html` / `/var/www/prod.blog.realpath.app/html`
+      (`python3 build.py --env <env>` copies `dist/`). Default branch is `main`.
 - [ ] `chat_web_app`: `npm run build:beta` / `build:prod` output to
       `/var/www/beta.chat.aimloperations/html` / `/var/www/prod.chat.aimloperations/html`.
 
@@ -294,11 +396,116 @@ do not).
 | scha | beta | `scha_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha_beta` |
 | chat_backend | prod | `chat_backend` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend` |
 | chat_backend | beta | `chat_backend_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend_beta` |
+| monica_site | prod | `monica_site` | `postgres://westfarn:<pw>@10.0.0.230:5432/monica_site` |
+| monica_site | beta | `monica_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/monica_site_beta` |
+| url_shortening_service | prod | `url_shortener` | `postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener` |
+| url_shortening_service | beta | `url_shortener_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener_beta` |
+| college_craft | prod | `college_craft` | `postgres://westfarn:<pw>@10.0.0.230:5432/college_craft` |
+| college_craft | beta | `college_craft_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/college_craft_beta` |
+| print_forge | prod | `print_forge` | `postgres://westfarn:<pw>@10.0.0.230:5432/print_forge` |
+| print_forge | beta | `print_forge_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/print_forge_beta` |
+| abc_be | beta | `abc_be_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/abc_be_beta` |
+| abc_worker | beta | *(same `abc_be_beta`)* | worker uses `DB_*` pointing at `abc_be_beta` — do **not** create a second DB |
 
 Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 `listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
 5432 to `10.0.0.0/24` only.
 
+`url_shortening_service` extra env (control-node secrets, not in git):
+`SHORT_PUBLIC_HOSTS` / `SHORT_API_HOSTS` / `SHORT_ADMIN_HOSTS` (admin = `localhost,127.0.0.1` only),
+`SHORTENER_API_TOKENS` (`monica:<token>`), `SHORT_ALLOWED_HOSTS`, `CLICK_IP_PEPPER`
+(distinct from `DJANGO_SECRET_KEY`). `DJANGO_DEBUG=false` for prod and beta.
+Caller `monica_site` uses `SHORTENER_BASE_URL=https://<api-host>` and Bearer mint;
+do not mint via the short hostname.
+
+`college_craft` extra env (control-node secrets, not in git):
+`~/Documents/secrets/college_craft/college_craft_prod.env` and
+`college_craft_beta.env`. Template: app repo `.env.prod.example`.
+`DJANGO_ENV` / `DJANGO_ALLOWED_HOSTS` / `DATABASE_URL` / `WEB_PORT` /
+`PUBLIC_SITE_URL` differ per env (prod `collegecraft.com` / `:8006`;
+beta `college-craft-preview.aimloperations.com` / `:8016`).
+`SITE_UNDER_CONSTRUCTION=true` on prod (holding page until launch), `false` on beta.
+`FEATURE_BLOG=true`; other `FEATURE_*` stay false until purchased. Prod also needs
+`TIANJI_WEBSITE_ID` + reCAPTCHA keys. If email/SMS or direct mail is turned on later,
+add a `college_craft:<token>` entry to `SHORTENER_API_TOKENS` on `url_shortening_service`.
+
+App-repo companion (`college_craft`, not this repo): deploy workflows still call
+`--app client_site` (template leftover). They must become `--app college_craft`
+before CI deploy will hit this catalog entry. Default branch is `master` (not `main`).
+
+`print_forge` extra env (control-node secrets, not in git) — **currently stopped**
+([#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)):
+`host_apps` rows stay with `enabled: false`. Bring back with `enabled` removed/true
+then `deploy.sh`. Secrets kept:
+`~/Documents/secrets/print_forge/print_forge_prod.env` and
+`print_forge_beta.env`. Template: app repo `.env.prod.example`.
+`DJANGO_ENV` / `DJANGO_ALLOWED_HOSTS` / `DATABASE_URL` / `WEB_PORT` /
+`PUBLIC_SITE_URL` differ per env (prod `printforgeprints.com` / `:8007`;
+beta `print-forge-preview.aimloperations.com` / `:8019`).
+`SITE_UNDER_CONSTRUCTION=true` on prod (holding page until launch), `false` on beta.
+Purchased flags: `FEATURE_EMAIL_SMS`, `FEATURE_PAYMENTS`, `FEATURE_SHOP`,
+`FEATURE_SHIPPING`. Other `FEATURE_*` stay false. Add a `print_forge:<token>`
+entry to `SHORTENER_API_TOKENS` on `url_shortening_service` (beta token first).
+`SHORTENER_BASE_URL` is prod shortener on prod, `https://shortener-beta.aimloperations.com`
+on beta. Default branch is `master`.
+
+Companion app ticket: [print_forge#1](https://git.aimloperations.com/ai_ml_operations/print_forge/issues/1)
+([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)).
+App Gitea workflow still calls `--app print_forge` on `master`; deploy **no-ops**
+while `enabled: false`.
+
+`abc_be` / `abc_worker` / `abc_fe` (GIS org, not this repo) — **beta only** ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)):
+
+Control-node secrets (never git, mode `600`):
+
+```text
+~/Documents/secrets/abc_be/abc_be_beta.env
+~/Documents/secrets/abc_worker/abc_worker_beta.env
+```
+
+Templates: `GIS/abc_be` `.env.beta.example`, `GIS/abc_worker` `.env.beta.example`.
+`abc_fe` has no secret file — public `VITE_*` URLs live in committed `.env.beta`.
+Create Postgres DB `abc_be_beta` and grant `westfarn` before first deploy.
+NPM: `beta.abc.aimloperations.com` → `:8085`, `beta.abc.be.aimloperations.com` → `:8017`,
+`beta.abc.worker.aimloperations.com` → `:8018` (enable WebSocket). No prod ABC
+`host_apps` rows; ports **8009** / 8008 / 8084 reserved (`print_forge` took prod **8007**).
+
+Companion workflows: [abc_be#22](https://git.aimloperations.com/GIS/abc_be/issues/22),
+[abc_worker#27](https://git.aimloperations.com/GIS/abc_worker/issues/27),
+[abc_fe#27](https://git.aimloperations.com/GIS/abc_fe/issues/27).
+
+`livekit` (GIS org, SFU on webservers — [abc_worker#14](https://git.aimloperations.com/GIS/abc_worker/issues/14)–[#17](https://git.aimloperations.com/GIS/abc_worker/issues/17)):
+
+Control-node secret (never git, mode `600`):
+
+```text
+~/Documents/secrets/livekit/livekit_beta.env
+```
+
+Template: `GIS/livekit` `.env.beta.example`. Same `LIVEKIT_API_KEY` /
+`LIVEKIT_API_SECRET` pair as `abc_be` / `abc_worker`; worker `LIVEKIT_URL=wss://<LIVEKIT_DOMAIN>`.
+
+Deploy: `./scripts/deploy.sh --app livekit --env beta` (all webservers, like
+`abc_worker`). NPM: `LIVEKIT_DOMAIN` → **one** host `:7880` with WebSocket
+(adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that same host.
+Do not active/active-balance LiveKit until Redis is in the compose.
+
+`dta_blog` ([#29](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/29)) —
+Python SSG, **no secrets file**. `npm run build:<env>` writes
+`/var/www/<env>.blog.realpath.app/html`. Prod (`:8086`) omits `demo: true` seed
+posts; beta (`:8087`) includes them. Header/footer links are baked at build
+(`https://realpath.app` vs `https://beta.realpath.app`); Tianji website ids
+are likewise baked (`cmtvvmf562afjzqumwt1yh2y8` / `cmtvvn6z62agdzqumtk8xijpy`).
+Default branch is `main`.
+
+NPM / DNS / Cloudflare (Ansible does not manage these): `blog.realpath.app` →
+`:8086`, `beta.blog.realpath.app` → `:8087`, TLS like the other RealPath names.
+Do **not** mount this on `realpath.app/blog`. Static ports stay LAN / NPM-only
+(UFW does not world-open 8086/8087).
+
+Companion app ticket: [dta_blog#1](https://git.aimloperations.com/Ditch_The_Agent/dta_blog/issues/1).
+Gitea deploy can later call `./scripts/deploy.sh --app dta_blog --env beta|prod`.
+
 ### One-time host bootstrap (per target)
 
 - [x] Gitea SSH key: the `gitea-key` role (in `site.yml`) generates a key per
@@ -309,7 +516,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
       (default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL`
       (see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port
       table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on
-      adama + roslin. Never committed to git.
+      adama + roslin + starbuck + apollo. Never committed to git.
 - [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs`
       role in `site.yml` (NodeSource, `node_major` default 20).
 
@@ -318,7 +525,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 **Recommended:** Single self-hosted runner on ai-server-4080.
 
 - One orchestration point.
-- App hosts (adama/roslin) run the workloads; no runner needed on them for deploy fan-out.
+- App hosts (adama/roslin/starbuck/apollo) run the workloads; no runner needed on them for deploy fan-out.
 - Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).
 
 ### Runner requirements on ai-server-4080
@@ -327,7 +534,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 |-------------|-----|
 | Ansible | Run `deploy-apps.yml` |
 | `server-infra` checkout | Playbooks + inventory |
-| SSH key to adama + roslin | Deploy fan-out |
+| SSH key to app hosts | Deploy fan-out |
 
 On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml`
 fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act
@@ -358,8 +565,8 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
 | # | Task | Status |
 |---|------|--------|
 | 1 | Create `server-infra` repo | Done |
-| 2 | Inventory with all 3 hosts | Done |
+| 2 | Inventory with all 5 hosts | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
-| 3 | Bootstrap SSH to adama + roslin | Manual |
+| 3 | Bootstrap SSH to app hosts | Manual |
 | 4 | `site.yml` → common, ufw, docker | Done |
 | 5 | Verify `ansible webservers -m ping` | Manual |
 | 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
@@ -371,12 +578,21 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
 | 10a | Register + deploy `scha` (all webservers, port 8002) | In progress ([scha#19](https://git.aimloperations.com/ai_ml_operations/scha/issues/19)) |
 | 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
 | 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
+| 10d | Register + deploy `monica_site` (django, ports 8004/8014) | Done ([#14](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/14)) |
+| 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | Done ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) |
+| 10f | Add starbuck + apollo as app hosts (same workloads as roslin) | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
+| 10g | Register + deploy `url_shortening_service` (django, ports 8005/8015) | Done ([#22](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/22)) |
+| 10h | Register + deploy `college_craft` (django, ports 8006/8016) | Done ([#24](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/24)) |
+| 10i | Register + deploy ABC beta (`abc_be` 8017, `abc_worker` 8018, `abc_fe` 8085) | Done ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)) |
+| 10j | Register + deploy `print_forge` (django, ports 8007/8019) | Done ([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)) |
+| 10k | Register + deploy `dta_blog` (node-static, ports 8086/8087) | Done ([#29](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/29)) |
+| 10l | Register GIS LiveKit SFU (`abc_worker`-style docker on webservers, port 7880) | This PR |
 | 11 | Gitea container registry (optional) | Future |
 
 ## Open Decisions
 
 1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
-2. **NPM load balancing** — confirm jc21 NPM can express adama+roslin upstreams (Advanced tab), else active/active is just two independent instances.
+2. **NPM load balancing** — confirm jc21 NPM can express all app-host upstreams (Advanced tab), else active/active is just independent instances.
 3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`).
 
 ## Adding a New VM

README.md

@@ -11,6 +11,8 @@ ansible-galaxy collection install -r requirements.yml
 # Bootstrap SSH key to each host (one-time, before Ansible)
 ssh-copy-id westfarn@10.0.0.77
 ssh-copy-id westfarn@10.0.0.176
+ssh-copy-id westfarn@10.0.0.44
+ssh-copy-id westfarn@10.0.0.7
 
 # First-time only: passwordless sudo on each new host (before first provision)
 ssh -t westfarn@10.0.0.176   # repeat for each host IP
@@ -31,6 +33,10 @@ ansible adama -m ping
 
 # Provision all hosts
 ./scripts/provision.sh
+
+# Stop a compose app (does not remove volumes/checkouts)
+./scripts/stop.sh --app print_forge --env prod
+./scripts/stop.sh --app print_forge --env beta
 ```
 
 See [IMPLEMENTATION.md](IMPLEMENTATION.md) for full architecture, CI/CD plan, and phase breakdown.
@@ -46,4 +52,6 @@ the central **Loki / Prometheus / Grafana** stack (`observability_stack: true`).
 |------|-----|------|
 | adama | 10.0.0.77 | app host |
 | roslin | 10.0.0.176 | app host |
+| starbuck | 10.0.0.44 | app host |
+| apollo | 10.0.0.7 | app host |
 | ai-server-4080 | 10.0.0.128 | control node + act runner |

docs/GRAFANA_USAGE.md

@@ -45,7 +45,7 @@ first; use text search (`|=`, `|~`) on logs second.
 
 | Label | Meaning | Examples |
 |-------|---------|----------|
-| `host` | Inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `env` | Deploy environment | `beta`, `prod`, `host` (journal), `infra` (stack containers) |
 | `app` | App / service name | `company_site`, `dta_service`, `dta_webapp`, `system` |
 | `job` | Collector | `docker`, `systemd` |
@@ -56,7 +56,7 @@ first; use text search (`|=`, `|~`) on logs second.
 
 | Label | Meaning | Examples |
 |-------|---------|----------|
-| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | Scrape job | `node` (host), `cadvisor` (containers) |
 | `env` / `app` | Parsed from Compose project `<app>_<env>` | `prod` / `dta_service` |
 | `name` | Container name (cAdvisor) | `company_site_prod-web-1` |

docs/OBSERVABILITY.md

@@ -10,6 +10,8 @@ flowchart LR
   subgraph hosts ["All webservers"]
     A["adama\nAlloy"]
     R["roslin\nAlloy"]
+    S["starbuck\nAlloy"]
+    Ap["apollo\nAlloy"]
     C["ai-server-4080\nAlloy"]
   end
 
@@ -24,9 +26,13 @@ flowchart LR
 
   A -->|logs| L
   R -->|logs| L
+  S -->|logs| L
+  Ap -->|logs| L
   C -->|logs| L
   A -->|metrics| P
   R -->|metrics| P
+  S -->|metrics| P
+  Ap -->|metrics| P
   C -->|metrics| P
   L --> G
   P --> G
@@ -36,7 +42,7 @@ flowchart LR
 
 | Piece | Where | Role |
 |-------|--------|------|
-| **Alloy** | every host (`adama`, `roslin`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
+| **Alloy** | every host (`adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
 | **Loki** | `ai-server-4080` only | Store and index logs |
 | **Prometheus** | `ai-server-4080` only | Store metrics (CPU, RAM, disk, container health) |
 | **Grafana** | `ai-server-4080` only | Explore logs/metrics, dashboards, alerts |
@@ -182,7 +188,7 @@ services:
     user: "0:0"
     command: -config.file=/etc/loki/loki-config.yml
     ports:
-      # Bind to all interfaces so Alloy on adama/roslin can push.
+      # Bind to all interfaces so Alloy on app hosts can push.
       # Firewall (UFW) should restrict who can connect — see 1.6.
       - "3100:3100"
     volumes:
@@ -376,7 +382,7 @@ Alloy runs on **every** host in `webservers`. It:
 
 | Label | Source | Example values |
 |-------|--------|----------------|
-| `host` | Ansible inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Ansible inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | collector name | `systemd`, `docker` |
 | `env` | Docker Compose project suffix | `beta`, `prod`, `host`, `infra` |
 | `app` | Compose project prefix | `company_site`, `dta_service`, `dta_webapp`, … |
@@ -387,7 +393,7 @@ Alloy runs on **every** host in `webservers`. It:
 
 | Label | Source | Example values |
 |-------|--------|----------------|
-| `host` | Added by Alloy relabel | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Added by Alloy relabel | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | scrape job name | `node`, `cadvisor` |
 | `name` | container name (cAdvisor) | `company_site_prod-web-1` |
 | `container_label_com_docker_compose_project` | Compose project | `dta_service_prod` |
@@ -407,7 +413,7 @@ container_memory_usage_bytes{host="adama", env="prod", app="dta_service"}
 
 ### 3.1 Install Alloy (manual — one host)
 
-Repeat on `adama`, `roslin`, and `ai-server-4080`. Example for **adama**:
+Repeat on `adama`, `roslin`, `starbuck`, `apollo`, and `ai-server-4080`. Example for **adama**:
 
 ```bash
 sudo mkdir -p /opt/apps/observability/alloy
@@ -415,7 +421,7 @@ sudo chown -R westfarn:westfarn /opt/apps/observability
 ```
 
 Create `/opt/apps/observability/alloy/config.alloy`. **Change every
-`host = "adama"`** on each machine (`adama` / `roslin` / `ai-server-4080`):
+`host = "adama"`** on each machine (`adama` / `roslin` / `starbuck` / `apollo` / `ai-server-4080`):
 
 ```river
 // Grafana Alloy — logs → Loki, metrics → Prometheus.
@@ -705,6 +711,8 @@ Healthy Alloy logs mention connecting / sending without repeated
 ```logql
 {host="adama"}
 {host="roslin"}
+{host="starbuck"}
+{host="apollo"}
 {host="ai-server-4080"}
 ```
 
@@ -816,6 +824,8 @@ observability_stack: true
 ./scripts/provision.sh ai-server-4080
 ./scripts/provision.sh adama
 ./scripts/provision.sh roslin
+./scripts/provision.sh starbuck
+./scripts/provision.sh apollo
 # or all (site.yml orders stack before Alloy):
 ./scripts/provision.sh
 ```
@@ -916,7 +926,7 @@ du -sh /opt/apps/observability/loki/data \
 - [ ] `GF_SERVER_ROOT_URL` matches public URL
 - [ ] Both Loki and Prometheus datasources green in Grafana
 
-### Alloy (each of adama, roslin, ai-server-4080)
+### Alloy (each of adama, roslin, starbuck, apollo, ai-server-4080)
 
 - [ ] `config.alloy` has correct `host = "..."` (or Ansible `inventory_hostname`)
 - [ ] Alloy container running privileged with host `/proc` `/sys` mounts

inventory/group_vars/all.yml

@@ -12,6 +12,15 @@ ufw_allowed_tcp_ports:
   - 80
   - 443
 
+# LiveKit on webservers. Signaling stays LAN (NPM → :7880). UDP must be
+# reachable from the internet via router DNAT to the NPM upstream host.
+ufw_lan_tcp_ports:
+  - 7880
+  - 7881
+ufw_extra_udp_ports:
+  - 3478
+  - "50000:60000"
+
 # Docker
 docker_users:
   - "{{ admin_user }}"
@@ -50,6 +59,7 @@ secrets_dir: "{{ lookup('ansible.builtin.env', 'HOME') }}/Documents/secrets"
 web_static_root: /var/www
 
 # Catalog of deployable apps. host_apps (per host_vars) references these by name.
+# Optional host_apps.enabled: false skips deploy (stop.sh still compose-downs).
 app_catalog:
   company_site:
     type: django
@@ -77,9 +87,18 @@ app_catalog:
     subdir: ditch-the-agent
     # Document root served by nginx and written by `npm run build:<env>`.
     # {env} is replaced with the entry's env (beta/prod).
-    webroot_pattern: "/var/www/{env}.app.ditchtheagent/html"
+    webroot_pattern: "/var/www/{env}.realpath.app/html"
     # deploy runs `npm ci` then `npm run build:<env>`; that script writes to
     # {{ web_static_root }}/<env>_dta_webapp (beta/prod), served by web-static.
+  dta_blog:
+    type: node-static
+    repo: "{{ git_base_url }}/Ditch_The_Agent/dta_blog.git"
+    default_branch: main
+    # Python SSG (stdlib). package.json build:<env> copies dist/ to this webroot
+    # so the existing node-static path (`npm ci` then `npm run build:<env>`) works.
+    webroot_pattern: "/var/www/{env}.blog.realpath.app/html"
+    # Real 404 page (not SPA index fallback). See roles/web-static nginx.conf.j2.
+    error_page_404: /404.html
   scha:
     type: django
     repo: "{{ git_base_url }}/ai_ml_operations/scha.git"
@@ -103,6 +122,69 @@ app_catalog:
     compose_file: docker-compose.prod.yml
     web_service: web
     migrate_cmd: "uv run python manage.py migrate --noinput"
+  monica_site:
+    type: django
+    repo: "{{ git_base_url }}/ai_ml_operations/monica_site.git"
+    default_branch: main
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
+    # dj-queue worker = compose profile `worker` (singleton). Deploy starts it
+    # via host_apps.compose_profiles on adama only (see host_vars/adama.yml).
+  url_shortening_service:
+    type: django
+    repo: "{{ git_base_url }}/ai_ml_operations/url_shortening_service.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
+    # No compose worker profile. Two public NPM hosts share one container/port.
+  college_craft:
+    type: django
+    repo: "{{ git_base_url }}/ai_ml_operations/college_craft.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
+    # dj-queue worker = compose profile `worker` (singleton). Deploy starts it
+    # via host_apps.compose_profiles on adama only (see host_vars/adama.yml).
+  print_forge:
+    type: django
+    repo: "{{ git_base_url }}/ai_ml_operations/print_forge.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
+    # dj-queue worker = compose profile `worker` (singleton). Deploy starts it
+    # via host_apps.compose_profiles on adama only (see host_vars/adama.yml).
+  abc_be:
+    type: django
+    repo: "{{ git_base_url }}/GIS/abc_be.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    migrate_cmd: "uv run python manage.py migrate --noinput"
+  abc_worker:
+    type: django
+    repo: "{{ git_base_url }}/GIS/abc_worker.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    # FastAPI — no Django migrations. `true` keeps the django deploy path happy.
+    migrate_cmd: "true"
+  abc_fe:
+    type: node-static
+    repo: "{{ git_base_url }}/GIS/abc_fe.git"
+    default_branch: master
+    webroot_pattern: "/var/www/{env}.abc.aimloperations/html"
+  livekit:
+    type: django
+    repo: "{{ git_base_url }}/GIS/livekit.git"
+    default_branch: master
+    compose_file: docker-compose.prod.yml
+    web_service: web
+    # SFU — no Django migrations. `true` keeps the django deploy path happy.
+    migrate_cmd: "true"
 
 # Deploy filter vars. CI passes these; manual runs may leave them undefined
 # to (re)deploy every app listed in the host's host_apps.

inventory/host_vars/adama.yml

@@ -1,7 +1,8 @@
 ---
-# Django services run active/active here and on roslin (shared external DB).
+# Django services run active/active here and on roslin/starbuck/apollo
-# dta_webapp static also runs active/active (built into /var/www, served by
+# (shared external DB). dta_webapp static also runs active/active (built into
-# the web-static nginx container). Ports MUST match roslin so NPM can balance.
+# /var/www, served by the web-static nginx container). Ports MUST match the
+# other app hosts so NPM can balance.
 # Each entry is one workload: django -> compose project <name>_<env> on port;
 # node-static -> /var/www/<env>_dta_webapp served on port.
 host_apps:
@@ -10,9 +11,25 @@ host_apps:
   - { name: dta_service, env: beta, port: 8011 }
   - { name: dta_webapp, env: prod, port: 8080 }
   - { name: dta_webapp, env: beta, port: 8081 }
+  - { name: dta_blog, env: prod, port: 8086 }
+  - { name: dta_blog, env: beta, port: 8087 }
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
   - { name: chat_web_app, env: prod, port: 8082 }
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
+  # compose_profiles: worker → dj-queue singleton (not on other app hosts).
+  - { name: monica_site, env: prod, port: 8004, compose_profiles: [worker] }
+  - { name: monica_site, env: beta, port: 8014, compose_profiles: [worker] }
+  - { name: url_shortening_service, env: prod, port: 8005 }
+  - { name: url_shortening_service, env: beta, port: 8015 }
+  - { name: college_craft, env: prod, port: 8006, compose_profiles: [worker] }
+  - { name: college_craft, env: beta, port: 8016, compose_profiles: [worker] }
+  # print_forge stopped (#33) — enabled: false keeps ports; stop.sh still matches.
+  - { name: print_forge, env: prod, port: 8007, compose_profiles: [worker], enabled: false }
+  - { name: print_forge, env: beta, port: 8019, compose_profiles: [worker], enabled: false }
+  - { name: abc_be, env: beta, port: 8017 }
+  - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_fe, env: beta, port: 8085 }
+  - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/ai-server-4080.yml

@@ -11,17 +11,24 @@ act_runner_enabled: true
 # Central Loki + Prometheus + Grafana (roles/observability).
 observability_stack: true
 
+# SearxNG JSON search API for chat_backend grounded retrieval (#10).
+# Host port 8088 — 8080 is dta_webapp on this host. chat_backend secret:
+#   SEARXNG_BASE_URL=http://10.0.0.128:8088
+searxng_stack: true
+
 # This host's pre-existing ~/.ssh/id_ed25519 is a personal key WITH a passphrase,
 # which hangs the (non-BatchMode) gitea access probe. Use a dedicated,
 # passphrase-less deploy key here instead.
 gitea_key_path: "/home/{{ admin_user }}/.ssh/gitea_deploy"
 
 # company_site + dta_webapp + dta_service for side testing and active/active.
-# Ports MUST match adama/roslin so NPM can balance all three upstreams.
+# Ports MUST match adama/roslin/starbuck/apollo so NPM can balance all upstreams.
 host_apps:
   - { name: company_site, env: prod, port: 8000 }
   - { name: dta_webapp, env: prod, port: 8080 }
   - { name: dta_webapp, env: beta, port: 8081 }
+  - { name: dta_blog, env: prod, port: 8086 }
+  - { name: dta_blog, env: beta, port: 8087 }
   - { name: dta_service, env: prod, port: 8001 }
   - { name: dta_service, env: beta, port: 8011 }
   - { name: scha, env: prod, port: 8002 }
@@ -29,3 +36,16 @@ host_apps:
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
+  - { name: monica_site, env: prod, port: 8004 }
+  - { name: monica_site, env: beta, port: 8014 }
+  - { name: url_shortening_service, env: prod, port: 8005 }
+  - { name: url_shortening_service, env: beta, port: 8015 }
+  - { name: college_craft, env: prod, port: 8006 }
+  - { name: college_craft, env: beta, port: 8016 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
+  - { name: abc_be, env: beta, port: 8017 }
+  - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_fe, env: beta, port: 8085 }
+  - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/apollo.yml

@@ -0,0 +1,30 @@
+---
+# Mirrors roslin for active/active. Ports MUST match adama/roslin so NPM
+# upstreams can balance apollo:PORT with the other app hosts.
+host_apps:
+  - { name: company_site, env: prod, port: 8000 }
+  - { name: dta_service, env: prod, port: 8001 }
+  - { name: dta_service, env: beta, port: 8011 }
+  - { name: dta_webapp, env: prod, port: 8080 }
+  - { name: dta_webapp, env: beta, port: 8081 }
+  - { name: dta_blog, env: prod, port: 8086 }
+  - { name: dta_blog, env: beta, port: 8087 }
+  - { name: scha, env: prod, port: 8002 }
+  # optional: - { name: scha, env: beta, port: 8012 }
+  - { name: chat_web_app, env: prod, port: 8082 }
+  - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_backend, env: prod, port: 8003 }
+  - { name: chat_backend, env: beta, port: 8013 }
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
+  - { name: monica_site, env: prod, port: 8004 }
+  - { name: monica_site, env: beta, port: 8014 }
+  - { name: url_shortening_service, env: prod, port: 8005 }
+  - { name: url_shortening_service, env: beta, port: 8015 }
+  - { name: college_craft, env: prod, port: 8006 }
+  - { name: college_craft, env: beta, port: 8016 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
+  - { name: abc_be, env: beta, port: 8017 }
+  - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_fe, env: beta, port: 8085 }
+  - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/roslin.yml

@@ -1,15 +1,30 @@
 ---
-# Mirrors adama for active/active. Ports MUST match adama so NPM upstreams
+# Mirrors adama for active/active. Ports MUST match adama/starbuck/apollo so
-# can balance adama:PORT and roslin:PORT for the same workload.
+# NPM upstreams can balance roslin:PORT with the other app hosts.
 host_apps:
   - { name: company_site, env: prod, port: 8000 }
   - { name: dta_service, env: prod, port: 8001 }
   - { name: dta_service, env: beta, port: 8011 }
   - { name: dta_webapp, env: prod, port: 8080 }
   - { name: dta_webapp, env: beta, port: 8081 }
+  - { name: dta_blog, env: prod, port: 8086 }
+  - { name: dta_blog, env: beta, port: 8087 }
   - { name: scha, env: prod, port: 8002 }
   # optional: - { name: scha, env: beta, port: 8012 }
   - { name: chat_web_app, env: prod, port: 8082 }
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
+  - { name: monica_site, env: prod, port: 8004 }
+  - { name: monica_site, env: beta, port: 8014 }
+  - { name: url_shortening_service, env: prod, port: 8005 }
+  - { name: url_shortening_service, env: beta, port: 8015 }
+  - { name: college_craft, env: prod, port: 8006 }
+  - { name: college_craft, env: beta, port: 8016 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
+  - { name: abc_be, env: beta, port: 8017 }
+  - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_fe, env: beta, port: 8085 }
+  - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/starbuck.yml

@@ -0,0 +1,30 @@
+---
+# Mirrors roslin for active/active. Ports MUST match adama/roslin so NPM
+# upstreams can balance starbuck:PORT with the other app hosts.
+host_apps:
+  - { name: company_site, env: prod, port: 8000 }
+  - { name: dta_service, env: prod, port: 8001 }
+  - { name: dta_service, env: beta, port: 8011 }
+  - { name: dta_webapp, env: prod, port: 8080 }
+  - { name: dta_webapp, env: beta, port: 8081 }
+  - { name: dta_blog, env: prod, port: 8086 }
+  - { name: dta_blog, env: beta, port: 8087 }
+  - { name: scha, env: prod, port: 8002 }
+  # optional: - { name: scha, env: beta, port: 8012 }
+  - { name: chat_web_app, env: prod, port: 8082 }
+  - { name: chat_web_app, env: beta, port: 8083 }
+  - { name: chat_backend, env: prod, port: 8003 }
+  - { name: chat_backend, env: beta, port: 8013 }
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
+  - { name: monica_site, env: prod, port: 8004 }
+  - { name: monica_site, env: beta, port: 8014 }
+  - { name: url_shortening_service, env: prod, port: 8005 }
+  - { name: url_shortening_service, env: beta, port: 8015 }
+  - { name: college_craft, env: prod, port: 8006 }
+  - { name: college_craft, env: beta, port: 8016 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
+  - { name: abc_be, env: beta, port: 8017 }
+  - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_fe, env: beta, port: 8085 }
+  - { name: livekit, env: beta, port: 7880 }

inventory/hosts.yml

@@ -7,5 +7,9 @@ all:
           ansible_host: 10.0.0.77
         roslin:
           ansible_host: 10.0.0.176
+        starbuck:
+          ansible_host: 10.0.0.44
+        apollo:
+          ansible_host: 10.0.0.7
         ai-server-4080:
           ansible_host: 10.0.0.128

playbooks/deploy-apps.yml

@@ -1,8 +1,9 @@
 ---
 # Phase 2: CI-triggered app deployment.
 #
-# Runs against all webservers; each host deploys only the apps listed in its
+# Runs against all webservers; each host deploys only enabled apps listed in
-# host_apps. Deploy exactly one app+env at a pinned ref via extra-vars:
+# its host_apps (enabled: false is skipped). Deploy exactly one app+env at a
+# pinned ref via extra-vars:
 #
 #   ansible-playbook playbooks/deploy-apps.yml \
 #     -e app=company_site -e app_env=prod -e app_ref=<sha>

playbooks/site.yml

@@ -21,6 +21,13 @@
     - role: observability
       when: observability_stack | default(false) | bool
 
+- name: Provision SearxNG (chat_backend grounded search)
+  hosts: ai-server-4080
+  become: true
+  roles:
+    - role: searxng
+      when: searxng_stack | default(false) | bool
+
 - name: Provision Alloy agents
   hosts: webservers
   become: true

playbooks/stop-apps.yml

@@ -0,0 +1,15 @@
+---
+# Stop one docker-compose app+env. Inverse of deploy-apps.yml start.
+#
+#   ansible-playbook playbooks/stop-apps.yml -e app=print_forge -e app_env=prod
+#
+# host_apps rows with enabled: false are still valid stop targets (so a
+# disabled app can be taken down). Node-static apps are skipped.
+- name: Stop applications
+  hosts: webservers
+  become: true
+  tasks:
+    - name: Stop requested compose apps
+      ansible.builtin.include_role:
+        name: app-deploy
+        tasks_from: stop.yml

roles/app-deploy/tasks/django.yml

@@ -65,6 +65,8 @@
     group: "{{ admin_user }}"
     mode: "0600"
 
+# COMPOSE_PROFILES must match the start step below: without it, profile-gated
+# services (e.g. the dj-queue worker) are skipped here and keep a stale image.
 - name: "django[{{ _project }}] build images"
   ansible.builtin.command:
     cmd: "docker compose -f {{ _spec.compose_file }} --env-file .env build"
@@ -72,6 +74,7 @@
   environment:
     COMPOSE_PROJECT_NAME: "{{ _project }}"
     WEB_PORT: "{{ app_item.port }}"
+    COMPOSE_PROFILES: "{{ (app_item.compose_profiles | default([])) | join(',') }}"
   become: true
   become_user: "{{ admin_user }}"
   changed_when: true
@@ -100,6 +103,9 @@
     - _legacy_systemd | default('') | length > 0
     - _legacy_systemd_probe.stdout | default('') | length > 0
 
+# Optional host_apps.compose_profiles (e.g. [worker]) activates compose profiles
+# on this host only. Used for monica_site / college_craft / print_forge dj-queue singleton on
+# adama — without it, plain up --remove-orphans can drop a manually started worker.
 - name: "django[{{ _project }}] start containers"
   ansible.builtin.command:
     cmd: "docker compose -f {{ _spec.compose_file }} --env-file .env up -d --remove-orphans"
@@ -107,6 +113,7 @@
   environment:
     COMPOSE_PROJECT_NAME: "{{ _project }}"
     WEB_PORT: "{{ app_item.port }}"
+    COMPOSE_PROFILES: "{{ (app_item.compose_profiles | default([])) | join(',') }}"
   become: true
   become_user: "{{ admin_user }}"
   changed_when: true

roles/app-deploy/tasks/main.yml

@@ -6,24 +6,21 @@
 # at branch master.
 #
 # host_apps (host_vars) lists what runs on THIS host; app_catalog (group_vars)
-# describes how each app is built.
+# describes how each app is built. host_apps.enabled: false (omit = true)
+# keeps the row for ports/stop.sh but skips deploy so CI cannot resurrect it.
 
 - name: Classify catalog by type
   ansible.builtin.set_fact:
     django_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'django') | map(attribute='key') | list }}"
     node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
 
-- name: Resolve deploy targets for {{ inventory_hostname }}
+- name: Resolve matching host_apps for {{ inventory_hostname }}
   ansible.builtin.set_fact:
-    deploy_targets: >-
+    matching_host_apps: >-
       {{ (host_apps | default([]) | selectattr('name', 'equalto', app | default('')) | selectattr('env', 'equalto', app_env | default('')) | list)
          if (app is defined and app_env is defined)
          else (host_apps | default([])) }}
 
-- name: Show deploy targets
-  ansible.builtin.debug:
-    msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
-
 # Explicit --app/--env with no host_apps match used to silently no-op Django
 # and still refresh web-static (looked like a "static-only" deploy). Fail loud.
 - name: Fail when requested app+env is not on this host
@@ -36,7 +33,32 @@
   when:
     - app is defined
     - app_env is defined
-    - deploy_targets | length == 0
+    - matching_host_apps | length == 0
+
+# rejectattr('enabled', …) raises when the key is omitted. Omit = enabled (#35).
+- name: Reset deploy targets
+  ansible.builtin.set_fact:
+    deploy_targets: []
+
+- name: "Drop disabled host_apps (enabled: false)"
+  ansible.builtin.set_fact:
+    deploy_targets: "{{ deploy_targets + [item] }}"
+  loop: "{{ matching_host_apps }}"
+  when: item.enabled | default(true)
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
+
+- name: Show skipped disabled apps
+  ansible.builtin.debug:
+    msg: "skip disabled {{ item.name }}/{{ item.env }} on {{ inventory_hostname }}"
+  loop: "{{ matching_host_apps }}"
+  when: not (item.enabled | default(true))
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
+
+- name: Show deploy targets
+  ansible.builtin.debug:
+    msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
 
 - name: Fail when requested app missing from app_catalog
   ansible.builtin.fail:

roles/app-deploy/tasks/stop.yml

@@ -0,0 +1,47 @@
+---
+# Stop one app+env. CI/manual must pass: app=<name> app_env=<beta|prod>
+#
+# Matches host_apps including enabled: false. Hosts without that row skip.
+# Django/compose → docker compose down (no -v). Node-static → message only.
+
+- name: Fail unless app and app_env extra-vars set
+  ansible.builtin.fail:
+    msg: "stop-apps.yml requires -e app=<name> -e app_env=<beta|prod>"
+  when: app is not defined or app_env is not defined
+
+- name: Classify catalog by type
+  ansible.builtin.set_fact:
+    django_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'django') | map(attribute='key') | list }}"
+    node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
+
+- name: Fail when requested app missing from app_catalog
+  ansible.builtin.fail:
+    msg: >-
+      app={{ app }} is not in app_catalog. Known: {{ app_catalog.keys() | list }}.
+  when: app not in app_catalog
+
+- name: Resolve stop targets for {{ inventory_hostname }}
+  ansible.builtin.set_fact:
+    stop_targets: "{{ host_apps | default([]) | selectattr('name', 'equalto', app) | selectattr('env', 'equalto', app_env) | list }}"
+
+- name: Skip {{ inventory_hostname }} — no host_apps row for {{ app }}/{{ app_env }}
+  ansible.builtin.debug:
+    msg: "skip {{ inventory_hostname }}: no host_apps entry for app={{ app }} env={{ app_env }}"
+  when: stop_targets | length == 0
+
+- name: Skip node-static {{ app }}/{{ app_env }} (no compose project)
+  ansible.builtin.debug:
+    msg: >-
+      {{ app }} is node-static. stop.sh only compose-downs Django apps.
+      Drop the host_apps row (or set enabled: false) and re-run deploy.sh
+      so web-static regenerates nginx without that vhost.
+  when:
+    - stop_targets | length > 0
+    - app in node_names
+
+- name: Stop Django (docker compose) apps
+  ansible.builtin.include_tasks: stop_django.yml
+  loop: "{{ stop_targets | selectattr('name', 'in', django_names) | list }}"
+  loop_control:
+    loop_var: app_item
+    label: "{{ app_item.name }}/{{ app_item.env }}"

roles/app-deploy/tasks/stop_django.yml

@@ -0,0 +1,72 @@
+---
+# Compose-down one Django app+env. Called per item with loop_var app_item.
+# No `down -v` — volumes stay. Checkout and secrets stay.
+
+- name: "stop[{{ app_item.name }}/{{ app_item.env }}] locals"
+  ansible.builtin.set_fact:
+    _spec: "{{ app_catalog[app_item.name] }}"
+    _src: "{{ apps_src_dir }}/{{ app_item.name }}_{{ app_item.env }}"
+    _project: "{{ app_item.name }}_{{ app_item.env }}"
+
+- name: "stop[{{ _project }}] checkout dir"
+  ansible.builtin.stat:
+    path: "{{ _src }}"
+  register: _src_stat
+
+- name: "stop[{{ _project }}] compose file"
+  ansible.builtin.stat:
+    path: "{{ _src }}/{{ _spec.compose_file }}"
+  register: _compose_stat
+  when: _src_stat.stat.exists
+
+- name: "stop[{{ _project }}] env file"
+  ansible.builtin.stat:
+    path: "{{ _src }}/.env"
+  register: _env_stat
+  when: _src_stat.stat.exists
+
+- name: "stop[{{ _project }}] use compose down"
+  ansible.builtin.set_fact:
+    _use_compose: "{{ _src_stat.stat.exists and _compose_stat is defined and _compose_stat.stat.exists }}"
+
+- name: "stop[{{ _project }}] compose down"
+  ansible.builtin.command:
+    cmd: >-
+      docker compose -f {{ _spec.compose_file }}
+      {{ '--env-file .env' if (_env_stat is defined and _env_stat.stat.exists) else '' }}
+      down
+    chdir: "{{ _src }}"
+  environment:
+    COMPOSE_PROJECT_NAME: "{{ _project }}"
+    WEB_PORT: "{{ app_item.port }}"
+    COMPOSE_PROFILES: "{{ (app_item.compose_profiles | default([])) | join(',') }}"
+  become: true
+  become_user: "{{ admin_user }}"
+  when: _use_compose | bool
+  changed_when: true
+
+- name: "stop[{{ _project }}] find leftover project containers"
+  ansible.builtin.command:
+    cmd: docker ps -aq --filter label=com.docker.compose.project={{ _project }}
+  register: _leftovers
+  changed_when: false
+  become: true
+  when: not (_use_compose | bool)
+
+- name: "stop[{{ _project }}] remove leftover project containers"
+  ansible.builtin.command:
+    cmd: "docker rm -f {{ _leftovers.stdout_lines | join(' ') }}"
+  become: true
+  when:
+    - not (_use_compose | bool)
+    - _leftovers is defined
+    - _leftovers.stdout_lines | length > 0
+  changed_when: true
+
+- name: "stop[{{ _project }}] nothing to stop (no checkout)"
+  ansible.builtin.debug:
+    msg: "no checkout at {{ _src }} and no containers for project {{ _project }}"
+  when:
+    - not (_use_compose | bool)
+    - _leftovers is defined
+    - _leftovers.stdout_lines | length == 0

roles/searxng/defaults/main.yml

@@ -0,0 +1,19 @@
+---
+# SearxNG for chat_backend grounded web search (#10).
+# Hosted on ai-server-4080 only (next to Ollama). LAN-only — not NPM public.
+
+searxng_dir: "{{ apps_base_dir }}/searxng"
+searxng_image: "searxng/searxng:latest"
+
+# Host port 8088 — 8080 is already dta_webapp prod on ai-server-4080.
+searxng_host_port: 8088
+searxng_container_port: 8080
+
+# Public base URL as seen by chat_backend containers on the LAN.
+searxng_base_url: "http://{{ ansible_host }}:{{ searxng_host_port }}/"
+
+# Override via host_vars or vault; must be stable across restarts.
+searxng_secret_key: "CHANGE_ME_SEARXNG_SECRET"
+
+# LAN CIDR allowed to hit the JSON API (same pattern as observability).
+searxng_ufw_from: "{{ ufw_ssh_allowed_network }}"

roles/searxng/tasks/main.yml

@@ -0,0 +1,64 @@
+---
+# SearxNG JSON search API for chat_backend grounded retrieval.
+# Enabled on ai-server-4080 via searxng_stack: true (issue #10).
+
+- name: searxng | ensure project directory
+  ansible.builtin.file:
+    path: "{{ searxng_dir }}"
+    state: directory
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0750"
+
+- name: searxng | settings.yml
+  ansible.builtin.template:
+    src: settings.yml.j2
+    dest: "{{ searxng_dir }}/settings.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0640"
+  register: _searxng_settings
+
+- name: searxng | compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ searxng_dir }}/docker-compose.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0644"
+  register: _searxng_compose
+
+- name: searxng | allow JSON API from LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ searxng_host_port }}"
+    proto: tcp
+    from_ip: "{{ searxng_ufw_from }}"
+    comment: SearxNG for chat_backend grounded search
+
+- name: searxng | start stack
+  ansible.builtin.command:
+    cmd: >-
+      docker compose up -d --remove-orphans
+      {{ '--force-recreate' if (
+        _searxng_compose is changed
+        or _searxng_settings is changed
+      ) else '' }}
+    chdir: "{{ searxng_dir }}"
+  become: true
+  become_user: "{{ admin_user }}"
+  register: _searxng_up
+  changed_when: >-
+    _searxng_up.rc == 0 and (
+      'Started' in (_searxng_up.stdout | default(''))
+      or 'Recreated' in (_searxng_up.stdout | default(''))
+      or 'Created' in (_searxng_up.stdout | default(''))
+      or _searxng_compose is changed
+      or _searxng_settings is changed
+    )
+  failed_when: _searxng_up.rc != 0
+
+- name: searxng | show compose failure output
+  ansible.builtin.debug:
+    msg: "{{ _searxng_up.stderr_lines | default(_searxng_up.stdout_lines) }}"
+  when: _searxng_up is failed

roles/searxng/templates/docker-compose.yml.j2

@@ -0,0 +1,23 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+services:
+  searxng:
+    image: {{ searxng_image }}
+    container_name: searxng
+    restart: unless-stopped
+    ports:
+      - "{{ searxng_host_port }}:{{ searxng_container_port }}"
+    volumes:
+      - ./settings.yml:/etc/searxng/settings.yml:rw
+    environment:
+      - SEARXNG_BASE_URL={{ searxng_base_url }}
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      - SETGID
+      - SETUID
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"

roles/searxng/templates/settings.yml.j2

@@ -0,0 +1,20 @@
+# Managed by Ansible (roles/searxng). Do not edit by hand.
+# Minimal overlay on SearxNG defaults — JSON format required by chat_backend (#62).
+
+use_default_settings: true
+
+server:
+  secret_key: "{{ searxng_secret_key }}"
+  limiter: false
+  image_proxy: false
+  port: {{ searxng_container_port }}
+  bind_address: "0.0.0.0"
+  base_url: "{{ searxng_base_url }}"
+
+search:
+  safe_search: 0
+  autocomplete: ""
+  default_lang: "en"
+  formats:
+    - html
+    - json

roles/ufw/tasks/main.yml

@@ -28,6 +28,28 @@
     proto: tcp
   loop: "{{ ufw_allowed_tcp_ports }}"
 
+- name: Allow extra TCP ports
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: tcp
+  loop: "{{ ufw_extra_tcp_ports | default([]) }}"
+
+- name: Allow extra TCP ports from LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: tcp
+    from_ip: "{{ ufw_ssh_allowed_network }}"
+  loop: "{{ ufw_lan_tcp_ports | default([]) }}"
+
+- name: Allow extra UDP ports or ranges
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: udp
+  loop: "{{ ufw_extra_udp_ports | default([]) }}"
+
 - name: Enable UFW
   community.general.ufw:
     state: enabled

roles/web-static/tasks/main.yml

@@ -6,9 +6,20 @@
   ansible.builtin.set_fact:
     _node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
 
+# rejectattr('enabled', …) raises when the key is omitted. Omit = enabled (#35).
+- name: web-static | reset static apps
+  ansible.builtin.set_fact:
+    _static_apps: []
+
 - name: web-static | this host's static apps
   ansible.builtin.set_fact:
-    _static_apps: "{{ host_apps | default([]) | selectattr('name', 'in', _node_names) | list }}"
+    _static_apps: "{{ _static_apps + [item] }}"
+  loop: "{{ host_apps | default([]) }}"
+  when:
+    - item.name in _node_names
+    - item.enabled | default(true)
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
 
 - name: web-static | configure and run
   when: _static_apps | length > 0
@@ -39,9 +50,34 @@
         mode: "0644"
       register: _compose_file
 
+    # Free host ports before bind. Catches leftover/orphan containers from older
+    # compose projects that still publish 8080/8081/etc. Includes the current
+    # web-static container when present — compose up recreates it next.
+    - name: web-static | find containers publishing static ports
+      ansible.builtin.command:
+        cmd: docker ps -q --filter publish={{ item }}
+      loop: "{{ _static_apps | map(attribute='port') | list | unique }}"
+      register: _port_holders
+      changed_when: false
+      become: true
+
+    - name: web-static | unique container ids on static ports
+      ansible.builtin.set_fact:
+        _stale_port_cids: "{{ _port_holders.results | map(attribute='stdout_lines') | flatten | unique | list }}"
+
+    - name: web-static | remove containers holding static ports
+      ansible.builtin.command:
+        cmd: "docker rm -f {{ _stale_port_cids | join(' ') }}"
+      when: _stale_port_cids | length > 0
+      become: true
+      changed_when: true
+      register: _stale_port_rm
+
     - name: web-static | start container
       ansible.builtin.command:
-        cmd: "docker compose up -d --remove-orphans{{ ' --force-recreate' if (_compose_file is changed) else '' }}"
+        cmd: >-
+          docker compose up -d --remove-orphans
+          {{ '--force-recreate' if (_compose_file is changed or (_stale_port_rm is changed)) else '' }}
         chdir: "{{ web_static_dir }}"
       become: true
       become_user: "{{ admin_user }}"
@@ -53,5 +89,8 @@
         chdir: "{{ web_static_dir }}"
       become: true
       become_user: "{{ admin_user }}"
-      when: _nginx_conf is changed and _compose_file is not changed
+      when:
+        - _nginx_conf is changed
+        - _compose_file is not changed
+        - _stale_port_rm is not changed
       changed_when: true

roles/web-static/templates/nginx.conf.j2

@@ -30,9 +30,17 @@ server {
     root {{ (app_catalog[a.name].webroot_pattern | default(web_static_root ~ '/{env}_' ~ a.name)) | replace('{env}', a.env) }};
     index index.html;
 
+{% if app_catalog[a.name].error_page_404 | default('') %}
+    error_page 404 {{ app_catalog[a.name].error_page_404 }};
+
+    location / {
+        try_files $uri $uri/ =404;
+    }
+{% else %}
     location / {
         try_files $uri $uri/ /index.html;
     }
+{% endif %}
 
     location ~* \.(?:js|css|woff2?|png|jpg|jpeg|gif|svg|ico)$ {
         expires 7d;

scripts/deploy.sh

@@ -14,10 +14,10 @@ Usage: $(basename "$0") [HOST] [OPTIONS]
 
 Deploy applications with deploy-apps.yml.
 
-  HOST              Optional. Limit to one host: adama, roslin, or ai-server-4080.
+  HOST              Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
 
 Options:
-  --app NAME        App to deploy (company_site, dta_service, dta_webapp, scha, chat_web_app, chat_backend)
+  --app NAME        App to deploy (company_site, dta_service, dta_webapp, dta_blog, scha, chat_web_app, chat_backend, monica_site, url_shortening_service, college_craft, print_forge, abc_be, abc_worker, abc_fe, livekit)
   --env ENV         Environment: beta or prod
   --ref REF         Git ref/sha to deploy (default: master)
   --check           Dry run
@@ -67,7 +67,7 @@ while [[ $# -gt 0 ]]; do
       EXTRA_VARS+=(-e "$2")
       shift 2
       ;;
-    adama|roslin|ai-server-4080)
+    adama|roslin|starbuck|apollo|ai-server-4080)
       LIMIT="$1"
       shift
       ;;

scripts/provision.sh

@@ -15,9 +15,10 @@ Provision server(s) with site.yml.
 
 Applies: common, ufw, docker, nodejs, gitea-key, tianji, alloy (every host).
 On ai-server-4080 also: observability (Loki + Prometheus + Grafana) when
-observability_stack is true in host_vars.
+observability_stack is true, and SearxNG when searxng_stack is true
+(chat_backend grounded search on :8088).
 
-  HOST              Optional. Limit to one host: adama, roslin, or ai-server-4080.
+  HOST              Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
                     Omit to run against all webservers.
 
 Options:
@@ -31,7 +32,7 @@ Examples:
   $(basename "$0") adama --check    # dry run on adama only
   $(basename "$0") adama            # provision adama (includes Alloy)
   $(basename "$0") adama --ask-pass  # first SSH login before ssh-copy-id
-  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana
+  $(basename "$0") ai-server-4080   # control node + Loki/Prometheus/Grafana + SearxNG
   $(basename "$0")                  # provision all hosts
 EOF
 }
@@ -58,7 +59,7 @@ while [[ $# -gt 0 ]]; do
       EXTRA_ARGS+=(--ask-become-pass)
       shift
       ;;
-    adama|roslin|ai-server-4080)
+    adama|roslin|starbuck|apollo|ai-server-4080)
       LIMIT="$1"
       shift
       ;;

scripts/stop.sh

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
+
+LIMIT=""
+EXTRA_ARGS=()
+EXTRA_VARS=()
+APP=""
+APP_ENV=""
+
+usage() {
+  cat <<EOF
+Usage: $(basename "$0") [HOST] --app NAME --env ENV [OPTIONS]
+
+Stop a docker-compose app (compose down). Inverse of deploy.sh.
+
+Does not remove volumes, checkouts, or secrets. Next deploy.sh of the same
+app+env will start it again unless host_apps sets enabled: false.
+
+Node-static apps have no compose project — stop.sh skips them.
+
+  HOST              Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
+
+Options:
+  --app NAME        Required. App to stop (same names as deploy.sh)
+  --env ENV         Required. Environment: beta or prod
+  --check           Dry run
+  --diff            Show diffs
+  --extra-vars V    Pass raw extra vars
+  -h, --help        Show this help
+
+Examples:
+  $(basename "$0") --app print_forge --env prod
+  $(basename "$0") --app print_forge --env beta
+  $(basename "$0") adama --app chat_backend --env beta --check
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    --app)
+      APP="$2"
+      EXTRA_VARS+=(-e "app=$2")
+      shift 2
+      ;;
+    --env)
+      APP_ENV="$2"
+      EXTRA_VARS+=(-e "app_env=$2")
+      shift 2
+      ;;
+    --check)
+      EXTRA_ARGS+=(--check)
+      shift
+      ;;
+    --diff)
+      EXTRA_ARGS+=(--diff)
+      shift
+      ;;
+    --extra-vars)
+      EXTRA_VARS+=(-e "$2")
+      shift 2
+      ;;
+    adama|roslin|starbuck|apollo|ai-server-4080)
+      LIMIT="$1"
+      shift
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 1
+      ;;
+    esac
+done
+
+if [[ -z "$APP" || -z "$APP_ENV" ]]; then
+  echo "error: --app and --env are required" >&2
+  usage >&2
+  exit 1
+fi
+
+CMD=(ansible-playbook playbooks/stop-apps.yml "${EXTRA_ARGS[@]}" "${EXTRA_VARS[@]}")
+if [[ -n "$LIMIT" ]]; then
+  CMD+=(--limit "$LIMIT")
+  echo "==> Stopping ${APP}/${APP_ENV} on: $LIMIT"
+else
+  echo "==> Stopping ${APP}/${APP_ENV} on all webservers"
+fi
+
+exec "${CMD[@]}"