Merge master and keep HeyCatch short links off the container port.

Master already has the public-page nginx from the 124 work, including the redirect that published port 8082. Keep that config and leave the shortlink Location relative.

IMPLEMENTATION.md

@@ -30,6 +30,7 @@ flowchart TB
 |----------|------|----------|---------------|
 | **Provision** | New VM, OS change, firewall, Docker install | `site.yml` | ai-server-4080 — run manually |
 | **Deploy** | Green unit tests on `master` | `deploy-apps.yml` | Gitea Act runner on ai-server-4080 |
+| **Stop** | Take a compose app down | `stop-apps.yml` (`scripts/stop.sh`) | ai-server-4080 — run manually |
 
 Both pipelines share the same inventory (`inventory/hosts.yml`).
 
@@ -57,15 +58,17 @@ server-infra/
 │   ├── hosts.yml
 │   ├── group_vars/
 │   │   └── all.yml         # vars + app_catalog
-│   └── host_vars/
+│   ├── host_vars/
-│       ├── adama.yml       # host_apps (django + static; monica worker)
+│   │   ├── adama.yml       # host_apps (django + static; monica worker)
-│       ├── roslin.yml      # host_apps (mirrors adama, no worker)
+│   │   ├── roslin.yml      # host_apps (mirrors adama, no worker)
-│       ├── starbuck.yml    # host_apps (mirrors roslin)
+│   │   ├── starbuck.yml    # host_apps (mirrors roslin)
-│       ├── apollo.yml      # host_apps (mirrors roslin)
+│   │   ├── apollo.yml      # host_apps (mirrors roslin)
-│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
+│   │   └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
+│   └── wordpress_sites.yml # WordPress NFS mounts + generated compose inputs
 ├── playbooks/
 │   ├── site.yml            # Phase 1: provision
-│   └── deploy-apps.yml     # Phase 2: CI deploy
+│   ├── deploy-apps.yml     # Phase 2: CI deploy
+│   └── stop-apps.yml       # compose down one app+env
 ├── roles/
 │   ├── common/             # Base packages
 │   ├── ufw/                # Firewall
@@ -77,10 +80,12 @@ server-infra/
 │   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
 │   ├── alloy/              # log/metrics shipper
 │   ├── app-deploy/         # django (docker) + node-static deploy
-│   └── web-static/         # nginx container serving /var/www builds
+│   ├── web-static/         # nginx container serving /var/www builds
+│   └── wordpress/          # NFS wp-content + compose rendered per site
 └── scripts/
     ├── provision.sh        # Wrapper with --limit support
-    └── deploy.sh           # Wrapper for deploy playbook
+    ├── deploy.sh           # Wrapper for deploy playbook
+    └── stop.sh             # Wrapper for stop playbook (compose down)
 ```
 
 ## Prerequisites (One-Time Bootstrap)
@@ -174,6 +179,18 @@ New hosts need the one-time passwordless sudo bootstrap in
 ./scripts/deploy.sh apollo
 ```
 
+### Stop a compose app
+
+```bash
+./scripts/stop.sh --app print_forge --env prod
+./scripts/stop.sh --app print_forge --env beta
+./scripts/stop.sh adama --app chat_backend --env beta --check
+```
+
+`stop.sh` is `docker compose down` for one `--app` + `--env` (no `-v`). Checkouts and
+secrets stay. Next `deploy.sh` of that app+env starts it again unless `host_apps`
+sets `enabled: false`. Node-static apps have no compose project; stop.sh skips them.
+
 Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`.
 
 ## Phase 1: Provision (`site.yml`)
@@ -182,7 +199,7 @@ Applies roles in order to the `webservers` group:
 
 | Role | Purpose |
 |------|---------|
-| `common` | apt update, git, python3, pip, curl, ca-certificates |
+| `common` | apt update, git, python3, pip, curl, ca-certificates, nfs-common |
 | `ufw` | Firewall: SSH from LAN only, HTTP/HTTPS public |
 | `docker` | Docker CE, compose plugin, add `westfarn` to `docker` group |
 | `nodejs` | Node.js + npm + npx (NodeSource) for `dta_webapp` builds |
@@ -218,9 +235,10 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 | `monica_site` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
 | `url_shortening_service` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres; **no worker**. Two public hosts, same container: short domain (`GET /`, `GET /<code>` 302) and API host (`/api/links/`, Bearer required). |
 | `college_craft` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha` / `monica_site`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434`; Nominatim `http://10.0.0.128:8089`; prod `SITE_UNDER_CONSTRUCTION=true` until launch |
-| `print_forge` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha` / `monica_site` / `college_craft`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Nominatim `http://10.0.0.128:8089`; prod `SITE_UNDER_CONSTRUCTION=true` until launch; prod NPM waits until launch |
+| `print_forge` | django (docker) | all webservers | beta + prod | **stopped** (`enabled: false`, [#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)). Ports reserved. Re-enable + `deploy.sh` to bring back. Worker was adama-only. |
 | `abc_be` | django (docker) | all webservers | **beta only** | AI Benefits Coach API; no bundled Postgres; Ollama `http://10.0.0.128:11434`; shares DB `abc_be_beta` with `abc_worker` |
 | `abc_worker` | django-type compose (FastAPI) | all webservers | **beta only** | same Postgres as `abc_be`; `migrate_cmd: true`; WS on host port 8018 |
+| `abc_voice` | django-type compose (FastAPI) | all webservers | **beta only** | Same repo as `abc_worker`, compose `docker-compose.voice.yml`, join API on host port 8020. One uvicorn worker. |
 | `abc_fe` | node-static (Vite) | all webservers | **beta only** | built to `/var/www/<env>.abc.aimloperations/html`; no prod `host_apps` row |
 | `livekit` | django-type compose (LiveKit SFU) | all webservers | **beta only** | Docker like `abc_worker`. Host network `:7880`. **Do not NPM-balance** until Redis — pick one upstream (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that host. Secrets `~/Documents/secrets/livekit/livekit_beta.env` |
 
@@ -234,6 +252,9 @@ a DB.
 - `host_apps` (`host_vars/<host>.yml`) — which app+env+port runs on that host.
   Optional `compose_profiles: [worker]` activates docker compose profiles on that
   host only (used for `monica_site` / `college_craft` / `print_forge` dj-queue singleton on adama).
+  Optional `enabled: false` skips deploy (CI included) but keeps the row for ports
+  and `stop.sh`. Omit the key (or `true`) to deploy as before. Do not use Jinja
+  `rejectattr('enabled')` — missing key is an error, not "enabled".
 - Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
   Ports match across app hosts so NPM can balance `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT`.
 
@@ -252,9 +273,10 @@ future beta replica.
 | monica_site | 8014 | 8004 | all webservers |
 | url_shortening_service | 8015 | 8005 | all webservers |
 | college_craft | 8016 | 8006 | all webservers |
-| print_forge | **8019** | **8007** | all webservers |
+| print_forge | **8019** (*stopped*) | **8007** (*stopped*) | all webservers (`enabled: false`) |
 | abc_be | **8017** | 8009 (*not deployed*) | all webservers |
 | abc_worker | **8018** | 8008 (*not deployed*) | all webservers |
+| abc_voice | **8020** | — | all webservers |
 | dta_webapp (nginx) | 8081 | 8080 | all webservers |
 | chat_web_app (nginx) | 8083 | 8082 | all webservers |
 | abc_fe (nginx) | **8085** | 8084 (*not deployed*) | all webservers |
@@ -285,7 +307,16 @@ chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.
      --app company_site --env prod --ref "${{ gitea.sha }}"
    ```
 3. `deploy-apps.yml` runs against `webservers`; each host deploys only the
-   matching app+env from its `host_apps`.
+   matching **enabled** app+env from its `host_apps` (`enabled: false` is skipped).
+
+To stop a compose app without deleting inventory:
+
+```bash
+./scripts/stop.sh --app print_forge --env prod
+```
+
+Set `enabled: false` on that `host_apps` row so the next CI deploy cannot start it
+again. `stop.sh` still matches disabled rows.
 
 ### `app-deploy` role behavior
 
@@ -329,10 +360,11 @@ point each domain at the backend(s):
 | url_shortening_service (short) | `aiml.pw` and/or `cidinn.li` (pick when DNS is ready); `short-beta.aimloperations.com` (beta). Proxy `/` + `/[a-z0-9]{4,8}` only — 404 `/api/`, `/admin/`, `/debug/` | `adama:8005` / `8015` + same on roslin / starbuck / apollo / ai-server-4080 |
 | url_shortening_service (API) | `shortener.aimloperations.com`; `shortener-beta.aimloperations.com` (beta). Proxy `/api/` only — 404 `/admin/` | same ports as short host (one container) |
 | college_craft | `collegecraft.com` (+ www); `college-craft-preview.aimloperations.com` (beta) | `adama:8006` / `8016` + same on roslin / starbuck / apollo / ai-server-4080 |
-| print_forge | `printforgeprints.com` (+ www) — **do not NPM-route prod until launch**; `print-forge-preview.aimloperations.com` (beta) | `adama:8007` / `8019` + same on roslin / starbuck / apollo / ai-server-4080 |
+| print_forge | `printforgeprints.com` (+ www); `print-forge-preview.aimloperations.com` (beta) — **app stopped** (`enabled: false`). Disable NPM hosts or expect 502. | `:8007` / `:8019` reserved |
 | abc_fe | `beta.abc.aimloperations.com` (beta only) | `adama:8085` + same on roslin / starbuck / apollo / ai-server-4080 |
 | abc_be | `beta.abc.be.aimloperations.com` (beta only) | `adama:8017` + same on roslin / starbuck / apollo / ai-server-4080 |
 | abc_worker | `beta.abc.worker.aimloperations.com` (beta only; HTTP + WebSocket upgrade) | `adama:8018` + same on roslin / starbuck / apollo / ai-server-4080 |
+| abc_voice | `beta.abc.voice.aimloperations.com` (beta only). Balance this host. Do not balance LiveKit. | `adama:8020` + same on roslin / starbuck / apollo / ai-server-4080 |
 | livekit | `LIVEKIT_DOMAIN` (e.g. `livekit.aimloperations.com`) — NPM Proxy Host, WebSocket, **single upstream** | `adama:7880` (do not balance until Redis) |
 
 ### Required changes IN each app repo (owned separately)
@@ -379,6 +411,7 @@ do not).
 | print_forge | beta | `print_forge_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/print_forge_beta` |
 | abc_be | beta | `abc_be_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/abc_be_beta` |
 | abc_worker | beta | *(same `abc_be_beta`)* | worker uses `DB_*` pointing at `abc_be_beta` — do **not** create a second DB |
+| abc_voice | beta | *(same `abc_be_beta`)* | voice container uses the worker DB settings — do **not** create a second DB |
 
 Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 `listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
@@ -406,7 +439,10 @@ App-repo companion (`college_craft`, not this repo): deploy workflows still call
 `--app client_site` (template leftover). They must become `--app college_craft`
 before CI deploy will hit this catalog entry. Default branch is `master` (not `main`).
 
-`print_forge` extra env (control-node secrets, not in git):
+`print_forge` extra env (control-node secrets, not in git) — **currently stopped**
+([#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)):
+`host_apps` rows stay with `enabled: false`. Bring back with `enabled` removed/true
+then `deploy.sh`. Secrets kept:
 `~/Documents/secrets/print_forge/print_forge_prod.env` and
 `print_forge_beta.env`. Template: app repo `.env.prod.example`.
 `DJANGO_ENV` / `DJANGO_ALLOWED_HOSTS` / `DATABASE_URL` / `WEB_PORT` /
@@ -421,7 +457,8 @@ on beta. Default branch is `master`.
 
 Companion app ticket: [print_forge#1](https://git.aimloperations.com/ai_ml_operations/print_forge/issues/1)
 ([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)).
-App Gitea workflow deploys beta on push to `master` (`--app print_forge`).
+App Gitea workflow still calls `--app print_forge` on `master`; deploy **no-ops**
+while `enabled: false`.
 
 `abc_be` / `abc_worker` / `abc_fe` (GIS org, not this repo) — **beta only** ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)):
 
@@ -430,13 +467,16 @@ Control-node secrets (never git, mode `600`):
 ```text
 ~/Documents/secrets/abc_be/abc_be_beta.env
 ~/Documents/secrets/abc_worker/abc_worker_beta.env
+~/Documents/secrets/abc_voice/abc_voice_beta.env
 ```
 
-Templates: `GIS/abc_be` `.env.beta.example`, `GIS/abc_worker` `.env.beta.example`.
+Templates: `GIS/abc_be` `.env.beta.example`, `GIS/abc_worker` `.env.beta.example`, `GIS/abc_worker` `.env.voice.beta.example`.
+`abc_voice` is the same checkout as `abc_worker` with `docker-compose.voice.yml`. Copy the worker beta env, set `VOICE_ENABLED=true`, `CARTESIA_API_KEY`, and the LiveKit keys. `abc_be` then uses `WORKER_AV_URL=https://beta.abc.voice.aimloperations.com`.
 `abc_fe` has no secret file — public `VITE_*` URLs live in committed `.env.beta`.
 Create Postgres DB `abc_be_beta` and grant `westfarn` before first deploy.
 NPM: `beta.abc.aimloperations.com` → `:8085`, `beta.abc.be.aimloperations.com` → `:8017`,
-`beta.abc.worker.aimloperations.com` → `:8018` (enable WebSocket). No prod ABC
+`beta.abc.worker.aimloperations.com` → `:8018` (enable WebSocket),
+`beta.abc.voice.aimloperations.com` → `:8020` on all five webservers. No prod ABC
 `host_apps` rows; ports **8009** / 8008 / 8084 reserved (`print_forge` took prod **8007**).
 
 Companion workflows: [abc_be#22](https://git.aimloperations.com/GIS/abc_be/issues/22),

README.md

@@ -33,12 +33,21 @@ ansible adama -m ping
 
 # Provision all hosts
 ./scripts/provision.sh
+
+# Stop a compose app (does not remove volumes/checkouts)
+./scripts/stop.sh --app print_forge --env prod
+./scripts/stop.sh --app print_forge --env beta
 ```
 
 See [IMPLEMENTATION.md](IMPLEMENTATION.md) for full architecture, CI/CD plan, and phase breakdown.
 
 Observability (Alloy → Loki / Prometheus → Grafana): [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md) · [docs/GRAFANA_USAGE.md](docs/GRAFANA_USAGE.md)
 
+WordPress (MariaDB on TrueNAS): [docs/WORDPRESS.md](docs/WORDPRESS.md).
+Provision mounts each site's `wp-content` and writes its compose file from
+[inventory/wordpress_sites.yml](inventory/wordpress_sites.yml). Start the
+containers by hand (adama first).
+
 Provision installs **Alloy** on every host. On **ai-server-4080** it also starts
 the central **Loki / Prometheus / Grafana** stack (`observability_stack: true`).
 

docs/WORDPRESS.md

@@ -0,0 +1,392 @@
+# Adding a WordPress site
+
+Manual procedure for a WordPress site on all five web servers, one MariaDB on
+TrueNAS, and Nginx Proxy Manager (NPM) in front. Ansible does **not** start
+the WordPress containers. When you run `scripts/provision.sh` it mounts each
+site's `wp-content` and, once that entry has a public `url`, writes its
+compose file from
+[inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml). Repeat
+[Per site](#per-site) for every new site. Do not install another MariaDB.
+
+## Where things live
+
+| Piece | Where |
+|-------|--------|
+| MariaDB | TrueNAS `10.0.0.230`, port **3306**, LAN only |
+| MariaDB login | `westfarn` @ `%` (the host the TrueNAS app created) |
+| NPM | `10.0.0.230` (not managed by Ansible) |
+| App hosts | adama `10.0.0.77`, roslin `10.0.0.176`, starbuck `10.0.0.44`, apollo `10.0.0.7`, ai-server-4080 `10.0.0.128` |
+| Secrets | `~/Documents/secrets/wordpress/` on the control node. Never commit these. |
+| Shared files | one NFS dataset per site per env, mounted on every app host |
+
+Prod and beta never share a database, an NFS dataset, a port, or a domain.
+The five hosts of one env do share that env's database and `wp-content`.
+
+## Already done
+
+Leave this alone when adding the next site.
+
+- MariaDB app is running. Its host-path dataset is owned by uid **999** (`netdata`). Without that, the container crash-loops with `Errcode: 13 "Permission denied"` on `/var/lib/mysql`.
+- Login `westfarn` exists at host `%`, not at `10.0.0.%`. `GRANT` to `'westfarn'@'10.0.0.%'` fails with `ERROR 1133` because that row is not in `mysql.user`.
+- Databases `relawfirm` and `relawfirm_beta` already exist.
+
+Confirm from **Apps → mariadb → Shell**:
+
+```bash
+mariadb -u root -p
+```
+
+```sql
+SELECT user, host FROM mysql.user WHERE user = 'westfarn';
+SHOW DATABASES;
+SHOW GRANTS FOR 'westfarn'@'%';
+```
+
+`relawfirm` still needs the grants below if `SHOW GRANTS` does not list both databases. New sites use the same `GRANT` shape with their own database names.
+
+```sql
+GRANT ALL PRIVILEGES ON relawfirm.* TO 'westfarn'@'%';
+GRANT ALL PRIVILEGES ON relawfirm_beta.* TO 'westfarn'@'%';
+FLUSH PRIVILEGES;
+```
+
+## Ports
+
+Same host port on every app server for a given site and env. NPM balances
+`adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT` + `ai-server-4080:PORT`.
+
+These are not in the Ansible `host_apps` tables. Do not reuse a port from
+[IMPLEMENTATION.md](../IMPLEMENTATION.md) (8000–8020 and 8080–8088 are taken;
+**8020** is `abc_voice`).
+
+| Site | prod port | beta port | prod database | beta database |
+|------|-----------|-----------|---------------|---------------|
+| relawfirm | **8021** | **8022** | `relawfirm` | `relawfirm_beta` |
+| *(next site)* | next free | next free | `<site>` | `<site>_beta` |
+
+---
+
+## Per site
+
+Fill this in before touching a host. `relawfirm` is the first site.
+
+| | prod | beta |
+|--|------|------|
+| Public URL | `https://<prod-domain>` | `https://<beta-domain>` |
+| Database | `relawfirm` | `relawfirm_beta` |
+| Host port | `8021` | `8022` |
+| NFS dataset | `/mnt/AIMLOperationsNAS/wordpress/relawfirm` | `/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta` |
+| Mount on app hosts | `/mnt/wordpress/relawfirm/wp-content` | `/mnt/wordpress/relawfirm_beta/wp-content` |
+| Env file | `relawfirm_prod.env` | `relawfirm_beta.env` |
+| Compose project | `relawfirm_prod` | `relawfirm_beta` |
+
+`<site>` below means that short name (`relawfirm`, then whatever the next site is). Use lowercase letters, numbers, and underscore only.
+
+### 1. Databases
+
+Skip this for `relawfirm` / `relawfirm_beta`. They already exist.
+
+In the MariaDB shell (`mariadb -u root -p`):
+
+```sql
+CREATE DATABASE <site> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+CREATE DATABASE <site>_beta CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+GRANT ALL PRIVILEGES ON <site>.* TO 'westfarn'@'%';
+GRANT ALL PRIVILEGES ON <site>_beta.* TO 'westfarn'@'%';
+FLUSH PRIVILEGES;
+```
+
+The account in `TO` must match `SELECT user, host FROM mysql.user`. Today that is `'westfarn'@'%'`. Do not create another MariaDB user unless you want a separate login; this matches the shared Postgres user.
+
+### 2. Salts and env files
+
+On the control node, generate salts **once per env**:
+
+```bash
+curl -fsS https://api.wordpress.org/secret-key/1.1/salt/
+```
+
+Copy the same eight values into the env file on every app host for that env. A different salt on one host logs people out as soon as NPM sends them to another host. Prod and beta should not share salts.
+
+Create the file locally (mode `600`). Do not commit it.
+
+`~/Documents/secrets/wordpress/<site>_prod.env`:
+
+```bash
+WORDPRESS_DB_HOST=10.0.0.230:3306
+WORDPRESS_DB_USER=westfarn
+WORDPRESS_DB_PASSWORD=<mariadb password for westfarn>
+WORDPRESS_DB_NAME=<site>
+WORDPRESS_AUTH_KEY=<from salt generator>
+WORDPRESS_SECURE_AUTH_KEY=<from salt generator>
+WORDPRESS_LOGGED_IN_KEY=<from salt generator>
+WORDPRESS_NONCE_KEY=<from salt generator>
+WORDPRESS_AUTH_SALT=<from salt generator>
+WORDPRESS_SECURE_AUTH_SALT=<from salt generator>
+WORDPRESS_LOGGED_IN_SALT=<from salt generator>
+WORDPRESS_NONCE_SALT=<from salt generator>
+```
+
+`<site>_beta.env` is the same except `WORDPRESS_DB_NAME=<site>_beta` and a second set of salts.
+
+`WORDPRESS_DB_PASSWORD` is the MariaDB password. The WordPress admin password is chosen later in the browser, on the first host only.
+
+### 3. Shared `wp-content` (TrueNAS NFS)
+
+Uploads, plugins, and themes are files. With five hosts, those files have to live on one share. A private Docker volume on each host 404s media as soon as NPM moves the next request.
+
+Create **one dataset per site per env**. Preset **Generic** (this dataset is an NFS share, not the MariaDB host path). Leave record size at the default. Case sensitive. Do not add an SMB share.
+
+Example paths:
+
+```text
+/mnt/AIMLOperationsNAS/wordpress/relawfirm
+/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta
+```
+
+**Shares → Unix Shares (NFS) → Add**, one share per dataset:
+
+| Setting | Value |
+|---------|--------|
+| Path | the dataset |
+| Authorized Networks | `10.0.0.0/24` |
+| Maproot User | `root` |
+| Maproot Group | `root` |
+
+Maproot root lets the WordPress entrypoint create files on an empty share. The export is limited to the LAN. Do not forward NFS to the internet.
+
+Do not edit `/etc/fstab` by hand. Add one entry per env to
+[inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml), then mount
+every host:
+
+```bash
+./scripts/provision.sh
+```
+
+`export` is the TrueNAS dataset. `mount` is the path on the app hosts. Prod
+and beta are separate entries (beta export ends in `_beta`, not `-beta`).
+
+```yaml
+wordpress_sites:
+  - name: <site>
+    port: 8021
+    url: https://<prod-domain>
+    export: /mnt/AIMLOperationsNAS/wordpress/<site>
+    mount: /mnt/wordpress/<site>/wp-content
+  - name: <site>_beta
+    port: 8022
+    url: https://<beta-domain>
+    export: /mnt/AIMLOperationsNAS/wordpress/<site>_beta
+    mount: /mnt/wordpress/<site>_beta/wp-content
+```
+
+`url` is the public NPM URL. Leave it empty and provision still mounts NFS,
+but it does not write a compose file. Set it, then provision again.
+
+One host is enough to check the new lines:
+
+```bash
+./scripts/provision.sh adama
+mount | grep wordpress
+touch /mnt/wordpress/<site>/wp-content/.write-test && rm /mnt/wordpress/<site>/wp-content/.write-test
+```
+
+The first container to start copies the stock themes and plugins into this directory. Mount it on all five hosts **before** that start, and start only one host while the directory is still empty. Beta uses its own dataset and its own yaml entry.
+
+### 4. Compose file
+
+Do not copy a compose file by hand. The template is
+[roles/wordpress/templates/docker-compose.yml.j2](../roles/wordpress/templates/docker-compose.yml.j2).
+`scripts/provision.sh` renders one file per inventory entry on every app host.
+Secrets stay in the env file. `WP_HOME` and `WP_SITEURL` come from that
+entry's `url`, not from `http://10.0.0.77:8021`.
+
+| `name` | Compose file | Project (`name:` in the file) | Env file |
+|--------|--------------|--------------------------------|----------|
+| `<site>` | `/opt/apps/wordpress/<site>/docker-compose.yml` | `<site>_prod` | `/opt/apps/env/<site>_prod.env` |
+| `<site>_beta` | `/opt/apps/wordpress/<site>-beta/docker-compose.yml` | `<site>_beta` | `/opt/apps/env/<site>_beta.env` |
+
+Prod and beta do not share a directory. Image default is `wordpress:6-apache`
+(`wordpress_image` in the role). Override with `image:` on one entry. The
+bind mount is that entry's `mount`.
+
+Copy the env file onto each host (mode `600`) before the first start.
+Provision creates `/opt/apps/env` (mode `700`) but does not copy secrets.
+
+### 5. Install in the browser, one host only
+
+Start **adama** first. Leave the other four stopped until the installer has finished. Five installers against one database will fight.
+
+On adama:
+
+```bash
+cd /opt/apps/wordpress/<site>
+docker compose -p <site>_prod up -d
+docker compose -p <site>_prod logs -f
+```
+
+Wait until Apache is up. The log should not show a database connection error.
+
+In NPM, add a normal Proxy Host (single forward target) **before** opening the installer:
+
+| Field | Value |
+|-------|--------|
+| Domain | prod domain |
+| Scheme | `http` |
+| Forward Hostname | `10.0.0.77` |
+| Forward Port | `8021` |
+| SSL | request a certificate, force SSL |
+| Websockets | off |
+
+Open `https://<prod-domain>` and finish the WordPress installer (site title, admin user, admin password). That admin password is only in WordPress. It is not the MariaDB password.
+
+Then confirm a page refresh stays on `https://<prod-domain>` and does not bounce to `http://10.0.0.77:8021`.
+
+### 6. Start the other four hosts
+
+Copy the same compose file and the same env file to roslin, starbuck, apollo, and ai-server-4080. NFS must already be mounted. Then, on each:
+
+```bash
+cd /opt/apps/wordpress/<site>
+docker compose -p <site>_prod up -d
+```
+
+There is no second browser install. These containers attach to the database adama already populated.
+
+### 7. Load balance in NPM
+
+NPM's proxy host form accepts one forward address. The upstream list has to live in NPM's custom **http** config, which is included outside the server block. An `upstream` directive in the proxy host Advanced tab is rejected (`upstream directive is not allowed here`).
+
+On the NPM host (TrueNAS shell, or the NPM app shell), edit the file the container sees as `/data/nginx/custom/http.conf`. On a TrueNAS install that path is under the NPM app's data dataset, usually `.../npm/data/nginx/custom/http.conf`. Create `custom/` if it is missing.
+
+Append one block per site per env. Name the upstream after the compose project.
+
+```nginx
+upstream relawfirm_prod {
+    least_conn;
+    server 10.0.0.77:8021 max_fails=3 fail_timeout=10s;
+    server 10.0.0.176:8021 max_fails=3 fail_timeout=10s;
+    server 10.0.0.44:8021 max_fails=3 fail_timeout=10s;
+    server 10.0.0.7:8021 max_fails=3 fail_timeout=10s;
+    server 10.0.0.128:8021 max_fails=3 fail_timeout=10s;
+}
+```
+
+In the proxy host **Advanced** tab, set:
+
+```nginx
+location ^~ / {
+    proxy_pass http://relawfirm_prod;
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_set_header X-Forwarded-Host $host;
+    client_max_body_size 64m;
+}
+```
+
+`location ^~ /` is what sends traffic at the upstream. The Details tab can keep `10.0.0.77:8021` as the saved forward target; the `^~` location is the one that matches.
+
+Reload and check the config from the NPM container:
+
+```bash
+docker exec <npm-container> nginx -t
+docker exec <npm-container> nginx -s reload
+```
+
+`nginx -t` must succeed. A duplicate-location error means this Advanced snippet was pasted twice.
+
+`client_max_body_size 64m` is the upload limit. Raise it in this block if the site needs larger media. A 413 from NPM is this limit, not MariaDB.
+
+Details tab: SSL on, Force SSL on, HTTP/2 on. Block Common Exploits can stay on.
+
+### 8. Cron, on adama only
+
+`DISABLE_WP_CRON` is set, so page views do not run scheduled jobs. One host runs them. Same idea as the dj-queue worker that stays on adama.
+
+On adama, `/etc/cron.d/wordpress-<site>`:
+
+```cron
+* * * * * westfarn curl -fsS https://<prod-domain>/wp-cron.php?doing_wp_cron >/dev/null
+```
+
+Beta gets its own cron file hitting the beta URL. Do not add this cron on the other four hosts.
+
+### 9. Plugins, themes, and upgrades
+
+Plugins and themes installed from the dashboard are written to the NFS share, so every host sees them. Run one update at a time.
+
+WordPress core in this setup comes from the `wordpress:6-apache` image, not from the dashboard updater. The dashboard updater writes core into that container only. To move core, change `wordpress_image` (or that entry's `image`) and run `./scripts/provision.sh` so every host gets the new tag. Then, on each host:
+
+```bash
+cd /opt/apps/wordpress/<site>
+docker compose pull
+docker compose up -d
+```
+
+The project name is already in the compose file, so `-p` is optional. The database and `wp-content` stay on MariaDB and NFS.
+
+---
+
+## Beta
+
+Repeat sections 2–8 with the beta column of the table:
+
+- database `<site>_beta`
+- port **8022** for relawfirm (next free port for the next site)
+- its own env file, salts, NFS dataset, compose project, NPM host, and adama cron
+
+Do not point beta `WORDPRESS_DB_NAME` at the prod database.
+
+---
+
+## Next site
+
+MariaDB is already the shared server. For the next site, start at [Per site](#per-site) and do not redo [Already done](#already-done).
+
+1. Pick a short name, a prod domain, a beta domain, and the next two free host ports. Add a row to the [Ports](#ports) table in this file.
+2. `CREATE DATABASE` / `GRANT` to `'westfarn'@'%'` for `<site>` and `<site>_beta`.
+3. New env files and a new salt set per env.
+4. New NFS datasets, then two entries in [inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml) (`name`, `port`, `url`, `export`, `mount`). Run `./scripts/provision.sh` so every host mounts them and writes compose. Do not reuse another site's `wp-content`.
+5. Start adama first, browser install once, then the other four.
+6. New `upstream` block in `http.conf` and a new NPM proxy host.
+7. New cron file on adama only.
+
+---
+
+## Checks
+
+On each app host:
+
+```bash
+docker compose -p <site>_prod ps
+curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8021/
+mount | grep wordpress
+```
+
+From the control node, each backend should answer:
+
+```bash
+for ip in 10.0.0.77 10.0.0.176 10.0.0.44 10.0.0.7 10.0.0.128; do
+  curl -fsS -o /dev/null -w "$ip %{http_code}\n" "http://$ip:8021/"
+done
+```
+
+Then load `https://<prod-domain>` several times. A media file uploaded while you are on the site should still load after a refresh (that request may have been served by a different host).
+
+## When something fails
+
+| Symptom | Cause |
+|---------|--------|
+| MariaDB `Errcode: 13` on `/var/lib/mysql` | Dataset not writable by uid 999. See [Already done](#already-done). |
+| `ERROR 1133` on `GRANT` | User host does not match. Use the host from `SELECT user, host FROM mysql.user`. |
+| `Can't create database; database exists` | Database was created by the TrueNAS wizard or a previous run. Continue at `GRANT`. |
+| Container exits, cannot connect to MySQL | `WORDPRESS_DB_*` host, user, password, or database name. Port is `10.0.0.230:3306`. |
+| Installer or login redirects to `http://10.0.0.x:8021` | `WP_HOME` / `WP_SITEURL` are not the public URL, or NPM is not sending `X-Forwarded-Proto`. |
+| Logged out on refresh | Salts differ between hosts. Recreate with one shared env file. |
+| Upload succeeds, image 404s on refresh | That host does not have the NFS mount, or the mount is not the one in the compose file. |
+| `413 Request Entity Too Large` | `client_max_body_size` in the NPM `location ^~ /` block. |
+| Scheduled posts never publish | Cron is missing on adama, or `wp-cron.php` is not reachable at the public URL. |
+| Plugin shows on one host only | It was installed onto a local disk instead of the NFS `wp-content`. |

inventory/group_vars/all.yml

@@ -14,9 +14,13 @@ ufw_allowed_tcp_ports:
 
 # LiveKit on webservers. Signaling stays LAN (NPM → :7880). UDP must be
 # reachable from the internet via router DNAT to the NPM upstream host.
+# 8020 is abc_voice. That process uses host networking, so Docker does not
+# publish the port and UFW drops it unless it is listed here. NPM
+# (abc.voice.aimloperations.com) reaches it from the LAN.
 ufw_lan_tcp_ports:
   - 7880
   - 7881
+  - 8020
 ufw_extra_udp_ports:
   - 3478
   - "50000:60000"
@@ -59,6 +63,7 @@ secrets_dir: "{{ lookup('ansible.builtin.env', 'HOME') }}/Documents/secrets"
 web_static_root: /var/www
 
 # Catalog of deployable apps. host_apps (per host_vars) references these by name.
+# Optional host_apps.enabled: false skips deploy (stop.sh still compose-downs).
 app_catalog:
   company_site:
     type: django
@@ -114,6 +119,25 @@ app_catalog:
     # Document root served by nginx and written by `npm run build:<env>`.
     # {env} is replaced with the entry's env (beta/prod).
     webroot_pattern: "/var/www/{env}.chat.aimloperations/html"
+    # hesychia.ai Stripe/RevenueCat webhooks hit this vhost. Proxy the prefix
+    # to chat_backend so POST is not answered 405 by the SPA index.html.
+    api_backend: chat_backend
+    api_proxy_prefix: /api/finance/webhooks/
+    # /robots.txt, /sitemap.xml, and /llms.txt must not fall through to index.html.
+    exact_root_files: true
+    # Homepage was missing these. Enforcing CSP stays loose so Tianji, the API,
+    # websockets, and inline styles keep working. Report-Only is the strict policy.
+    security_headers: true
+    # Unknown URLs are HTTP 404 with public/404.html, not 200 + the SPA shell.
+    # Slash routes are directory index.html files from scripts/stampCanonicals.js.
+    # These extensionless routes are the matching files (keep in sync with
+    # llm-fe/src/llm-fe/utils/canonicals.js paths that do not end in /).
+    error_page_404: /404.html
+    html_exact_paths:
+      - /legal/terms-of-service
+      - /billing/success
+      - /billing/cancel
+      - /document_storage
   chat_backend:
     type: django
     repo: "{{ git_base_url }}/ai_ml_operations/chat_backend.git"
@@ -121,6 +145,11 @@ app_catalog:
     compose_file: docker-compose.prod.yml
     web_service: web
     migrate_cmd: "uv run python manage.py migrate --noinput"
+    # Host header the static-site webhook proxy sends. Must be in Django
+    # ALLOWED_HOSTS (see chat_backend DJANGO_ALLOWED_HOSTS).
+    public_hosts:
+      prod: chatbackend.aimloperations.com
+      beta: beta.chatbackend.aimloperations.com
   monica_site:
     type: django
     repo: "{{ git_base_url }}/ai_ml_operations/monica_site.git"
@@ -159,22 +188,30 @@ app_catalog:
   abc_be:
     type: django
     repo: "{{ git_base_url }}/GIS/abc_be.git"
-    default_branch: master
+    default_branch: Development
     compose_file: docker-compose.prod.yml
     web_service: web
     migrate_cmd: "uv run python manage.py migrate --noinput"
   abc_worker:
     type: django
     repo: "{{ git_base_url }}/GIS/abc_worker.git"
-    default_branch: master
+    default_branch: Development
     compose_file: docker-compose.prod.yml
     web_service: web
     # FastAPI — no Django migrations. `true` keeps the django deploy path happy.
     migrate_cmd: "true"
+  # Same git repo as abc_worker, second container (voice join). Not the text worker.
+  abc_voice:
+    type: django
+    repo: "{{ git_base_url }}/GIS/abc_worker.git"
+    default_branch: Development
+    compose_file: docker-compose.voice.yml
+    web_service: web
+    migrate_cmd: "true"
   abc_fe:
     type: node-static
     repo: "{{ git_base_url }}/GIS/abc_fe.git"
-    default_branch: master
+    default_branch: Development
     webroot_pattern: "/var/www/{env}.abc.aimloperations/html"
   livekit:
     type: django

inventory/host_vars/adama.yml

@@ -26,9 +26,11 @@ host_apps:
   - { name: url_shortening_service, env: beta, port: 8015 }
   - { name: college_craft, env: prod, port: 8006, compose_profiles: [worker] }
   - { name: college_craft, env: beta, port: 8016, compose_profiles: [worker] }
-  - { name: print_forge, env: prod, port: 8007, compose_profiles: [worker] }
+  # print_forge stopped (#33) — enabled: false keeps ports; stop.sh still matches.
-  - { name: print_forge, env: beta, port: 8019, compose_profiles: [worker] }
+  - { name: print_forge, env: prod, port: 8007, compose_profiles: [worker], enabled: false }
+  - { name: print_forge, env: beta, port: 8019, compose_profiles: [worker], enabled: false }
   - { name: abc_be, env: beta, port: 8017 }
   - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/ai-server-4080.yml

@@ -43,9 +43,10 @@ host_apps:
   - { name: url_shortening_service, env: beta, port: 8015 }
   - { name: college_craft, env: prod, port: 8006 }
   - { name: college_craft, env: beta, port: 8016 }
-  - { name: print_forge, env: prod, port: 8007 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
-  - { name: print_forge, env: beta, port: 8019 }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
   - { name: abc_be, env: beta, port: 8017 }
   - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/apollo.yml

@@ -22,9 +22,10 @@ host_apps:
   - { name: url_shortening_service, env: beta, port: 8015 }
   - { name: college_craft, env: prod, port: 8006 }
   - { name: college_craft, env: beta, port: 8016 }
-  - { name: print_forge, env: prod, port: 8007 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
-  - { name: print_forge, env: beta, port: 8019 }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
   - { name: abc_be, env: beta, port: 8017 }
   - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/roslin.yml

@@ -22,9 +22,10 @@ host_apps:
   - { name: url_shortening_service, env: beta, port: 8015 }
   - { name: college_craft, env: prod, port: 8006 }
   - { name: college_craft, env: beta, port: 8016 }
-  - { name: print_forge, env: prod, port: 8007 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
-  - { name: print_forge, env: beta, port: 8019 }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
   - { name: abc_be, env: beta, port: 8017 }
   - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }

inventory/host_vars/starbuck.yml

@@ -22,9 +22,10 @@ host_apps:
   - { name: url_shortening_service, env: beta, port: 8015 }
   - { name: college_craft, env: prod, port: 8006 }
   - { name: college_craft, env: beta, port: 8016 }
-  - { name: print_forge, env: prod, port: 8007 }
+  - { name: print_forge, env: prod, port: 8007, enabled: false }
-  - { name: print_forge, env: beta, port: 8019 }
+  - { name: print_forge, env: beta, port: 8019, enabled: false }
   - { name: abc_be, env: beta, port: 8017 }
   - { name: abc_worker, env: beta, port: 8018 }
+  - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }

inventory/wordpress_sites.yml

@@ -0,0 +1,32 @@
+---
+# WordPress sites on every webserver. scripts/provision.sh mounts wp-content
+# and writes one compose file per entry. Do not hand-edit /etc/fstab or the
+# generated compose file on the hosts.
+#
+# name:   site key. Prod is the short name (relawfirm). Beta is that name
+#         plus _beta (relawfirm_beta). Names use lowercase, numbers, underscore.
+# export: dataset path on the TrueNAS NFS server
+# mount:  local path compose bind-mounts to /var/www/html/wp-content
+# port:   host port published to container port 80
+# url:    public https URL (WP_HOME and WP_SITEURL). Empty skips compose.
+#
+# From name, provision derives:
+#   relawfirm      -> /opt/apps/wordpress/relawfirm/docker-compose.yml
+#                     project relawfirm_prod, env /opt/apps/env/relawfirm_prod.env
+#   relawfirm_beta -> /opt/apps/wordpress/relawfirm-beta/docker-compose.yml
+#                     project relawfirm_beta, env /opt/apps/env/relawfirm_beta.env
+
+wordpress_nfs_server: 10.0.0.230
+wordpress_nfs_opts: "nfsvers=4,tcp,_netdev"
+
+wordpress_sites:
+  - name: relawfirm
+    port: 8021
+    url: ""
+    export: /mnt/AIMLOperationsNAS/wordpress/relawfirm
+    mount: /mnt/wordpress/relawfirm/wp-content
+  - name: relawfirm_beta
+    port: 8022
+    url: http://relawfirm-preview.aimloperations.com
+    export: /mnt/AIMLOperationsNAS/wordpress/relawfirm_beta
+    mount: /mnt/wordpress/relawfirm_beta/wp-content

playbooks/deploy-apps.yml

@@ -1,8 +1,9 @@
 ---
 # Phase 2: CI-triggered app deployment.
 #
-# Runs against all webservers; each host deploys only the apps listed in its
+# Runs against all webservers; each host deploys only enabled apps listed in
-# host_apps. Deploy exactly one app+env at a pinned ref via extra-vars:
+# its host_apps (enabled: false is skipped). Deploy exactly one app+env at a
+# pinned ref via extra-vars:
 #
 #   ansible-playbook playbooks/deploy-apps.yml \
 #     -e app=company_site -e app_env=prod -e app_ref=<sha>

playbooks/site.yml

@@ -13,6 +13,9 @@
     - nodejs
     - gitea-key
     - tianji
+    # NFS wp-content and generated compose for inventory/wordpress_sites.yml.
+    - role: wordpress
+      tags: [wordpress]
 
 - name: Provision central observability stack
   hosts: ai-server-4080

playbooks/stop-apps.yml

@@ -0,0 +1,15 @@
+---
+# Stop one docker-compose app+env. Inverse of deploy-apps.yml start.
+#
+#   ansible-playbook playbooks/stop-apps.yml -e app=print_forge -e app_env=prod
+#
+# host_apps rows with enabled: false are still valid stop targets (so a
+# disabled app can be taken down). Node-static apps are skipped.
+- name: Stop applications
+  hosts: webservers
+  become: true
+  tasks:
+    - name: Stop requested compose apps
+      ansible.builtin.include_role:
+        name: app-deploy
+        tasks_from: stop.yml

requirements.yml

@@ -2,3 +2,5 @@
 collections:
   - name: community.general
     version: ">=8.0.0"
+  - name: ansible.posix
+    version: ">=1.5.0"

roles/app-deploy/tasks/main.yml

@@ -6,24 +6,21 @@
 # at branch master.
 #
 # host_apps (host_vars) lists what runs on THIS host; app_catalog (group_vars)
-# describes how each app is built.
+# describes how each app is built. host_apps.enabled: false (omit = true)
+# keeps the row for ports/stop.sh but skips deploy so CI cannot resurrect it.
 
 - name: Classify catalog by type
   ansible.builtin.set_fact:
     django_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'django') | map(attribute='key') | list }}"
     node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
 
-- name: Resolve deploy targets for {{ inventory_hostname }}
+- name: Resolve matching host_apps for {{ inventory_hostname }}
   ansible.builtin.set_fact:
-    deploy_targets: >-
+    matching_host_apps: >-
       {{ (host_apps | default([]) | selectattr('name', 'equalto', app | default('')) | selectattr('env', 'equalto', app_env | default('')) | list)
          if (app is defined and app_env is defined)
          else (host_apps | default([])) }}
 
-- name: Show deploy targets
-  ansible.builtin.debug:
-    msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
-
 # Explicit --app/--env with no host_apps match used to silently no-op Django
 # and still refresh web-static (looked like a "static-only" deploy). Fail loud.
 - name: Fail when requested app+env is not on this host
@@ -36,7 +33,32 @@
   when:
     - app is defined
     - app_env is defined
-    - deploy_targets | length == 0
+    - matching_host_apps | length == 0
+
+# rejectattr('enabled', …) raises when the key is omitted. Omit = enabled (#35).
+- name: Reset deploy targets
+  ansible.builtin.set_fact:
+    deploy_targets: []
+
+- name: "Drop disabled host_apps (enabled: false)"
+  ansible.builtin.set_fact:
+    deploy_targets: "{{ deploy_targets + [item] }}"
+  loop: "{{ matching_host_apps }}"
+  when: item.enabled | default(true)
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
+
+- name: Show skipped disabled apps
+  ansible.builtin.debug:
+    msg: "skip disabled {{ item.name }}/{{ item.env }} on {{ inventory_hostname }}"
+  loop: "{{ matching_host_apps }}"
+  when: not (item.enabled | default(true))
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
+
+- name: Show deploy targets
+  ansible.builtin.debug:
+    msg: "ref={{ app_ref | default('(per-app default branch)') }} targets={{ deploy_targets | map(attribute='name') | zip(deploy_targets | map(attribute='env')) | list }}"
 
 - name: Fail when requested app missing from app_catalog
   ansible.builtin.fail:

roles/app-deploy/tasks/stop.yml

@@ -0,0 +1,47 @@
+---
+# Stop one app+env. CI/manual must pass: app=<name> app_env=<beta|prod>
+#
+# Matches host_apps including enabled: false. Hosts without that row skip.
+# Django/compose → docker compose down (no -v). Node-static → message only.
+
+- name: Fail unless app and app_env extra-vars set
+  ansible.builtin.fail:
+    msg: "stop-apps.yml requires -e app=<name> -e app_env=<beta|prod>"
+  when: app is not defined or app_env is not defined
+
+- name: Classify catalog by type
+  ansible.builtin.set_fact:
+    django_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'django') | map(attribute='key') | list }}"
+    node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
+
+- name: Fail when requested app missing from app_catalog
+  ansible.builtin.fail:
+    msg: >-
+      app={{ app }} is not in app_catalog. Known: {{ app_catalog.keys() | list }}.
+  when: app not in app_catalog
+
+- name: Resolve stop targets for {{ inventory_hostname }}
+  ansible.builtin.set_fact:
+    stop_targets: "{{ host_apps | default([]) | selectattr('name', 'equalto', app) | selectattr('env', 'equalto', app_env) | list }}"
+
+- name: Skip {{ inventory_hostname }} — no host_apps row for {{ app }}/{{ app_env }}
+  ansible.builtin.debug:
+    msg: "skip {{ inventory_hostname }}: no host_apps entry for app={{ app }} env={{ app_env }}"
+  when: stop_targets | length == 0
+
+- name: Skip node-static {{ app }}/{{ app_env }} (no compose project)
+  ansible.builtin.debug:
+    msg: >-
+      {{ app }} is node-static. stop.sh only compose-downs Django apps.
+      Drop the host_apps row (or set enabled: false) and re-run deploy.sh
+      so web-static regenerates nginx without that vhost.
+  when:
+    - stop_targets | length > 0
+    - app in node_names
+
+- name: Stop Django (docker compose) apps
+  ansible.builtin.include_tasks: stop_django.yml
+  loop: "{{ stop_targets | selectattr('name', 'in', django_names) | list }}"
+  loop_control:
+    loop_var: app_item
+    label: "{{ app_item.name }}/{{ app_item.env }}"

roles/app-deploy/tasks/stop_django.yml

@@ -0,0 +1,72 @@
+---
+# Compose-down one Django app+env. Called per item with loop_var app_item.
+# No `down -v` — volumes stay. Checkout and secrets stay.
+
+- name: "stop[{{ app_item.name }}/{{ app_item.env }}] locals"
+  ansible.builtin.set_fact:
+    _spec: "{{ app_catalog[app_item.name] }}"
+    _src: "{{ apps_src_dir }}/{{ app_item.name }}_{{ app_item.env }}"
+    _project: "{{ app_item.name }}_{{ app_item.env }}"
+
+- name: "stop[{{ _project }}] checkout dir"
+  ansible.builtin.stat:
+    path: "{{ _src }}"
+  register: _src_stat
+
+- name: "stop[{{ _project }}] compose file"
+  ansible.builtin.stat:
+    path: "{{ _src }}/{{ _spec.compose_file }}"
+  register: _compose_stat
+  when: _src_stat.stat.exists
+
+- name: "stop[{{ _project }}] env file"
+  ansible.builtin.stat:
+    path: "{{ _src }}/.env"
+  register: _env_stat
+  when: _src_stat.stat.exists
+
+- name: "stop[{{ _project }}] use compose down"
+  ansible.builtin.set_fact:
+    _use_compose: "{{ _src_stat.stat.exists and _compose_stat is defined and _compose_stat.stat.exists }}"
+
+- name: "stop[{{ _project }}] compose down"
+  ansible.builtin.command:
+    cmd: >-
+      docker compose -f {{ _spec.compose_file }}
+      {{ '--env-file .env' if (_env_stat is defined and _env_stat.stat.exists) else '' }}
+      down
+    chdir: "{{ _src }}"
+  environment:
+    COMPOSE_PROJECT_NAME: "{{ _project }}"
+    WEB_PORT: "{{ app_item.port }}"
+    COMPOSE_PROFILES: "{{ (app_item.compose_profiles | default([])) | join(',') }}"
+  become: true
+  become_user: "{{ admin_user }}"
+  when: _use_compose | bool
+  changed_when: true
+
+- name: "stop[{{ _project }}] find leftover project containers"
+  ansible.builtin.command:
+    cmd: docker ps -aq --filter label=com.docker.compose.project={{ _project }}
+  register: _leftovers
+  changed_when: false
+  become: true
+  when: not (_use_compose | bool)
+
+- name: "stop[{{ _project }}] remove leftover project containers"
+  ansible.builtin.command:
+    cmd: "docker rm -f {{ _leftovers.stdout_lines | join(' ') }}"
+  become: true
+  when:
+    - not (_use_compose | bool)
+    - _leftovers is defined
+    - _leftovers.stdout_lines | length > 0
+  changed_when: true
+
+- name: "stop[{{ _project }}] nothing to stop (no checkout)"
+  ansible.builtin.debug:
+    msg: "no checkout at {{ _src }} and no containers for project {{ _project }}"
+  when:
+    - not (_use_compose | bool)
+    - _leftovers is defined
+    - _leftovers.stdout_lines | length == 0

roles/common/tasks/main.yml

@@ -23,4 +23,5 @@
       - gnupg
       - apt-transport-https
       - software-properties-common
+      - nfs-common
     state: present

roles/web-static/tasks/main.yml

@@ -6,9 +6,20 @@
   ansible.builtin.set_fact:
     _node_names: "{{ app_catalog | dict2items | selectattr('value.type', 'equalto', 'node-static') | map(attribute='key') | list }}"
 
+# rejectattr('enabled', …) raises when the key is omitted. Omit = enabled (#35).
+- name: web-static | reset static apps
+  ansible.builtin.set_fact:
+    _static_apps: []
+
 - name: web-static | this host's static apps
   ansible.builtin.set_fact:
-    _static_apps: "{{ host_apps | default([]) | selectattr('name', 'in', _node_names) | list }}"
+    _static_apps: "{{ _static_apps + [item] }}"
+  loop: "{{ host_apps | default([]) }}"
+  when:
+    - item.name in _node_names
+    - item.enabled | default(true)
+  loop_control:
+    label: "{{ item.name }}/{{ item.env }}"
 
 - name: web-static | configure and run
   when: _static_apps | length > 0

roles/web-static/templates/docker-compose.yml.j2

@@ -3,6 +3,9 @@ services:
   web-static:
     image: {{ web_static_image }}
     restart: unless-stopped
+    # Webhook locations proxy to chat_backend on the host-published port.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     logging:
       driver: json-file
       options:

roles/web-static/templates/nginx.conf.j2

@@ -17,6 +17,14 @@ log_format observability escape=json
     '"user_agent":"$http_user_agent"'
   '}';
 
+{% macro security_headers() %}
+    add_header X-Content-Type-Options "nosniff" always;
+    add_header X-Frame-Options "DENY" always;
+    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
+    add_header Content-Security-Policy "default-src 'self' https: wss: data: blob: 'unsafe-inline'; frame-ancestors 'none'; base-uri 'self'; object-src 'none'" always;
+    add_header Content-Security-Policy-Report-Only "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; font-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; base-uri 'self'; object-src 'none'; form-action 'self'" always;
+{% endmacro %}
+
 {% for a in _static_apps %}
 server {
     listen {{ a.port }};
@@ -29,6 +37,73 @@ server {
 
     root {{ (app_catalog[a.name].webroot_pattern | default(web_static_root ~ '/{env}_' ~ a.name)) | replace('{env}', a.env) }};
     index index.html;
+{% if app_catalog[a.name].security_headers | default(false) %}
+{{ security_headers() }}
+{% endif %}
+
+{%- set api_backend_name = app_catalog[a.name].api_backend | default('') %}
+{%- if api_backend_name %}
+{%- set api_proxy_prefix = app_catalog[a.name].api_proxy_prefix | default('/api/') %}
+{%- set api_match = namespace(port=0, ok=false) %}
+{%- for item in host_apps | default([]) %}
+{%- if item.name == api_backend_name and item.env == a.env and (item.enabled | default(true)) %}
+{%- set api_match.port = item.port %}
+{%- set api_match.ok = true %}
+{%- endif %}
+{%- endfor %}
+{%- if api_match.ok %}
+{%- set api_host = (app_catalog[api_backend_name].public_hosts | default({}))[a.env] | default('') %}
+
+    # Finance webhooks (Stripe) arrive on the public product host. The SPA
+    # try_files fallback is a static file and returns 405 for POST.
+    location {{ api_proxy_prefix }} {
+        proxy_pass http://host.docker.internal:{{ api_match.port }};
+        proxy_http_version 1.1;
+        proxy_set_header Host {{ api_host if api_host else '$host' }};
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_read_timeout 60s;
+        client_max_body_size 2m;
+    }
+{%- endif %}
+{%- endif %}
+
+{% if app_catalog[a.name].exact_root_files | default(false) %}
+    # Crawlers must get these files, never the SPA index.html fallback.
+    location = /robots.txt {
+        default_type text/plain;
+        try_files /robots.txt =404;
+    }
+    location = /sitemap.xml {
+        default_type application/xml;
+        try_files /sitemap.xml =404;
+    }
+    location = /llms.txt {
+        default_type text/plain;
+        try_files /llms.txt =404;
+    }
+{% endif %}
+
+{% if app_catalog[a.name].html_exact_paths | default([]) %}
+{% for p in app_catalog[a.name].html_exact_paths %}
+    location = {{ p }} {
+        default_type text/html;
+        try_files {{ p }} =404;
+    }
+{% endfor %}
+{% endif %}
+
+{% if a.name == 'chat_web_app' and a.env == 'prod' %}
+    # HeyCatch short links.
+    # f Facebook, h Threads, i Instagram, l LinkedIn, r Reddit, x X, y YouTube.
+    # Relative Location: this vhost listens on the container port. An absolute
+    # redirect would publish that port (hesychia.ai:8082).
+    location ~ ^/([a-z0-9])/?$ {
+        absolute_redirect off;
+        return 302 /?utm_source=heycatch&utm_campaign=$1;
+    }
+{% endif %}
 
 {% if app_catalog[a.name].error_page_404 | default('') %}
     error_page 404 {{ app_catalog[a.name].error_page_404 }};
@@ -45,6 +120,9 @@ server {
     location ~* \.(?:js|css|woff2?|png|jpg|jpeg|gif|svg|ico)$ {
         expires 7d;
         add_header Cache-Control "public";
+{% if app_catalog[a.name].security_headers | default(false) %}
+{{ security_headers() }}
+{% endif %}
     }
 }
 {% endfor %}

roles/wordpress/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+wordpress_image: "wordpress:6-apache"
+wordpress_root: "{{ apps_base_dir }}/wordpress"

roles/wordpress/tasks/main.yml

@@ -0,0 +1,77 @@
+---
+# Mount each site's shared wp-content and write its compose file.
+# List lives in inventory/wordpress_sites.yml. Containers are not started
+# here: the first browser install must run on one host only.
+
+- name: Load WordPress site list
+  ansible.builtin.include_vars:
+    file: "{{ inventory_dir }}/wordpress_sites.yml"
+
+- name: Install NFS client
+  ansible.builtin.apt:
+    name: nfs-common
+    state: present
+
+- name: Create WordPress mount points
+  ansible.builtin.file:
+    path: "{{ item.mount }}"
+    state: directory
+    mode: "0755"
+  loop: "{{ wordpress_sites }}"
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Mount WordPress wp-content
+  ansible.posix.mount:
+    src: "{{ wordpress_nfs_server }}:{{ item.export }}"
+    path: "{{ item.mount }}"
+    fstype: nfs
+    opts: "{{ wordpress_nfs_opts }}"
+    dump: "0"
+    passno: "0"
+    state: mounted
+  loop: "{{ wordpress_sites }}"
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Ensure WordPress env directory
+  ansible.builtin.file:
+    path: "{{ apps_env_dir }}"
+    state: directory
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0700"
+
+- name: Ensure WordPress compose directories
+  ansible.builtin.file:
+    path: "{{ wordpress_root }}/{{ item.name | regex_replace('_beta$', '-beta') }}"
+    state: directory
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0755"
+  loop: "{{ wordpress_sites }}"
+  when: (item.url | default('') | length) > 0
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Note WordPress sites waiting on a public URL
+  ansible.builtin.debug:
+    msg: >-
+      WordPress {{ item.name }} has no url. NFS is mounted.
+      Compose is not written until url is set in inventory/wordpress_sites.yml.
+  loop: "{{ wordpress_sites }}"
+  when: (item.url | default('') | length) == 0
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Write WordPress compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ wordpress_root }}/{{ item.name | regex_replace('_beta$', '-beta') }}/docker-compose.yml"
+    owner: "{{ admin_user }}"
+    group: "{{ admin_user }}"
+    mode: "0644"
+  loop: "{{ wordpress_sites }}"
+  when: (item.url | default('') | length) > 0
+  loop_control:
+    label: "{{ item.name }}"

roles/wordpress/templates/docker-compose.yml.j2

@@ -0,0 +1,21 @@
+# Managed by Ansible (roles/wordpress). Do not edit by hand.
+name: {{ item.name if (item.name is match('.*_beta$')) else (item.name ~ '_prod') }}
+services:
+  wordpress:
+    image: {{ item.image | default(wordpress_image) }}
+    restart: unless-stopped
+    env_file:
+      - {{ apps_env_dir }}/{{ item.name if (item.name is match('.*_beta$')) else (item.name ~ '_prod') }}.env
+    ports:
+      - "{{ item.port }}:80"
+    environment:
+      WORDPRESS_CONFIG_EXTRA: |
+        define('WP_HOME', '{{ item.url }}');
+        define('WP_SITEURL', '{{ item.url }}');
+        define('DISABLE_WP_CRON', true);
+        if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && strpos($_SERVER['HTTP_X_FORWARDED_PROTO'], 'https') !== false) {
+            $_SERVER['HTTPS'] = 'on';
+        }
+        define('FORCE_SSL_ADMIN', true);
+    volumes:
+      - {{ item.mount }}:/var/www/html/wp-content

scripts/provision.sh

@@ -14,6 +14,11 @@ Usage: $(basename "$0") [HOST] [OPTIONS]
 Provision server(s) with site.yml.
 
 Applies: common, ufw, docker, nodejs, gitea-key, tianji, alloy (every host).
+common installs nfs-common with the other base packages.
+Also mounts WordPress wp-content (NFS) and, when that entry has a public
+url, writes its compose file. Site list is inventory/wordpress_sites.yml.
+Add a site there; the next provision mounts it on every webserver.
+Containers are not started.
 On ai-server-4080 also: observability (Loki + Prometheus + Grafana) when
 observability_stack is true, and SearxNG when searxng_stack is true
 (chat_backend grounded search on :8088).

scripts/stop.sh

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
+
+LIMIT=""
+EXTRA_ARGS=()
+EXTRA_VARS=()
+APP=""
+APP_ENV=""
+
+usage() {
+  cat <<EOF
+Usage: $(basename "$0") [HOST] --app NAME --env ENV [OPTIONS]
+
+Stop a docker-compose app (compose down). Inverse of deploy.sh.
+
+Does not remove volumes, checkouts, or secrets. Next deploy.sh of the same
+app+env will start it again unless host_apps sets enabled: false.
+
+Node-static apps have no compose project — stop.sh skips them.
+
+  HOST              Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
+
+Options:
+  --app NAME        Required. App to stop (same names as deploy.sh)
+  --env ENV         Required. Environment: beta or prod
+  --check           Dry run
+  --diff            Show diffs
+  --extra-vars V    Pass raw extra vars
+  -h, --help        Show this help
+
+Examples:
+  $(basename "$0") --app print_forge --env prod
+  $(basename "$0") --app print_forge --env beta
+  $(basename "$0") adama --app chat_backend --env beta --check
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    --app)
+      APP="$2"
+      EXTRA_VARS+=(-e "app=$2")
+      shift 2
+      ;;
+    --env)
+      APP_ENV="$2"
+      EXTRA_VARS+=(-e "app_env=$2")
+      shift 2
+      ;;
+    --check)
+      EXTRA_ARGS+=(--check)
+      shift
+      ;;
+    --diff)
+      EXTRA_ARGS+=(--diff)
+      shift
+      ;;
+    --extra-vars)
+      EXTRA_VARS+=(-e "$2")
+      shift 2
+      ;;
+    adama|roslin|starbuck|apollo|ai-server-4080)
+      LIMIT="$1"
+      shift
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 1
+      ;;
+    esac
+done
+
+if [[ -z "$APP" || -z "$APP_ENV" ]]; then
+  echo "error: --app and --env are required" >&2
+  usage >&2
+  exit 1
+fi
+
+CMD=(ansible-playbook playbooks/stop-apps.yml "${EXTRA_ARGS[@]}" "${EXTRA_VARS[@]}")
+if [[ -n "$LIMIT" ]]; then
+  CMD+=(--limit "$LIMIT")
+  echo "==> Stopping ${APP}/${APP_ENV} on: $LIMIT"
+else
+  echo "==> Stopping ${APP}/${APP_ENV} on all webservers"
+fi
+
+exec "${CMD[@]}"