server-infra/docs/WORDPRESS.md

# Adding a WordPress site

Manual procedure for a WordPress site on all five web servers, one MariaDB on
TrueNAS, and Nginx Proxy Manager (NPM) in front. Ansible does **not** start
the WordPress containers. When you run `scripts/provision.sh` it mounts each
site's `wp-content` and, once that entry has a public `url`, writes its
compose file from
[inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml). Repeat
[Per site](#per-site) for every new site. Do not install another MariaDB.

## Where things live

| Piece | Where |
|-------|--------|
| MariaDB | TrueNAS `10.0.0.230`, port **3306**, LAN only |
| MariaDB login | `westfarn` @ `%` (the host the TrueNAS app created) |
| NPM | `10.0.0.230` (not managed by Ansible) |
| App hosts | adama `10.0.0.77`, roslin `10.0.0.176`, starbuck `10.0.0.44`, apollo `10.0.0.7`, ai-server-4080 `10.0.0.128` |
| Secrets | `~/Documents/secrets/wordpress/` on the control node. Never commit these. |
| Shared files | one NFS dataset per site per env, mounted on every app host |

Prod and beta never share a database, an NFS dataset, a port, or a domain.
The five hosts of one env do share that env's database and `wp-content`.

## Already done

Leave this alone when adding the next site.

- MariaDB app is running. Its host-path dataset is owned by uid **999** (`netdata`). Without that, the container crash-loops with `Errcode: 13 "Permission denied"` on `/var/lib/mysql`.
- Login `westfarn` exists at host `%`, not at `10.0.0.%`. `GRANT` to `'westfarn'@'10.0.0.%'` fails with `ERROR 1133` because that row is not in `mysql.user`.
- Databases `relawfirm` and `relawfirm_beta` already exist.

Confirm from **Apps → mariadb → Shell**:

```bash
mariadb -u root -p
```

```sql
SELECT user, host FROM mysql.user WHERE user = 'westfarn';
SHOW DATABASES;
SHOW GRANTS FOR 'westfarn'@'%';
```

`relawfirm` still needs the grants below if `SHOW GRANTS` does not list both databases. New sites use the same `GRANT` shape with their own database names.

```sql
GRANT ALL PRIVILEGES ON relawfirm.* TO 'westfarn'@'%';
GRANT ALL PRIVILEGES ON relawfirm_beta.* TO 'westfarn'@'%';
FLUSH PRIVILEGES;
```

## Ports

Same host port on every app server for a given site and env. NPM balances
`adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT` + `ai-server-4080:PORT`.

These are not in the Ansible `host_apps` tables. Do not reuse a port from
[IMPLEMENTATION.md](../IMPLEMENTATION.md) (8000–8020, **8023–8024**, and 8080–8088
are taken; **8020** is `abc_voice`, **8023/8024** are `disney_hotel_service`).
Next WordPress site starts at **8025**.

| Site | prod port | beta port | prod database | beta database |
|------|-----------|-----------|---------------|---------------|
| relawfirm | **8021** | **8022** | `relawfirm` | `relawfirm_beta` |
| *(next site)* | next free | next free | `<site>` | `<site>_beta` |

---

## Per site

Fill this in before touching a host. `relawfirm` is the first site.

| | prod | beta |
|--|------|------|
| Public URL | `https://<prod-domain>` | `https://<beta-domain>` |
| Database | `relawfirm` | `relawfirm_beta` |
| Host port | `8021` | `8022` |
| NFS dataset | `/mnt/AIMLOperationsNAS/wordpress/relawfirm` | `/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta` |
| Mount on app hosts | `/mnt/wordpress/relawfirm/wp-content` | `/mnt/wordpress/relawfirm_beta/wp-content` |
| Env file | `relawfirm_prod.env` | `relawfirm_beta.env` |
| Compose project | `relawfirm_prod` | `relawfirm_beta` |

`<site>` below means that short name (`relawfirm`, then whatever the next site is). Use lowercase letters, numbers, and underscore only.

### 1. Databases

Skip this for `relawfirm` / `relawfirm_beta`. They already exist.

In the MariaDB shell (`mariadb -u root -p`):

```sql
CREATE DATABASE <site> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE DATABASE <site>_beta CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON <site>.* TO 'westfarn'@'%';
GRANT ALL PRIVILEGES ON <site>_beta.* TO 'westfarn'@'%';
FLUSH PRIVILEGES;
```

The account in `TO` must match `SELECT user, host FROM mysql.user`. Today that is `'westfarn'@'%'`. Do not create another MariaDB user unless you want a separate login; this matches the shared Postgres user.

### 2. Salts and env files

On the control node, generate salts **once per env**:

```bash
curl -fsS https://api.wordpress.org/secret-key/1.1/salt/
```

Copy the same eight values into the env file on every app host for that env. A different salt on one host logs people out as soon as NPM sends them to another host. Prod and beta should not share salts.

Create the file locally (mode `600`). Do not commit it.

`~/Documents/secrets/wordpress/<site>_prod.env`:

```bash
WORDPRESS_DB_HOST=10.0.0.230:3306
WORDPRESS_DB_USER=westfarn
WORDPRESS_DB_PASSWORD=<mariadb password for westfarn>
WORDPRESS_DB_NAME=<site>
WORDPRESS_AUTH_KEY=<from salt generator>
WORDPRESS_SECURE_AUTH_KEY=<from salt generator>
WORDPRESS_LOGGED_IN_KEY=<from salt generator>
WORDPRESS_NONCE_KEY=<from salt generator>
WORDPRESS_AUTH_SALT=<from salt generator>
WORDPRESS_SECURE_AUTH_SALT=<from salt generator>
WORDPRESS_LOGGED_IN_SALT=<from salt generator>
WORDPRESS_NONCE_SALT=<from salt generator>
```

`<site>_beta.env` is the same except `WORDPRESS_DB_NAME=<site>_beta` and a second set of salts.

`WORDPRESS_DB_PASSWORD` is the MariaDB password. The WordPress admin password is chosen later in the browser, on the first host only.

### 3. Shared `wp-content` (TrueNAS NFS)

Uploads, plugins, and themes are files. With five hosts, those files have to live on one share. A private Docker volume on each host 404s media as soon as NPM moves the next request.

Create **one dataset per site per env**. Preset **Generic** (this dataset is an NFS share, not the MariaDB host path). Leave record size at the default. Case sensitive. Do not add an SMB share.

Example paths:

```text
/mnt/AIMLOperationsNAS/wordpress/relawfirm
/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta
```

**Shares → Unix Shares (NFS) → Add**, one share per dataset:

| Setting | Value |
|---------|--------|
| Path | the dataset |
| Authorized Networks | `10.0.0.0/24` |
| Maproot User | `root` |
| Maproot Group | `root` |

Maproot root lets the WordPress entrypoint create files on an empty share. The export is limited to the LAN. Do not forward NFS to the internet.

Do not edit `/etc/fstab` by hand. Add one entry per env to
[inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml), then mount
every host:

```bash
./scripts/provision.sh
```

`export` is the TrueNAS dataset. `mount` is the path on the app hosts. Prod
and beta are separate entries (beta export ends in `_beta`, not `-beta`).

```yaml
wordpress_sites:
  - name: <site>
    port: 8021
    url: https://<prod-domain>
    export: /mnt/AIMLOperationsNAS/wordpress/<site>
    mount: /mnt/wordpress/<site>/wp-content
  - name: <site>_beta
    port: 8022
    url: https://<beta-domain>
    export: /mnt/AIMLOperationsNAS/wordpress/<site>_beta
    mount: /mnt/wordpress/<site>_beta/wp-content
```

`url` is the public NPM URL. Leave it empty and provision still mounts NFS,
but it does not write a compose file. Set it, then provision again.

One host is enough to check the new lines:

```bash
./scripts/provision.sh adama
mount | grep wordpress
touch /mnt/wordpress/<site>/wp-content/.write-test && rm /mnt/wordpress/<site>/wp-content/.write-test
```

The first container to start copies the stock themes and plugins into this directory. Mount it on all five hosts **before** that start, and start only one host while the directory is still empty. Beta uses its own dataset and its own yaml entry.

### 4. Compose file

Do not copy a compose file by hand. The template is
[roles/wordpress/templates/docker-compose.yml.j2](../roles/wordpress/templates/docker-compose.yml.j2).
`scripts/provision.sh` renders one file per inventory entry on every app host.
Secrets stay in the env file. `WP_HOME` and `WP_SITEURL` come from that
entry's `url`, not from `http://10.0.0.77:8021`.

| `name` | Compose file | Project (`name:` in the file) | Env file |
|--------|--------------|--------------------------------|----------|
| `<site>` | `/opt/apps/wordpress/<site>/docker-compose.yml` | `<site>_prod` | `/opt/apps/env/<site>_prod.env` |
| `<site>_beta` | `/opt/apps/wordpress/<site>-beta/docker-compose.yml` | `<site>_beta` | `/opt/apps/env/<site>_beta.env` |

Prod and beta do not share a directory. Image default is `wordpress:6-apache`
(`wordpress_image` in the role). Override with `image:` on one entry. The
bind mount is that entry's `mount`.

Copy the env file onto each host (mode `600`) before the first start.
Provision creates `/opt/apps/env` (mode `700`) but does not copy secrets.

### 5. Install in the browser, one host only

Start **adama** first. Leave the other four stopped until the installer has finished. Five installers against one database will fight.

On adama:

```bash
cd /opt/apps/wordpress/<site>
docker compose -p <site>_prod up -d
docker compose -p <site>_prod logs -f
```

Wait until Apache is up. The log should not show a database connection error.

In NPM, add a normal Proxy Host (single forward target) **before** opening the installer:

| Field | Value |
|-------|--------|
| Domain | prod domain |
| Scheme | `http` |
| Forward Hostname | `10.0.0.77` |
| Forward Port | `8021` |
| SSL | request a certificate, force SSL |
| Websockets | off |

Open `https://<prod-domain>` and finish the WordPress installer (site title, admin user, admin password). That admin password is only in WordPress. It is not the MariaDB password.

Then confirm a page refresh stays on `https://<prod-domain>` and does not bounce to `http://10.0.0.77:8021`.

### 6. Start the other four hosts

Copy the same compose file and the same env file to roslin, starbuck, apollo, and ai-server-4080. NFS must already be mounted. Then, on each:

```bash
cd /opt/apps/wordpress/<site>
docker compose -p <site>_prod up -d
```

There is no second browser install. These containers attach to the database adama already populated.

### 7. Load balance in NPM

NPM's proxy host form accepts one forward address. The upstream list has to live in NPM's custom **http** config, which is included outside the server block. An `upstream` directive in the proxy host Advanced tab is rejected (`upstream directive is not allowed here`).

On the NPM host (TrueNAS shell, or the NPM app shell), edit the file the container sees as `/data/nginx/custom/http.conf`. On a TrueNAS install that path is under the NPM app's data dataset, usually `.../npm/data/nginx/custom/http.conf`. Create `custom/` if it is missing.

Append one block per site per env. Name the upstream after the compose project.

```nginx
upstream relawfirm_prod {
    least_conn;
    server 10.0.0.77:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.176:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.44:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.7:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.128:8021 max_fails=3 fail_timeout=10s;
}
```

In the proxy host **Advanced** tab, set:

```nginx
location ^~ / {
    proxy_pass http://relawfirm_prod;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $host;
    client_max_body_size 64m;
}
```

`location ^~ /` is what sends traffic at the upstream. The Details tab can keep `10.0.0.77:8021` as the saved forward target; the `^~` location is the one that matches.

Reload and check the config from the NPM container:

```bash
docker exec <npm-container> nginx -t
docker exec <npm-container> nginx -s reload
```

`nginx -t` must succeed. A duplicate-location error means this Advanced snippet was pasted twice.

`client_max_body_size 64m` is the upload limit. Raise it in this block if the site needs larger media. A 413 from NPM is this limit, not MariaDB.

Details tab: SSL on, Force SSL on, HTTP/2 on. Block Common Exploits can stay on.

### 8. Cron, on adama only

`DISABLE_WP_CRON` is set, so page views do not run scheduled jobs. One host runs them. Same idea as the dj-queue worker that stays on adama.

On adama, `/etc/cron.d/wordpress-<site>`:

```cron
* * * * * westfarn curl -fsS https://<prod-domain>/wp-cron.php?doing_wp_cron >/dev/null
```

Beta gets its own cron file hitting the beta URL. Do not add this cron on the other four hosts.

### 9. Plugins, themes, and upgrades

Plugins and themes installed from the dashboard are written to the NFS share, so every host sees them. Run one update at a time.

WordPress core in this setup comes from the `wordpress:6-apache` image, not from the dashboard updater. The dashboard updater writes core into that container only. To move core, change `wordpress_image` (or that entry's `image`) and run `./scripts/provision.sh` so every host gets the new tag. Then, on each host:

```bash
cd /opt/apps/wordpress/<site>
docker compose pull
docker compose up -d
```

The project name is already in the compose file, so `-p` is optional. The database and `wp-content` stay on MariaDB and NFS.

---

## Beta

Repeat sections 2–8 with the beta column of the table:

- database `<site>_beta`
- port **8022** for relawfirm (next free WordPress port is **8025**; **8023/8024** are `disney_hotel_service`)
- its own env file, salts, NFS dataset, compose project, NPM host, and adama cron

Do not point beta `WORDPRESS_DB_NAME` at the prod database.

---

## Next site

MariaDB is already the shared server. For the next site, start at [Per site](#per-site) and do not redo [Already done](#already-done).

1. Pick a short name, a prod domain, a beta domain, and the next two free host ports. Add a row to the [Ports](#ports) table in this file.
2. `CREATE DATABASE` / `GRANT` to `'westfarn'@'%'` for `<site>` and `<site>_beta`.
3. New env files and a new salt set per env.
4. New NFS datasets, then two entries in [inventory/wordpress_sites.yml](../inventory/wordpress_sites.yml) (`name`, `port`, `url`, `export`, `mount`). Run `./scripts/provision.sh` so every host mounts them and writes compose. Do not reuse another site's `wp-content`.
5. Start adama first, browser install once, then the other four.
6. New `upstream` block in `http.conf` and a new NPM proxy host.
7. New cron file on adama only.

---

## Checks

On each app host:

```bash
docker compose -p <site>_prod ps
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8021/
mount | grep wordpress
```

From the control node, each backend should answer:

```bash
for ip in 10.0.0.77 10.0.0.176 10.0.0.44 10.0.0.7 10.0.0.128; do
  curl -fsS -o /dev/null -w "$ip %{http_code}\n" "http://$ip:8021/"
done
```

Then load `https://<prod-domain>` several times. A media file uploaded while you are on the site should still load after a refresh (that request may have been served by a different host).

## When something fails

| Symptom | Cause |
|---------|--------|
| MariaDB `Errcode: 13` on `/var/lib/mysql` | Dataset not writable by uid 999. See [Already done](#already-done). |
| `ERROR 1133` on `GRANT` | User host does not match. Use the host from `SELECT user, host FROM mysql.user`. |
| `Can't create database; database exists` | Database was created by the TrueNAS wizard or a previous run. Continue at `GRANT`. |
| Container exits, cannot connect to MySQL | `WORDPRESS_DB_*` host, user, password, or database name. Port is `10.0.0.230:3306`. |
| Installer or login redirects to `http://10.0.0.x:8021` | `WP_HOME` / `WP_SITEURL` are not the public URL, or NPM is not sending `X-Forwarded-Proto`. |
| Logged out on refresh | Salts differ between hosts. Recreate with one shared env file. |
| Upload succeeds, image 404s on refresh | That host does not have the NFS mount, or the mount is not the one in the compose file. |
| `413 Request Entity Too Large` | `client_max_body_size` in the NPM `location ^~ /` block. |
| Scheduled posts never publish | Cron is missing on adama, or `wp-cron.php` is not reachable at the public URL. |
| Plugin shows on one host only | It was installed onto a local disk instead of the NFS `wp-content`. |