server-infra/docs/WORDPRESS.md

Adding a WordPress site

Manual procedure for a WordPress site on all five web servers, one MariaDB on TrueNAS, and Nginx Proxy Manager (NPM) in front. Ansible does not start the WordPress containers. When you run scripts/provision.sh it mounts each site's wp-content and, once that entry has a public url, writes its compose file from inventory/wordpress_sites.yml. Repeat Per site for every new site. Do not install another MariaDB.

Where things live

Piece	Where
MariaDB	TrueNAS 10.0.0.230, port 3306, LAN only
MariaDB login	westfarn @ % (the host the TrueNAS app created)
NPM	10.0.0.230 (not managed by Ansible)
App hosts	adama 10.0.0.77, roslin 10.0.0.176, starbuck 10.0.0.44, apollo 10.0.0.7, ai-server-4080 10.0.0.128
Secrets	~/Documents/secrets/wordpress/ on the control node. Never commit these.
Shared files	one NFS dataset per site per env, mounted on every app host

Prod and beta never share a database, an NFS dataset, a port, or a domain. The five hosts of one env do share that env's database and wp-content.

Already done

Leave this alone when adding the next site.

• MariaDB app is running. Its host-path dataset is owned by uid 999 (netdata). Without that, the container crash-loops with Errcode: 13 "Permission denied" on /var/lib/mysql.
• Login westfarn exists at host %, not at 10.0.0.%. GRANT to 'westfarn'@'10.0.0.%' fails with ERROR 1133 because that row is not in mysql.user.
• Databases relawfirm and relawfirm_beta already exist.

Confirm from Apps → mariadb → Shell:

mariadb -u root -p

SELECT user, host FROM mysql.user WHERE user = 'westfarn';
SHOW DATABASES;
SHOW GRANTS FOR 'westfarn'@'%';

relawfirm still needs the grants below if SHOW GRANTS does not list both databases. New sites use the same GRANT shape with their own database names.

GRANT ALL PRIVILEGES ON relawfirm.* TO 'westfarn'@'%';
GRANT ALL PRIVILEGES ON relawfirm_beta.* TO 'westfarn'@'%';
FLUSH PRIVILEGES;

Ports

Same host port on every app server for a given site and env. NPM balances adama:PORT + roslin:PORT + starbuck:PORT + apollo:PORT + ai-server-4080:PORT.

These are not in the Ansible host_apps tables. Do not reuse a port from IMPLEMENTATION.md (8000–8020, 8023–8024, and 8080–8088 are taken; 8020 is abc_voice, 8023/8024 are disney_hotel_service). Next WordPress site starts at 8025.

Site	prod port	beta port	prod database	beta database
relawfirm	8021	8022	relawfirm	relawfirm_beta
(next site)	next free	next free	<site>	<site>_beta

---

Per site

Fill this in before touching a host. relawfirm is the first site.

	prod	beta
Public URL	https://<prod-domain>	https://<beta-domain>
Database	relawfirm	relawfirm_beta
Host port	8021	8022
NFS dataset	/mnt/AIMLOperationsNAS/wordpress/relawfirm	/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta
Mount on app hosts	/mnt/wordpress/relawfirm/wp-content	/mnt/wordpress/relawfirm_beta/wp-content
Env file	relawfirm_prod.env	relawfirm_beta.env
Compose project	relawfirm_prod	relawfirm_beta

<site> below means that short name (relawfirm, then whatever the next site is). Use lowercase letters, numbers, and underscore only.

1. Databases

Skip this for relawfirm / relawfirm_beta. They already exist.

In the MariaDB shell (mariadb -u root -p):

CREATE DATABASE <site> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE DATABASE <site>_beta CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON <site>.* TO 'westfarn'@'%';
GRANT ALL PRIVILEGES ON <site>_beta.* TO 'westfarn'@'%';
FLUSH PRIVILEGES;

The account in TO must match SELECT user, host FROM mysql.user. Today that is 'westfarn'@'%'. Do not create another MariaDB user unless you want a separate login; this matches the shared Postgres user.

2. Salts and env files

On the control node, generate salts once per env:

curl -fsS https://api.wordpress.org/secret-key/1.1/salt/

Copy the same eight values into the env file on every app host for that env. A different salt on one host logs people out as soon as NPM sends them to another host. Prod and beta should not share salts.

Create the file locally (mode 600). Do not commit it.

~/Documents/secrets/wordpress/<site>_prod.env:

WORDPRESS_DB_HOST=10.0.0.230:3306
WORDPRESS_DB_USER=westfarn
WORDPRESS_DB_PASSWORD=<mariadb password for westfarn>
WORDPRESS_DB_NAME=<site>
WORDPRESS_AUTH_KEY=<from salt generator>
WORDPRESS_SECURE_AUTH_KEY=<from salt generator>
WORDPRESS_LOGGED_IN_KEY=<from salt generator>
WORDPRESS_NONCE_KEY=<from salt generator>
WORDPRESS_AUTH_SALT=<from salt generator>
WORDPRESS_SECURE_AUTH_SALT=<from salt generator>
WORDPRESS_LOGGED_IN_SALT=<from salt generator>
WORDPRESS_NONCE_SALT=<from salt generator>

<site>_beta.env is the same except WORDPRESS_DB_NAME=<site>_beta and a second set of salts.

WORDPRESS_DB_PASSWORD is the MariaDB password. The WordPress admin password is chosen later in the browser, on the first host only.

3. Shared wp-content (TrueNAS NFS)

Uploads, plugins, and themes are files. With five hosts, those files have to live on one share. A private Docker volume on each host 404s media as soon as NPM moves the next request.

Create one dataset per site per env. Preset Generic (this dataset is an NFS share, not the MariaDB host path). Leave record size at the default. Case sensitive. Do not add an SMB share.

Example paths:

/mnt/AIMLOperationsNAS/wordpress/relawfirm
/mnt/AIMLOperationsNAS/wordpress/relawfirm_beta

Shares → Unix Shares (NFS) → Add, one share per dataset:

Setting	Value
Path	the dataset
Authorized Networks	10.0.0.0/24
Maproot User	root
Maproot Group	root

Maproot root lets the WordPress entrypoint create files on an empty share. The export is limited to the LAN. Do not forward NFS to the internet.

Do not edit /etc/fstab by hand. Add one entry per env to inventory/wordpress_sites.yml, then mount every host:

./scripts/provision.sh

export is the TrueNAS dataset. mount is the path on the app hosts. Prod and beta are separate entries (beta export ends in _beta, not -beta).

wordpress_sites:
  - name: <site>
    port: 8021
    url: https://<prod-domain>
    export: /mnt/AIMLOperationsNAS/wordpress/<site>
    mount: /mnt/wordpress/<site>/wp-content
  - name: <site>_beta
    port: 8022
    url: https://<beta-domain>
    export: /mnt/AIMLOperationsNAS/wordpress/<site>_beta
    mount: /mnt/wordpress/<site>_beta/wp-content

url is the public NPM URL. Leave it empty and provision still mounts NFS, but it does not write a compose file. Set it, then provision again.

One host is enough to check the new lines:

./scripts/provision.sh adama
mount | grep wordpress
touch /mnt/wordpress/<site>/wp-content/.write-test && rm /mnt/wordpress/<site>/wp-content/.write-test

The first container to start copies the stock themes and plugins into this directory. Mount it on all five hosts before that start, and start only one host while the directory is still empty. Beta uses its own dataset and its own yaml entry.

4. Compose file

Do not copy a compose file by hand. The template is roles/wordpress/templates/docker-compose.yml.j2. scripts/provision.sh renders one file per inventory entry on every app host. Secrets stay in the env file. WP_HOME and WP_SITEURL come from that entry's url, not from http://10.0.0.77:8021.

name	Compose file	Project (name: in the file)	Env file
<site>	/opt/apps/wordpress/<site>/docker-compose.yml	<site>_prod	/opt/apps/env/<site>_prod.env
<site>_beta	/opt/apps/wordpress/<site>-beta/docker-compose.yml	<site>_beta	/opt/apps/env/<site>_beta.env

Prod and beta do not share a directory. Image default is wordpress:6-apache (wordpress_image in the role). Override with image: on one entry. The bind mount is that entry's mount.

Copy the env file onto each host (mode 600) before the first start. Provision creates /opt/apps/env (mode 700) but does not copy secrets.

5. Install in the browser, one host only

Start adama first. Leave the other four stopped until the installer has finished. Five installers against one database will fight.

On adama:

cd /opt/apps/wordpress/<site>
docker compose -p <site>_prod up -d
docker compose -p <site>_prod logs -f

Wait until Apache is up. The log should not show a database connection error.

In NPM, add a normal Proxy Host (single forward target) before opening the installer:

Field	Value
Domain	prod domain
Scheme	http
Forward Hostname	10.0.0.77
Forward Port	8021
SSL	request a certificate, force SSL
Websockets	off

Open https://<prod-domain> and finish the WordPress installer (site title, admin user, admin password). That admin password is only in WordPress. It is not the MariaDB password.

Then confirm a page refresh stays on https://<prod-domain> and does not bounce to http://10.0.0.77:8021.

6. Start the other four hosts

Copy the same compose file and the same env file to roslin, starbuck, apollo, and ai-server-4080. NFS must already be mounted. Then, on each:

cd /opt/apps/wordpress/<site>
docker compose -p <site>_prod up -d

There is no second browser install. These containers attach to the database adama already populated.

7. Load balance in NPM

NPM's proxy host form accepts one forward address. The upstream list has to live in NPM's custom http config, which is included outside the server block. An upstream directive in the proxy host Advanced tab is rejected (upstream directive is not allowed here).

On the NPM host (TrueNAS shell, or the NPM app shell), edit the file the container sees as /data/nginx/custom/http.conf. On a TrueNAS install that path is under the NPM app's data dataset, usually .../npm/data/nginx/custom/http.conf. Create custom/ if it is missing.

Append one block per site per env. Name the upstream after the compose project.

upstream relawfirm_prod {
    least_conn;
    server 10.0.0.77:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.176:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.44:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.7:8021 max_fails=3 fail_timeout=10s;
    server 10.0.0.128:8021 max_fails=3 fail_timeout=10s;
}

In the proxy host Advanced tab, set:

location ^~ / {
    proxy_pass http://relawfirm_prod;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $host;
    client_max_body_size 64m;
}

location ^~ / is what sends traffic at the upstream. The Details tab can keep 10.0.0.77:8021 as the saved forward target; the ^~ location is the one that matches.

Reload and check the config from the NPM container:

docker exec <npm-container> nginx -t
docker exec <npm-container> nginx -s reload

nginx -t must succeed. A duplicate-location error means this Advanced snippet was pasted twice.

client_max_body_size 64m is the upload limit. Raise it in this block if the site needs larger media. A 413 from NPM is this limit, not MariaDB.

Details tab: SSL on, Force SSL on, HTTP/2 on. Block Common Exploits can stay on.

8. Cron, on adama only

DISABLE_WP_CRON is set, so page views do not run scheduled jobs. One host runs them. Same idea as the dj-queue worker that stays on adama.

On adama, /etc/cron.d/wordpress-<site>:

* * * * * westfarn curl -fsS https://<prod-domain>/wp-cron.php?doing_wp_cron >/dev/null

Beta gets its own cron file hitting the beta URL. Do not add this cron on the other four hosts.

9. Plugins, themes, and upgrades

Plugins and themes installed from the dashboard are written to the NFS share, so every host sees them. Run one update at a time.

WordPress core in this setup comes from the wordpress:6-apache image, not from the dashboard updater. The dashboard updater writes core into that container only. To move core, change wordpress_image (or that entry's image) and run ./scripts/provision.sh so every host gets the new tag. Then, on each host:

cd /opt/apps/wordpress/<site>
docker compose pull
docker compose up -d

The project name is already in the compose file, so -p is optional. The database and wp-content stay on MariaDB and NFS.

---

Beta

Repeat sections 2–8 with the beta column of the table:

• database <site>_beta
• port 8022 for relawfirm (next free WordPress port is 8025; 8023/8024 are disney_hotel_service)
• its own env file, salts, NFS dataset, compose project, NPM host, and adama cron

Do not point beta WORDPRESS_DB_NAME at the prod database.

---

Next site

MariaDB is already the shared server. For the next site, start at Per site and do not redo Already done.

1. Pick a short name, a prod domain, a beta domain, and the next two free host ports. Add a row to the Ports table in this file.
2. CREATE DATABASE / GRANT to 'westfarn'@'%' for <site> and <site>_beta.
3. New env files and a new salt set per env.
4. New NFS datasets, then two entries in inventory/wordpress_sites.yml (name, port, url, export, mount). Run ./scripts/provision.sh so every host mounts them and writes compose. Do not reuse another site's wp-content.
5. Start adama first, browser install once, then the other four.
6. New upstream block in http.conf and a new NPM proxy host.
7. New cron file on adama only.

---

Checks

On each app host:

docker compose -p <site>_prod ps
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8021/
mount | grep wordpress

From the control node, each backend should answer:

for ip in 10.0.0.77 10.0.0.176 10.0.0.44 10.0.0.7 10.0.0.128; do
  curl -fsS -o /dev/null -w "$ip %{http_code}\n" "http://$ip:8021/"
done

Then load https://<prod-domain> several times. A media file uploaded while you are on the site should still load after a refresh (that request may have been served by a different host).

When something fails

Symptom	Cause
MariaDB Errcode: 13 on /var/lib/mysql	Dataset not writable by uid 999. See Already done.
ERROR 1133 on GRANT	User host does not match. Use the host from SELECT user, host FROM mysql.user.
Can't create database; database exists	Database was created by the TrueNAS wizard or a previous run. Continue at GRANT.
Container exits, cannot connect to MySQL	WORDPRESS_DB_* host, user, password, or database name. Port is 10.0.0.230:3306.
Installer or login redirects to http://10.0.0.x:8021	WP_HOME / WP_SITEURL are not the public URL, or NPM is not sending X-Forwarded-Proto.
Logged out on refresh	Salts differ between hosts. Recreate with one shared env file.
Upload succeeds, image 404s on refresh	That host does not have the NFS mount, or the mount is not the one in the compose file.
413 Request Entity Too Large	client_max_body_size in the NPM location ^~ / block.
Scheduled posts never publish	Cron is missing on adama, or wp-cron.php is not reachable at the public URL.
Plugin shows on one host only	It was installed onto a local disk instead of the NFS wp-content.