server-infra/IMPLEMENTATION.md

Server Infrastructure — Implementation Guide

Ansible-based provisioning and deployment for homelab web servers.

Architecture

flowchart TB
    subgraph provision ["Provisioning (manual / rare)"]
        Control1["ai-server-4080\n(control node)"]
        Control1 -->|ansible-playbook site.yml| Adama
        Control1 -->|ansible-playbook site.yml| Roslin
        Control1 -->|ansible-playbook site.yml| Starbuck
        Control1 -->|ansible-playbook site.yml| Apollo
    end

    subgraph cicd ["CI/CD (every merge to master)"]
        Gitea["Gitea push"]
        Gitea --> Tests["Act: unit tests"]
        Tests --> Deploy["Act: deploy job"]
        Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
        AnsibleDeploy --> Adama2["adama"]
        AnsibleDeploy --> Roslin2["roslin"]
        AnsibleDeploy --> Starbuck2["starbuck"]
        AnsibleDeploy --> Apollo2["apollo"]
    end

Pipeline	When	Playbook	Where it runs
Provision	New VM, OS change, firewall, Docker install	site.yml	ai-server-4080 — run manually
Deploy	Green unit tests on master	deploy-apps.yml	Gitea Act runner on ai-server-4080
Stop	Take a compose app down	stop-apps.yml (scripts/stop.sh)	ai-server-4080 — run manually

Both pipelines share the same inventory (inventory/hosts.yml).

Servers

Name	IP	Role
adama	10.0.0.77	Ubuntu Server VM (Proxmox) — app host
roslin	10.0.0.176	Ubuntu Server VM (Proxmox) — app host
starbuck	10.0.0.44	Ubuntu Server VM (Proxmox) — app host
apollo	10.0.0.7	Ubuntu Server VM (Proxmox) — app host
ai-server-4080	10.0.0.128	Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas

Hostname on this machine: ryan-development-1

Repo Layout

server-infra/
├── IMPLEMENTATION.md       # This file
├── README.md               # Quick start
├── ansible.cfg
├── requirements.yml        # Ansible Galaxy collections
├── inventory/
│   ├── hosts.yml
│   ├── group_vars/
│   │   └── all.yml         # vars + app_catalog
│   └── host_vars/
│       ├── adama.yml       # host_apps (django + static; monica worker)
│       ├── roslin.yml      # host_apps (mirrors adama, no worker)
│       ├── starbuck.yml    # host_apps (mirrors roslin)
│       ├── apollo.yml      # host_apps (mirrors roslin)
│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
├── playbooks/
│   ├── site.yml            # Phase 1: provision
│   ├── deploy-apps.yml     # Phase 2: CI deploy
│   └── stop-apps.yml       # compose down one app+env
├── roles/
│   ├── common/             # Base packages
│   ├── ufw/                # Firewall
│   ├── docker/             # Docker CE + compose plugin
│   ├── nodejs/             # Node.js + npm + npx (NodeSource)
│   ├── gitea-key/          # per-server SSH key + Gitea access probe
│   ├── tianji/             # Monitoring reporter
│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
│   ├── alloy/              # log/metrics shipper
│   ├── app-deploy/         # django (docker) + node-static deploy
│   └── web-static/         # nginx container serving /var/www builds
└── scripts/
    ├── provision.sh        # Wrapper with --limit support
    ├── deploy.sh           # Wrapper for deploy playbook
    └── stop.sh             # Wrapper for stop playbook (compose down)

Prerequisites (One-Time Bootstrap)

Ansible needs SSH + sudo on each target before playbooks work.

1. Create westfarn on each VM with sudo membership.
2. Copy your SSH public key from the control node (ai-server-4080):

   ssh-copy-id westfarn@10.0.0.77
   ssh-copy-id westfarn@10.0.0.176
   ssh-copy-id westfarn@10.0.0.44
   ssh-copy-id westfarn@10.0.0.7
   

3. Confirm passwordless SSH:

   ssh westfarn@10.0.0.77
   ssh westfarn@10.0.0.176
   ssh westfarn@10.0.0.44
   ssh westfarn@10.0.0.7
   

4. First-time only — grant passwordless sudo on each new host before the first provision.sh run. Ubuntu 26.04 ships sudo-rs by default; Ansible's --ask-become-pass does not recognize its password prompt, so bootstrap sudo manually over SSH instead:

   ssh -t westfarn@10.0.0.176   # repeat for each host IP
   

   On the host:

   echo 'westfarn ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/westfarn
   sudo chmod 440 /etc/sudoers.d/westfarn
   exit
   

   The common role writes the same file on later runs; this one-time step is only needed before Ansible can escalate privileges the first time.
5. On ai-server-4080 (control node), install Ansible:

   sudo apt update && sudo apt install -y ansible
   # or: pip install ansible
   

6. Install Galaxy collections:

   cd ~/Documents/repos/server-infra
   ansible-galaxy collection install -r requirements.yml
   

7. Update inventory/host_vars/ai-server-4080.yml with this machine's LAN IP (ansible_host).

Testing on a Single Server

Use --limit to target one host without touching the others. Helper scripts wrap this.

Ping one host

./scripts/provision.sh adama --check   # dry run
ansible adama -m ping

Provision one host

New hosts need the one-time passwordless sudo bootstrap in Prerequisites before the first run.

# Dry run (no changes)
./scripts/provision.sh adama --check

# Apply for real
./scripts/provision.sh adama

# Same for other hosts
./scripts/provision.sh roslin
./scripts/provision.sh starbuck
./scripts/provision.sh apollo
./scripts/provision.sh ai-server-4080

Provision all hosts

./scripts/provision.sh

Deploy to one host (Phase 2)

./scripts/deploy.sh adama
./scripts/deploy.sh --check roslin
./scripts/deploy.sh starbuck
./scripts/deploy.sh apollo

Stop a compose app

./scripts/stop.sh --app print_forge --env prod
./scripts/stop.sh --app print_forge --env beta
./scripts/stop.sh adama --app chat_backend --env beta --check

stop.sh is docker compose down for one --app + --env (no -v). Checkouts and secrets stay. Next deploy.sh of that app+env starts it again unless host_apps sets enabled: false. Node-static apps have no compose project; stop.sh skips them.

Under the hood, scripts pass --limit <hostname> to ansible-playbook.

Phase 1: Provision (site.yml)

Applies roles in order to the webservers group:

Role	Purpose
common	apt update, git, python3, pip, curl, ca-certificates
ufw	Firewall: SSH from LAN only, HTTP/HTTPS public
docker	Docker CE, compose plugin, add westfarn to docker group
nodejs	Node.js + npm + npx (NodeSource) for dta_webapp builds
gitea-key	Per-server SSH key + Gitea access probe
tianji	Monitoring reporter

UFW rules

Port	Source	Purpose
22	10.0.0.0/24	SSH (LAN only)
80	anywhere	HTTP
443	anywhere	HTTPS
default	deny incoming	Block everything else

Warning: Test UFW on one host first (./scripts/provision.sh adama). Keep a Proxmox console session open in case SSH rules lock you out.

After Docker install, re-SSH so the docker group membership takes effect.

Phase 2: CI Deploy (deploy-apps.yml)

Apps

App	Type	Hosts	Envs	Notes
company_site	django (docker)	all webservers	prod	active/active behind NPM; beta port reserved
dta_service	django (docker)	all webservers	beta + prod	active/active behind NPM
dta_webapp	node/vite static	all webservers	beta + prod	active/active; built to /var/www/<env>.realpath.app/html, served by web-static nginx
dta_blog	node-static (Python SSG)	all webservers	beta + prod	active/active; built to /var/www/<env>.blog.realpath.app/html; own hosts (blog.realpath.app / beta.blog.realpath.app), not realpath.app/blog; Tianji ids baked at build
scha	django (docker)	all webservers	prod	active/active behind NPM; beta port reserved
chat_web_app	node-static (CRA)	all webservers	beta + prod	active/active; built to /var/www/<env>.chat.aimloperations/html, served by web-static nginx
chat_backend	django (docker)	all webservers	beta + prod	active/active behind NPM; Ollama http://10.0.0.128:11434; SearxNG http://10.0.0.128:8088 (SEARXNG_BASE_URL)
monica_site	django (docker)	all webservers	beta + prod	active/active behind NPM; no bundled Postgres (like scha); dj-queue worker singleton on adama only (compose --profile worker); Ollama social drafting via 10.0.0.128:11434
url_shortening_service	django (docker)	all webservers	beta + prod	active/active behind NPM; no bundled Postgres; no worker. Two public hosts, same container: short domain (GET /, GET /<code> 302) and API host (/api/links/, Bearer required).
college_craft	django (docker)	all webservers	beta + prod	active/active behind NPM; no bundled Postgres (like scha / monica_site); dj-queue worker singleton on adama only (compose --profile worker); Ollama social drafting via 10.0.0.128:11434; Nominatim http://10.0.0.128:8089; prod SITE_UNDER_CONSTRUCTION=true until launch
print_forge	django (docker)	all webservers	beta + prod	stopped (enabled: false, #33). Ports reserved. Re-enable + deploy.sh to bring back. Worker was adama-only.
abc_be	django (docker)	all webservers	beta only	AI Benefits Coach API; no bundled Postgres; Ollama http://10.0.0.128:11434; shares DB abc_be_beta with abc_worker
abc_worker	django-type compose (FastAPI)	all webservers	beta only	same Postgres as abc_be; migrate_cmd: true; WS on host port 8018
abc_fe	node-static (Vite)	all webservers	beta only	built to /var/www/<env>.abc.aimloperations/html; no prod host_apps row
livekit	django-type compose (LiveKit SFU)	all webservers	beta only	Docker like abc_worker. Host network :7880. Do not NPM-balance until Redis — pick one upstream (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that host. Secrets ~/Documents/secrets/livekit/livekit_beta.env

Django apps use a shared external Postgres (via DATABASE_URL in each host's env file) so active/active replicas share one database. Beta and prod never share a DB.

Data model

• app_catalog (group_vars/all.yml) — how each app is built (repo, type, compose file, migrate cmd).
• host_apps (host_vars/<host>.yml) — which app+env+port runs on that host. Optional compose_profiles: [worker] activates docker compose profiles on that host only (used for monica_site / college_craft / print_forge dj-queue singleton on adama). Optional enabled: false skips deploy (CI included) but keeps the row for ports and stop.sh. Omit the key (or true) to deploy as before. Do not use Jinja rejectattr('enabled') — missing key is an error, not "enabled".
• Django app = one compose project per env: project name <app>_<env>, host port from host_apps. Ports match across app hosts so NPM can balance adama:PORT + roslin:PORT + starbuck:PORT + apollo:PORT.

Ports

Reserved host ports for NPM upstreams. Ports must match across every host that serves the same app+env. Rows marked not deployed keep the port free for a future beta replica.

App	beta	prod	Deployed on
company_site	8010 (not deployed)	8000	all webservers
dta_service	8011	8001	all webservers
scha	8012 (not deployed)	8002	all webservers
chat_backend	8013	8003	all webservers
monica_site	8014	8004	all webservers
url_shortening_service	8015	8005	all webservers
college_craft	8016	8006	all webservers
print_forge	8019 (stopped)	8007 (stopped)	all webservers (enabled: false)
abc_be	8017	8009 (not deployed)	all webservers
abc_worker	8018	8008 (not deployed)	all webservers
dta_webapp (nginx)	8081	8080	all webservers
chat_web_app (nginx)	8083	8082	all webservers
abc_fe (nginx)	8085	8084 (not deployed)	all webservers
dta_blog (nginx)	8087	8086	all webservers
livekit	7880	—	all webservers (signaling). UDP 3478 + 50000–60000 + TCP 7881 via router DNAT to the NPM upstream host
SearxNG (LAN only)	—	8088	ai-server-4080 only (searxng_stack); not an NPM upstream

Host-local services on ai-server-4080 (not balanced by NPM):

Service	Port	Notes
Ollama	11434	Not Ansible-managed today; GPU host (monica_site / college_craft social drafting)
Nominatim	8089	Not Ansible-managed today; LAN address autocomplete for college_craft / print_forge
SearxNG	8088	roles/searxng (#10); JSON API for chat_backend grounded search
Loki	3100	roles/observability
Prometheus	9090	roles/observability
Grafana	3000	roles/observability

Port clash warning: do not bind SearxNG to 8080 — that is dta_webapp prod. chat_backend secrets must use SEARXNG_BASE_URL=http://10.0.0.128:8088.

Flow

1. Gitea push to master → repo's .gitea/workflows runs tests.
2. On green, deploy job on the self-hosted runner calls:

   ~/Documents/repos/server-infra/scripts/deploy.sh \
     --app company_site --env prod --ref "${{ gitea.sha }}"
   

3. deploy-apps.yml runs against webservers; each host deploys only the matching enabled app+env from its host_apps (enabled: false is skipped).

To stop a compose app without deleting inventory:

./scripts/stop.sh --app print_forge --env prod

Set enabled: false on that host_apps row so the next CI deploy cannot start it again. stop.sh still matches disabled rows.

app-deploy role behavior

• django: push per-app secret from control node {{ secrets_dir }}/<app>/<app>_<env>.env to host {{ apps_env_dir }} → git checkout at ref → copy .env into checkout → docker compose build → up -d (with COMPOSE_PROFILES from optional host_apps.compose_profiles) → migrate (run once, shared DB). monica_site / college_craft / print_forge worker: adama host_apps sets compose_profiles: [worker] so deploy starts dj-queue with web. Other hosts omit profiles (web only).
• node-static: git checkout at ref → npm ci → npm run build:<env> (writes to the app's webroot_pattern, e.g. /var/www/{env}.realpath.app/html, /var/www/{env}.blog.realpath.app/html, or /var/www/{env}.chat.aimloperations/html). Optional catalog error_page_404 (used by dta_blog) serves that file instead of the SPA /index.html fallback.
• web-static role: one nginx container per app host serving the static roots on their ports (from host_apps); NPM balances across hosts. Before compose up, removes any container currently publishing those host ports (docker ps --filter publish=<port>) so leftovers cannot block the bind, then recreates web-static.

Reverse proxy / load balancing (NPM at 10.0.0.230)

Ansible does not manage NPM. It only guarantees stable host ports. In NPM you point each domain at the backend(s):

• Single host: standard Proxy Host → adama:PORT.
• Active/active: jc21 NPM's UI Proxy Host is single-target. To balance app hosts you need the Advanced tab with a custom upstream {} block (or a real LB). Confirm this before relying on active/active.

App	Domains	Backends
company_site	aimloperations.com (+ www)	adama:8000 + roslin:8000 + starbuck:8000 + apollo:8000
dta_service	(see DTA NPM hosts)	adama:8001 / 8011 + same on roslin / starbuck / apollo / ai-server-4080
dta_webapp	(see DTA NPM hosts)	adama:8080 / 8081 + same on roslin / starbuck / apollo
dta_blog	blog.realpath.app (prod); beta.blog.realpath.app (beta). Do not reverse-proxy onto realpath.app/blog	adama:8086 / 8087 + same on roslin / starbuck / apollo / ai-server-4080
scha	schawheaton.aimloperations.com, schawheaton.com (+ www)	adama:8002 + roslin:8002 + starbuck:8002 + apollo:8002 (+ ai-server-4080:8002)
chat_web_app	chat.aimloperations.com (+ www); beta.chat.aimloperations.com	adama:8082 / 8083 + same on roslin / starbuck / apollo / ai-server-4080
chat_backend	chatbackend.aimloperations.com; beta.chatbackend.aimloperations.com	adama:8003 / 8013 + same on roslin / starbuck / apollo / ai-server-4080
monica_site	mkdrealtor.com (+ www); monica-preview.aimloperations.com (beta)	adama:8004 / 8014 + same on roslin / starbuck / apollo / ai-server-4080
url_shortening_service (short)	aiml.pw and/or cidinn.li (pick when DNS is ready); short-beta.aimloperations.com (beta). Proxy / + /[a-z0-9]{4,8} only — 404 /api/, /admin/, /debug/	adama:8005 / 8015 + same on roslin / starbuck / apollo / ai-server-4080
url_shortening_service (API)	shortener.aimloperations.com; shortener-beta.aimloperations.com (beta). Proxy /api/ only — 404 /admin/	same ports as short host (one container)
college_craft	collegecraft.com (+ www); college-craft-preview.aimloperations.com (beta)	adama:8006 / 8016 + same on roslin / starbuck / apollo / ai-server-4080
print_forge	printforgeprints.com (+ www); print-forge-preview.aimloperations.com (beta) — app stopped (enabled: false). Disable NPM hosts or expect 502.	:8007 / :8019 reserved
abc_fe	beta.abc.aimloperations.com (beta only)	adama:8085 + same on roslin / starbuck / apollo / ai-server-4080
abc_be	beta.abc.be.aimloperations.com (beta only)	adama:8017 + same on roslin / starbuck / apollo / ai-server-4080
abc_worker	beta.abc.worker.aimloperations.com (beta only; HTTP + WebSocket upgrade)	adama:8018 + same on roslin / starbuck / apollo / ai-server-4080
livekit	LIVEKIT_DOMAIN (e.g. livekit.aimloperations.com) — NPM Proxy Host, WebSocket, single upstream	adama:7880 (do not balance until Redis)

Required changes IN each app repo (owned separately)

• docker-compose.prod.yml: drop the bundled db service; web reads DATABASE_URL / DB_HOST pointing at the shared external Postgres.
• Each app has its own database + user on the shared Postgres.
• .gitea/workflows/deploy.yml: replace the local scripts/deploy.sh step with a call to server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha> (keep the test/docker jobs).
• dta_webapp: npm run build:beta / build:prod output to /var/www/beta.realpath.app/html / /var/www/prod.realpath.app/html.
• dta_blog: npm run build:beta / build:prod output to /var/www/beta.blog.realpath.app/html / /var/www/prod.blog.realpath.app/html (python3 build.py --env <env> copies dist/). Default branch is main.
• chat_web_app: npm run build:beta / build:prod output to /var/www/beta.chat.aimloperations/html / /var/www/prod.chat.aimloperations/html.

Companion chat_web_app frontend is already registered in this infrastructure repo.

Shared Postgres (10.0.0.230, same box as NPM)

One shared instance; each app+env gets its own database (beta and prod MUST NOT share a DB — active/active replicas of the same env share one DB, different envs do not).

app	env	database	DATABASE_URL
company_site	prod	company_site	postgres://westfarn:<pw>@10.0.0.230:5432/company_site
company_site	beta	company_site_beta	postgres://westfarn:<pw>@10.0.0.230:5432/company_site_beta
dta_service	prod	dta_service	postgres://westfarn:<pw>@10.0.0.230:5432/dta_service
dta_service	beta	dta_service_beta	postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta
scha	prod	scha	postgres://westfarn:<pw>@10.0.0.230:5432/scha
scha	beta	scha_beta	postgres://westfarn:<pw>@10.0.0.230:5432/scha_beta
chat_backend	prod	chat_backend	postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend
chat_backend	beta	chat_backend_beta	postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend_beta
monica_site	prod	monica_site	postgres://westfarn:<pw>@10.0.0.230:5432/monica_site
monica_site	beta	monica_site_beta	postgres://westfarn:<pw>@10.0.0.230:5432/monica_site_beta
url_shortening_service	prod	url_shortener	postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener
url_shortening_service	beta	url_shortener_beta	postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener_beta
college_craft	prod	college_craft	postgres://westfarn:<pw>@10.0.0.230:5432/college_craft
college_craft	beta	college_craft_beta	postgres://westfarn:<pw>@10.0.0.230:5432/college_craft_beta
print_forge	prod	print_forge	postgres://westfarn:<pw>@10.0.0.230:5432/print_forge
print_forge	beta	print_forge_beta	postgres://westfarn:<pw>@10.0.0.230:5432/print_forge_beta
abc_be	beta	abc_be_beta	postgres://westfarn:<pw>@10.0.0.230:5432/abc_be_beta
abc_worker	beta	(same abc_be_beta)	worker uses DB_* pointing at abc_be_beta — do not create a second DB

Server prereqs on 10.0.0.230: create each DB + grant westfarn; listen_addresses covers LAN; pg_hba.conf allows 10.0.0.0/24; firewall opens 5432 to 10.0.0.0/24 only.

url_shortening_service extra env (control-node secrets, not in git): SHORT_PUBLIC_HOSTS / SHORT_API_HOSTS / SHORT_ADMIN_HOSTS (admin = localhost,127.0.0.1 only), SHORTENER_API_TOKENS (monica:<token>), SHORT_ALLOWED_HOSTS, CLICK_IP_PEPPER (distinct from DJANGO_SECRET_KEY). DJANGO_DEBUG=false for prod and beta. Caller monica_site uses SHORTENER_BASE_URL=https://<api-host> and Bearer mint; do not mint via the short hostname.

college_craft extra env (control-node secrets, not in git): ~/Documents/secrets/college_craft/college_craft_prod.env and college_craft_beta.env. Template: app repo .env.prod.example. DJANGO_ENV / DJANGO_ALLOWED_HOSTS / DATABASE_URL / WEB_PORT / PUBLIC_SITE_URL differ per env (prod collegecraft.com / :8006; beta college-craft-preview.aimloperations.com / :8016). SITE_UNDER_CONSTRUCTION=true on prod (holding page until launch), false on beta. FEATURE_BLOG=true; other FEATURE_* stay false until purchased. Prod also needs TIANJI_WEBSITE_ID + reCAPTCHA keys. If email/SMS or direct mail is turned on later, add a college_craft:<token> entry to SHORTENER_API_TOKENS on url_shortening_service.

App-repo companion (college_craft, not this repo): deploy workflows still call --app client_site (template leftover). They must become --app college_craft before CI deploy will hit this catalog entry. Default branch is master (not main).

print_forge extra env (control-node secrets, not in git) — currently stopped (#33): host_apps rows stay with enabled: false. Bring back with enabled removed/true then deploy.sh. Secrets kept: ~/Documents/secrets/print_forge/print_forge_prod.env and print_forge_beta.env. Template: app repo .env.prod.example. DJANGO_ENV / DJANGO_ALLOWED_HOSTS / DATABASE_URL / WEB_PORT / PUBLIC_SITE_URL differ per env (prod printforgeprints.com / :8007; beta print-forge-preview.aimloperations.com / :8019). SITE_UNDER_CONSTRUCTION=true on prod (holding page until launch), false on beta. Purchased flags: FEATURE_EMAIL_SMS, FEATURE_PAYMENTS, FEATURE_SHOP, FEATURE_SHIPPING. Other FEATURE_* stay false. Add a print_forge:<token> entry to SHORTENER_API_TOKENS on url_shortening_service (beta token first). SHORTENER_BASE_URL is prod shortener on prod, https://shortener-beta.aimloperations.com on beta. Default branch is master.

Companion app ticket: print_forge#1 (#27). App Gitea workflow still calls --app print_forge on master; deploy no-ops while enabled: false.

abc_be / abc_worker / abc_fe (GIS org, not this repo) — beta only (#26):

Control-node secrets (never git, mode 600):

~/Documents/secrets/abc_be/abc_be_beta.env
~/Documents/secrets/abc_worker/abc_worker_beta.env

Templates: GIS/abc_be .env.beta.example, GIS/abc_worker .env.beta.example. abc_fe has no secret file — public VITE_* URLs live in committed .env.beta. Create Postgres DB abc_be_beta and grant westfarn before first deploy. NPM: beta.abc.aimloperations.com → :8085, beta.abc.be.aimloperations.com → :8017, beta.abc.worker.aimloperations.com → :8018 (enable WebSocket). No prod ABC host_apps rows; ports 8009 / 8008 / 8084 reserved (print_forge took prod 8007).

Companion workflows: abc_be#22, abc_worker#27, abc_fe#27.

livekit (GIS org, SFU on webservers — abc_worker#14–#17):

Control-node secret (never git, mode 600):

~/Documents/secrets/livekit/livekit_beta.env

Template: GIS/livekit .env.beta.example. Same LIVEKIT_API_KEY / LIVEKIT_API_SECRET pair as abc_be / abc_worker; worker LIVEKIT_URL=wss://<LIVEKIT_DOMAIN>.

Deploy: ./scripts/deploy.sh --app livekit --env beta (all webservers, like abc_worker). NPM: LIVEKIT_DOMAIN → one host :7880 with WebSocket (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that same host. Do not active/active-balance LiveKit until Redis is in the compose.

dta_blog (#29) — Python SSG, no secrets file. npm run build:<env> writes /var/www/<env>.blog.realpath.app/html. Prod (:8086) omits demo: true seed posts; beta (:8087) includes them. Header/footer links are baked at build (https://realpath.app vs https://beta.realpath.app); Tianji website ids are likewise baked (cmtvvmf562afjzqumwt1yh2y8 / cmtvvn6z62agdzqumtk8xijpy). Default branch is main.

NPM / DNS / Cloudflare (Ansible does not manage these): blog.realpath.app → :8086, beta.blog.realpath.app → :8087, TLS like the other RealPath names. Do not mount this on realpath.app/blog. Static ports stay LAN / NPM-only (UFW does not world-open 8086/8087).

Companion app ticket: dta_blog#1. Gitea deploy can later call ./scripts/deploy.sh --app dta_blog --env beta|prod.

One-time host bootstrap (per target)

• Gitea SSH key: the gitea-key role (in site.yml) generates a key per server, configures SSH for port 30009, probes access, and — if the server can't reach Gitea yet — prints the public key to add and stops. Add the key (Gitea user SSH keys, or repo Deploy Keys) and re-run provisioning.
• Create control-node secrets {{ secrets_dir }}/<app>/<app>_<env>.env (default ~/Documents/secrets/<app>/<app>_<env>.env) with DATABASE_URL (see table), DJANGO_ENV, DJANGO_SECRET_KEY, WEB_PORT (matching the port table). Deploy pushes these to /opt/apps/env/<app>_<env>.env (mode 600) on adama + roslin + starbuck + apollo. Never committed to git.
• Node.js/npm/npx for the dta_webapp build — installed by the nodejs role in site.yml (NodeSource, node_major default 20).

Gitea Act Runner

Recommended: Single self-hosted runner on ai-server-4080.

• One orchestration point.
• App hosts (adama/roslin/starbuck/apollo) run the workloads; no runner needed on them for deploy fan-out.
• Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).

Runner requirements on ai-server-4080

Requirement	Why
Ansible	Run deploy-apps.yml
server-infra checkout	Playbooks + inventory
SSH key to app hosts	Deploy fan-out

On every push or merged PR to master, .gitea/workflows/sync-checkout.yml fast-forward pulls this repo at ~/Documents/repos/server-infra on the Act runner so playbooks and inventory stay current without a manual git pull.

SSH Keys for CI Deploy

Key	Used by	Purpose
Personal key	You	Manual provisioning
Deploy key (runner)	Act → Ansible → hosts	Automated deploy

Consider a dedicated deploy user with limited sudo (docker only) — future hardening step.

Secrets (Phase 2)

Use Ansible Vault for production secrets. Do not commit plaintext.

ansible-vault create inventory/group_vars/webservers/vault.yml
ansible-playbook playbooks/site.yml --ask-vault-pass

Store vault password for CI in a file readable only by the Act runner (e.g. ~/.ansible-vault-pass, mode 600).

Implementation Order

#	Task	Status
1	Create server-infra repo	Done
2	Inventory with all 5 hosts	Done (#20)
3	Bootstrap SSH to app hosts	Manual
4	site.yml → common, ufw, docker	Done
5	Verify ansible webservers -m ping	Manual
6	Test on single server: ./scripts/provision.sh adama	Manual
7	Provision all: ./scripts/provision.sh	Manual
8	Deploy SSH key for Act runner	Future
8a	Auto-sync runner checkout on master (.gitea/workflows/sync-checkout.yml)	Done
9	Stub deploy-apps.yml + update company_site workflow	Future
10	Dockerize company_site	Future (separate ticket)
10a	Register + deploy scha (all webservers, port 8002)	In progress (scha#19)
10b	Register + deploy chat_web_app (node-static, ports 8082/8083)	Done (prod); beta (#7, chat_web_app#35)
10c	Register + deploy chat_backend (django, ports 8003/8013)	Done (prod); beta (#7, chat_backend#26)
10d	Register + deploy monica_site (django, ports 8004/8014)	Done (#14)
10e	Auto-start monica_site dj-queue worker on adama (compose_profiles)	Done (#17)
10f	Add starbuck + apollo as app hosts (same workloads as roslin)	Done (#20)
10g	Register + deploy url_shortening_service (django, ports 8005/8015)	Done (#22)
10h	Register + deploy college_craft (django, ports 8006/8016)	Done (#24)
10i	Register + deploy ABC beta (abc_be 8017, abc_worker 8018, abc_fe 8085)	Done (#26)
10j	Register + deploy print_forge (django, ports 8007/8019)	Done (#27)
10k	Register + deploy dta_blog (node-static, ports 8086/8087)	Done (#29)
10l	Register GIS LiveKit SFU (abc_worker-style docker on webservers, port 7880)	This PR
11	Gitea container registry (optional)	Future

Open Decisions

1. Deploy user — westfarn vs dedicated deploy for CI.
2. NPM load balancing — confirm jc21 NPM can express all app-host upstreams (Advanced tab), else active/active is just independent instances.
3. Secrets — Ansible Vault vs per-host env files (currently per-host /opt/apps/env/*.env).

Adding a New VM

1. Add host to inventory/hosts.yml under webservers.
2. Bootstrap SSH: ssh-copy-id westfarn@<new-ip>.
3. Test: ./scripts/provision.sh <hostname> --check.
4. Provision: ./scripts/provision.sh <hostname>.
5. Deploys automatically include new host once in webservers group.