server-infra/IMPLEMENTATION.md

# Server Infrastructure — Implementation Guide

Ansible-based provisioning and deployment for homelab web servers.

## Architecture

```mermaid
flowchart TB
    subgraph provision ["Provisioning (manual / rare)"]
        Control1["ai-server-4080\n(control node)"]
        Control1 -->|ansible-playbook site.yml| Adama
        Control1 -->|ansible-playbook site.yml| Roslin
        Control1 -->|ansible-playbook site.yml| Starbuck
        Control1 -->|ansible-playbook site.yml| Apollo
    end

    subgraph cicd ["CI/CD (every merge to master)"]
        Gitea["Gitea push"]
        Gitea --> Tests["Act: unit tests"]
        Tests --> Deploy["Act: deploy job"]
        Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
        AnsibleDeploy --> Adama2["adama"]
        AnsibleDeploy --> Roslin2["roslin"]
        AnsibleDeploy --> Starbuck2["starbuck"]
        AnsibleDeploy --> Apollo2["apollo"]
    end
```

| Pipeline | When | Playbook | Where it runs |
|----------|------|----------|---------------|
| **Provision** | New VM, OS change, firewall, Docker install | `site.yml` | ai-server-4080 — run manually |
| **Deploy** | Green unit tests on `master` | `deploy-apps.yml` | Gitea Act runner on ai-server-4080 |
| **Stop** | Take a compose app down | `stop-apps.yml` (`scripts/stop.sh`) | ai-server-4080 — run manually |

Both pipelines share the same inventory (`inventory/hosts.yml`).

## Servers

| Name | IP | Role |
|------|-----|------|
| adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
| roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
| starbuck | 10.0.0.44 | Ubuntu Server VM (Proxmox) — app host |
| apollo | 10.0.0.7 | Ubuntu Server VM (Proxmox) — app host |
| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |

Hostname on this machine: `ryan-development-1`

## Repo Layout

```
server-infra/
├── IMPLEMENTATION.md       # This file
├── README.md               # Quick start
├── ansible.cfg
├── requirements.yml        # Ansible Galaxy collections
├── inventory/
│   ├── hosts.yml
│   ├── group_vars/
│   │   └── all.yml         # vars + app_catalog
│   └── host_vars/
│       ├── adama.yml       # host_apps (django + static; monica worker)
│       ├── roslin.yml      # host_apps (mirrors adama, no worker)
│       ├── starbuck.yml    # host_apps (mirrors roslin)
│       ├── apollo.yml      # host_apps (mirrors roslin)
│       └── ai-server-4080.yml  # control node / act runner / SearxNG / observability
├── playbooks/
│   ├── site.yml            # Phase 1: provision
│   ├── deploy-apps.yml     # Phase 2: CI deploy
│   └── stop-apps.yml       # compose down one app+env
├── roles/
│   ├── common/             # Base packages
│   ├── ufw/                # Firewall
│   ├── docker/             # Docker CE + compose plugin
│   ├── nodejs/             # Node.js + npm + npx (NodeSource)
│   ├── gitea-key/          # per-server SSH key + Gitea access probe
│   ├── tianji/             # Monitoring reporter
│   ├── observability/      # Loki + Prometheus + Grafana (ai-server-4080)
│   ├── searxng/            # SearxNG JSON API for chat_backend (#10)
│   ├── alloy/              # log/metrics shipper
│   ├── app-deploy/         # django (docker) + node-static deploy
│   └── web-static/         # nginx container serving /var/www builds
└── scripts/
    ├── provision.sh        # Wrapper with --limit support
    ├── deploy.sh           # Wrapper for deploy playbook
    └── stop.sh             # Wrapper for stop playbook (compose down)
```

## Prerequisites (One-Time Bootstrap)

Ansible needs SSH + sudo on each target before playbooks work.

1. Create `westfarn` on each VM with sudo membership.
2. Copy your SSH public key from the control node (ai-server-4080):
   ```bash
   ssh-copy-id westfarn@10.0.0.77
   ssh-copy-id westfarn@10.0.0.176
   ssh-copy-id westfarn@10.0.0.44
   ssh-copy-id westfarn@10.0.0.7
   ```
3. Confirm passwordless SSH:
   ```bash
   ssh westfarn@10.0.0.77
   ssh westfarn@10.0.0.176
   ssh westfarn@10.0.0.44
   ssh westfarn@10.0.0.7
   ```
4. **First-time only** — grant passwordless sudo on each new host before the first
   `provision.sh` run. Ubuntu 26.04 ships `sudo-rs` by default; Ansible's
   `--ask-become-pass` does not recognize its password prompt, so bootstrap sudo
   manually over SSH instead:
   ```bash
   ssh -t westfarn@10.0.0.176   # repeat for each host IP
   ```
   On the host:
   ```bash
   echo 'westfarn ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/westfarn
   sudo chmod 440 /etc/sudoers.d/westfarn
   exit
   ```
   The `common` role writes the same file on later runs; this one-time step is only
   needed before Ansible can escalate privileges the first time.
5. On ai-server-4080 (control node), install Ansible:
   ```bash
   sudo apt update && sudo apt install -y ansible
   # or: pip install ansible
   ```
6. Install Galaxy collections:
   ```bash
   cd ~/Documents/repos/server-infra
   ansible-galaxy collection install -r requirements.yml
   ```
7. Update `inventory/host_vars/ai-server-4080.yml` with this machine's LAN IP (`ansible_host`).

## Testing on a Single Server

Use `--limit` to target one host without touching the others. Helper scripts wrap this.

### Ping one host

```bash
./scripts/provision.sh adama --check   # dry run
ansible adama -m ping
```

### Provision one host

New hosts need the one-time passwordless sudo bootstrap in
[Prerequisites](#prerequisites-one-time-bootstrap) before the first run.

```bash
# Dry run (no changes)
./scripts/provision.sh adama --check

# Apply for real
./scripts/provision.sh adama

# Same for other hosts
./scripts/provision.sh roslin
./scripts/provision.sh starbuck
./scripts/provision.sh apollo
./scripts/provision.sh ai-server-4080
```

### Provision all hosts

```bash
./scripts/provision.sh
```

### Deploy to one host (Phase 2)

```bash
./scripts/deploy.sh adama
./scripts/deploy.sh --check roslin
./scripts/deploy.sh starbuck
./scripts/deploy.sh apollo
```

### Stop a compose app

```bash
./scripts/stop.sh --app print_forge --env prod
./scripts/stop.sh --app print_forge --env beta
./scripts/stop.sh adama --app chat_backend --env beta --check
```

`stop.sh` is `docker compose down` for one `--app` + `--env` (no `-v`). Checkouts and
secrets stay. Next `deploy.sh` of that app+env starts it again unless `host_apps`
sets `enabled: false`. Node-static apps have no compose project; stop.sh skips them.

Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`.

## Phase 1: Provision (`site.yml`)

Applies roles in order to the `webservers` group:

| Role | Purpose |
|------|---------|
| `common` | apt update, git, python3, pip, curl, ca-certificates |
| `ufw` | Firewall: SSH from LAN only, HTTP/HTTPS public |
| `docker` | Docker CE, compose plugin, add `westfarn` to `docker` group |
| `nodejs` | Node.js + npm + npx (NodeSource) for `dta_webapp` builds |
| `gitea-key` | Per-server SSH key + Gitea access probe |
| `tianji` | Monitoring reporter |

### UFW rules

| Port | Source | Purpose |
|------|--------|---------|
| 22 | `10.0.0.0/24` | SSH (LAN only) |
| 80 | anywhere | HTTP |
| 443 | anywhere | HTTPS |
| default | deny incoming | Block everything else |

**Warning:** Test UFW on one host first (`./scripts/provision.sh adama`). Keep a Proxmox console session open in case SSH rules lock you out.

After Docker install, re-SSH so the `docker` group membership takes effect.

## Phase 2: CI Deploy (`deploy-apps.yml`)

### Apps

| App | Type | Hosts | Envs | Notes |
|-----|------|-------|------|-------|
| `company_site` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `dta_service` | django (docker) | all webservers | beta + prod | active/active behind NPM |
| `dta_webapp` | node/vite static | all webservers | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx |
| `dta_blog` | node-static (Python SSG) | all webservers | beta + prod | active/active; built to `/var/www/<env>.blog.realpath.app/html`; **own hosts** (`blog.realpath.app` / `beta.blog.realpath.app`), not `realpath.app/blog`; Tianji ids baked at build |
| `scha` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `chat_web_app` | node-static (CRA) | all webservers | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
| `chat_backend` | django (docker) | all webservers | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
| `monica_site` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
| `url_shortening_service` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres; **no worker**. Two public hosts, same container: short domain (`GET /`, `GET /<code>` 302) and API host (`/api/links/`, Bearer required). |
| `college_craft` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha` / `monica_site`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434`; Nominatim `http://10.0.0.128:8089`; prod `SITE_UNDER_CONSTRUCTION=true` until launch |
| `print_forge` | django (docker) | all webservers | beta + prod | **stopped** (`enabled: false`, [#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)). Ports reserved. Re-enable + `deploy.sh` to bring back. Worker was adama-only. |
| `abc_be` | django (docker) | all webservers | **beta only** | AI Benefits Coach API; no bundled Postgres; Ollama `http://10.0.0.128:11434`; shares DB `abc_be_beta` with `abc_worker` |
| `abc_worker` | django-type compose (FastAPI) | all webservers | **beta only** | same Postgres as `abc_be`; `migrate_cmd: true`; WS on host port 8018 |
| `abc_fe` | node-static (Vite) | all webservers | **beta only** | built to `/var/www/<env>.abc.aimloperations/html`; no prod `host_apps` row |
| `livekit` | django-type compose (LiveKit SFU) | all webservers | **beta only** | Docker like `abc_worker`. Host network `:7880`. **Do not NPM-balance** until Redis — pick one upstream (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that host. Secrets `~/Documents/secrets/livekit/livekit_beta.env` |

Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
env file) so active/active replicas share one database. Beta and prod never share
a DB.

### Data model

- `app_catalog` (`group_vars/all.yml`) — how each app is built (repo, type, compose file, migrate cmd).
- `host_apps` (`host_vars/<host>.yml`) — which app+env+port runs on that host.
  Optional `compose_profiles: [worker]` activates docker compose profiles on that
  host only (used for `monica_site` / `college_craft` / `print_forge` dj-queue singleton on adama).
  Optional `enabled: false` skips deploy (CI included) but keeps the row for ports
  and `stop.sh`. Omit the key (or `true`) to deploy as before. Do not use Jinja
  `rejectattr('enabled')` — missing key is an error, not "enabled".
- Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
  Ports match across app hosts so NPM can balance `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT`.

### Ports

Reserved host ports for NPM upstreams. Ports must match across every host that
serves the same app+env. Rows marked *not deployed* keep the port free for a
future beta replica.

| App | beta | prod | Deployed on |
|-----|------|------|-------------|
| company_site | 8010 (*not deployed*) | 8000 | all webservers |
| dta_service | 8011 | 8001 | all webservers |
| scha | 8012 (*not deployed*) | 8002 | all webservers |
| chat_backend | 8013 | 8003 | all webservers |
| monica_site | 8014 | 8004 | all webservers |
| url_shortening_service | 8015 | 8005 | all webservers |
| college_craft | 8016 | 8006 | all webservers |
| print_forge | **8019** (*stopped*) | **8007** (*stopped*) | all webservers (`enabled: false`) |
| abc_be | **8017** | 8009 (*not deployed*) | all webservers |
| abc_worker | **8018** | 8008 (*not deployed*) | all webservers |
| dta_webapp (nginx) | 8081 | 8080 | all webservers |
| chat_web_app (nginx) | 8083 | 8082 | all webservers |
| abc_fe (nginx) | **8085** | 8084 (*not deployed*) | all webservers |
| dta_blog (nginx) | **8087** | **8086** | all webservers |
| livekit | **7880** | — | all webservers (signaling). UDP 3478 + 50000–60000 + TCP 7881 via router DNAT to the NPM upstream host |
| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |

Host-local services on ai-server-4080 (not balanced by NPM):

| Service | Port | Notes |
|---------|------|-------|
| Ollama | 11434 | Not Ansible-managed today; GPU host (`monica_site` / `college_craft` social drafting) |
| Nominatim | 8089 | Not Ansible-managed today; LAN address autocomplete for `college_craft` / `print_forge` |
| SearxNG | 8088 | `roles/searxng` (#10); JSON API for chat_backend grounded search |
| Loki | 3100 | `roles/observability` |
| Prometheus | 9090 | `roles/observability` |
| Grafana | 3000 | `roles/observability` |

**Port clash warning:** do **not** bind SearxNG to `8080` — that is `dta_webapp` prod.
chat_backend secrets must use `SEARXNG_BASE_URL=http://10.0.0.128:8088`.

### Flow

1. Gitea push to `master` → repo's `.gitea/workflows` runs tests.
2. On green, deploy job on the self-hosted runner calls:
   ```bash
   ~/Documents/repos/server-infra/scripts/deploy.sh \
     --app company_site --env prod --ref "${{ gitea.sha }}"
   ```
3. `deploy-apps.yml` runs against `webservers`; each host deploys only the
   matching **enabled** app+env from its `host_apps` (`enabled: false` is skipped).

To stop a compose app without deleting inventory:

```bash
./scripts/stop.sh --app print_forge --env prod
```

Set `enabled: false` on that `host_apps` row so the next CI deploy cannot start it
again. `stop.sh` still matches disabled rows.

### `app-deploy` role behavior

- **django**: push per-app secret from control node `{{ secrets_dir }}/<app>/<app>_<env>.env`
  to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
  `docker compose build` → `up -d` (with `COMPOSE_PROFILES` from optional
  `host_apps.compose_profiles`) → migrate (run once, shared DB).
  **`monica_site` / `college_craft` / `print_forge` worker:** adama `host_apps` sets
  `compose_profiles: [worker]` so deploy starts dj-queue with web. Other hosts
  omit profiles (web only).
- **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
  (writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.realpath.app/html`,
  `/var/www/{env}.blog.realpath.app/html`, or `/var/www/{env}.chat.aimloperations/html`).
  Optional catalog `error_page_404` (used by `dta_blog`) serves that file instead
  of the SPA `/index.html` fallback.
- **web-static** role: one nginx container per app host serving the static roots
  on their ports (from `host_apps`); NPM balances across hosts. Before `compose up`,
  removes any container currently publishing those host ports (`docker ps --filter
  publish=<port>`) so leftovers cannot block the bind, then recreates web-static.

### Reverse proxy / load balancing (NPM at 10.0.0.230)

Ansible does **not** manage NPM. It only guarantees stable host ports. In NPM you
point each domain at the backend(s):

- Single host: standard Proxy Host → `adama:PORT`.
- Active/active: jc21 NPM's UI Proxy Host is single-target. To balance
  app hosts you need the **Advanced** tab with a custom `upstream {}` block
  (or a real LB). Confirm this before relying on active/active.

| App | Domains | Backends |
|-----|---------|----------|
| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` + `starbuck:8000` + `apollo:8000` |
| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / starbuck / apollo / ai-server-4080 |
| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin / starbuck / apollo |
| dta_blog | `blog.realpath.app` (prod); `beta.blog.realpath.app` (beta). **Do not** reverse-proxy onto `realpath.app/blog` | `adama:8086` / `8087` + same on roslin / starbuck / apollo / ai-server-4080 |
| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` + `starbuck:8002` + `apollo:8002` (+ `ai-server-4080:8002`) |
| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / starbuck / apollo / ai-server-4080 |
| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / starbuck / apollo / ai-server-4080 |
| monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / starbuck / apollo / ai-server-4080 |
| url_shortening_service (short) | `aiml.pw` and/or `cidinn.li` (pick when DNS is ready); `short-beta.aimloperations.com` (beta). Proxy `/` + `/[a-z0-9]{4,8}` only — 404 `/api/`, `/admin/`, `/debug/` | `adama:8005` / `8015` + same on roslin / starbuck / apollo / ai-server-4080 |
| url_shortening_service (API) | `shortener.aimloperations.com`; `shortener-beta.aimloperations.com` (beta). Proxy `/api/` only — 404 `/admin/` | same ports as short host (one container) |
| college_craft | `collegecraft.com` (+ www); `college-craft-preview.aimloperations.com` (beta) | `adama:8006` / `8016` + same on roslin / starbuck / apollo / ai-server-4080 |
| print_forge | `printforgeprints.com` (+ www); `print-forge-preview.aimloperations.com` (beta) — **app stopped** (`enabled: false`). Disable NPM hosts or expect 502. | `:8007` / `:8019` reserved |
| abc_fe | `beta.abc.aimloperations.com` (beta only) | `adama:8085` + same on roslin / starbuck / apollo / ai-server-4080 |
| abc_be | `beta.abc.be.aimloperations.com` (beta only) | `adama:8017` + same on roslin / starbuck / apollo / ai-server-4080 |
| abc_worker | `beta.abc.worker.aimloperations.com` (beta only; HTTP + WebSocket upgrade) | `adama:8018` + same on roslin / starbuck / apollo / ai-server-4080 |
| livekit | `LIVEKIT_DOMAIN` (e.g. `livekit.aimloperations.com`) — NPM Proxy Host, WebSocket, **single upstream** | `adama:7880` (do not balance until Redis) |

### Required changes IN each app repo (owned separately)

- [ ] `docker-compose.prod.yml`: drop the bundled `db` service; `web` reads
      `DATABASE_URL` / `DB_HOST` pointing at the shared external Postgres.
- [ ] Each app has its own database + user on the shared Postgres.
- [ ] `.gitea/workflows/deploy.yml`: replace the local `scripts/deploy.sh` step
      with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
      (keep the test/docker jobs).
- [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
      `/var/www/beta.realpath.app/html` / `/var/www/prod.realpath.app/html`.
- [ ] `dta_blog`: `npm run build:beta` / `build:prod` output to
      `/var/www/beta.blog.realpath.app/html` / `/var/www/prod.blog.realpath.app/html`
      (`python3 build.py --env <env>` copies `dist/`). Default branch is `main`.
- [ ] `chat_web_app`: `npm run build:beta` / `build:prod` output to
      `/var/www/beta.chat.aimloperations/html` / `/var/www/prod.chat.aimloperations/html`.

Companion `chat_web_app` frontend is already registered in this infrastructure repo.

### Shared Postgres (10.0.0.230, same box as NPM)

One shared instance; each app+env gets its own database (beta and prod MUST NOT
share a DB — active/active replicas of the same env share one DB, different envs
do not).

| app | env | database | DATABASE_URL |
|-----|-----|----------|--------------|
| company_site | prod | `company_site` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site` |
| company_site | beta | `company_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site_beta` |
| dta_service | prod | `dta_service` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service` |
| dta_service | beta | `dta_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta` |
| scha | prod | `scha` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha` |
| scha | beta | `scha_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/scha_beta` |
| chat_backend | prod | `chat_backend` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend` |
| chat_backend | beta | `chat_backend_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/chat_backend_beta` |
| monica_site | prod | `monica_site` | `postgres://westfarn:<pw>@10.0.0.230:5432/monica_site` |
| monica_site | beta | `monica_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/monica_site_beta` |
| url_shortening_service | prod | `url_shortener` | `postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener` |
| url_shortening_service | beta | `url_shortener_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/url_shortener_beta` |
| college_craft | prod | `college_craft` | `postgres://westfarn:<pw>@10.0.0.230:5432/college_craft` |
| college_craft | beta | `college_craft_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/college_craft_beta` |
| print_forge | prod | `print_forge` | `postgres://westfarn:<pw>@10.0.0.230:5432/print_forge` |
| print_forge | beta | `print_forge_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/print_forge_beta` |
| abc_be | beta | `abc_be_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/abc_be_beta` |
| abc_worker | beta | *(same `abc_be_beta`)* | worker uses `DB_*` pointing at `abc_be_beta` — do **not** create a second DB |

Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
`listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
5432 to `10.0.0.0/24` only.

`url_shortening_service` extra env (control-node secrets, not in git):
`SHORT_PUBLIC_HOSTS` / `SHORT_API_HOSTS` / `SHORT_ADMIN_HOSTS` (admin = `localhost,127.0.0.1` only),
`SHORTENER_API_TOKENS` (`monica:<token>`), `SHORT_ALLOWED_HOSTS`, `CLICK_IP_PEPPER`
(distinct from `DJANGO_SECRET_KEY`). `DJANGO_DEBUG=false` for prod and beta.
Caller `monica_site` uses `SHORTENER_BASE_URL=https://<api-host>` and Bearer mint;
do not mint via the short hostname.

`college_craft` extra env (control-node secrets, not in git):
`~/Documents/secrets/college_craft/college_craft_prod.env` and
`college_craft_beta.env`. Template: app repo `.env.prod.example`.
`DJANGO_ENV` / `DJANGO_ALLOWED_HOSTS` / `DATABASE_URL` / `WEB_PORT` /
`PUBLIC_SITE_URL` differ per env (prod `collegecraft.com` / `:8006`;
beta `college-craft-preview.aimloperations.com` / `:8016`).
`SITE_UNDER_CONSTRUCTION=true` on prod (holding page until launch), `false` on beta.
`FEATURE_BLOG=true`; other `FEATURE_*` stay false until purchased. Prod also needs
`TIANJI_WEBSITE_ID` + reCAPTCHA keys. If email/SMS or direct mail is turned on later,
add a `college_craft:<token>` entry to `SHORTENER_API_TOKENS` on `url_shortening_service`.

App-repo companion (`college_craft`, not this repo): deploy workflows still call
`--app client_site` (template leftover). They must become `--app college_craft`
before CI deploy will hit this catalog entry. Default branch is `master` (not `main`).

`print_forge` extra env (control-node secrets, not in git) — **currently stopped**
([#33](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/33)):
`host_apps` rows stay with `enabled: false`. Bring back with `enabled` removed/true
then `deploy.sh`. Secrets kept:
`~/Documents/secrets/print_forge/print_forge_prod.env` and
`print_forge_beta.env`. Template: app repo `.env.prod.example`.
`DJANGO_ENV` / `DJANGO_ALLOWED_HOSTS` / `DATABASE_URL` / `WEB_PORT` /
`PUBLIC_SITE_URL` differ per env (prod `printforgeprints.com` / `:8007`;
beta `print-forge-preview.aimloperations.com` / `:8019`).
`SITE_UNDER_CONSTRUCTION=true` on prod (holding page until launch), `false` on beta.
Purchased flags: `FEATURE_EMAIL_SMS`, `FEATURE_PAYMENTS`, `FEATURE_SHOP`,
`FEATURE_SHIPPING`. Other `FEATURE_*` stay false. Add a `print_forge:<token>`
entry to `SHORTENER_API_TOKENS` on `url_shortening_service` (beta token first).
`SHORTENER_BASE_URL` is prod shortener on prod, `https://shortener-beta.aimloperations.com`
on beta. Default branch is `master`.

Companion app ticket: [print_forge#1](https://git.aimloperations.com/ai_ml_operations/print_forge/issues/1)
([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)).
App Gitea workflow still calls `--app print_forge` on `master`; deploy **no-ops**
while `enabled: false`.

`abc_be` / `abc_worker` / `abc_fe` (GIS org, not this repo) — **beta only** ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)):

Control-node secrets (never git, mode `600`):

```text
~/Documents/secrets/abc_be/abc_be_beta.env
~/Documents/secrets/abc_worker/abc_worker_beta.env
```

Templates: `GIS/abc_be` `.env.beta.example`, `GIS/abc_worker` `.env.beta.example`.
`abc_fe` has no secret file — public `VITE_*` URLs live in committed `.env.beta`.
Create Postgres DB `abc_be_beta` and grant `westfarn` before first deploy.
NPM: `beta.abc.aimloperations.com` → `:8085`, `beta.abc.be.aimloperations.com` → `:8017`,
`beta.abc.worker.aimloperations.com` → `:8018` (enable WebSocket). No prod ABC
`host_apps` rows; ports **8009** / 8008 / 8084 reserved (`print_forge` took prod **8007**).

Companion workflows: [abc_be#22](https://git.aimloperations.com/GIS/abc_be/issues/22),
[abc_worker#27](https://git.aimloperations.com/GIS/abc_worker/issues/27),
[abc_fe#27](https://git.aimloperations.com/GIS/abc_fe/issues/27).

`livekit` (GIS org, SFU on webservers — [abc_worker#14](https://git.aimloperations.com/GIS/abc_worker/issues/14)–[#17](https://git.aimloperations.com/GIS/abc_worker/issues/17)):

Control-node secret (never git, mode `600`):

```text
~/Documents/secrets/livekit/livekit_beta.env
```

Template: `GIS/livekit` `.env.beta.example`. Same `LIVEKIT_API_KEY` /
`LIVEKIT_API_SECRET` pair as `abc_be` / `abc_worker`; worker `LIVEKIT_URL=wss://<LIVEKIT_DOMAIN>`.

Deploy: `./scripts/deploy.sh --app livekit --env beta` (all webservers, like
`abc_worker`). NPM: `LIVEKIT_DOMAIN` → **one** host `:7880` with WebSocket
(adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that same host.
Do not active/active-balance LiveKit until Redis is in the compose.

`dta_blog` ([#29](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/29)) —
Python SSG, **no secrets file**. `npm run build:<env>` writes
`/var/www/<env>.blog.realpath.app/html`. Prod (`:8086`) omits `demo: true` seed
posts; beta (`:8087`) includes them. Header/footer links are baked at build
(`https://realpath.app` vs `https://beta.realpath.app`); Tianji website ids
are likewise baked (`cmtvvmf562afjzqumwt1yh2y8` / `cmtvvn6z62agdzqumtk8xijpy`).
Default branch is `main`.

NPM / DNS / Cloudflare (Ansible does not manage these): `blog.realpath.app` →
`:8086`, `beta.blog.realpath.app` → `:8087`, TLS like the other RealPath names.
Do **not** mount this on `realpath.app/blog`. Static ports stay LAN / NPM-only
(UFW does not world-open 8086/8087).

Companion app ticket: [dta_blog#1](https://git.aimloperations.com/Ditch_The_Agent/dta_blog/issues/1).
Gitea deploy can later call `./scripts/deploy.sh --app dta_blog --env beta|prod`.

### One-time host bootstrap (per target)

- [x] Gitea SSH key: the `gitea-key` role (in `site.yml`) generates a key per
      server, configures SSH for port 30009, probes access, and — if the server
      can't reach Gitea yet — prints the public key to add and stops. Add the key
      (Gitea user SSH keys, or repo Deploy Keys) and re-run provisioning.
- [ ] Create control-node secrets `{{ secrets_dir }}/<app>/<app>_<env>.env`
      (default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL`
      (see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port
      table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on
      adama + roslin + starbuck + apollo. Never committed to git.
- [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs`
      role in `site.yml` (NodeSource, `node_major` default 20).

## Gitea Act Runner

**Recommended:** Single self-hosted runner on ai-server-4080.

- One orchestration point.
- App hosts (adama/roslin/starbuck/apollo) run the workloads; no runner needed on them for deploy fan-out.
- Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).

### Runner requirements on ai-server-4080

| Requirement | Why |
|-------------|-----|
| Ansible | Run `deploy-apps.yml` |
| `server-infra` checkout | Playbooks + inventory |
| SSH key to app hosts | Deploy fan-out |

On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml`
fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act
runner so playbooks and inventory stay current without a manual `git pull`.

## SSH Keys for CI Deploy

| Key | Used by | Purpose |
|-----|---------|---------|
| Personal key | You | Manual provisioning |
| Deploy key (runner) | Act → Ansible → hosts | Automated deploy |

Consider a dedicated `deploy` user with limited sudo (docker only) — future hardening step.

## Secrets (Phase 2)

Use Ansible Vault for production secrets. Do not commit plaintext.

```bash
ansible-vault create inventory/group_vars/webservers/vault.yml
ansible-playbook playbooks/site.yml --ask-vault-pass
```

Store vault password for CI in a file readable only by the Act runner (e.g. `~/.ansible-vault-pass`, mode 600).

## Implementation Order

| # | Task | Status |
|---|------|--------|
| 1 | Create `server-infra` repo | Done |
| 2 | Inventory with all 5 hosts | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 3 | Bootstrap SSH to app hosts | Manual |
| 4 | `site.yml` → common, ufw, docker | Done |
| 5 | Verify `ansible webservers -m ping` | Manual |
| 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
| 7 | Provision all: `./scripts/provision.sh` | Manual |
| 8 | Deploy SSH key for Act runner | Future |
| 8a | Auto-sync runner checkout on `master` (`.gitea/workflows/sync-checkout.yml`) | Done |
| 9 | Stub `deploy-apps.yml` + update `company_site` workflow | Future |
| 10 | Dockerize `company_site` | Future (separate ticket) |
| 10a | Register + deploy `scha` (all webservers, port 8002) | In progress ([scha#19](https://git.aimloperations.com/ai_ml_operations/scha/issues/19)) |
| 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
| 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
| 10d | Register + deploy `monica_site` (django, ports 8004/8014) | Done ([#14](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/14)) |
| 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | Done ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) |
| 10f | Add starbuck + apollo as app hosts (same workloads as roslin) | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 10g | Register + deploy `url_shortening_service` (django, ports 8005/8015) | Done ([#22](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/22)) |
| 10h | Register + deploy `college_craft` (django, ports 8006/8016) | Done ([#24](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/24)) |
| 10i | Register + deploy ABC beta (`abc_be` 8017, `abc_worker` 8018, `abc_fe` 8085) | Done ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)) |
| 10j | Register + deploy `print_forge` (django, ports 8007/8019) | Done ([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)) |
| 10k | Register + deploy `dta_blog` (node-static, ports 8086/8087) | Done ([#29](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/29)) |
| 10l | Register GIS LiveKit SFU (`abc_worker`-style docker on webservers, port 7880) | This PR |
| 11 | Gitea container registry (optional) | Future |

## Open Decisions

1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
2. **NPM load balancing** — confirm jc21 NPM can express all app-host upstreams (Advanced tab), else active/active is just independent instances.
3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`).

## Adding a New VM

1. Add host to `inventory/hosts.yml` under `webservers`.
2. Bootstrap SSH: `ssh-copy-id westfarn@<new-ip>`.
3. Test: `./scripts/provision.sh <hostname> --check`.
4. Provision: `./scripts/provision.sh <hostname>`.
5. Deploys automatically include new host once in `webservers` group.