ai_ml_operations/server-infra
2
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Add starbuck and apollo as active/active app hosts.
Sync runner checkout / sync (pull_request) Successful in 7s
Details

Browse Source 
Mirror roslin host_apps (no monica worker) so NPM can balance the same
ports on 10.0.0.44 and 10.0.0.7 (closes #20).
This commit is contained in:
westfarn
2026-08-16 09:20:08 -05:00
parent 2e3fe13b39
commit 3016d3877a
12 changed files with 124 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
IMPLEMENTATION.md
+49 -32
View File
@@ -10,6 +10,8 @@ flowchart TB
Control1["ai-server-4080\n(control node)"]
Control1 -->|ansible-playbook site.yml| Adama
Control1 -->|ansible-playbook site.yml| Roslin
Control1 -->|ansible-playbook site.yml| Starbuck
Control1 -->|ansible-playbook site.yml| Apollo
end
subgraph cicd ["CI/CD (every merge to master)"]
@@ -19,6 +21,8 @@ flowchart TB
Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
AnsibleDeploy --> Adama2["adama"]
AnsibleDeploy --> Roslin2["roslin"]
AnsibleDeploy --> Starbuck2["starbuck"]
AnsibleDeploy --> Apollo2["apollo"]
end
```
@@ -35,6 +39,8 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
|------|-----|------|
| adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
| roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
| starbuck | 10.0.0.44 | Ubuntu Server VM (Proxmox) — app host |
| apollo | 10.0.0.7 | Ubuntu Server VM (Proxmox) — app host |
| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
Hostname on this machine: `ryan-development-1`
@@ -52,8 +58,10 @@ server-infra/
│ ├── group_vars/
│ │ └── all.yml # vars + app_catalog
│ └── host_vars/
│ ├── adama.yml # host_apps (django + dta_webapp)
│ ├── roslin.yml # host_apps (mirrors adama)
│ ├── adama.yml # host_apps (django + static; monica worker)
│ ├── roslin.yml # host_apps (mirrors adama, no worker)
│ ├── starbuck.yml # host_apps (mirrors roslin)
│ ├── apollo.yml # host_apps (mirrors roslin)
│ └── ai-server-4080.yml # control node / act runner / SearxNG / observability
├── playbooks/
│ ├── site.yml # Phase 1: provision
@@ -84,11 +92,15 @@ Ansible needs SSH + sudo on each target before playbooks work.
```bash
ssh-copy-id westfarn@10.0.0.77
ssh-copy-id westfarn@10.0.0.176
ssh-copy-id westfarn@10.0.0.44
ssh-copy-id westfarn@10.0.0.7
```
3. Confirm passwordless SSH:
```bash
ssh westfarn@10.0.0.77
ssh westfarn@10.0.0.176
ssh westfarn@10.0.0.44
ssh westfarn@10.0.0.7
```
4. **First-time only** — grant passwordless sudo on each new host before the first
`provision.sh` run. Ubuntu 26.04 ships `sudo-rs` by default; Ansible's
@@ -142,6 +154,8 @@ New hosts need the one-time passwordless sudo bootstrap in
# Same for other hosts
./scripts/provision.sh roslin
./scripts/provision.sh starbuck
./scripts/provision.sh apollo
./scripts/provision.sh ai-server-4080
```
@@ -156,6 +170,8 @@ New hosts need the one-time passwordless sudo bootstrap in
```bash
./scripts/deploy.sh adama
./scripts/deploy.sh --check roslin
./scripts/deploy.sh starbuck
./scripts/deploy.sh apollo
```
Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`.
@@ -192,13 +208,13 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
| App | Type | Hosts | Envs | Notes |
|-----|------|-------|------|-------|
| `company_site` | django (docker) | adama + roslin (+ ai-server-4080) | prod | active/active behind NPM; beta port reserved |
| `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM |
| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx |
| `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved |
| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
| `monica_site` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
| `company_site` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `dta_service` | django (docker) | all webservers | beta + prod | active/active behind NPM |
| `dta_webapp` | node/vite static | all webservers | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx |
| `scha` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `chat_web_app` | node-static (CRA) | all webservers | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
| `chat_backend` | django (docker) | all webservers | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
| `monica_site` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
env file) so active/active replicas share one database. Beta and prod never share
@@ -211,7 +227,7 @@ a DB.
Optional `compose_profiles: [worker]` activates docker compose profiles on that
host only (used for `monica_site` dj-queue singleton on adama).
- Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
Ports match across adama/roslin so NPM can balance `adama:PORT` + `roslin:PORT`.
Ports match across app hosts so NPM can balance `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT`.
### Ports
@@ -221,13 +237,13 @@ future beta replica.
| App | beta | prod | Deployed on |
|-----|------|------|-------------|
| company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 |
| dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 |
| scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 |
| chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 |
| monica_site | 8014 | 8004 | adama, roslin, ai-server-4080 |
| dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 |
| chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 |
| company_site | 8010 (*not deployed*) | 8000 | all webservers |
| dta_service | 8011 | 8001 | all webservers |
| scha | 8012 (*not deployed*) | 8002 | all webservers |
| chat_backend | 8013 | 8003 | all webservers |
| monica_site | 8014 | 8004 | all webservers |
| dta_webapp (nginx) | 8081 | 8080 | all webservers |
| chat_web_app (nginx) | 8083 | 8082 | all webservers |
| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
Host-local services on ai-server-4080 (not balanced by NPM):
@@ -277,18 +293,18 @@ point each domain at the backend(s):
- Single host: standard Proxy Host → `adama:PORT`.
- Active/active: jc21 NPM's UI Proxy Host is single-target. To balance
adama+roslin you need the **Advanced** tab with a custom `upstream {}` block
app hosts you need the **Advanced** tab with a custom `upstream {}` block
(or a real LB). Confirm this before relying on active/active.
| App | Domains | Backends |
|-----|---------|----------|
| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` |
| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 |
| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin |
| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) |
| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / ai-server-4080 |
| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / ai-server-4080 |
| monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / ai-server-4080 |
| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` + `starbuck:8000` + `apollo:8000` |
| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / starbuck / apollo / ai-server-4080 |
| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin / starbuck / apollo |
| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` + `starbuck:8002` + `apollo:8002` (+ `ai-server-4080:8002`) |
| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / starbuck / apollo / ai-server-4080 |
| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / starbuck / apollo / ai-server-4080 |
| monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / starbuck / apollo / ai-server-4080 |
### Required changes IN each app repo (owned separately)
@@ -338,7 +354,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
(default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL`
(see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port
table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on
adama + roslin. Never committed to git.
adama + roslin + starbuck + apollo. Never committed to git.
- [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs`
role in `site.yml` (NodeSource, `node_major` default 20).
@@ -347,7 +363,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
**Recommended:** Single self-hosted runner on ai-server-4080.
- One orchestration point.
- App hosts (adama/roslin) run the workloads; no runner needed on them for deploy fan-out.
- App hosts (adama/roslin/starbuck/apollo) run the workloads; no runner needed on them for deploy fan-out.
- Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).
### Runner requirements on ai-server-4080
@@ -356,7 +372,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
|-------------|-----|
| Ansible | Run `deploy-apps.yml` |
| `server-infra` checkout | Playbooks + inventory |
| SSH key to adama + roslin | Deploy fan-out |
| SSH key to app hosts | Deploy fan-out |
On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml`
fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act
@@ -387,8 +403,8 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
| # | Task | Status |
|---|------|--------|
| 1 | Create `server-infra` repo | Done |
| 2 | Inventory with all 3 hosts | Done |
| 3 | Bootstrap SSH to adama + roslin | Manual |
| 2 | Inventory with all 5 hosts | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 3 | Bootstrap SSH to app hosts | Manual |
| 4 | `site.yml` → common, ufw, docker | Done |
| 5 | Verify `ansible webservers -m ping` | Manual |
| 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
@@ -401,13 +417,14 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
| 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
| 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
| 10d | Register + deploy `monica_site` (django, ports 8004/8014) | Done ([#14](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/14)) |
| 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | In progress ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) |
| 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | Done ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) |
| 10f | Add starbuck + apollo as app hosts (same workloads as roslin) | This PR ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 11 | Gitea container registry (optional) | Future |
## Open Decisions
1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
2. **NPM load balancing** — confirm jc21 NPM can express adama+roslin upstreams (Advanced tab), else active/active is just two independent instances.
2. **NPM load balancing** — confirm jc21 NPM can express all app-host upstreams (Advanced tab), else active/active is just independent instances.
3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`).
## Adding a New VM