ai_ml_operations/server-infra
2
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Add starbuck and apollo as active/active app hosts.
Sync runner checkout / sync (pull_request) Successful in 7s
Details

Browse Source 
Mirror roslin host_apps (no monica worker) so NPM can balance the same
ports on 10.0.0.44 and 10.0.0.7 (closes #20).
This commit is contained in:
westfarn
2026-08-16 09:20:08 -05:00
parent 2e3fe13b39
commit 3016d3877a
12 changed files with 124 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
IMPLEMENTATION.md
+49 -32
View File
@@ -10,6 +10,8 @@ flowchart TB
Control1["ai-server-4080\n(control node)"] Control1["ai-server-4080\n(control node)"]
Control1 -->|ansible-playbook site.yml| Adama Control1 -->|ansible-playbook site.yml| Adama
Control1 -->|ansible-playbook site.yml| Roslin Control1 -->|ansible-playbook site.yml| Roslin
Control1 -->|ansible-playbook site.yml| Starbuck
Control1 -->|ansible-playbook site.yml| Apollo
end end
subgraph cicd ["CI/CD (every merge to master)"] subgraph cicd ["CI/CD (every merge to master)"]
@@ -19,6 +21,8 @@ flowchart TB
Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"] Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
AnsibleDeploy --> Adama2["adama"] AnsibleDeploy --> Adama2["adama"]
AnsibleDeploy --> Roslin2["roslin"] AnsibleDeploy --> Roslin2["roslin"]
AnsibleDeploy --> Starbuck2["starbuck"]
AnsibleDeploy --> Apollo2["apollo"]
end end
``` ```
@@ -35,6 +39,8 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
|------|-----|------| |------|-----|------|
| adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host | | adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
| roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host | | roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
| starbuck | 10.0.0.44 | Ubuntu Server VM (Proxmox) — app host |
| apollo | 10.0.0.7 | Ubuntu Server VM (Proxmox) — app host |
| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas | | ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner + Ollama + SearxNG + observability; also runs app replicas |
Hostname on this machine: `ryan-development-1` Hostname on this machine: `ryan-development-1`
@@ -52,8 +58,10 @@ server-infra/
│ ├── group_vars/ │ ├── group_vars/
│ │ └── all.yml # vars + app_catalog │ │ └── all.yml # vars + app_catalog
│ └── host_vars/ │ └── host_vars/
│ ├── adama.yml # host_apps (django + dta_webapp) │ ├── adama.yml # host_apps (django + static; monica worker)
│ ├── roslin.yml # host_apps (mirrors adama) │ ├── roslin.yml # host_apps (mirrors adama, no worker)
│ ├── starbuck.yml # host_apps (mirrors roslin)
│ ├── apollo.yml # host_apps (mirrors roslin)
│ └── ai-server-4080.yml # control node / act runner / SearxNG / observability │ └── ai-server-4080.yml # control node / act runner / SearxNG / observability
├── playbooks/ ├── playbooks/
│ ├── site.yml # Phase 1: provision │ ├── site.yml # Phase 1: provision
@@ -84,11 +92,15 @@ Ansible needs SSH + sudo on each target before playbooks work.
```bash ```bash
ssh-copy-id westfarn@10.0.0.77 ssh-copy-id westfarn@10.0.0.77
ssh-copy-id westfarn@10.0.0.176 ssh-copy-id westfarn@10.0.0.176
ssh-copy-id westfarn@10.0.0.44
ssh-copy-id westfarn@10.0.0.7
``` ```
3. Confirm passwordless SSH: 3. Confirm passwordless SSH:
```bash ```bash
ssh westfarn@10.0.0.77 ssh westfarn@10.0.0.77
ssh westfarn@10.0.0.176 ssh westfarn@10.0.0.176
ssh westfarn@10.0.0.44
ssh westfarn@10.0.0.7
``` ```
4. **First-time only** — grant passwordless sudo on each new host before the first 4. **First-time only** — grant passwordless sudo on each new host before the first
`provision.sh` run. Ubuntu 26.04 ships `sudo-rs` by default; Ansible's `provision.sh` run. Ubuntu 26.04 ships `sudo-rs` by default; Ansible's
@@ -142,6 +154,8 @@ New hosts need the one-time passwordless sudo bootstrap in
# Same for other hosts # Same for other hosts
./scripts/provision.sh roslin ./scripts/provision.sh roslin
./scripts/provision.sh starbuck
./scripts/provision.sh apollo
./scripts/provision.sh ai-server-4080 ./scripts/provision.sh ai-server-4080
``` ```
@@ -156,6 +170,8 @@ New hosts need the one-time passwordless sudo bootstrap in
```bash ```bash
./scripts/deploy.sh adama ./scripts/deploy.sh adama
./scripts/deploy.sh --check roslin ./scripts/deploy.sh --check roslin
./scripts/deploy.sh starbuck
./scripts/deploy.sh apollo
``` ```
Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`. Under the hood, scripts pass `--limit <hostname>` to `ansible-playbook`.
@@ -192,13 +208,13 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
| App | Type | Hosts | Envs | Notes | | App | Type | Hosts | Envs | Notes |
|-----|------|-------|------|-------| |-----|------|-------|------|-------|
| `company_site` | django (docker) | adama + roslin (+ ai-server-4080) | prod | active/active behind NPM; beta port reserved | | `company_site` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `dta_service` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM | | `dta_service` | django (docker) | all webservers | beta + prod | active/active behind NPM |
| `dta_webapp` | node/vite static | adama + roslin (+ ai-server-4080) | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx | | `dta_webapp` | node/vite static | all webservers | beta + prod | active/active; built to `/var/www/<env>.realpath.app/html`, served by web-static nginx |
| `scha` | django (docker) | adama + roslin + ai-server-4080 | prod | active/active behind NPM; beta port reserved | | `scha` | django (docker) | all webservers | prod | active/active behind NPM; beta port reserved |
| `chat_web_app` | node-static (CRA) | adama + roslin + ai-server-4080 | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx | | `chat_web_app` | node-static (CRA) | all webservers | beta + prod | active/active; built to `/var/www/<env>.chat.aimloperations/html`, served by web-static nginx |
| `chat_backend` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) | | `chat_backend` | django (docker) | all webservers | beta + prod | active/active behind NPM; Ollama `http://10.0.0.128:11434`; SearxNG `http://10.0.0.128:8088` (`SEARXNG_BASE_URL`) |
| `monica_site` | django (docker) | adama + roslin + ai-server-4080 | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` | | `monica_site` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres (like `scha`); dj-queue **worker singleton on adama** only (`compose --profile worker`); Ollama social drafting via `10.0.0.128:11434` |
Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
env file) so active/active replicas share one database. Beta and prod never share env file) so active/active replicas share one database. Beta and prod never share
@@ -211,7 +227,7 @@ a DB.
Optional `compose_profiles: [worker]` activates docker compose profiles on that Optional `compose_profiles: [worker]` activates docker compose profiles on that
host only (used for `monica_site` dj-queue singleton on adama). host only (used for `monica_site` dj-queue singleton on adama).
- Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`. - Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
Ports match across adama/roslin so NPM can balance `adama:PORT` + `roslin:PORT`. Ports match across app hosts so NPM can balance `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT`.
### Ports ### Ports
@@ -221,13 +237,13 @@ future beta replica.
| App | beta | prod | Deployed on | | App | beta | prod | Deployed on |
|-----|------|------|-------------| |-----|------|------|-------------|
| company_site | 8010 (*not deployed*) | 8000 | adama, roslin, ai-server-4080 | | company_site | 8010 (*not deployed*) | 8000 | all webservers |
| dta_service | 8011 | 8001 | adama, roslin, ai-server-4080 | | dta_service | 8011 | 8001 | all webservers |
| scha | 8012 (*not deployed*) | 8002 | adama, roslin, ai-server-4080 | | scha | 8012 (*not deployed*) | 8002 | all webservers |
| chat_backend | 8013 | 8003 | adama, roslin, ai-server-4080 | | chat_backend | 8013 | 8003 | all webservers |
| monica_site | 8014 | 8004 | adama, roslin, ai-server-4080 | | monica_site | 8014 | 8004 | all webservers |
| dta_webapp (nginx) | 8081 | 8080 | adama, roslin, ai-server-4080 | | dta_webapp (nginx) | 8081 | 8080 | all webservers |
| chat_web_app (nginx) | 8083 | 8082 | adama, roslin, ai-server-4080 | | chat_web_app (nginx) | 8083 | 8082 | all webservers |
| SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream | | SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
Host-local services on ai-server-4080 (not balanced by NPM): Host-local services on ai-server-4080 (not balanced by NPM):
@@ -277,18 +293,18 @@ point each domain at the backend(s):
- Single host: standard Proxy Host → `adama:PORT`. - Single host: standard Proxy Host → `adama:PORT`.
- Active/active: jc21 NPM's UI Proxy Host is single-target. To balance - Active/active: jc21 NPM's UI Proxy Host is single-target. To balance
adama+roslin you need the **Advanced** tab with a custom `upstream {}` block app hosts you need the **Advanced** tab with a custom `upstream {}` block
(or a real LB). Confirm this before relying on active/active. (or a real LB). Confirm this before relying on active/active.
| App | Domains | Backends | | App | Domains | Backends |
|-----|---------|----------| |-----|---------|----------|
| company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` | | company_site | aimloperations.com (+ www) | `adama:8000` + `roslin:8000` + `starbuck:8000` + `apollo:8000` |
| dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / ai-server-4080 | | dta_service | (see DTA NPM hosts) | `adama:8001` / `8011` + same on roslin / starbuck / apollo / ai-server-4080 |
| dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin | | dta_webapp | (see DTA NPM hosts) | `adama:8080` / `8081` + same on roslin / starbuck / apollo |
| scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` (+ `ai-server-4080:8002`) | | scha | `schawheaton.aimloperations.com`, `schawheaton.com` (+ www) | `adama:8002` + `roslin:8002` + `starbuck:8002` + `apollo:8002` (+ `ai-server-4080:8002`) |
| chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / ai-server-4080 | | chat_web_app | `chat.aimloperations.com` (+ www); `beta.chat.aimloperations.com` | `adama:8082` / `8083` + same on roslin / starbuck / apollo / ai-server-4080 |
| chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / ai-server-4080 | | chat_backend | `chatbackend.aimloperations.com`; `beta.chatbackend.aimloperations.com` | `adama:8003` / `8013` + same on roslin / starbuck / apollo / ai-server-4080 |
| monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / ai-server-4080 | | monica_site | `mkdrealtor.com` (+ www); `monica-preview.aimloperations.com` (beta) | `adama:8004` / `8014` + same on roslin / starbuck / apollo / ai-server-4080 |
### Required changes IN each app repo (owned separately) ### Required changes IN each app repo (owned separately)
@@ -338,7 +354,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
(default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL` (default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL`
(see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port (see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port
table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on
adama + roslin. Never committed to git. adama + roslin + starbuck + apollo. Never committed to git.
- [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs` - [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs`
role in `site.yml` (NodeSource, `node_major` default 20). role in `site.yml` (NodeSource, `node_major` default 20).
@@ -347,7 +363,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
**Recommended:** Single self-hosted runner on ai-server-4080. **Recommended:** Single self-hosted runner on ai-server-4080.
- One orchestration point. - One orchestration point.
- App hosts (adama/roslin) run the workloads; no runner needed on them for deploy fan-out. - App hosts (adama/roslin/starbuck/apollo) run the workloads; no runner needed on them for deploy fan-out.
- Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later). - Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).
### Runner requirements on ai-server-4080 ### Runner requirements on ai-server-4080
@@ -356,7 +372,7 @@ Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
|-------------|-----| |-------------|-----|
| Ansible | Run `deploy-apps.yml` | | Ansible | Run `deploy-apps.yml` |
| `server-infra` checkout | Playbooks + inventory | | `server-infra` checkout | Playbooks + inventory |
| SSH key to adama + roslin | Deploy fan-out | | SSH key to app hosts | Deploy fan-out |
On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml` On every push or merged PR to `master`, `.gitea/workflows/sync-checkout.yml`
fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act fast-forward pulls this repo at `~/Documents/repos/server-infra` on the Act
@@ -387,8 +403,8 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
| # | Task | Status | | # | Task | Status |
|---|------|--------| |---|------|--------|
| 1 | Create `server-infra` repo | Done | | 1 | Create `server-infra` repo | Done |
| 2 | Inventory with all 3 hosts | Done | | 2 | Inventory with all 5 hosts | Done ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 3 | Bootstrap SSH to adama + roslin | Manual | | 3 | Bootstrap SSH to app hosts | Manual |
| 4 | `site.yml` → common, ufw, docker | Done | | 4 | `site.yml` → common, ufw, docker | Done |
| 5 | Verify `ansible webservers -m ping` | Manual | | 5 | Verify `ansible webservers -m ping` | Manual |
| 6 | Test on single server: `./scripts/provision.sh adama` | Manual | | 6 | Test on single server: `./scripts/provision.sh adama` | Manual |
@@ -401,13 +417,14 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
| 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) | | 10b | Register + deploy `chat_web_app` (node-static, ports 8082/8083) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_web_app#35](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/35)) |
| 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) | | 10c | Register + deploy `chat_backend` (django, ports 8003/8013) | Done (prod); beta ([#7](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/7), [chat_backend#26](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/26)) |
| 10d | Register + deploy `monica_site` (django, ports 8004/8014) | Done ([#14](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/14)) | | 10d | Register + deploy `monica_site` (django, ports 8004/8014) | Done ([#14](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/14)) |
| 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | In progress ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) | | 10e | Auto-start `monica_site` dj-queue worker on adama (`compose_profiles`) | Done ([#17](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/17)) |
| 10f | Add starbuck + apollo as app hosts (same workloads as roslin) | This PR ([#20](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/20)) |
| 11 | Gitea container registry (optional) | Future | | 11 | Gitea container registry (optional) | Future |
## Open Decisions ## Open Decisions
1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI. 1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
2. **NPM load balancing** — confirm jc21 NPM can express adama+roslin upstreams (Advanced tab), else active/active is just two independent instances. 2. **NPM load balancing** — confirm jc21 NPM can express all app-host upstreams (Advanced tab), else active/active is just independent instances.
3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`). 3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`).
## Adding a New VM ## Adding a New VM
README.md
+4
View File
@@ -11,6 +11,8 @@ ansible-galaxy collection install -r requirements.yml
# Bootstrap SSH key to each host (one-time, before Ansible) # Bootstrap SSH key to each host (one-time, before Ansible)
ssh-copy-id westfarn@10.0.0.77 ssh-copy-id westfarn@10.0.0.77
ssh-copy-id westfarn@10.0.0.176 ssh-copy-id westfarn@10.0.0.176
ssh-copy-id westfarn@10.0.0.44
ssh-copy-id westfarn@10.0.0.7
# First-time only: passwordless sudo on each new host (before first provision) # First-time only: passwordless sudo on each new host (before first provision)
ssh -t westfarn@10.0.0.176 # repeat for each host IP ssh -t westfarn@10.0.0.176 # repeat for each host IP
@@ -46,4 +48,6 @@ the central **Loki / Prometheus / Grafana** stack (`observability_stack: true`).
|------|-----|------| |------|-----|------|
| adama | 10.0.0.77 | app host | | adama | 10.0.0.77 | app host |
| roslin | 10.0.0.176 | app host | | roslin | 10.0.0.176 | app host |
| starbuck | 10.0.0.44 | app host |
| apollo | 10.0.0.7 | app host |
| ai-server-4080 | 10.0.0.128 | control node + act runner | | ai-server-4080 | 10.0.0.128 | control node + act runner |
docs/GRAFANA_USAGE.md
+2 -2
View File
@@ -45,7 +45,7 @@ first; use text search (`|=`, `|~`) on logs second.
| Label | Meaning | Examples | | Label | Meaning | Examples |
|-------|---------|----------| |-------|---------|----------|
| `host` | Inventory hostname | `adama`, `roslin`, `ai-server-4080` | | `host` | Inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
| `env` | Deploy environment | `beta`, `prod`, `host` (journal), `infra` (stack containers) | | `env` | Deploy environment | `beta`, `prod`, `host` (journal), `infra` (stack containers) |
| `app` | App / service name | `company_site`, `dta_service`, `dta_webapp`, `system` | | `app` | App / service name | `company_site`, `dta_service`, `dta_webapp`, `system` |
| `job` | Collector | `docker`, `systemd` | | `job` | Collector | `docker`, `systemd` |
@@ -56,7 +56,7 @@ first; use text search (`|=`, `|~`) on logs second.
| Label | Meaning | Examples | | Label | Meaning | Examples |
|-------|---------|----------| |-------|---------|----------|
| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `ai-server-4080` | | `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
| `job` | Scrape job | `node` (host), `cadvisor` (containers) | | `job` | Scrape job | `node` (host), `cadvisor` (containers) |
| `env` / `app` | Parsed from Compose project `<app>_<env>` | `prod` / `dta_service` | | `env` / `app` | Parsed from Compose project `<app>_<env>` | `prod` / `dta_service` |
| `name` | Container name (cAdvisor) | `company_site_prod-web-1` | | `name` | Container name (cAdvisor) | `company_site_prod-web-1` |
docs/OBSERVABILITY.md
+17 -7
View File
@@ -10,6 +10,8 @@ flowchart LR
subgraph hosts ["All webservers"] subgraph hosts ["All webservers"]
A["adama\nAlloy"] A["adama\nAlloy"]
R["roslin\nAlloy"] R["roslin\nAlloy"]
S["starbuck\nAlloy"]
Ap["apollo\nAlloy"]
C["ai-server-4080\nAlloy"] C["ai-server-4080\nAlloy"]
end end
@@ -24,9 +26,13 @@ flowchart LR
A -->|logs| L A -->|logs| L
R -->|logs| L R -->|logs| L
S -->|logs| L
Ap -->|logs| L
C -->|logs| L C -->|logs| L
A -->|metrics| P A -->|metrics| P
R -->|metrics| P R -->|metrics| P
S -->|metrics| P
Ap -->|metrics| P
C -->|metrics| P C -->|metrics| P
L --> G L --> G
P --> G P --> G
@@ -36,7 +42,7 @@ flowchart LR
| Piece | Where | Role | | Piece | Where | Role |
|-------|--------|------| |-------|--------|------|
| **Alloy** | every host (`adama`, `roslin`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus | | **Alloy** | every host (`adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
| **Loki** | `ai-server-4080` only | Store and index logs | | **Loki** | `ai-server-4080` only | Store and index logs |
| **Prometheus** | `ai-server-4080` only | Store metrics (CPU, RAM, disk, container health) | | **Prometheus** | `ai-server-4080` only | Store metrics (CPU, RAM, disk, container health) |
| **Grafana** | `ai-server-4080` only | Explore logs/metrics, dashboards, alerts | | **Grafana** | `ai-server-4080` only | Explore logs/metrics, dashboards, alerts |
@@ -182,7 +188,7 @@ services:
user: "0:0" user: "0:0"
command: -config.file=/etc/loki/loki-config.yml command: -config.file=/etc/loki/loki-config.yml
ports: ports:
# Bind to all interfaces so Alloy on adama/roslin can push. # Bind to all interfaces so Alloy on app hosts can push.
# Firewall (UFW) should restrict who can connect — see 1.6. # Firewall (UFW) should restrict who can connect — see 1.6.
- "3100:3100" - "3100:3100"
volumes: volumes:
@@ -376,7 +382,7 @@ Alloy runs on **every** host in `webservers`. It:
| Label | Source | Example values | | Label | Source | Example values |
|-------|--------|----------------| |-------|--------|----------------|
| `host` | Ansible inventory hostname | `adama`, `roslin`, `ai-server-4080` | | `host` | Ansible inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
| `job` | collector name | `systemd`, `docker` | | `job` | collector name | `systemd`, `docker` |
| `env` | Docker Compose project suffix | `beta`, `prod`, `host`, `infra` | | `env` | Docker Compose project suffix | `beta`, `prod`, `host`, `infra` |
| `app` | Compose project prefix | `company_site`, `dta_service`, `dta_webapp`, … | | `app` | Compose project prefix | `company_site`, `dta_service`, `dta_webapp`, … |
@@ -387,7 +393,7 @@ Alloy runs on **every** host in `webservers`. It:
| Label | Source | Example values | | Label | Source | Example values |
|-------|--------|----------------| |-------|--------|----------------|
| `host` | Added by Alloy relabel | `adama`, `roslin`, `ai-server-4080` | | `host` | Added by Alloy relabel | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
| `job` | scrape job name | `node`, `cadvisor` | | `job` | scrape job name | `node`, `cadvisor` |
| `name` | container name (cAdvisor) | `company_site_prod-web-1` | | `name` | container name (cAdvisor) | `company_site_prod-web-1` |
| `container_label_com_docker_compose_project` | Compose project | `dta_service_prod` | | `container_label_com_docker_compose_project` | Compose project | `dta_service_prod` |
@@ -407,7 +413,7 @@ container_memory_usage_bytes{host="adama", env="prod", app="dta_service"}
### 3.1 Install Alloy (manual — one host) ### 3.1 Install Alloy (manual — one host)
Repeat on `adama`, `roslin`, and `ai-server-4080`. Example for **adama**: Repeat on `adama`, `roslin`, `starbuck`, `apollo`, and `ai-server-4080`. Example for **adama**:
```bash ```bash
sudo mkdir -p /opt/apps/observability/alloy sudo mkdir -p /opt/apps/observability/alloy
@@ -415,7 +421,7 @@ sudo chown -R westfarn:westfarn /opt/apps/observability
``` ```
Create `/opt/apps/observability/alloy/config.alloy`. **Change every Create `/opt/apps/observability/alloy/config.alloy`. **Change every
`host = "adama"`** on each machine (`adama` / `roslin` / `ai-server-4080`): `host = "adama"`** on each machine (`adama` / `roslin` / `starbuck` / `apollo` / `ai-server-4080`):
```river ```river
// Grafana Alloy — logs → Loki, metrics → Prometheus. // Grafana Alloy — logs → Loki, metrics → Prometheus.
@@ -705,6 +711,8 @@ Healthy Alloy logs mention connecting / sending without repeated
```logql ```logql
{host="adama"} {host="adama"}
{host="roslin"} {host="roslin"}
{host="starbuck"}
{host="apollo"}
{host="ai-server-4080"} {host="ai-server-4080"}
``` ```
@@ -816,6 +824,8 @@ observability_stack: true
./scripts/provision.sh ai-server-4080 ./scripts/provision.sh ai-server-4080
./scripts/provision.sh adama ./scripts/provision.sh adama
./scripts/provision.sh roslin ./scripts/provision.sh roslin
./scripts/provision.sh starbuck
./scripts/provision.sh apollo
# or all (site.yml orders stack before Alloy): # or all (site.yml orders stack before Alloy):
./scripts/provision.sh ./scripts/provision.sh
``` ```
@@ -916,7 +926,7 @@ du -sh /opt/apps/observability/loki/data \
- [ ] `GF_SERVER_ROOT_URL` matches public URL - [ ] `GF_SERVER_ROOT_URL` matches public URL
- [ ] Both Loki and Prometheus datasources green in Grafana - [ ] Both Loki and Prometheus datasources green in Grafana
### Alloy (each of adama, roslin, ai-server-4080) ### Alloy (each of adama, roslin, starbuck, apollo, ai-server-4080)
- [ ] `config.alloy` has correct `host = "..."` (or Ansible `inventory_hostname`) - [ ] `config.alloy` has correct `host = "..."` (or Ansible `inventory_hostname`)
- [ ] Alloy container running privileged with host `/proc` `/sys` mounts - [ ] Alloy container running privileged with host `/proc` `/sys` mounts
inventory/host_vars/adama.yml
+5 -4
View File
@@ -1,7 +1,8 @@
--- ---
# Django services run active/active here and on roslin (shared external DB). # Django services run active/active here and on roslin/starbuck/apollo
# dta_webapp static also runs active/active (built into /var/www, served by # (shared external DB). dta_webapp static also runs active/active (built into
# the web-static nginx container). Ports MUST match roslin so NPM can balance. # /var/www, served by the web-static nginx container). Ports MUST match the
# other app hosts so NPM can balance.
# Each entry is one workload: django -> compose project <name>_<env> on port; # Each entry is one workload: django -> compose project <name>_<env> on port;
# node-static -> /var/www/<env>_dta_webapp served on port. # node-static -> /var/www/<env>_dta_webapp served on port.
host_apps: host_apps:
@@ -16,6 +17,6 @@ host_apps:
- { name: chat_web_app, env: beta, port: 8083 } - { name: chat_web_app, env: beta, port: 8083 }
- { name: chat_backend, env: prod, port: 8003 } - { name: chat_backend, env: prod, port: 8003 }
- { name: chat_backend, env: beta, port: 8013 } - { name: chat_backend, env: beta, port: 8013 }
# compose_profiles: worker → dj-queue singleton (not on roslin / ai-server-4080). # compose_profiles: worker → dj-queue singleton (not on other app hosts).
- { name: monica_site, env: prod, port: 8004, compose_profiles: [worker] } - { name: monica_site, env: prod, port: 8004, compose_profiles: [worker] }
- { name: monica_site, env: beta, port: 8014, compose_profiles: [worker] } - { name: monica_site, env: beta, port: 8014, compose_profiles: [worker] }
inventory/host_vars/ai-server-4080.yml
+1 -1
View File
@@ -22,7 +22,7 @@ searxng_stack: true
gitea_key_path: "/home/{{ admin_user }}/.ssh/gitea_deploy" gitea_key_path: "/home/{{ admin_user }}/.ssh/gitea_deploy"
# company_site + dta_webapp + dta_service for side testing and active/active. # company_site + dta_webapp + dta_service for side testing and active/active.
# Ports MUST match adama/roslin so NPM can balance all three upstreams. # Ports MUST match adama/roslin/starbuck/apollo so NPM can balance all upstreams.
host_apps: host_apps:
- { name: company_site, env: prod, port: 8000 } - { name: company_site, env: prod, port: 8000 }
- { name: dta_webapp, env: prod, port: 8080 } - { name: dta_webapp, env: prod, port: 8080 }
inventory/host_vars/apollo.yml
+18
View File
@@ -0,0 +1,18 @@
---
# Mirrors roslin for active/active. Ports MUST match adama/roslin so NPM
# upstreams can balance apollo:PORT with the other app hosts.
host_apps:
- { name: company_site, env: prod, port: 8000 }
- { name: dta_service, env: prod, port: 8001 }
- { name: dta_service, env: beta, port: 8011 }
- { name: dta_webapp, env: prod, port: 8080 }
- { name: dta_webapp, env: beta, port: 8081 }
- { name: scha, env: prod, port: 8002 }
# optional: - { name: scha, env: beta, port: 8012 }
- { name: chat_web_app, env: prod, port: 8082 }
- { name: chat_web_app, env: beta, port: 8083 }
- { name: chat_backend, env: prod, port: 8003 }
- { name: chat_backend, env: beta, port: 8013 }
# monica_site worker NOT here — adama host_apps sets compose_profiles: [worker].
- { name: monica_site, env: prod, port: 8004 }
- { name: monica_site, env: beta, port: 8014 }
inventory/host_vars/roslin.yml
+2 -2
View File
@@ -1,6 +1,6 @@
--- ---
# Mirrors adama for active/active. Ports MUST match adama so NPM upstreams # Mirrors adama for active/active. Ports MUST match adama/starbuck/apollo so
# can balance adama:PORT and roslin:PORT for the same workload. # NPM upstreams can balance roslin:PORT with the other app hosts.
host_apps: host_apps:
- { name: company_site, env: prod, port: 8000 } - { name: company_site, env: prod, port: 8000 }
- { name: dta_service, env: prod, port: 8001 } - { name: dta_service, env: prod, port: 8001 }
inventory/host_vars/starbuck.yml
+18
View File
@@ -0,0 +1,18 @@
---
# Mirrors roslin for active/active. Ports MUST match adama/roslin so NPM
# upstreams can balance starbuck:PORT with the other app hosts.
host_apps:
- { name: company_site, env: prod, port: 8000 }
- { name: dta_service, env: prod, port: 8001 }
- { name: dta_service, env: beta, port: 8011 }
- { name: dta_webapp, env: prod, port: 8080 }
- { name: dta_webapp, env: beta, port: 8081 }
- { name: scha, env: prod, port: 8002 }
# optional: - { name: scha, env: beta, port: 8012 }
- { name: chat_web_app, env: prod, port: 8082 }
- { name: chat_web_app, env: beta, port: 8083 }
- { name: chat_backend, env: prod, port: 8003 }
- { name: chat_backend, env: beta, port: 8013 }
# monica_site worker NOT here — adama host_apps sets compose_profiles: [worker].
- { name: monica_site, env: prod, port: 8004 }
- { name: monica_site, env: beta, port: 8014 }
inventory/hosts.yml
+4
View File
@@ -7,5 +7,9 @@ all:
ansible_host: 10.0.0.77 ansible_host: 10.0.0.77
roslin: roslin:
ansible_host: 10.0.0.176 ansible_host: 10.0.0.176
starbuck:
ansible_host: 10.0.0.44
apollo:
ansible_host: 10.0.0.7
ai-server-4080: ai-server-4080:
ansible_host: 10.0.0.128 ansible_host: 10.0.0.128
scripts/deploy.sh
+2 -2
View File
@@ -14,7 +14,7 @@ Usage: $(basename "$0") [HOST] [OPTIONS]
Deploy applications with deploy-apps.yml. Deploy applications with deploy-apps.yml.
HOST Optional. Limit to one host: adama, roslin, or ai-server-4080. HOST Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
Options: Options:
--app NAME App to deploy (company_site, dta_service, dta_webapp, scha, chat_web_app, chat_backend, monica_site) --app NAME App to deploy (company_site, dta_service, dta_webapp, scha, chat_web_app, chat_backend, monica_site)
@@ -67,7 +67,7 @@ while [[ $# -gt 0 ]]; do
EXTRA_VARS+=(-e "$2") EXTRA_VARS+=(-e "$2")
shift 2 shift 2
;; ;;
adama|roslin|ai-server-4080) adama|roslin|starbuck|apollo|ai-server-4080)
LIMIT="$1" LIMIT="$1"
shift shift
;; ;;
scripts/provision.sh
+2 -2
View File
@@ -18,7 +18,7 @@ On ai-server-4080 also: observability (Loki + Prometheus + Grafana) when
observability_stack is true, and SearxNG when searxng_stack is true observability_stack is true, and SearxNG when searxng_stack is true
(chat_backend grounded search on :8088). (chat_backend grounded search on :8088).
HOST Optional. Limit to one host: adama, roslin, or ai-server-4080. HOST Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
Omit to run against all webservers. Omit to run against all webservers.
Options: Options:
@@ -59,7 +59,7 @@ while [[ $# -gt 0 ]]; do
EXTRA_ARGS+=(--ask-become-pass) EXTRA_ARGS+=(--ask-become-pass)
shift shift
;; ;;
adama|roslin|ai-server-4080) adama|roslin|starbuck|apollo|ai-server-4080)
LIMIT="$1" LIMIT="$1"
shift shift
;; ;;