ai_ml_operations/server-infra
2
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Add starbuck (10.0.0.44) and apollo (10.0.0.7) as app hosts like roslin/adama #20

New Issue
Closed
opened 2026-08-16 07:17:28 -07:00 by westfarn · 0 comments
westfarn commented 2026-08-16 07:17:28 -07:00
Owner
Copy Link
Copy Source

Why

Add two new Proxmox Ubuntu app VMs to the Ansible inventory so they run the same active/active workloads as adama / roslin.

Host IP Role
starbuck 10.0.0.44 Ubuntu Server VM (Proxmox) — app host
apollo 10.0.0.7 Ubuntu Server VM (Proxmox) — app host

These are app hosts, not a control node. Do not enable observability_stack, searxng_stack, or act_runner_enabled. Do not copy ai-server-4080.yml.

Template

Copy inventory/host_vars/roslin.yml, not adama.

Roslin already mirrors adama's apps/ports for NPM, but omits monica_site compose_profiles: [worker]. The dj-queue worker stays a singleton on adama (#17). New hosts get web replicas only.

Ports must match adama/roslin so NPM can add starbuck:PORT and apollo:PORT as extra upstreams.

Apps / ports (same as roslin)

host_apps:
  - { name: company_site, env: prod, port: 8000 }
  - { name: dta_service, env: prod, port: 8001 }
  - { name: dta_service, env: beta, port: 8011 }
  - { name: dta_webapp, env: prod, port: 8080 }
  - { name: dta_webapp, env: beta, port: 8081 }
  - { name: scha, env: prod, port: 8002 }
  - { name: chat_web_app, env: prod, port: 8082 }
  - { name: chat_web_app, env: beta, port: 8083 }
  - { name: chat_backend, env: prod, port: 8003 }
  - { name: chat_backend, env: beta, port: 8013 }
  # monica_site worker NOT here — adama host_apps sets compose_profiles: [worker].
  - { name: monica_site, env: prod, port: 8004 }
  - { name: monica_site, env: beta, port: 8014 }

Provision roles (via webservers in site.yml): common, ufw, docker, nodejs, gitea-key, tianji, alloy.

Ansible / repo changes

  1. inventory/hosts.yml — add both under webservers: 
    starbuck:
  ansible_host: 10.0.0.44
apollo:
  ansible_host: 10.0.0.7
  2. inventory/host_vars/starbuck.yml and inventory/host_vars/apollo.yml — copy roslin.yml (same host_apps, no worker profiles).
  3. scripts/provision.sh and scripts/deploy.sh — extend the host allowlist (adama|roslin|ai-server-4080) and help text to include starbuck and apollo.
  4. Docs — add both hosts to server tables and examples:
    • README.md
    • IMPLEMENTATION.md (architecture diagram, servers table, host_vars layout, app host lists, NPM backends, CI fan-out notes)
    • docs/OBSERVABILITY.md / docs/GRAFANA_USAGE.md (Alloy host label examples)

Manual bootstrap (before first provision)

Same one-time steps as adama/roslin (IMPLEMENTATION.md Prerequisites). VMs assumed already exist at those IPs.

From ai-server-4080:

ssh-copy-id westfarn@10.0.0.44
ssh-copy-id westfarn@10.0.0.7

Passwordless sudo on each new host (Ubuntu 26.04 sudo-rs does not work with Ansible --ask-become-pass):

ssh -t westfarn@10.0.0.44   # then apollo at 10.0.0.7
echo 'westfarn ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/westfarn
sudo chmod 440 /etc/sudoers.d/westfarn

Keep a Proxmox console open when first enabling UFW.

Provision + deploy

ansible starbuck,apollo -m ping

./scripts/provision.sh starbuck --check
./scripts/provision.sh starbuck
./scripts/provision.sh apollo --check
./scripts/provision.sh apollo

# After docker group / SSH re-login as needed:
./scripts/deploy.sh starbuck
./scripts/deploy.sh apollo

Secrets already live on the control node (~/Documents/secrets/<app>/...); app-deploy pushes them. No new DBs — shared external Postgres, same as adama/roslin.

Out of Ansible (manual)

NPM at 10.0.0.230: add starbuck and apollo as extra upstreams on the existing active/active Proxy Hosts (Advanced upstream {}), same ports as adama/roslin. Ansible does not manage NPM.

Acceptance criteria

  • starbuck (10.0.0.44) and apollo (10.0.0.7) are in inventory/hosts.yml under webservers.
  • host_vars match roslin.yml (same apps/ports; no monica_site worker profiles).
  • ./scripts/provision.sh starbuck and ./scripts/provision.sh apollo succeed (UFW, Docker, Node, Gitea key, Tianji, Alloy).
  • ./scripts/deploy.sh starbuck and ./scripts/deploy.sh apollo bring up the same app+env set as roslin.
  • monica_site dj-queue worker still runs only on adama.
  • Alloy logs/metrics appear in Grafana/Loki with host="starbuck" and host="apollo".
  • Wrapper scripts accept starbuck / apollo as --limit hosts.
  • Docs list both hosts; NPM backends documented (actual NPM edit is manual).
## Why Add two new Proxmox Ubuntu app VMs to the Ansible inventory so they run the same active/active workloads as **adama** / **roslin**. | Host | IP | Role | |------|----|------| | starbuck | 10.0.0.44 | Ubuntu Server VM (Proxmox) — app host | | apollo | 10.0.0.7 | Ubuntu Server VM (Proxmox) — app host | These are **app hosts**, not a control node. Do **not** enable `observability_stack`, `searxng_stack`, or `act_runner_enabled`. Do **not** copy `ai-server-4080.yml`. ## Template Copy **`inventory/host_vars/roslin.yml`**, not adama. Roslin already mirrors adama's apps/ports for NPM, but **omits** `monica_site` `compose_profiles: [worker]`. The dj-queue worker stays a **singleton on adama** (#17). New hosts get web replicas only. Ports **must** match adama/roslin so NPM can add `starbuck:PORT` and `apollo:PORT` as extra upstreams. ## Apps / ports (same as roslin) ```yaml host_apps: - { name: company_site, env: prod, port: 8000 } - { name: dta_service, env: prod, port: 8001 } - { name: dta_service, env: beta, port: 8011 } - { name: dta_webapp, env: prod, port: 8080 } - { name: dta_webapp, env: beta, port: 8081 } - { name: scha, env: prod, port: 8002 } - { name: chat_web_app, env: prod, port: 8082 } - { name: chat_web_app, env: beta, port: 8083 } - { name: chat_backend, env: prod, port: 8003 } - { name: chat_backend, env: beta, port: 8013 } # monica_site worker NOT here — adama host_apps sets compose_profiles: [worker]. - { name: monica_site, env: prod, port: 8004 } - { name: monica_site, env: beta, port: 8014 } ``` Provision roles (via `webservers` in `site.yml`): `common`, `ufw`, `docker`, `nodejs`, `gitea-key`, `tianji`, `alloy`. ## Ansible / repo changes 1. **`inventory/hosts.yml`** — add both under `webservers`: ```yaml starbuck: ansible_host: 10.0.0.44 apollo: ansible_host: 10.0.0.7 ``` 2. **`inventory/host_vars/starbuck.yml`** and **`inventory/host_vars/apollo.yml`** — copy `roslin.yml` (same `host_apps`, no worker profiles). 3. **`scripts/provision.sh`** and **`scripts/deploy.sh`** — extend the host allowlist (`adama|roslin|ai-server-4080`) and help text to include `starbuck` and `apollo`. 4. **Docs** — add both hosts to server tables and examples: - `README.md` - `IMPLEMENTATION.md` (architecture diagram, servers table, host_vars layout, app host lists, NPM backends, CI fan-out notes) - `docs/OBSERVABILITY.md` / `docs/GRAFANA_USAGE.md` (Alloy `host` label examples) ## Manual bootstrap (before first provision) Same one-time steps as adama/roslin (`IMPLEMENTATION.md` Prerequisites). VMs assumed already exist at those IPs. From **ai-server-4080**: ```bash ssh-copy-id westfarn@10.0.0.44 ssh-copy-id westfarn@10.0.0.7 ``` Passwordless sudo on each new host (Ubuntu 26.04 `sudo-rs` does not work with Ansible `--ask-become-pass`): ```bash ssh -t westfarn@10.0.0.44 # then apollo at 10.0.0.7 echo 'westfarn ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/westfarn sudo chmod 440 /etc/sudoers.d/westfarn ``` Keep a **Proxmox console** open when first enabling UFW. ## Provision + deploy ```bash ansible starbuck,apollo -m ping ./scripts/provision.sh starbuck --check ./scripts/provision.sh starbuck ./scripts/provision.sh apollo --check ./scripts/provision.sh apollo # After docker group / SSH re-login as needed: ./scripts/deploy.sh starbuck ./scripts/deploy.sh apollo ``` Secrets already live on the control node (`~/Documents/secrets/<app>/...`); `app-deploy` pushes them. No new DBs — shared external Postgres, same as adama/roslin. ## Out of Ansible (manual) **NPM at 10.0.0.230:** add `starbuck` and `apollo` as extra upstreams on the existing active/active Proxy Hosts (Advanced `upstream {}`), same ports as adama/roslin. Ansible does not manage NPM. ## Acceptance criteria - [ ] `starbuck` (`10.0.0.44`) and `apollo` (`10.0.0.7`) are in `inventory/hosts.yml` under `webservers`. - [ ] `host_vars` match `roslin.yml` (same apps/ports; **no** `monica_site` worker profiles). - [ ] `./scripts/provision.sh starbuck` and `./scripts/provision.sh apollo` succeed (UFW, Docker, Node, Gitea key, Tianji, Alloy). - [ ] `./scripts/deploy.sh starbuck` and `./scripts/deploy.sh apollo` bring up the same app+env set as roslin. - [ ] `monica_site` dj-queue worker still runs **only** on adama. - [ ] Alloy logs/metrics appear in Grafana/Loki with `host="starbuck"` and `host="apollo"`. - [ ] Wrapper scripts accept `starbuck` / `apollo` as `--limit` hosts. - [ ] Docs list both hosts; NPM backends documented (actual NPM edit is manual).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/server-infra#20