Add starbuck and apollo as active/active app hosts (#21)

## Summary

- Closes [#20](#20).
- Register **starbuck** (`10.0.0.44`) and **apollo** (`10.0.0.7`) as Proxmox app hosts in `webservers`, with `host_apps` copied from **roslin** (same apps/ports; **no** `monica_site` worker — that stays a singleton on **adama**).
- Allow `starbuck` / `apollo` in `provision.sh` and `deploy.sh`; document inventory, NPM backends, and Alloy `host` labels.

## Test plan

- [ ] `ansible-inventory --list` shows both hosts under `webservers` at the expected IPs.
- [ ] Bootstrap SSH + passwordless sudo on each VM (keep a Proxmox console open for first UFW enable).
- [ ] `ansible starbuck,apollo -m ping`
- [ ] `./scripts/provision.sh starbuck --check` then `./scripts/provision.sh starbuck`
- [ ] `./scripts/provision.sh apollo --check` then `./scripts/provision.sh apollo`
- [ ] `./scripts/deploy.sh starbuck` and `./scripts/deploy.sh apollo` bring up the same app+env set as roslin.
- [ ] `monica_site` dj-queue worker still runs **only** on adama.
- [ ] Grafana/Loki show `host="starbuck"` and `host="apollo"`.
- [ ] NPM Advanced upstreams (manual): add `starbuck:PORT` and `apollo:PORT` beside adama/roslin.Reviewed-on: #21

docs/GRAFANA_USAGE.md

@@ -45,7 +45,7 @@ first; use text search (`|=`, `|~`) on logs second.
 
 | Label | Meaning | Examples |
 |-------|---------|----------|
-| `host` | Inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `env` | Deploy environment | `beta`, `prod`, `host` (journal), `infra` (stack containers) |
 | `app` | App / service name | `company_site`, `dta_service`, `dta_webapp`, `system` |
 | `job` | Collector | `docker`, `systemd` |
@@ -56,7 +56,7 @@ first; use text search (`|=`, `|~`) on logs second.
 
 | Label | Meaning | Examples |
 |-------|---------|----------|
-| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | Scrape job | `node` (host), `cadvisor` (containers) |
 | `env` / `app` | Parsed from Compose project `<app>_<env>` | `prod` / `dta_service` |
 | `name` | Container name (cAdvisor) | `company_site_prod-web-1` |

docs/OBSERVABILITY.md

@@ -10,6 +10,8 @@ flowchart LR
   subgraph hosts ["All webservers"]
     A["adama\nAlloy"]
     R["roslin\nAlloy"]
+    S["starbuck\nAlloy"]
+    Ap["apollo\nAlloy"]
     C["ai-server-4080\nAlloy"]
   end
 
@@ -24,9 +26,13 @@ flowchart LR
 
   A -->|logs| L
   R -->|logs| L
+  S -->|logs| L
+  Ap -->|logs| L
   C -->|logs| L
   A -->|metrics| P
   R -->|metrics| P
+  S -->|metrics| P
+  Ap -->|metrics| P
   C -->|metrics| P
   L --> G
   P --> G
@@ -36,7 +42,7 @@ flowchart LR
 
 | Piece | Where | Role |
 |-------|--------|------|
-| **Alloy** | every host (`adama`, `roslin`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
+| **Alloy** | every host (`adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
 | **Loki** | `ai-server-4080` only | Store and index logs |
 | **Prometheus** | `ai-server-4080` only | Store metrics (CPU, RAM, disk, container health) |
 | **Grafana** | `ai-server-4080` only | Explore logs/metrics, dashboards, alerts |
@@ -182,7 +188,7 @@ services:
     user: "0:0"
     command: -config.file=/etc/loki/loki-config.yml
     ports:
-      # Bind to all interfaces so Alloy on adama/roslin can push.
+      # Bind to all interfaces so Alloy on app hosts can push.
       # Firewall (UFW) should restrict who can connect — see 1.6.
       - "3100:3100"
     volumes:
@@ -376,7 +382,7 @@ Alloy runs on **every** host in `webservers`. It:
 
 | Label | Source | Example values |
 |-------|--------|----------------|
-| `host` | Ansible inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Ansible inventory hostname | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | collector name | `systemd`, `docker` |
 | `env` | Docker Compose project suffix | `beta`, `prod`, `host`, `infra` |
 | `app` | Compose project prefix | `company_site`, `dta_service`, `dta_webapp`, … |
@@ -387,7 +393,7 @@ Alloy runs on **every** host in `webservers`. It:
 
 | Label | Source | Example values |
 |-------|--------|----------------|
-| `host` | Added by Alloy relabel | `adama`, `roslin`, `ai-server-4080` |
+| `host` | Added by Alloy relabel | `adama`, `roslin`, `starbuck`, `apollo`, `ai-server-4080` |
 | `job` | scrape job name | `node`, `cadvisor` |
 | `name` | container name (cAdvisor) | `company_site_prod-web-1` |
 | `container_label_com_docker_compose_project` | Compose project | `dta_service_prod` |
@@ -407,7 +413,7 @@ container_memory_usage_bytes{host="adama", env="prod", app="dta_service"}
 
 ### 3.1 Install Alloy (manual — one host)
 
-Repeat on `adama`, `roslin`, and `ai-server-4080`. Example for **adama**:
+Repeat on `adama`, `roslin`, `starbuck`, `apollo`, and `ai-server-4080`. Example for **adama**:
 
 ```bash
 sudo mkdir -p /opt/apps/observability/alloy
@@ -415,7 +421,7 @@ sudo chown -R westfarn:westfarn /opt/apps/observability
 ```
 
 Create `/opt/apps/observability/alloy/config.alloy`. **Change every
-`host = "adama"`** on each machine (`adama` / `roslin` / `ai-server-4080`):
+`host = "adama"`** on each machine (`adama` / `roslin` / `starbuck` / `apollo` / `ai-server-4080`):
 
 ```river
 // Grafana Alloy — logs → Loki, metrics → Prometheus.
@@ -705,6 +711,8 @@ Healthy Alloy logs mention connecting / sending without repeated
 ```logql
 {host="adama"}
 {host="roslin"}
+{host="starbuck"}
+{host="apollo"}
 {host="ai-server-4080"}
 ```
 
@@ -816,6 +824,8 @@ observability_stack: true
 ./scripts/provision.sh ai-server-4080
 ./scripts/provision.sh adama
 ./scripts/provision.sh roslin
+./scripts/provision.sh starbuck
+./scripts/provision.sh apollo
 # or all (site.yml orders stack before Alloy):
 ./scripts/provision.sh
 ```
@@ -916,7 +926,7 @@ du -sh /opt/apps/observability/loki/data \
 - [ ] `GF_SERVER_ROOT_URL` matches public URL
 - [ ] Both Loki and Prometheus datasources green in Grafana
 
-### Alloy (each of adama, roslin, ai-server-4080)
+### Alloy (each of adama, roslin, starbuck, apollo, ai-server-4080)
 
 - [ ] `config.alloy` has correct `host = "..."` (or Ansible `inventory_hostname`)
 - [ ] Alloy container running privileged with host `/proc` `/sys` mounts