diff --git a/README.md b/README.md
index c8958f0..4038f8e 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,8 @@ ansible adama -m ping
 
 See [IMPLEMENTATION.md](IMPLEMENTATION.md) for full architecture, CI/CD plan, and phase breakdown.
 
+Observability (Alloy → Loki / Prometheus → Grafana): [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md) · [docs/GRAFANA_USAGE.md](docs/GRAFANA_USAGE.md)
+
 ## Servers
 
 | Host | IP | Role |
diff --git a/docs/GRAFANA_USAGE.md b/docs/GRAFANA_USAGE.md
new file mode 100644
index 0000000..8e3440b
--- /dev/null
+++ b/docs/GRAFANA_USAGE.md
@@ -0,0 +1,774 @@
+# Grafana — Using Logs, Metrics, Dashboards, and Alerts
+
+Companion to [OBSERVABILITY.md](OBSERVABILITY.md). Assumes Alloy is shipping
+**logs → Loki** and **metrics → Prometheus**, and Grafana is reachable (LAN
+or your public URL).
+
+This guide is for day-to-day use: Explore logs and metrics, build **container
+health** and **system status** dashboards, and send alerts via **SMTP2GO**
+(email and SMS-via-email gateways).
+
+---
+
+## 1. First login and basics
+
+1. Open Grafana (`https://grafana.YOURDOMAIN` or `http://10.0.0.128:3000`).
+2. Log in as `admin` (password you set during bootstrap).
+3. Confirm **Connections → Data sources**:
+   - **Prometheus** — green “Data source is working”
+   - **Loki** — green “Data source is working”
+
+### UI map (what you will use most)
+
+| Area | Path | Use for |
+|------|------|---------|
+| Explore | left menu → **Explore** | Ad-hoc log search (Loki) or metric queries (Prometheus) |
+| Dashboards | **Dashboards** | Container health, system status, log overviews |
+| Alerting | **Alerting** | Rules, contact points, notification policies |
+| Admin | **Administration** | Users, SMTP, org settings |
+
+Pick the datasource at the top of Explore / each panel:
+
+| Want… | Datasource |
+|-------|------------|
+| Log lines, error text | **Loki** |
+| CPU / RAM / disk / container up | **Prometheus** |
+
+---
+
+## 2. Labels you can filter on
+
+Alloy attaches these labels (see OBSERVABILITY.md). Prefer **label filters**
+first; use text search (`|=`, `|~`) on logs second.
+
+### Logs (Loki)
+
+| Label | Meaning | Examples |
+|-------|---------|----------|
+| `host` | Inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `env` | Deploy environment | `beta`, `prod`, `host` (journal), `infra` (stack containers) |
+| `app` | App / service name | `company_site`, `dta_service`, `dta_webapp`, `system` |
+| `job` | Collector | `docker`, `systemd` |
+| `unit` | systemd unit (journal) | `docker.service`, `ssh.service` |
+| `container` | Docker container name | `company_site_prod-web-1` |
+
+### Metrics (Prometheus)
+
+| Label | Meaning | Examples |
+|-------|---------|----------|
+| `host` | Inventory hostname (stamped by Alloy) | `adama`, `roslin`, `ai-server-4080` |
+| `job` | Scrape job | `node` (host), `cadvisor` (containers) |
+| `env` / `app` | Parsed from Compose project `<app>_<env>` | `prod` / `dta_service` |
+| `name` | Container name (cAdvisor) | `company_site_prod-web-1` |
+| `device` / `mountpoint` | Disk (node metrics) | `sda`, `/` |
+
+### Mental model
+
+```text
+LOGS:    {host, env, app, job}  ← filters
+         |= "ERROR"             ← text search after labels
+
+METRICS: metric_name{host, env, app, name, job}  ← PromQL selectors
+         rate(...[5m]) / sum by (...)            ← aggregations
+```
+
+---
+
+## 3. Explore — logs (LogQL)
+
+Open **Explore** → pick **Loki** → switch to **Code** mode (easier while learning).
+
+### 3.1 Starter queries
+
+```logql
+# All recent logs from one host
+{host="adama"}
+
+# One app in prod on every host (active/active)
+{app="dta_service", env="prod"}
+
+# Beta only on roslin
+{host="roslin", env="beta"}
+
+# Host OS / systemd (not containers)
+{job="systemd", host="ai-server-4080"}
+
+# Docker only
+{job="docker", host="adama"}
+```
+
+### 3.2 Text filters (after labels)
+
+```logql
+# Lines containing ERROR (case-sensitive)
+{app="company_site", env="prod"} |= "ERROR"
+
+# Case-insensitive
+{app="dta_service", env="prod"} |~ "(?i)error|exception|traceback"
+
+# Exclude noise
+{job="docker", app="dta_webapp"} != "healthcheck" != "favicon"
+```
+
+### 3.3 Parse JSON logs (if an app logs JSON)
+
+```logql
+{app="dta_service", env="prod"}
+  | json
+  | level="ERROR"
+```
+
+Only works when the **line** is JSON. If not, stick to `|=` / `|~`.
+
+### 3.4 Metrics from logs (for graphs / alerts)
+
+```logql
+# Log lines per second, by host
+sum by (host) (rate({job="docker"}[5m]))
+
+# Error-ish lines per minute for one app
+sum by (host) (
+  count_over_time({app="dta_service", env="prod"} |~ "(?i)error|exception"[1m])
+)
+```
+
+Tips:
+
+- Time range picker (top right): start with **Last 15 minutes**.
+- Click a log line → expand → see all labels.
+- **Add to dashboard** (panel menu) once a query looks useful.
+- Live tail: Explore → enable live (good while reproducing a bug).
+
+---
+
+## 4. Explore — metrics (PromQL)
+
+Open **Explore** → pick **Prometheus** → **Code** mode.
+
+### 4.1 Are agents reporting?
+
+```promql
+# 1 = healthy scrape path from Alloy → Prometheus
+up{job="node"}
+up{job="cadvisor"}
+
+# Count series by host
+count by (host, job) (up)
+```
+
+Every host should show `up == 1` for both `node` and `cadvisor`.
+
+### 4.2 System status (per host)
+
+```promql
+# CPU % busy
+100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+
+# Load (1 / 5 / 15 minute)
+node_load1
+node_load5
+node_load15
+
+# Memory % used
+100 * (1 - (
+  node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
+))
+
+# Root disk % used (path may be /host/root when Alloy uses rootfs_path)
+100 - (
+  (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
+    / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"})
+  * 100
+)
+
+# Network receive/transmit bytes/sec (exclude lo)
+sum by (host, device) (
+  rate(node_network_receive_bytes_total{device!="lo"}[5m])
+)
+sum by (host, device) (
+  rate(node_network_transmit_bytes_total{device!="lo"}[5m])
+)
+```
+
+If disk queries return nothing, open Explore → Metrics browser → search
+`node_filesystem` and check which `mountpoint` label your hosts use
+(sometimes `/`, sometimes `/host/root`).
+
+### 4.3 Container health and resource usage
+
+cAdvisor exposes many series. Filter with `name!=""` to skip blank/cgroup roots.
+
+```promql
+# Memory working set (best "RSS-like" view) per container
+sum by (host, name, app, env) (
+  container_memory_working_set_bytes{name!=""}
+)
+
+# CPU usage in cores (1.0 = one full core)
+sum by (host, name, app, env) (
+  rate(container_cpu_usage_seconds_total{name!=""}[5m])
+)
+
+# Network I/O
+sum by (host, name) (
+  rate(container_network_receive_bytes_total{name!=""}[5m])
+)
+sum by (host, name) (
+  rate(container_network_transmit_bytes_total{name!=""}[5m])
+)
+
+# One app across the fleet
+sum by (host, name) (
+  container_memory_working_set_bytes{app="dta_service", env="prod"}
+)
+
+# Container last-seen / presence (1 while metrics flow)
+up{job="cadvisor"}
+```
+
+**“Is the container healthy?”** in this stack means:
+
+| Signal | How to read it |
+|--------|----------------|
+| Metrics still arriving for `name=…` | Container (or cgroup) is running and Alloy can see it |
+| CPU / memory suddenly zero then missing | Container stopped or host/Alloy down |
+| Memory climbing without bound | Leak / need limit |
+| Logs show crash loop + memory resets | Restart loop — correlate with Loki |
+
+Docker’s own `healthcheck` status is **not** a first-class Prometheus metric
+from cAdvisor. For true HEALTHCHECK state later you can add
+`docker events` scraping or an exporter; for now combine **metrics presence +
+resource graphs + Loki errors**.
+
+### 4.4 Split-pane: metrics + logs for one incident
+
+1. Explore → Prometheus → CPU/memory query for `{app="dta_service", env="prod"}`.
+2. Click **Split** (top) → second pane → Loki →
+   `{app="dta_service", env="prod"} |~ "(?i)error|traceback"`.
+3. Align time ranges — spike on left should line up with errors on right.
+
+---
+
+## 5. Dashboards
+
+Create a folder **Homelab** (Dashboards → New → New folder). Build these three
+first.
+
+### 5.1 Dashboard: System status (fleet)
+
+**Name:** `Fleet — System status`  
+**Datasource:** Prometheus
+
+1. **Dashboards → New → New dashboard**.
+2. Gear → **Variables**:
+
+| Name | Type | Definition |
+|------|------|------------|
+| `host` | Query | Prometheus → Label values → `host` (from metric `up`) · Multi · Include All |
+
+3. Add panels (all Prometheus, filter `host=~"$host"`):
+
+| Panel | Type | Query |
+|-------|------|-------|
+| CPU % busy | Time series | `100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle", host=~"$host"}[5m])) * 100)` |
+| Load 1m | Time series | `node_load1{host=~"$host"}` |
+| Memory % used | Time series | `100 * (1 - (node_memory_MemAvailable_bytes{host=~"$host"} / node_memory_MemTotal_bytes{host=~"$host"}))` |
+| Disk % used | Time series | See §4.2 disk query; add `host=~"$host"` |
+| Network RX/TX | Time series | `sum by (host, device) (rate(node_network_receive_bytes_total{host=~"$host", device!="lo"}[5m]))` (and transmit) |
+| Agent up | Stat | `up{job="node", host=~"$host"}` — thresholds: 1 green, 0 red |
+
+4. Save.
+
+Optional unit settings: CPU → `percent (0-100)`; memory/disk → percent;
+network → `bytes/sec (SI)`.
+
+### 5.2 Dashboard: Docker container health
+
+**Name:** `Fleet — Docker containers`  
+**Datasource:** Prometheus
+
+Variables:
+
+| Name | Type | Definition |
+|------|------|------------|
+| `host` | Query | label_values(`host`) · Multi · All |
+| `env` | Query | label_values(`env`) · Multi · All |
+| `app` | Query | label_values(`app`) · Multi · All |
+
+Common selector used below:
+
+```text
+host=~"$host", env=~"$env", app=~"$app", name!=""
+```
+
+| Panel | Type | Query / notes |
+|-------|------|----------------|
+| Containers reporting | Stat | `count(container_memory_working_set_bytes{host=~"$host", env=~"$env", app=~"$app", name!=""})` |
+| CPU by container | Time series | `sum by (host, name, app, env) (rate(container_cpu_usage_seconds_total{host=~"$host", env=~"$env", app=~"$app", name!=""}[5m]))` · Legend: `{{host}} / {{name}}` · Unit: short / cores |
+| Memory working set | Time series | `sum by (host, name, app, env) (container_memory_working_set_bytes{host=~"$host", env=~"$env", app=~"$app", name!=""})` · Unit: bytes (IEC) |
+| Memory top table | Table | Same memory query → Transform **Sort by** → Instant query · useful “who’s fattest” |
+| Network RX | Time series | `sum by (host, name) (rate(container_network_receive_bytes_total{host=~"$host", env=~"$env", app=~"$app", name!=""}[5m]))` |
+| Network TX | Time series | transmit counterpart |
+| Prod apps only (row) | — | Set variable `env=prod` or duplicate panels with `env="prod"` hard-coded |
+
+**Layout tip:** top row = Stat “how many containers” + Stat “hosts with cadvisor up”;
+middle = CPU + Memory; bottom = Network.
+
+### 5.3 Dashboard: Per-app (logs + metrics mixed)
+
+**Name:** `App — dta_service` (clone per app)
+
+Variables: `host`, `env` (default `prod`).
+
+| Panel | DS | Query |
+|-------|-----|-------|
+| CPU | Prometheus | `sum by (host, name) (rate(container_cpu_usage_seconds_total{app="dta_service", env=~"$env", host=~"$host", name!=""}[5m]))` |
+| Memory | Prometheus | `sum by (host, name) (container_memory_working_set_bytes{app="dta_service", env=~"$env", host=~"$host", name!=""})` |
+| Error log rate | Loki | `sum by (host) (count_over_time({app="dta_service", env=~"$env", host=~"$host"} \|~ "(?i)error\|exception\|traceback"[1m]))` |
+| Recent errors | Loki Logs | `{app="dta_service", env=~"$env", host=~"$host"} \|~ "(?i)error\|exception\|traceback"` |
+
+Clone dashboard → change `app="…"` in every panel for `company_site` /
+`dta_webapp`.
+
+### 5.4 Dashboard: Fleet logs (optional)
+
+Same as before — Loki-only overview:
+
+1. Logs panel: `{env="prod"}`
+2. Time series: `sum by (host) (rate({job="docker", env="prod"}[5m]))`
+3. Errors: `{env="prod"} |~ "(?i)error|exception"`
+
+### 5.5 Import community dashboards (fast start)
+
+**Dashboards → New → Import** → enter ID → choose **Prometheus** datasource.
+
+| ID | Name | Notes |
+|----|------|-------|
+| [1860](https://grafana.com/grafana/dashboards/1860) | Node Exporter Full | Excellent host CPU/RAM/disk/net. Our Alloy stamps `host`; you may need to edit panel queries from `instance` → `host`, or set the dashboard’s instance variable to match. |
+| [14282](https://grafana.com/grafana/dashboards/14282) | Cadvisor exporter | Container CPU/mem. Filter/adjust labels to `name`, `host`, `app`, `env`. |
+| [193](https://grafana.com/grafana/dashboards/193) | Docker monitoring (older) | Often needs label tweaks; prefer 14282 or the hand-built §5.2 board. |
+
+After import:
+
+1. Open a panel → Edit → fix label matchers (`instance` → `host=~"$host"`).
+2. Delete panels you do not care about (keeps the board readable).
+3. Save under folder **Homelab**.
+
+Hand-built §5.1 / §5.2 boards already use your `host` / `env` / `app` labels —
+prefer those if imports fight you.
+
+### 5.6 What “healthy” looks like on the Docker dashboard
+
+| Green | Investigate |
+|-------|-------------|
+| Container count stable for that env | Count dropped → container exited / host down |
+| CPU flat or gently varying | CPU pegged at ~cores available → thrash / loop |
+| Memory flat or sawtooth (GC) | Steady climb over hours → leak |
+| Network quiet except traffic spikes | Continuous TX flood → scrape/abuse/misconfig |
+| Loki error rate ~0 | Sustained errors → open split Explore |
+
+You do **not** need a separate “monitor” product — a **dashboard panel**
+visualizes; an **alert rule** watches a query and notifies you.
+
+---
+
+## 6. Alerting overview
+
+Grafana Alerting has three pieces:
+
+```text
+Alert rule          →  what to watch (LogQL / metric query + threshold)
+Contact point       →  how to notify (email, webhook, …)
+Notification policy →  which rules go to which contact points
+```
+
+Flow:
+
+1. Configure **SMTP** (SMTP2GO) so Grafana can send email.
+2. Create a **Contact point** (email, and optionally SMS-via-email).
+3. Create **Alert rules** that fire on PromQL (preferred for health) or LogQL rates.
+4. Route them with a **Notification policy** (default route is enough at first).
+
+---
+
+## 7. SMTP2GO — email (and SMS) from Grafana
+
+SMTP2GO is an SMTP relay. Grafana sends ordinary email through it. SMS is
+usually “email to carrier gateway” or SMTP2GO’s own SMS product — both end up
+as an email destination Grafana can use.
+
+### 7.1 Gather SMTP2GO settings
+
+From the SMTP2GO dashboard you need:
+
+| Setting | Typical SMTP2GO value |
+|---------|------------------------|
+| Host | `mail.smtp2go.com` |
+| Port | `587` (STARTTLS) or `465` (TLS) |
+| Username | SMTP2GO SMTP user |
+| Password | SMTP2GO SMTP password |
+| From address | A verified sender in SMTP2GO (e.g. `alerts@yourdomain.com`) |
+
+Confirm the **From** domain/sender is verified in SMTP2GO or mail will be rejected.
+
+### 7.2 Configure SMTP in Grafana (UI)
+
+1. **Administration → Default preferences** is not enough — use config or env.
+2. Easiest on this host: set env vars in
+   `/opt/apps/observability/docker-compose.yml` under `grafana.environment`,
+   then recreate the container.
+
+Example additions:
+
+```yaml
+environment:
+  GF_SECURITY_ADMIN_USER: admin
+  GF_SECURITY_ADMIN_PASSWORD: "…already set…"
+  GF_USERS_ALLOW_SIGN_UP: "false"
+  GF_SERVER_ROOT_URL: "https://grafana.aimloperations.com"
+  GF_SERVER_DOMAIN: "grafana.aimloperations.com"
+
+  # --- SMTP2GO ---
+  GF_SMTP_ENABLED: "true"
+  GF_SMTP_HOST: "mail.smtp2go.com:587"
+  GF_SMTP_USER: "your-smtp2go-username"
+  GF_SMTP_PASSWORD: "your-smtp2go-password"
+  GF_SMTP_FROM_ADDRESS: "alerts@yourdomain.com"
+  GF_SMTP_FROM_NAME: "Grafana Alerts"
+  GF_SMTP_STARTTLS_POLICY: "OpportunisticStartTLS"
+```
+
+**Do not commit** the password. Prefer an env file:
+
+```bash
+# /opt/apps/observability/grafana/smtp.env  (mode 600, not in git)
+GF_SMTP_ENABLED=true
+GF_SMTP_HOST=mail.smtp2go.com:587
+GF_SMTP_USER=...
+GF_SMTP_PASSWORD=...
+GF_SMTP_FROM_ADDRESS=alerts@yourdomain.com
+GF_SMTP_FROM_NAME=Grafana Alerts
+GF_SMTP_STARTTLS_POLICY=OpportunisticStartTLS
+```
+
+Wire it in compose:
+
+```yaml
+grafana:
+  env_file:
+    - ./grafana/smtp.env
+  environment:
+    GF_SECURITY_ADMIN_USER: admin
+    # …
+```
+
+Apply:
+
+```bash
+chmod 600 /opt/apps/observability/grafana/smtp.env
+cd /opt/apps/observability && docker compose up -d grafana
+```
+
+### 7.3 Contact point — email
+
+1. **Alerting → Contact points → Add contact point**.
+2. Name: `email-ops`.
+3. Integration: **Email**.
+4. Addresses: your inbox (comma-separated for several people).
+5. **Test** → you should receive a message via SMTP2GO.
+6. Save.
+
+### 7.4 Contact point — SMS via email gateway
+
+Carriers expose addresses like `number@txt.att.net`. Grafana still sends
+**email**; the carrier turns it into SMS.
+
+Examples (US — confirm with your carrier):
+
+| Carrier | Gateway pattern |
+|---------|-----------------|
+| AT&T | `10digit@txt.att.net` |
+| T-Mobile | `10digit@tmomail.net` |
+| Verizon | `10digit@vtext.com` |
+
+1. Add another contact point, e.g. `sms-ryan`.
+2. Integration: **Email**.
+3. Addresses: `5551234567@tmomail.net` (your number + gateway).
+4. Keep the message short — SMS truncates.
+5. **Test**.
+
+Alternatively, if you use **SMTP2GO SMS** (separate product/API), use a
+Grafana **Webhook** contact point to their SMS API instead of Email. Email-to-SMS
+is simpler and enough for most homelabs.
+
+### 7.5 Notification policy
+
+1. **Alerting → Notification policies**.
+2. Default policy → set **Contact point** to `email-ops`.
+3. Optional child policy:
+   - Matcher: `severity = critical`
+   - Contact point: `sms-ryan` (and/or email)
+4. Save.
+
+At small scale, routing everything to email and only critical rules to SMS is
+enough.
+
+---
+
+## 8. Create alert rules (practical examples)
+
+Path: **Alerting → Alert rules → New alert rule**.
+
+Prefer **Prometheus** rules for health/resources; use **Loki** for error-text
+spikes.
+
+### 8.1 Anatomy of an alert
+
+1. **Rule name**: e.g. `host high CPU` or `dta_service prod errors`
+2. **Query**: PromQL or LogQL that returns a **number** (not a raw log stream).
+3. **Reduce**: usually **Last** or **Mean**.
+4. **Threshold**: e.g. `IS ABOVE 90` or `IS BELOW 1`.
+5. **Evaluation**: folder + group; interval e.g. `1m`; pending period e.g. `5m`.
+6. **Labels**: `severity=warning|critical` for routing.
+7. **Summary**: include `{{ $labels.host }}` / `{{ $labels.name }}`.
+
+### 8.2 Example A — Alloy / host metrics missing (critical)
+
+**Name:** `host metrics down`  
+**Datasource:** Prometheus
+
+```promql
+up{job="node"}
+```
+
+**Condition:** WHEN last of A IS BELOW `1`  
+**For:** `5m`  
+**Labels:** `severity=critical`  
+**Summary:** `No node metrics from {{ $labels.host }} — host or Alloy down`
+
+Repeat pattern for cAdvisor:
+
+```promql
+up{job="cadvisor"}
+```
+
+### 8.3 Example B — expected prod container missing
+
+After you know the stable container `name` (from the Docker dashboard legend),
+alert when its memory series disappears.
+
+**Name:** `dta_service prod container missing`  
+**Datasource:** Prometheus
+
+```promql
+sum by (host) (
+  container_memory_working_set_bytes{
+    app="dta_service",
+    env="prod",
+    name!=""
+  }
+)
+```
+
+**Condition:** WHEN last of A IS BELOW `1` (bytes — effectively “no series” /
+near-zero; better: use **Alerting → No data** handling = **Alerting** on this
+query, or per-host:
+
+```promql
+sum(
+  container_memory_working_set_bytes{
+    host="adama",
+    app="dta_service",
+    env="prod",
+    name!=""
+  }
+)
+```
+
+**For:** `5m`  
+**Labels:** `severity=critical`, `app=dta_service`  
+**Summary:** `dta_service prod not reporting memory on {{ $labels.host }}`
+
+Create one rule per host you care about until you are comfortable with
+`sum by (host)` + no-data behaviour.
+
+### 8.4 Example C — host CPU high
+
+```promql
+100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+**Condition:** IS ABOVE `90`  
+**For:** `10m`  
+**severity:** `warning`  
+**Summary:** `CPU {{ $values.A }}% on {{ $labels.host }}`
+
+### 8.5 Example D — host disk almost full
+
+```promql
+100 - (
+  (
+    node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
+    / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}
+  ) * 100
+)
+```
+
+**Condition:** IS ABOVE `90`  
+**For:** `15m`  
+**severity:** `critical`  
+**Summary:** `Disk {{ $values.A }}% full on {{ $labels.host }} ({{ $labels.mountpoint }})`
+
+Confirm `mountpoint` labels in Explore first; restrict with
+`mountpoint="/"` or `mountpoint="/host/root"` if needed.
+
+### 8.6 Example E — container memory high
+
+```promql
+sum by (host, name, app, env) (
+  container_memory_working_set_bytes{env="prod", name!=""}
+)
+```
+
+**Condition:** IS ABOVE `1073741824` (1 GiB) — tune per app  
+**For:** `15m`  
+**severity:** `warning`  
+**Summary:** `{{ $labels.app }} {{ $labels.name }} using high memory on {{ $labels.host }}`
+
+### 8.7 Example F — app error spike (Loki)
+
+**Name:** `dta_service prod error spike`  
+**Datasource:** Loki
+
+```logql
+sum by (host) (
+  count_over_time(
+    {app="dta_service", env="prod"} |~ "(?i)error|exception|traceback"[5m]
+  )
+)
+```
+
+**Condition:** WHEN last of A IS ABOVE `5`  
+**For:** `5m`  
+**Labels:** `severity=warning`, `app=dta_service`  
+**Summary:** `dta_service prod errors on {{ $labels.host }}`
+
+### 8.8 Example G — SSH auth failures (Loki / journal)
+
+```logql
+sum by (host) (
+  count_over_time(
+    {job="systemd", unit="ssh.service"} |= "Failed password"[5m]
+  )
+)
+```
+
+Threshold: `IS ABOVE 20` over `5m`, severity `warning`.
+
+### 8.9 After saving a rule
+
+1. Wait for evaluation (**Normal** / **Pending** / **Firing**).
+2. Contact point **Test** already proved SMTP.
+3. Optionally lower threshold briefly to force a fire, confirm email/SMS, restore.
+
+---
+
+## 9. Alert hygiene (avoid pager fatigue)
+
+- Prefer PromQL health rules (`up`, disk, CPU) over noisy log greps.
+- Always **label-filter** before text-matching on Loki.
+- Use a **For** duration (`5m`–`15m`) so blips do not page you.
+- Start with **email only**; add SMS for `severity=critical` after a week.
+- Separate **beta** and **prod** rules — beta is noisier.
+- Mute windows: **Alerting → Silence** during planned deploys.
+- Revisit thresholds after a few days of real volume.
+
+---
+
+## 10. Users and access (optional)
+
+**Administration → Users**:
+
+- Invite a read-only viewer for others who only need Explore/Dashboards.
+- Keep `admin` for you; do not reuse the admin password elsewhere.
+- If Grafana is on the public internet via NPM, consider also:
+  - Strong admin password (required)
+  - Disabling sign-up (`GF_USERS_ALLOW_SIGN_UP=false` — already in compose)
+  - Optional: NPM Access List / Authelia later if you harden further
+
+---
+
+## 11. Workflow cheat sheet
+
+### Debug a production issue
+
+1. Open **Fleet — Docker containers** → set `app` / `env=prod` — CPU/mem weird?
+2. Explore → Prometheus → same selectors for detail.
+3. Split Explore → Loki → `{app="dta_service", env="prod"} |~ "(?i)error"`
+4. Narrow host if active/active: add `host="adama"`
+5. Optional: pin panels on an incident dashboard for the postmortem
+
+### After deploying a new app env
+
+1. Confirm Compose project is `<app>_<env>` so Alloy labels `app` / `env`
+2. Docker dashboard → container appears with CPU/mem
+3. Loki Explore → `{app="new_app", env="beta"}` shows lines
+4. Clone alert rules; change `app` matcher
+
+### Weekly health pass
+
+1. **Fleet — System status** — CPU / RAM / disk sane on all hosts?
+2. **Fleet — Docker containers** — expected prod count; no memory climbers
+3. **Alerting → Alert rules** — anything lingering in Firing?
+4. Disk on 4080:
+
+```bash
+du -sh /opt/apps/observability/loki/data \
+      /opt/apps/observability/prometheus/data
+```
+
+---
+
+## 12. Troubleshooting alerts / mail
+
+| Symptom | Check |
+|---------|--------|
+| Contact point Test fails | SMTP user/pass, From address verified in SMTP2GO, port 587 not blocked outbound |
+| PromQL rule never fires | Query in Explore returns data? Threshold direction correct (`ABOVE` vs `BELOW`)? |
+| Loki rule never fires | Needs a **number** — wrap with `count_over_time` / `rate` |
+| Rule always firing | Threshold too low; noisy pattern; raise threshold or lengthen **For** |
+| Container “missing” false alarm | Deploy renamed container; update `name`/`app` matchers |
+| SMS never arrives | Gateway address wrong; carrier blocks; try email first |
+| Duplicate notifications | Multiple contact points on default policy; simplify policy tree |
+| No labels in alert text | Use `{{ $labels.host }}`; ensure query uses `sum by (host)` / `by (name)` |
+
+Grafana container logs:
+
+```bash
+cd /opt/apps/observability && docker compose logs -f --tail=100 grafana
+```
+
+SMTP2GO activity log (in their web UI) shows whether the message was accepted
+and delivered.
+
+---
+
+## 13. What this stack does *not* cover (yet)
+
+| Need | Later addition |
+|------|----------------|
+| Docker HEALTHCHECK status as a metric | Small exporter / script, or Docker events → Alloy |
+| Blackbox HTTP uptime (public URL checks) | Blackbox exporter, Grafana Synthetic Monitoring, or keep Tianji |
+| On-call scheduling | Grafana OnCall / external PagerDuty |
+| Distributed traces | Tempo + app instrumentation |
+| Long-term cheap metrics at huge scale | Mimir / remote storage (not needed at 3 hosts) |
+
+You already have host + container **resource metrics**, **presence**, and
+**logs** — enough for solid health dashboards and alerts.
+
+---
+
+## Related docs
+
+- [OBSERVABILITY.md](OBSERVABILITY.md) — install Alloy / Loki / Prometheus / Grafana, NPM, Ansible plan
+- [IMPLEMENTATION.md](../IMPLEMENTATION.md) — server roles and deploy conventions (`<app>_<env>`)
diff --git a/docs/OBSERVABILITY.md b/docs/OBSERVABILITY.md
new file mode 100644
index 0000000..dce0778
--- /dev/null
+++ b/docs/OBSERVABILITY.md
@@ -0,0 +1,1030 @@
+# Observability — Alloy → Loki / Prometheus → Grafana
+
+Centralized **logs + metrics** for the homelab. Every managed host runs
+**Grafana Alloy** (collector). **Loki** (logs), **Prometheus** (metrics), and
+**Grafana** (UI / dashboards / alerts) run on **ai-server-4080**. Grafana is
+exposed through Nginx Proxy Manager at `10.0.0.230` on an external domain.
+
+```mermaid
+flowchart LR
+  subgraph hosts ["All webservers"]
+    A["adama\nAlloy"]
+    R["roslin\nAlloy"]
+    C["ai-server-4080\nAlloy"]
+  end
+
+  subgraph stack ["ai-server-4080"]
+    L["Loki :3100"]
+    P["Prometheus :9090"]
+    G["Grafana :3000"]
+  end
+
+  NPM["NPM\n10.0.0.230"]
+  DNS["grafana.YOURDOMAIN"]
+
+  A -->|logs| L
+  R -->|logs| L
+  C -->|logs| L
+  A -->|metrics| P
+  R -->|metrics| P
+  C -->|metrics| P
+  L --> G
+  P --> G
+  G --> NPM
+  DNS --> NPM
+```
+
+| Piece | Where | Role |
+|-------|--------|------|
+| **Alloy** | every host (`adama`, `roslin`, `ai-server-4080`) | Ship journald + Docker **logs** to Loki; scrape host + container **metrics** → Prometheus |
+| **Loki** | `ai-server-4080` only | Store and index logs |
+| **Prometheus** | `ai-server-4080` only | Store metrics (CPU, RAM, disk, container health) |
+| **Grafana** | `ai-server-4080` only | Explore logs/metrics, dashboards, alerts |
+| **NPM** | `10.0.0.230` (external) | TLS + public hostname → Grafana |
+
+**What you get in Grafana**
+
+| View | Datasource | Examples |
+|------|------------|----------|
+| App / host logs | Loki | `{host="adama", env="prod", app="dta_service"}` |
+| Container health | Prometheus | Up/down, restarts, CPU %, memory, network I/O per container |
+| System status | Prometheus | Host CPU, RAM, disk, load, network |
+
+This guide is written for a first-time setup. Do the **manual bootstrap** on
+ai-server-4080 first so you can see logs **and** metrics flowing. Then wire
+Alloy into Ansible provision (`site.yml`) so new hosts get it automatically.
+
+---
+
+## Prerequisites
+
+- Docker + Compose already on all three hosts (`roles/docker` via `./scripts/provision.sh`)
+- LAN reachability: hosts can talk to `10.0.0.128:3100` (Loki) and
+  `10.0.0.128:9090` (Prometheus remote-write)
+- Access to Nginx Proxy Manager at `10.0.0.230`
+- A DNS name you control (example used below: `grafana.aimloperations.com` — replace with yours)
+- SMTP2GO account (for alerts; covered in [GRAFANA_USAGE.md](GRAFANA_USAGE.md))
+
+Suggested ports (keep free on ai-server-4080):
+
+| Service | Host port | Who can reach it |
+|---------|-----------|------------------|
+| Loki | `3100` | LAN only (`10.0.0.0/24`) |
+| Prometheus | `9090` | LAN only (`10.0.0.0/24`) — Alloy remote-write + Grafana |
+| Grafana | `3000` | LAN (NPM proxies to it); do **not** open to the public internet directly |
+
+---
+
+## Part 1 — Central stack on ai-server-4080 (Loki + Prometheus + Grafana)
+
+Run these steps **on ai-server-4080** as `westfarn`.
+
+### 1.1 Create directories
+
+```bash
+sudo mkdir -p /opt/apps/observability/{loki,prometheus,grafana,alloy}
+sudo chown -R westfarn:westfarn /opt/apps/observability
+mkdir -p /opt/apps/observability/loki/{data,rules}
+mkdir -p /opt/apps/observability/prometheus/data
+mkdir -p /opt/apps/observability/grafana/data
+```
+
+### 1.2 Loki config
+
+Create `/opt/apps/observability/loki/loki-config.yml`:
+
+```yaml
+# Managed manually for first bootstrap; later: roles/observability template.
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+  log_level: info
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+schema_config:
+  configs:
+    - from: "2024-01-01"
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+# Keep labels low-cardinality. host / env / app / job are fine.
+# Do NOT put request IDs, user IDs, or full paths in labels.
+limits_config:
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  ingestion_rate_mb: 16
+  ingestion_burst_size_mb: 32
+  max_query_series: 500
+  retention_period: 744h   # 31 days
+
+compactor:
+  working_directory: /loki/compactor
+  compaction_interval: 10m
+  retention_enabled: true
+  retention_delete_delay: 2h
+  delete_request_store: filesystem
+
+ruler:
+  alertmanager_url: http://localhost:9093   # unused for now; Grafana handles alerts
+```
+
+### 1.3 Prometheus config
+
+Create `/opt/apps/observability/prometheus/prometheus.yml`:
+
+```yaml
+# Managed manually for first bootstrap; later: roles/observability template.
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+# Alloy on each host *pushes* metrics here (remote_write).
+# We still keep a tiny local scrape so Prometheus has a self-health target.
+scrape_configs:
+  - job_name: prometheus
+    static_configs:
+      - targets: ["localhost:9090"]
+        labels:
+          host: ai-server-4080
+```
+
+Prometheus must accept remote-write from Alloy — that is enabled with the
+`--web.enable-remote-write-receiver` flag in compose (below).
+
+### 1.4 Docker Compose for Loki + Prometheus + Grafana
+
+Create `/opt/apps/observability/docker-compose.yml`:
+
+```yaml
+# Managed manually for first bootstrap; later: roles/observability template.
+services:
+  loki:
+    image: grafana/loki:3.4.2
+    container_name: loki
+    restart: unless-stopped
+    user: "0:0"
+    command: -config.file=/etc/loki/loki-config.yml
+    ports:
+      # Bind to all interfaces so Alloy on adama/roslin can push.
+      # Firewall (UFW) should restrict who can connect — see 1.6.
+      - "3100:3100"
+    volumes:
+      - ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
+      - ./loki/data:/loki
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3100/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+
+  prometheus:
+    image: prom/prometheus:v3.2.1
+    container_name: prometheus
+    restart: unless-stopped
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.path=/prometheus
+      - --storage.tsdb.retention.time=31d
+      - --web.enable-remote-write-receiver
+      - --web.enable-lifecycle
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus/data:/prometheus
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9090/-/ready || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+
+  grafana:
+    image: grafana/grafana:11.5.2
+    container_name: grafana
+    restart: unless-stopped
+    depends_on:
+      loki:
+        condition: service_healthy
+      prometheus:
+        condition: service_healthy
+    ports:
+      # LAN only is fine; NPM on 10.0.0.230 will proxy to this.
+      - "3000:3000"
+    environment:
+      GF_SECURITY_ADMIN_USER: admin
+      # Change immediately after first login. Prefer a secrets file later.
+      GF_SECURITY_ADMIN_PASSWORD: "CHANGE_ME_ON_FIRST_LOGIN"
+      GF_USERS_ALLOW_SIGN_UP: "false"
+      GF_SERVER_ROOT_URL: "https://grafana.aimloperations.com"
+      GF_SERVER_DOMAIN: "grafana.aimloperations.com"
+      # SMTP is configured in the UI or via GF_SMTP_* — see GRAFANA_USAGE.md
+    volumes:
+      - ./grafana/data:/var/lib/grafana
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+```
+
+Replace `grafana.aimloperations.com` with your real hostname in
+`GF_SERVER_ROOT_URL` and `GF_SERVER_DOMAIN`.
+
+### 1.5 Provision Grafana datasources (Loki + Prometheus)
+
+Create provisioning so Grafana auto-connects on every restart:
+
+```bash
+mkdir -p /opt/apps/observability/grafana/provisioning/datasources
+```
+
+Create `/opt/apps/observability/grafana/provisioning/datasources/datasources.yml`:
+
+```yaml
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    isDefault: false
+    editable: false
+    jsonData:
+      maxLines: 1000
+
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false
+    jsonData:
+      timeInterval: 15s
+```
+
+Prometheus is the default datasource so new panels default to metrics;
+switch to Loki in Explore when searching logs.
+
+### 1.6 Firewall — allow Loki + Prometheus from LAN only
+
+On **ai-server-4080**, open ingest ports to the LAN (not the world):
+
+```bash
+sudo ufw allow from 10.0.0.0/24 to any port 3100 proto tcp comment 'Loki ingest from Alloy'
+sudo ufw allow from 10.0.0.0/24 to any port 9090 proto tcp comment 'Prometheus remote-write from Alloy'
+sudo ufw allow from 10.0.0.0/24 to any port 3000 proto tcp comment 'Grafana for NPM'
+sudo ufw status numbered
+```
+
+Later, when this is Ansible-managed, add these ports to a host-specific UFW
+allow list in `inventory/host_vars/ai-server-4080.yml` rather than opening
+them on every webserver via `ufw_allowed_tcp_ports`.
+
+### 1.7 Start the stack
+
+```bash
+cd /opt/apps/observability
+docker compose up -d
+docker compose ps
+curl -s http://127.0.0.1:3100/ready          # expect: ready
+curl -s http://127.0.0.1:9090/-/ready        # expect: Prometheus Server is Ready.
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3000/login   # expect: 200
+```
+
+Open `http://10.0.0.128:3000` from your LAN, log in with `admin` /
+`CHANGE_ME_ON_FIRST_LOGIN`, and set a strong password when prompted.
+
+Confirm both datasources: **Connections → Data sources** → Loki and
+Prometheus both **Save & test** green.
+
+---
+
+## Part 2 — Expose Grafana on an external domain (NPM)
+
+Nginx Proxy Manager lives at **10.0.0.230** and is **not** managed by this
+repo. Do this in the NPM UI.
+
+### 2.1 DNS
+
+Create an A (or CNAME) record for your chosen hostname pointing at the public
+IP that reaches NPM (same pattern as your other `*.aimloperations.com` hosts).
+
+Example: `grafana.aimloperations.com` → your public IP / NPM.
+
+### 2.2 Proxy Host in NPM
+
+1. Log into Nginx Proxy Manager.
+2. **Hosts → Proxy Hosts → Add Proxy Host**.
+3. **Details**
+   - Domain Names: `grafana.aimloperations.com` (your name)
+   - Scheme: `http`
+   - Forward Hostname / IP: `10.0.0.128` (ai-server-4080)
+   - Forward Port: `3000`
+   - Cache Assets: off
+   - Block Common Exploits: on
+   - Websockets Support: **on** (Grafana live features need this)
+4. **SSL**
+   - Request a new Let's Encrypt certificate
+   - Force SSL: on
+   - HTTP/2: on
+5. Save.
+
+### 2.3 Confirm
+
+```bash
+curl -sI https://grafana.aimloperations.com/login
+```
+
+You should get HTTP 200 (or a redirect to login). Update
+`GF_SERVER_ROOT_URL` if the hostname differs, then:
+
+```bash
+cd /opt/apps/observability && docker compose up -d grafana
+```
+
+---
+
+## Part 3 — Alloy on each machine (logs + metrics)
+
+Alloy runs on **every** host in `webservers`. It:
+
+1. Reads **systemd journal** (host / service logs) → Loki
+2. Reads **Docker container logs** → Loki
+3. Scrapes **host metrics** (`prometheus.exporter.unix` ≈ node_exporter) → Prometheus
+4. Scrapes **container metrics** (`prometheus.exporter.cadvisor`) → Prometheus
+5. Attaches labels: `host`, `env`, `app`, `job`, `unit` / `container` / `name`
+6. Pushes logs to `http://10.0.0.128:3100/loki/api/v1/push`
+7. Remote-writes metrics to `http://10.0.0.128:9090/api/v1/write`
+
+### Labels we care about
+
+**Logs (Loki)**
+
+| Label | Source | Example values |
+|-------|--------|----------------|
+| `host` | Ansible inventory hostname | `adama`, `roslin`, `ai-server-4080` |
+| `job` | collector name | `systemd`, `docker` |
+| `env` | Docker Compose project suffix | `beta`, `prod`, `host`, `infra` |
+| `app` | Compose project prefix | `company_site`, `dta_service`, `dta_webapp`, … |
+| `unit` | systemd unit (journal only) | `docker.service`, `ssh.service` |
+| `container` | Docker container name | `company_site_prod-web-1` |
+
+**Metrics (Prometheus)** — same `host` / `env` / `app` idea, plus cAdvisor fields:
+
+| Label | Source | Example values |
+|-------|--------|----------------|
+| `host` | Added by Alloy relabel | `adama`, `roslin`, `ai-server-4080` |
+| `job` | scrape job name | `node`, `cadvisor` |
+| `name` | container name (cAdvisor) | `company_site_prod-web-1` |
+| `container_label_com_docker_compose_project` | Compose project | `dta_service_prod` |
+| `env` / `app` | Parsed from Compose project by Alloy | `prod` / `dta_service` |
+| `instance` | scrape target | usually the Alloy exporter address |
+
+Your deploy convention already names Compose projects `<app>_<env>`
+(e.g. `company_site_prod`). Alloy parses that for both logs and metrics:
+
+```logql
+{host="adama", env="prod", app="dta_service"}
+```
+
+```promql
+container_memory_usage_bytes{host="adama", env="prod", app="dta_service"}
+```
+
+### 3.1 Install Alloy (manual — one host)
+
+Repeat on `adama`, `roslin`, and `ai-server-4080`. Example for **adama**:
+
+```bash
+sudo mkdir -p /opt/apps/observability/alloy
+sudo chown -R westfarn:westfarn /opt/apps/observability
+```
+
+Create `/opt/apps/observability/alloy/config.alloy`. **Change every
+`host = "adama"`** on each machine (`adama` / `roslin` / `ai-server-4080`):
+
+```river
+// Grafana Alloy — logs → Loki, metrics → Prometheus.
+// host label MUST match inventory hostname.
+
+// =====================================================================
+// LOGS
+// =====================================================================
+
+// ---- systemd journal ----
+loki.source.journal "system" {
+  forward_to    = [loki.process.journal_labels.receiver]
+  relabel_rules = discovery.relabel.journal.rules
+  labels        = {
+    job  = "systemd",
+    host = "adama",   // <<< CHANGE PER HOST
+  }
+}
+
+discovery.relabel "journal" {
+  targets = []
+
+  rule {
+    source_labels = ["__journal__systemd_unit"]
+    target_label  = "unit"
+  }
+}
+
+loki.process "journal_labels" {
+  forward_to = [loki.write.default.receiver]
+
+  stage.static_labels {
+    values = {
+      env = "host",
+      app = "system",
+    }
+  }
+}
+
+// ---- Docker containers (logs) ----
+discovery.docker "containers" {
+  host = "unix:///var/run/docker.sock"
+}
+
+discovery.relabel "docker" {
+  targets = discovery.docker.containers.targets
+
+  rule {
+    source_labels = ["__meta_docker_container_name"]
+    regex         = "/(.*)"
+    target_label  = "container"
+  }
+
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
+    target_label  = "compose_project"
+  }
+
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
+    target_label  = "compose_service"
+  }
+}
+
+loki.source.docker "containers" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.docker.output
+  forward_to = [loki.process.docker_labels.receiver]
+  labels     = {
+    job  = "docker",
+    host = "adama",   // <<< CHANGE PER HOST
+  }
+}
+
+loki.process "docker_labels" {
+  forward_to = [loki.write.default.receiver]
+
+  // compose_project is like "company_site_prod" or "dta_service_beta"
+  stage.regex {
+    source     = "compose_project"
+    expression = "^(?P<app>[a-z0-9_]+)_(?P<env>beta|prod)$"
+  }
+
+  stage.labels {
+    values = {
+      app = "",
+      env = "",
+    }
+  }
+
+  // Containers not named <app>_<env> (loki, grafana, web-static, …)
+  stage.template {
+    source   = "env"
+    template = `{{ if .env }}{{ .env }}{{ else }}infra{{ end }}`
+  }
+
+  stage.template {
+    source   = "app"
+    template = `{{ if .app }}{{ .app }}{{ else }}{{ .compose_project }}{{ end }}`
+  }
+
+  stage.labels {
+    values = {
+      app = "",
+      env = "",
+    }
+  }
+}
+
+loki.write "default" {
+  endpoint {
+    url = "http://10.0.0.128:3100/loki/api/v1/push"
+  }
+}
+
+// =====================================================================
+// METRICS — host (CPU / RAM / disk / load / network)
+// =====================================================================
+
+prometheus.exporter.unix "node" {
+  // When Alloy runs in Docker, point at the mounted host filesystem.
+  // Matches the volume mounts in alloy/docker-compose.yml below.
+  procfs_path = "/host/proc"
+  sysfs_path  = "/host/sys"
+  rootfs_path = "/host/root"
+}
+
+prometheus.scrape "node" {
+  targets         = prometheus.exporter.unix.node.targets
+  forward_to      = [prometheus.relabel.add_host.receiver]
+  scrape_interval = "15s"
+  job_name        = "node"
+}
+
+// =====================================================================
+// METRICS — Docker containers (cAdvisor)
+// =====================================================================
+
+prometheus.exporter.cadvisor "docker" {
+  docker_host       = "unix:///var/run/docker.sock"
+  docker_only       = true
+  storage_duration  = "5m"
+  // Keep cardinality down: only promote Compose labels we care about.
+  store_container_labels = false
+  allowlisted_container_labels = [
+    "com.docker.compose.project",
+    "com.docker.compose.service",
+  ]
+}
+
+prometheus.scrape "cadvisor" {
+  targets         = prometheus.exporter.cadvisor.docker.targets
+  forward_to      = [prometheus.relabel.cadvisor_labels.receiver]
+  scrape_interval = "15s"
+  job_name        = "cadvisor"
+}
+
+// Parse compose project → app + env; stamp inventory hostname.
+prometheus.relabel "cadvisor_labels" {
+  forward_to = [prometheus.remote_write.default.receiver]
+
+  rule {
+    target_label = "host"
+    replacement  = "adama"   // <<< CHANGE PER HOST
+  }
+
+  // container_label_com_docker_compose_project → compose_project helper
+  rule {
+    source_labels = ["container_label_com_docker_compose_project"]
+    target_label  = "compose_project"
+  }
+
+  // company_site_prod → app=company_site
+  rule {
+    source_labels = ["compose_project"]
+    regex         = "^([a-z0-9_]+)_(beta|prod)$"
+    target_label  = "app"
+    replacement   = "${1}"
+  }
+
+  // company_site_prod → env=prod
+  rule {
+    source_labels = ["compose_project"]
+    regex         = "^([a-z0-9_]+)_(beta|prod)$"
+    target_label  = "env"
+    replacement   = "${2}"
+  }
+
+  // Non-app containers (loki, grafana, alloy, web-static, …)
+  rule {
+    source_labels = ["env"]
+    regex         = "^$"
+    target_label  = "env"
+    replacement   = "infra"
+  }
+
+  rule {
+    source_labels = ["app"]
+    regex         = "^$"
+    target_label  = "app"
+    replacement   = "infra"
+  }
+}
+
+// Stamp host on node metrics too.
+prometheus.relabel "add_host" {
+  forward_to = [prometheus.remote_write.default.receiver]
+
+  rule {
+    target_label = "host"
+    replacement  = "adama"   // <<< CHANGE PER HOST
+  }
+}
+
+prometheus.remote_write "default" {
+  endpoint {
+    url = "http://10.0.0.128:9090/api/v1/write"
+  }
+}
+```
+
+### 3.2 Run Alloy with Docker Compose (per host)
+
+cAdvisor + node metrics need host filesystem mounts and privileged mode.
+Create `/opt/apps/observability/alloy/docker-compose.yml` on **each** host:
+
+```yaml
+services:
+  alloy:
+    image: grafana/alloy:v1.7.1
+    container_name: alloy
+    restart: unless-stopped
+    privileged: true
+    pid: host
+    command:
+      - run
+      - /etc/alloy/config.alloy
+      - --storage.path=/var/lib/alloy/data
+      - --server.http.listen-addr=0.0.0.0:12345
+    volumes:
+      - ./config.alloy:/etc/alloy/config.alloy:ro
+      - alloy-data:/var/lib/alloy/data
+      # Logs
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/log/journal:/var/log/journal:ro
+      - /etc/machine-id:/etc/machine-id:ro
+      - /run/systemd/journal:/run/systemd/journal:ro
+      # Metrics (node + cAdvisor) — host views
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/host/root:ro
+      - /var/run:/var/run:ro
+      - /var/lib/docker:/var/lib/docker:ro
+      - /dev/disk:/dev/disk:ro
+    # No published ports required — Alloy pushes outbound to Loki + Prometheus.
+    # Optional: expose Alloy's own UI/metrics on LAN for debugging:
+    # ports:
+    #   - "12345:12345"
+
+volumes:
+  alloy-data:
+```
+
+Start:
+
+```bash
+cd /opt/apps/observability/alloy
+docker compose up -d
+docker compose logs -f --tail=50
+```
+
+Healthy Alloy logs mention connecting / sending without repeated
+`connection refused` to either `:3100` or `:9090`.
+
+### 3.3 Verify logs in Grafana (Loki)
+
+1. Open Grafana → **Explore** → datasource **Loki**.
+2. Query:
+
+```logql
+{host=~".+"}
+```
+
+3. Confirm label browser shows `host`, `env`, `app`, `job`.
+4. Spot-check each machine:
+
+```logql
+{host="adama"}
+{host="roslin"}
+{host="ai-server-4080"}
+```
+
+5. Spot-check env / app:
+
+```logql
+{env="prod", app="company_site"}
+{env="beta", app="dta_service"}
+{job="systemd", host="adama"}
+```
+
+If Docker logs show `env="infra"`, that container is not a `<app>_<env>` Compose
+project (expected for `loki`, `grafana`, `alloy`, `web-static`).
+
+### 3.4 Verify metrics in Grafana (Prometheus)
+
+1. **Explore** → datasource **Prometheus**.
+2. Host metrics present:
+
+```promql
+up{job="node"}
+node_load1{host=~".+"}
+100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+3. Container metrics present:
+
+```promql
+up{job="cadvisor"}
+container_memory_usage_bytes{host=~".+", name!=""}
+rate(container_cpu_usage_seconds_total{host=~".+", name!=""}[5m])
+```
+
+4. Labels `host`, `env`, `app` on containers:
+
+```promql
+container_memory_usage_bytes{env="prod", app="dta_service"}
+```
+
+5. Quick check from the shell on ai-server-4080:
+
+```bash
+curl -sG 'http://127.0.0.1:9090/api/v1/label/host/values' | head
+curl -sG 'http://127.0.0.1:9090/api/v1/query' \
+  --data-urlencode 'query=count(up{job="cadvisor"})'
+```
+
+You should see all three hostnames under `host`, and `up` series for
+`job="node"` and `job="cadvisor"` from each Alloy.
+
+Dashboard build steps (container health, system status, imports) are in
+[GRAFANA_USAGE.md](GRAFANA_USAGE.md) §4–§5.
+
+---
+
+## Part 4 — Ansible provision (make Alloy automatic)
+
+Manual steps above prove the pipeline. Next, encode Alloy into provision so
+`./scripts/provision.sh` installs/updates it on every host, and keep Loki +
+Prometheus + Grafana only on the control node.
+
+### 4.1 Target layout (recommended)
+
+```
+roles/
+├── alloy/                         # every webserver
+│   ├── defaults/main.yml
+│   ├── tasks/main.yml
+│   └── templates/
+│       ├── config.alloy.j2
+│       └── docker-compose.yml.j2
+└── observability/                 # ai-server-4080 only
+    ├── defaults/main.yml
+    ├── tasks/main.yml
+    └── templates/
+        ├── docker-compose.yml.j2
+        ├── loki-config.yml.j2
+        ├── prometheus.yml.j2
+        └── grafana-datasources.yml.j2
+```
+
+### 4.2 Inventory vars (`inventory/group_vars/all.yml`)
+
+Add something like:
+
+```yaml
+# Observability
+loki_url: "http://10.0.0.128:3100"
+loki_push_url: "{{ loki_url }}/loki/api/v1/push"
+prometheus_url: "http://10.0.0.128:9090"
+prometheus_remote_write_url: "{{ prometheus_url }}/api/v1/write"
+alloy_image: "grafana/alloy:v1.7.1"
+alloy_dir: "{{ apps_base_dir }}/observability/alloy"
+observability_dir: "{{ apps_base_dir }}/observability"
+loki_image: "grafana/loki:3.4.2"
+prometheus_image: "prom/prometheus:v3.2.1"
+grafana_image: "grafana/grafana:11.5.2"
+grafana_public_url: "https://grafana.aimloperations.com"
+```
+
+### 4.3 Host flag (`inventory/host_vars/ai-server-4080.yml`)
+
+```yaml
+observability_stack: true   # run Loki + Prometheus + Grafana on this host
+```
+
+Other hosts omit the flag (or set `false`).
+
+### 4.4 Playbook wiring (`playbooks/site.yml`)
+
+```yaml
+---
+- name: Provision webservers
+  hosts: webservers
+  become: true
+  roles:
+    - common
+    - ufw
+    - docker
+    - nodejs
+    - gitea-key
+    - tianji
+    - alloy
+
+- name: Provision central observability stack
+  hosts: ai-server-4080
+  become: true
+  roles:
+    - role: observability
+      when: observability_stack | default(false)
+```
+
+Order matters: **docker** before **alloy** / **observability**.
+
+### 4.5 Alloy template essentials
+
+In `config.alloy.j2`, set host from inventory — never hardcode:
+
+```jinja
+labels = {
+  job  = "systemd",
+  host = "{{ inventory_hostname }}",
+}
+```
+
+Same for Docker log labels and every `prometheus.relabel` `replacement`
+for `host`. That is how `adama` / `roslin` / `ai-server-4080` stay correct
+without editing each file by hand.
+
+### 4.6 UFW for Loki + Prometheus (control node only)
+
+Do **not** put `3100` / `9090` in global `ufw_allowed_tcp_ports` (that opens
+on every host). Prefer a small task in `roles/observability` or a host_vars
+list:
+
+```yaml
+# host_vars/ai-server-4080.yml
+ufw_extra_rules:
+  - { port: 3100, proto: tcp, from_ip: "10.0.0.0/24", comment: "Loki" }
+  - { port: 9090, proto: tcp, from_ip: "10.0.0.0/24", comment: "Prometheus" }
+  - { port: 3000, proto: tcp, from_ip: "10.0.0.0/24", comment: "Grafana" }
+```
+
+(Extend `roles/ufw` to loop `ufw_extra_rules` when you implement this.)
+
+### 4.7 Secrets
+
+- Grafana admin password: store under `~/Documents/secrets/observability/`
+  (same pattern as app env files), push or template at provision time. **Never
+  commit** passwords or SMTP2GO keys to git.
+- SMTP2GO credentials: configure in Grafana UI or via env file on the host —
+  see [GRAFANA_USAGE.md](GRAFANA_USAGE.md).
+
+### 4.8 Apply
+
+```bash
+# After roles exist:
+./scripts/provision.sh ai-server-4080 --check
+./scripts/provision.sh ai-server-4080
+./scripts/provision.sh adama
+./scripts/provision.sh roslin
+# or all:
+./scripts/provision.sh
+```
+
+---
+
+## Part 5 — Day-2 operations
+
+### Useful LogQL starters
+
+```logql
+# Everything from one host
+{host="adama"}
+
+# One app in prod across all hosts
+{app="dta_service", env="prod"}
+
+# Errors (text match — tune per app)
+{app="company_site", env="prod"} |= "ERROR"
+
+# Rate of log lines per host (Explore → Build a dashboard)
+sum by (host) (rate({job="docker"}[5m]))
+```
+
+### Useful PromQL starters
+
+```promql
+# Host CPU % busy
+100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+
+# Host memory % used
+100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))
+
+# Disk % used (root)
+100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"}
+  * 100) / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"})
+
+# Container CPU cores used
+sum by (host, name, app, env) (
+  rate(container_cpu_usage_seconds_total{name!=""}[5m])
+)
+
+# Container memory working set
+sum by (host, name, app, env) (
+  container_memory_working_set_bytes{name!=""}
+)
+
+# Containers that disappeared / stopped reporting (up == 0)
+up{job="cadvisor"} == 0
+```
+
+### Check Alloy → Loki / Prometheus path
+
+On a shipper host:
+
+```bash
+cd /opt/apps/observability/alloy && docker compose logs --tail=100 alloy
+```
+
+On ai-server-4080:
+
+```bash
+curl -s "http://127.0.0.1:3100/loki/api/v1/label/host/values"
+curl -s "http://127.0.0.1:9090/api/v1/label/host/values"
+curl -sG "http://127.0.0.1:9090/api/v1/query" \
+  --data-urlencode 'query=count by (host, job) (up)'
+```
+
+### Restart / update images
+
+```bash
+cd /opt/apps/observability && docker compose pull && docker compose up -d
+cd /opt/apps/observability/alloy && docker compose pull && docker compose up -d
+```
+
+### Disk
+
+Loki + Prometheus retention are both ~31 days in the sample configs. Watch:
+
+```bash
+du -sh /opt/apps/observability/loki/data \
+      /opt/apps/observability/prometheus/data
+```
+
+---
+
+## Part 6 — Checklist
+
+### Central stack (ai-server-4080)
+
+- [ ] `/opt/apps/observability` created
+- [ ] Loki + Prometheus configs + Grafana datasource provisioning in place
+- [ ] `docker compose up -d` — Loki ready, Prometheus ready, Grafana login on `:3000`
+- [ ] Prometheus started with `--web.enable-remote-write-receiver`
+- [ ] UFW allows `3100`, `9090`, and `3000` from `10.0.0.0/24` only
+- [ ] Admin password changed from default
+- [ ] DNS + NPM proxy host with SSL + websockets
+- [ ] `GF_SERVER_ROOT_URL` matches public URL
+- [ ] Both Loki and Prometheus datasources green in Grafana
+
+### Alloy (each of adama, roslin, ai-server-4080)
+
+- [ ] `config.alloy` has correct `host = "..."` (or Ansible `inventory_hostname`)
+- [ ] Alloy container running privileged with host `/proc` `/sys` mounts
+- [ ] Can reach `10.0.0.128:3100` and `10.0.0.128:9090`
+- [ ] Grafana Explore (Loki) shows that host’s log labels
+- [ ] Grafana Explore (Prometheus) shows `up{job="node"}` and `up{job="cadvisor"}` for that host
+- [ ] Docker apps show `env` of `beta` or `prod` when Compose project is `<app>_<env>`
+
+### Dashboards / alerts
+
+- [ ] Container health + system status dashboards — [GRAFANA_USAGE.md](GRAFANA_USAGE.md)
+- [ ] SMTP2GO + alert rules for down containers / disk / CPU
+
+### Follow-ups
+
+- [ ] Encode `roles/alloy` + `roles/observability` and add to `site.yml`
+- [ ] Optional: drop or keep Tianji side-by-side (they do not conflict)
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---------|--------------|-----|
+| Alloy: connection refused to Loki | Loki down or UFW blocking | `docker compose ps` on 4080; `ufw status`; `curl 10.0.0.128:3100/ready` from shipper |
+| Alloy: connection refused to Prometheus | Prometheus down, no remote-write flag, or UFW | Confirm `--web.enable-remote-write-receiver`; `curl 10.0.0.128:9090/-/ready`; open `9090` from LAN |
+| No container metrics / empty cAdvisor | Missing privileged / mounts | Alloy compose needs `privileged: true`, `/sys`, `/var/lib/docker`, docker.sock |
+| Node metrics look like container’s own tiny FS | Wrong `rootfs_path` / mounts | Use `/host/proc`, `/host/sys`, `/host/root` mounts as in §3.2 |
+| No `env`/`app` on Docker logs/metrics | Container not from Compose, or missing compose labels | `docker inspect <id>` for `com.docker.compose.project` |
+| Wrong `host` label | Hardcoded wrong in `config.alloy` | Fix host string or use Ansible template |
+| Grafana behind NPM shows blank / websocket errors | Websockets off in NPM | Enable Websockets on the Proxy Host |
+| Grafana redirects to wrong host | `GF_SERVER_ROOT_URL` mismatch | Set to `https://your.domain` and recreate container |
+| Disk filling on 4080 | Retention / volume growth | Lower Loki `retention_period` / Prometheus `--storage.tsdb.retention.time` |
+
+---
+
+## Related docs
+
+- [GRAFANA_USAGE.md](GRAFANA_USAGE.md) — Explore, container/system dashboards, alerts, SMTP2GO
+- [IMPLEMENTATION.md](../IMPLEMENTATION.md) — overall Ansible architecture
+- [README.md](../README.md) — provision quick start