Updates for grafana stack
This commit is contained in:
+25
-93
@@ -760,13 +760,18 @@ Dashboard build steps (container health, system status, imports) are in
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Ansible provision (make Alloy automatic)
|
||||
## Part 4 — Ansible provision (implemented)
|
||||
|
||||
Manual steps above prove the pipeline. Next, encode Alloy into provision so
|
||||
`./scripts/provision.sh` installs/updates it on every host, and keep Loki +
|
||||
Prometheus + Grafana only on the control node.
|
||||
Roles are in the repo. `./scripts/provision.sh` runs `site.yml`, which:
|
||||
|
||||
### 4.1 Target layout (recommended)
|
||||
1. Base roles on every host (`common` → `tianji`)
|
||||
2. **`observability`** on `ai-server-4080` when `observability_stack: true`
|
||||
(Loki + Prometheus + Grafana + UFW for `3100`/`9090`/`3000`)
|
||||
3. **`alloy`** on every host (logs → Loki, metrics → Prometheus)
|
||||
|
||||
Alloy is Grafana’s Promtail successor; one agent covers logs **and** metrics.
|
||||
|
||||
### 4.1 Layout
|
||||
|
||||
```
|
||||
roles/
|
||||
@@ -786,107 +791,32 @@ roles/
|
||||
└── grafana-datasources.yml.j2
|
||||
```
|
||||
|
||||
### 4.2 Inventory vars (`inventory/group_vars/all.yml`)
|
||||
### 4.2 Inventory
|
||||
|
||||
Add something like:
|
||||
Vars in `inventory/group_vars/all.yml` (`loki_url`, `prometheus_url`,
|
||||
`grafana_public_url`, …). Flag in `inventory/host_vars/ai-server-4080.yml`:
|
||||
|
||||
```yaml
|
||||
# Observability
|
||||
loki_url: "http://10.0.0.128:3100"
|
||||
loki_push_url: "{{ loki_url }}/loki/api/v1/push"
|
||||
prometheus_url: "http://10.0.0.128:9090"
|
||||
prometheus_remote_write_url: "{{ prometheus_url }}/api/v1/write"
|
||||
alloy_image: "grafana/alloy:v1.7.1"
|
||||
alloy_dir: "{{ apps_base_dir }}/observability/alloy"
|
||||
observability_dir: "{{ apps_base_dir }}/observability"
|
||||
loki_image: "grafana/loki:3.4.2"
|
||||
prometheus_image: "prom/prometheus:v3.2.1"
|
||||
grafana_image: "grafana/grafana:11.5.2"
|
||||
grafana_public_url: "https://grafana.aimloperations.com"
|
||||
observability_stack: true
|
||||
```
|
||||
|
||||
### 4.3 Host flag (`inventory/host_vars/ai-server-4080.yml`)
|
||||
`playbooks/site.yml` order: base → observability (4080) → alloy (all).
|
||||
|
||||
```yaml
|
||||
observability_stack: true # run Loki + Prometheus + Grafana on this host
|
||||
```
|
||||
### 4.3 Secrets
|
||||
|
||||
Other hosts omit the flag (or set `false`).
|
||||
- Default Grafana password is `CHANGE_ME_ON_FIRST_LOGIN` (role default). Change
|
||||
on first login, or set `observability_grafana_admin_password` in host_vars /
|
||||
vault (never commit real passwords).
|
||||
- SMTP2GO: see [GRAFANA_USAGE.md](GRAFANA_USAGE.md).
|
||||
|
||||
### 4.4 Playbook wiring (`playbooks/site.yml`)
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: Provision webservers
|
||||
hosts: webservers
|
||||
become: true
|
||||
roles:
|
||||
- common
|
||||
- ufw
|
||||
- docker
|
||||
- nodejs
|
||||
- gitea-key
|
||||
- tianji
|
||||
- alloy
|
||||
|
||||
- name: Provision central observability stack
|
||||
hosts: ai-server-4080
|
||||
become: true
|
||||
roles:
|
||||
- role: observability
|
||||
when: observability_stack | default(false)
|
||||
```
|
||||
|
||||
Order matters: **docker** before **alloy** / **observability**.
|
||||
|
||||
### 4.5 Alloy template essentials
|
||||
|
||||
In `config.alloy.j2`, set host from inventory — never hardcode:
|
||||
|
||||
```jinja
|
||||
labels = {
|
||||
job = "systemd",
|
||||
host = "{{ inventory_hostname }}",
|
||||
}
|
||||
```
|
||||
|
||||
Same for Docker log labels and every `prometheus.relabel` `replacement`
|
||||
for `host`. That is how `adama` / `roslin` / `ai-server-4080` stay correct
|
||||
without editing each file by hand.
|
||||
|
||||
### 4.6 UFW for Loki + Prometheus (control node only)
|
||||
|
||||
Do **not** put `3100` / `9090` in global `ufw_allowed_tcp_ports` (that opens
|
||||
on every host). Prefer a small task in `roles/observability` or a host_vars
|
||||
list:
|
||||
|
||||
```yaml
|
||||
# host_vars/ai-server-4080.yml
|
||||
ufw_extra_rules:
|
||||
- { port: 3100, proto: tcp, from_ip: "10.0.0.0/24", comment: "Loki" }
|
||||
- { port: 9090, proto: tcp, from_ip: "10.0.0.0/24", comment: "Prometheus" }
|
||||
- { port: 3000, proto: tcp, from_ip: "10.0.0.0/24", comment: "Grafana" }
|
||||
```
|
||||
|
||||
(Extend `roles/ufw` to loop `ufw_extra_rules` when you implement this.)
|
||||
|
||||
### 4.7 Secrets
|
||||
|
||||
- Grafana admin password: store under `~/Documents/secrets/observability/`
|
||||
(same pattern as app env files), push or template at provision time. **Never
|
||||
commit** passwords or SMTP2GO keys to git.
|
||||
- SMTP2GO credentials: configure in Grafana UI or via env file on the host —
|
||||
see [GRAFANA_USAGE.md](GRAFANA_USAGE.md).
|
||||
|
||||
### 4.8 Apply
|
||||
### 4.4 Apply
|
||||
|
||||
```bash
|
||||
# After roles exist:
|
||||
./scripts/provision.sh ai-server-4080 --check
|
||||
./scripts/provision.sh ai-server-4080
|
||||
./scripts/provision.sh adama
|
||||
./scripts/provision.sh roslin
|
||||
# or all:
|
||||
# or all (site.yml orders stack before Alloy):
|
||||
./scripts/provision.sh
|
||||
```
|
||||
|
||||
@@ -1002,8 +932,10 @@ du -sh /opt/apps/observability/loki/data \
|
||||
|
||||
### Follow-ups
|
||||
|
||||
- [ ] Encode `roles/alloy` + `roles/observability` and add to `site.yml`
|
||||
- [x] Encode `roles/alloy` + `roles/observability` and add to `site.yml`
|
||||
- [ ] Optional: drop or keep Tianji side-by-side (they do not conflict)
|
||||
- [ ] NPM proxy + change Grafana admin password
|
||||
- [ ] SMTP2GO + dashboards — [GRAFANA_USAGE.md](GRAFANA_USAGE.md)
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user