ai_ml_operations/server-infra
2
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
westfarn
2026-07-08 05:59:11 -05:00
parent f848420d8f
commit 589462e6d0
27 changed files with 771 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
IMPLEMENTATION.md
+125 -56
View File
@@ -7,10 +7,9 @@ Ansible-based provisioning and deployment for homelab web servers.
```mermaid
flowchart TB
subgraph provision ["Provisioning (manual / rare)"]
Desktop1["Ubuntu Desktop\n(control node)"]
Desktop1 -->|ansible-playbook site.yml| Adama
Desktop1 -->|ansible-playbook site.yml| Roslin
Desktop1 -->|ansible-playbook site.yml| DesktopTarget
Control1["ai-server-4080\n(control node)"]
Control1 -->|ansible-playbook site.yml| Adama
Control1 -->|ansible-playbook site.yml| Roslin
end
subgraph cicd ["CI/CD (every merge to master)"]
@@ -20,14 +19,13 @@ flowchart TB
Deploy --> AnsibleDeploy["ansible-playbook deploy-apps.yml"]
AnsibleDeploy --> Adama2["adama"]
AnsibleDeploy --> Roslin2["roslin"]
AnsibleDeploy --> Desktop2["desktop"]
end
```
| Pipeline | When | Playbook | Where it runs |
|----------|------|----------|---------------|
| **Provision** | New VM, OS change, firewall, Docker install | `site.yml` | Desktop — run manually |
| **Deploy** | Green unit tests on `master` | `deploy-apps.yml` | Gitea Act runner on desktop |
| **Provision** | New VM, OS change, firewall, Docker install | `site.yml` | ai-server-4080 — run manually |
| **Deploy** | Green unit tests on `master` | `deploy-apps.yml` | Gitea Act runner on ai-server-4080 |
Both pipelines share the same inventory (`inventory/hosts.yml`).
@@ -35,9 +33,9 @@ Both pipelines share the same inventory (`inventory/hosts.yml`).
| Name | IP | Role |
|------|-----|------|
| adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) |
| roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) |
| desktop | *see `host_vars/desktop.yml`* | Ubuntu Desktop — control node + deployment target |
| adama | 10.0.0.77 | Ubuntu Server VM (Proxmox) — app host |
| roslin | 10.0.0.176 | Ubuntu Server VM (Proxmox) — app host |
| ai-server-4080 | 10.0.0.128 | Control node + Gitea act runner (no app workloads) |
Hostname on this machine: `ryan-development-1`
@@ -52,18 +50,23 @@ server-infra/
├── inventory/
│ ├── hosts.yml
│ ├── group_vars/
│ │ ├── all.yml
│ │ └── webservers.yml
│ │ └── all.yml # vars + app_catalog
│ └── host_vars/
│ └── desktop.yml
│ ├── adama.yml # host_apps (django + dta_webapp)
│ ├── roslin.yml # host_apps (mirrors adama)
│ └── ai-server-4080.yml # control node / act runner, no workloads
├── playbooks/
│ ├── site.yml # Phase 1: provision
│ └── deploy-apps.yml # Phase 2: CI deploy (stub)
│ └── deploy-apps.yml # Phase 2: CI deploy
├── roles/
│ ├── common/ # Base packages
│ ├── ufw/ # Firewall
│ ├── docker/ # Docker CE + compose plugin
│ └── app-deploy/ # App deploy (stub for Phase 2)
│ ├── nodejs/ # Node.js + npm + npx (NodeSource)
│ ├── gitea-key/ # per-server SSH key + Gitea access probe
│ ├── tianji/ # Monitoring reporter
│ ├── app-deploy/ # django (docker) + node-static deploy
│ └── web-static/ # nginx container serving /var/www builds
└── scripts/
├── provision.sh # Wrapper with --limit support
└── deploy.sh # Wrapper for deploy playbook
@@ -74,7 +77,7 @@ server-infra/
Ansible needs SSH + sudo on each target before playbooks work.
1. Create `westfarn` on each VM with sudo membership.
2. Copy your SSH public key from the desktop:
2. Copy your SSH public key from the control node (ai-server-4080):
```bash
ssh-copy-id westfarn@10.0.0.77
ssh-copy-id westfarn@10.0.0.176
@@ -84,7 +87,7 @@ Ansible needs SSH + sudo on each target before playbooks work.
ssh westfarn@10.0.0.77
ssh westfarn@10.0.0.176
```
4. On the desktop (control node), install Ansible:
4. On ai-server-4080 (control node), install Ansible:
```bash
sudo apt update && sudo apt install -y ansible
# or: pip install ansible
@@ -94,7 +97,7 @@ Ansible needs SSH + sudo on each target before playbooks work.
cd ~/Documents/repos/server-infra
ansible-galaxy collection install -r requirements.yml
```
6. Update `inventory/host_vars/desktop.yml` with this machine's LAN IP.
6. Update `inventory/host_vars/ai-server-4080.yml` with this machine's LAN IP (`ansible_host`).
## Testing on a Single Server
@@ -118,7 +121,7 @@ ansible adama -m ping
# Same for other hosts
./scripts/provision.sh roslin
./scripts/provision.sh desktop
./scripts/provision.sh ai-server-4080
```
### Provision all hosts
@@ -145,6 +148,9 @@ Applies roles in order to the `webservers` group:
| `common` | apt update, git, python3, pip, curl, ca-certificates |
| `ufw` | Firewall: SSH from LAN only, HTTP/HTTPS public |
| `docker` | Docker CE, compose plugin, add `westfarn` to `docker` group |
| `nodejs` | Node.js + npm + npx (NodeSource) for `dta_webapp` builds |
| `gitea-key` | Per-server SSH key + Gitea access probe |
| `tianji` | Monitoring reporter |
### UFW rules
@@ -161,56 +167,121 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
## Phase 2: CI Deploy (`deploy-apps.yml`)
Not fully implemented yet. Planned flow:
### Apps
1. Gitea push triggers unit tests.
2. On success, Act runner on desktop runs `deploy-apps.yml`.
3. Ansible fans out to all `webservers` hosts.
| App | Type | Hosts | Notes |
|-----|------|-------|-------|
| `company_site` | django (docker) | adama + roslin | active/active behind NPM |
| `dta_service` | django (docker) | adama + roslin | active/active behind NPM |
| `dta_webapp` | node/vite static | adama + roslin | active/active; built to `/var/www/<env>_dta_webapp`, served by web-static nginx |
### Planned `company_site` workflow change
Both environments (`beta`, `prod`) are deployed. Django apps use a **shared external
Postgres** (via `DATABASE_URL` in each host's env file) so active/active replicas
share one database.
```yaml
# company_site/.gitea/workflows/deploy.yml (future)
jobs:
deploy:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
with:
ref: ${{ gitea.event.workflow_run.head_sha }}
### Data model
- name: Deploy to all webservers
run: |
~/Documents/repos/server-infra/scripts/deploy.sh \
--extra-vars "app_ref=${{ gitea.event.workflow_run.head_sha }}"
```
- `app_catalog` (`group_vars/all.yml`) — how each app is built (repo, type, compose file, migrate cmd).
- `host_apps` (`host_vars/<host>.yml`) — which app+env+port runs on that host.
- Django app = one compose project per env: project name `<app>_<env>`, host port from `host_apps`.
Ports match across adama/roslin so NPM can balance `adama:PORT` + `roslin:PORT`.
### Planned `app-deploy` role (post-dockerize)
### Ports
Per host:
| App | beta | prod |
|-----|------|------|
| company_site | 8010 | 8000 |
| dta_service | 8011 | 8001 |
| dta_webapp (nginx) | 8081 | 8080 |
1. Clone or pull app repo at pinned SHA.
2. `docker compose pull && docker compose up -d`.
3. Optional health check.
### Flow
Pre-dockerize interim: role can rsync/systemd like current `company_site/scripts/deploy.sh`.
1. Gitea push to `master` → repo's `.gitea/workflows` runs tests.
2. On green, deploy job on the self-hosted runner calls:
```bash
~/Documents/repos/server-infra/scripts/deploy.sh \
--app company_site --env prod --ref "${{ gitea.sha }}"
```
3. `deploy-apps.yml` runs against `webservers`; each host deploys only the
matching app+env from its `host_apps`.
### `app-deploy` role behavior
- **django**: push per-app secret from control node `{{ secrets_dir }}/<app>/<app>_<env>.env`
to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
`docker compose build` → `up -d` → migrate (run once, shared DB).
- **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
(writes to `/var/www/<env>_dta_webapp`).
- **web-static** role: one nginx container per app host (adama + roslin) serving
the static roots on their ports; NPM balances across both.
### Reverse proxy / load balancing (NPM at 10.0.0.230)
Ansible does **not** manage NPM. It only guarantees stable host ports. In NPM you
point each domain at the backend(s):
- Single host: standard Proxy Host → `adama:PORT`.
- Active/active: jc21 NPM's UI Proxy Host is single-target. To balance
adama+roslin you need the **Advanced** tab with a custom `upstream {}` block
(or a real LB). Confirm this before relying on active/active.
### Required changes IN each app repo (owned separately)
- [ ] `docker-compose.prod.yml`: drop the bundled `db` service; `web` reads
`DATABASE_URL` / `DB_HOST` pointing at the shared external Postgres.
- [ ] Each app has its own database + user on the shared Postgres.
- [ ] `.gitea/workflows/deploy.yml`: replace the local `scripts/deploy.sh` step
with a call to `server-infra/scripts/deploy.sh --app <name> --env <env> --ref <sha>`
(keep the test/docker jobs).
- [ ] `dta_webapp`: `npm run build:beta` / `build:prod` output to
`/var/www/beta_dta_webapp` / `/var/www/prod_dta_webapp`.
### Shared Postgres (10.0.0.230, same box as NPM)
One shared instance; each app+env gets its own database (beta and prod MUST NOT
share a DB — active/active replicas of the same env share one DB, different envs
do not).
| app | env | database | DATABASE_URL |
|-----|-----|----------|--------------|
| company_site | prod | `company_site` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site` |
| company_site | beta | `company_site_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/company_site_beta` |
| dta_service | prod | `dta_service` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service` |
| dta_service | beta | `dta_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/dta_service_beta` |
Server prereqs on 10.0.0.230: create the 4 DBs + grant `westfarn`;
`listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
5432 to `10.0.0.0/24` only.
### One-time host bootstrap (per target)
- [x] Gitea SSH key: the `gitea-key` role (in `site.yml`) generates a key per
server, configures SSH for port 30009, probes access, and — if the server
can't reach Gitea yet — prints the public key to add and stops. Add the key
(Gitea user SSH keys, or repo Deploy Keys) and re-run provisioning.
- [ ] Create control-node secrets `{{ secrets_dir }}/<app>/<app>_<env>.env`
(default `~/Documents/secrets/<app>/<app>_<env>.env`) with `DATABASE_URL`
(see table), `DJANGO_ENV`, `DJANGO_SECRET_KEY`, `WEB_PORT` (matching the port
table). Deploy pushes these to `/opt/apps/env/<app>_<env>.env` (mode 600) on
adama + roslin. Never committed to git.
- [x] Node.js/npm/npx for the `dta_webapp` build — installed by the `nodejs`
role in `site.yml` (NodeSource, `node_major` default 20).
## Gitea Act Runner
**Recommended:** Single self-hosted runner on the desktop.
**Recommended:** Single self-hosted runner on ai-server-4080.
- One build artifact, one orchestration point.
- VMs only run containers; no runner needed on them for deploy fan-out.
- One orchestration point.
- App hosts (adama/roslin) run the workloads; no runner needed on them for deploy fan-out.
- Runner needs: Ansible, this repo checked out, SSH key to all hosts, vault password (later).
### Runner requirements on desktop
### Runner requirements on ai-server-4080
| Requirement | Why |
|-------------|-----|
| Ansible | Run `deploy-apps.yml` |
| `server-infra` checkout | Playbooks + inventory |
| SSH key to all hosts | Including loopback to desktop |
| Docker | Build images before push to hosts (Phase 2) |
| SSH key to adama + roslin | Deploy fan-out |
## SSH Keys for CI Deploy
@@ -237,7 +308,7 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
| # | Task | Status |
|---|------|--------|
| 1 | Create `server-infra` repo | Done |
| 2 | Inventory with all 3 hosts | Done — update desktop IP |
| 2 | Inventory with all 3 hosts | Done |
| 3 | Bootstrap SSH to adama + roslin | Manual |
| 4 | `site.yml` → common, ufw, docker | Done |
| 5 | Verify `ansible webservers -m ping` | Manual |
@@ -250,11 +321,9 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
## Open Decisions
1. **Desktop LAN IP** — set in `inventory/host_vars/desktop.yml`.
2. **Same app on all three?** — prod mirror vs adama=prod / roslin=staging / desktop=dev.
3. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
4. **Gitea URL** — for clone URLs in `app-deploy` role.
5. **Reverse proxy** — Caddy/nginx on host before containers? Affects Phase 2.
1. **Deploy user** — `westfarn` vs dedicated `deploy` for CI.
2. **NPM load balancing** — confirm jc21 NPM can express adama+roslin upstreams (Advanced tab), else active/active is just two independent instances.
3. **Secrets** — Ansible Vault vs per-host env files (currently per-host `/opt/apps/env/*.env`).
## Adding a New VM