Closes#35.
## Summary
- `#33` used Jinja `rejectattr('enabled', 'equalto', false)` to skip stopped apps. That filter **requires** the key, so every `host_apps` row that omits `enabled` (the documented default) crashed deploy on all webservers before git/compose.
- Filter with `item.enabled | default(true)` instead, in `app-deploy` and `web-static`. `print_forge` (`enabled: false`) still skipped. `url_shortening_service` and every other omitted-key row deploy again.
## Test plan
- [ ] `ansible-playbook playbooks/deploy-apps.yml --syntax-check`
- [ ] Redeploy `url_shortening_service` beta: `deploy.sh --app url_shortening_service --env beta --ref <sha>`
- [ ] Confirm `print_forge` is still not started (no compose up on 8007/8019)
Reviewed-on: #36
## Summary
- Closes [#27](#27).
- Register `print_forge` in `app_catalog` (`type: django`, repo `ai_ml_operations/print_forge`, default branch `master`).
- Add `host_apps` prod `:8007` / beta `:8019` on **adama**, **roslin**, **starbuck**, **apollo**, and **ai-server-4080**.
- dj-queue **worker singleton on adama** only (`compose_profiles: [worker]`); other hosts web-only.
- Document ports, NPM hosts, Postgres DBs, secrets, Nominatim (`10.0.0.128:8089`), and the app-repo companion ([print_forge#1](ai_ml_operations/print_forge#1)).
- Update `scripts/deploy.sh` `--app` help.
- Move unused `abc_be` prod port reserve **8007 → 8009** so it does not collide with `print_forge`.
## Test plan
- [ ] Confirm `print_forge` appears in `app_catalog` and `--app print_forge` is listed in `deploy.sh --help`
- [ ] Confirm beta `host_apps` on all five hosts (port **8019**); adama has `compose_profiles: [worker]`
- [ ] Confirm prod port **8007** reserved on those hosts (no NPM until launch)
- [ ] After merge (ops, not this PR): create Postgres DBs `print_forge_beta` + `print_forge` on `10.0.0.230`, grant `westfarn`
- [ ] After merge (ops): write `~/Documents/secrets/print_forge/print_forge_{beta,prod}.env` from app `.env.prod.example`; add `print_forge:<token>` to `SHORTENER_API_TOKENS` (beta first)
- [ ] After merge (ops): NPM `print-forge-preview.aimloperations.com` → `:8019` active/active; do **not** NPM-route `printforgeprints.com` until launch
- [ ] After secrets + DBs: `~/Documents/repos/server-infra/scripts/deploy.sh --app print_forge --env beta --ref master` and hit `/healthz/`
Reviewed-on: #28
## Summary
- Closes [#20](#20).
- Register **starbuck** (`10.0.0.44`) and **apollo** (`10.0.0.7`) as Proxmox app hosts in `webservers`, with `host_apps` copied from **roslin** (same apps/ports; **no** `monica_site` worker — that stays a singleton on **adama**).
- Allow `starbuck` / `apollo` in `provision.sh` and `deploy.sh`; document inventory, NPM backends, and Alloy `host` labels.
## Test plan
- [ ] `ansible-inventory --list` shows both hosts under `webservers` at the expected IPs.
- [ ] Bootstrap SSH + passwordless sudo on each VM (keep a Proxmox console open for first UFW enable).
- [ ] `ansible starbuck,apollo -m ping`
- [ ] `./scripts/provision.sh starbuck --check` then `./scripts/provision.sh starbuck`
- [ ] `./scripts/provision.sh apollo --check` then `./scripts/provision.sh apollo`
- [ ] `./scripts/deploy.sh starbuck` and `./scripts/deploy.sh apollo` bring up the same app+env set as roslin.
- [ ] `monica_site` dj-queue worker still runs **only** on adama.
- [ ] Grafana/Loki show `host="starbuck"` and `host="apollo"`.
- [ ] NPM Advanced upstreams (manual): add `starbuck:PORT` and `apollo:PORT` beside adama/roslin.Reviewed-on: #21
## Summary
The django deploy sets `COMPOSE_PROFILES` on the **start** step but not on the **build** step, so `docker compose build` skips profile-gated services. `up -d` then reuses whatever image already exists and the container silently keeps running old code.
This adds `COMPOSE_PROFILES` to the build step so it matches the start step. Only affects hosts that set `host_apps.compose_profiles` (today: the `monica_site` dj-queue worker on adama).
## Symptom this fixes
On adama the beta worker image was 20 hours stale while web was current:
| Image | Built | Postgres driver |
|---|---|---|
| `monica_site_beta-web` | today | psycopg 3.3.4 |
| `monica_site_beta-worker` | Aug 8 | psycopg2 2.9.12 |
So the worker crash-looped on LISTEN/NOTIFY (`TypeError: 'list' object is not callable` in `dj_queue/runtime/notify.py`) long after `monica_site` had moved to psycopg3, because its image was never rebuilt.
Branch is merged up with `master`, which already carries the worker auto-start from [#18](#18); the diff here is just the build step.
## Test plan
- [x] Manual `COMPOSE_PROFILES=worker docker compose build worker` on adama produced an image with psycopg 3.3.4 and the notify errors stopped.
- [ ] Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild.
- [ ] Hosts without `compose_profiles` (roslin, ai-server-4080) still build/start web only.Reviewed-on: #19
## Summary
- Change `dta_webapp` `webroot_pattern` from `/var/www/{env}.app.ditchtheagent/html` → `/var/www/{env}.realpath.app/html`
- Update `IMPLEMENTATION.md` references to match
- **web-static:** before `compose up`, `docker rm -f` any container publishing the static host ports (`docker ps --filter publish=<port>`), then force-recreate when cleanup ran — clears orphan/legacy binders while keeping the same ports
Closes#12
Coordinates with [dta_webapp#48](Ditch_The_Agent/dta_webapp#48) / [PR #49](Ditch_The_Agent/dta_webapp#49) (`build:beta`/`build:prod` write to the same dirs; prod uses `prod.realpath.app`).
## Test plan
- [ ] Confirm `app_catalog.dta_webapp.webroot_pattern` expands to beta/prod realpath paths
- [ ] On a host with a leftover container on 8080/8081, run deploy and confirm it is removed and web-static rebinds
- [ ] After merge, redeploy `dta_webapp` beta (and prod when ready) and verify files under `/var/www/{env}.realpath.app/html`
- [ ] Hit web-static ports and confirm new build content servedReviewed-on: #13
## Summary
- Closes [#10](#10).
- Deploys **SearxNG** on **ai-server-4080** (`10.0.0.128`) for [chat_backend#62](ai_ml_operations/chat_backend#62) / [PR #65](ai_ml_operations/chat_backend#65) grounded search.
- New `roles/searxng/` (compose + JSON-enabled `settings.yml`), gated by `searxng_stack: true`, wired into `site.yml`.
- **Host port 8088** (not 8080 — that is `dta_webapp` on this host). UFW allows `10.0.0.0/24` → `8088/tcp` only.
## Ops after merge
```bash
./scripts/provision.sh ai-server-4080
# or targeted:
ansible-playbook playbooks/site.yml --limit ai-server-4080 --tags never # full site play includes searxng when searxng_stack
```
Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:
```text
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```
Smoke test from any app host:
```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```
## Test plan
- [ ] Provision ai-server-4080; confirm `docker ps` shows `searxng`
- [ ] Confirm `:8088` responds with JSON; `:8080` still serves dta_webapp
- [ ] Confirm UFW rule is LAN-only
- [ ] From adama/roslin container network, curl SearxNG succeeds
- [ ] Update chat_backend secrets to `:8088` and redeploy betaReviewed-on: #11
## Summary
- Fail when `--app` / `--env` match no `host_apps` row (was silent no-op)
- Fail when `--app` missing from `app_catalog`
Prevents "deploy succeeded" that only refreshed web-static nginx while skipping Django — what happened on [chat_backend run 205](https://git.aimloperations.com/ai_ml_operations/chat_backend/actions/runs/205) before `chat_backend` was registered on master.
## Test plan
- [ ] `./scripts/deploy.sh --app does_not_exist --env prod` should fail with catalog message
- [ ] `./scripts/deploy.sh --app chat_backend --env prod` should proceed with Django once catalog/host_apps presentReviewed-on: #6
Add app_catalog entry, host_apps on adama/roslin/ai-server-4080 (prod :8082),
and document ports/NPM backends for the CRA static frontend.Reviewed-on: #4
## Summary
- Register `scha` in `app_catalog` and `host_apps` on adama, roslin, and ai-server-4080 (prod port **8002**; beta **8012** reserved).
- Update `IMPLEMENTATION.md` ports, apps, Postgres, and NPM tables so every app lists beta + prod.
- Include `scha` in `deploy.sh` help.
Part A (dockerize / CI in the `scha` repo) is tracked separately in [scha#19](ai_ml_operations/scha#19).
## Test plan
- [ ] Confirm inventory YAML loads: `ansible-inventory --list` shows `scha` under each host
- [ ] After Part A merges + `~/Documents/secrets/scha/scha_prod.env` exists: dry-run `./scripts/deploy.sh --app scha --env prod --check`
- [ ] Create DB `scha` on shared Postgres (`10.0.0.230`) and point NPM at `adama:8002` + `roslin:8002` (+ optional `ai-server-4080:8002`)
Reviewed-on: #3
## Summary
Implements #1: Gitea Act workflow that fast-forward pulls `~/Documents/repos/server-infra` on the self-hosted runner whenever `master` changes.
- Adds `.gitea/workflows/sync-checkout.yml`
- Triggers on direct push to `master` and on merged PRs targeting `master`
- Refuses to pull if the working tree is dirty
- Documents the workflow in `IMPLEMENTATION.md`
## Test plan
- [ ] Merge PR → workflow runs on self-hosted runner
- [ ] Runner checkout at `/home/westfarn/Documents/repos/server-infra` advances to latest `master` commit
- [ ] Dirty working tree on runner causes workflow to fail (no silent overwrite)
- [ ] Direct push to `master` also triggers sync
Closes#1
Reviewed-on: #2
ai-server-4080 runs dta_service active/active with adama and roslin.
Disable beta_dta_wsgi/prod_dta_wsgi so port 8001/8011 are free for compose.
Co-authored-by: Cursor <cursoragent@cursor.com>
westfarn
