Catalog + host_apps on all webservers (prod 8086 / beta 8087) with webroots
under /var/www/{env}.blog.realpath.app/html. Nginx serves the SSG 404 page
instead of SPA index fallback.
## Summary
- Closes [#27](#27).
- Register `print_forge` in `app_catalog` (`type: django`, repo `ai_ml_operations/print_forge`, default branch `master`).
- Add `host_apps` prod `:8007` / beta `:8019` on **adama**, **roslin**, **starbuck**, **apollo**, and **ai-server-4080**.
- dj-queue **worker singleton on adama** only (`compose_profiles: [worker]`); other hosts web-only.
- Document ports, NPM hosts, Postgres DBs, secrets, Nominatim (`10.0.0.128:8089`), and the app-repo companion ([print_forge#1](ai_ml_operations/print_forge#1)).
- Update `scripts/deploy.sh` `--app` help.
- Move unused `abc_be` prod port reserve **8007 → 8009** so it does not collide with `print_forge`.
## Test plan
- [ ] Confirm `print_forge` appears in `app_catalog` and `--app print_forge` is listed in `deploy.sh --help`
- [ ] Confirm beta `host_apps` on all five hosts (port **8019**); adama has `compose_profiles: [worker]`
- [ ] Confirm prod port **8007** reserved on those hosts (no NPM until launch)
- [ ] After merge (ops, not this PR): create Postgres DBs `print_forge_beta` + `print_forge` on `10.0.0.230`, grant `westfarn`
- [ ] After merge (ops): write `~/Documents/secrets/print_forge/print_forge_{beta,prod}.env` from app `.env.prod.example`; add `print_forge:<token>` to `SHORTENER_API_TOKENS` (beta first)
- [ ] After merge (ops): NPM `print-forge-preview.aimloperations.com` → `:8019` active/active; do **not** NPM-route `printforgeprints.com` until launch
- [ ] After secrets + DBs: `~/Documents/repos/server-infra/scripts/deploy.sh --app print_forge --env beta --ref master` and hit `/healthz/`
Reviewed-on: #28
## Summary
The django deploy sets `COMPOSE_PROFILES` on the **start** step but not on the **build** step, so `docker compose build` skips profile-gated services. `up -d` then reuses whatever image already exists and the container silently keeps running old code.
This adds `COMPOSE_PROFILES` to the build step so it matches the start step. Only affects hosts that set `host_apps.compose_profiles` (today: the `monica_site` dj-queue worker on adama).
## Symptom this fixes
On adama the beta worker image was 20 hours stale while web was current:
| Image | Built | Postgres driver |
|---|---|---|
| `monica_site_beta-web` | today | psycopg 3.3.4 |
| `monica_site_beta-worker` | Aug 8 | psycopg2 2.9.12 |
So the worker crash-looped on LISTEN/NOTIFY (`TypeError: 'list' object is not callable` in `dj_queue/runtime/notify.py`) long after `monica_site` had moved to psycopg3, because its image was never rebuilt.
Branch is merged up with `master`, which already carries the worker auto-start from [#18](#18); the diff here is just the build step.
## Test plan
- [x] Manual `COMPOSE_PROFILES=worker docker compose build worker` on adama produced an image with psycopg 3.3.4 and the notify errors stopped.
- [ ] Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild.
- [ ] Hosts without `compose_profiles` (roslin, ai-server-4080) still build/start web only.Reviewed-on: #19
## Summary
- Change `dta_webapp` `webroot_pattern` from `/var/www/{env}.app.ditchtheagent/html` → `/var/www/{env}.realpath.app/html`
- Update `IMPLEMENTATION.md` references to match
- **web-static:** before `compose up`, `docker rm -f` any container publishing the static host ports (`docker ps --filter publish=<port>`), then force-recreate when cleanup ran — clears orphan/legacy binders while keeping the same ports
Closes#12
Coordinates with [dta_webapp#48](Ditch_The_Agent/dta_webapp#48) / [PR #49](Ditch_The_Agent/dta_webapp#49) (`build:beta`/`build:prod` write to the same dirs; prod uses `prod.realpath.app`).
## Test plan
- [ ] Confirm `app_catalog.dta_webapp.webroot_pattern` expands to beta/prod realpath paths
- [ ] On a host with a leftover container on 8080/8081, run deploy and confirm it is removed and web-static rebinds
- [ ] After merge, redeploy `dta_webapp` beta (and prod when ready) and verify files under `/var/www/{env}.realpath.app/html`
- [ ] Hit web-static ports and confirm new build content servedReviewed-on: #13
## Summary
- Closes [#10](#10).
- Deploys **SearxNG** on **ai-server-4080** (`10.0.0.128`) for [chat_backend#62](ai_ml_operations/chat_backend#62) / [PR #65](ai_ml_operations/chat_backend#65) grounded search.
- New `roles/searxng/` (compose + JSON-enabled `settings.yml`), gated by `searxng_stack: true`, wired into `site.yml`.
- **Host port 8088** (not 8080 — that is `dta_webapp` on this host). UFW allows `10.0.0.0/24` → `8088/tcp` only.
## Ops after merge
```bash
./scripts/provision.sh ai-server-4080
# or targeted:
ansible-playbook playbooks/site.yml --limit ai-server-4080 --tags never # full site play includes searxng when searxng_stack
```
Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:
```text
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```
Smoke test from any app host:
```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```
## Test plan
- [ ] Provision ai-server-4080; confirm `docker ps` shows `searxng`
- [ ] Confirm `:8088` responds with JSON; `:8080` still serves dta_webapp
- [ ] Confirm UFW rule is LAN-only
- [ ] From adama/roslin container network, curl SearxNG succeeds
- [ ] Update chat_backend secrets to `:8088` and redeploy betaReviewed-on: #11
## Summary
- Fail when `--app` / `--env` match no `host_apps` row (was silent no-op)
- Fail when `--app` missing from `app_catalog`
Prevents "deploy succeeded" that only refreshed web-static nginx while skipping Django — what happened on [chat_backend run 205](https://git.aimloperations.com/ai_ml_operations/chat_backend/actions/runs/205) before `chat_backend` was registered on master.
## Test plan
- [ ] `./scripts/deploy.sh --app does_not_exist --env prod` should fail with catalog message
- [ ] `./scripts/deploy.sh --app chat_backend --env prod` should proceed with Django once catalog/host_apps presentReviewed-on: #6
ai-server-4080 runs dta_service active/active with adama and roslin.
Disable beta_dta_wsgi/prod_dta_wsgi so port 8001/8011 are free for compose.
Co-authored-by: Cursor <cursoragent@cursor.com>
westfarn
