Commit Graph
9 Commits
Author SHA1 Message Date
westfarn 726ad971cb Add stop.sh (compose down) and disable print_forge (#34)
Sync runner checkout / sync (push) Successful in 6s
## Summary
- Closes [#33](#33).
- Add `scripts/stop.sh` → `playbooks/stop-apps.yml` → `docker compose down` for one `--app` + `--env` (no `-v`; checkouts/secrets stay).
- `host_apps.enabled: false` skips deploy (including CI `--app`) so a stopped app cannot come back. Rows stay for ports and `stop.sh`.
- Disable `print_forge` beta + prod on all app hosts. After merge (or from this branch): `./scripts/stop.sh --app print_forge --env prod` and `--env beta`.

## Test plan
- [ ] `./scripts/stop.sh --help` shows required `--app` / `--env`
- [ ] `ansible-playbook playbooks/stop-apps.yml --syntax-check -e app=print_forge -e app_env=prod`
- [ ] `./scripts/stop.sh --app print_forge --env prod` compose-downs `print_forge_prod` on all webservers
- [ ] `./scripts/stop.sh --app print_forge --env beta` compose-downs `print_forge_beta` (including adama worker)
- [ ] `./scripts/deploy.sh --app print_forge --env prod --check` skips disabled rows (does not start containers)
- [ ] NPM: `print-forge-preview.aimloperations.com` will 502 until the proxy host is disabled (out of scope)

Reviewed-on: #34
2026-09-16 03:11:23 -07:00
westfarn 2abcdd7c58 Register print_forge for django deploy (closes #27) (#28)
Sync runner checkout / sync (push) Successful in 6s
## Summary
- Closes [#27](#27).
- Register `print_forge` in `app_catalog` (`type: django`, repo `ai_ml_operations/print_forge`, default branch `master`).
- Add `host_apps` prod `:8007` / beta `:8019` on **adama**, **roslin**, **starbuck**, **apollo**, and **ai-server-4080**.
- dj-queue **worker singleton on adama** only (`compose_profiles: [worker]`); other hosts web-only.
- Document ports, NPM hosts, Postgres DBs, secrets, Nominatim (`10.0.0.128:8089`), and the app-repo companion ([print_forge#1](ai_ml_operations/print_forge#1)).
- Update `scripts/deploy.sh` `--app` help.
- Move unused `abc_be` prod port reserve **8007 → 8009** so it does not collide with `print_forge`.

## Test plan
- [ ] Confirm `print_forge` appears in `app_catalog` and `--app print_forge` is listed in `deploy.sh --help`
- [ ] Confirm beta `host_apps` on all five hosts (port **8019**); adama has `compose_profiles: [worker]`
- [ ] Confirm prod port **8007** reserved on those hosts (no NPM until launch)
- [ ] After merge (ops, not this PR): create Postgres DBs `print_forge_beta` + `print_forge` on `10.0.0.230`, grant `westfarn`
- [ ] After merge (ops): write `~/Documents/secrets/print_forge/print_forge_{beta,prod}.env` from app `.env.prod.example`; add `print_forge:<token>` to `SHORTENER_API_TOKENS` (beta first)
- [ ] After merge (ops): NPM `print-forge-preview.aimloperations.com` → `:8019` active/active; do **not** NPM-route `printforgeprints.com` until launch
- [ ] After secrets + DBs: `~/Documents/repos/server-infra/scripts/deploy.sh --app print_forge --env beta --ref master` and hit `/healthz/`

Reviewed-on: #28
2026-09-06 06:35:48 -07:00
westfarn 95fbae8db4 Register college_craft for django deploy (closes #24) (#25)
Sync runner checkout / sync (push) Successful in 6s
## Summary
- Closes [#24](#24).
- Register `college_craft` in `app_catalog` (`type: django`, repo `ai_ml_operations/college_craft`, default branch `master`).
- Add `host_apps` prod `:8006` / beta `:8016` on **adama**, **roslin**, **starbuck**, **apollo**, and **ai-server-4080**.
- dj-queue **worker singleton on adama** only (`compose_profiles: [worker]`); other hosts web-only.
- Document ports, NPM hosts, Postgres DBs, secrets, Nominatim (`10.0.0.128:8089`), and the app-repo `--app client_site` leftover.
- Add `college_craft` to `scripts/deploy.sh` `--app` help.

## Out of scope (ops, after merge)
- Create Postgres DBs `college_craft` / `college_craft_beta` on `10.0.0.230` and grant `westfarn`.
- Write control-node secrets `~/Documents/secrets/college_craft/college_craft_{prod,beta}.env`.
- NPM: `collegecraft.com` (+ www) → `:8006`; `college-craft-preview.aimloperations.com` → `:8016`.
- Smoke deploy after secrets + DBs exist:
  ```bash
  ~/Documents/repos/server-infra/scripts/deploy.sh --app college_craft --env beta --ref master
  ~/Documents/repos/server-infra/scripts/deploy.sh --app college_craft --env prod --ref master
  ```

## Test plan
- [ ] Confirm `app_catalog.college_craft` matches other django apps (`compose_file`, `migrate_cmd`).
- [ ] Confirm ports 8006/8016 unused elsewhere and match across all five hosts.
- [ ] Confirm only adama has `compose_profiles: [worker]` for college_craft.
- [ ] After secrets + DBs: deploy beta then prod; `/healthz/` up on both; prod holding page, beta full site.

Reviewed-on: #25
2026-09-02 07:39:03 -07:00
westfarn 2e3fe13b39 Build profile-gated services so workers do not run stale images (#19)
Sync runner checkout / sync (push) Successful in 7s
## Summary

The django deploy sets `COMPOSE_PROFILES` on the **start** step but not on the **build** step, so `docker compose build` skips profile-gated services. `up -d` then reuses whatever image already exists and the container silently keeps running old code.

This adds `COMPOSE_PROFILES` to the build step so it matches the start step. Only affects hosts that set `host_apps.compose_profiles` (today: the `monica_site` dj-queue worker on adama).

## Symptom this fixes

On adama the beta worker image was 20 hours stale while web was current:

| Image | Built | Postgres driver |
|---|---|---|
| `monica_site_beta-web` | today | psycopg 3.3.4 |
| `monica_site_beta-worker` | Aug 8 | psycopg2 2.9.12 |

So the worker crash-looped on LISTEN/NOTIFY (`TypeError: 'list' object is not callable` in `dj_queue/runtime/notify.py`) long after `monica_site` had moved to psycopg3, because its image was never rebuilt.

Branch is merged up with `master`, which already carries the worker auto-start from [#18](#18); the diff here is just the build step.

## Test plan

- [x] Manual `COMPOSE_PROFILES=worker docker compose build worker` on adama produced an image with psycopg 3.3.4 and the notify errors stopped.
- [ ] Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild.
- [ ] Hosts without `compose_profiles` (roslin, ai-server-4080) still build/start web only.Reviewed-on: #19
2026-08-09 04:36:51 -07:00
westfarn b07e8cb4c3 Auto-start monica_site dj-queue worker on adama (#18)
Sync runner checkout / sync (push) Successful in 7s
## Summary
- Closes [#17](#17).
- Django deploy passes optional `host_apps.compose_profiles` as `COMPOSE_PROFILES` so compose profiles survive `up -d --remove-orphans`.
- Adama `monica_site` beta+prod set `compose_profiles: [worker]` (dj-queue singleton). Roslin / ai-server-4080 stay web-only.

## Why
Real campaign sends enqueue dj-queue tasks; without the worker they stay **Queued**. Test email worked because it is synchronous SMTP.

## Test plan
- [ ] Merge + deploy `monica_site` beta (and/or prod) via `deploy.sh`
- [ ] On **adama**, confirm worker container up for `monica_site_beta` / `monica_site_prod`
- [ ] On roslin / ai-server-4080, confirm **no** worker
- [ ] Portal: send campaign → recipient leaves Queued → SentReviewed-on: #18
2026-08-08 12:01:18 -07:00
westfarn d49c6bfe2d Fail loud when deploy --app matches nothing (#6)
Sync runner checkout / sync (push) Successful in 6s
## Summary

- Fail when `--app` / `--env` match no `host_apps` row (was silent no-op)
- Fail when `--app` missing from `app_catalog`

Prevents "deploy succeeded" that only refreshed web-static nginx while skipping Django — what happened on [chat_backend run 205](https://git.aimloperations.com/ai_ml_operations/chat_backend/actions/runs/205) before `chat_backend` was registered on master.

## Test plan

- [ ] `./scripts/deploy.sh --app does_not_exist --env prod` should fail with catalog message
- [ ] `./scripts/deploy.sh --app chat_backend --env prod` should proceed with Django once catalog/host_apps presentReviewed-on: #6
2026-07-27 08:27:29 -07:00
westfarnandCursor b1b9b3bafe Stop legacy dta_service systemd units before Docker deploy (#52).
ai-server-4080 runs dta_service active/active with adama and roslin.
Disable beta_dta_wsgi/prod_dta_wsgi so port 8001/8011 are free for compose.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:45:03 -05:00
westfarn 589462e6d0 updates 2026-07-08 05:59:11 -05:00
westfarn f848420d8f Initial server-infra setup 2026-07-06 15:34:49 -05:00