ai_ml_operations/server-infra
2
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
21 Commits 2 Branches 0 Tags
2e3fe13b391560f7db8e910320fc2d368fda52d7
Commit Graph
10 Commits
Author SHA1 Message Date
westfarn 2e3fe13b39 Build profile-gated services so workers do not run stale images (#19)
Sync runner checkout / sync (push) Successful in 7s
Details 
## Summary

The django deploy sets `COMPOSE_PROFILES` on the **start** step but not on the **build** step, so `docker compose build` skips profile-gated services. `up -d` then reuses whatever image already exists and the container silently keeps running old code.

This adds `COMPOSE_PROFILES` to the build step so it matches the start step. Only affects hosts that set `host_apps.compose_profiles` (today: the `monica_site` dj-queue worker on adama).

## Symptom this fixes

On adama the beta worker image was 20 hours stale while web was current:

| Image | Built | Postgres driver |
|---|---|---|
| `monica_site_beta-web` | today | psycopg 3.3.4 |
| `monica_site_beta-worker` | Aug 8 | psycopg2 2.9.12 |

So the worker crash-looped on LISTEN/NOTIFY (`TypeError: 'list' object is not callable` in `dj_queue/runtime/notify.py`) long after `monica_site` had moved to psycopg3, because its image was never rebuilt.

Branch is merged up with `master`, which already carries the worker auto-start from [#18](#18); the diff here is just the build step.

## Test plan

- [x] Manual `COMPOSE_PROFILES=worker docker compose build worker` on adama produced an image with psycopg 3.3.4 and the notify errors stopped.
- [ ] Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild.
- [ ] Hosts without `compose_profiles` (roslin, ai-server-4080) still build/start web only.Reviewed-on: #19
 2026-08-09 04:36:51 -07:00
westfarn b07e8cb4c3 Auto-start monica_site dj-queue worker on adama (#18)
Sync runner checkout / sync (push) Successful in 7s
Details 
## Summary
- Closes [#17](#17).
- Django deploy passes optional `host_apps.compose_profiles` as `COMPOSE_PROFILES` so compose profiles survive `up -d --remove-orphans`.
- Adama `monica_site` beta+prod set `compose_profiles: [worker]` (dj-queue singleton). Roslin / ai-server-4080 stay web-only.

## Why
Real campaign sends enqueue dj-queue tasks; without the worker they stay **Queued**. Test email worked because it is synchronous SMTP.

## Test plan
- [ ] Merge + deploy `monica_site` beta (and/or prod) via `deploy.sh`
- [ ] On **adama**, confirm worker container up for `monica_site_beta` / `monica_site_prod`
- [ ] On roslin / ai-server-4080, confirm **no** worker
- [ ] Portal: send campaign → recipient leaves Queued → SentReviewed-on: #18
 2026-08-08 12:01:18 -07:00
westfarn 829a63cbf2 Point dta_webapp webroot at realpath.app paths (closes #12) (#13)
Sync runner checkout / sync (push) Successful in 7s
Details 
## Summary
- Change `dta_webapp` `webroot_pattern` from `/var/www/{env}.app.ditchtheagent/html` → `/var/www/{env}.realpath.app/html`
- Update `IMPLEMENTATION.md` references to match
- **web-static:** before `compose up`, `docker rm -f` any container publishing the static host ports (`docker ps --filter publish=<port>`), then force-recreate when cleanup ran — clears orphan/legacy binders while keeping the same ports

Closes #12

Coordinates with [dta_webapp#48](Ditch_The_Agent/dta_webapp#48) / [PR #49](Ditch_The_Agent/dta_webapp#49) (`build:beta`/`build:prod` write to the same dirs; prod uses `prod.realpath.app`).

## Test plan
- [ ] Confirm `app_catalog.dta_webapp.webroot_pattern` expands to beta/prod realpath paths
- [ ] On a host with a leftover container on 8080/8081, run deploy and confirm it is removed and web-static rebinds
- [ ] After merge, redeploy `dta_webapp` beta (and prod when ready) and verify files under `/var/www/{env}.realpath.app/html`
- [ ] Hit web-static ports and confirm new build content servedReviewed-on: #13
 2026-08-06 08:46:33 -07:00
westfarn 7e48c22e3e Deploy SearxNG on ai-server-4080 for chat_backend grounded search (#10) (#11)
Sync runner checkout / sync (push) Successful in 7s
Details 
## Summary
- Closes [#10](#10).
- Deploys **SearxNG** on **ai-server-4080** (`10.0.0.128`) for [chat_backend#62](ai_ml_operations/chat_backend#62) / [PR #65](ai_ml_operations/chat_backend#65) grounded search.
- New `roles/searxng/` (compose + JSON-enabled `settings.yml`), gated by `searxng_stack: true`, wired into `site.yml`.
- **Host port 8088** (not 8080 — that is `dta_webapp` on this host). UFW allows `10.0.0.0/24` → `8088/tcp` only.

## Ops after merge

```bash
./scripts/provision.sh ai-server-4080
# or targeted:
ansible-playbook playbooks/site.yml --limit ai-server-4080 --tags never  # full site play includes searxng when searxng_stack
```

Then set in `chat_backend_prod.env` / `chat_backend_beta.env`:

```text
SEARCH_PROVIDER=searxng
SEARCH_FAILOVER_PROVIDER=ddgs
SEARXNG_BASE_URL=http://10.0.0.128:8088
```

Smoke test from any app host:

```bash
curl -sG 'http://10.0.0.128:8088/search' --data-urlencode 'q=test' -d 'format=json' | head
```

## Test plan
- [ ] Provision ai-server-4080; confirm `docker ps` shows `searxng`
- [ ] Confirm `:8088` responds with JSON; `:8080` still serves dta_webapp
- [ ] Confirm UFW rule is LAN-only
- [ ] From adama/roslin container network, curl SearxNG succeeds
- [ ] Update chat_backend secrets to `:8088` and redeploy betaReviewed-on: #11
 2026-08-02 11:33:38 -07:00
westfarn d49c6bfe2d Fail loud when deploy --app matches nothing (#6)
Sync runner checkout / sync (push) Successful in 6s
Details 
## Summary

- Fail when `--app` / `--env` match no `host_apps` row (was silent no-op)
- Fail when `--app` missing from `app_catalog`

Prevents "deploy succeeded" that only refreshed web-static nginx while skipping Django — what happened on [chat_backend run 205](https://git.aimloperations.com/ai_ml_operations/chat_backend/actions/runs/205) before `chat_backend` was registered on master.

## Test plan

- [ ] `./scripts/deploy.sh --app does_not_exist --env prod` should fail with catalog message
- [ ] `./scripts/deploy.sh --app chat_backend --env prod` should proceed with Django once catalog/host_apps presentReviewed-on: #6
 2026-07-27 08:27:29 -07:00
westfarn 974e848d91 updating to remove company_site beta 2026-07-10 12:31:32 -05:00
westfarn f37ec41530 Updates for grafana stack 2026-07-10 06:50:52 -05:00
westfarnandCursor b1b9b3bafe Stop legacy dta_service systemd units before Docker deploy (#52). 
ai-server-4080 runs dta_service active/active with adama and roslin.
Disable beta_dta_wsgi/prod_dta_wsgi so port 8001/8011 are free for compose.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-09 11:45:03 -05:00
westfarn 589462e6d0 updates 2026-07-08 05:59:11 -05:00
westfarn f848420d8f Initial server-infra setup 2026-07-06 15:34:49 -05:00