Build profile-gated services so workers do not run stale images #19

2026-08-09T04:23:34-07:00

westfarn commented

2026-08-09 04:23:34 -07:00

Summary

The django deploy sets COMPOSE_PROFILES on the start step but not on the build step, so docker compose build skips profile-gated services. up -d then reuses whatever image already exists and the container silently keeps running old code.

This adds COMPOSE_PROFILES to the build step so it matches the start step. Only affects hosts that set host_apps.compose_profiles (today: the monica_site dj-queue worker on adama).

Symptom this fixes

On adama the beta worker image was 20 hours stale while web was current:

Image	Built	Postgres driver
`monica_site_beta-web`	today	psycopg 3.3.4
`monica_site_beta-worker`	Aug 8	psycopg2 2.9.12

So the worker crash-looped on LISTEN/NOTIFY (TypeError: 'list' object is not callable in dj_queue/runtime/notify.py) long after monica_site had moved to psycopg3, because its image was never rebuilt.

Branch is merged up with master, which already carries the worker auto-start from #18; the diff here is just the build step.

Test plan

Manual COMPOSE_PROFILES=worker docker compose build worker on adama produced an image with psycopg 3.3.4 and the notify errors stopped.
Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild.
Hosts without compose_profiles (roslin, ai-server-4080) still build/start web only.

## Summary The django deploy sets `COMPOSE_PROFILES` on the **start** step but not on the **build** step, so `docker compose build` skips profile-gated services. `up -d` then reuses whatever image already exists and the container silently keeps running old code. This adds `COMPOSE_PROFILES` to the build step so it matches the start step. Only affects hosts that set `host_apps.compose_profiles` (today: the `monica_site` dj-queue worker on adama). ## Symptom this fixes On adama the beta worker image was 20 hours stale while web was current: | Image | Built | Postgres driver | |---|---|---| | `monica_site_beta-web` | today | psycopg 3.3.4 | | `monica_site_beta-worker` | Aug 8 | psycopg2 2.9.12 | So the worker crash-looped on LISTEN/NOTIFY (`TypeError: 'list' object is not callable` in `dj_queue/runtime/notify.py`) long after `monica_site` had moved to psycopg3, because its image was never rebuilt. Branch is merged up with `master`, which already carries the worker auto-start from [#18](https://git.aimloperations.com/ai_ml_operations/server-infra/pulls/18); the diff here is just the build step. ## Test plan - [x] Manual `COMPOSE_PROFILES=worker docker compose build worker` on adama produced an image with psycopg 3.3.4 and the notify errors stopped. - [ ] Beta deploy from this branch recreates the worker with a fresh image, no manual rebuild. - [ ] Hosts without `compose_profiles` (roslin, ai-server-4080) still build/start web only.

westfarn added 2 commits 2026-08-09 04:23:34 -07:00

Auto-start monica_site dj-queue worker on adama deploy.

Sync runner checkout / sync (pull_request) Successful in 6s

Details

9a0cbf80d9

Campaign sends stay Queued without the worker; wire compose_profiles into
django deploy so adama brings up the singleton with web (closes #17).

Build profile-gated services so workers do not run stale images. 853eff020d

The build step omitted COMPOSE_PROFILES, so docker compose skipped the
dj-queue worker and up -d reused an image from an earlier deploy; the
monica_site beta worker kept psycopg2 and crash-looped on LISTEN/NOTIFY.

westfarn added 1 commit 2026-08-09 04:24:16 -07:00

Merge remote-tracking branch 'origin/master' into issue-17-monica-dj-queue-worker

Sync runner checkout / sync (pull_request) Successful in 7s

Details

63cb6ae45f

westfarn changed title from ~~Auto-start and rebuild the monica_site dj-queue worker (closes #17)~~ to Build profile-gated services so workers do not run stale images

2026-08-09 04:24:31 -07:00

westfarn merged commit 2e3fe13b39 into master

2026-08-09 04:36:52 -07:00

westfarn deleted branch issue-17-monica-dj-queue-worker

2026-08-09 04:36:52 -07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ai_ml_operations/server-infra#19