ai_ml_operations/monica_site
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
westfarn
2026-07-16 05:48:12 -05:00
parent c36d46c3b3
commit e9be65df8c
558 changed files with 74087 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/monica-site-design.md
+103 -65
View File
@@ -64,13 +64,12 @@ flowchart TB
end
subgraph worker [Background - single host]
WorkerProc["Celery worker"]
Beat["Celery beat (scheduler)"]
TaskWorker["django.tasks worker process"]
DispatchTick["dispatch_due management command / tick"]
end
subgraph data [Shared services 10.0.0.230]
PG[("Postgres - monica_site DB")]
Redis[("Redis - broker + cache")]
PG[("Postgres - monica_site DB + task queue store")]
end
subgraph external [External APIs]
@@ -85,39 +84,69 @@ flowchart TB
Realtor(["Realtor (portal)"]) --> NPM
NPM --> Web1 & Web2
Web1 & Web2 --> PG
Web1 & Web2 --> Redis
Web1 & Web2 --> Recaptcha
WorkerProc --> PG
WorkerProc --> Redis
Beat --> Redis
WorkerProc --> SMTP2GO & Postcard & Meta & LinkedIn
TaskWorker --> PG
DispatchTick --> PG
TaskWorker --> SMTP2GO & Postcard & Meta & LinkedIn
```
Key points:
- **Web tier is stateless** and runs on every app host (matches `company_site`: active/active
behind NPM, `SECURE_PROXY_SSL_HEADER`, `USE_X_FORWARDED_HOST`).
- **Background tier is a singleton**: exactly one Celery worker + one beat, so scheduled
postcards/social posts and bulk sends fire once (not once per replica). Pinned to one host.
- **Shared external Postgres + Redis** live on the existing `10.0.0.230` box alongside NPM,
consistent with the `server-infra` shared-Postgres model.
- **Background work uses Django 6.0 Tasks (`django.tasks`)** — define with `@task`, enqueue with
`.enqueue()`. Queue store lives in **Postgres** (same DB as the app). No Redis, no Celery.
- **Worker is a singleton**: Django's Tasks API does not run tasks itself; a separate worker
process (from a production-capable third-party backend) claims and executes queued tasks.
Exactly one worker host so scheduled sends don't duplicate.
- **Shared external Postgres** on `10.0.0.230` (same pattern as `company_site`).
---
## 4. Why background jobs (and the Django 6.0 angle)
## 4. Background jobs — Django Tasks (not Celery/Redis)
Postcard rendering/mailing, social scheduling, and bulk email/SMS are slow, rate-limited, and
retry-prone — they must run outside the request cycle.
Postcard mailing, social publishing, and bulk email/SMS must run outside the request cycle.
- **Recommended: Celery + Redis + Celery Beat.** Mature, supports scheduled (cron-like) jobs,
retries with backoff, and rate limiting per provider. Beat runs on one host only.
- **Django 6.0 Tasks framework (`django.tasks`)** is a clean native way to *enqueue* background
work and will be used for the app-facing API (`enqueue()` calls), but it is a task-execution
abstraction, not a cron scheduler — so Beat still owns time-based triggers. We wrap enqueue
calls thinly so we can swap backends later.
### Decision
Use **Django 6.0's built-in Tasks framework** (`django.tasks`) as the only background API.
> Decision: use Celery/Redis for execution + scheduling now; keep enqueue calls behind a small
> `messaging.tasks`/`social.tasks` seam so a future move to pure `django.tasks` is mechanical.
| Piece | Choice |
|-------|--------|
| Define / enqueue | `@task` + `.enqueue()` / `aenqueue()` |
| Dev / tests | `ImmediateBackend` (or `DummyBackend` in tests) |
| Production queue | Third-party **database-backed** Task backend (Postgres queue store) + its worker CLI |
| Deferred / scheduled | Prefer `using(run_after=...)` when the backend supports defer; otherwise poll due rows via a small `dispatch_due` management command that enqueues Tasks |
Django guarantees task definition, validation, queuing, and result handling. **Execution** is
always a separate worker process — pick a backend from the
[Django Tasks community ecosystem](https://docs.djangoproject.com/en/6.0/topics/tasks/) /
Django Packages grid that provides a durable queue + worker. Do **not** introduce Redis or
Celery for this app.
### Pattern
```python
from functools import partial
from django.db import transaction
from django.tasks import task
@task
def send_campaign_message(message_id: int) -> None:
...
# After DB writes commit — workers use a separate connection
with transaction.atomic():
message = Message.objects.create(...)
transaction.on_commit(partial(send_campaign_message.enqueue, message_id=message.pk))
```
Scheduled social/campaign work:
1. Persist `scheduled_for` on the model when the realtor schedules.
2. Either enqueue immediately with `run_after=scheduled_for` (if backend supports defer), **or**
run `manage.py dispatch_due` on a short interval (same host as the worker) to find due rows
and `.enqueue()` publish/send tasks.
3. Worker executes; updates status / `provider_message_id` / `remote_id` on success or failure.
> No Redis broker. Task queue + app data share Postgres. One less moving part vs Celery.
---
## 5. Django Apps & Responsibilities
@@ -237,21 +266,23 @@ realtor with a fixed set of accounts.
app review / posting-scope approval — start this early.
- Connectors live in `social/connectors/{meta,linkedin}.py` behind a common
`SocialConnector` interface (`publish(post, target) -> remote_id`, `refresh_token()`).
- OAuth tokens stored encrypted; Beat enqueues due `SocialPost`s; worker fans out to each
`SocialPostTarget` with per-platform retry.
- OAuth tokens stored encrypted; due `SocialPost`s are enqueued as Django Tasks
(`run_after` or `dispatch_due`); the task worker fans out to each `SocialPostTarget`
with per-platform retry.
```mermaid
sequenceDiagram
participant R as Realtor
participant D as Dashboard
participant DB as Postgres
participant B as Celery Beat
participant W as Worker
participant T as django.tasks enqueue
participant W as Task worker
participant P as Platform API
R->>D: Compose post, pick FB/IG/LI, schedule
D->>DB: SocialPost + SocialPostTargets (status=scheduled)
B->>DB: poll due posts
B->>W: enqueue publish(target)
D->>T: enqueue publish with run_after or via dispatch_due
T->>DB: task row in queue store
W->>DB: claim due task
W->>P: publish via connector
P-->>W: remote_id / error
W->>DB: update target status + remote_id
@@ -278,9 +309,9 @@ db**, `scripts/docker-entrypoint.sh`, `.gitea/workflows`).
```
monica_site/ # repo root
├── Dockerfile # python:3.12-slim + uv (copy of company_site pattern)
├── docker-compose.yml # local dev: web + db + redis
├── docker-compose.prod.yml # prod: web only (+ worker/beat); external PG + Redis
├── pyproject.toml # uv-managed deps (django==6.0, ...)
├── docker-compose.yml # local dev: web + db (+ optional task worker)
├── docker-compose.prod.yml # prod: web on all hosts; task worker on one host; external PG
├── pyproject.toml # uv-managed deps (django==6.0, db task backend, ...)
├── uv.lock
├── .env.example / .env.prod.example # documented env (extends company_site's)
├── .dockerignore / .gitignore
@@ -290,8 +321,7 @@ monica_site/ # repo root
│ └── monica-site-design.md # this document
├── scripts/
│ ├── docker-entrypoint.sh # migrate + collectstatic + gunicorn (web role)
│ ├── worker-entrypoint.sh # celery worker
│ ├── beat-entrypoint.sh # celery beat
│ ├── worker-entrypoint.sh # django.tasks backend worker (+ optional dispatch_due loop)
│ └── validate-env.sh
├── .gitea/workflows/
│ ├── ci.yml # PR: uv sync + manage.py test
@@ -303,17 +333,18 @@ monica_site/ # repo root
│ ├── __init__.py
│ ├── settings/
│ │ ├── __init__.py # dispatch on DJANGO_ENV (dev|beta|prod)
│ │ ├── base.py # shared (DATABASE_URL parse, recaptcha, smtp2go, celery)
│ │ ├── dev.py
│ │ ├── base.py # shared (DATABASE_URL, TASKS, recaptcha, smtp2go)
│ │ ├── dev.py # ImmediateBackend for local
│ │ ├── beta.py
│ │ ├── prod.py # DEBUG off, proxy SSL header, secure cookies
│ │ ├── prod.py # DEBUG off, proxy SSL header, secure cookies, DB task backend
│ │ └── logging.py
│ ├── celery.py # Celery app
│ ├── storage.py # TolerantManifestStaticFilesStorage (reuse)
│ ├── urls.py
│ ├── asgi.py
│ └── wsgi.py
├── core/ # base models/mixins/utils
│ └── management/commands/
│ └── dispatch_due.py # poll scheduled rows → enqueue django.tasks (if needed)
├── public/ # marketing pages + contact form (+ templates/static)
├── accounts/ # realtor auth + RealtorProfile
├── dashboard/ # private portal shell
@@ -322,7 +353,7 @@ monica_site/ # repo root
├── analytics/ # UTMVisit, Attribution + reports
├── messaging/ # templates, campaigns, messages, tasks
│ ├── models.py
│ ├── tasks.py # celery tasks (send email/sms/postcard batches)
│ ├── tasks.py # @task senders (email/sms/postcard batches)
│ ├── channels/ # email.py, sms.py, postcard.py (dispatch)
│ └── providers/
│ ├── email/smtp2go.py
@@ -330,7 +361,7 @@ monica_site/ # repo root
│ └── postcard/{base.py,lob.py,click2mail.py,postgrid.py}
└── social/ # accounts, posts, scheduling
├── models.py
├── tasks.py
├── tasks.py # @task publish_social_target, etc.
└── connectors/{base.py,meta.py,linkedin.py}
```
@@ -349,10 +380,11 @@ We plug in by mirroring `company_site` and adding one catalog entry.
- **Dockerfile**: identical pattern to `company_site` — `python:3.12-slim`, `uv sync --frozen
--no-dev`, copy project, `scripts/docker-entrypoint.sh` runs
`migrate → collectstatic → gunicorn monica_site.wsgi:application`.
- **`docker-compose.prod.yml`**: `web` service only for the app hosts (external Postgres +
Redis via `DATABASE_URL` / `REDIS_URL`; **drop the bundled `db`**, per the server-infra
"required changes in each app repo" checklist). Add `worker` + `beat` services deployed to a
single host (or a separate `docker-compose.worker.yml`) so schedulers are singletons.
- **`docker-compose.prod.yml`**: `web` on app hosts (external Postgres via `DATABASE_URL`;
**drop the bundled `db`**, per the server-infra checklist). Add a **`worker`** service on
exactly one host (or `docker-compose.worker.yml`) that runs the chosen django.tasks backend
worker. Optionally the same process/host loops `dispatch_due` for schedule polling if the
backend lacks `run_after` defer support.
### 8.2 server-infra wiring (repo reviewed)
- Add to `inventory/group_vars/all.yml` → `app_catalog`:
@@ -370,10 +402,10 @@ monica_site:
- Assign host ports in `host_vars/<host>.yml` `host_apps` (reserve a new port pair, e.g.
`prod 8003 / beta 8013`, matching across adama+roslin so NPM can balance).
- Create control-node secret `~/Documents/secrets/monica_site/monica_site_prod.env`
(`DATABASE_URL`, `REDIS_URL`, `DJANGO_SECRET_KEY`, `WEB_PORT`, provider keys). Never committed;
(`DATABASE_URL`, `DJANGO_SECRET_KEY`, `WEB_PORT`, provider keys). Never committed;
Ansible pushes it to `/opt/apps/env/monica_site_prod.env` (mode 600) at deploy.
- Shared Postgres prereq on `10.0.0.230`: create `monica_site` DB + user (and `monica_site_beta`
for beta), like the other apps.
for beta), like the other apps. Task queue tables live in the same DB.
### 8.3 CI/CD flow
Same as `company_site`:
@@ -391,14 +423,14 @@ flowchart LR
UT -->|green| DEP["deploy.yml -> server-infra deploy.sh"]
DEP --> A["adama: web :8003"]
DEP --> R["roslin: web :8003"]
DEP --> WK["worker+beat (single host)"]
DEP --> WK["django.tasks worker (single host)"]
A & R --> NPM["NPM balances monica domain"]
```
### 8.4 New infra bits beyond company_site
- **Redis** service on `10.0.0.230` (broker + cache) — small addition to the shared box.
- **Worker/beat placement**: one host runs the `worker`+`beat` compose project; document that it
must be exactly one to avoid duplicate scheduled sends.
- **Task worker placement**: exactly one host runs the django.tasks worker (and `dispatch_due`
if used). Document this so active/active web replicas do not each run a worker.
- **No Redis** required for background jobs — queue store is Postgres.
---
@@ -414,9 +446,9 @@ DATABASE_URL=postgres://monica_site:<pw>@10.0.0.230:5432/monica_site
WEB_PORT=8003
GUNICORN_WORKERS=2
# Redis / Celery
REDIS_URL=redis://10.0.0.230:6379/0
CELERY_BROKER_URL=${REDIS_URL}
# Django Tasks — prod uses a DB-backed third-party backend + worker.
# Dev defaults to ImmediateBackend in settings/dev.py (no worker needed).
# TASKS is configured in settings, not usually via env; override only if needed.
# reCAPTCHA v3 (same as company_site)
RECAPTCHA_PUBLIC_KEY=...
@@ -454,12 +486,14 @@ TIANJI_WEBSITE_ID=...
### Phase 0 — Scaffolding & deploy skeleton
- Create `monica_site` Django 6.0 project via `uv`; `settings/` dispatch; Dockerfile,
compose (dev with db+redis), entrypoints, `.gitea/workflows`, pre-commit.
compose (dev with db), entrypoints, `.gitea/workflows`, pre-commit.
- Configure `TASKS` (ImmediateBackend in `dev`, DB-backed backend in `prod`); add
`worker-entrypoint.sh` for the task worker.
- Add `core` app (`TimeStampedModel`), health check, base template.
- Wire `server-infra` `app_catalog` entry + host ports + secrets; get a "hello world" deploying
to adama+roslin behind NPM on a staging domain.
- Add Redis to `10.0.0.230`; create Postgres DB/user.
- **Exit:** empty app deploys via CI to prod hosts, health check green.
- Create Postgres DB/user on `10.0.0.230`.
- **Exit:** empty app deploys via CI to prod hosts, health check green; worker can claim a no-op task.
### Phase 1 — Public site + contact form + leads + UTM
- `public`: marketing pages (home/about/contact), `django_recaptcha` v3 contact form.
@@ -470,25 +504,25 @@ TIANJI_WEBSITE_ID=...
- **Exit:** realtor logs in, sees leads with source attribution; visitors submit safely.
### Phase 2 — Mailing list + Email/SMS outreach + opt-out
- `messaging`: `MessageTemplate`, `Campaign`, `Message`, `ProviderEvent`; Celery + Beat online.
- `messaging`: `MessageTemplate`, `Campaign`, `Message`, `ProviderEvent`; `@task` senders online.
- Email via SMTP2GO with unsubscribe links; SMS via SMTP2GO SMS API.
- Opt-out views + SMS `STOP` webhook → `Suppression`; consent enforced on every send.
- Dashboard: contact list management, segment/select recipients, compose + schedule campaign,
delivery status.
delivery status (enqueue via `transaction.on_commit` + django.tasks).
- **Exit:** realtor sends a real email + SMS campaign to consented contacts; opt-out works.
### Phase 3 — Postcards
- `messaging/providers/postcard/` interface + Lob adapter (+ Click2Mail adapter as the
low-volume option); `POSTCARD_PROVIDER` switch.
- Postcard campaign flow: pick template/artwork, select recipients (address required + Lob
address verification), send batch via worker, poll delivery status.
address verification), enqueue batch as Django Tasks, poll delivery status.
- Client prerequisites called out: Lob account, A2P 10DLC for SMS.
- **Exit:** realtor mails a postcard batch and sees delivery tracking.
### Phase 4 — Social automation
- `social`: `SocialAccount` OAuth connect (Meta + LinkedIn), encrypted token storage.
- Connectors for Meta (FB Page + IG Business) and LinkedIn; `SocialPost` + `SocialPostTarget`
compose/schedule; Beat dispatches, worker fans out with retries.
compose/schedule; enqueue via `run_after` or `dispatch_due`; task worker fans out with retries.
- Dashboard: composer with per-platform preview, calendar/scheduled view, post status.
- **Kick off LinkedIn app review in Phase 0–1** given 2–4 week approval lead time.
- **Exit:** realtor schedules one post to all three platforms; it publishes and reports status.
@@ -512,8 +546,12 @@ flowchart LR
## 11. Key Risks & Decisions
- **Singleton scheduler:** must run exactly one Beat + worker across the active/active fleet, or
scheduled sends duplicate. Pin to one host; documented in deploy.
- **Singleton task worker:** exactly one django.tasks worker across the active/active fleet, or
duplicate claims / double-sends are possible depending on backend locking. Pin to one host.
- **Django Tasks has no built-in production worker:** must pick and pin a third-party DB backend
+ worker early in Phase 0; built-in backends are Immediate/Dummy only.
- **Scheduling:** if the chosen backend lacks `supports_defer` / `run_after`, rely on
`dispatch_due` polling — keep the tick interval short and idempotent.
- **LinkedIn approval latency (2–4 wks):** start the app-review process at project kickoff.
- **SMS compliance (A2P 10DLC/TCPA):** client-side registration required before SMS launch;
consent + `STOP` handling are built in but registration is a prerequisite.
@@ -527,10 +565,10 @@ flowchart LR
## 12. Cost Snapshot (why it beats PostcardMania)
- **Hosting:** runs on existing adama/roslin fleet — near-zero marginal infra (add Redis).
- **Hosting:** runs on existing adama/roslin fleet — near-zero marginal infra (Postgres only;
no Redis/Celery stack).
- **Email/SMS:** SMTP2GO usage-based (already an org account).
- **Postcards:** ~$0.35–0.77/4x6 at provider cost vs PostcardMania's bundled campaign packages;
no forced campaign minimums.
- **Social:** $0 API fees (native), vs PostcardMania add-ons / aggregator per-profile fees.
- **No PostcardMania platform/campaign markup**; realtor pays provider cost + our hosting.
```