monica_site/docs/monica-site-design.md

Monica Site — Architecture & Design

A Django 6.0 web platform to replace PostcardMania for a single realtor client: a public marketing site with a captcha-protected contact form, a private portal for leads and UTM analytics, a managed mailing list with multi-channel outreach (email, SMS, postcard), and direct social-media automation for Facebook, Instagram, and LinkedIn.

Built to deploy the same way as the existing company_site app: a Docker image using uv, Postgres in production, and CI-driven deploys through the server-infra Ansible pipeline onto multiple active/active instances behind Nginx Proxy Manager (NPM).

Repo: monica_site (mirrors the company_site naming convention). Single-tenant now, but models and app boundaries are drawn so a future Realtor/Organization foreign key can be added without a rewrite.

---

1. Goals & Non-Goals

Goals

• Public site to market the realtor, with a contact form protected by reCAPTCHA v3.
• Private portal: lead inbox, UTM/campaign analytics, mailing-list management, campaign composer, social scheduler.
• One mailing list of contacts. Form submissions auto-create/append contacts.
• Per-channel, per-contact consent with easy opt-out (email unsubscribe link, SMS STOP).
• Outreach over Email + SMS (SMTP2GO) and Postcard (pluggable provider, default Lob).
• Social posting/scheduling to Facebook, Instagram, LinkedIn via native APIs.
• Cheaper than PostcardMania + adds social automation.

Non-Goals (initial)

• Multi-tenant SaaS (structured for it later, not built now).
• Full CRM (deal pipelines, MLS/IDX listing sync) — future phase.
• In-app billing/subscriptions.
• Building our own print/mail logistics (we integrate a provider).

---

2. Feature Breakdown (mapped to the request)

• 1. Public site + contact form w/ reCAPTCHA → public app. Reuse the proven django_recaptcha v3 setup from company_site (RECAPTCHA_PUBLIC_KEY / RECAPTCHA_PRIVATE_KEY, django_recaptcha/widget_v3.html).
• 2. Private side: leads + UTM analytics → dashboard (portal shell + auth) + leads (lead records, statuses) + analytics (UTM capture + reporting).
• 3. Mailing list + multi-channel outreach + opt-out → contacts (people, consent) + messaging (channels, templates, campaigns, provider adapters). Email/SMS via SMTP2GO, postcards via a pluggable provider interface.
• 4. Social automation (FB/IG/LinkedIn) → social (connected accounts, posts, scheduling, native connectors).

---

3. High-Level Architecture

flowchart TB
    subgraph edge [Edge]
        NPM["Nginx Proxy Manager 10.0.0.230 (TLS + load balance)"]
    end

    subgraph hosts [App hosts adama + roslin - active/active]
        Web1["monica_site web (gunicorn) :PORT"]
        Web2["monica_site web (gunicorn) :PORT"]
    end

    subgraph worker [Background - single host]
        WorkerProc["Celery worker"]
        Beat["Celery beat (scheduler)"]
    end

    subgraph data [Shared services 10.0.0.230]
        PG[("Postgres - monica_site DB")]
        Redis[("Redis - broker + cache")]
    end

    subgraph external [External APIs]
        SMTP2GO["SMTP2GO (email + SMS)"]
        Postcard["Postcard provider (Lob default)"]
        Meta["Meta Graph API (FB + IG)"]
        LinkedIn["LinkedIn API"]
        Recaptcha["Google reCAPTCHA"]
    end

    Visitor(["Public visitor"]) --> NPM
    Realtor(["Realtor (portal)"]) --> NPM
    NPM --> Web1 & Web2
    Web1 & Web2 --> PG
    Web1 & Web2 --> Redis
    Web1 & Web2 --> Recaptcha
    WorkerProc --> PG
    WorkerProc --> Redis
    Beat --> Redis
    WorkerProc --> SMTP2GO & Postcard & Meta & LinkedIn

Key points:

• Web tier is stateless and runs on every app host (matches company_site: active/active behind NPM, SECURE_PROXY_SSL_HEADER, USE_X_FORWARDED_HOST).
• Background tier is a singleton: exactly one Celery worker + one beat, so scheduled postcards/social posts and bulk sends fire once (not once per replica). Pinned to one host.
• Shared external Postgres + Redis live on the existing 10.0.0.230 box alongside NPM, consistent with the server-infra shared-Postgres model.

---

4. Why background jobs (and the Django 6.0 angle)

Postcard rendering/mailing, social scheduling, and bulk email/SMS are slow, rate-limited, and retry-prone — they must run outside the request cycle.

• Recommended: Celery + Redis + Celery Beat. Mature, supports scheduled (cron-like) jobs, retries with backoff, and rate limiting per provider. Beat runs on one host only.
• Django 6.0 Tasks framework (django.tasks) is a clean native way to enqueue background work and will be used for the app-facing API (enqueue() calls), but it is a task-execution abstraction, not a cron scheduler — so Beat still owns time-based triggers. We wrap enqueue calls thinly so we can swap backends later.

Decision: use Celery/Redis for execution + scheduling now; keep enqueue calls behind a small messaging.tasks/social.tasks seam so a future move to pure django.tasks is mechanical.

---

5. Django Apps & Responsibilities

App	Responsibility	Key models
core	Shared base classes, mixins, utils	TimeStampedModel (created/last_modified, like company_site.TimeInfoBase)
public	Public marketing pages, contact form + reCAPTCHA, sitemap/robots/SEO	— (writes Lead + Contact)
accounts	Realtor auth, portal login, profile	uses django.contrib.auth; RealtorProfile
dashboard	Private portal shell, nav, home widgets	— (read-only aggregates)
leads	Lead capture + lifecycle (new/contacted/won/lost), notes	Lead, LeadNote
contacts	The mailing list: people + per-channel consent + opt-out	Contact, ConsentRecord, Suppression
analytics	UTM capture, attribution, campaign reporting	UTMVisit, Attribution
messaging	Templates, campaigns, per-recipient messages, channel adapters	MessageTemplate, Campaign, Message, ProviderEvent
social	Connected social accounts, posts, scheduling, native connectors	SocialAccount, SocialPost, SocialPostTarget

Data model sketch

erDiagram
    Contact ||--o{ ConsentRecord : has
    Contact ||--o{ Message : receives
    Lead }o--|| Contact : "linked to"
    Lead ||--o{ LeadNote : has
    Lead }o--o| Attribution : "attributed by"
    UTMVisit }o--o| Lead : "converts to"
    Campaign ||--o{ Message : sends
    MessageTemplate ||--o{ Campaign : "used by"
    SocialPost ||--o{ SocialPostTarget : "fans out to"
    SocialAccount ||--o{ SocialPostTarget : "posted via"

    Contact {
        uuid id
        string email
        string phone
        string first_name
        string last_name
        json postal_address
        string source
    }
    ConsentRecord {
        string channel "email|sms|postcard"
        bool opted_in
        datetime changed_at
        string reason
    }
    Message {
        string channel
        string status "queued|sent|delivered|failed|bounced"
        string provider
        string provider_message_id
        datetime scheduled_for
    }
    SocialPost {
        text body
        json media
        datetime scheduled_for
        string status
    }
    SocialPostTarget {
        string platform "facebook|instagram|linkedin"
        string status
        string remote_id
    }

Consent model (important): consent is per (contact, channel). Every send checks for an opted-in ConsentRecord and no active Suppression. Opt-out writes an immutable audit trail (ConsentRecord change + Suppression row). This keeps us defensible on CAN-SPAM/TCPA and lets a contact opt out of SMS while keeping email.

---

6. Channel Integrations

6.1 Email + SMS — SMTP2GO

• Email: Django SMTP backend to mail.smtp2go.com (already the company_site default: EMAIL_HOST, EMAIL_PORT=2525, TLS). Every marketing email includes an unsubscribe link (signed token → one-click opt-out view).
• SMS: SMTP2GO SMS REST API. Inbound STOP/UNSUBSCRIBE handled via SMTP2GO webhook → writes a Suppression. A2P 10DLC registration is a client onboarding prerequisite (call out in Phase 3).

6.2 Postcards — pluggable provider (default Lob)

A small provider interface so the realtor isn't locked in and we can shop on price:

class PostcardProvider(Protocol):
    def send_postcard(self, *, to: PostalAddress, from_: PostalAddress,
                      front: Asset, back: Asset, idempotency_key: str) -> ProviderResult: ...
    def get_status(self, provider_id: str) -> DeliveryStatus: ...

Adapters live in messaging/providers/postcard/. Selected via POSTCARD_PROVIDER env var.

Provider	Model	~4x6 postcard	Best for	Notes
Lob (default)	API-first, tiered	Free dev tier ~$0.77; ~$0.51 on $260/mo Startup	Clean API, address verification, in-transit tracking	Best DX; monthly fee only worth it at volume
Click2Mail	Pay-per-piece, no subscription	~$0.35–0.70	Low/occasional volume, no monthly fee	API less polished; great when volume is small
PostGrid	API + dashboard, subscription	Contact sales (from ~$250/mo)	Compliance-heavy, templates	Overkill unless compliance-driven
Stannp	Marketer-friendly, no minimums	Transparent per-piece	Non-dev fallback, EU	Good dashboard
USPS EDDM	Postage-only saturation	Postage only	Whole-route blasts (no list)	No per-address API; manual/bulk option

Recommendation: start on Lob Developer (free) tier to validate, keep Click2Mail as the low-volume cost option behind the same interface. Revisit once monthly volume is known — the abstraction makes switching a config change.

6.3 Social — native APIs (Facebook, Instagram, LinkedIn)

Chosen over an aggregator (Ayrshare) to avoid per-profile monthly fees, since this is one realtor with a fixed set of accounts.

• Facebook + Instagram: Meta Graph API (one integration covers both). IG posting requires an IG Business/Creator account linked to a Facebook Page. Plan for quarterly Graph API version bumps.
• LinkedIn: LinkedIn API (organization + member posting). Budget 2–4 weeks lead time for app review / posting-scope approval — start this early.
• Connectors live in social/connectors/{meta,linkedin}.py behind a common SocialConnector interface (publish(post, target) -> remote_id, refresh_token()).
• OAuth tokens stored encrypted; Beat enqueues due SocialPosts; worker fans out to each SocialPostTarget with per-platform retry.

sequenceDiagram
    participant R as Realtor
    participant D as Dashboard
    participant DB as Postgres
    participant B as Celery Beat
    participant W as Worker
    participant P as Platform API
    R->>D: Compose post, pick FB/IG/LI, schedule
    D->>DB: SocialPost + SocialPostTargets (status=scheduled)
    B->>DB: poll due posts
    B->>W: enqueue publish(target)
    W->>P: publish via connector
    P-->>W: remote_id / error
    W->>DB: update target status + remote_id

6.4 UTM Analytics

• Landing views capture utm_source/medium/campaign/term/content + referrer into UTMVisit (cookie/session correlation id).
• On contact-form submit, the correlation id ties the UTMVisit → Lead → Attribution.
• Dashboard reports leads by source/campaign over time (Chart.js — already vendored in company_site static; reuse the material-dashboard theme).
• Reuse existing Tianji (self-hosted analytics already run at tianji.aimloperations.com) for pageview-level traffic; our analytics app owns conversion attribution that Tianji can't tie to CRM records.

---

7. Proposed Folder / File Layout

Mirrors company_site exactly for deploy compatibility (Dockerfile at repo root, uv, settings/ package with DJANGO_ENV dispatch, docker-compose.prod.yml with no bundled db, scripts/docker-entrypoint.sh, .gitea/workflows).

monica_site/                           # repo root
├── Dockerfile                         # python:3.12-slim + uv (copy of company_site pattern)
├── docker-compose.yml                 # local dev: web + db + redis
├── docker-compose.prod.yml            # prod: web only (+ worker/beat); external PG + Redis
├── pyproject.toml                     # uv-managed deps (django==6.0, ...)
├── uv.lock
├── .env.example / .env.prod.example   # documented env (extends company_site's)
├── .dockerignore / .gitignore
├── .pre-commit-config.yaml
├── README.md
├── docs/
│   └── monica-site-design.md          # this document
├── scripts/
│   ├── docker-entrypoint.sh           # migrate + collectstatic + gunicorn (web role)
│   ├── worker-entrypoint.sh           # celery worker
│   ├── beat-entrypoint.sh             # celery beat
│   └── validate-env.sh
├── .gitea/workflows/
│   ├── ci.yml                         # PR: uv sync + manage.py test
│   ├── unittests.yml                  # master: tests gate deploy
│   └── deploy.yml                     # calls server-infra/scripts/deploy.sh --app monica_site
└── monica_site/                       # Django project dir (manage.py lives here)
    ├── manage.py
    ├── monica_site/                   # project package
    │   ├── __init__.py
    │   ├── settings/
    │   │   ├── __init__.py            # dispatch on DJANGO_ENV (dev|beta|prod)
    │   │   ├── base.py                # shared (DATABASE_URL parse, recaptcha, smtp2go, celery)
    │   │   ├── dev.py
    │   │   ├── beta.py
    │   │   ├── prod.py                # DEBUG off, proxy SSL header, secure cookies
    │   │   └── logging.py
    │   ├── celery.py                  # Celery app
    │   ├── storage.py                 # TolerantManifestStaticFilesStorage (reuse)
    │   ├── urls.py
    │   ├── asgi.py
    │   └── wsgi.py
    ├── core/                          # base models/mixins/utils
    ├── public/                        # marketing pages + contact form (+ templates/static)
    ├── accounts/                      # realtor auth + RealtorProfile
    ├── dashboard/                     # private portal shell
    ├── leads/                         # Lead, LeadNote
    ├── contacts/                      # Contact, ConsentRecord, Suppression
    ├── analytics/                     # UTMVisit, Attribution + reports
    ├── messaging/                     # templates, campaigns, messages, tasks
    │   ├── models.py
    │   ├── tasks.py                   # celery tasks (send email/sms/postcard batches)
    │   ├── channels/                  # email.py, sms.py, postcard.py (dispatch)
    │   └── providers/
    │       ├── email/smtp2go.py
    │       ├── sms/smtp2go.py
    │       └── postcard/{base.py,lob.py,click2mail.py,postgrid.py}
    └── social/                        # accounts, posts, scheduling
        ├── models.py
        ├── tasks.py
        └── connectors/{base.py,meta.py,linkedin.py}

Each Django app follows the standard admin.py / apps.py / models.py / views.py / urls.py / forms.py / migrations/ / templates/<app>/ / static/<app>/ / tests.py layout used in company_site (e.g. public/, financial/).

---

8. Deployment (matches server-infra pipeline)

The existing pipeline already knows how to deploy Dockerized Django apps to active/active hosts. We plug in by mirroring company_site and adding one catalog entry.

8.1 Container & compose

• Dockerfile: identical pattern to company_site — python:3.12-slim, uv sync --frozen --no-dev, copy project, scripts/docker-entrypoint.sh runs migrate → collectstatic → gunicorn monica_site.wsgi:application.
• docker-compose.prod.yml: web service only for the app hosts (external Postgres + Redis via DATABASE_URL / REDIS_URL; drop the bundled db, per the server-infra "required changes in each app repo" checklist). Add worker + beat services deployed to a single host (or a separate docker-compose.worker.yml) so schedulers are singletons.

8.2 server-infra wiring (repo reviewed)

• Add to inventory/group_vars/all.yml → app_catalog:

monica_site:
  type: django
  repo: "{{ git_base_url }}/ai_ml_operations/monica_site.git"
  default_branch: main
  compose_file: docker-compose.prod.yml
  web_service: web
  migrate_cmd: "uv run python manage.py migrate --noinput"

• Assign host ports in host_vars/<host>.yml host_apps (reserve a new port pair, e.g. prod 8003 / beta 8013, matching across adama+roslin so NPM can balance).
• Create control-node secret ~/Documents/secrets/monica_site/monica_site_prod.env (DATABASE_URL, REDIS_URL, DJANGO_SECRET_KEY, WEB_PORT, provider keys). Never committed; Ansible pushes it to /opt/apps/env/monica_site_prod.env (mode 600) at deploy.
• Shared Postgres prereq on 10.0.0.230: create monica_site DB + user (and monica_site_beta for beta), like the other apps.

8.3 CI/CD flow

Same as company_site:

1. PR → ci.yml runs uv sync + manage.py test.
2. Merge to main → unittests.yml (containerized tests) → on green, deploy.yml calls server-infra/scripts/deploy.sh --app monica_site --env prod --ref <sha>.
3. deploy-apps.yml checks out the ref on each app host, injects .env, docker compose build, up -d, runs migrate once.

flowchart LR
    PR["PR to main"] --> CI["ci.yml: uv sync + tests"]
    CI --> Merge["merge main"]
    Merge --> UT["unittests.yml (docker tests)"]
    UT -->|green| DEP["deploy.yml -> server-infra deploy.sh"]
    DEP --> A["adama: web :8003"]
    DEP --> R["roslin: web :8003"]
    DEP --> WK["worker+beat (single host)"]
    A & R --> NPM["NPM balances monica domain"]

8.4 New infra bits beyond company_site

• Redis service on 10.0.0.230 (broker + cache) — small addition to the shared box.
• Worker/beat placement: one host runs the worker+beat compose project; document that it must be exactly one to avoid duplicate scheduled sends.

---

9. Environment Variables (extends company_site's set)

# Core (same as company_site)
DJANGO_ENV=prod
DJANGO_DEBUG=false
DJANGO_SECRET_KEY=...
DJANGO_ALLOWED_HOSTS=monicadomain.com,www.monicadomain.com
DATABASE_URL=postgres://monica_site:<pw>@10.0.0.230:5432/monica_site
WEB_PORT=8003
GUNICORN_WORKERS=2

# Redis / Celery
REDIS_URL=redis://10.0.0.230:6379/0
CELERY_BROKER_URL=${REDIS_URL}

# reCAPTCHA v3 (same as company_site)
RECAPTCHA_PUBLIC_KEY=...
RECAPTCHA_PRIVATE_KEY=...

# SMTP2GO email (same defaults as company_site)
EMAIL_HOST=mail.smtp2go.com
EMAIL_HOST_USER=...
EMAIL_HOST_PASSWORD=...
EMAIL_PORT=2525
EMAIL_USE_TLS=true

# SMTP2GO SMS
SMTP2GO_SMS_API_KEY=...

# Postcards
POSTCARD_PROVIDER=lob            # lob|click2mail|postgrid
LOB_API_KEY=...

# Social (native)
META_APP_ID=...
META_APP_SECRET=...
LINKEDIN_CLIENT_ID=...
LINKEDIN_CLIENT_SECRET=...
SOCIAL_TOKEN_ENCRYPTION_KEY=...

# Analytics (reuse Tianji)
TIANJI_ENABLED=true
TIANJI_WEBSITE_ID=...

---

10. Phased Implementation Plan

Phase 0 — Scaffolding & deploy skeleton

• Create monica_site Django 6.0 project via uv; settings/ dispatch; Dockerfile, compose (dev with db+redis), entrypoints, .gitea/workflows, pre-commit.
• Add core app (TimeStampedModel), health check, base template.
• Wire server-infra app_catalog entry + host ports + secrets; get a "hello world" deploying to adama+roslin behind NPM on a staging domain.
• Add Redis to 10.0.0.230; create Postgres DB/user.
• Exit: empty app deploys via CI to prod hosts, health check green.

Phase 1 — Public site + contact form + leads + UTM

• public: marketing pages (home/about/contact), django_recaptcha v3 contact form.
• contacts: Contact + ConsentRecord (default opt-in on submit, with clear notice).
• leads: Lead + statuses; form submit creates/updates Contact and a Lead.
• analytics: UTMVisit capture middleware + attribution on submit.
• accounts + dashboard: realtor login, portal shell, lead inbox, basic UTM report (Chart.js).
• Exit: realtor logs in, sees leads with source attribution; visitors submit safely.

Phase 2 — Mailing list + Email/SMS outreach + opt-out

• messaging: MessageTemplate, Campaign, Message, ProviderEvent; Celery + Beat online.
• Email via SMTP2GO with unsubscribe links; SMS via SMTP2GO SMS API.
• Opt-out views + SMS STOP webhook → Suppression; consent enforced on every send.
• Dashboard: contact list management, segment/select recipients, compose + schedule campaign, delivery status.
• Exit: realtor sends a real email + SMS campaign to consented contacts; opt-out works.

Phase 3 — Postcards

• messaging/providers/postcard/ interface + Lob adapter (+ Click2Mail adapter as the low-volume option); POSTCARD_PROVIDER switch.
• Postcard campaign flow: pick template/artwork, select recipients (address required + Lob address verification), send batch via worker, poll delivery status.
• Client prerequisites called out: Lob account, A2P 10DLC for SMS.
• Exit: realtor mails a postcard batch and sees delivery tracking.

Phase 4 — Social automation

• social: SocialAccount OAuth connect (Meta + LinkedIn), encrypted token storage.
• Connectors for Meta (FB Page + IG Business) and LinkedIn; SocialPost + SocialPostTarget compose/schedule; Beat dispatches, worker fans out with retries.
• Dashboard: composer with per-platform preview, calendar/scheduled view, post status.
• Kick off LinkedIn app review in Phase 0–1 given 2–4 week approval lead time.
• Exit: realtor schedules one post to all three platforms; it publishes and reports status.

Phase 5 — Hardening & polish

• Rate limiting/backoff per provider, dead-letter handling, admin dashboards for failures.
• Analytics polish (campaign ROI, source trends), email/SMS engagement events.
• Backups for Postgres, monitoring via existing Alloy/Tianji, docs + runbook.

flowchart LR
    P0["P0 Scaffold + deploy"] --> P1["P1 Public + leads + UTM"]
    P1 --> P2["P2 Mailing list + email/SMS"]
    P2 --> P3["P3 Postcards"]
    P1 --> P4["P4 Social"]
    P3 --> P5["P5 Harden"]
    P4 --> P5

---

11. Key Risks & Decisions

• Singleton scheduler: must run exactly one Beat + worker across the active/active fleet, or scheduled sends duplicate. Pin to one host; documented in deploy.
• LinkedIn approval latency (2–4 wks): start the app-review process at project kickoff.
• SMS compliance (A2P 10DLC/TCPA): client-side registration required before SMS launch; consent + STOP handling are built in but registration is a prerequisite.
• Postcard cost/volume: Lob's monthly tiers only pay off at volume; the provider abstraction lets us default to Lob's free tier and drop to Click2Mail (pay-per-piece) if volume stays low.
• Meta Graph versioning: budget quarterly maintenance for API version bumps.
• Single-tenant → multi-tenant later: every tenant-scoped model gets a nullable owner/realtor FK path reserved so a later migration adds isolation without a rewrite.

---

12. Cost Snapshot (why it beats PostcardMania)

• Hosting: runs on existing adama/roslin fleet — near-zero marginal infra (add Redis).
• Email/SMS: SMTP2GO usage-based (already an org account).
• Postcards: ~$0.35–0.77/4x6 at provider cost vs PostcardMania's bundled campaign packages; no forced campaign minimums.
• Social: $0 API fees (native), vs PostcardMania add-ons / aggregator per-profile fees.
• No PostcardMania platform/campaign markup; realtor pays provider cost + our hosting.