Implement v1 URL shortener (Bearer API, public 302, landing, CI) (#2)

## Summary - Standalone Django 6 shortener: Bearer `/api/links/` (create/list/detail/disable) and public `GET /<code>` 302 - Host split, target-host allowlist, named rotatable tokens; `short_url` from `PUBLIC_SHORT_URL` - Landing page, DEBUG-only `/debug/` mint form, Django admin - Docker/compose (host **8005**), Gitea CI like monica_site (PR tests, beta on merge, prod button) - Caller contract in `API.md` Closes #1. Infra follow-up: [server-infra#22](ai_ml_operations/server-infra#22). ## Test plan - [ ] `cd site && uv run python manage.py test` - [ ] `docker compose up --build` → http://127.0.0.1:8005/ - [ ] `POST /api/links/` with `Bearer monica:dev-only-token` → 201 - [ ] `GET /<code>` → 302 to allowlisted https URL - [ ] No Bearer → 401; non-allowlisted host → 400 - [ ] `/debug/` only when `DEBUG=true` Reviewed-on: #2
2026-08-30 04:55:33 -07:00
parent 4baaa4b33c
commit 630b770c12
48 changed files with 3469 additions and 2 deletions
@@ -0,0 +1,504 @@
+# URL Shortening Service — Implementation
+
+Django 6 service in Docker, packaged with `uv`. First caller is `monica_site` (SMS campaign
+bodies and ad-hoc texts). A short public domain will be pointed at this service later;
+until then a subdomain (for example `go.mkdrealtor.com`) is enough.
+
+This document is the build spec. Implement in this repo. Do not fold the shortener into
+`monica_site`.
+
+---
+
+## 1. Goals & non-goals
+
+### Goals
+
+- Mint a short HTTPS URL that 302s to a long HTTPS URL.
+- Public `GET /<code>` so SMS recipients can tap the link with no token.
+- Create / list / disable **only** via a Bearer-authenticated API (own hostname; may be public DNS).
+- Named, rotatable tokens so `monica_site` can be revoked without rotating every caller.
+- Target-host allowlist so a stolen token cannot mint open redirects off the short domain.
+- Click counts for campaign reporting.
+- Same deploy shape as `monica_site`: `uv`, Docker, Postgres in prod, Gitea CI, `server-infra`.
+
+### Non-goals (v1)
+
+- Public web UI or Django admin exposed on the short domain.
+- Query-string tokens (`?token=`). Header only.
+- Per-recipient unique codes (who clicked). Add later if needed.
+- Custom vanity slugs (`/openhouse`). Random codes only in v1.
+- Multi-tenant SaaS, rate-limit dashboards, QR generation.
+
+---
+
+## 2. Security model (do this first)
+
+Two surfaces. Mixing them is the failure mode.
+
+| Surface | Who | Auth | Network |
+|---------|-----|------|---------|
+| `GET /<code>` | anyone with the SMS | **none** | public short hostname (NPM + TLS) |
+| `/api/links/` (all methods) | `monica_site` and other trusted callers | **Bearer** | own hostname — **may be public DNS**. Token is the lock. |
+
+### 2.1 Public redirect
+
+SMS recipients have no token. Do **not** put Bearer on `GET /<code>`. That path is the
+product.
+
+Use **302**, not 301. 301 is cached forever and later click counts (and disable) stop
+working.
+
+### 2.2 Authenticated API
+
+- `Authorization: Bearer <named-token>` on every `/api/` request.
+- If `SHORTENER_API_TOKENS` is empty, API returns **503**. Fail closed.
+- Constant-time compare (`hmac.compare_digest`) against every configured token.
+- Named tokens: `monica:<secret>`, `admin:<secret>`. Store the name on the `ShortLink`
+  row as `created_by_token`.
+- Generate secrets with `python -c "import secrets; print(secrets.token_urlsafe(32))"`.
+- Never put the token in the short URL, logs, or git. Do not reuse
+  `DJANGO_SECRET_KEY` or any `monica_site` webhook secret.
+- Do not log the `Authorization` header (Gunicorn/NPM access logs). Strip or disable it.
+
+### 2.3 Two public hostnames (Bearer is the lock on /api/)
+
+`/api/links/` may have its own public DNS record. Anyone who finds that URL still
+cannot mint links without a configured Bearer token. Empty `SHORTENER_API_TOKENS`
+→ **503**. Missing/wrong token → **401**. Do not add CORS `*` (server-to-server only).
+
+```
+Internet
+   │
+   ├─ NPM: SHORT_DOMAIN (mkd.to / go.mkdrealtor.com)
+   │    location ~*^/[a-z0-9]{4,8}$  →  gunicorn   (redirects only)
+   │    /api/*                       →  404 / drop
+   │    /admin/                      →  404 / drop
+   │
+   └─ NPM: API hostname (e.g. shortener.aimloperations.com)
+        /api/links/  →  gunicorn   (Bearer required)
+        /admin/      →  404 / drop
+```
+
+- Public NPM for `SHORT_DOMAIN` must not proxy `/api/` or `/admin/`.
+- Public NPM for the API hostname proxies `/api/` only. Do not expose `/admin/`.
+- `monica_site` calls `SHORTENER_BASE_URL` (the API hostname), never the short
+  hostname, to create links.
+- Django admin stays on `SHORT_ADMIN_HOSTS` (localhost). Not on the public API host.
+
+### 2.4 Target allowlist
+
+A valid Bearer still must not mint arbitrary redirects.
+
+On create:
+
+- `https` only (reject `http`, `javascript:`, `data:`, protocol-relative `//evil.com`).
+- Hostname must match `SHORT_ALLOWED_HOSTS` (exact or suffix, e.g. `mkdrealtor.com`).
+- Normalize / reject credentials in the URL (`https://user:pass@host`).
+
+### 2.5 Host routing inside the app
+
+Even if NPM is misconfigured, the Django process must refuse the wrong surface:
+
+- `request.get_host()` in `SHORT_PUBLIC_HOSTS` → redirect view only. `/api/` → 404.
+- `request.get_host()` in `SHORT_API_HOSTS` → `/api/` (still Bearer). May be public DNS.
+- `request.get_host()` in `SHORT_ADMIN_HOSTS` → `/admin/` (localhost only by default).
+- `/healthz/` allowed on both public hosts. No secrets in the body.
+
+---
+
+## 3. Architecture
+
+```
+Phone SMS:  https://mkd.to/a3k9
+                │
+                ▼
+         Nginx Proxy Manager (TLS)
+                │
+                ▼
+     url_shortening_service (this repo)
+     Docker: gunicorn + uv
+                │
+         Host header?
+           ├─ public short host  →  GET /<code> 302 + Click row
+           └─ API host (own DNS) →  /api/links/ (Bearer)
+                │
+                ▼
+         Postgres (own DATABASE_URL)
+```
+
+Caller (`monica_site`) is an HTTP client. It does not share this database.
+
+Until the purchased short domain exists:
+
+- Set `SHORT_DOMAIN=go.mkdrealtor.com` (or `localhost:8005` locally).
+- Codes do not change when DNS is swapped. Only `SHORT_DOMAIN` / `PUBLIC_SHORT_URL` change.
+
+---
+
+## 4. Stack
+
+Mirror `monica_site` so deploy muscle memory is the same.
+
+| Piece | Choice |
+|-------|--------|
+| Python | 3.12 |
+| Package manager | `uv` (`pyproject.toml` + committed `uv.lock`) |
+| Framework | Django 6 |
+| WSGI | gunicorn in prod/beta; `runserver` when `DJANGO_ENV=dev` |
+| Static | WhiteNoise (almost unused; keep for admin if enabled internally) |
+| DB | Postgres via `DATABASE_URL`; SQLite when unset (tests / local) |
+| Image | `python:3.12-slim` + copy `uv` from `ghcr.io/astral-sh/uv` |
+| Queue | none in v1 (no worker) |
+| Extra deps | none for codes (`secrets`); `psycopg[binary]`, `gunicorn`, `whitenoise` |
+
+No Redis, Celery, or frontend build.
+
+---
+
+## 5. Project layout
+
+```
+url_shortening_service/
+  Implementation.md          ← this file
+  README.md
+  pyproject.toml
+  uv.lock
+  Dockerfile
+  docker-compose.yml          # web + postgres, hot reload
+  docker-compose.prod.yml    # no bundled postgres; DATABASE_URL from env
+  .env.example
+  .env.prod.example
+  .gitignore
+  .gitea/workflows/ci.yml
+  scripts/docker-entrypoint.sh
+  site/
+    manage.py
+    shortener/               # Django project package
+      settings/
+        __init__.py
+        base.py
+        dev.py
+        beta.py
+        prod.py
+      urls.py
+      wsgi.py
+      asgi.py
+    links/                   # the app
+      models.py
+      views.py
+      api.py
+      auth.py
+      services.py
+      urls.py
+      tests.py
+      admin.py               # optional; only reachable on API host if enabled
+    core/
+      views.py               # healthz
+```
+
+Settings split by `DJANGO_ENV` like `monica_site`.
+
+---
+
+## 6. Data model
+
+### `ShortLink`
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `id` | UUID PK | |
+| `code` | `CharField(8)`, unique, indexed | 4–6 chars from alphabet below |
+| `target_url` | `URLField(2048)` | stored canonical https URL |
+| `title` | `CharField(200)`, blank | caller-supplied note |
+| `created_by_token` | `CharField(64)` | token **name** (`monica`), never the secret |
+| `external_ref` | `CharField(64)`, blank, indexed | optional caller id (campaign UUID) |
+| `is_active` | bool, default True | |
+| `expires_at` | datetime, nullable | |
+| `click_count` | positive int, default 0 | denormalized |
+| `created_at` / `updated_at` | timestamps | |
+
+Disable without deleting. Inactive or expired codes 404 (no redirect). Do not retarget an
+existing code to a new URL in v1 (prevents swapping a live SMS onto a new destination).
+
+### `Click`
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `id` | UUID PK | |
+| `link` | FK `ShortLink` | |
+| `occurred_at` | datetime | |
+| `ip_hash` | `CharField(64)`, blank | HMAC-SHA256 of remote IP with a server pepper; never store raw IP |
+| `user_agent` | `CharField(512)`, blank | truncated |
+| `referrer` | `CharField(1024)`, blank | |
+
+Increment `ShortLink.click_count` in the same request as inserting `Click` (or
+`F('click_count') + 1`). Do not block the 302 on analytics failure: log and still redirect
+if the click insert fails.
+
+### Code alphabet
+
+```
+23456789abcdefghjkmnpqrstuvwxyz
+```
+
+No `0/O`, `1/l/I`. Length 6 by default (`SHORT_CODE_LENGTH=6`). Collision: retry a small
+number of times, then 500.
+
+Do not accept caller-supplied codes in v1.
+
+---
+
+## 7. HTTP API (internal host only)
+
+All `/api/` routes require Bearer. JSON in/out. CSRF exempt (token auth, no cookies).
+
+Prefix: `/api/links/`
+
+### `POST /api/links/`
+
+Create.
+
+Request:
+
+```json
+{
+  "target_url": "https://mkdrealtor.com/listings/oak-st?utm_source=monica&utm_medium=sms",
+  "title": "Oak St listing",
+  "external_ref": "campaign-uuid-optional",
+  "expires_at": null
+}
+```
+
+Response `201`:
+
+```json
+{
+  "code": "a3k9xm",
+  "short_url": "https://go.mkdrealtor.com/a3k9xm",
+  "target_url": "https://mkdrealtor.com/listings/oak-st?utm_source=monica&utm_medium=sms",
+  "title": "Oak St listing",
+  "is_active": true,
+  "click_count": 0,
+  "created_at": "2026-08-30T10:00:00Z"
+}
+```
+
+`short_url` is built from `PUBLIC_SHORT_URL` (the public origin phones will hit), **not**
+from the internal `Host` header.
+
+Errors:
+
+- `400` invalid URL / host not allowlisted
+- `401` missing/wrong Bearer
+- `503` no tokens configured
+
+Idempotency (v1, keep simple): if the same `target_url` + `external_ref` + still-active
+link exists, return that row `200` instead of minting a duplicate. If `external_ref` is
+empty, always mint a new code.
+
+### `GET /api/links/`
+
+List. Query: `?external_ref=`, `?is_active=true`, pagination (`limit`/`offset`, cap 100).
+
+### `GET /api/links/<code>/`
+
+Detail including `click_count`.
+
+### `POST /api/links/<code>/disable/`
+
+Set `is_active=false`. Idempotent. `200`.
+
+No DELETE in v1. No PATCH of `target_url`.
+
+### Auth helper
+
+```python
+# links/auth.py
+# Parse Authorization: Bearer <token>
+# Split named tokens from SHORTENER_API_TOKENS (comma-separated name:secret)
+# compare_digest each secret
+# Return token name or None
+```
+
+Decorator / mixin on all API views. Wrong or missing → `401` with
+`WWW-Authenticate: Bearer`. Do not distinguish "unknown token" vs "malformed" in the body.
+
+---
+
+## 8. Public redirect
+
+`GET /<code>` on the public host.
+
+1. Lookup code (404 if missing, inactive, or expired).
+2. Insert `Click`, bump `click_count` (best-effort).
+3. `HttpResponseRedirect(target_url)` — **302**.
+
+Reject codes that do not match `^[a-z0-9]{4,8}$` with 404 (no extra work).
+
+`GET /` on the public host: landing page (“URL shortening service”, credit
+`aimloperations.com`). Does not advertise the API.
+
+`GET /debug/`: mint form, **DEBUG only**, never on `SHORT_PUBLIC_HOSTS`.
+
+`GET /healthz/`: `{"status": "ok"}`.
+
+---
+
+## 9. Environment
+
+### This service
+
+| Var | Purpose |
+|-----|---------|
+| `DJANGO_ENV` | `dev` / `beta` / `prod` |
+| `DJANGO_SECRET_KEY` | Django signing; **not** an API token |
+| `DJANGO_DEBUG` | false in prod |
+| `DJANGO_ALLOWED_HOSTS` | public short host **and** API hostname |
+| `DATABASE_URL` | Postgres; empty → SQLite |
+| `SHORT_DOMAIN` | public hostname phones use (`go.mkdrealtor.com` then `mkd.to`) |
+| `PUBLIC_SHORT_URL` | origin for minted URLs, e.g. `https://go.mkdrealtor.com` (no trailing slash) |
+| `SHORT_PUBLIC_HOSTS` | comma list; Host values that only serve redirects |
+| `SHORT_API_HOSTS` | comma list; Host values that serve `/api/` (public API DNS and/or docker name) |
+| `SHORT_ADMIN_HOSTS` | comma list; Host values that serve `/admin/` (default localhost only) |
+| `SHORTENER_API_TOKENS` | `name:secret,name:secret` — required for API |
+| `SHORT_ALLOWED_HOSTS` | allowlist for `target_url` hosts |
+| `SHORT_CODE_LENGTH` | default `6` |
+| `CLICK_IP_PEPPER` | HMAC key for `ip_hash`; distinct from `DJANGO_SECRET_KEY` |
+| `GUNICORN_WORKERS` | default `2` |
+
+`.env.example` documents all of these. Prod secrets live in
+`~/Documents/secrets/url_shortening_service/` on the control node, same convention as
+`monica_site`.
+
+### Caller (`monica_site`) — later, other repo
+
+```text
+SHORTENER_BASE_URL=https://shortener.aimloperations.com
+SHORTENER_API_TOKEN=monica:<same-secret>
+```
+
+`monica_site` sends `Authorization: Bearer monica:<secret>` to
+`POST {SHORTENER_BASE_URL}/api/links/` and substitutes `short_url` into SMS bodies
+**after** UTM is attached.
+
+Do not implement the caller in this repo. Document the contract only.
+
+---
+
+## 10. Docker
+
+### `Dockerfile`
+
+Same pattern as `monica_site`:
+
+- `FROM python:3.12-slim`
+- install `libpq5`, copy `uv`
+- `uv sync --frozen --no-dev`
+- copy `site/` + entrypoint
+- `EXPOSE 8000`
+
+### `docker-compose.yml` (dev)
+
+- `db`: Postgres 16
+- `web`: build `.`, mount `./site`, `DJANGO_ENV=dev` → runserver
+- **Do not** publish the host port to the LAN unless you are testing redirects.
+  Prefer `127.0.0.1:8005:8000` (container still listens on 8000).
+- Internal API hostname: service name `web` (or `url-shortener`).
+
+### `docker-compose.prod.yml`
+
+- `web` only. External Postgres via `DATABASE_URL`.
+- `ports: "${WEB_PORT:-8005}:8000"` (pick a free host port; document it).
+- `env_file: .env`
+- No worker profile.
+
+### Entrypoint
+
+Wait for DB → `migrate --noinput` → if `DJANGO_ENV=dev` then `runserver`, else
+`collectstatic` + gunicorn.
+
+---
+
+## 11. NPM / deploy notes (ops, not code)
+
+When the domains exist:
+
+1. Public NPM proxy host = `SHORT_DOMAIN` → this container’s `WEB_PORT`.
+2. Proxy `/` (landing) and the code regex. `/api/`, `/admin/`, `/debug/` → 404.
+3. TLS like other apps.
+4. Add `SHORT_DOMAIN` to `DJANGO_ALLOWED_HOSTS` and `SHORT_PUBLIC_HOSTS`.
+5. `PUBLIC_SHORT_URL=https://<that-domain>`.
+6. Second NPM proxy host = API hostname → same container. Proxy `/api/` only.
+7. Add the API hostname to `DJANGO_ALLOWED_HOSTS` and `SHORT_API_HOSTS`.
+
+CI (same split as `monica_site`, default branch `master`):
+
+- `.gitea/workflows/ci.yml` — unit tests on pull requests
+- `.gitea/workflows/deploy-beta.yml` — on push to `master`: unit tests → compose
+  tests → `server-infra/scripts/deploy.sh --app url_shortening_service --env beta`
+- `.gitea/workflows/deploy-prod.yml` — `workflow_dispatch` (Actions button): same
+  tests, then `--env prod`
+
+---
+
+## 12. Tests (required before merge)
+
+Django `TestCase` / `SimpleTestCase`. No live network.
+
+- Auth: missing Bearer → 401; wrong token → 401; empty `SHORTENER_API_TOKENS` → 503;
+  matching named token → 201.
+- Host split: request to short host `/api/links/` → 404 even with valid Bearer.
+  Public API host without Bearer → 401; valid Bearer → 201. `/admin/` 404 on API host.
+- Allowlist: `https://mkdrealtor.com/x` ok; `http://…` 400; `https://evil.com` 400;
+  `javascript:alert(1)` 400.
+- Redirect: active code → 302 to target; inactive/expired/unknown → 404.
+- Redirect is 302, not 301.
+- Click row created and `click_count` incremented.
+- Idempotent create with same `target_url` + `external_ref`.
+- `short_url` uses `PUBLIC_SHORT_URL`, not the API Host header.
+- Code charset: generated codes only use the unambiguous alphabet.
+
+---
+
+## 13. Implementation order
+
+1. `uv init` / `pyproject.toml` + Django project + settings/env helpers.
+2. `links` models + migration.
+3. `links/auth.py` + API views (create/list/detail/disable).
+4. Redirect view + host-split middleware.
+5. Dockerfile, compose, entrypoint, `.env.example`.
+6. Tests in §12.
+7. README: how to run locally, how a caller authenticates, what is public vs internal.
+8. Stop. Caller integration lives in `monica_site` after this service is up.
+
+Phase 2 (not this pass): SMS auto-shorten in `monica_site` composer, custom slugs,
+per-recipient codes, internal-only admin UI.
+
+---
+
+## 14. Local run (target)
+
+```bash
+cp .env.example .env
+# set SHORTENER_API_TOKENS=monica:dev-only-token
+docker compose up --build
+# or:
+uv sync
+cd site && uv run python manage.py migrate
+uv run python manage.py runserver
+```
+
+Mint (API host / localhost):
+
+```bash
+curl -sS -X POST http://127.0.0.1:8005/api/links/ \
+  -H "Authorization: Bearer monica:dev-only-token" \
+  -H "Content-Type: application/json" \
+  -d '{"target_url":"https://mkdrealtor.com/","title":"test"}'
+```
+
+Follow (public path):
+
+```bash
+curl -sSI http://127.0.0.1:8005/<code>
+```