Serve Hesychia crawler files as static text instead of the app shell.

Exact locations return robots.txt, sitemap.xml, and llms.txt, or 404.
Closes #43.

Co-authored-by: Cursor <cursoragent@cursor.com>

IMPLEMENTATION.md

@@ -241,7 +241,6 @@ After Docker install, re-SSH so the `docker` group membership takes effect.
 | `abc_voice` | django-type compose (FastAPI) | all webservers | **beta only** | Same repo as `abc_worker`, compose `docker-compose.voice.yml`, join API on host port 8020. One uvicorn worker. |
 | `abc_fe` | node-static (Vite) | all webservers | **beta only** | built to `/var/www/<env>.abc.aimloperations/html`; no prod `host_apps` row |
 | `livekit` | django-type compose (LiveKit SFU) | all webservers | **beta only** | Docker like `abc_worker`. Host network `:7880`. **Do not NPM-balance** until Redis — pick one upstream (adama). Router DNAT UDP 3478 + 50000–60000 and TCP 7881 to that host. Secrets `~/Documents/secrets/livekit/livekit_beta.env` |
-| `disney_hotel_service` | django (docker) | all webservers | beta + prod | active/active behind NPM; no bundled Postgres. Alert worker (`db_worker`) **singleton on adama** only (`compose --profile worker`). Repo `DisneyHotel/disney_hotel_service` (local folder `disney_reservation_finder`). Reads `POSTGRES_*`, not `DATABASE_URL`. |
 
 Django apps use a **shared external Postgres** (via `DATABASE_URL` in each host's
 env file) so active/active replicas share one database. Beta and prod never share
@@ -252,8 +251,7 @@ a DB.
 - `app_catalog` (`group_vars/all.yml`) — how each app is built (repo, type, compose file, migrate cmd).
 - `host_apps` (`host_vars/<host>.yml`) — which app+env+port runs on that host.
   Optional `compose_profiles: [worker]` activates docker compose profiles on that
-  host only (used for `monica_site` / `college_craft` / `print_forge` dj-queue
+  host only (used for `monica_site` / `college_craft` / `print_forge` dj-queue singleton on adama).
-  and `disney_hotel_service` alert-worker singleton on adama).
   Optional `enabled: false` skips deploy (CI included) but keeps the row for ports
   and `stop.sh`. Omit the key (or `true`) to deploy as before. Do not use Jinja
   `rejectattr('enabled')` — missing key is an error, not "enabled".
@@ -284,7 +282,6 @@ future beta replica.
 | abc_fe (nginx) | **8085** | 8084 (*not deployed*) | all webservers |
 | dta_blog (nginx) | **8087** | **8086** | all webservers |
 | livekit | **7880** | — | all webservers (signaling). UDP 3478 + 50000–60000 + TCP 7881 via router DNAT to the NPM upstream host |
-| disney_hotel_service | **8024** | **8023** | all webservers. **8021/8022** are WordPress `relawfirm` — do not reuse |
 | SearxNG (LAN only) | — | **8088** | ai-server-4080 only (`searxng_stack`); not an NPM upstream |
 
 Host-local services on ai-server-4080 (not balanced by NPM):
@@ -327,9 +324,9 @@ again. `stop.sh` still matches disabled rows.
   to host `{{ apps_env_dir }}` → git checkout at ref → copy `.env` into checkout →
   `docker compose build` → `up -d` (with `COMPOSE_PROFILES` from optional
   `host_apps.compose_profiles`) → migrate (run once, shared DB).
-  **`monica_site` / `college_craft` / `print_forge` / `disney_hotel_service` worker:**
+  **`monica_site` / `college_craft` / `print_forge` worker:** adama `host_apps` sets
-  adama `host_apps` sets `compose_profiles: [worker]` so deploy starts the worker
+  `compose_profiles: [worker]` so deploy starts dj-queue with web. Other hosts
-  with web. Other hosts omit profiles (web only).
+  omit profiles (web only).
 - **node-static**: git checkout at ref → `npm ci` → `npm run build:<env>`
   (writes to the app's `webroot_pattern`, e.g. `/var/www/{env}.realpath.app/html`,
   `/var/www/{env}.blog.realpath.app/html`, or `/var/www/{env}.chat.aimloperations/html`).
@@ -369,7 +366,6 @@ point each domain at the backend(s):
 | abc_worker | `beta.abc.worker.aimloperations.com` (beta only; HTTP + WebSocket upgrade) | `adama:8018` + same on roslin / starbuck / apollo / ai-server-4080 |
 | abc_voice | `beta.abc.voice.aimloperations.com` (beta only). Balance this host. Do not balance LiveKit. | `adama:8020` + same on roslin / starbuck / apollo / ai-server-4080 |
 | livekit | `LIVEKIT_DOMAIN` (e.g. `livekit.aimloperations.com`) — NPM Proxy Host, WebSocket, **single upstream** | `adama:7880` (do not balance until Redis) |
-| disney_hotel_service | `disneyhotel.app` (+ www); `beta.disneyhotel.app` (beta) | `adama:8023` / `8024` + same on roslin / starbuck / apollo / ai-server-4080 |
 
 ### Required changes IN each app repo (owned separately)
 
@@ -416,45 +412,11 @@ do not).
 | abc_be | beta | `abc_be_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/abc_be_beta` |
 | abc_worker | beta | *(same `abc_be_beta`)* | worker uses `DB_*` pointing at `abc_be_beta` — do **not** create a second DB |
 | abc_voice | beta | *(same `abc_be_beta`)* | voice container uses the worker DB settings — do **not** create a second DB |
-| disney_hotel_service | prod | `disney_hotel_service` | `postgres://westfarn:<pw>@10.0.0.230:5432/disney_hotel_service` |
-| disney_hotel_service | beta | `disney_hotel_service_beta` | `postgres://westfarn:<pw>@10.0.0.230:5432/disney_hotel_service_beta` |
 
 Server prereqs on 10.0.0.230: create each DB + grant `westfarn`;
 `listen_addresses` covers LAN; `pg_hba.conf` allows `10.0.0.0/24`; firewall opens
 5432 to `10.0.0.0/24` only.
 
-`disney_hotel_service` secrets (control node, never git, mode `600`) — [#47](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/47):
-
-```text
-~/Documents/secrets/disney_hotel_service/disney_hotel_service_prod.env
-~/Documents/secrets/disney_hotel_service/disney_hotel_service_beta.env
-```
-
-Ansible copies those to `/opt/apps/env/disney_hotel_service_<env>.env` on each
-app host. Create them on the machine that runs `deploy.sh` (ai-server-4080).
-Unix dotenv: `KEY=value`, one per line, no `export`. This app reads `POSTGRES_*`
-(`config/settings.py` and `scripts/entrypoint.sh`). It ignores `DATABASE_URL`.
-
-Same keys in both files. These values differ:
-
-| | prod | beta |
-|--|------|------|
-| `POSTGRES_DB` | `disney_hotel_service` | `disney_hotel_service_beta` |
-| `WEB_PORT` | `8023` | `8024` |
-| `DJANGO_ALLOWED_HOSTS` | `disneyhotel.app,www.disneyhotel.app` | `beta.disneyhotel.app` |
-| `DJANGO_CSRF_TRUSTED_ORIGINS` | `https://disneyhotel.app,https://www.disneyhotel.app` | `https://beta.disneyhotel.app` |
-| `SITE_BASE_URL` | `https://disneyhotel.app` | `https://beta.disneyhotel.app` |
-
-Shared DB settings: `POSTGRES_USER=westfarn`, `POSTGRES_HOST=10.0.0.230`,
-`POSTGRES_PORT=5432`. `DJANGO_DEBUG=0`. Leave `RUN_MIGRATIONS=0` and
-`RUN_SEED=0` (deploy runs migrate once; seed parks by hand after the first
-deploy). Blank `STRIPE_SECRET_KEY` skips payment enforcement. Blank
-`SMTP2GO_API_KEY` logs email to the container console.
-
-App repo still needs `docker-compose.prod.yml` on `master` (no bundled `db`,
-`ports: ["${WEB_PORT}:8000"]`, `worker` under profile `worker`) before the
-first deploy succeeds.
-
 `url_shortening_service` extra env (control-node secrets, not in git):
 `SHORT_PUBLIC_HOSTS` / `SHORT_API_HOSTS` / `SHORT_ADMIN_HOSTS` (admin = `localhost,127.0.0.1` only),
 `SHORTENER_API_TOKENS` (`monica:<token>`), `SHORT_ALLOWED_HOSTS`, `CLICK_IP_PEPPER`
@@ -633,8 +595,7 @@ Store vault password for CI in a file readable only by the Act runner (e.g. `~/.
 | 10i | Register + deploy ABC beta (`abc_be` 8017, `abc_worker` 8018, `abc_fe` 8085) | Done ([#26](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/26)) |
 | 10j | Register + deploy `print_forge` (django, ports 8007/8019) | Done ([#27](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/27)) |
 | 10k | Register + deploy `dta_blog` (node-static, ports 8086/8087) | Done ([#29](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/29)) |
-| 10l | Register GIS LiveKit SFU (`abc_worker`-style docker on webservers, port 7880) | Done |
+| 10l | Register GIS LiveKit SFU (`abc_worker`-style docker on webservers, port 7880) | This PR |
-| 10m | Register + deploy `disney_hotel_service` (django, ports 8023/8024) | This PR ([#47](https://git.aimloperations.com/ai_ml_operations/server-infra/issues/47)) |
 | 11 | Gitea container registry (optional) | Future |
 
 ## Open Decisions

docs/WORDPRESS.md

@@ -56,9 +56,8 @@ Same host port on every app server for a given site and env. NPM balances
 `adama:PORT` + `roslin:PORT` + `starbuck:PORT` + `apollo:PORT` + `ai-server-4080:PORT`.
 
 These are not in the Ansible `host_apps` tables. Do not reuse a port from
-[IMPLEMENTATION.md](../IMPLEMENTATION.md) (8000–8020, **8023–8024**, and 8080–8088
+[IMPLEMENTATION.md](../IMPLEMENTATION.md) (8000–8020 and 8080–8088 are taken;
-are taken; **8020** is `abc_voice`, **8023/8024** are `disney_hotel_service`).
+**8020** is `abc_voice`).
-Next WordPress site starts at **8025**.
 
 | Site | prod port | beta port | prod database | beta database |
 |------|-----------|-----------|---------------|---------------|
@@ -336,7 +335,7 @@ The project name is already in the compose file, so `-p` is optional. The databa
 Repeat sections 2–8 with the beta column of the table:
 
 - database `<site>_beta`
-- port **8022** for relawfirm (next free WordPress port is **8025**; **8023/8024** are `disney_hotel_service`)
+- port **8022** for relawfirm (next free port for the next site)
 - its own env file, salts, NFS dataset, compose project, NPM host, and adama cron
 
 Do not point beta `WORDPRESS_DB_NAME` at the prod database.

inventory/group_vars/all.yml

@@ -119,25 +119,8 @@ app_catalog:
     # Document root served by nginx and written by `npm run build:<env>`.
     # {env} is replaced with the entry's env (beta/prod).
     webroot_pattern: "/var/www/{env}.chat.aimloperations/html"
-    # hesychia.ai Stripe/RevenueCat webhooks hit this vhost. Proxy the prefix
-    # to chat_backend so POST is not answered 405 by the SPA index.html.
-    api_backend: chat_backend
-    api_proxy_prefix: /api/finance/webhooks/
     # /robots.txt, /sitemap.xml, and /llms.txt must not fall through to index.html.
     exact_root_files: true
-    # Homepage was missing these. Enforcing CSP stays loose so Tianji, the API,
-    # websockets, and inline styles keep working. Report-Only is the strict policy.
-    security_headers: true
-    # Unknown URLs are HTTP 404 with public/404.html, not 200 + the SPA shell.
-    # Slash routes are directory index.html files from scripts/stampCanonicals.js.
-    # These extensionless routes are the matching files (keep in sync with
-    # llm-fe/src/llm-fe/utils/canonicals.js paths that do not end in /).
-    error_page_404: /404.html
-    html_exact_paths:
-      - /legal/terms-of-service
-      - /billing/success
-      - /billing/cancel
-      - /document_storage
   chat_backend:
     type: django
     repo: "{{ git_base_url }}/ai_ml_operations/chat_backend.git"
@@ -145,11 +128,6 @@ app_catalog:
     compose_file: docker-compose.prod.yml
     web_service: web
     migrate_cmd: "uv run python manage.py migrate --noinput"
-    # Host header the static-site webhook proxy sends. Must be in Django
-    # ALLOWED_HOSTS (see chat_backend DJANGO_ALLOWED_HOSTS).
-    public_hosts:
-      prod: chatbackend.aimloperations.com
-      beta: beta.chatbackend.aimloperations.com
   monica_site:
     type: django
     repo: "{{ git_base_url }}/ai_ml_operations/monica_site.git"
@@ -221,15 +199,6 @@ app_catalog:
     web_service: web
     # SFU — no Django migrations. `true` keeps the django deploy path happy.
     migrate_cmd: "true"
-  disney_hotel_service:
-    type: django
-    repo: "{{ git_base_url }}/DisneyHotel/disney_hotel_service.git"
-    default_branch: master
-    compose_file: docker-compose.prod.yml
-    web_service: web
-    migrate_cmd: "uv run python manage.py migrate --noinput"
-    # Alert worker (db_worker + enqueue_due_alerts) = compose profile `worker`
-    # (singleton). Deploy starts it via host_apps.compose_profiles on adama only.
 
 # Deploy filter vars. CI passes these; manual runs may leave them undefined
 # to (re)deploy every app listed in the host's host_apps.

inventory/host_vars/adama.yml

@@ -34,6 +34,3 @@ host_apps:
   - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }
-  # Alert worker singleton (db_worker). Other hosts omit compose_profiles.
-  - { name: disney_hotel_service, env: prod, port: 8023, compose_profiles: [worker] }
-  - { name: disney_hotel_service, env: beta, port: 8024, compose_profiles: [worker] }

inventory/host_vars/ai-server-4080.yml

@@ -36,7 +36,7 @@ host_apps:
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
-  # monica_site / college_craft / print_forge / disney_hotel_service workers NOT here — adama sets compose_profiles: [worker].
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
   - { name: monica_site, env: prod, port: 8004 }
   - { name: monica_site, env: beta, port: 8014 }
   - { name: url_shortening_service, env: prod, port: 8005 }
@@ -49,6 +49,4 @@ host_apps:
   - { name: abc_worker, env: beta, port: 8018 }
   - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
-  - { name: livekit, env: beta, port: 7880 }
+  - { name: livekit, env: beta, port: 7880 }
-  - { name: disney_hotel_service, env: prod, port: 8023 }
-  - { name: disney_hotel_service, env: beta, port: 8024 }

inventory/host_vars/apollo.yml

@@ -15,7 +15,7 @@ host_apps:
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
-  # monica_site / college_craft / print_forge / disney_hotel_service workers NOT here — adama sets compose_profiles: [worker].
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
   - { name: monica_site, env: prod, port: 8004 }
   - { name: monica_site, env: beta, port: 8014 }
   - { name: url_shortening_service, env: prod, port: 8005 }
@@ -29,5 +29,3 @@ host_apps:
   - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }
-  - { name: disney_hotel_service, env: prod, port: 8023 }
-  - { name: disney_hotel_service, env: beta, port: 8024 }

inventory/host_vars/roslin.yml

@@ -15,7 +15,7 @@ host_apps:
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
-  # monica_site / college_craft / print_forge / disney_hotel_service workers NOT here — adama sets compose_profiles: [worker].
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
   - { name: monica_site, env: prod, port: 8004 }
   - { name: monica_site, env: beta, port: 8014 }
   - { name: url_shortening_service, env: prod, port: 8005 }
@@ -29,5 +29,3 @@ host_apps:
   - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }
-  - { name: disney_hotel_service, env: prod, port: 8023 }
-  - { name: disney_hotel_service, env: beta, port: 8024 }

inventory/host_vars/starbuck.yml

@@ -15,7 +15,7 @@ host_apps:
   - { name: chat_web_app, env: beta, port: 8083 }
   - { name: chat_backend, env: prod, port: 8003 }
   - { name: chat_backend, env: beta, port: 8013 }
-  # monica_site / college_craft / print_forge / disney_hotel_service workers NOT here — adama sets compose_profiles: [worker].
+  # monica_site / college_craft / print_forge workers NOT here — adama sets compose_profiles: [worker].
   - { name: monica_site, env: prod, port: 8004 }
   - { name: monica_site, env: beta, port: 8014 }
   - { name: url_shortening_service, env: prod, port: 8005 }
@@ -29,5 +29,3 @@ host_apps:
   - { name: abc_voice, env: beta, port: 8020 }
   - { name: abc_fe, env: beta, port: 8085 }
   - { name: livekit, env: beta, port: 7880 }
-  - { name: disney_hotel_service, env: prod, port: 8023 }
-  - { name: disney_hotel_service, env: beta, port: 8024 }

roles/app-deploy/tasks/django.yml

@@ -104,9 +104,8 @@
     - _legacy_systemd_probe.stdout | default('') | length > 0
 
 # Optional host_apps.compose_profiles (e.g. [worker]) activates compose profiles
-# on this host only. Used for monica_site / college_craft / print_forge /
+# on this host only. Used for monica_site / college_craft / print_forge dj-queue singleton on
-# disney_hotel_service worker singleton on adama — without it, plain up
+# adama — without it, plain up --remove-orphans can drop a manually started worker.
-# --remove-orphans can drop a manually started worker.
 - name: "django[{{ _project }}] start containers"
   ansible.builtin.command:
     cmd: "docker compose -f {{ _spec.compose_file }} --env-file .env up -d --remove-orphans"

roles/web-static/templates/docker-compose.yml.j2

@@ -3,9 +3,6 @@ services:
   web-static:
     image: {{ web_static_image }}
     restart: unless-stopped
-    # Webhook locations proxy to chat_backend on the host-published port.
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
     logging:
       driver: json-file
       options:

roles/web-static/templates/nginx.conf.j2

@@ -17,14 +17,6 @@ log_format observability escape=json
     '"user_agent":"$http_user_agent"'
   '}';
 
-{% macro security_headers() %}
-    add_header X-Content-Type-Options "nosniff" always;
-    add_header X-Frame-Options "DENY" always;
-    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
-    add_header Content-Security-Policy "default-src 'self' https: wss: data: blob: 'unsafe-inline'; frame-ancestors 'none'; base-uri 'self'; object-src 'none'" always;
-    add_header Content-Security-Policy-Report-Only "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; font-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; base-uri 'self'; object-src 'none'; form-action 'self'" always;
-{% endmacro %}
-
 {% for a in _static_apps %}
 server {
     listen {{ a.port }};
@@ -37,37 +29,6 @@ server {
 
     root {{ (app_catalog[a.name].webroot_pattern | default(web_static_root ~ '/{env}_' ~ a.name)) | replace('{env}', a.env) }};
     index index.html;
-{% if app_catalog[a.name].security_headers | default(false) %}
-{{ security_headers() }}
-{% endif %}
-
-{%- set api_backend_name = app_catalog[a.name].api_backend | default('') %}
-{%- if api_backend_name %}
-{%- set api_proxy_prefix = app_catalog[a.name].api_proxy_prefix | default('/api/') %}
-{%- set api_match = namespace(port=0, ok=false) %}
-{%- for item in host_apps | default([]) %}
-{%- if item.name == api_backend_name and item.env == a.env and (item.enabled | default(true)) %}
-{%- set api_match.port = item.port %}
-{%- set api_match.ok = true %}
-{%- endif %}
-{%- endfor %}
-{%- if api_match.ok %}
-{%- set api_host = (app_catalog[api_backend_name].public_hosts | default({}))[a.env] | default('') %}
-
-    # Finance webhooks (Stripe) arrive on the public product host. The SPA
-    # try_files fallback is a static file and returns 405 for POST.
-    location {{ api_proxy_prefix }} {
-        proxy_pass http://host.docker.internal:{{ api_match.port }};
-        proxy_http_version 1.1;
-        proxy_set_header Host {{ api_host if api_host else '$host' }};
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_read_timeout 60s;
-        client_max_body_size 2m;
-    }
-{%- endif %}
-{%- endif %}
 
 {% if app_catalog[a.name].exact_root_files | default(false) %}
     # Crawlers must get these files, never the SPA index.html fallback.
@@ -85,26 +46,6 @@ server {
     }
 {% endif %}
 
-{% if app_catalog[a.name].html_exact_paths | default([]) %}
-{% for p in app_catalog[a.name].html_exact_paths %}
-    location = {{ p }} {
-        default_type text/html;
-        try_files {{ p }} =404;
-    }
-{% endfor %}
-{% endif %}
-
-{% if a.name == 'chat_web_app' and a.env == 'prod' %}
-    # HeyCatch short links.
-    # f Facebook, h Threads, i Instagram, l LinkedIn, r Reddit, x X, y YouTube.
-    # Relative Location: this vhost listens on the container port. An absolute
-    # redirect would publish that port (hesychia.ai:8082).
-    location ~ ^/([a-z0-9])/?$ {
-        absolute_redirect off;
-        return 302 /?utm_source=heycatch&utm_campaign=$1;
-    }
-{% endif %}
-
 {% if app_catalog[a.name].error_page_404 | default('') %}
     error_page 404 {{ app_catalog[a.name].error_page_404 }};
 
@@ -120,9 +61,6 @@ server {
     location ~* \.(?:js|css|woff2?|png|jpg|jpeg|gif|svg|ico)$ {
         expires 7d;
         add_header Cache-Control "public";
-{% if app_catalog[a.name].security_headers | default(false) %}
-{{ security_headers() }}
-{% endif %}
     }
 }
 {% endfor %}

scripts/deploy.sh

@@ -17,7 +17,7 @@ Deploy applications with deploy-apps.yml.
   HOST              Optional. Limit to one host: adama, roslin, starbuck, apollo, or ai-server-4080.
 
 Options:
-  --app NAME        App to deploy (company_site, dta_service, dta_webapp, dta_blog, scha, chat_web_app, chat_backend, monica_site, url_shortening_service, college_craft, print_forge, abc_be, abc_worker, abc_fe, livekit, disney_hotel_service)
+  --app NAME        App to deploy (company_site, dta_service, dta_webapp, dta_blog, scha, chat_web_app, chat_backend, monica_site, url_shortening_service, college_craft, print_forge, abc_be, abc_worker, abc_fe, livekit)
   --env ENV         Environment: beta or prod
   --ref REF         Git ref/sha to deploy (default: master)
   --check           Dry run