Serve Hesychia crawler files as static text #44

2026-10-17T05:19:51-07:00

westfarn commented

2026-10-17 05:19:51 -07:00

Summary

chat_web_app nginx now has exact locations for /robots.txt, /sitemap.xml, and /llms.txt.
Each location serves the static file or returns 404. It does not fall through to index.html.
Other static apps are unchanged. The flag is exact_root_files on chat_web_app only.
Merged current master, so the Stripe webhook proxy stays in place next to the new locations.
Closes #43.

The file contents ship in the chat_web_app public-site PR.

Test plan

Render the web-static nginx template and confirm only chat_web_app gets the three exact locations
Webhook location /api/finance/webhooks/ is still present
After deploy, curl -I https://hesychia.ai/robots.txt is text/plain, and a missing file would be 404 rather than HTML

## Summary - `chat_web_app` nginx now has exact locations for `/robots.txt`, `/sitemap.xml`, and `/llms.txt`. - Each location serves the static file or returns 404. It does not fall through to `index.html`. - Other static apps are unchanged. The flag is `exact_root_files` on `chat_web_app` only. - Merged current `master`, so the Stripe webhook proxy stays in place next to the new locations. - Closes #43. The file contents ship in the chat_web_app public-site PR. ## Test plan - [ ] Render the web-static nginx template and confirm only `chat_web_app` gets the three exact locations - [ ] Webhook location `/api/finance/webhooks/` is still present - [ ] After deploy, `curl -I https://hesychia.ai/robots.txt` is `text/plain`, and a missing file would be 404 rather than HTML

westfarn added 1 commit 2026-10-17 05:19:51 -07:00

Serve Hesychia crawler files as static text instead of the app shell. dce0879e62

Exact locations return robots.txt, sitemap.xml, and llms.txt, or 404.
Closes #43.

Co-authored-by: Cursor <cursoragent@cursor.com>

westfarn added 1 commit 2026-10-17 05:20:49 -07:00

Merge master and keep both the webhook proxy and crawler file locations. 3174c34a6f

Co-authored-by: Cursor <cursoragent@cursor.com>

westfarn referenced this pull request

2026-10-19 07:57:24 -07:00

Serve Hesychia pages with 404s and security headers #45

westfarn referenced this issue from a commit

2026-10-19 08:02:52 -07:00

Serve Hesychia pages with 404s and security headers (#45)

westfarn added 1 commit 2026-10-19 08:02:52 -07:00

Serve Hesychia pages with 404s and security headers (#45 )

Sync runner checkout / sync (pull_request) Successful in 7s

Details

5e9878632e

## Summary
- Unknown Hesychia URLs return 404 with `404.html` instead of 200 and the app shell.
- Extensionless routes (`/legal/terms-of-service`, billing return paths, `/document_storage`) are served as HTML.
- `chat_web_app` sends the security headers. The enforcing CSP stays loose enough for Tianji, the API, and the new analytics hosts. A stricter policy is report-only.
- Production only: one-character paths redirect for HeyCatch short links.

Depends on #44. Merge that first, then retarget this to `master` if Gitea does not do it automatically.

## Test plan
- [ ] After #44 and this change are deployed, `curl -sI https://hesychia.ai/not-a-page` is 404.
- [ ] `curl -sI https://hesychia.ai/pricing/` is 200 once the frontend shells are published.
- [ ] Response headers include `Content-Security-Policy`, `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, and `Referrer-Policy`.
- [ ] `curl -sI https://hesychia.ai/x` is a 302 to `/?utm_source=heycatch&utm_campaign=x`.

Reviewed-on: #45

westfarn merged commit dc08229f21 into master

2026-10-19 08:07:47 -07:00

westfarn deleted branch feat/43-static-crawler-files

2026-10-19 08:07:47 -07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ai_ml_operations/server-infra#44