Add robots.txt, sitemap.xml, and llms.txt endpoints so crawlers and AI agents can discover public pages
Fix accessibility tree issues: contact form labels, semantic nav/profile dropdown buttons, cookie consent dialog ARIA, and dead href="#" links
Reduce layout shift on homepage/contact via hero min-heights, trusted-by marquee sizing, and fixed cookie banner
Add regression tests and docs/agentic-browsing.md for Lighthouse agentic-browsing audits
Closes#5. Also addresses overlap with #3 (sitemap + robots.txt).
Test plan
python manage.py test public.tests (14 tests pass)
Verify GET /robots.txt, /sitemap.xml, /llms.txt return 200 in staging/prod
Confirm contact form labels visible and submittable on /contact
Run Lighthouse --only-categories=agentic-browsing on homepage and contact page
Smoke test mobile nav dropdown and cookie consent banner
## Summary
- Add `robots.txt`, `sitemap.xml`, and `llms.txt` endpoints so crawlers and AI agents can discover public pages
- Fix accessibility tree issues: contact form labels, semantic nav/profile dropdown buttons, cookie consent dialog ARIA, and dead `href="#"` links
- Reduce layout shift on homepage/contact via hero min-heights, trusted-by marquee sizing, and fixed cookie banner
- Add regression tests and `docs/agentic-browsing.md` for Lighthouse agentic-browsing audits
Closes #5. Also addresses overlap with #3 (sitemap + robots.txt).
## Test plan
- [x] `python manage.py test public.tests` (14 tests pass)
- [ ] Verify `GET /robots.txt`, `/sitemap.xml`, `/llms.txt` return 200 in staging/prod
- [ ] Confirm contact form labels visible and submittable on `/contact`
- [ ] Run Lighthouse `--only-categories=agentic-browsing` on homepage and contact page
- [ ] Smoke test mobile nav dropdown and cookie consent banner
Add robots.txt, sitemap.xml, and llms.txt for crawlers and AI agents, fix accessibility tree gaps in forms and navigation, reduce layout shift on key pages, and document Lighthouse regression checks.
Co-authored-by: Cursor <cursoragent@cursor.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
robots.txt,sitemap.xml, andllms.txtendpoints so crawlers and AI agents can discover public pageshref="#"linksdocs/agentic-browsing.mdfor Lighthouse agentic-browsing auditsCloses #5. Also addresses overlap with #3 (sitemap + robots.txt).
Test plan
python manage.py test public.tests(14 tests pass)GET /robots.txt,/sitemap.xml,/llms.txtreturn 200 in staging/prod/contact--only-categories=agentic-browsingon homepage and contact page