Commit Graph
11 Commits
Author SHA1 Message Date
westfarn d54094f5e0 Tier-gated RAG + Drive document sources (#42) (#54)
Unit Tests / test (push) Successful in 10s
Deploy Beta / unit-tests (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 40s
## Summary

Implements epic [#42](#42) (children #43–#53) and advances [#11](#11).

- **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag`
- **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")`
- **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE
- **Workspaces:** auto-create default company workspace; fail-closed scoping
- **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command
- **Docs/env:** README + `.env*.example` updated

Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85).

## Test plan

- [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK)
- [ ] Migrate finance `0004` + chat_backend `0028` on beta
- [ ] Verify Standard user: Documents API 403 + no RAG retrieval
- [ ] Verify Founders/Pro: upload + list + active toggle
- [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync
- [ ] Company manager: `link_company_drive`; non-manager 403
- [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54
2026-08-01 14:02:36 -07:00
westfarn eedc842b08 Add account self-delete and subscription lifecycle sync (#34) (#39)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 26s
Deploy Beta / deploy-beta (push) Successful in 6m46s
## Summary
- Closes [#34](#34)
- Companion for [chat_web_app#75](ai_ml_operations/chat_web_app#75) (portal cancel/change local sync)
- Soft-delete `DELETE /api/user/` for the authenticated user only: `deleted=True`, `is_active=False`, hide conversations, blacklist outstanding refresh tokens; staff self-delete rejected
- Stripe `customer.subscription.updated` / `deleted` webhooks sync plan status, `cancel_at_period_end`, and `current_period_end`; checkout assigns plan from `metadata.plan_slug`
- **UserAuthEvent audit**: `account_deleted`, `subscription_started` (first active plan), `subscription_updated` (plan/status/cancel changes) — visible on user admin
- Document FE contract in README (endpoint, response, post-delete logout)

## Test plan
- [ ] `uv run python manage.py test chat_backend.tests.test_views_users.CustomUserSelfDeleteTestCase finance.tests`
- [ ] Authenticated `DELETE /api/user/` soft-deletes self, hides conversations, blocks re-login, writes `account_deleted` auth event
- [ ] Checkout / Backer assign writes `subscription_started`; portal cancel/change writes `subscription_updated`
- [ ] Anonymous / staff self-delete rejected; body cannot target another user
- [ ] After portal cancel, webhook sets `cancel_at_period_end` / `canceled` on `GET /finance/subscription/`Reviewed-on: #39
2026-08-01 12:24:17 -07:00
westfarn 67f16565e9 Add Stripe Customer Portal session API for account billing (#35)
Deploy Beta / unit-tests (push) Successful in 9s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 26s
Deploy Beta / deploy-beta (push) Successful in 6m49s
## Summary
- Companion to [chat_web_app#33](ai_ml_operations/chat_web_app#33) (Account billing + Customer Portal)
- Follow-on from finance MVP [#21](#21): add authenticated `POST /api/finance/portal/` that creates a Stripe Billing Portal session and returns `portal_url`
- Resolve Stripe customer from the user's latest `Invoice.stripe_customer_id`; return `400` when missing (user must complete Checkout first)
- Document `STRIPE_PORTAL_RETURN_URL` (default `{FRONTEND_BASE_URL}/account/`) in settings + env examples

## Test plan
- [ ] `manage.py test finance.tests.test_portal finance.tests.test_checkout`
- [ ] Authenticated portal create with invoice that has `stripe_customer_id` → `201` + `portal_url`
- [ ] No customer / unpaid user → `400` with clear detail
- [ ] Missing `STRIPE_SECRET_KEY` → `503`
- [ ] Unauthenticated → `401`
- [ ] Custom `return_url` in body overrides default portal return URLReviewed-on: #35
2026-07-31 03:54:28 -07:00
westfarn 5d5b448868 Enable beta auto-deploy on master; manual prod button (#26) (#30)
Unit Tests / test (push) Successful in 9s
Deploy Beta / unit-tests (push) Successful in 10s
Deploy Beta / docker (push) Successful in 17s
Deploy Beta / deploy-beta (push) Successful in 1m3s
## Summary

Closes #26.

- Replace auto-prod `deploy.yml` with `deploy-beta.yml` (push to `master` → tests → `--env beta`) and `deploy-prod.yml` (`workflow_dispatch` only → `--env prod`), matching `dta_service` / `chat_web_app`
- Default `ALLOWED_HOSTS` / `CORS_ALLOWED_ORIGINS` include beta API + SPA hosts so beta frontend (and Tianji-instrumented pages) can call the API
- Expand `.env.prod.example` beta block (CORS, CSRF, OAuth callbacks, `FRONTEND_BASE_URL`, Tianji note)
- Harden `validate-env.sh` for beta secrets; README documents beta auto / prod button

## Tianji

Backend does not load `tracker.js`. FE already owns wiring (`chat_web_app#35`, closed).

- Beta SPA origin: `https://beta.chat.aimloperations.com`
- Beta Tianji website ID (FE `.env.beta`): `cms38bw671mf9n5jjw3xp1j3q`

## Coordination

- Infra: [server-infra#7](ai_ml_operations/server-infra#7) (`host_apps` beta `:8013`, `chat_backend_beta` DB, secret, NPM)
- FE companion: [chat_web_app#35](ai_ml_operations/chat_web_app#35)

## Test plan

- [ ] PR CI / unit tests green
- [ ] Merge to `master` triggers **Deploy Beta** only (not prod)
- [ ] Manual **Deploy Prod** `workflow_dispatch` still deploys `--env prod`
- [ ] After infra#7: beta container healthy on **8013** with `chat_backend_beta`
- [ ] Beta hosts / CSRF / CORS allow `https://beta.chat.aimloperations.com`Reviewed-on: #30
2026-07-27 08:35:07 -07:00
westfarn 30ce3d048d Gate self-serve registration behind ENABLE_ACCOUNT_REGISTRATION (#25)
Unit Tests / test (push) Successful in 9s
## Summary
- Add `ENABLE_ACCOUNT_REGISTRATION` (default `false`) for self-serve sign-up, set via env / control-node secret (`chat_backend_<env>.env`)
- Harden `POST /api/user/create/` to create company + user, return JWTs, and reject when the flag is off
- Expose `GET /api/public/settings/` with `enable_account_registration` for the SPA

Supports frontend: [chat_web_app#31](ai_ml_operations/chat_web_app#31)

## Test plan
- [ ] `cd llm_be && uv run python manage.py test chat_backend.tests.test_views_users`
- [ ] Confirm `POST /api/user/create/` returns 403 when flag unset/false
- [ ] With `ENABLE_ACCOUNT_REGISTRATION=true`, register returns access/refresh + creates company
- [ ] `GET /api/public/settings/` returns the flag
- [ ] Add `ENABLE_ACCOUNT_REGISTRATION=false` to control-node `chat_backend_prod.env` / `chat_backend_beta.env` before deploy (see `.env.prod.example`)Reviewed-on: #25
2026-07-27 04:19:59 -07:00
westfarn ad44359804 Add finance app with Stripe Checkout subscriptions (#21) (#23)
Unit Tests / test (push) Successful in 10s
## Summary
- Closes #21 — new Django `finance` app with Stripe as payment provider
- Subscription price defaults to **$10 USD / month** via `SUBSCRIPTION_PRICE_AMOUNT_CENTS = 1000` in `settings.py` (env-overridable)
- Persists **Invoice** and **Payment** rows; both registered in Django admin (with payment inline on invoices)
- Checkout Session API redirects users to Stripe hosted payment; webhook verifies signatures and upserts ledger idempotently

## API
- `POST /api/finance/checkout/` — JWT auth → `{ checkout_url, session_id }`
- `GET /api/finance/invoices/` / `GET /api/finance/payments/` — own records
- `POST /api/finance/webhooks/stripe/` — Stripe signature-verified webhook

## Config
Documented in `.env.example` / `.env.prod.example`:
`STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY`, `STRIPE_WEBHOOK_SECRET`, optional `STRIPE_PRICE_ID`, `FRONTEND_BASE_URL`

## Test plan
- [x] `uv run python manage.py test finance` (17 tests)
- [ ] Set Stripe test keys locally; create checkout session; complete payment in Stripe test mode
- [ ] Confirm Invoice/Payment appear in `/admin/`
- [ ] Point Stripe webhook to `/api/finance/webhooks/stripe/` and verify `checkout.session.completed` / `invoice.paid`Reviewed-on: #23
2026-07-26 17:35:06 -07:00
westfarn 9984d1c340 Name AI assistant Hesychia in system prompts (#22)
Unit Tests / test (push) Successful in 9s
## Summary
- Closes #20 — name the AI assistant **Hesychia** in system / generation prompts
- Add shared `assistant_identity.py` with `ASSISTANT_NAME` + concise calm/stillness tone
- Prepend identity to chat (`llm_service`), RAG, data analysis, and the views system message
- Document that identity lives in code (not env); add unit coverage

## Test plan
- [x] `uv run python manage.py test chat_backend.tests.test_assistant_identity chat_backend.tests.test_services_llm chat_backend.tests.test_services_data_analysis`
- [ ] Fresh chat: ask "who are you?" → responds as Hesychia
- [ ] Confirm classifiers/moderators/title generator unchanged (not assistant identity)

Related: companion frontend rebrand `chat_web_app#29`Reviewed-on: #22
2026-07-26 17:08:08 -07:00
westfarn 0525f9559b Add offline unit test suite for chat_backend (#12)
Unit Tests / test (push) Successful in 9s
Closes #5

## Summary

- Replaces the three scattered test modules (`chat_backend/tests.py`, `services/tests.py`, `services/prompt_classifier/tests.py`) with a `chat_backend/tests/` package: **242 deterministic tests plus 6 opt-in live-Ollama checks**, up from 10 tests (3 of which were skipped and 4 of which were never even discovered).
- The suite runs fully offline — no Ollama, Chroma, SMTP or network access. LangChain runnables are replaced by a small `FakeChain`, Chroma/embeddings are mocked, email uses Django's locmem backend, and blobs go through `DatabaseStorage`.
- New `llm_be/test_runner.py` (wired via `TEST_RUNNER`) sets `SKIP_RAG_INIT=1` and an MD5 password hasher, so the suite cannot accidentally reach a model server and finishes in ~5s on SQLite (~17s on Postgres) instead of ~30s.

## Coverage

| Area | File |
|------|------|
| `TimeInfoBase.save`, slugs, `get_duration`, `file_exists`, cascades | `test_models.py` |
| `DatabaseStorage` save/open/exists/size/listdir/delete/times | `test_storage.py` |
| JWT claim, prompt/user/feedback/document serializers | `test_serializers.py` |
| auth + token, invite, feedback, company users, set-password, TOS | `test_views_users.py` |
| conversation list/order/create/detail/soft-delete | `test_views_conversations.py` |
| all four analytics endpoints, including empty-month behaviour | `test_views_analytics.py` |
| workspace + document upload/list/detail, 404 and 400 paths | `test_views_documents.py` |
| prompt classifier rules/parsing, moderation fail-safe, title cleanup | `test_services_classifiers.py` |
| CSV/XLSX/DOCX/PDF analysis, plot generation, error payloads | `test_services_data_analysis.py` |
| loader selection, filename sanitising, ingest, temp-file cleanup, search filters | `test_services_rag.py` |
| history formatting and streaming | `test_services_llm.py` |
| document re-index on create/delete, `SKIP_RAG_INIT` guard | `test_signals.py` |
| consumer DB helpers, LangGraph nodes (moderation, classification, generation, search flags), websocket routes | `test_consumers.py` |

Live checks (non-deterministic, need a model server):

```bash
cd llm_be
RUN_LIVE_OLLAMA_TESTS=1 uv run python manage.py test chat_backend.tests.test_live_ollama
```

## Bugs the tests surfaced (fixed here)

1. **`ConversationDetailView.post` silently dropped every prompt.** `import datetime` shadowed `from datetime import datetime`, so `datetime.now()` raised `AttributeError` inside a bare `except` and the endpoint returned 200 without saving. Now uses `timezone.now()`.
2. **Prompt attachments never reached the LLM.** `get_conversation_file_async` (both consumers) did `sync_to_async(prompt.file.read)` — with `DatabaseStorage` the attribute access itself opens the blob, i.e. a DB query in async context, raising `SynchronousOnlyOperation` that was swallowed and returned `(None, None)`. The read now happens inside the thread.
3. **`DatabaseStorage._save` crashed on a str-backed `ContentFile`** (`TypeError: sequence item 0: expected a bytes-like object`); chunks are encoded when needed.
4. **`services/prompt_classifier/__init__.,py`** (note the comma) meant the directory was only an implicit namespace package, which is why its test module was never collected. Renamed, and its duplicate live-Ollama tests folded into `test_live_ollama.py`.

Known-broken paths deliberately left untested and unchanged: `reset_password` / `ResetUserPassword` reference an unimported `requests` plus undefined locals, and `DocumentDetailView.get` references an undefined `workspaces` on its success path. Worth a follow-up ticket.

## Test plan

- [x] `cd llm_be && uv run python manage.py test` → 248 tests, OK (6 skipped, all opt-in live)
- [x] Same suite against Postgres 16 (`DATABASE_URL=postgres://…`) → OK, matching the containerized run in `deploy.yml`
- [x] `uv run black` clean on all added files
- [ ] Gitea Actions **Unit Tests** + **CI** green on this PRReviewed-on: #12
2026-07-26 05:00:17 -07:00
westfarn d1660792ad Dockerize chat_backend + Ollama LAN + DB file storage (#6) (#7)
Unit Tests / test (push) Successful in 13s
## Summary

Implements [chat_backend#6](#6) Part A:

- **uv** packaging (`pyproject.toml` + `uv.lock`), Docker/compose (dev + prod), entrypoint/validate-env, Gitea unit-test + auto-deploy workflows (mirror `scha`)
- Env-driven Django settings (`DJANGO_*`, `DATABASE_URL`, CSRF/CORS)
- **`OLLAMA_BASE_URL`** wired through all Ollama/LangChain clients (prod → `http://10.0.0.128:11434`)
- **DatabaseStorage** — prompt/document file blobs in Postgres (`StoredFile`), not container FS; RAG materializes temp paths for loaders
- ASGI via `gunicorn` + `UvicornWorker` (HTTP + WebSockets)

Companion server-infra PR registers `app_catalog` / `host_apps` (port **8003**).

## Test plan

- [ ] `uv sync && cd llm_be && SKIP_RAG_INIT=1 uv run python manage.py test`
- [ ] `docker compose build && docker compose up` against bundled Postgres
- [ ] Confirm Ollama calls use `OLLAMA_BASE_URL` (not hardcoded localhost)
- [ ] Upload a document / prompt file → row in `chat_backend_storedfile`, no disk under `media/`
- [ ] After server-infra merge + secret/Postgres/NPM: deploy via `deploy.sh --app chat_backend --env prod`Reviewed-on: #7
2026-07-25 05:23:33 -07:00
westfarn 57695353d0 temp checkin 2025-03-07 12:23:00 -06:00
westfarn 058f961b0d Initial commit 2025-03-03 10:53:23 +00:00