093e5462a5ef6669eb0b3cd25af9b0e9f2ab44b0
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 westfarn
|
e1e086a474 |
Monetization app + RevenueCat webhooks (store IAP ledger) (#69)
## Summary - Rename `finance` → **`monetization`** Django app (keep `finance_*` tables via `label = "finance"`) - Add `services/stripe.py` + `services/revenuecat.py`; RevenueCat webhook upserts **subscription + Invoice/Payment** (billing history parity with Stripe) - Mount `/api/monetization/` + keep `/api/finance/` alias - Extend `Source`/`Provider` with `revenuecat`; product→plan mapping via `revenuecat_product_id` / `REVENUECAT_PRODUCT_PLAN_MAP` Closes #68. Companion to [chat_web_app#100](ai_ml_operations/chat_web_app#100). ## Test plan - [x] `manage.py test monetization.tests` (54 OK) - [x] Smoke `chat_backend.tests.test_views_documents` + `test_oauth` - [ ] Deploy: set `REVENUECAT_WEBHOOK_SECRET`; point RC webhook at `/api/finance/webhooks/revenuecat/` - [ ] Map store product IDs on `SubscriptionPlan.revenuecat_product_id` (or env JSON map) - [ ] Sandbox INITIAL_PURCHASE → subscription `source=revenuecat` + invoice in `/finance/invoices/`Reviewed-on: #69 |
||
|
westfarn
|
d8f5b8ebf2 |
Always-on grounded retrieval + role-scoped Ollama models (#62 Phases 1–3) (#65)
## Summary - Closes Phases 1–3 of [#62](#62) (Phase 4 eval harness left for a follow-up). - **Accuracy:** Retrieval is decided every turn (`GroundingDecider`, fails open). `FAST` no longer skips search — it only selects `OLLAMA_MODEL_FAST`. Search failures surface an explicit error instead of hallucinating from parametric memory. - **Search:** Pluggable `services/search/` with **SearxNG primary** + DDGS failover, ranking/dedupe/rumour filtering, numbered dated source blocks, citations persisted on `Prompt.citations` and emitted as `{"v":1,"type":"citations",...}` after stream end. - **Models:** Role-scoped `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` / `OLLAMA_EMBED_MODEL=nomic-embed-text`, configurable `num_ctx`, real model name on `PromptMetric`, `reindex_embeddings` management command + loud embedding-dimension mismatch. ## SearxNG (ops) See README **SearxNG** section. Short version: run `searxng/searxng` on the GPU host, enable `json` in `settings.yml`, set `SEARXNG_BASE_URL=http://10.0.0.128:8080` in prod/beta secrets, open `:8080` on the LAN firewall like Ollama. ## Test plan - [x] `SKIP_RAG_INIT=1 python manage.py test chat_backend.tests` — 442 OK (6 skipped) - [ ] Deploy beta with updated secrets (`OLLAMA_MODEL_*`, `OLLAMA_EMBED_MODEL=nomic-embed-text`, `SEARXNG_BASE_URL`) - [ ] After embed change: `python manage.py reindex_embeddings` - [ ] Verify `did Taylor Swift get married` in FAST and THINKING returns grounded answer with citations frame - [ ] Kill SearxNG and confirm factual turns return search_unavailable (not Joe Alwyn hallucination); non-factual chat still worksReviewed-on: #65 |
||
|
westfarn
|
fd12bb972d |
Async Drive sync via Django tasks (#57) (#58)
## Summary - Closes [#57](#57) - Companion FE: [chat_web_app#90](ai_ml_operations/chat_web_app#90) / [PR #91](ai_ml_operations/chat_web_app#91) - `POST /api/drive/connections/<id>/sync/` enqueues via Django 6 Tasks (`drive_tasks.enqueue_drive_sync`) and returns **202** with `last_sync_status=pending` - Webhooks + `manage.py sync_drive_connections` use the same enqueue path (`--sync-now` for inline/cron) - With default `ImmediateBackend`, sync still runs in-process but on a **daemon thread** so HTTP returns quickly; swap `TASKS` later for a durable worker - Duplicate syncs while already `pending` are skipped (unless `force=True`) ## Test plan - [x] `manage.py test chat_backend.tests.test_drive_tasks chat_backend.tests.test_views_drive chat_backend.tests.test_management_drive_sync` - [ ] Manual: Sync now returns fast; connection flips pending → ok/error - [ ] Manual: Google Drive API disabled → `last_sync_status=error` + message in `last_sync_error` - [ ] Manual: second Sync while pending does not stack jobsReviewed-on: #58 |
||
|
westfarn
|
7025dab857 |
Personal Drive/RAG without a company (#55) (#56)
## Summary - Closes [#55](#55) (related [#46](#46)) - **Personal** Google Drive / OneDrive works for users **without** a company (personal RAG) - **Company** Drive still requires company + manager - Schema: nullable `DriveConnection.company`, personal `DocumentWorkspace.user`, ownership check constraints + conditional uniques - Runtime: `ensure_personal_workspace`, personal sync → personal WS, chat/document APIs fall back to personal WS when `company_id` is null - Supersedes the interim "reject with `no_company`" approach (wrong for personal connect) ## Test plan - [x] OAuth: personal Drive callback with `user.company=NULL` succeeds (`company_id=NULL` on connection) - [x] OAuth: company Drive still returns `no_company` / `forbidden` appropriately - [x] Chat tenant scope creates personal workspace for solo users - [x] Drive sync + document view suites (`91` related tests) - [ ] Manual: solo entitled user connects Google Drive → success, sync lands in personal workspace - [ ] Manual: company manager company Drive still works - [ ] Migrate prod/staging with `0029_personal_drive_rag_without_company`Reviewed-on: #56 |
||
|
westfarn
|
d54094f5e0 |
Tier-gated RAG + Drive document sources (#42) (#54)
## Summary Implements epic [#42](#42) (children #43–#53) and advances [#11](#11). - **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag` - **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")` - **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE - **Workspaces:** auto-create default company workspace; fail-closed scoping - **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command - **Docs/env:** README + `.env*.example` updated Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85). ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK) - [ ] Migrate finance `0004` + chat_backend `0028` on beta - [ ] Verify Standard user: Documents API 403 + no RAG retrieval - [ ] Verify Founders/Pro: upload + list + active toggle - [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync - [ ] Company manager: `link_company_drive`; non-manager 403 - [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54 |
||
|
westfarn
|
eedc842b08 |
Add account self-delete and subscription lifecycle sync (#34) (#39)
## Summary - Closes [#34](#34) - Companion for [chat_web_app#75](ai_ml_operations/chat_web_app#75) (portal cancel/change local sync) - Soft-delete `DELETE /api/user/` for the authenticated user only: `deleted=True`, `is_active=False`, hide conversations, blacklist outstanding refresh tokens; staff self-delete rejected - Stripe `customer.subscription.updated` / `deleted` webhooks sync plan status, `cancel_at_period_end`, and `current_period_end`; checkout assigns plan from `metadata.plan_slug` - **UserAuthEvent audit**: `account_deleted`, `subscription_started` (first active plan), `subscription_updated` (plan/status/cancel changes) — visible on user admin - Document FE contract in README (endpoint, response, post-delete logout) ## Test plan - [ ] `uv run python manage.py test chat_backend.tests.test_views_users.CustomUserSelfDeleteTestCase finance.tests` - [ ] Authenticated `DELETE /api/user/` soft-deletes self, hides conversations, blocks re-login, writes `account_deleted` auth event - [ ] Checkout / Backer assign writes `subscription_started`; portal cancel/change writes `subscription_updated` - [ ] Anonymous / staff self-delete rejected; body cannot target another user - [ ] After portal cancel, webhook sets `cancel_at_period_end` / `canceled` on `GET /finance/subscription/`Reviewed-on: #39 |
||
|
westfarn
|
67f16565e9 |
Add Stripe Customer Portal session API for account billing (#35)
## Summary - Companion to [chat_web_app#33](ai_ml_operations/chat_web_app#33) (Account billing + Customer Portal) - Follow-on from finance MVP [#21](#21): add authenticated `POST /api/finance/portal/` that creates a Stripe Billing Portal session and returns `portal_url` - Resolve Stripe customer from the user's latest `Invoice.stripe_customer_id`; return `400` when missing (user must complete Checkout first) - Document `STRIPE_PORTAL_RETURN_URL` (default `{FRONTEND_BASE_URL}/account/`) in settings + env examples ## Test plan - [ ] `manage.py test finance.tests.test_portal finance.tests.test_checkout` - [ ] Authenticated portal create with invoice that has `stripe_customer_id` → `201` + `portal_url` - [ ] No customer / unpaid user → `400` with clear detail - [ ] Missing `STRIPE_SECRET_KEY` → `503` - [ ] Unauthenticated → `401` - [ ] Custom `return_url` in body overrides default portal return URLReviewed-on: #35 |
||
|
westfarn
|
5d5b448868 |
Enable beta auto-deploy on master; manual prod button (#26) (#30)
## Summary Closes #26. - Replace auto-prod `deploy.yml` with `deploy-beta.yml` (push to `master` → tests → `--env beta`) and `deploy-prod.yml` (`workflow_dispatch` only → `--env prod`), matching `dta_service` / `chat_web_app` - Default `ALLOWED_HOSTS` / `CORS_ALLOWED_ORIGINS` include beta API + SPA hosts so beta frontend (and Tianji-instrumented pages) can call the API - Expand `.env.prod.example` beta block (CORS, CSRF, OAuth callbacks, `FRONTEND_BASE_URL`, Tianji note) - Harden `validate-env.sh` for beta secrets; README documents beta auto / prod button ## Tianji Backend does not load `tracker.js`. FE already owns wiring (`chat_web_app#35`, closed). - Beta SPA origin: `https://beta.chat.aimloperations.com` - Beta Tianji website ID (FE `.env.beta`): `cms38bw671mf9n5jjw3xp1j3q` ## Coordination - Infra: [server-infra#7](ai_ml_operations/server-infra#7) (`host_apps` beta `:8013`, `chat_backend_beta` DB, secret, NPM) - FE companion: [chat_web_app#35](ai_ml_operations/chat_web_app#35) ## Test plan - [ ] PR CI / unit tests green - [ ] Merge to `master` triggers **Deploy Beta** only (not prod) - [ ] Manual **Deploy Prod** `workflow_dispatch` still deploys `--env prod` - [ ] After infra#7: beta container healthy on **8013** with `chat_backend_beta` - [ ] Beta hosts / CSRF / CORS allow `https://beta.chat.aimloperations.com`Reviewed-on: #30 |
||
|
westfarn
|
30ce3d048d |
Gate self-serve registration behind ENABLE_ACCOUNT_REGISTRATION (#25)
Unit Tests / test (push) Successful in 9s
## Summary - Add `ENABLE_ACCOUNT_REGISTRATION` (default `false`) for self-serve sign-up, set via env / control-node secret (`chat_backend_<env>.env`) - Harden `POST /api/user/create/` to create company + user, return JWTs, and reject when the flag is off - Expose `GET /api/public/settings/` with `enable_account_registration` for the SPA Supports frontend: [chat_web_app#31](ai_ml_operations/chat_web_app#31) ## Test plan - [ ] `cd llm_be && uv run python manage.py test chat_backend.tests.test_views_users` - [ ] Confirm `POST /api/user/create/` returns 403 when flag unset/false - [ ] With `ENABLE_ACCOUNT_REGISTRATION=true`, register returns access/refresh + creates company - [ ] `GET /api/public/settings/` returns the flag - [ ] Add `ENABLE_ACCOUNT_REGISTRATION=false` to control-node `chat_backend_prod.env` / `chat_backend_beta.env` before deploy (see `.env.prod.example`)Reviewed-on: #25 |
||
|
westfarn
|
ad44359804 |
Add finance app with Stripe Checkout subscriptions (#21) (#23)
Unit Tests / test (push) Successful in 10s
## Summary - Closes #21 — new Django `finance` app with Stripe as payment provider - Subscription price defaults to **$10 USD / month** via `SUBSCRIPTION_PRICE_AMOUNT_CENTS = 1000` in `settings.py` (env-overridable) - Persists **Invoice** and **Payment** rows; both registered in Django admin (with payment inline on invoices) - Checkout Session API redirects users to Stripe hosted payment; webhook verifies signatures and upserts ledger idempotently ## API - `POST /api/finance/checkout/` — JWT auth → `{ checkout_url, session_id }` - `GET /api/finance/invoices/` / `GET /api/finance/payments/` — own records - `POST /api/finance/webhooks/stripe/` — Stripe signature-verified webhook ## Config Documented in `.env.example` / `.env.prod.example`: `STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY`, `STRIPE_WEBHOOK_SECRET`, optional `STRIPE_PRICE_ID`, `FRONTEND_BASE_URL` ## Test plan - [x] `uv run python manage.py test finance` (17 tests) - [ ] Set Stripe test keys locally; create checkout session; complete payment in Stripe test mode - [ ] Confirm Invoice/Payment appear in `/admin/` - [ ] Point Stripe webhook to `/api/finance/webhooks/stripe/` and verify `checkout.session.completed` / `invoice.paid`Reviewed-on: #23 |
||
|
westfarn
|
9984d1c340 |
Name AI assistant Hesychia in system prompts (#22)
Unit Tests / test (push) Successful in 9s
## Summary - Closes #20 — name the AI assistant **Hesychia** in system / generation prompts - Add shared `assistant_identity.py` with `ASSISTANT_NAME` + concise calm/stillness tone - Prepend identity to chat (`llm_service`), RAG, data analysis, and the views system message - Document that identity lives in code (not env); add unit coverage ## Test plan - [x] `uv run python manage.py test chat_backend.tests.test_assistant_identity chat_backend.tests.test_services_llm chat_backend.tests.test_services_data_analysis` - [ ] Fresh chat: ask "who are you?" → responds as Hesychia - [ ] Confirm classifiers/moderators/title generator unchanged (not assistant identity) Related: companion frontend rebrand `chat_web_app#29`Reviewed-on: #22 |
||
|
westfarn
|
0525f9559b |
Add offline unit test suite for chat_backend (#12)
Unit Tests / test (push) Successful in 9s
Closes #5 ## Summary - Replaces the three scattered test modules (`chat_backend/tests.py`, `services/tests.py`, `services/prompt_classifier/tests.py`) with a `chat_backend/tests/` package: **242 deterministic tests plus 6 opt-in live-Ollama checks**, up from 10 tests (3 of which were skipped and 4 of which were never even discovered). - The suite runs fully offline — no Ollama, Chroma, SMTP or network access. LangChain runnables are replaced by a small `FakeChain`, Chroma/embeddings are mocked, email uses Django's locmem backend, and blobs go through `DatabaseStorage`. - New `llm_be/test_runner.py` (wired via `TEST_RUNNER`) sets `SKIP_RAG_INIT=1` and an MD5 password hasher, so the suite cannot accidentally reach a model server and finishes in ~5s on SQLite (~17s on Postgres) instead of ~30s. ## Coverage | Area | File | |------|------| | `TimeInfoBase.save`, slugs, `get_duration`, `file_exists`, cascades | `test_models.py` | | `DatabaseStorage` save/open/exists/size/listdir/delete/times | `test_storage.py` | | JWT claim, prompt/user/feedback/document serializers | `test_serializers.py` | | auth + token, invite, feedback, company users, set-password, TOS | `test_views_users.py` | | conversation list/order/create/detail/soft-delete | `test_views_conversations.py` | | all four analytics endpoints, including empty-month behaviour | `test_views_analytics.py` | | workspace + document upload/list/detail, 404 and 400 paths | `test_views_documents.py` | | prompt classifier rules/parsing, moderation fail-safe, title cleanup | `test_services_classifiers.py` | | CSV/XLSX/DOCX/PDF analysis, plot generation, error payloads | `test_services_data_analysis.py` | | loader selection, filename sanitising, ingest, temp-file cleanup, search filters | `test_services_rag.py` | | history formatting and streaming | `test_services_llm.py` | | document re-index on create/delete, `SKIP_RAG_INIT` guard | `test_signals.py` | | consumer DB helpers, LangGraph nodes (moderation, classification, generation, search flags), websocket routes | `test_consumers.py` | Live checks (non-deterministic, need a model server): ```bash cd llm_be RUN_LIVE_OLLAMA_TESTS=1 uv run python manage.py test chat_backend.tests.test_live_ollama ``` ## Bugs the tests surfaced (fixed here) 1. **`ConversationDetailView.post` silently dropped every prompt.** `import datetime` shadowed `from datetime import datetime`, so `datetime.now()` raised `AttributeError` inside a bare `except` and the endpoint returned 200 without saving. Now uses `timezone.now()`. 2. **Prompt attachments never reached the LLM.** `get_conversation_file_async` (both consumers) did `sync_to_async(prompt.file.read)` — with `DatabaseStorage` the attribute access itself opens the blob, i.e. a DB query in async context, raising `SynchronousOnlyOperation` that was swallowed and returned `(None, None)`. The read now happens inside the thread. 3. **`DatabaseStorage._save` crashed on a str-backed `ContentFile`** (`TypeError: sequence item 0: expected a bytes-like object`); chunks are encoded when needed. 4. **`services/prompt_classifier/__init__.,py`** (note the comma) meant the directory was only an implicit namespace package, which is why its test module was never collected. Renamed, and its duplicate live-Ollama tests folded into `test_live_ollama.py`. Known-broken paths deliberately left untested and unchanged: `reset_password` / `ResetUserPassword` reference an unimported `requests` plus undefined locals, and `DocumentDetailView.get` references an undefined `workspaces` on its success path. Worth a follow-up ticket. ## Test plan - [x] `cd llm_be && uv run python manage.py test` → 248 tests, OK (6 skipped, all opt-in live) - [x] Same suite against Postgres 16 (`DATABASE_URL=postgres://…`) → OK, matching the containerized run in `deploy.yml` - [x] `uv run black` clean on all added files - [ ] Gitea Actions **Unit Tests** + **CI** green on this PRReviewed-on: #12 |
||
|
westfarn
|
d1660792ad |
Dockerize chat_backend + Ollama LAN + DB file storage (#6) (#7)
Unit Tests / test (push) Successful in 13s
## Summary Implements [chat_backend#6](#6) Part A: - **uv** packaging (`pyproject.toml` + `uv.lock`), Docker/compose (dev + prod), entrypoint/validate-env, Gitea unit-test + auto-deploy workflows (mirror `scha`) - Env-driven Django settings (`DJANGO_*`, `DATABASE_URL`, CSRF/CORS) - **`OLLAMA_BASE_URL`** wired through all Ollama/LangChain clients (prod → `http://10.0.0.128:11434`) - **DatabaseStorage** — prompt/document file blobs in Postgres (`StoredFile`), not container FS; RAG materializes temp paths for loaders - ASGI via `gunicorn` + `UvicornWorker` (HTTP + WebSockets) Companion server-infra PR registers `app_catalog` / `host_apps` (port **8003**). ## Test plan - [ ] `uv sync && cd llm_be && SKIP_RAG_INIT=1 uv run python manage.py test` - [ ] `docker compose build && docker compose up` against bundled Postgres - [ ] Confirm Ollama calls use `OLLAMA_BASE_URL` (not hardcoded localhost) - [ ] Upload a document / prompt file → row in `chat_backend_storedfile`, no disk under `media/` - [ ] After server-infra merge + secret/Postgres/NPM: deploy via `deploy.sh --app chat_backend --env prod`Reviewed-on: #7 |
||
|
westfarn
|
57695353d0 | temp checkin | ||
|
westfarn
|
058f961b0d | Initial commit |