Commit Graph
13 Commits
Author SHA1 Message Date
westfarnandCursor c21f052d5b Read attached chat images with a vision model.
CI / test (pull_request) Successful in 9s
Unit Tests / test (pull_request) Successful in 10s
Image uploads were treated as unknown documents and never shown to the model. PNG, JPEG, WebP, and GIF now go to Ollama vision, and history returns the image bytes.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-20 08:11:42 -05:00
westfarn e1e086a474 Monetization app + RevenueCat webhooks (store IAP ledger) (#69)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 11s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 50s
## Summary
- Rename `finance` → **`monetization`** Django app (keep `finance_*` tables via `label = "finance"`)
- Add `services/stripe.py` + `services/revenuecat.py`; RevenueCat webhook upserts **subscription + Invoice/Payment** (billing history parity with Stripe)
- Mount `/api/monetization/` + keep `/api/finance/` alias
- Extend `Source`/`Provider` with `revenuecat`; product→plan mapping via `revenuecat_product_id` / `REVENUECAT_PRODUCT_PLAN_MAP`

Closes #68. Companion to [chat_web_app#100](ai_ml_operations/chat_web_app#100).

## Test plan
- [x] `manage.py test monetization.tests` (54 OK)
- [x] Smoke `chat_backend.tests.test_views_documents` + `test_oauth`
- [ ] Deploy: set `REVENUECAT_WEBHOOK_SECRET`; point RC webhook at `/api/finance/webhooks/revenuecat/`
- [ ] Map store product IDs on `SubscriptionPlan.revenuecat_product_id` (or env JSON map)
- [ ] Sandbox INITIAL_PURCHASE → subscription `source=revenuecat` + invoice in `/finance/invoices/`Reviewed-on: #69
2026-08-04 03:40:15 -07:00
westfarn 2aeb95136a Add PromptFeedback API for per-message thumbs ratings (#67) (#70)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 11s
Deploy Beta / docker (push) Successful in 22s
Deploy Beta / deploy-beta (push) Successful in 49s
## Summary
- Closes [#67](#67) — new `PromptFeedback` model (unique on `(prompt, user)`) with `rating` (`up`|`down`), optional `reason` / `comment`, and timestamps via `TimeInfoBase`.
- `POST /api/prompt_feedback` upserts `{ prompt_id, rating, reason?, comment? }`; `DELETE /api/prompt_feedback?prompt_id=` clears the caller's vote.
- `GET conversation_details` now nests the caller's `feedback: { rating, reason, comment }` (or `null`) on each prompt so [chat_web_app#101](ai_ml_operations/chat_web_app#101) can rehydrate thumbs UI.
- Auth required; users can only rate assistant prompts in their own non-deleted conversations. Distinct from app-wide `POST /feedbacks/`.
- Joinable to `PromptMetric` via `prompt_id` for per-model accuracy slices.

## Test plan
- [ ] `uv run python manage.py test chat_backend.tests.test_views_prompt_feedback`
- [ ] Upsert thumbs up, then down with reason/comment — one row updated
- [ ] DELETE clears vote; second DELETE → 404
- [ ] Rating another user's prompt → 404; rating a user message → 400
- [ ] Reload conversation details → assistant prompts show caller feedback only
- [ ] Companion FE [chat_web_app#101](ai_ml_operations/chat_web_app#101) thumbs + reason popover against this APIReviewed-on: #70
2026-08-04 03:23:58 -07:00
westfarn d8f5b8ebf2 Always-on grounded retrieval + role-scoped Ollama models (#62 Phases 1–3) (#65)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 30s
Deploy Beta / deploy-beta (push) Successful in 6m49s
## Summary
- Closes Phases 1–3 of [#62](#62) (Phase 4 eval harness left for a follow-up).
- **Accuracy:** Retrieval is decided every turn (`GroundingDecider`, fails open). `FAST` no longer skips search — it only selects `OLLAMA_MODEL_FAST`. Search failures surface an explicit error instead of hallucinating from parametric memory.
- **Search:** Pluggable `services/search/` with **SearxNG primary** + DDGS failover, ranking/dedupe/rumour filtering, numbered dated source blocks, citations persisted on `Prompt.citations` and emitted as `{"v":1,"type":"citations",...}` after stream end.
- **Models:** Role-scoped `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` / `OLLAMA_EMBED_MODEL=nomic-embed-text`, configurable `num_ctx`, real model name on `PromptMetric`, `reindex_embeddings` management command + loud embedding-dimension mismatch.

## SearxNG (ops)
See README **SearxNG** section. Short version: run `searxng/searxng` on the GPU host, enable `json` in `settings.yml`, set `SEARXNG_BASE_URL=http://10.0.0.128:8080` in prod/beta secrets, open `:8080` on the LAN firewall like Ollama.

## Test plan
- [x] `SKIP_RAG_INIT=1 python manage.py test chat_backend.tests` — 442 OK (6 skipped)
- [ ] Deploy beta with updated secrets (`OLLAMA_MODEL_*`, `OLLAMA_EMBED_MODEL=nomic-embed-text`, `SEARXNG_BASE_URL`)
- [ ] After embed change: `python manage.py reindex_embeddings`
- [ ] Verify `did Taylor Swift get married` in FAST and THINKING returns grounded answer with citations frame
- [ ] Kill SearxNG and confirm factual turns return search_unavailable (not Joe Alwyn hallucination); non-factual chat still worksReviewed-on: #65
2026-08-02 11:46:02 -07:00
westfarn 57a2350b8e Drive sync progress + documents list API + prompt heatmap (#59, #60, #94) (#61)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 48s
## Summary
- **#59** — Persist and expose Drive sync progress (`sync_total`, `sync_processed`, `sync_added`, `sync_updated`, `sync_failed`) during sync so the FE can show a progress bar.
- **#60** — Paginate/search/sort `GET /documents/` with `workspace=personal|company` scope isolation.
- **#94 (API)** — Add `GET /analytics/user_prompt_heatmap/?tz=` weekday × hour bins for user-entered prompts.

## Test plan
- [x] `manage.py test` drive sync / documents / analytics / drive_tasks suites
- [ ] Manual: trigger Drive sync and poll connections → progress fields update while pending
- [ ] Manual: `GET /documents/?workspace=personal&page=1&search=…&ordering=name`
- [ ] Deploy FE companion PR for #92/#93/#94 UI

Closes #59
Closes #60Reviewed-on: #61
2026-08-02 04:49:23 -07:00
westfarn d54094f5e0 Tier-gated RAG + Drive document sources (#42) (#54)
Unit Tests / test (push) Successful in 10s
Deploy Beta / unit-tests (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 40s
## Summary

Implements epic [#42](#42) (children #43–#53) and advances [#11](#11).

- **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag`
- **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")`
- **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE
- **Workspaces:** auto-create default company workspace; fail-closed scoping
- **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command
- **Docs/env:** README + `.env*.example` updated

Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85).

## Test plan

- [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK)
- [ ] Migrate finance `0004` + chat_backend `0028` on beta
- [ ] Verify Standard user: Documents API 403 + no RAG retrieval
- [ ] Verify Founders/Pro: upload + list + active toggle
- [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync
- [ ] Company manager: `link_company_drive`; non-manager 403
- [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54
2026-08-01 14:02:36 -07:00
westfarn eedc842b08 Add account self-delete and subscription lifecycle sync (#34) (#39)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 26s
Deploy Beta / deploy-beta (push) Successful in 6m46s
## Summary
- Closes [#34](#34)
- Companion for [chat_web_app#75](ai_ml_operations/chat_web_app#75) (portal cancel/change local sync)
- Soft-delete `DELETE /api/user/` for the authenticated user only: `deleted=True`, `is_active=False`, hide conversations, blacklist outstanding refresh tokens; staff self-delete rejected
- Stripe `customer.subscription.updated` / `deleted` webhooks sync plan status, `cancel_at_period_end`, and `current_period_end`; checkout assigns plan from `metadata.plan_slug`
- **UserAuthEvent audit**: `account_deleted`, `subscription_started` (first active plan), `subscription_updated` (plan/status/cancel changes) — visible on user admin
- Document FE contract in README (endpoint, response, post-delete logout)

## Test plan
- [ ] `uv run python manage.py test chat_backend.tests.test_views_users.CustomUserSelfDeleteTestCase finance.tests`
- [ ] Authenticated `DELETE /api/user/` soft-deletes self, hides conversations, blocks re-login, writes `account_deleted` auth event
- [ ] Checkout / Backer assign writes `subscription_started`; portal cancel/change writes `subscription_updated`
- [ ] Anonymous / staff self-delete rejected; body cannot target another user
- [ ] After portal cancel, webhook sets `cancel_at_period_end` / `canceled` on `GET /finance/subscription/`Reviewed-on: #39
2026-08-01 12:24:17 -07:00
westfarn 841c0962d9 Multi-plan subscriptions, quotas, and token usage APIs (#16 #17 #36) (#37)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 9s
Deploy Beta / docker (push) Successful in 18s
Deploy Beta / deploy-beta (push) Successful in 46s
## Summary
Implements [#16](#16), [#17](#17), and [#36](#36) in one backend PR.

- **#36 Multi-plan catalog**: Founders ($10, public), Standard ($15), Pro ($40), Business ($99), Backer ($0). Future tiers seeded but hidden/`is_selectable=false`. Backer email whitelist auto-assigns Founders-level access with no checkout.
- **#36 Feature + prompt gating**: plan feature flags (text vs image); rolling **6h** prompt windows (100 / 200 / 300 / 300 / 300). Enforced in both chat consumers when `ENFORCE_SUBSCRIPTION_GATES=true`.
- **#17 Token-period quotas**: optional `monthly_token_quota` on plans + per-user override; calendar-month aggregation from `PromptMetric`; warn/block when reported token totals exceed cap. Null provider usage never fabricated as 0; tracked via `turns_missing_token_usage`.
- **#16 Token API exposure**: `tokens_in` / `tokens_out` on conversation + prompt serializers (null when unknown). `GET /api/finance/subscription/` returns plan + usage snapshot for the FE.
- Checkout defaults to **Founders**; Stripe paid webhooks assign Founders. Registration/OAuth redeem Backer whitelist and return `needs_checkout`.

Companion FE PR: `chat_web_app` branch `feature/plans-quotas-token-usage`.

## Test plan
- [ ] `manage.py migrate` seeds five plans; admin can add Backer emails
- [ ] Public `GET /api/finance/plans/` returns only Founders
- [ ] Register with Backer email → active Backer, `needs_checkout=false`, checkout rejected
- [ ] Founders checkout + paid webhook → active Founders subscription
- [ ] Chat turn blocked without subscription / when prompt window exceeded / when token period exceeded
- [ ] Standard plan denies image feature; Pro/Founders/Backer allow
- [ ] Conversation/prompt API returns `null` tokens when unreported, sums when present
- [ ] `finance.tests.test_plans_quotas` + existing finance/checkout tests passReviewed-on: #37
2026-07-31 04:24:20 -07:00
westfarn ee3d47c8c3 Ignore WS heartbeats and reject empty chat messages (#32)
Deploy Beta / docker (push) Successful in 25s
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 9s
Deploy Beta / deploy-beta (push) Successful in 56s
## Summary
- Closes #31
- Ignore WebSocket `type: ping` heartbeats so keepalives no longer create conversations or hit title/LLM pipelines
- Reject empty/whitespace user messages in both chat consumers, `PromptSerializer`, and REST conversation prompt POST

## Test plan
- [x] `UserPromptGuardTestCase`, `PromptSerializerTestCase` blank/whitespace cases
- [x] `WebSocketReceiveGuardTestCase` ping ignore + empty message rejection (both WS routes)
- [ ] Deploy to beta; leave idle tab open and confirm no new rogue conversations
- [ ] Confirm normal chat send still works

Related FE: https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/51Reviewed-on: #32
2026-07-28 05:12:13 -07:00
westfarn 30ce3d048d Gate self-serve registration behind ENABLE_ACCOUNT_REGISTRATION (#25)
Unit Tests / test (push) Successful in 9s
## Summary
- Add `ENABLE_ACCOUNT_REGISTRATION` (default `false`) for self-serve sign-up, set via env / control-node secret (`chat_backend_<env>.env`)
- Harden `POST /api/user/create/` to create company + user, return JWTs, and reject when the flag is off
- Expose `GET /api/public/settings/` with `enable_account_registration` for the SPA

Supports frontend: [chat_web_app#31](ai_ml_operations/chat_web_app#31)

## Test plan
- [ ] `cd llm_be && uv run python manage.py test chat_backend.tests.test_views_users`
- [ ] Confirm `POST /api/user/create/` returns 403 when flag unset/false
- [ ] With `ENABLE_ACCOUNT_REGISTRATION=true`, register returns access/refresh + creates company
- [ ] `GET /api/public/settings/` returns the flag
- [ ] Add `ENABLE_ACCOUNT_REGISTRATION=false` to control-node `chat_backend_prod.env` / `chat_backend_beta.env` before deploy (see `.env.prod.example`)Reviewed-on: #25
2026-07-27 04:19:59 -07:00
westfarn a85f1222eb Syncing with updates from prod and formatted 2025-05-18 06:15:07 -05:00
westfarn f5d29166a6 RAG implementation, content moderation, prompt classification, new LLM chain, document storage 2025-05-14 03:27:38 -05:00
westfarn 57695353d0 temp checkin 2025-03-07 12:23:00 -06:00