Commit Graph
13 Commits
Author SHA1 Message Date
westfarn fc06202a1f Read attached chat images with a vision model (#82)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 9s
Deploy Beta / docker (push) Successful in 23s
Deploy Beta / deploy-beta (push) Successful in 1m22s
## Summary
- Image attachments (PNG, JPEG, WebP, GIF) skip the document parser and go to a vision-capable Ollama model (`OLLAMA_MODEL_VISION`, default `gemma4:latest`).
- The image is stored on the prompt. `PromptSerializer` returns `file_type` and base64 `image` so history can redraw it.
- Unsupported image types and files over 8 MB return a websocket error and are not saved.
- A later text turn does not reuse a previous image as a spreadsheet.

Pairs with `chat_web_app` `feat/128-vision-image-prompt` ([#128](ai_ml_operations/chat_web_app#128)).

## Test plan
- [ ] `manage.py test chat_backend.tests.test_vision chat_backend.tests.test_consumers.GraphNodeTestCase`
- [ ] Send a prompt plus PNG over `ws/chat_again/` and confirm the vision model is called.
- [ ] `GET conversation_details` includes `image` and `file_type` for that turn.
- [ ] CSV upload still follows the document path.

Reviewed-on: #82
2026-10-20 06:13:31 -07:00
westfarn e1e086a474 Monetization app + RevenueCat webhooks (store IAP ledger) (#69)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 11s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 50s
## Summary
- Rename `finance` → **`monetization`** Django app (keep `finance_*` tables via `label = "finance"`)
- Add `services/stripe.py` + `services/revenuecat.py`; RevenueCat webhook upserts **subscription + Invoice/Payment** (billing history parity with Stripe)
- Mount `/api/monetization/` + keep `/api/finance/` alias
- Extend `Source`/`Provider` with `revenuecat`; product→plan mapping via `revenuecat_product_id` / `REVENUECAT_PRODUCT_PLAN_MAP`

Closes #68. Companion to [chat_web_app#100](ai_ml_operations/chat_web_app#100).

## Test plan
- [x] `manage.py test monetization.tests` (54 OK)
- [x] Smoke `chat_backend.tests.test_views_documents` + `test_oauth`
- [ ] Deploy: set `REVENUECAT_WEBHOOK_SECRET`; point RC webhook at `/api/finance/webhooks/revenuecat/`
- [ ] Map store product IDs on `SubscriptionPlan.revenuecat_product_id` (or env JSON map)
- [ ] Sandbox INITIAL_PURCHASE → subscription `source=revenuecat` + invoice in `/finance/invoices/`Reviewed-on: #69
2026-08-04 03:40:15 -07:00
westfarn 2aeb95136a Add PromptFeedback API for per-message thumbs ratings (#67) (#70)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 11s
Deploy Beta / docker (push) Successful in 22s
Deploy Beta / deploy-beta (push) Successful in 49s
## Summary
- Closes [#67](#67) — new `PromptFeedback` model (unique on `(prompt, user)`) with `rating` (`up`|`down`), optional `reason` / `comment`, and timestamps via `TimeInfoBase`.
- `POST /api/prompt_feedback` upserts `{ prompt_id, rating, reason?, comment? }`; `DELETE /api/prompt_feedback?prompt_id=` clears the caller's vote.
- `GET conversation_details` now nests the caller's `feedback: { rating, reason, comment }` (or `null`) on each prompt so [chat_web_app#101](ai_ml_operations/chat_web_app#101) can rehydrate thumbs UI.
- Auth required; users can only rate assistant prompts in their own non-deleted conversations. Distinct from app-wide `POST /feedbacks/`.
- Joinable to `PromptMetric` via `prompt_id` for per-model accuracy slices.

## Test plan
- [ ] `uv run python manage.py test chat_backend.tests.test_views_prompt_feedback`
- [ ] Upsert thumbs up, then down with reason/comment — one row updated
- [ ] DELETE clears vote; second DELETE → 404
- [ ] Rating another user's prompt → 404; rating a user message → 400
- [ ] Reload conversation details → assistant prompts show caller feedback only
- [ ] Companion FE [chat_web_app#101](ai_ml_operations/chat_web_app#101) thumbs + reason popover against this APIReviewed-on: #70
2026-08-04 03:23:58 -07:00
westfarn d8f5b8ebf2 Always-on grounded retrieval + role-scoped Ollama models (#62 Phases 1–3) (#65)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 30s
Deploy Beta / deploy-beta (push) Successful in 6m49s
## Summary
- Closes Phases 1–3 of [#62](#62) (Phase 4 eval harness left for a follow-up).
- **Accuracy:** Retrieval is decided every turn (`GroundingDecider`, fails open). `FAST` no longer skips search — it only selects `OLLAMA_MODEL_FAST`. Search failures surface an explicit error instead of hallucinating from parametric memory.
- **Search:** Pluggable `services/search/` with **SearxNG primary** + DDGS failover, ranking/dedupe/rumour filtering, numbered dated source blocks, citations persisted on `Prompt.citations` and emitted as `{"v":1,"type":"citations",...}` after stream end.
- **Models:** Role-scoped `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` / `OLLAMA_EMBED_MODEL=nomic-embed-text`, configurable `num_ctx`, real model name on `PromptMetric`, `reindex_embeddings` management command + loud embedding-dimension mismatch.

## SearxNG (ops)
See README **SearxNG** section. Short version: run `searxng/searxng` on the GPU host, enable `json` in `settings.yml`, set `SEARXNG_BASE_URL=http://10.0.0.128:8080` in prod/beta secrets, open `:8080` on the LAN firewall like Ollama.

## Test plan
- [x] `SKIP_RAG_INIT=1 python manage.py test chat_backend.tests` — 442 OK (6 skipped)
- [ ] Deploy beta with updated secrets (`OLLAMA_MODEL_*`, `OLLAMA_EMBED_MODEL=nomic-embed-text`, `SEARXNG_BASE_URL`)
- [ ] After embed change: `python manage.py reindex_embeddings`
- [ ] Verify `did Taylor Swift get married` in FAST and THINKING returns grounded answer with citations frame
- [ ] Kill SearxNG and confirm factual turns return search_unavailable (not Joe Alwyn hallucination); non-factual chat still worksReviewed-on: #65
2026-08-02 11:46:02 -07:00
westfarn 57a2350b8e Drive sync progress + documents list API + prompt heatmap (#59, #60, #94) (#61)
Deploy Beta / unit-tests (push) Successful in 11s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 48s
## Summary
- **#59** — Persist and expose Drive sync progress (`sync_total`, `sync_processed`, `sync_added`, `sync_updated`, `sync_failed`) during sync so the FE can show a progress bar.
- **#60** — Paginate/search/sort `GET /documents/` with `workspace=personal|company` scope isolation.
- **#94 (API)** — Add `GET /analytics/user_prompt_heatmap/?tz=` weekday × hour bins for user-entered prompts.

## Test plan
- [x] `manage.py test` drive sync / documents / analytics / drive_tasks suites
- [ ] Manual: trigger Drive sync and poll connections → progress fields update while pending
- [ ] Manual: `GET /documents/?workspace=personal&page=1&search=…&ordering=name`
- [ ] Deploy FE companion PR for #92/#93/#94 UI

Closes #59
Closes #60Reviewed-on: #61
2026-08-02 04:49:23 -07:00
westfarn d54094f5e0 Tier-gated RAG + Drive document sources (#42) (#54)
Unit Tests / test (push) Successful in 10s
Deploy Beta / unit-tests (push) Successful in 10s
Deploy Beta / docker (push) Successful in 21s
Deploy Beta / deploy-beta (push) Successful in 40s
## Summary

Implements epic [#42](#42) (children #43–#53) and advances [#11](#11).

- **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag`
- **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")`
- **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE
- **Workspaces:** auto-create default company workspace; fail-closed scoping
- **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command
- **Docs/env:** README + `.env*.example` updated

Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85).

## Test plan

- [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK)
- [ ] Migrate finance `0004` + chat_backend `0028` on beta
- [ ] Verify Standard user: Documents API 403 + no RAG retrieval
- [ ] Verify Founders/Pro: upload + list + active toggle
- [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync
- [ ] Company manager: `link_company_drive`; non-manager 403
- [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54
2026-08-01 14:02:36 -07:00
westfarn eedc842b08 Add account self-delete and subscription lifecycle sync (#34) (#39)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 26s
Deploy Beta / deploy-beta (push) Successful in 6m46s
## Summary
- Closes [#34](#34)
- Companion for [chat_web_app#75](ai_ml_operations/chat_web_app#75) (portal cancel/change local sync)
- Soft-delete `DELETE /api/user/` for the authenticated user only: `deleted=True`, `is_active=False`, hide conversations, blacklist outstanding refresh tokens; staff self-delete rejected
- Stripe `customer.subscription.updated` / `deleted` webhooks sync plan status, `cancel_at_period_end`, and `current_period_end`; checkout assigns plan from `metadata.plan_slug`
- **UserAuthEvent audit**: `account_deleted`, `subscription_started` (first active plan), `subscription_updated` (plan/status/cancel changes) — visible on user admin
- Document FE contract in README (endpoint, response, post-delete logout)

## Test plan
- [ ] `uv run python manage.py test chat_backend.tests.test_views_users.CustomUserSelfDeleteTestCase finance.tests`
- [ ] Authenticated `DELETE /api/user/` soft-deletes self, hides conversations, blocks re-login, writes `account_deleted` auth event
- [ ] Checkout / Backer assign writes `subscription_started`; portal cancel/change writes `subscription_updated`
- [ ] Anonymous / staff self-delete rejected; body cannot target another user
- [ ] After portal cancel, webhook sets `cancel_at_period_end` / `canceled` on `GET /finance/subscription/`Reviewed-on: #39
2026-08-01 12:24:17 -07:00
westfarn 841c0962d9 Multi-plan subscriptions, quotas, and token usage APIs (#16 #17 #36) (#37)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 9s
Deploy Beta / docker (push) Successful in 18s
Deploy Beta / deploy-beta (push) Successful in 46s
## Summary
Implements [#16](#16), [#17](#17), and [#36](#36) in one backend PR.

- **#36 Multi-plan catalog**: Founders ($10, public), Standard ($15), Pro ($40), Business ($99), Backer ($0). Future tiers seeded but hidden/`is_selectable=false`. Backer email whitelist auto-assigns Founders-level access with no checkout.
- **#36 Feature + prompt gating**: plan feature flags (text vs image); rolling **6h** prompt windows (100 / 200 / 300 / 300 / 300). Enforced in both chat consumers when `ENFORCE_SUBSCRIPTION_GATES=true`.
- **#17 Token-period quotas**: optional `monthly_token_quota` on plans + per-user override; calendar-month aggregation from `PromptMetric`; warn/block when reported token totals exceed cap. Null provider usage never fabricated as 0; tracked via `turns_missing_token_usage`.
- **#16 Token API exposure**: `tokens_in` / `tokens_out` on conversation + prompt serializers (null when unknown). `GET /api/finance/subscription/` returns plan + usage snapshot for the FE.
- Checkout defaults to **Founders**; Stripe paid webhooks assign Founders. Registration/OAuth redeem Backer whitelist and return `needs_checkout`.

Companion FE PR: `chat_web_app` branch `feature/plans-quotas-token-usage`.

## Test plan
- [ ] `manage.py migrate` seeds five plans; admin can add Backer emails
- [ ] Public `GET /api/finance/plans/` returns only Founders
- [ ] Register with Backer email → active Backer, `needs_checkout=false`, checkout rejected
- [ ] Founders checkout + paid webhook → active Founders subscription
- [ ] Chat turn blocked without subscription / when prompt window exceeded / when token period exceeded
- [ ] Standard plan denies image feature; Pro/Founders/Backer allow
- [ ] Conversation/prompt API returns `null` tokens when unreported, sums when present
- [ ] `finance.tests.test_plans_quotas` + existing finance/checkout tests passReviewed-on: #37
2026-07-31 04:24:20 -07:00
westfarn ee3d47c8c3 Ignore WS heartbeats and reject empty chat messages (#32)
Deploy Beta / docker (push) Successful in 25s
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 9s
Deploy Beta / deploy-beta (push) Successful in 56s
## Summary
- Closes #31
- Ignore WebSocket `type: ping` heartbeats so keepalives no longer create conversations or hit title/LLM pipelines
- Reject empty/whitespace user messages in both chat consumers, `PromptSerializer`, and REST conversation prompt POST

## Test plan
- [x] `UserPromptGuardTestCase`, `PromptSerializerTestCase` blank/whitespace cases
- [x] `WebSocketReceiveGuardTestCase` ping ignore + empty message rejection (both WS routes)
- [ ] Deploy to beta; leave idle tab open and confirm no new rogue conversations
- [ ] Confirm normal chat send still works

Related FE: https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/51Reviewed-on: #32
2026-07-28 05:12:13 -07:00
westfarn 30ce3d048d Gate self-serve registration behind ENABLE_ACCOUNT_REGISTRATION (#25)
Unit Tests / test (push) Successful in 9s
## Summary
- Add `ENABLE_ACCOUNT_REGISTRATION` (default `false`) for self-serve sign-up, set via env / control-node secret (`chat_backend_<env>.env`)
- Harden `POST /api/user/create/` to create company + user, return JWTs, and reject when the flag is off
- Expose `GET /api/public/settings/` with `enable_account_registration` for the SPA

Supports frontend: [chat_web_app#31](ai_ml_operations/chat_web_app#31)

## Test plan
- [ ] `cd llm_be && uv run python manage.py test chat_backend.tests.test_views_users`
- [ ] Confirm `POST /api/user/create/` returns 403 when flag unset/false
- [ ] With `ENABLE_ACCOUNT_REGISTRATION=true`, register returns access/refresh + creates company
- [ ] `GET /api/public/settings/` returns the flag
- [ ] Add `ENABLE_ACCOUNT_REGISTRATION=false` to control-node `chat_backend_prod.env` / `chat_backend_beta.env` before deploy (see `.env.prod.example`)Reviewed-on: #25
2026-07-27 04:19:59 -07:00
westfarn a85f1222eb Syncing with updates from prod and formatted 2025-05-18 06:15:07 -05:00
westfarn f5d29166a6 RAG implementation, content moderation, prompt classification, new LLM chain, document storage 2025-05-14 03:27:38 -05:00
westfarn 57695353d0 temp checkin 2025-03-07 12:23:00 -06:00