c21f052d5b0a33f890a86baf7f826cf9c33b9f40
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  westfarnandCursor
|
c21f052d5b |
Read attached chat images with a vision model.
Image uploads were treated as unknown documents and never shown to the model. PNG, JPEG, WebP, and GIF now go to Ollama vision, and history returns the image bytes. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
westfarn
|
ab3dfffa6b |
Add opt-in use_conversation_context flag for chat history (#33) (#73)
## Summary - Closes [#33](#33) — per-user `use_conversation_context` boolean on `CustomUser` (default `false`, opt-in). - Migration `0034_customuser_use_conversation_context`. - `GET/POST /api/conversation_preferences` reads/writes the flag; POST sets absolute boolean without toggling `conversation_order` when only this field is sent. - Chat + document RAG paths skip prior-turn history when the flag is off (`AsyncLLMService`, `AsyncRAGService`, both WS consumers). ## Frontend contract - Field: `use_conversation_context` (boolean, default false) - Read: `GET /api/conversation_preferences` → `{ order, use_conversation_context }` - Write: `POST /api/conversation_preferences` with `{ "use_conversation_context": true|false }` - Tooltip copy: use previous conversations to better customize the experience - Counterpart: [chat_web_app#66](ai_ml_operations/chat_web_app#66) ## Test plan - [ ] `manage.py test chat_backend.tests.test_models chat_backend.tests.test_views_conversations.ConversationPreferencesTestCase chat_backend.tests.test_services_llm chat_backend.tests.test_services_rag` - [ ] New user / unset flag → `use_conversation_context` is false; history empty in LLM/RAG prompts - [ ] Enable via preferences API → prior turns appear in history - [ ] Disable again → history skipped; order preference unchanged when posting only the context flagReviewed-on: #73 |
||
|
westfarn
|
bef151c92c |
Fix agent progress frames on live WebSocket (#63) (#72)
## Summary - Follow-up to [#71](#71) / [#63](#63). - **Bugfix:** `run_agentic_turn` discarded `ws_send`, so `run_started` / `plan_ready` / `step_*` / `run_completed` frames only hit an empty Redis group and never the connected client during inline agent runs. - Thread `ws_send` through `execute_agent_run` / `_broadcast` (WS + channel-layer group). - Route agentic turns in `consumers_graph` the same way as `consumers.py`. - Add offline `test_agent_orchestrator.py` coverage. ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test chat_backend.tests.test_agent_orchestrator chat_backend.tests.test_agent_tools chat_backend.tests.test_consumers` - [ ] Manual: `ALLOW_AGENTIC_TASKS=true`, multi-step prompt — confirm agent frames arrive on the live WS - [ ] Manual: graph consumer path (`ws/conditional_chat/`) also routes agentic turnsReviewed-on: #72 |
||
|
westfarn
|
093e5462a5 |
Eval harness (#62 P4), status frames, and agentic runs (#63) (#71)
## Summary - Closes Phase 4 of [#62](#62): `evals/suite.json` (≥40 graded questions), `run_evals` management command, and manually-triggered `.gitea/workflows/run-evals.yml`. - Emits versioned WS `status` frames during grounded chat (evaluating / searching / reading_sources / refining / writing) for [chat_web_app#96](ai_ml_operations/chat_web_app#96). - Implements [#63](#63): Redis/Celery optional infra, `AgentRun`/`AgentStep`, tool registry (SSRF-safe `fetch_url`, tenant-scoped docs), LangGraph orchestrator, progress frames, REST `GET/POST /api/agent_runs/…`, gated by `ALLOW_AGENTIC_TASKS` (default off). ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test` for evals, ws frames, agent tools, consumers, grounding - [ ] Manual: with `ALLOW_AGENTIC_TASKS=false`, chat identical to today - [ ] Manual: status frames visible in FE with #96 branch - [ ] Manual (GPU): `python manage.py run_evals --runs 3` - [ ] Manual: `ALLOW_AGENTIC_TASKS=true` multi-step research prompt creates AgentRun + framesReviewed-on: #71 |
||
|
westfarn
|
e1e086a474 |
Monetization app + RevenueCat webhooks (store IAP ledger) (#69)
## Summary - Rename `finance` → **`monetization`** Django app (keep `finance_*` tables via `label = "finance"`) - Add `services/stripe.py` + `services/revenuecat.py`; RevenueCat webhook upserts **subscription + Invoice/Payment** (billing history parity with Stripe) - Mount `/api/monetization/` + keep `/api/finance/` alias - Extend `Source`/`Provider` with `revenuecat`; product→plan mapping via `revenuecat_product_id` / `REVENUECAT_PRODUCT_PLAN_MAP` Closes #68. Companion to [chat_web_app#100](ai_ml_operations/chat_web_app#100). ## Test plan - [x] `manage.py test monetization.tests` (54 OK) - [x] Smoke `chat_backend.tests.test_views_documents` + `test_oauth` - [ ] Deploy: set `REVENUECAT_WEBHOOK_SECRET`; point RC webhook at `/api/finance/webhooks/revenuecat/` - [ ] Map store product IDs on `SubscriptionPlan.revenuecat_product_id` (or env JSON map) - [ ] Sandbox INITIAL_PURCHASE → subscription `source=revenuecat` + invoice in `/finance/invoices/`Reviewed-on: #69 |
||
|
westfarn
|
d8f5b8ebf2 |
Always-on grounded retrieval + role-scoped Ollama models (#62 Phases 1–3) (#65)
## Summary - Closes Phases 1–3 of [#62](#62) (Phase 4 eval harness left for a follow-up). - **Accuracy:** Retrieval is decided every turn (`GroundingDecider`, fails open). `FAST` no longer skips search — it only selects `OLLAMA_MODEL_FAST`. Search failures surface an explicit error instead of hallucinating from parametric memory. - **Search:** Pluggable `services/search/` with **SearxNG primary** + DDGS failover, ranking/dedupe/rumour filtering, numbered dated source blocks, citations persisted on `Prompt.citations` and emitted as `{"v":1,"type":"citations",...}` after stream end. - **Models:** Role-scoped `OLLAMA_MODEL_THINKING` / `_FAST` / `_UTILITY` / `OLLAMA_EMBED_MODEL=nomic-embed-text`, configurable `num_ctx`, real model name on `PromptMetric`, `reindex_embeddings` management command + loud embedding-dimension mismatch. ## SearxNG (ops) See README **SearxNG** section. Short version: run `searxng/searxng` on the GPU host, enable `json` in `settings.yml`, set `SEARXNG_BASE_URL=http://10.0.0.128:8080` in prod/beta secrets, open `:8080` on the LAN firewall like Ollama. ## Test plan - [x] `SKIP_RAG_INIT=1 python manage.py test chat_backend.tests` — 442 OK (6 skipped) - [ ] Deploy beta with updated secrets (`OLLAMA_MODEL_*`, `OLLAMA_EMBED_MODEL=nomic-embed-text`, `SEARXNG_BASE_URL`) - [ ] After embed change: `python manage.py reindex_embeddings` - [ ] Verify `did Taylor Swift get married` in FAST and THINKING returns grounded answer with citations frame - [ ] Kill SearxNG and confirm factual turns return search_unavailable (not Joe Alwyn hallucination); non-factual chat still worksReviewed-on: #65 |
||
|
westfarn
|
d54094f5e0 |
Tier-gated RAG + Drive document sources (#42) (#54)
## Summary Implements epic [#42](#42) (children #43–#53) and advances [#11](#11). - **Entitlement:** `allows_rag` on plans (founders / backer / pro / business; not standard); exposed as `features.rag` - **Gates:** document REST + WS `PromptType.RAG` use `assert_feature_allowed(..., "rag")` - **Lifecycle:** dedupe ingest, delete vectors by `document_id`, honor `active`, fix document detail PATCH/DELETE - **Workspaces:** auto-create default company workspace; fail-closed scoping - **Drive:** personal + company Google/Microsoft connect (`link_drive` / `link_company_drive`), resource selection, sync, webhooks stubs, `sync_drive_connections` management command - **Docs/env:** README + `.env*.example` updated Companion FE: `chat_web_app` branch `feature/rag-epic-42-ui` (#81–#85). ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test` (457 OK) - [ ] Migrate finance `0004` + chat_backend `0028` on beta - [ ] Verify Standard user: Documents API 403 + no RAG retrieval - [ ] Verify Founders/Pro: upload + list + active toggle - [ ] Connect Google/Microsoft Drive (incremental scopes) and Sync - [ ] Company manager: `link_company_drive`; non-manager 403 - [ ] Run `manage.py sync_drive_connections`Reviewed-on: #54 |
||
|
westfarn
|
2e9e95e16c |
Stove-pipe RAG retrieval to prevent cross-tenant leakage (#40) (#41)
## Summary
- Closes [#40](#40)
- Aligns chat/RAG with the abc_worker stove-pipe pattern ([b13cec8](
|
||
|
westfarn
|
cc45ae5808 |
Persist Ollama token usage from streamed LLM responses (#16) (#38)
## Summary - Fixes token tracking for [#16](#16): streaming chat never persisted `PromptMetric.tokens_in` / `tokens_out` (admin + account usage showed `—`). - Drop `StrOutputParser` on async LLM/RAG/data-analysis chains so Ollama `generation_info` (`prompt_eval_count` / `eval_count`) survives; collect usage while streaming via `TokenUsageCollector`. - Stop calling `self.close()` in `disconnect` (fixes Grafana `Unexpected ASGI message 'websocket.close'`). ## Test plan - [x] Unit tests: `test_utils`, consumers, LLM/RAG/data-analysis services, finance quotas - [ ] Deploy / local: send a chat prompt, confirm admin Prompt Metrics shows Tokens In/Out - [ ] Reload Account usage card — in/out no longer `—` for new turns - [ ] Confirm WS disconnect no longer raises double-close in logsReviewed-on: #38 |
||
|
westfarn
|
841c0962d9 |
Multi-plan subscriptions, quotas, and token usage APIs (#16 #17 #36) (#37)
## Summary Implements [#16](#16), [#17](#17), and [#36](#36) in one backend PR. - **#36 Multi-plan catalog**: Founders ($10, public), Standard ($15), Pro ($40), Business ($99), Backer ($0). Future tiers seeded but hidden/`is_selectable=false`. Backer email whitelist auto-assigns Founders-level access with no checkout. - **#36 Feature + prompt gating**: plan feature flags (text vs image); rolling **6h** prompt windows (100 / 200 / 300 / 300 / 300). Enforced in both chat consumers when `ENFORCE_SUBSCRIPTION_GATES=true`. - **#17 Token-period quotas**: optional `monthly_token_quota` on plans + per-user override; calendar-month aggregation from `PromptMetric`; warn/block when reported token totals exceed cap. Null provider usage never fabricated as 0; tracked via `turns_missing_token_usage`. - **#16 Token API exposure**: `tokens_in` / `tokens_out` on conversation + prompt serializers (null when unknown). `GET /api/finance/subscription/` returns plan + usage snapshot for the FE. - Checkout defaults to **Founders**; Stripe paid webhooks assign Founders. Registration/OAuth redeem Backer whitelist and return `needs_checkout`. Companion FE PR: `chat_web_app` branch `feature/plans-quotas-token-usage`. ## Test plan - [ ] `manage.py migrate` seeds five plans; admin can add Backer emails - [ ] Public `GET /api/finance/plans/` returns only Founders - [ ] Register with Backer email → active Backer, `needs_checkout=false`, checkout rejected - [ ] Founders checkout + paid webhook → active Founders subscription - [ ] Chat turn blocked without subscription / when prompt window exceeded / when token period exceeded - [ ] Standard plan denies image feature; Pro/Founders/Backer allow - [ ] Conversation/prompt API returns `null` tokens when unreported, sums when present - [ ] `finance.tests.test_plans_quotas` + existing finance/checkout tests passReviewed-on: #37 |
||
|
westfarn
|
ee3d47c8c3 |
Ignore WS heartbeats and reject empty chat messages (#32)
## Summary - Closes #31 - Ignore WebSocket `type: ping` heartbeats so keepalives no longer create conversations or hit title/LLM pipelines - Reject empty/whitespace user messages in both chat consumers, `PromptSerializer`, and REST conversation prompt POST ## Test plan - [x] `UserPromptGuardTestCase`, `PromptSerializerTestCase` blank/whitespace cases - [x] `WebSocketReceiveGuardTestCase` ping ignore + empty message rejection (both WS routes) - [ ] Deploy to beta; leave idle tab open and confirm no new rogue conversations - [ ] Confirm normal chat send still works Related FE: https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/51Reviewed-on: #32 |
||
|
westfarn
|
a049e4f685 |
Track token in/out per prompt on PromptMetric (#18)
Unit Tests / test (push) Successful in 9s
Closes #15 ## Summary - Add nullable `tokens_in` / `tokens_out` `IntegerField`s to `PromptMetric` to record real prompt/completion token counts per turn. - New `extract_token_usage()` helper parses provider usage payloads (LangChain `usage_metadata`, OpenAI-style `prompt_tokens`/`completion_tokens`, Ollama `prompt_eval_count`/`eval_count`). When a provider reports no usage, the fields stay **null** — counts are never estimated/fabricated. - `create_prompt_metric` / `finish_prompt_metric` in both `consumers.py` and `consumers_graph.py` accept and persist optional `tokens_in` / `tokens_out` (added to `update_fields` only when present). - Admin panel (this ticket's deliverable): - `PromptMetricAdmin` lists `tokens_in` / `tokens_out` and adds `event` / `model_name` / `has_file` filters. - `ConversationAdmin` shows summed `tokens_in` / `tokens_out` / `tokens_total` per conversation. - Migration `0023_promptmetric_tokens_in_promptmetric_tokens_out` (existing rows remain valid — null). ## Note on live capture The streaming chat path uses LangChain `StrOutputParser`, which yields plain string chunks with no usage metadata, so live turns currently persist `null` tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it. ## Follow-ups - #16 — Show token in/out in chat web app UI (FE + API exposure) - #17 — Token-based billing, quotas, and enforcement ## Test plan - [x] `uv run python manage.py test` — full suite green (266 tests, 6 skipped) - [x] Model: token fields default null + persist when set - [x] `extract_token_usage`: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None) - [x] Metric lifecycle: tokens persist when provided, stay null when absent (both consumers) - [x] Admin: conversation token totals sum across metrics and ignore other conversationsReviewed-on: #18 |
||
|
westfarn
|
0525f9559b |
Add offline unit test suite for chat_backend (#12)
Unit Tests / test (push) Successful in 9s
Closes #5 ## Summary - Replaces the three scattered test modules (`chat_backend/tests.py`, `services/tests.py`, `services/prompt_classifier/tests.py`) with a `chat_backend/tests/` package: **242 deterministic tests plus 6 opt-in live-Ollama checks**, up from 10 tests (3 of which were skipped and 4 of which were never even discovered). - The suite runs fully offline — no Ollama, Chroma, SMTP or network access. LangChain runnables are replaced by a small `FakeChain`, Chroma/embeddings are mocked, email uses Django's locmem backend, and blobs go through `DatabaseStorage`. - New `llm_be/test_runner.py` (wired via `TEST_RUNNER`) sets `SKIP_RAG_INIT=1` and an MD5 password hasher, so the suite cannot accidentally reach a model server and finishes in ~5s on SQLite (~17s on Postgres) instead of ~30s. ## Coverage | Area | File | |------|------| | `TimeInfoBase.save`, slugs, `get_duration`, `file_exists`, cascades | `test_models.py` | | `DatabaseStorage` save/open/exists/size/listdir/delete/times | `test_storage.py` | | JWT claim, prompt/user/feedback/document serializers | `test_serializers.py` | | auth + token, invite, feedback, company users, set-password, TOS | `test_views_users.py` | | conversation list/order/create/detail/soft-delete | `test_views_conversations.py` | | all four analytics endpoints, including empty-month behaviour | `test_views_analytics.py` | | workspace + document upload/list/detail, 404 and 400 paths | `test_views_documents.py` | | prompt classifier rules/parsing, moderation fail-safe, title cleanup | `test_services_classifiers.py` | | CSV/XLSX/DOCX/PDF analysis, plot generation, error payloads | `test_services_data_analysis.py` | | loader selection, filename sanitising, ingest, temp-file cleanup, search filters | `test_services_rag.py` | | history formatting and streaming | `test_services_llm.py` | | document re-index on create/delete, `SKIP_RAG_INIT` guard | `test_signals.py` | | consumer DB helpers, LangGraph nodes (moderation, classification, generation, search flags), websocket routes | `test_consumers.py` | Live checks (non-deterministic, need a model server): ```bash cd llm_be RUN_LIVE_OLLAMA_TESTS=1 uv run python manage.py test chat_backend.tests.test_live_ollama ``` ## Bugs the tests surfaced (fixed here) 1. **`ConversationDetailView.post` silently dropped every prompt.** `import datetime` shadowed `from datetime import datetime`, so `datetime.now()` raised `AttributeError` inside a bare `except` and the endpoint returned 200 without saving. Now uses `timezone.now()`. 2. **Prompt attachments never reached the LLM.** `get_conversation_file_async` (both consumers) did `sync_to_async(prompt.file.read)` — with `DatabaseStorage` the attribute access itself opens the blob, i.e. a DB query in async context, raising `SynchronousOnlyOperation` that was swallowed and returned `(None, None)`. The read now happens inside the thread. 3. **`DatabaseStorage._save` crashed on a str-backed `ContentFile`** (`TypeError: sequence item 0: expected a bytes-like object`); chunks are encoded when needed. 4. **`services/prompt_classifier/__init__.,py`** (note the comma) meant the directory was only an implicit namespace package, which is why its test module was never collected. Renamed, and its duplicate live-Ollama tests folded into `test_live_ollama.py`. Known-broken paths deliberately left untested and unchanged: `reset_password` / `ResetUserPassword` reference an unimported `requests` plus undefined locals, and `DocumentDetailView.get` references an undefined `workspaces` on its success path. Worth a follow-up ticket. ## Test plan - [x] `cd llm_be && uv run python manage.py test` → 248 tests, OK (6 skipped, all opt-in live) - [x] Same suite against Postgres 16 (`DATABASE_URL=postgres://…`) → OK, matching the containerized run in `deploy.yml` - [x] `uv run black` clean on all added files - [ ] Gitea Actions **Unit Tests** + **CI** green on this PRReviewed-on: #12 |
||
|
westfarn
|
77d7edd0dc |
Closes #4
Added site tracking Can pick the model the use Better handle llm model based on debug or not |
||
|
westfarn
|
eed1abedc8 | updates |