ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

[TICKET] Answer accuracy overhaul: always-on grounded retrieval, better models, and citations #62

New Issue
Open
opened 2026-08-02 06:45:38 -07:00 by westfarn · 1 comment
westfarn commented 2026-08-02 06:45:38 -07:00
Owner
Copy Link
Copy Source

Priority

P0 — highest priority. Accuracy is the top concern for this cycle. Ship this before the agentic work (see the agentic execution ticket).

Problem

The same factual question returns different — and often fabricated — answers on repeated asks. Reproduced end-to-end against the production Ollama host (http://10.0.0.128:11434) with the production model.

Reproduction 1 — no-search path fabricates, inconsistently

Prompt "did Taylor Swift get married" sent straight to llama3.2 with the production sampling params (temperature=0.7, top_k=50, top_p=0.9, repeat_penalty=1.1, num_ctx=4096), three consecutive runs:

run 0: "As of my knowledge cutoff in December 2023, Taylor Swift is not married."
run 1: "Yes, Taylor Swift is married. She tied the knot with Joe Alwyn, a British
        actor, on March 18, 2023."          <-- entirely fabricated
run 2: "Taylor Swift is not married."

This is exactly the reported "three different answers, one correct" behaviour. Run 1 invents a wedding that never happened, with a specific date, stated confidently.

Reproduction 2 — search results are non-deterministic and sometimes poisoned

DuckDuckGoSearchRun().run("did Taylor Swift get married"), three consecutive calls, returned three materially different blobs:

  • run 0 (1082 chars) — correct: married Travis Kelce, July 3, Madison Square Garden.
  • run 1 (714 chars) — poisoned: "fake, AI-generated photos of Taylor Swift and Travis Kelce in wedding attire", "Rumors are running wild online following a new blind item claim", plus an unrelated couple (Hurtado and Bochman) who married at a Swift concert. A small model handed only this blob will reasonably answer "no, those were fakes / just rumours".
  • run 2 (1136 chars) — correct.

The blob has no URLs, no per-result delimiters, no publication dates, and no ranking. It is injected verbatim as HumanMessage(content=f"Search Results: {search_results}").

Reproduction 3 — when search does land cleanly, answers are correct and stable

Four end-to-end runs through the real AsyncLLMService template with a fresh search each time all produced the correct answer (married Travis Kelce, July 3, Madison Square Garden). Grounding works. The problem is that grounding is optional, silent when it fails, and unstructured when it succeeds.

Root causes

1. Search is gated behind three independent conditions, any of which silently disables it

llm_be/chat_backend/consumers.py:481-497 (and the mirror in consumers_graph.py:280-294):

if prompt_type == PromptType.SEARCH:
    # Check modelName first - if FAST, we skip search regardless of settings
    if input_dict.get("model_name") == "FAST":
        pass # Skip search
    elif getattr(settings, "ALLOW_INTERNET_ACCESS", False):
        try:
            search = DuckDuckGoSearchRun()
            search_results = search.run(input_dict["message"])
            messages.append(HumanMessage(content=f"Search Results: {search_results}"))
        except Exception as e:
            logger.error(f"Search failed: {e}")
            pass

Search runs only if all of: classifier returns SEARCH, and modelName != "FAST", and ALLOW_INTERNET_ACCESS. Any miss falls through to Reproduction 1. The except swallows failures with no user-visible signal, so a DuckDuckGo rate-limit degrades straight to confident hallucination.

2. Production runs the smallest model on the box, for everything

Production secrets:

OLLAMA_BASE_URL=http://10.0.0.128:11434
OLLAMA_MODEL=llama3.2
OLLAMA_EMBED_MODEL=llama3.2

llama3.2:latest is 3.2B params, Q4_K_M, 2.0 GB. It backs every LLM call: chat generation, moderation, prompt classification, title generation, RAG synthesis, and data analysis.

Meanwhile settings.py:154-157 defaults to gpt-oss:20b when DEBUG is on. Development runs a 20B thinking model; production runs a 3B model. Nothing that works in dev is validated against what users actually hit.

Available on the GPU host today (/api/tags):

Model Params Size Capabilities Context
gpt-oss:20b 20.9B 13.8 GB completion, tools, thinking 131072
llama3.3:latest 70.6B 42.5 GB completion, tools 131072
gemma4:latest 8.0B 9.6 GB completion, tools, thinking —
qwen2.5-coder:7b 7.6B 4.7 GB completion, tools, insert 32768
llama3.2:latest 3.2B 2.0 GB completion, tools 131072
nomic-embed-text 137M 274 MB embedding (768d) 2048
embeddinggemma 308M 622 MB embedding (768d) 2048

3. A causal LM is being used as the embedding model

OLLAMA_EMBED_MODEL=llama3.2 means RAG embeddings come from mean-pooled hidden states of a chat model, while two purpose-built embedding models (nomic-embed-text, embeddinggemma) sit unused on the same host. This degrades every retrieval in the RAG path. settings.py:158 makes this the silent default: OLLAMA_EMBED_MODEL = env("OLLAMA_EMBED_MODEL", OLLAMA_MODEL).

4. num_ctx=4096 hard-coded, and history is rendered into the prompt twice

services/llm_service.py:19-26 pins num_ctx=4096 even though llama3.2 and gpt-oss:20b both advertise 131072.

Worse, _setup_chain (llm_service.py:92-123) fills {context} from _format_history(conversation) (all messages) and {recent_history} from _get_recent_messages(conversation[-6:]). The search blob is a member of messages, so it is serialised into the prompt twice. On a long conversation this overflows 4096 tokens and Ollama truncates from the front — dropping the system prompt and the search block while keeping the trailing question.

Secondary: the template says "Last 3 messages" but slices [-6:].

5. No citations anywhere

Neither the web-search path nor the RAG path attaches source URLs, titles, or dates. Users cannot tell a grounded answer from a fabricated one — which is precisely how Reproduction 1 run 1 slips through.

6. Metrics cannot distinguish models

consumers.py:55 and consumers_graph.py:46 hard-code model_name="llama3.2" on PromptMetric. Any model change is invisible in analytics, so we cannot measure whether this ticket worked.

Proposed implementation

Phase 1 — Configuration and model routing

1.1 Split model configuration by role. Replace the single OLLAMA_MODEL with role-scoped settings in llm_be/llm_be/settings.py, each independently overridable:

OLLAMA_MODEL_THINKING  = env("OLLAMA_MODEL_THINKING",  "gpt-oss:20b")
OLLAMA_MODEL_FAST      = env("OLLAMA_MODEL_FAST",      "gemma4:latest")
OLLAMA_MODEL_UTILITY   = env("OLLAMA_MODEL_UTILITY",   "llama3.2")   # classify/moderate/title
OLLAMA_EMBED_MODEL     = env("OLLAMA_EMBED_MODEL",     "nomic-embed-text")

Keep OLLAMA_MODEL honoured as a fallback for all four so existing deploys do not break. Remove the DEBUG-conditional default at settings.py:154-157 so dev and prod resolve identically unless explicitly overridden.

Rationale for OLLAMA_MODEL_UTILITY=llama3.2: classification was measured stable on the small model (see Testing below), and keeping it small preserves latency on the hot path.

1.2 Confirm VRAM headroom on 10.0.0.128 before promoting gpt-oss:20b (13.8 GB) as the THINKING default, accounting for a concurrently loaded FAST model and the embedding model. If headroom is tight, set OLLAMA_KEEP_ALIVE and stagger, or fall back to gemma4:latest for THINKING. Record the measured numbers on this ticket.

1.3 Migrate embeddings to nomic-embed-text. This changes vector dimensionality, so existing Chroma collections are invalid. Ship a management command reindex_embeddings that recreates the collection and re-ingests every active Document, preserving the workspace_id / company_id / document_id / active / source metadata written in rag_services.py:118-124. Refuse to start (loud error, not a silent fallback) if the persisted collection dimensionality does not match the configured embedding model.

1.4 Thread the real model name into metrics. Replace the hard-coded "llama3.2" at consumers.py:55 and consumers_graph.py:46 with the model actually resolved for the request. Without this, none of the acceptance criteria below are measurable.

Phase 2 — Always-on grounded retrieval

2.1 Invert the default: retrieval is on unless proven unnecessary. Replace the PromptType.SEARCH-only trigger with an explicit groundedness decision that runs for every turn. A query needs retrieval unless it is self-contained — creative writing, math, code, chit-chat, or a pure follow-up on text already in the conversation.

Implement as a small dedicated GroundingDecider on OLLAMA_MODEL_UTILITY at temperature=0.0, returning strict JSON:

{"needs_retrieval": true, "reason": "asks about a real-world fact that can change", "queries": ["Taylor Swift Travis Kelce wedding date"]}

Bias the prompt explicitly toward retrieval: any question about a person, organisation, product, price, event, date, statistic, or anything post-training-cutoff must set needs_retrieval=true. Add a deterministic pre-pass that forces true on temporal markers (latest, current, today, now, this year, did … yet, a year >= training cutoff) so the model cannot veto obvious cases. On parse failure or timeout, fail open to retrieval — the expensive error is skipping search, not running it.

2.2 Remove the FAST search bypass. FAST must select a smaller/faster model, not disable grounding. This directly fixes "Flash mode can't answer because it was trained before it happened." Delete the if input_dict.get("model_name") == "FAST": pass branch in both consumers.

2.3 Generate multiple focused search queries rather than passing the raw user message to the search engine. queries from 2.1, capped at 3, executed concurrently.

2.4 Replace DuckDuckGoSearchRun with a structured, pluggable search layer. Introduce services/search/ with a SearchProvider protocol returning structured results — title, url, snippet, published_at, rank — never a flat string. Implement:

  • DDGSProvider — use ddgs directly (already a dependency, pyproject.toml:44) so we get per-result fields instead of the concatenated blob DuckDuckGoSearchRun produces.
  • SearxNGProvider — self-hosted, no rate limit, deterministic. Recommended as the production default given the observed DDG variance and the Impersonate 'edge_131' does not exist, using 'random' fallback warnings.

Select via SEARCH_PROVIDER env var, with automatic failover to the secondary provider.

2.5 Rank, deduplicate, and date-filter results. Deduplicate by registrable domain, prefer results with a parseable publication date, and for temporally-marked queries sort recent-first. Drop results whose snippet is dominated by hedging/rumour markers (rumor, speculation, AI-generated, blind item, fake) when better-scored alternatives exist — this is what would have suppressed the Reproduction 2 run-1 poisoning.

2.6 Inject results as a delimited, numbered, dated context block — not f"Search Results: {blob}":

[1] "Taylor Swift and Travis Kelce Are Married" — people.com — 2026-07-03
    Taylor Swift and Travis Kelce, both 36, married on Friday, July 3 at
    Madison Square Garden in New York City.

[2] ...

Carry the source list alongside the generation so it can be emitted as citations (2.8) rather than being reconstructed from the model's prose.

2.7 Add a grounded-answer system prompt. Extend services/assistant_identity.py with a retrieval-specific instruction: answer only from the numbered sources; cite the source index inline; if the sources do not settle the question, say so explicitly rather than filling the gap from memory; prefer the most recent source on conflict; never state a date or number that does not appear in the sources.

2.8 Stream citations to the client. Emit a structured citations frame in the WebSocket protocol after END_OF_THE_STREAM_ENDER_GAME_42, carrying [{index, title, url, published_at}], and persist it on the assistant Prompt row so citations survive a page reload. Coordinate the frame shape with the frontend progress work (chat_web_app progress-feedback issue) — both add frames to the same stream and should land together.

2.9 Make search failure visible. When retrieval was required but every provider failed, do not silently fall through to parametric generation. Either surface a non-fatal notice alongside a clearly-hedged answer, or return an explicit "couldn't reach live sources" state. Log the failure with the query at WARNING.

Phase 3 — Context window and prompt hygiene

3.1 Make num_ctx configurable per role (OLLAMA_NUM_CTX_THINKING, OLLAMA_NUM_CTX_FAST), defaulting to at least 16384 for the THINKING path. Remove the hard-coded 4096 in llm_service.py:25.

3.2 Stop double-rendering history. {context} and {recent_history} currently serialise overlapping message sets. Send one windowed history. Fix the "Last 3 messages" / [-6:] mismatch.

3.3 Budget the prompt. Compute an approximate token budget and trim oldest history first, never the system prompt and never the retrieved-source block. Today truncation eats exactly the parts that make the answer correct.

3.4 Lower sampling temperature on factual turns. When the grounding decider says needs_retrieval=true, generate at temperature<=0.3. 0.7 on a factual lookup is a large part of the run-to-run variance in Reproduction 1.

Phase 4 — Evaluation harness

4.1 Add llm_be/chat_backend/evals/ with a YAML/JSON set of at least 40 graded questions: post-cutoff facts (the Taylor Swift case included verbatim), stable facts that must not regress, questions the system should refuse or hedge, RAG questions against fixture documents, and multi-turn follow-ups.

4.2 Add a run_evals management command that executes the suite N times (default 3) per question against a configurable model and reports accuracy, self-consistency (identical verdict across runs), citation coverage, hallucinated-fact rate, and p50/p95 latency.

4.3 Wire it into .gitea/workflows/ as a manually-triggered job — it needs the GPU host, so it should not gate every PR, but it must be one click before any model or prompt change ships.

Acceptance criteria

Accuracy and grounding

  • The exact prompt "did Taylor Swift get married" returns the correct answer (married Travis Kelce, 2026-07-03, Madison Square Garden) on 10 out of 10 consecutive runs, in both FAST and THINKING modes.
  • The same prompt returns the correct answer with at least one search provider forced offline, exercising failover.
  • With all search providers forced offline, the system explicitly states it could not reach live sources. It must not answer from parametric memory, and must never produce the fabricated "married Joe Alwyn on March 18, 2023" class of response.
  • Eval suite accuracy on post-cutoff factual questions is >= 90%, up from the current measured baseline (record the baseline on this ticket before changing anything).
  • Eval suite self-consistency is >= 95% across 3 runs per question.
  • Stable-fact questions show no regression against baseline.

Retrieval behaviour

  • Grounding is evaluated on every turn; PromptType.SEARCH is no longer the sole trigger.
  • FAST mode never disables retrieval. Verified by test asserting the removed bypass.
  • The grounding decider fails open — a timeout, exception, or unparseable response results in retrieval running. Covered by unit test.
  • Temporal-marker pre-pass forces retrieval regardless of model output. Covered by unit test over a table of phrasings.
  • Search results reach the prompt as structured, numbered, dated, delimited entries with URLs — never as a single concatenated string. Asserted on the rendered prompt.
  • Provider failover is covered by a test that makes the primary raise and asserts the secondary is used.
  • Retrieval failure after all providers is surfaced to the user, not swallowed.

Citations

  • Every answer produced from retrieval carries at least one citation with a resolvable URL.
  • Citations are emitted as a structured WebSocket frame and persisted on the assistant Prompt, surviving reload.
  • RAG answers cite source documents using the existing source metadata.

Models and configuration

  • OLLAMA_MODEL_THINKING, OLLAMA_MODEL_FAST, OLLAMA_MODEL_UTILITY, OLLAMA_EMBED_MODEL are independently configurable and documented in .env.example, .env.prod.example, and README.md.
  • OLLAMA_EMBED_MODEL defaults to nomic-embed-text and never silently falls back to a chat model.
  • Startup fails loudly if the persisted Chroma collection dimensionality does not match the configured embedding model.
  • reindex_embeddings re-ingests all active documents with metadata preserved, verified against a fixture workspace.
  • Dev and prod resolve to the same model defaults; the DEBUG-conditional default is removed.
  • VRAM headroom on 10.0.0.128 is measured and recorded on this ticket for the chosen THINKING + FAST + embedding combination.
  • PromptMetric.model_name records the model actually used, in both consumers. No hard-coded "llama3.2" remains.

Context handling

  • num_ctx is configurable per role and defaults to >= 16384 for THINKING.
  • Conversation history is serialised into the prompt exactly once.
  • Under prompt-budget pressure, oldest history is dropped first; the system prompt and retrieved-source block are never truncated. Covered by a test with an oversized history.
  • Factual (retrieval-backed) turns generate at temperature <= 0.3.

Tooling

  • run_evals reports accuracy, self-consistency, citation coverage, hallucinated-fact rate, and p50/p95 latency.
  • The eval suite contains >= 40 questions across all five listed categories.
  • A manually-triggered Gitea workflow runs the suite against the GPU host.
  • Existing offline unit tests still pass without a live Ollama or network (extend the SKIP_RAG_INIT pattern to search).

Testing notes

Measurements already taken against http://10.0.0.128:11434 on 2026-08-02, for reference when building the baseline:

  • Prompt classifier is not the bottleneck. llama3.2 at temperature=0.1 returned SEARCH on 15/15 runs across three phrasings of the Taylor Swift question. Classification is stable and correct; the failure is entirely downstream. Keeping the small model for utility roles is safe.
  • Grounded generation is not the bottleneck either. Four end-to-end runs with a fresh successful search all produced the correct answer.
  • The failure is the ungrounded path and the unstructured blob. See Reproductions 1 and 2.

Out of scope

  • Multi-step agentic tool loops — separate ticket.
  • Frontend rendering of citations beyond agreeing the WebSocket frame shape — see the chat_web_app progress-feedback issue.
  • Broader file-format parsing — see the file-parsing coverage issue.
## Priority **P0 — highest priority.** Accuracy is the top concern for this cycle. Ship this before the agentic work (see the agentic execution ticket). --- ## Problem The same factual question returns different — and often fabricated — answers on repeated asks. Reproduced end-to-end against the production Ollama host (`http://10.0.0.128:11434`) with the production model. ### Reproduction 1 — no-search path fabricates, inconsistently Prompt `"did Taylor Swift get married"` sent straight to `llama3.2` with the production sampling params (`temperature=0.7, top_k=50, top_p=0.9, repeat_penalty=1.1, num_ctx=4096`), three consecutive runs: ``` run 0: "As of my knowledge cutoff in December 2023, Taylor Swift is not married." run 1: "Yes, Taylor Swift is married. She tied the knot with Joe Alwyn, a British actor, on March 18, 2023." <-- entirely fabricated run 2: "Taylor Swift is not married." ``` This is exactly the reported "three different answers, one correct" behaviour. Run 1 invents a wedding that never happened, with a specific date, stated confidently. ### Reproduction 2 — search results are non-deterministic and sometimes poisoned `DuckDuckGoSearchRun().run("did Taylor Swift get married")`, three consecutive calls, returned three materially different blobs: - **run 0** (1082 chars) — correct: married Travis Kelce, July 3, Madison Square Garden. - **run 1** (714 chars) — poisoned: *"fake, AI-generated photos of Taylor Swift and Travis Kelce in wedding attire"*, *"Rumors are running wild online following a new blind item claim"*, plus an unrelated couple (Hurtado and Bochman) who married at a Swift concert. A small model handed only this blob will reasonably answer "no, those were fakes / just rumours". - **run 2** (1136 chars) — correct. The blob has no URLs, no per-result delimiters, no publication dates, and no ranking. It is injected verbatim as `HumanMessage(content=f"Search Results: {search_results}")`. ### Reproduction 3 — when search does land cleanly, answers are correct and stable Four end-to-end runs through the real `AsyncLLMService` template with a fresh search each time all produced the correct answer (married Travis Kelce, July 3, Madison Square Garden). **Grounding works. The problem is that grounding is optional, silent when it fails, and unstructured when it succeeds.** --- ## Root causes ### 1. Search is gated behind three independent conditions, any of which silently disables it `llm_be/chat_backend/consumers.py:481-497` (and the mirror in `consumers_graph.py:280-294`): ```python if prompt_type == PromptType.SEARCH: # Check modelName first - if FAST, we skip search regardless of settings if input_dict.get("model_name") == "FAST": pass # Skip search elif getattr(settings, "ALLOW_INTERNET_ACCESS", False): try: search = DuckDuckGoSearchRun() search_results = search.run(input_dict["message"]) messages.append(HumanMessage(content=f"Search Results: {search_results}")) except Exception as e: logger.error(f"Search failed: {e}") pass ``` Search runs only if **all** of: classifier returns `SEARCH`, **and** `modelName != "FAST"`, **and** `ALLOW_INTERNET_ACCESS`. Any miss falls through to Reproduction 1. The `except` swallows failures with no user-visible signal, so a DuckDuckGo rate-limit degrades straight to confident hallucination. ### 2. Production runs the smallest model on the box, for everything Production secrets: ``` OLLAMA_BASE_URL=http://10.0.0.128:11434 OLLAMA_MODEL=llama3.2 OLLAMA_EMBED_MODEL=llama3.2 ``` `llama3.2:latest` is **3.2B params, Q4_K_M, 2.0 GB**. It backs *every* LLM call: chat generation, moderation, prompt classification, title generation, RAG synthesis, and data analysis. Meanwhile `settings.py:154-157` defaults to `gpt-oss:20b` when `DEBUG` is on. **Development runs a 20B thinking model; production runs a 3B model.** Nothing that works in dev is validated against what users actually hit. Available on the GPU host today (`/api/tags`): | Model | Params | Size | Capabilities | Context | |---|---|---|---|---| | `gpt-oss:20b` | 20.9B | 13.8 GB | completion, **tools**, **thinking** | 131072 | | `llama3.3:latest` | 70.6B | 42.5 GB | completion, tools | 131072 | | `gemma4:latest` | 8.0B | 9.6 GB | completion, **tools**, **thinking** | — | | `qwen2.5-coder:7b` | 7.6B | 4.7 GB | completion, tools, insert | 32768 | | `llama3.2:latest` | 3.2B | 2.0 GB | completion, tools | 131072 | | `nomic-embed-text` | 137M | 274 MB | **embedding** (768d) | 2048 | | `embeddinggemma` | 308M | 622 MB | **embedding** (768d) | 2048 | ### 3. A causal LM is being used as the embedding model `OLLAMA_EMBED_MODEL=llama3.2` means RAG embeddings come from mean-pooled hidden states of a chat model, while two purpose-built embedding models (`nomic-embed-text`, `embeddinggemma`) sit unused on the same host. This degrades every retrieval in the RAG path. `settings.py:158` makes this the silent default: `OLLAMA_EMBED_MODEL = env("OLLAMA_EMBED_MODEL", OLLAMA_MODEL)`. ### 4. `num_ctx=4096` hard-coded, and history is rendered into the prompt twice `services/llm_service.py:19-26` pins `num_ctx=4096` even though `llama3.2` and `gpt-oss:20b` both advertise 131072. Worse, `_setup_chain` (`llm_service.py:92-123`) fills `{context}` from `_format_history(conversation)` (**all** messages) and `{recent_history}` from `_get_recent_messages(conversation[-6:])`. The search blob is a member of `messages`, so it is serialised into the prompt **twice**. On a long conversation this overflows 4096 tokens and Ollama truncates from the front — dropping the system prompt and the search block while keeping the trailing question. Secondary: the template says "Last 3 messages" but slices `[-6:]`. ### 5. No citations anywhere Neither the web-search path nor the RAG path attaches source URLs, titles, or dates. Users cannot tell a grounded answer from a fabricated one — which is precisely how Reproduction 1 run 1 slips through. ### 6. Metrics cannot distinguish models `consumers.py:55` and `consumers_graph.py:46` hard-code `model_name="llama3.2"` on `PromptMetric`. Any model change is invisible in analytics, so we cannot measure whether this ticket worked. --- ## Proposed implementation ### Phase 1 — Configuration and model routing **1.1 Split model configuration by role.** Replace the single `OLLAMA_MODEL` with role-scoped settings in `llm_be/llm_be/settings.py`, each independently overridable: ```python OLLAMA_MODEL_THINKING = env("OLLAMA_MODEL_THINKING", "gpt-oss:20b") OLLAMA_MODEL_FAST = env("OLLAMA_MODEL_FAST", "gemma4:latest") OLLAMA_MODEL_UTILITY = env("OLLAMA_MODEL_UTILITY", "llama3.2") # classify/moderate/title OLLAMA_EMBED_MODEL = env("OLLAMA_EMBED_MODEL", "nomic-embed-text") ``` Keep `OLLAMA_MODEL` honoured as a fallback for all four so existing deploys do not break. Remove the `DEBUG`-conditional default at `settings.py:154-157` so dev and prod resolve identically unless explicitly overridden. Rationale for `OLLAMA_MODEL_UTILITY=llama3.2`: classification was measured stable on the small model (see Testing below), and keeping it small preserves latency on the hot path. **1.2 Confirm VRAM headroom on 10.0.0.128** before promoting `gpt-oss:20b` (13.8 GB) as the THINKING default, accounting for a concurrently loaded FAST model and the embedding model. If headroom is tight, set `OLLAMA_KEEP_ALIVE` and stagger, or fall back to `gemma4:latest` for THINKING. Record the measured numbers on this ticket. **1.3 Migrate embeddings to `nomic-embed-text`.** This changes vector dimensionality, so existing Chroma collections are invalid. Ship a management command `reindex_embeddings` that recreates the collection and re-ingests every active `Document`, preserving the `workspace_id` / `company_id` / `document_id` / `active` / `source` metadata written in `rag_services.py:118-124`. Refuse to start (loud error, not a silent fallback) if the persisted collection dimensionality does not match the configured embedding model. **1.4 Thread the real model name into metrics.** Replace the hard-coded `"llama3.2"` at `consumers.py:55` and `consumers_graph.py:46` with the model actually resolved for the request. Without this, none of the acceptance criteria below are measurable. ### Phase 2 — Always-on grounded retrieval **2.1 Invert the default: retrieval is on unless proven unnecessary.** Replace the `PromptType.SEARCH`-only trigger with an explicit *groundedness* decision that runs for every turn. A query needs retrieval unless it is self-contained — creative writing, math, code, chit-chat, or a pure follow-up on text already in the conversation. Implement as a small dedicated `GroundingDecider` on `OLLAMA_MODEL_UTILITY` at `temperature=0.0`, returning strict JSON: ```json {"needs_retrieval": true, "reason": "asks about a real-world fact that can change", "queries": ["Taylor Swift Travis Kelce wedding date"]} ``` Bias the prompt explicitly toward retrieval: any question about a person, organisation, product, price, event, date, statistic, or anything post-training-cutoff must set `needs_retrieval=true`. Add a deterministic pre-pass that forces `true` on temporal markers (`latest`, `current`, `today`, `now`, `this year`, `did … yet`, a year >= training cutoff) so the model cannot veto obvious cases. On parse failure or timeout, **fail open to retrieval** — the expensive error is skipping search, not running it. **2.2 Remove the `FAST` search bypass.** FAST must select a smaller/faster model, not disable grounding. This directly fixes "Flash mode can't answer because it was trained before it happened." Delete the `if input_dict.get("model_name") == "FAST": pass` branch in both consumers. **2.3 Generate multiple focused search queries** rather than passing the raw user message to the search engine. `queries` from 2.1, capped at 3, executed concurrently. **2.4 Replace `DuckDuckGoSearchRun` with a structured, pluggable search layer.** Introduce `services/search/` with a `SearchProvider` protocol returning structured results — `title`, `url`, `snippet`, `published_at`, `rank` — never a flat string. Implement: - `DDGSProvider` — use `ddgs` directly (already a dependency, `pyproject.toml:44`) so we get per-result fields instead of the concatenated blob `DuckDuckGoSearchRun` produces. - `SearxNGProvider` — self-hosted, no rate limit, deterministic. Recommended as the production default given the observed DDG variance and the `Impersonate 'edge_131' does not exist, using 'random'` fallback warnings. Select via `SEARCH_PROVIDER` env var, with automatic failover to the secondary provider. **2.5 Rank, deduplicate, and date-filter results.** Deduplicate by registrable domain, prefer results with a parseable publication date, and for temporally-marked queries sort recent-first. Drop results whose snippet is dominated by hedging/rumour markers (`rumor`, `speculation`, `AI-generated`, `blind item`, `fake`) when better-scored alternatives exist — this is what would have suppressed the Reproduction 2 run-1 poisoning. **2.6 Inject results as a delimited, numbered, dated context block** — not `f"Search Results: {blob}"`: ``` [1] "Taylor Swift and Travis Kelce Are Married" — people.com — 2026-07-03 Taylor Swift and Travis Kelce, both 36, married on Friday, July 3 at Madison Square Garden in New York City. [2] ... ``` Carry the source list alongside the generation so it can be emitted as citations (2.8) rather than being reconstructed from the model's prose. **2.7 Add a grounded-answer system prompt.** Extend `services/assistant_identity.py` with a retrieval-specific instruction: answer only from the numbered sources; cite the source index inline; if the sources do not settle the question, say so explicitly rather than filling the gap from memory; prefer the most recent source on conflict; never state a date or number that does not appear in the sources. **2.8 Stream citations to the client.** Emit a structured citations frame in the WebSocket protocol after `END_OF_THE_STREAM_ENDER_GAME_42`, carrying `[{index, title, url, published_at}]`, and persist it on the assistant `Prompt` row so citations survive a page reload. Coordinate the frame shape with the frontend progress work (`chat_web_app` progress-feedback issue) — both add frames to the same stream and should land together. **2.9 Make search failure visible.** When retrieval was required but every provider failed, do **not** silently fall through to parametric generation. Either surface a non-fatal notice alongside a clearly-hedged answer, or return an explicit "couldn't reach live sources" state. Log the failure with the query at `WARNING`. ### Phase 3 — Context window and prompt hygiene **3.1 Make `num_ctx` configurable per role** (`OLLAMA_NUM_CTX_THINKING`, `OLLAMA_NUM_CTX_FAST`), defaulting to at least 16384 for the THINKING path. Remove the hard-coded `4096` in `llm_service.py:25`. **3.2 Stop double-rendering history.** `{context}` and `{recent_history}` currently serialise overlapping message sets. Send one windowed history. Fix the "Last 3 messages" / `[-6:]` mismatch. **3.3 Budget the prompt.** Compute an approximate token budget and trim *oldest history first*, never the system prompt and never the retrieved-source block. Today truncation eats exactly the parts that make the answer correct. **3.4 Lower sampling temperature on factual turns.** When the grounding decider says `needs_retrieval=true`, generate at `temperature<=0.3`. `0.7` on a factual lookup is a large part of the run-to-run variance in Reproduction 1. ### Phase 4 — Evaluation harness **4.1 Add `llm_be/chat_backend/evals/`** with a YAML/JSON set of at least 40 graded questions: post-cutoff facts (the Taylor Swift case included verbatim), stable facts that must *not* regress, questions the system should refuse or hedge, RAG questions against fixture documents, and multi-turn follow-ups. **4.2 Add a `run_evals` management command** that executes the suite N times (default 3) per question against a configurable model and reports **accuracy**, **self-consistency** (identical verdict across runs), **citation coverage**, **hallucinated-fact rate**, and **p50/p95 latency**. **4.3 Wire it into `.gitea/workflows/` as a manually-triggered job** — it needs the GPU host, so it should not gate every PR, but it must be one click before any model or prompt change ships. --- ## Acceptance criteria ### Accuracy and grounding - [ ] The exact prompt `"did Taylor Swift get married"` returns the correct answer (married Travis Kelce, 2026-07-03, Madison Square Garden) on **10 out of 10** consecutive runs, in **both** FAST and THINKING modes. - [ ] The same prompt returns the correct answer with at least one search provider forced offline, exercising failover. - [ ] With **all** search providers forced offline, the system explicitly states it could not reach live sources. It must **not** answer from parametric memory, and must never produce the fabricated "married Joe Alwyn on March 18, 2023" class of response. - [ ] Eval suite accuracy on post-cutoff factual questions is **>= 90%**, up from the current measured baseline (record the baseline on this ticket before changing anything). - [ ] Eval suite self-consistency is **>= 95%** across 3 runs per question. - [ ] Stable-fact questions show **no regression** against baseline. ### Retrieval behaviour - [ ] Grounding is evaluated on every turn; `PromptType.SEARCH` is no longer the sole trigger. - [ ] `FAST` mode never disables retrieval. Verified by test asserting the removed bypass. - [ ] The grounding decider fails **open** — a timeout, exception, or unparseable response results in retrieval running. Covered by unit test. - [ ] Temporal-marker pre-pass forces retrieval regardless of model output. Covered by unit test over a table of phrasings. - [ ] Search results reach the prompt as structured, numbered, dated, delimited entries with URLs — never as a single concatenated string. Asserted on the rendered prompt. - [ ] Provider failover is covered by a test that makes the primary raise and asserts the secondary is used. - [ ] Retrieval failure after all providers is surfaced to the user, not swallowed. ### Citations - [ ] Every answer produced from retrieval carries at least one citation with a resolvable URL. - [ ] Citations are emitted as a structured WebSocket frame and persisted on the assistant `Prompt`, surviving reload. - [ ] RAG answers cite source documents using the existing `source` metadata. ### Models and configuration - [ ] `OLLAMA_MODEL_THINKING`, `OLLAMA_MODEL_FAST`, `OLLAMA_MODEL_UTILITY`, `OLLAMA_EMBED_MODEL` are independently configurable and documented in `.env.example`, `.env.prod.example`, and `README.md`. - [ ] `OLLAMA_EMBED_MODEL` defaults to `nomic-embed-text` and **never** silently falls back to a chat model. - [ ] Startup fails loudly if the persisted Chroma collection dimensionality does not match the configured embedding model. - [ ] `reindex_embeddings` re-ingests all active documents with metadata preserved, verified against a fixture workspace. - [ ] Dev and prod resolve to the same model defaults; the `DEBUG`-conditional default is removed. - [ ] VRAM headroom on 10.0.0.128 is measured and recorded on this ticket for the chosen THINKING + FAST + embedding combination. - [ ] `PromptMetric.model_name` records the model actually used, in both consumers. No hard-coded `"llama3.2"` remains. ### Context handling - [ ] `num_ctx` is configurable per role and defaults to >= 16384 for THINKING. - [ ] Conversation history is serialised into the prompt exactly once. - [ ] Under prompt-budget pressure, oldest history is dropped first; the system prompt and retrieved-source block are never truncated. Covered by a test with an oversized history. - [ ] Factual (retrieval-backed) turns generate at `temperature <= 0.3`. ### Tooling - [ ] `run_evals` reports accuracy, self-consistency, citation coverage, hallucinated-fact rate, and p50/p95 latency. - [ ] The eval suite contains >= 40 questions across all five listed categories. - [ ] A manually-triggered Gitea workflow runs the suite against the GPU host. - [ ] Existing offline unit tests still pass without a live Ollama or network (extend the `SKIP_RAG_INIT` pattern to search). --- ## Testing notes Measurements already taken against `http://10.0.0.128:11434` on 2026-08-02, for reference when building the baseline: - **Prompt classifier is not the bottleneck.** `llama3.2` at `temperature=0.1` returned `SEARCH` on **15/15** runs across three phrasings of the Taylor Swift question. Classification is stable and correct; the failure is entirely downstream. Keeping the small model for utility roles is safe. - **Grounded generation is not the bottleneck either.** Four end-to-end runs with a fresh successful search all produced the correct answer. - **The failure is the ungrounded path and the unstructured blob.** See Reproductions 1 and 2. --- ## Out of scope - Multi-step agentic tool loops — separate ticket. - Frontend rendering of citations beyond agreeing the WebSocket frame shape — see the `chat_web_app` progress-feedback issue. - Broader file-format parsing — see the file-parsing coverage issue.
westfarn commented 2026-08-02 12:12:25 -07:00
Author
Owner
Copy Link
Copy Source

Frontend citation rendering ticket: chat_web_app#98 — Sources list + clickable [n] for the WS/persisted citations frame from this epic.

Also related: activity frames chat_web_app#96 (same versioned envelope; citations out of scope there).

Frontend citation rendering ticket: [chat_web_app#98](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/98) — Sources list + clickable `[n]` for the WS/persisted citations frame from this epic. Also related: activity frames [chat_web_app#96](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/96) (same versioned envelope; citations out of scope there).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#62