ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

[TICKET] Long-running agentic task execution: tool-calling loop, sub-agents, and durable runs #63

New Issue
Open
opened 2026-08-02 06:47:08 -07:00 by westfarn · 1 comment
westfarn commented 2026-08-02 06:47:08 -07:00
Owner
Copy Link
Copy Source

Priority

P1. Depends on #62 (accuracy overhaul). #62 establishes the model routing, structured search layer, and citation plumbing this ticket builds on. Start design in parallel, but land #62 first.

Goal

Support multi-step, long-running tasks the way Gemini and ChatGPT do — the assistant decomposes a request into steps, spawns sub-tasks, calls tools, and rolls the results up into a single synthesised answer, with the user watching progress the whole time.

Two concrete examples that must work end to end:

  • "Research the top 5 competitors to our product, summarise each one's pricing, and give me a comparison table." — needs N parallel research sub-tasks, each doing its own searches, then a synthesis pass.
  • "Read the three PDFs in my workspace, extract every contractual deadline, and tell me which ones fall in Q4." — needs document retrieval, per-document extraction, and a filtering/aggregation pass.

Neither is possible today.

Current state

There is no agent loop

consumers_graph.py:329-341 is a fixed three-node LangGraph, executed exactly once:

workflow = StateGraph(ChatState)
workflow.add_node("moderation", moderation_node)
workflow.add_node("classification", classification_node)
workflow.add_node("generation", generation_node)
workflow.set_entry_point("moderation")
workflow.add_edge("moderation", "classification")
workflow.add_edge("classification", "generation")
workflow.add_edge("generation", END)
app = workflow.compile()

No conditional edges, no cycles, no ToolNode. The model cannot decide to call a tool, cannot iterate, and cannot revisit a step.

Tools are hardcoded call sites, not bound tools

Web search, RAG retrieval, plotting, and (stubbed) image generation are all if-branches inside generate_response_step. None are registered as tool schemas; the model is never given a tool list and never emits a tool call. OllamaLLM (the text-completion class) is used throughout — it cannot do tool calling at all. Tool calling requires ChatOllama.

There is no durable execution

settings.py:291-297 uses the Django Tasks ImmediateBackend — tasks run in-process, synchronously:

TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}}

CHANNEL_LAYERS is InMemoryChannelLayer (settings.py:279-283), so progress cannot be fanned out across processes. A long task is bound to one WebSocket connection in one worker: refresh the page or drop the connection and the work is lost. drive_tasks.py:35-109 already works around this by spawning a raw daemon thread when it detects ImmediateBackend — a sign the infrastructure gap is already being felt.

There is no way to show progress

The stream protocol is CONVERSATION_ID → id → START_OF_THE_STREAM_ENDER_GAME_42 → raw text chunks → END_OF_THE_STREAM_ENDER_GAME_42. There is no frame type for "started step 2 of 5".

Which Ollama models can do this

Queried http://10.0.0.128:11434/api/tags on 2026-08-02. The capabilities array is authoritative — a model needs tools to participate in an agent loop, and thinking gives traceable intermediate reasoning.

Model Params Size tools thinking Context Verdict for agentic use
gpt-oss:20b 20.9B 13.8 GB yes yes 131072 Recommended orchestrator. Best installed combination of tool calling, reasoning traces, and context. Caveat: no parallel tool calling — emits one tool call per turn (ollama/ollama#12159), so the loop must serialise or fan out at the sub-agent layer.
gemma4:latest 8.0B 9.6 GB yes yes — Recommended sub-agent worker. Small enough to run several concurrently; thinking mode aids debugging.
llama3.3:latest 70.6B 42.5 GB yes no 131072 Strong tool selection, but 42.5 GB and no thinking traces. Only viable if VRAM allows and it does not evict the workers.
qwen2.5-coder:7b 7.6B 4.7 GB yes no 32768 Narrow. Reserve for a future code-execution tool.
llama3.2:latest 3.2B 2.0 GB yes no 131072 Advertises tools but too small for reliable multi-step planning. Keep for utility roles only (see #62).
nomic-embed-text 137M 274 MB — — 2048 Embeddings only.
embeddinggemma 308M 622 MB — — 2048 Embeddings only.

Worth evaluating but not installed: the Qwen 3.x family is widely reported as the strongest open-weight tool-caller per parameter and supports genuine parallel tool calling, which gpt-oss:20b does not. Pull qwen3 (or the current 3.x MoE variant that fits the host's VRAM) and bench it against gpt-oss:20b on the tool-selection suite in Phase 5 before locking the orchestrator default.

Starting recommendation: orchestrator gpt-oss:20b, sub-agent workers gemma4:latest, with both overridable by env var and the choice revisited once the Phase 5 benchmark exists.

Proposed implementation

Phase 1 — Durable execution infrastructure

This is the prerequisite. Without it, a five-minute task cannot survive a page refresh.

1.1 Replace InMemoryChannelLayer with channels_redis.core.RedisChannelLayer. Required for any cross-process progress fan-out. Add REDIS_URL to settings and both env examples.

1.2 Replace the Django Tasks ImmediateBackend with a real worker backend (Celery with a Redis broker, or the Django Tasks database backend plus a worker process, whichever fits the existing deploy). Add the worker to the systemd/compose units used by .gitea/workflows/deploy-beta.yml and deploy-prod.yml. Remove the daemon-thread workaround in drive_tasks.py:35-109 once a real backend exists.

1.3 Add an AgentRun model persisting the full lifecycle so a run is recoverable independent of any connection:

  • id, conversation, user, status (queued / planning / running / synthesising / succeeded / failed / cancelled), goal, plan (JSON), created_at, started_at, finished_at, error
  • total_tokens_in / total_tokens_out, model_name — quotas must cover agent runs, not just single turns

1.4 Add AgentStep — run (FK), index, parent_step (FK, nullable, for sub-agent nesting), kind (plan / tool_call / sub_agent / synthesis), title (short human-readable label for the UI), status, tool_name, tool_input (JSON), tool_output (JSON), started_at, finished_at, error, tokens_in, tokens_out.

1.5 Add REST endpoints GET /api/agent_runs and GET /api/agent_runs/<id> returning the run with its nested steps, so the frontend can rehydrate after a reload rather than depending on having been connected.

Phase 2 — Tool-calling layer

2.1 Switch the agent path from OllamaLLM to ChatOllama. OllamaLLM is a text-completion wrapper with no bind_tools. This is a hard blocker — nothing else in this ticket works without it. Leave the existing non-agent chat path on its current class until #62 has settled, then converge.

2.2 Create services/tools/ with a registry of @tool-decorated callables, each with a typed Pydantic argument schema and a clear docstring (the docstring is the tool description the model sees, so it materially affects selection accuracy). Initial set:

Tool Backed by
web_search(query, recency_days=None) the structured SearchProvider layer from #62
fetch_url(url) new; fetch and extract readable text, with size cap and SSRF guard
search_documents(query, workspace_id) existing Chroma retriever in rag_services.py
read_document(document_id) existing Document storage
analyse_dataframe(document_id, question) existing AsyncDataAnalysisService
make_plot(document_id, spec) existing _generate_plot in data_analysis_service.py:103-145

Every tool must be tenant-scoped through the existing resolve_chat_company_scope() / _workspace_filter path. A tool must never be able to read across workspaces — this is the same class of bug fixed in the stove-piping work, and an agent loop is a much larger attack surface than a single turn.

2.3 Enforce per-tool timeouts, output size caps, and a per-run tool-call budget. Truncate oversized tool output with an explicit marker rather than letting it blow the context window.

Phase 3 — Orchestrator: plan → execute → synthesise

3.1 Add a planner node that turns the goal into an ordered list of steps, each with a title, an assigned tool or sub-agent, and its dependencies. Emit strict JSON. Cap the plan at a configurable maximum (default 8 steps). For a simple conversational turn the planner must return a single-step plan so the fast path stays fast — the agent pipeline must not add latency to ordinary chat.

3.2 Add an executor node driving the actual agent loop — a cyclic LangGraph with a conditional edge:

planner -> executor -> (tool_calls?) -> tools -> executor   [cycle]
                    -> (no tool_calls?) -> synthesiser -> END

Bound by a max-iteration count (default 12) and a wall-clock deadline (default 10 minutes), both configurable. On breach, stop cleanly and synthesise from whatever completed — never hang, never return nothing.

3.3 Add sub-agents for parallelisable steps. Steps with no interdependency run as concurrent child runs on OLLAMA_MODEL_SUBAGENT, each with its own scratchpad and its own AgentStep subtree via parent_step. Bound concurrency (default 3) so the GPU host is not oversubscribed. Sub-agents return a structured summary plus citations to the parent — not their full transcript, which would flood the orchestrator's context.

This also sidesteps gpt-oss:20b's lack of parallel tool calling: parallelism happens at the sub-agent layer rather than within a single model turn.

3.4 Add a synthesiser node producing the final user-facing answer from the collected step outputs, carrying citations forward from #62 and streaming tokens over the existing stream markers so the final answer renders exactly like a normal reply.

3.5 Add retry with backoff on transient tool failure (default 2 retries). A permanently failed step must be recorded on AgentStep and reported in the final answer rather than silently dropped — same principle as #62's rule against silent search failure.

Phase 4 — Progress streaming

4.1 Extend the WebSocket protocol with JSON frames for run_started, plan_ready (full step list up front, so the UI can render the whole checklist immediately), step_started, step_completed, step_failed, and run_completed. Each carries run_id, step_id, parent_step_id, title, status.

Define the frame shape jointly with the chat_web_app progress-feedback issue and with #62's citation frame — all three add frames to the same stream and must not conflict. A versioned envelope ({"v": 1, "type": ..., "data": ...}) is preferable to more bare sentinel strings like START_OF_THE_STREAM_ENDER_GAME_42.

4.2 Publish progress through the Redis channel layer, keyed by run, so any connected client for that user receives it and a reconnecting client can resume mid-run.

4.3 Add a cancel endpoint / frame. A user must be able to stop a long run, and cancellation must actually halt the worker and mark the run cancelled.

Phase 5 — Evaluation and rollout

5.1 Add a tool-selection benchmark: 30+ prompts with the expected tool sequence, scored on correct tool chosen, correct arguments, and correctly choosing not to call a tool. Run against gpt-oss:20b, gemma4:latest, llama3.3:latest, and a pulled qwen3 to settle the orchestrator default with data rather than the starting recommendation above.

5.2 Add end-to-end agent scenarios covering both worked examples at the top of this ticket, asserting plan shape, tool usage, citation presence, and completion within the deadline.

5.3 Gate the feature. Add ALLOW_AGENTIC_TASKS (default off) alongside the existing ALLOW_INTERNET_ACCESS / ALLOW_IMAGE_GENERATION flags, and a plan-tier feature gate via the existing enforce_feature_gate mechanism — agent runs are far more expensive than single turns and must be tier-limited and quota-counted.

Acceptance criteria

Infrastructure

  • Channel layer is Redis-backed; progress from a worker process reaches a client connected to a different web process.
  • A real (non-immediate) task backend runs agent work in a worker process, deployed by both the beta and prod workflows.
  • AgentRun and AgentStep persist the full lifecycle, including nested sub-agent steps via parent_step.
  • GET /api/agent_runs/<id> returns a run with nested steps and is tenant-scoped.
  • A run survives a page refresh: disconnect mid-run, reload, and both completed and in-flight steps are visible, with the final answer arriving on the reconnected client.
  • The drive_tasks.py daemon-thread workaround is removed.

Tool calling

  • The agent path uses ChatOllama with bind_tools; no agent-path call goes through OllamaLLM.
  • All six Phase 2 tools are registered with typed schemas and docstring descriptions.
  • Every tool is tenant-scoped. A test asserts a tool invoked in workspace A cannot return data from workspace B.
  • fetch_url rejects private/link-local/loopback addresses and enforces a response size cap. Covered by an SSRF test.
  • Per-tool timeout, output cap, and per-run tool-call budget are enforced and configurable.

Orchestration

  • The graph contains a genuine cycle with a conditional edge; the model drives tool invocation.
  • The planner produces a step list with titles and dependencies, capped at the configured maximum.
  • A simple conversational turn produces a single-step plan and shows no measurable latency regression versus today's chat path. This is a hard requirement.
  • Independent steps execute concurrently as sub-agents, bounded by the concurrency limit.
  • Sub-agents return structured summaries plus citations, not raw transcripts.
  • Max-iteration and wall-clock deadline both terminate cleanly and still synthesise a partial answer.
  • Transient tool failures retry with backoff; permanent failures are recorded and surfaced in the final answer.
  • Both worked examples from the Goal section complete end to end with correct tool usage and citations.

Progress streaming

  • All six frame types are emitted in the correct order for a multi-step run.
  • plan_ready delivers the complete step list before execution starts.
  • The frame envelope is versioned and agreed with the chat_web_app progress-feedback issue and #62's citation frame; no conflicting frame types across the three.
  • Frames publish via the Redis channel layer and reach a reconnecting client.
  • Cancellation halts the worker and marks the run cancelled; no orphaned work continues.

Models and gating

  • OLLAMA_MODEL_ORCHESTRATOR and OLLAMA_MODEL_SUBAGENT are configurable and documented in .env.example, .env.prod.example, and README.md.
  • The tool-selection benchmark is run across all four candidate models and the results are recorded on this ticket.
  • The chosen orchestrator scores >= 90% correct tool selection on the benchmark.
  • ALLOW_AGENTIC_TASKS defaults to off; agent runs are plan-gated via enforce_feature_gate and counted against quotas.
  • Token usage for a whole run, including all sub-agents, is aggregated onto AgentRun and reflected in existing usage reporting.

Regression safety

  • With ALLOW_AGENTIC_TASKS=False, chat behaves exactly as it does today.
  • The existing offline unit suite passes without Redis, a worker, a live Ollama, or network access.

Open questions

  • Celery versus the Django Tasks database backend — decide against the existing deploy topology in server-infra before starting Phase 1.
  • Is there VRAM headroom on 10.0.0.128 for an orchestrator plus three concurrent sub-agents plus the embedding model? Measure before setting the concurrency default. Overlaps with the VRAM task in #62.
  • Should sub-agent progress be shown nested in the UI or flattened? Depends on the design in the chat_web_app progress-feedback issue.
## Priority **P1.** Depends on #62 (accuracy overhaul). #62 establishes the model routing, structured search layer, and citation plumbing this ticket builds on. Start design in parallel, but land #62 first. --- ## Goal Support multi-step, long-running tasks the way Gemini and ChatGPT do — the assistant decomposes a request into steps, spawns sub-tasks, calls tools, and rolls the results up into a single synthesised answer, with the user watching progress the whole time. Two concrete examples that must work end to end: - *"Research the top 5 competitors to our product, summarise each one's pricing, and give me a comparison table."* — needs N parallel research sub-tasks, each doing its own searches, then a synthesis pass. - *"Read the three PDFs in my workspace, extract every contractual deadline, and tell me which ones fall in Q4."* — needs document retrieval, per-document extraction, and a filtering/aggregation pass. Neither is possible today. --- ## Current state ### There is no agent loop `consumers_graph.py:329-341` is a fixed three-node LangGraph, executed exactly once: ```python workflow = StateGraph(ChatState) workflow.add_node("moderation", moderation_node) workflow.add_node("classification", classification_node) workflow.add_node("generation", generation_node) workflow.set_entry_point("moderation") workflow.add_edge("moderation", "classification") workflow.add_edge("classification", "generation") workflow.add_edge("generation", END) app = workflow.compile() ``` No conditional edges, no cycles, no `ToolNode`. The model cannot decide to call a tool, cannot iterate, and cannot revisit a step. ### Tools are hardcoded call sites, not bound tools Web search, RAG retrieval, plotting, and (stubbed) image generation are all `if`-branches inside `generate_response_step`. None are registered as tool schemas; the model is never given a tool list and never emits a tool call. `OllamaLLM` (the text-completion class) is used throughout — it cannot do tool calling at all. Tool calling requires `ChatOllama`. ### There is no durable execution `settings.py:291-297` uses the Django Tasks `ImmediateBackend` — tasks run in-process, synchronously: ```python TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}} ``` `CHANNEL_LAYERS` is `InMemoryChannelLayer` (`settings.py:279-283`), so progress cannot be fanned out across processes. A long task is bound to one WebSocket connection in one worker: refresh the page or drop the connection and the work is lost. `drive_tasks.py:35-109` already works around this by spawning a raw daemon thread when it detects `ImmediateBackend` — a sign the infrastructure gap is already being felt. ### There is no way to show progress The stream protocol is `CONVERSATION_ID` → id → `START_OF_THE_STREAM_ENDER_GAME_42` → raw text chunks → `END_OF_THE_STREAM_ENDER_GAME_42`. There is no frame type for "started step 2 of 5". --- ## Which Ollama models can do this Queried `http://10.0.0.128:11434/api/tags` on 2026-08-02. The `capabilities` array is authoritative — a model needs `tools` to participate in an agent loop, and `thinking` gives traceable intermediate reasoning. | Model | Params | Size | `tools` | `thinking` | Context | Verdict for agentic use | |---|---|---|---|---|---|---| | `gpt-oss:20b` | 20.9B | 13.8 GB | yes | **yes** | 131072 | **Recommended orchestrator.** Best installed combination of tool calling, reasoning traces, and context. Caveat: no parallel tool calling — emits one tool call per turn (ollama/ollama#12159), so the loop must serialise or fan out at the sub-agent layer. | | `gemma4:latest` | 8.0B | 9.6 GB | yes | **yes** | — | **Recommended sub-agent worker.** Small enough to run several concurrently; thinking mode aids debugging. | | `llama3.3:latest` | 70.6B | 42.5 GB | yes | no | 131072 | Strong tool selection, but 42.5 GB and no thinking traces. Only viable if VRAM allows and it does not evict the workers. | | `qwen2.5-coder:7b` | 7.6B | 4.7 GB | yes | no | 32768 | Narrow. Reserve for a future code-execution tool. | | `llama3.2:latest` | 3.2B | 2.0 GB | yes | no | 131072 | Advertises `tools` but too small for reliable multi-step planning. Keep for utility roles only (see #62). | | `nomic-embed-text` | 137M | 274 MB | — | — | 2048 | Embeddings only. | | `embeddinggemma` | 308M | 622 MB | — | — | 2048 | Embeddings only. | **Worth evaluating but not installed:** the Qwen 3.x family is widely reported as the strongest open-weight tool-caller per parameter and supports genuine parallel tool calling, which `gpt-oss:20b` does not. Pull `qwen3` (or the current 3.x MoE variant that fits the host's VRAM) and bench it against `gpt-oss:20b` on the tool-selection suite in Phase 5 before locking the orchestrator default. **Starting recommendation:** orchestrator `gpt-oss:20b`, sub-agent workers `gemma4:latest`, with both overridable by env var and the choice revisited once the Phase 5 benchmark exists. --- ## Proposed implementation ### Phase 1 — Durable execution infrastructure This is the prerequisite. Without it, a five-minute task cannot survive a page refresh. **1.1 Replace `InMemoryChannelLayer` with `channels_redis.core.RedisChannelLayer`.** Required for any cross-process progress fan-out. Add `REDIS_URL` to settings and both env examples. **1.2 Replace the Django Tasks `ImmediateBackend` with a real worker backend** (Celery with a Redis broker, or the Django Tasks database backend plus a worker process, whichever fits the existing deploy). Add the worker to the systemd/compose units used by `.gitea/workflows/deploy-beta.yml` and `deploy-prod.yml`. Remove the daemon-thread workaround in `drive_tasks.py:35-109` once a real backend exists. **1.3 Add an `AgentRun` model** persisting the full lifecycle so a run is recoverable independent of any connection: - `id`, `conversation`, `user`, `status` (`queued` / `planning` / `running` / `synthesising` / `succeeded` / `failed` / `cancelled`), `goal`, `plan` (JSON), `created_at`, `started_at`, `finished_at`, `error` - `total_tokens_in` / `total_tokens_out`, `model_name` — quotas must cover agent runs, not just single turns **1.4 Add `AgentStep`** — `run` (FK), `index`, `parent_step` (FK, nullable, for sub-agent nesting), `kind` (`plan` / `tool_call` / `sub_agent` / `synthesis`), `title` (short human-readable label for the UI), `status`, `tool_name`, `tool_input` (JSON), `tool_output` (JSON), `started_at`, `finished_at`, `error`, `tokens_in`, `tokens_out`. **1.5 Add REST endpoints** `GET /api/agent_runs` and `GET /api/agent_runs/<id>` returning the run with its nested steps, so the frontend can rehydrate after a reload rather than depending on having been connected. ### Phase 2 — Tool-calling layer **2.1 Switch the agent path from `OllamaLLM` to `ChatOllama`.** `OllamaLLM` is a text-completion wrapper with no `bind_tools`. This is a hard blocker — nothing else in this ticket works without it. Leave the existing non-agent chat path on its current class until #62 has settled, then converge. **2.2 Create `services/tools/` with a registry of `@tool`-decorated callables**, each with a typed Pydantic argument schema and a clear docstring (the docstring *is* the tool description the model sees, so it materially affects selection accuracy). Initial set: | Tool | Backed by | |---|---| | `web_search(query, recency_days=None)` | the structured `SearchProvider` layer from #62 | | `fetch_url(url)` | new; fetch and extract readable text, with size cap and SSRF guard | | `search_documents(query, workspace_id)` | existing Chroma retriever in `rag_services.py` | | `read_document(document_id)` | existing `Document` storage | | `analyse_dataframe(document_id, question)` | existing `AsyncDataAnalysisService` | | `make_plot(document_id, spec)` | existing `_generate_plot` in `data_analysis_service.py:103-145` | Every tool must be tenant-scoped through the existing `resolve_chat_company_scope()` / `_workspace_filter` path. **A tool must never be able to read across workspaces** — this is the same class of bug fixed in the stove-piping work, and an agent loop is a much larger attack surface than a single turn. **2.3 Enforce per-tool timeouts, output size caps, and a per-run tool-call budget.** Truncate oversized tool output with an explicit marker rather than letting it blow the context window. ### Phase 3 — Orchestrator: plan → execute → synthesise **3.1 Add a `planner` node** that turns the goal into an ordered list of steps, each with a title, an assigned tool or sub-agent, and its dependencies. Emit strict JSON. Cap the plan at a configurable maximum (default 8 steps). For a simple conversational turn the planner must return a single-step plan so the fast path stays fast — the agent pipeline must not add latency to ordinary chat. **3.2 Add an `executor` node** driving the actual agent loop — a cyclic LangGraph with a conditional edge: ``` planner -> executor -> (tool_calls?) -> tools -> executor [cycle] -> (no tool_calls?) -> synthesiser -> END ``` Bound by a max-iteration count (default 12) **and** a wall-clock deadline (default 10 minutes), both configurable. On breach, stop cleanly and synthesise from whatever completed — never hang, never return nothing. **3.3 Add sub-agents for parallelisable steps.** Steps with no interdependency run as concurrent child runs on `OLLAMA_MODEL_SUBAGENT`, each with its own scratchpad and its own `AgentStep` subtree via `parent_step`. Bound concurrency (default 3) so the GPU host is not oversubscribed. Sub-agents return a structured summary plus citations to the parent — **not** their full transcript, which would flood the orchestrator's context. This also sidesteps `gpt-oss:20b`'s lack of parallel tool calling: parallelism happens at the sub-agent layer rather than within a single model turn. **3.4 Add a `synthesiser` node** producing the final user-facing answer from the collected step outputs, carrying citations forward from #62 and streaming tokens over the existing stream markers so the final answer renders exactly like a normal reply. **3.5 Add retry with backoff on transient tool failure** (default 2 retries). A permanently failed step must be recorded on `AgentStep` and reported in the final answer rather than silently dropped — same principle as #62's rule against silent search failure. ### Phase 4 — Progress streaming **4.1 Extend the WebSocket protocol with JSON frames** for `run_started`, `plan_ready` (full step list up front, so the UI can render the whole checklist immediately), `step_started`, `step_completed`, `step_failed`, and `run_completed`. Each carries `run_id`, `step_id`, `parent_step_id`, `title`, `status`. Define the frame shape **jointly with the `chat_web_app` progress-feedback issue and with #62's citation frame** — all three add frames to the same stream and must not conflict. A versioned envelope (`{"v": 1, "type": ..., "data": ...}`) is preferable to more bare sentinel strings like `START_OF_THE_STREAM_ENDER_GAME_42`. **4.2 Publish progress through the Redis channel layer**, keyed by run, so any connected client for that user receives it and a reconnecting client can resume mid-run. **4.3 Add a cancel endpoint / frame.** A user must be able to stop a long run, and cancellation must actually halt the worker and mark the run `cancelled`. ### Phase 5 — Evaluation and rollout **5.1 Add a tool-selection benchmark**: 30+ prompts with the expected tool sequence, scored on correct tool chosen, correct arguments, and correctly choosing *not* to call a tool. Run against `gpt-oss:20b`, `gemma4:latest`, `llama3.3:latest`, and a pulled `qwen3` to settle the orchestrator default with data rather than the starting recommendation above. **5.2 Add end-to-end agent scenarios** covering both worked examples at the top of this ticket, asserting plan shape, tool usage, citation presence, and completion within the deadline. **5.3 Gate the feature.** Add `ALLOW_AGENTIC_TASKS` (default off) alongside the existing `ALLOW_INTERNET_ACCESS` / `ALLOW_IMAGE_GENERATION` flags, and a plan-tier feature gate via the existing `enforce_feature_gate` mechanism — agent runs are far more expensive than single turns and must be tier-limited and quota-counted. --- ## Acceptance criteria ### Infrastructure - [ ] Channel layer is Redis-backed; progress from a worker process reaches a client connected to a different web process. - [ ] A real (non-immediate) task backend runs agent work in a worker process, deployed by both the beta and prod workflows. - [ ] `AgentRun` and `AgentStep` persist the full lifecycle, including nested sub-agent steps via `parent_step`. - [ ] `GET /api/agent_runs/<id>` returns a run with nested steps and is tenant-scoped. - [ ] **A run survives a page refresh**: disconnect mid-run, reload, and both completed and in-flight steps are visible, with the final answer arriving on the reconnected client. - [ ] The `drive_tasks.py` daemon-thread workaround is removed. ### Tool calling - [ ] The agent path uses `ChatOllama` with `bind_tools`; no agent-path call goes through `OllamaLLM`. - [ ] All six Phase 2 tools are registered with typed schemas and docstring descriptions. - [ ] Every tool is tenant-scoped. A test asserts a tool invoked in workspace A **cannot** return data from workspace B. - [ ] `fetch_url` rejects private/link-local/loopback addresses and enforces a response size cap. Covered by an SSRF test. - [ ] Per-tool timeout, output cap, and per-run tool-call budget are enforced and configurable. ### Orchestration - [ ] The graph contains a genuine cycle with a conditional edge; the model drives tool invocation. - [ ] The planner produces a step list with titles and dependencies, capped at the configured maximum. - [ ] A simple conversational turn produces a single-step plan and shows **no measurable latency regression** versus today's chat path. This is a hard requirement. - [ ] Independent steps execute concurrently as sub-agents, bounded by the concurrency limit. - [ ] Sub-agents return structured summaries plus citations, not raw transcripts. - [ ] Max-iteration and wall-clock deadline both terminate cleanly and still synthesise a partial answer. - [ ] Transient tool failures retry with backoff; permanent failures are recorded and surfaced in the final answer. - [ ] Both worked examples from the Goal section complete end to end with correct tool usage and citations. ### Progress streaming - [ ] All six frame types are emitted in the correct order for a multi-step run. - [ ] `plan_ready` delivers the complete step list before execution starts. - [ ] The frame envelope is versioned and agreed with the `chat_web_app` progress-feedback issue and #62's citation frame; no conflicting frame types across the three. - [ ] Frames publish via the Redis channel layer and reach a reconnecting client. - [ ] Cancellation halts the worker and marks the run `cancelled`; no orphaned work continues. ### Models and gating - [ ] `OLLAMA_MODEL_ORCHESTRATOR` and `OLLAMA_MODEL_SUBAGENT` are configurable and documented in `.env.example`, `.env.prod.example`, and `README.md`. - [ ] The tool-selection benchmark is run across all four candidate models and the results are recorded on this ticket. - [ ] The chosen orchestrator scores **>= 90%** correct tool selection on the benchmark. - [ ] `ALLOW_AGENTIC_TASKS` defaults to off; agent runs are plan-gated via `enforce_feature_gate` and counted against quotas. - [ ] Token usage for a whole run, including all sub-agents, is aggregated onto `AgentRun` and reflected in existing usage reporting. ### Regression safety - [ ] With `ALLOW_AGENTIC_TASKS=False`, chat behaves exactly as it does today. - [ ] The existing offline unit suite passes without Redis, a worker, a live Ollama, or network access. --- ## Open questions - Celery versus the Django Tasks database backend — decide against the existing deploy topology in `server-infra` before starting Phase 1. - Is there VRAM headroom on 10.0.0.128 for an orchestrator plus three concurrent sub-agents plus the embedding model? Measure before setting the concurrency default. Overlaps with the VRAM task in #62. - Should sub-agent progress be shown nested in the UI or flattened? Depends on the design in the `chat_web_app` progress-feedback issue.
westfarn commented 2026-08-02 12:12:26 -07:00
Author
Owner
Copy Link
Copy Source

Related FE frame consumers (same WS envelope family):

Related FE frame consumers (same WS envelope family): - Status/activity: [chat_web_app#96](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/96) - Citations Sources UI: [chat_web_app#98](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/98) - Grounding/citations emit: [chat_backend#62](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/62)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#63