ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

Eval harness (#62 P4), status frames, and agentic runs (#63) #71

Merged
westfarn merged 1 commits from feat/62-p4-evals-63-agentic into master 2026-08-04 04:08:50 -07:00
Conversation 0 Commits 1 Files Changed 46
+4669 -117
westfarn commented 2026-08-04 04:07:54 -07:00
Owner
Copy Link
Copy Source

Summary

  • Closes Phase 4 of #62: evals/suite.json (≥40 graded questions), run_evals management command, and manually-triggered .gitea/workflows/run-evals.yml.
  • Emits versioned WS status frames during grounded chat (evaluating / searching / reading_sources / refining / writing) for chat_web_app#96.
  • Implements #63: Redis/Celery optional infra, AgentRun/AgentStep, tool registry (SSRF-safe fetch_url, tenant-scoped docs), LangGraph orchestrator, progress frames, REST GET/POST /api/agent_runs/…, gated by ALLOW_AGENTIC_TASKS (default off).

Test plan

  • SKIP_RAG_INIT=1 uv run python manage.py test for evals, ws frames, agent tools, consumers, grounding
  • Manual: with ALLOW_AGENTIC_TASKS=false, chat identical to today
  • Manual: status frames visible in FE with #96 branch
  • Manual (GPU): python manage.py run_evals --runs 3
  • Manual: ALLOW_AGENTIC_TASKS=true multi-step research prompt creates AgentRun + frames
## Summary - Closes Phase 4 of [#62](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/62): `evals/suite.json` (≥40 graded questions), `run_evals` management command, and manually-triggered `.gitea/workflows/run-evals.yml`. - Emits versioned WS `status` frames during grounded chat (evaluating / searching / reading_sources / refining / writing) for [chat_web_app#96](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/96). - Implements [#63](https://git.aimloperations.com/ai_ml_operations/chat_backend/issues/63): Redis/Celery optional infra, `AgentRun`/`AgentStep`, tool registry (SSRF-safe `fetch_url`, tenant-scoped docs), LangGraph orchestrator, progress frames, REST `GET/POST /api/agent_runs/…`, gated by `ALLOW_AGENTIC_TASKS` (default off). ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test` for evals, ws frames, agent tools, consumers, grounding - [ ] Manual: with `ALLOW_AGENTIC_TASKS=false`, chat identical to today - [ ] Manual: status frames visible in FE with #96 branch - [ ] Manual (GPU): `python manage.py run_evals --runs 3` - [ ] Manual: `ALLOW_AGENTIC_TASKS=true` multi-step research prompt creates AgentRun + frames
westfarn added 1 commit 2026-08-04 04:07:54 -07:00
Add eval harness (#62 P4), status frames (#96), and agentic runs (#63).
CI / test (pull_request) Successful in 11s
Details
Unit Tests / test (pull_request) Successful in 10s
Details
9c0b648db3 
Ship the Phase 4 accuracy eval suite with a manual Gitea workflow, emit
versioned WS status frames during grounded chat, and introduce opt-in
agent infrastructure (Redis/Celery, AgentRun/Step, tools, orchestrator)
gated by ALLOW_AGENTIC_TASKS so default chat behaviour stays unchanged.
westfarn merged commit 093e5462a5 into master 2026-08-04 04:08:50 -07:00
westfarn deleted branch feat/62-p4-evals-63-agentic 2026-08-04 04:08:50 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#71