Eval harness (#62 P4), status frames, and agentic runs (#63) (#71)

## Summary - Closes Phase 4 of [#62](#62): `evals/suite.json` (≥40 graded questions), `run_evals` management command, and manually-triggered `.gitea/workflows/run-evals.yml`. - Emits versioned WS `status` frames during grounded chat (evaluating / searching / reading_sources / refining / writing) for [chat_web_app#96](ai_ml_operations/chat_web_app#96). - Implements [#63](#63): Redis/Celery optional infra, `AgentRun`/`AgentStep`, tool registry (SSRF-safe `fetch_url`, tenant-scoped docs), LangGraph orchestrator, progress frames, REST `GET/POST /api/agent_runs/…`, gated by `ALLOW_AGENTIC_TASKS` (default off). ## Test plan - [x] `SKIP_RAG_INIT=1 uv run python manage.py test` for evals, ws frames, agent tools, consumers, grounding - [ ] Manual: with `ALLOW_AGENTIC_TASKS=false`, chat identical to today - [ ] Manual: status frames visible in FE with #96 branch - [ ] Manual (GPU): `python manage.py run_evals --runs 3` - [ ] Manual: `ALLOW_AGENTIC_TASKS=true` multi-step research prompt creates AgentRun + framesReviewed-on: #71
2026-08-04 04:08:50 -07:00
parent e1e086a474
commit 093e5462a5
46 changed files with 4669 additions and 117 deletions
@@ -0,0 +1,60 @@
+name: Run Evals
+
+# Manual eval harness — does not gate PRs (#62 Phase 4).
+# Requires self-hosted runner with GPU/Ollama and live search when SKIP_LIVE is unset.
+on:
+  workflow_dispatch: {}
+
+jobs:
+  run-evals:
+    runs-on: self-hosted
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Install dependencies
+        run: uv sync --frozen
+
+      - name: Validate eval suite (offline)
+        env:
+          DJANGO_ENV: dev
+          DJANGO_SECRET_KEY: test-secret-key
+          DJANGO_DEBUG: "true"
+          DJANGO_ALLOWED_HOSTS: localhost,127.0.0.1,testserver
+          DATABASE_URL: ""
+          DB_HOST: ""
+          SKIP_RAG_INIT: "1"
+          SKIP_LIVE: "1"
+          OLLAMA_BASE_URL: http://127.0.0.1:11434
+        working-directory: llm_be
+        run: uv run python manage.py run_evals --dry-run
+
+      - name: Run eval harness
+        env:
+          DJANGO_ENV: dev
+          DJANGO_SECRET_KEY: test-secret-key
+          DJANGO_DEBUG: "true"
+          DJANGO_ALLOWED_HOSTS: localhost,127.0.0.1,testserver
+          DATABASE_URL: ""
+          DB_HOST: ""
+          SKIP_RAG_INIT: "1"
+          RUN_EVALS: "1"
+          ALLOW_INTERNET_ACCESS: "true"
+          OLLAMA_BASE_URL: http://127.0.0.1:11434
+        working-directory: llm_be
+        run: |
+          uv run python manage.py run_evals \
+            --runs 3 \
+            --output "../eval-report.json"
+
+      - name: Upload eval report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-report
+          path: eval-report.json