ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

Track token in/out per prompt on PromptMetric #18

Merged
westfarn merged 1 commits from feature/token-in-out-tracking into master 2026-07-26 08:22:34 -07:00
Conversation 0 Commits 1 Files Changed 10
+310 -8
westfarn commented 2026-07-26 06:01:28 -07:00
Owner
Copy Link
Copy Source

Closes #15

Summary

  • Add nullable tokens_in / tokens_out IntegerFields to PromptMetric to record real prompt/completion token counts per turn.
  • New extract_token_usage() helper parses provider usage payloads (LangChain usage_metadata, OpenAI-style prompt_tokens/completion_tokens, Ollama prompt_eval_count/eval_count). When a provider reports no usage, the fields stay null — counts are never estimated/fabricated.
  • create_prompt_metric / finish_prompt_metric in both consumers.py and consumers_graph.py accept and persist optional tokens_in / tokens_out (added to update_fields only when present).
  • Admin panel (this ticket's deliverable):
    • PromptMetricAdmin lists tokens_in / tokens_out and adds event / model_name / has_file filters.
    • ConversationAdmin shows summed tokens_in / tokens_out / tokens_total per conversation.
  • Migration 0023_promptmetric_tokens_in_promptmetric_tokens_out (existing rows remain valid — null).

Note on live capture

The streaming chat path uses LangChain StrOutputParser, which yields plain string chunks with no usage metadata, so live turns currently persist null tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it.

Follow-ups

  • #16 — Show token in/out in chat web app UI (FE + API exposure)
  • #17 — Token-based billing, quotas, and enforcement

Test plan

  • uv run python manage.py test — full suite green (266 tests, 6 skipped)
  • Model: token fields default null + persist when set
  • extract_token_usage: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None)
  • Metric lifecycle: tokens persist when provided, stay null when absent (both consumers)
  • Admin: conversation token totals sum across metrics and ignore other conversations
Closes #15 ## Summary - Add nullable `tokens_in` / `tokens_out` `IntegerField`s to `PromptMetric` to record real prompt/completion token counts per turn. - New `extract_token_usage()` helper parses provider usage payloads (LangChain `usage_metadata`, OpenAI-style `prompt_tokens`/`completion_tokens`, Ollama `prompt_eval_count`/`eval_count`). When a provider reports no usage, the fields stay **null** — counts are never estimated/fabricated. - `create_prompt_metric` / `finish_prompt_metric` in both `consumers.py` and `consumers_graph.py` accept and persist optional `tokens_in` / `tokens_out` (added to `update_fields` only when present). - Admin panel (this ticket's deliverable): - `PromptMetricAdmin` lists `tokens_in` / `tokens_out` and adds `event` / `model_name` / `has_file` filters. - `ConversationAdmin` shows summed `tokens_in` / `tokens_out` / `tokens_total` per conversation. - Migration `0023_promptmetric_tokens_in_promptmetric_tokens_out` (existing rows remain valid — null). ## Note on live capture The streaming chat path uses LangChain `StrOutputParser`, which yields plain string chunks with no usage metadata, so live turns currently persist `null` tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it. ## Follow-ups - #16 — Show token in/out in chat web app UI (FE + API exposure) - #17 — Token-based billing, quotas, and enforcement ## Test plan - [x] `uv run python manage.py test` — full suite green (266 tests, 6 skipped) - [x] Model: token fields default null + persist when set - [x] `extract_token_usage`: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None) - [x] Metric lifecycle: tokens persist when provided, stay null when absent (both consumers) - [x] Admin: conversation token totals sum across metrics and ignore other conversations
westfarn added 1 commit 2026-07-26 06:01:28 -07:00
Track token in/out per prompt on PromptMetric
CI / test (pull_request) Successful in 10s
Details
Unit Tests / test (pull_request) Successful in 9s
Details
d4bc3fa71f 
Closes #15

Add nullable tokens_in / tokens_out IntegerFields to PromptMetric so we
record real prompt/completion token counts per turn. Values come from the
LLM provider usage payload via a new extract_token_usage helper; when a
provider does not report usage the fields stay null (never estimated).

- consumers/consumers_graph create_prompt_metric + finish_prompt_metric
  now accept and persist optional tokens_in / tokens_out.
- Admin: PromptMetric lists tokens_in/tokens_out (+ event/model filters);
  Conversation admin shows summed tokens_in/out/total per conversation.
- Tests cover the model fields, extract_token_usage parsing, metric
  lifecycle token persistence, and admin aggregation.

Follow-ups: #16 (frontend token display) and #17 (billing/quotas).
Co-authored-by: Cursor <cursoragent@cursor.com>
westfarn merged commit a049e4f685 into master 2026-07-26 08:22:34 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#18