Track token in/out per prompt on PromptMetric (#18)
Unit Tests / test (push) Successful in 9s

Closes #15

## Summary
- Add nullable `tokens_in` / `tokens_out` `IntegerField`s to `PromptMetric` to record real prompt/completion token counts per turn.
- New `extract_token_usage()` helper parses provider usage payloads (LangChain `usage_metadata`, OpenAI-style `prompt_tokens`/`completion_tokens`, Ollama `prompt_eval_count`/`eval_count`). When a provider reports no usage, the fields stay **null** — counts are never estimated/fabricated.
- `create_prompt_metric` / `finish_prompt_metric` in both `consumers.py` and `consumers_graph.py` accept and persist optional `tokens_in` / `tokens_out` (added to `update_fields` only when present).
- Admin panel (this ticket's deliverable):
  - `PromptMetricAdmin` lists `tokens_in` / `tokens_out` and adds `event` / `model_name` / `has_file` filters.
  - `ConversationAdmin` shows summed `tokens_in` / `tokens_out` / `tokens_total` per conversation.
- Migration `0023_promptmetric_tokens_in_promptmetric_tokens_out` (existing rows remain valid — null).

## Note on live capture
The streaming chat path uses LangChain `StrOutputParser`, which yields plain string chunks with no usage metadata, so live turns currently persist `null` tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it.

## Follow-ups
- #16 — Show token in/out in chat web app UI (FE + API exposure)
- #17 — Token-based billing, quotas, and enforcement

## Test plan
- [x] `uv run python manage.py test` — full suite green (266 tests, 6 skipped)
- [x] Model: token fields default null + persist when set
- [x] `extract_token_usage`: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None)
- [x] Metric lifecycle: tokens persist when provided, stay null when absent (both consumers)
- [x] Admin: conversation token totals sum across metrics and ignore other conversationsReviewed-on: #18
This commit was merged in pull request #18.
This commit is contained in:
2026-07-26 08:22:34 -07:00
parent 85637e3db6
commit a049e4f685
10 changed files with 310 additions and 8 deletions
+16
View File
@@ -216,6 +216,22 @@ class PromptMetric(TimeInfoBase):
reponse_length = models.IntegerField(
blank=True, null=True, help_text="How many characters are in the response"
)
tokens_in = models.IntegerField(
blank=True,
null=True,
help_text=(
"Prompt/input tokens reported by the LLM provider usage payload. "
"Null when the provider did not report usage (never estimated)."
),
)
tokens_out = models.IntegerField(
blank=True,
null=True,
help_text=(
"Completion/output tokens reported by the LLM provider usage payload. "
"Null when the provider did not report usage (never estimated)."
),
)
has_file = models.BooleanField(help_text="Is there a file")
file_type = models.CharField(
max_length=16, help_text="The file type, if any", blank=True, null=True