Track token in / token out per conversation (admin panel) #15

New Issue

2026-07-26T05:48:24-07:00

westfarn commented

2026-07-26 05:48:24 -07:00

Summary

Persist and expose token_in (prompt/input tokens) and token_out (completion/output tokens) for conversations so we can measure real LLM usage. Ship visibility in the Django admin panel first; chat frontend UI can follow in a later ticket.

Motivation

Today PromptMetric only stores character lengths (prompt_length, reponse_length), not actual tokenizer/API token counts. Without token in/out we cannot accurately track usage, cost, or per-conversation consumption.

Requirements

Data model

Capture tokens_in and tokens_out for each LLM turn (ideally on Prompt and/or PromptMetric; roll up totals on Conversation).
Prefer values from the LLM/provider response usage payload when available (e.g. Ollama/OpenAI-style prompt_tokens / completion_tokens or equivalent).
If a path does not return usage, document fallback behavior (leave null vs estimate) — prefer null over fake numbers.
Support aggregation: conversation-level totals = sum of turn-level tokens.

Capture

Record tokens when a prompt finishes successfully (same lifecycle as existing PromptMetric FINISHED/response-length recording).
Cover main chat completion paths used by the websocket/LLM services.

Admin panel (this ticket)

Update Django admin so operators can see token usage without a frontend change:
- ConversationAdmin: show tokens_in, tokens_out (and optionally total) in list_display / detail.
- PromptAdmin and/or PromptMetricAdmin: show per-turn tokens_in / tokens_out.
Make fields searchable/filterable where useful (e.g. filter conversations with high usage).

Out of scope (follow-up)

Chat web app UI for showing tokens to end users.
Billing/quotas/enforcement (tracking only for now).

Acceptance criteria

Token in and token out persisted per LLM turn
Conversation-level totals available (stored or derived) and visible in admin
Admin list/detail views show tokens for Conversation and Prompt/PromptMetric
Values come from provider usage when present; missing usage does not invent counts
Migration(s) included; existing rows remain valid (null/0 as appropriate)
Tests cover recording + admin-visible fields / aggregation

Notes

Current related pieces:

PromptMetric already tracks char lengths and prompt/conversation ids — natural place to extend, or add fields on Prompt/Conversation.
ConversationAdmin currently lists title, get_user_email, deleted only.
Frontend token display deferred; API exposure optional if needed later for the FE ticket.

## Summary Persist and expose **token_in** (prompt/input tokens) and **token_out** (completion/output tokens) for conversations so we can measure real LLM usage. Ship visibility in the **Django admin panel** first; chat frontend UI can follow in a later ticket. ## Motivation Today `PromptMetric` only stores character lengths (`prompt_length`, `reponse_length`), not actual tokenizer/API token counts. Without token in/out we cannot accurately track usage, cost, or per-conversation consumption. ## Requirements ### Data model - Capture **tokens_in** and **tokens_out** for each LLM turn (ideally on `Prompt` and/or `PromptMetric`; roll up totals on `Conversation`). - Prefer values from the LLM/provider response usage payload when available (e.g. Ollama/OpenAI-style `prompt_tokens` / `completion_tokens` or equivalent). - If a path does not return usage, document fallback behavior (leave null vs estimate) — prefer null over fake numbers. - Support aggregation: conversation-level totals = sum of turn-level tokens. ### Capture - Record tokens when a prompt finishes successfully (same lifecycle as existing `PromptMetric` FINISHED/response-length recording). - Cover main chat completion paths used by the websocket/LLM services. ### Admin panel (this ticket) - Update Django admin so operators can see token usage without a frontend change: - `ConversationAdmin`: show `tokens_in`, `tokens_out` (and optionally total) in `list_display` / detail. - `PromptAdmin` and/or `PromptMetricAdmin`: show per-turn tokens_in / tokens_out. - Make fields searchable/filterable where useful (e.g. filter conversations with high usage). ### Out of scope (follow-up) - Chat web app UI for showing tokens to end users. - Billing/quotas/enforcement (tracking only for now). ## Acceptance criteria - [ ] Token in and token out persisted per LLM turn - [ ] Conversation-level totals available (stored or derived) and visible in admin - [ ] Admin list/detail views show tokens for Conversation and Prompt/PromptMetric - [ ] Values come from provider usage when present; missing usage does not invent counts - [ ] Migration(s) included; existing rows remain valid (null/0 as appropriate) - [ ] Tests cover recording + admin-visible fields / aggregation ## Notes Current related pieces: - `PromptMetric` already tracks char lengths and prompt/conversation ids — natural place to extend, or add fields on `Prompt`/`Conversation`. - `ConversationAdmin` currently lists `title`, `get_user_email`, `deleted` only. - Frontend token display deferred; API exposure optional if needed later for the FE ticket.

westfarn referenced this issue

2026-07-26 06:00:33 -07:00

Show token in / token out usage in chat web app UI #16

westfarn referenced this issue

2026-07-26 06:00:39 -07:00

Token-based billing, quotas, and usage enforcement #17

westfarn referenced this issue from a commit

2026-07-26 06:01:10 -07:00

Track token in/out per prompt on PromptMetric

westfarn referenced a pull request that will close this issue

2026-07-26 06:01:28 -07:00

Track token in/out per prompt on PromptMetric #18