Track token in/out per prompt on PromptMetric (#18)
Unit Tests / test (push) Successful in 9s
Unit Tests / test (push) Successful in 9s
Closes #15 ## Summary - Add nullable `tokens_in` / `tokens_out` `IntegerField`s to `PromptMetric` to record real prompt/completion token counts per turn. - New `extract_token_usage()` helper parses provider usage payloads (LangChain `usage_metadata`, OpenAI-style `prompt_tokens`/`completion_tokens`, Ollama `prompt_eval_count`/`eval_count`). When a provider reports no usage, the fields stay **null** — counts are never estimated/fabricated. - `create_prompt_metric` / `finish_prompt_metric` in both `consumers.py` and `consumers_graph.py` accept and persist optional `tokens_in` / `tokens_out` (added to `update_fields` only when present). - Admin panel (this ticket's deliverable): - `PromptMetricAdmin` lists `tokens_in` / `tokens_out` and adds `event` / `model_name` / `has_file` filters. - `ConversationAdmin` shows summed `tokens_in` / `tokens_out` / `tokens_total` per conversation. - Migration `0023_promptmetric_tokens_in_promptmetric_tokens_out` (existing rows remain valid — null). ## Note on live capture The streaming chat path uses LangChain `StrOutputParser`, which yields plain string chunks with no usage metadata, so live turns currently persist `null` tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it. ## Follow-ups - #16 — Show token in/out in chat web app UI (FE + API exposure) - #17 — Token-based billing, quotas, and enforcement ## Test plan - [x] `uv run python manage.py test` — full suite green (266 tests, 6 skipped) - [x] Model: token fields default null + persist when set - [x] `extract_token_usage`: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None) - [x] Metric lifecycle: tokens persist when provided, stay null when absent (both consumers) - [x] Admin: conversation token totals sum across metrics and ignore other conversationsReviewed-on: #18
This commit was merged in pull request #18.
This commit is contained in:
1 parent
85637e3db6
commit
a049e4f685
10 files changed
+310
-8
No files matched your search
@@ -114,11 +114,12 @@ def save_generated_message(conversation_id, message):
|
||||
print(serializer.errors)
|
||||
|
||||
@database_sync_to_async
|
||||
def create_prompt_metric(prompt_id, prompt, has_file, file_type, model_name, conversation_id):
|
||||
def create_prompt_metric(prompt_id, prompt, has_file, file_type, model_name, conversation_id, tokens_in=None):
|
||||
prompt_metric = PromptMetric.objects.create(
|
||||
prompt_id=prompt_id,
|
||||
start_time=timezone.now(),
|
||||
prompt_length=len(prompt),
|
||||
tokens_in=tokens_in,
|
||||
has_file=has_file,
|
||||
file_type=file_type,
|
||||
model_name=model_name,
|
||||
@@ -127,11 +128,18 @@ def create_prompt_metric(prompt_id, prompt, has_file, file_type, model_name, con
|
||||
return prompt_metric
|
||||
|
||||
@database_sync_to_async
|
||||
def finish_prompt_metric(prompt_metric, response_length):
|
||||
def finish_prompt_metric(prompt_metric, response_length, tokens_in=None, tokens_out=None):
|
||||
prompt_metric.end_time = timezone.now()
|
||||
prompt_metric.reponse_length = response_length
|
||||
prompt_metric.event = "FINISHED"
|
||||
prompt_metric.save(update_fields=["end_time", "reponse_length", "event"])
|
||||
update_fields = ["end_time", "reponse_length", "event"]
|
||||
if tokens_in is not None:
|
||||
prompt_metric.tokens_in = tokens_in
|
||||
update_fields.append("tokens_in")
|
||||
if tokens_out is not None:
|
||||
prompt_metric.tokens_out = tokens_out
|
||||
update_fields.append("tokens_out")
|
||||
prompt_metric.save(update_fields=update_fields)
|
||||
|
||||
async def get_conversation_file_async(conversation_id):
|
||||
try:
|
||||
|
||||
Reference in new issue
Block a user