Track token in/out per prompt on PromptMetric (#18)
Unit Tests / test (push) Successful in 9s
Unit Tests / test (push) Successful in 9s
Closes #15 ## Summary - Add nullable `tokens_in` / `tokens_out` `IntegerField`s to `PromptMetric` to record real prompt/completion token counts per turn. - New `extract_token_usage()` helper parses provider usage payloads (LangChain `usage_metadata`, OpenAI-style `prompt_tokens`/`completion_tokens`, Ollama `prompt_eval_count`/`eval_count`). When a provider reports no usage, the fields stay **null** — counts are never estimated/fabricated. - `create_prompt_metric` / `finish_prompt_metric` in both `consumers.py` and `consumers_graph.py` accept and persist optional `tokens_in` / `tokens_out` (added to `update_fields` only when present). - Admin panel (this ticket's deliverable): - `PromptMetricAdmin` lists `tokens_in` / `tokens_out` and adds `event` / `model_name` / `has_file` filters. - `ConversationAdmin` shows summed `tokens_in` / `tokens_out` / `tokens_total` per conversation. - Migration `0023_promptmetric_tokens_in_promptmetric_tokens_out` (existing rows remain valid — null). ## Note on live capture The streaming chat path uses LangChain `StrOutputParser`, which yields plain string chunks with no usage metadata, so live turns currently persist `null` tokens (honest, per acceptance criteria — no fabricated counts). The plumbing + helper are in place so wiring real provider usage is a drop-in once the services expose it. ## Follow-ups - #16 — Show token in/out in chat web app UI (FE + API exposure) - #17 — Token-based billing, quotas, and enforcement ## Test plan - [x] `uv run python manage.py test` — full suite green (266 tests, 6 skipped) - [x] Model: token fields default null + persist when set - [x] `extract_token_usage`: LangChain / OpenAI / Ollama key variants, attribute sources, bool/float handling, missing usage → (None, None) - [x] Metric lifecycle: tokens persist when provided, stay null when absent (both consumers) - [x] Admin: conversation token totals sum across metrics and ignore other conversationsReviewed-on: #18
This commit was merged in pull request #18.
This commit is contained in:
@@ -1,4 +1,5 @@
|
||||
from django.contrib import admin
|
||||
from django.db.models import Sum
|
||||
from .models import (
|
||||
CustomUser,
|
||||
Announcement,
|
||||
@@ -60,10 +61,35 @@ class PromptInline(admin.TabularInline):
|
||||
|
||||
class ConversationAdmin(admin.ModelAdmin):
|
||||
model = Conversation
|
||||
list_display = ("title", "get_user_email", "deleted")
|
||||
list_display = (
|
||||
"title",
|
||||
"get_user_email",
|
||||
"deleted",
|
||||
"tokens_in",
|
||||
"tokens_out",
|
||||
"tokens_total",
|
||||
)
|
||||
search_fields = ("title",)
|
||||
inlines = [PromptInline,]
|
||||
|
||||
def _token_sum(self, conversation, field):
|
||||
total = PromptMetric.objects.filter(
|
||||
conversation_id=conversation.id
|
||||
).aggregate(total=Sum(field))["total"]
|
||||
return total or 0
|
||||
|
||||
@admin.display(description="Tokens in")
|
||||
def tokens_in(self, conversation):
|
||||
return self._token_sum(conversation, "tokens_in")
|
||||
|
||||
@admin.display(description="Tokens out")
|
||||
def tokens_out(self, conversation):
|
||||
return self._token_sum(conversation, "tokens_out")
|
||||
|
||||
@admin.display(description="Tokens total")
|
||||
def tokens_total(self, conversation):
|
||||
return self.tokens_in(conversation) + self.tokens_out(conversation)
|
||||
|
||||
|
||||
class PromptAdmin(admin.ModelAdmin):
|
||||
model = Prompt
|
||||
@@ -79,11 +105,14 @@ class PromptMetricAdmin(admin.ModelAdmin):
|
||||
"model_name",
|
||||
"prompt_length",
|
||||
"reponse_length",
|
||||
"tokens_in",
|
||||
"tokens_out",
|
||||
"has_file",
|
||||
"file_type",
|
||||
"get_duration",
|
||||
"created"
|
||||
)
|
||||
list_filter = ("event", "model_name", "has_file")
|
||||
|
||||
|
||||
class DocumentWorkspaceAdmin(admin.ModelAdmin):
|
||||
|
||||
Reference in New Issue
Block a user