ai_ml_operations/chat_backend
2
0 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

Read attached chat images with a vision model #82

Merged
westfarn merged 1 commits from feat/128-vision-image-prompt into master 2026-10-20 06:13:32 -07:00
Conversation 0 Commits 1 Files Changed 11
+387 -44
westfarn commented 2026-10-20 06:12:13 -07:00
Owner
Copy link
Copy source

Summary

  • Image attachments (PNG, JPEG, WebP, GIF) skip the document parser and go to a vision-capable Ollama model (OLLAMA_MODEL_VISION, default gemma4:latest).
  • The image is stored on the prompt. PromptSerializer returns file_type and base64 image so history can redraw it.
  • Unsupported image types and files over 8 MB return a websocket error and are not saved.
  • A later text turn does not reuse a previous image as a spreadsheet.

Pairs with chat_web_app feat/128-vision-image-prompt (#128).

Test plan

  • manage.py test chat_backend.tests.test_vision chat_backend.tests.test_consumers.GraphNodeTestCase
  • Send a prompt plus PNG over ws/chat_again/ and confirm the vision model is called.
  • GET conversation_details includes image and file_type for that turn.
  • CSV upload still follows the document path.
## Summary - Image attachments (PNG, JPEG, WebP, GIF) skip the document parser and go to a vision-capable Ollama model (`OLLAMA_MODEL_VISION`, default `gemma4:latest`). - The image is stored on the prompt. `PromptSerializer` returns `file_type` and base64 `image` so history can redraw it. - Unsupported image types and files over 8 MB return a websocket error and are not saved. - A later text turn does not reuse a previous image as a spreadsheet. Pairs with `chat_web_app` `feat/128-vision-image-prompt` ([#128](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/128)). ## Test plan - [ ] `manage.py test chat_backend.tests.test_vision chat_backend.tests.test_consumers.GraphNodeTestCase` - [ ] Send a prompt plus PNG over `ws/chat_again/` and confirm the vision model is called. - [ ] `GET conversation_details` includes `image` and `file_type` for that turn. - [ ] CSV upload still follows the document path.
westfarn added 1 commit 2026-10-20 06:12:13 -07:00
Read attached chat images with a vision model.
CI / test (pull_request) Successful in 9s
Details
Unit Tests / test (pull_request) Successful in 10s
Details
c21f052d5b 
Image uploads were treated as unknown documents and never shown to the model. PNG, JPEG, WebP, and GIF now go to Ollama vision, and history returns the image bytes.

Co-authored-by: Cursor <cursoragent@cursor.com>
westfarn merged commit fc06202a1f into master 2026-10-20 06:13:32 -07:00
westfarn deleted branch feat/128-vision-image-prompt 2026-10-20 06:13:32 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#82