Vision: attach an image with a prompt and have the model read it #128

New Issue

2026-10-20T04:30:11-07:00

westfarn commented

2026-10-20 04:30:11 -07:00

Summary

Let a user attach an image to a chat prompt and get an answer about that image. Landing copy already promises this (pageProse.js roadmapVision: "Vision is on the roadmap, so you can ask questions about an image you upload."). This is input vision, not image generation (#7).

Today the composer only accepts documents (.csv, .xlsx, .txt, .pdf). sendMessage in llm-fe/src/llm-fe/contexts/WebSocketContext.js already base64-encodes a file + fileType onto the websocket payload. The backend (chat_backend consumers_graph.py) treats that payload as a document (csv/xlsx/docx/pdf/txt) and feeds decoded text into generate_response. Image MIME types fall through as "Not Sure" and are not read as pictures.

User flow

User picks or pastes an image (png, jpeg, webp, gif) in the chat composer alongside a text prompt.
Composer shows a thumbnail preview and the filename before send.
User sends. The turn stores the image with the prompt so it still shows when the conversation is reopened.
The model answers using both the prompt and the image contents (text in the image, objects, layout, charts).
A prompt with no image still behaves as today. A document attachment still follows the existing file path.

Scope

Web (`chat_web_app`)

Accept image types on the chat file input in AsyncDashboard2 (and AsyncChat if that composer is still live). Keep the existing document accept list.
Preview the selected image before send. Clear it on reset with file / fileType.
Render the user's image on the prompt bubble, and again when loading history.
Do not send an image-only turn with an empty prompt unless we explicitly allow it. Current sendMessage rejects empty text.
Update roadmap copy once this ships (pageProse.js, landing tests, public/llms.txt).

Backend (`chat_backend`, required for this to work)

When fileType is an image MIME type, do not run the document parser.
Call a vision-capable model with the prompt plus the image bytes.
Persist the image on the prompt the same way other prompt files are stored, so history can reload it.
Reject oversized or unsupported images with a clear error the web client can show.

Out of scope

Generating images from a prompt (#7).
Multi-image turns.
OCR-only pipelines that ignore the rest of the picture.
Using the image as a RAG document.

Acceptance

Composer accepts png, jpeg, webp, and gif in addition to current document types
Selected image shows a preview before send
Prompt + image returns an answer that depends on the image (for example, reading text in a screenshot)
Image still appears on the user turn after reload
Document attachments (csv/xlsx/txt/pdf) still use the existing path
Text-only prompts unchanged
Unsupported or oversized images show an error instead of a silent failure
Landing / llms.txt no longer list vision as roadmap-only once shipped

## Summary Let a user attach an image to a chat prompt and get an answer about that image. Landing copy already promises this (`pageProse.js` `roadmapVision`: "Vision is on the roadmap, so you can ask questions about an image you upload."). This is input vision, not image generation ([#7](https://git.aimloperations.com/ai_ml_operations/chat_web_app/issues/7)). Today the composer only accepts documents (`.csv`, `.xlsx`, `.txt`, `.pdf`). `sendMessage` in `llm-fe/src/llm-fe/contexts/WebSocketContext.js` already base64-encodes a `file` + `fileType` onto the websocket payload. The backend (`chat_backend` `consumers_graph.py`) treats that payload as a document (csv/xlsx/docx/pdf/txt) and feeds decoded text into `generate_response`. Image MIME types fall through as `"Not Sure"` and are not read as pictures. ## User flow 1. User picks or pastes an image (png, jpeg, webp, gif) in the chat composer alongside a text prompt. 2. Composer shows a thumbnail preview and the filename before send. 3. User sends. The turn stores the image with the prompt so it still shows when the conversation is reopened. 4. The model answers using both the prompt and the image contents (text in the image, objects, layout, charts). 5. A prompt with no image still behaves as today. A document attachment still follows the existing file path. ## Scope ### Web (`chat_web_app`) - Accept image types on the chat file input in `AsyncDashboard2` (and `AsyncChat` if that composer is still live). Keep the existing document accept list. - Preview the selected image before send. Clear it on reset with `file` / `fileType`. - Render the user's image on the prompt bubble, and again when loading history. - Do not send an image-only turn with an empty prompt unless we explicitly allow it. Current `sendMessage` rejects empty text. - Update roadmap copy once this ships (`pageProse.js`, landing tests, `public/llms.txt`). ### Backend (`chat_backend`, required for this to work) - When `fileType` is an image MIME type, do not run the document parser. - Call a vision-capable model with the prompt plus the image bytes. - Persist the image on the prompt the same way other prompt files are stored, so history can reload it. - Reject oversized or unsupported images with a clear error the web client can show. ## Out of scope - Generating images from a prompt (#7). - Multi-image turns. - OCR-only pipelines that ignore the rest of the picture. - Using the image as a RAG document. ## Acceptance - [ ] Composer accepts png, jpeg, webp, and gif in addition to current document types - [ ] Selected image shows a preview before send - [ ] Prompt + image returns an answer that depends on the image (for example, reading text in a screenshot) - [ ] Image still appears on the user turn after reload - [ ] Document attachments (csv/xlsx/txt/pdf) still use the existing path - [ ] Text-only prompts unchanged - [ ] Unsupported or oversized images show an error instead of a silent failure - [ ] Landing / `llms.txt` no longer list vision as roadmap-only once shipped

westfarn referenced this issue from a commit

2026-10-20 06:12:05 -07:00

Let a chat prompt include an image the model can read.

westfarn referenced a pull request that will close this issue

2026-10-20 06:12:11 -07:00

Vision: attach an image with a prompt and have the model read it #129

westfarn referenced this issue from ai_ml_operations/chat_backend

2026-10-20 06:12:13 -07:00

Read attached chat images with a vision model #82

westfarn referenced this issue from a commit

2026-10-20 06:13:32 -07:00

Read attached chat images with a vision model (#82)