ai_ml_operations/chat_backend
2
0
Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases Wiki Activity

File parsing: fix DOCX/PPTX MIME misrouting, unify the two parser paths, and widen format coverage #64

New Issue
Open
opened 2026-08-02 06:49:45 -07:00 by westfarn · 0 comments
westfarn commented 2026-08-02 06:49:45 -07:00
Owner
Copy Link
Copy Source

Summary

Audit of how uploaded files are parsed, prompted by the accuracy review in #62. Two confirmed bugs and a substantial coverage gap. There are two entirely separate parsing implementations that support different formats and behave differently, and the chat-attachment one silently corrupts Word documents.

Bug 1 — DOCX and PPTX chat attachments are misrouted to the Excel reader

llm_be/chat_backend/consumers.py:410-425 normalises the browser MIME type:

if "csv" in file_type:
    file_type = "csv"
elif "xmlformats-officedocument" in file_type:
    file_type = "xlsx"
elif "word" in file_type:
    file_type = "docx"
elif "pdf" in file_type:
    file_type = "pdf"
elif "text" in file_type:
    file_type = "txt"
else:
    file_type = "Not Sure"

The DOCX MIME type is application/vnd.openxmlformats-officedocument.wordprocessingml.document. It contains the substring xmlformats-officedocument, so it matches the second branch and is labelled xlsx. The elif "word" branch is unreachable dead code.

AsyncDataAnalysisService.generate_response then dispatches on that label (data_analysis_service.py:163-175) and calls pd.read_excel() on a Word document, which raises and yields {"type": "error", "content": "An error occurred: ..."}. The _read_docx method exists and is never reached from this path.

Verified against the real MIME strings:

docx  application/vnd.openxmlformats-officedocument.wordprocessingml.document    -> xlsx  WRONG
pptx  application/vnd.openxmlformats-officedocument.presentationml.presentation  -> xlsx  WRONG
xlsx  application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          -> xlsx  ok
csv   text/csv                                                                   -> csv   ok
pdf   application/pdf                                                            -> pdf   ok
txt   text/plain                                                                 -> txt   ok
xls   application/vnd.ms-excel                                                   -> Not Sure
json  application/json                                                           -> Not Sure
rtf   application/rtf                                                            -> Not Sure

Fix: match on exact MIME types (with the file extension as a tiebreaker), most specific first. Never substring-match a prefix shared by every OOXML format.

Bug 2 — Client-supplied MIME type is trusted outright

file_type comes from the browser's file.type (AsyncDashboard2.tsx:554-573), which is unreliable and trivially spoofable. .csv files frequently arrive as application/vnd.ms-excel, and a file with no recognised extension arrives as "".

Fix: sniff the actual content (magic bytes via python-magic or filetype), fall back to the extension, and treat the client MIME as the last resort. Reject on mismatch between sniffed type and claimed type.

The two parser paths are inconsistent

Chat attachment RAG document upload
Entry WebSocket file (base64) POST /documents/ multipart
Parser AsyncDataAnalysisService (data_analysis_service.py:157-175) rag_services.py loader_mapping (lines 59-65)
Libraries pandas, pypdf, python-docx PyPDFLoader, Docx2txtLoader, TextLoader, UnstructuredFileLoader
Formats csv, xlsx, docx*, pdf pdf, docx, txt, plus UnstructuredFileLoader fallback
Unknown type Hard error to the user Falls back to unstructured
Frontend accept .csv,.xlsx,.txt,.pdf,.PDF no filter at all (DocumentStoragePage.tsx:515-567)

* broken, see Bug 1.

So .txt is accepted by the chat file picker but the data-analysis service has no txt branch — it falls to else and errors. Meanwhile the RAG page accepts anything with no client-side validation.

Fix: extract a single services/parsing/ module with one registry keyed by canonical format, used by both paths. One place to add a format, one place to test it.

Formats to support

Tier 1 — fix or add now

Format Status Work
.docx broken in chat Fix Bug 1. Extract tables, headers/footers, and footnotes — Docx2txtLoader and the current _read_docx both walk doc.paragraphs only and silently drop every table.
.pptx broken in chat, unstructured fallback in RAG Add a real parser: slide text, speaker notes, per-slide boundaries.
.txt accepted by picker, errors in chat Add a branch. Detect encoding with charset-normalizer; TextLoader assumes UTF-8 and throws on Latin-1.
.pdf works, low quality pypdf returns no text for scanned PDFs and mangles multi-column layouts and tables. See PDF section below.
.csv works Sniff the delimiter (tab/semicolon-separated files are common) and the encoding. Currently assumes comma + UTF-8.
.xlsx partial pd.read_excel reads only the first sheet. Read all sheets, name them, and surface formulas as values.
.xls unsupported Legacy Excel is still common. Add via xlrd.

Tier 2 — add next

Format Rationale
.md Arrives as text/markdown; currently Not Sure. Should preserve heading structure for chunking.
.json Currently Not Sure. Pretty-print and summarise structure rather than dumping raw.
.rtf Currently Not Sure.
.odt / .ods LibreOffice formats; unstructured handles them but only on the RAG path.
.html Strip boilerplate, keep tables. Shares an extractor with the fetch_url tool in #63.
.eml / .msg Email export is a common enterprise ask.
Images (.png, .jpg) OCR via pytesseract; see below.

Tier 3 — later

.epub, .parquet, .tsv, .xml, .zip (expand and parse members), .py/.js/.ts and other source files (parse with language-aware chunking rather than as flat text).

PDF quality — the biggest single win

pypdf / PyPDFLoader is the weakest link, and PDFs are the most-uploaded format:

  • Scanned PDFs return empty text. No OCR anywhere. The user gets a confident answer synthesised from nothing.
  • Tables are destroyed. Text extraction flattens a table into space-separated runs with no row or column structure.
  • Multi-column layouts interleave, producing scrambled sentences.
  • No page metadata is attached to chunks, so a RAG citation cannot say which page it came from.

Proposed: evaluate pymupdf (better layout fidelity) or unstructured's hi_res strategy against a fixture set of real documents — a scanned contract, a multi-column paper, a table-heavy report, and a plain text document. Add pytesseract OCR triggered when extracted text falls below a per-page character threshold. Attach page_number to chunk metadata alongside the existing source.

Cross-cutting gaps

  • No size limits. A large PDF is base64-encoded over the WebSocket and parsed inline in the consumer, blocking the event loop for the whole worker. Add a max file size, and move parsing off the event loop.
  • No page/row caps. A 5,000-page PDF or a million-row CSV will exhaust memory.
  • Parse errors leak internals. data_analysis_service.py:197-198 yields f"An error occurred: {e}" straight to the user. Return actionable messages ("This PDF appears to be scanned and contains no extractable text") and log the exception server-side.
  • print() used instead of logging at consumers.py:518, data_analysis_service.py:161, rag_services.py:113, and prompt_classifier.py:167.
  • Encrypted/password-protected files are not detected; they surface as generic errors.
  • No frontend/backend agreement on accepted types. The chat picker allows .csv,.xlsx,.txt,.pdf,.PDF while the RAG page allows everything. Both should be driven by one server-published list.

Acceptance criteria

Correctness

  • A .docx chat attachment is parsed as a Word document, not routed to pd.read_excel. Covered by a test over the full MIME table above.
  • A .pptx chat attachment is parsed as a presentation.
  • No dead branches remain in the MIME normaliser; every branch is reachable and asserted.
  • File type is determined by content sniffing first, extension second, client MIME last. Covered by a test with a deliberately mislabelled file.
  • A .csv arriving as application/vnd.ms-excel is parsed as CSV.
  • A .txt chat attachment parses instead of erroring.

Unification

  • A single services/parsing/ registry serves both the chat-attachment and RAG paths.
  • Both paths support the same Tier 1 format set; a test asserts parity.
  • The accepted-format list is published by the backend and consumed by both frontend file pickers.

Coverage

  • All Tier 1 formats parse correctly, each with a fixture file and a test.
  • DOCX extraction includes tables, headers/footers, and footnotes — not just doc.paragraphs.
  • XLSX reads all sheets, not just the first.
  • .xls is supported.
  • CSV delimiter and encoding are sniffed; a semicolon-delimited Latin-1 file parses correctly.
  • Tier 2 formats are either implemented or filed as follow-ups with the fixture set in place.

PDF

  • The extractor comparison is run against all four fixture categories and the results recorded on this issue.
  • A scanned PDF produces text via OCR, or an explicit "no extractable text" message — never a silent empty parse.
  • Table structure survives extraction well enough that a table-lookup question answers correctly.
  • page_number is attached to RAG chunk metadata and available for citations in #62.

Safety and robustness

  • Max file size enforced, with a clear user-facing message on breach.
  • Page and row caps enforced with explicit truncation notices.
  • Parsing does not block the WebSocket event loop; verified by a concurrent-upload test.
  • Encrypted or corrupt files produce actionable messages; raw exception text is never sent to the user.
  • Every print() in the parsing paths is replaced with proper logging.

Related

  • #62 — accuracy overhaul; page-level metadata here feeds RAG citations there.
  • #63 — agentic execution; the read_document and analyse_dataframe tools call this parsing layer, so its format coverage bounds what those tools can do.
  • chat_web_app#97 — export formats; the parse and export format lists should stay aligned.
## Summary Audit of how uploaded files are parsed, prompted by the accuracy review in #62. Two confirmed bugs and a substantial coverage gap. There are **two entirely separate parsing implementations** that support different formats and behave differently, and the chat-attachment one silently corrupts Word documents. --- ## Bug 1 — DOCX and PPTX chat attachments are misrouted to the Excel reader `llm_be/chat_backend/consumers.py:410-425` normalises the browser MIME type: ```python if "csv" in file_type: file_type = "csv" elif "xmlformats-officedocument" in file_type: file_type = "xlsx" elif "word" in file_type: file_type = "docx" elif "pdf" in file_type: file_type = "pdf" elif "text" in file_type: file_type = "txt" else: file_type = "Not Sure" ``` The DOCX MIME type is `application/vnd.openxmlformats-officedocument.wordprocessingml.document`. It contains the substring `xmlformats-officedocument`, so it matches the **second** branch and is labelled `xlsx`. **The `elif "word"` branch is unreachable dead code.** `AsyncDataAnalysisService.generate_response` then dispatches on that label (`data_analysis_service.py:163-175`) and calls `pd.read_excel()` on a Word document, which raises and yields `{"type": "error", "content": "An error occurred: ..."}`. The `_read_docx` method exists and is never reached from this path. Verified against the real MIME strings: ``` docx application/vnd.openxmlformats-officedocument.wordprocessingml.document -> xlsx WRONG pptx application/vnd.openxmlformats-officedocument.presentationml.presentation -> xlsx WRONG xlsx application/vnd.openxmlformats-officedocument.spreadsheetml.sheet -> xlsx ok csv text/csv -> csv ok pdf application/pdf -> pdf ok txt text/plain -> txt ok xls application/vnd.ms-excel -> Not Sure json application/json -> Not Sure rtf application/rtf -> Not Sure ``` **Fix:** match on exact MIME types (with the file extension as a tiebreaker), most specific first. Never substring-match a prefix shared by every OOXML format. ## Bug 2 — Client-supplied MIME type is trusted outright `file_type` comes from the browser's `file.type` (`AsyncDashboard2.tsx:554-573`), which is unreliable and trivially spoofable. `.csv` files frequently arrive as `application/vnd.ms-excel`, and a file with no recognised extension arrives as `""`. **Fix:** sniff the actual content (magic bytes via `python-magic` or `filetype`), fall back to the extension, and treat the client MIME as the last resort. Reject on mismatch between sniffed type and claimed type. --- ## The two parser paths are inconsistent | | Chat attachment | RAG document upload | |---|---|---| | Entry | WebSocket `file` (base64) | `POST /documents/` multipart | | Parser | `AsyncDataAnalysisService` (`data_analysis_service.py:157-175`) | `rag_services.py` `loader_mapping` (lines 59-65) | | Libraries | `pandas`, `pypdf`, `python-docx` | `PyPDFLoader`, `Docx2txtLoader`, `TextLoader`, `UnstructuredFileLoader` | | Formats | csv, xlsx, docx*, pdf | pdf, docx, txt, plus `UnstructuredFileLoader` fallback | | Unknown type | Hard error to the user | Falls back to `unstructured` | | Frontend `accept` | `.csv,.xlsx,.txt,.pdf,.PDF` | no filter at all (`DocumentStoragePage.tsx:515-567`) | \* broken, see Bug 1. So `.txt` is accepted by the chat file picker but the data-analysis service has no `txt` branch — it falls to `else` and errors. Meanwhile the RAG page accepts *anything* with no client-side validation. **Fix:** extract a single `services/parsing/` module with one registry keyed by canonical format, used by both paths. One place to add a format, one place to test it. --- ## Formats to support ### Tier 1 — fix or add now | Format | Status | Work | |---|---|---| | `.docx` | **broken** in chat | Fix Bug 1. Extract tables, headers/footers, and footnotes — `Docx2txtLoader` and the current `_read_docx` both walk `doc.paragraphs` only and silently drop every table. | | `.pptx` | **broken** in chat, `unstructured` fallback in RAG | Add a real parser: slide text, speaker notes, per-slide boundaries. | | `.txt` | accepted by picker, errors in chat | Add a branch. Detect encoding with `charset-normalizer`; `TextLoader` assumes UTF-8 and throws on Latin-1. | | `.pdf` | works, low quality | `pypdf` returns no text for scanned PDFs and mangles multi-column layouts and tables. See PDF section below. | | `.csv` | works | Sniff the delimiter (tab/semicolon-separated files are common) and the encoding. Currently assumes comma + UTF-8. | | `.xlsx` | partial | `pd.read_excel` reads only the **first sheet**. Read all sheets, name them, and surface formulas as values. | | `.xls` | unsupported | Legacy Excel is still common. Add via `xlrd`. | ### Tier 2 — add next | Format | Rationale | |---|---| | `.md` | Arrives as `text/markdown`; currently `Not Sure`. Should preserve heading structure for chunking. | | `.json` | Currently `Not Sure`. Pretty-print and summarise structure rather than dumping raw. | | `.rtf` | Currently `Not Sure`. | | `.odt` / `.ods` | LibreOffice formats; `unstructured` handles them but only on the RAG path. | | `.html` | Strip boilerplate, keep tables. Shares an extractor with the `fetch_url` tool in #63. | | `.eml` / `.msg` | Email export is a common enterprise ask. | | Images (`.png`, `.jpg`) | OCR via `pytesseract`; see below. | ### Tier 3 — later `.epub`, `.parquet`, `.tsv`, `.xml`, `.zip` (expand and parse members), `.py`/`.js`/`.ts` and other source files (parse with language-aware chunking rather than as flat text). --- ## PDF quality — the biggest single win `pypdf` / `PyPDFLoader` is the weakest link, and PDFs are the most-uploaded format: - **Scanned PDFs return empty text.** No OCR anywhere. The user gets a confident answer synthesised from nothing. - **Tables are destroyed.** Text extraction flattens a table into space-separated runs with no row or column structure. - **Multi-column layouts interleave**, producing scrambled sentences. - **No page metadata** is attached to chunks, so a RAG citation cannot say which page it came from. Proposed: evaluate `pymupdf` (better layout fidelity) or `unstructured`'s `hi_res` strategy against a fixture set of real documents — a scanned contract, a multi-column paper, a table-heavy report, and a plain text document. Add `pytesseract` OCR triggered when extracted text falls below a per-page character threshold. Attach `page_number` to chunk metadata alongside the existing `source`. --- ## Cross-cutting gaps - **No size limits.** A large PDF is base64-encoded over the WebSocket and parsed inline in the consumer, blocking the event loop for the whole worker. Add a max file size, and move parsing off the event loop. - **No page/row caps.** A 5,000-page PDF or a million-row CSV will exhaust memory. - **Parse errors leak internals.** `data_analysis_service.py:197-198` yields `f"An error occurred: {e}"` straight to the user. Return actionable messages ("This PDF appears to be scanned and contains no extractable text") and log the exception server-side. - **`print()` used instead of logging** at `consumers.py:518`, `data_analysis_service.py:161`, `rag_services.py:113`, and `prompt_classifier.py:167`. - **Encrypted/password-protected files** are not detected; they surface as generic errors. - **No frontend/backend agreement on accepted types.** The chat picker allows `.csv,.xlsx,.txt,.pdf,.PDF` while the RAG page allows everything. Both should be driven by one server-published list. --- ## Acceptance criteria ### Correctness - [ ] A `.docx` chat attachment is parsed as a Word document, not routed to `pd.read_excel`. Covered by a test over the full MIME table above. - [ ] A `.pptx` chat attachment is parsed as a presentation. - [ ] No dead branches remain in the MIME normaliser; every branch is reachable and asserted. - [ ] File type is determined by content sniffing first, extension second, client MIME last. Covered by a test with a deliberately mislabelled file. - [ ] A `.csv` arriving as `application/vnd.ms-excel` is parsed as CSV. - [ ] A `.txt` chat attachment parses instead of erroring. ### Unification - [ ] A single `services/parsing/` registry serves both the chat-attachment and RAG paths. - [ ] Both paths support the same Tier 1 format set; a test asserts parity. - [ ] The accepted-format list is published by the backend and consumed by both frontend file pickers. ### Coverage - [ ] All Tier 1 formats parse correctly, each with a fixture file and a test. - [ ] DOCX extraction includes tables, headers/footers, and footnotes — not just `doc.paragraphs`. - [ ] XLSX reads **all** sheets, not just the first. - [ ] `.xls` is supported. - [ ] CSV delimiter and encoding are sniffed; a semicolon-delimited Latin-1 file parses correctly. - [ ] Tier 2 formats are either implemented or filed as follow-ups with the fixture set in place. ### PDF - [ ] The extractor comparison is run against all four fixture categories and the results recorded on this issue. - [ ] A scanned PDF produces text via OCR, or an explicit "no extractable text" message — never a silent empty parse. - [ ] Table structure survives extraction well enough that a table-lookup question answers correctly. - [ ] `page_number` is attached to RAG chunk metadata and available for citations in #62. ### Safety and robustness - [ ] Max file size enforced, with a clear user-facing message on breach. - [ ] Page and row caps enforced with explicit truncation notices. - [ ] Parsing does not block the WebSocket event loop; verified by a concurrent-upload test. - [ ] Encrypted or corrupt files produce actionable messages; raw exception text is never sent to the user. - [ ] Every `print()` in the parsing paths is replaced with proper logging. ## Related - #62 — accuracy overhaul; page-level metadata here feeds RAG citations there. - #63 — agentic execution; the `read_document` and `analyse_dataframe` tools call this parsing layer, so its format coverage bounds what those tools can do. - `chat_web_app`#97 — export formats; the parse and export format lists should stay aligned.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
twestfall westfarn (Ryan Westfall)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ai_ml_operations/chat_backend#64