Fix reindex_embeddings: install unstructured[xlsx] extras (#66)
Deploy Beta / unit-tests (push) Successful in 10s
Unit Tests / test (push) Successful in 10s
Deploy Beta / docker (push) Successful in 28s
Deploy Beta / deploy-beta (push) Successful in 6m58s

## Summary

`reindex_embeddings` crashed with `ModuleNotFoundError: No module named 'networkx'` when Unstructured hit spreadsheet files via `partition/xlsx.py`. Reindex clears Chroma first, so a mid-run crash leaves an empty/partial vector store.

### True fix
Depend on **`unstructured[xlsx]==0.18.21`** (not bare `unstructured`). That extra pulls the required spreadsheet partition deps: `networkx`, `msoffcrypto-tool`, `xlrd` (plus openpyxl/pandas already present).

### Resilience
Also catch per-document ingest failures so one corrupt/unsupported file cannot abort a full rebuild after Chroma was cleared. Successful files still ingest; failures are logged.

## Test plan
- [x] CI / unit tests on this PR
- [ ] Merge + redeploy image to all hosts
- [ ] On each host: `docker compose -p chat_backend_prod exec web bash -lc 'cd /app/llm_be && SKIP_RAG_INIT=1 uv run python manage.py reindex_embeddings'`
- [ ] Confirm non-zero chunk count and no `networkx` / `msoffcrypto` / `xlrd` import errors

## Ops workaround (running containers only, until deploy)
```bash
uv pip install 'unstructured[xlsx]==0.18.21'
cd /app/llm_be && SKIP_RAG_INIT=1 uv run python manage.py reindex_embeddings
```Reviewed-on: #66
This commit was merged in pull request #66.
This commit is contained in:
2026-08-02 11:57:10 -07:00
parent d8f5b8ebf2
commit bf3fffa343
4 changed files with 66 additions and 4 deletions
@@ -46,7 +46,18 @@ class Command(BaseCommand):
self.stdout.write("Clearing Chroma collection…")
rag.clear_vector_store()
self.stdout.write("Re-ingesting documents…")
rag.ingest_documents()
try:
rag.ingest_documents()
except Exception as exc:
# Partial ingest may still have written some chunks; report and re-raise.
count = rag.vector_store._collection.count()
self.stderr.write(
self.style.ERROR(
f"Reindex failed after clearing Chroma ({exc}). "
f"Vector chunks currently: {count}. Fix the error and re-run."
)
)
raise
count = rag.vector_store._collection.count()
self.stdout.write(
self.style.SUCCESS(f"Reindex complete. Vector chunks now: {count}")
@@ -161,6 +161,13 @@ class RAGService(BaseService):
)
if chunks:
self.vector_store.add_documents(chunks)
except Exception:
# Keep reindex/ingest moving; one bad file must not wipe progress.
logger.exception(
"Failed to ingest document_id=%s file=%s",
doc.id,
doc.file.name,
)
finally:
if os.path.exists(tmp_path):
os.unlink(tmp_path)