RAG (read PDFs and other documents)

[thisandthat1]

To enable more usability, can we add a feature to upload the docs and let the model understand the same, so that we can chat about the document.

Document formats like .txt, .docx, .csv, .pdf

[cztomsik]

this is planned, .txt (and images) will happen first

[cztomsik]

This will first happen with BM25 (classic fulltext) and then also with embeddings.

There will be 2 versions of this

1. per-session RAG with uploaded PDF (scope of the original ticket)

   • either from the last message or from all uploaded files in the conversation

2. global section, called datasets/datasources, where you upload files (or select folders) and it will be available for chatting (and namespaced, so you can also select which source(s) to use)