Chat & answer generation¶

RAGMill's chat feature is retrieval-augmented generation: it searches your store for the most relevant chunks, then asks an LLM to answer using only those chunks, citing the source filename. This keeps answers grounded in your documents instead of the model's memory.

The single entry point is generate_answer(query, chunks). The backend is chosen at call time from RAGMILL_CHAT_BACKEND, so switching providers is an env-var change — no code change.

Backends¶

Backend	Extra	Needs a key?	Notes
`local` (default)	`ragmill setup-chat`	No	`Qwen2.5-1.5B-Instruct` via `llama-cpp-python`. Downloads once (~1 GB) to `~/.cache/ragmill/models`, then fully offline.
`gemini`	`ragmill[chat-gemini]`	`GEMINI_API_KEY` (or `GOOGLE_API_KEY`)	Google Gemini API. Best quality when online.
`openai`	`ragmill[chat-openai]`	`OPENAI_API_KEY`	OpenAI Chat Completions (ChatGPT).

Nothing is fine-tuned

Both local models are off-the-shelf pretrained models — RAGMill does no fine-tuning. Quality comes from retrieval (feeding the right chunks), not from a custom model.

Local (default, offline)¶

The chat extra is deliberately not part of [all] — llama-cpp-python has no PyPI wheels, so pip would build it from source. Install the prebuilt wheel with:

ragmill setup-chat
ragmill chat

setup-chat prints the package, the index it comes from, and the exact pip command, then asks before installing anything (--yes skips the prompt). To run it yourself:

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu \
  --only-binary llama-cpp-python

--only-binary is not optional. --extra-index-url merges both indexes and pip picks the highest version across them; PyPI carries a newer sdist-only release than the wheel index carries wheels, so without the flag pip downloads the 70 MB source archive and compiles it.

pip install "ragmill[chat]" also works, but only if you have CMake and a C++ compiler — and on Windows, long paths enabled. See Installation for the details and the exact error you'll get without them.

First call downloads the GGUF model; subsequent calls are offline. Override the model with RAGMILL_CHAT_MODEL_REPO / RAGMILL_CHAT_MODEL_FILE, and the context window with RAGMILL_CHAT_N_CTX.

Gemini¶

pip install "ragmill[chat-gemini]"
export RAGMILL_CHAT_BACKEND=gemini
export GEMINI_API_KEY=your-key
ragmill chat

OpenAI / ChatGPT¶

pip install "ragmill[chat-openai]"
export RAGMILL_CHAT_BACKEND=openai
export OPENAI_API_KEY=your-key
ragmill chat

From Python¶

generate_answer takes a query and the list of chunks returned by store.search():

from ragmill.embeddings import EmbeddingModel
from ragmill.vector_store import SQLiteVectorStore
from ragmill.chat import generate_answer

model = EmbeddingModel()
store = SQLiteVectorStore("kb.db")

question = "what is the refund window?"
qvec = model.embed([question])[0]
chunks = store.search(qvec, top_k=5)

answer = generate_answer(question, chunks)   # backend from RAGMILL_CHAT_BACKEND
print(answer)

The system prompt¶

All backends share one instruction: answer only from the provided context, lead with a direct answer and then a brief (2–4 sentence) explanation, and say so plainly when the context doesn't contain the answer rather than guessing.

Answers no longer embed inline bracketed citation markers (e.g. [report.pdf] or [1]) — they read cleanly, and the source filenames are returned separately alongside the answer (the CLI prints them, and the REST /chat endpoint returns them in a sources array), so results stay auditable without cluttering the prose.