Running RAG Completely Offline: What Nobody Tells You Until It Fails

There's a specific kind of frustration that comes from building something that works perfectly in a tutorial and then completely falls apart the moment you use it on a real document. That's where I spent most of my time building both the Physics Local Chatbot and LocalMind AI.

Both systems use RAG — retrieval-augmented generation — to let users ask questions over their own documents without sending anything to a cloud server. No OpenAI API, no Pinecone subscription, no internet connection required. Just a PDF, a local LLM running through Ollama, and FAISS handling the similarity search.

In theory, the architecture is straightforward. In practice, I hit three distinct failure modes in sequence. Here's what they were, and what actually fixed them.

The chunking problem nobody talks about

Every tutorial on RAG starts with RecursiveCharacterTextSplitter at 500 characters with a 50-character overlap. For prose content — articles, documentation, general writing — this is workable. For a physics textbook, it's a disaster.

Physics text has equations, theorem statements, and formal definitions that are semantically atomic. Splitting "Newton's second law states F = ma, where F is the net force..." in the middle of that sentence gives you two fragments that are individually useless. When a student asks what Newton's second law says, the retriever pulls both fragments and the LLM stitches together something that sounds plausible but may be subtly wrong. And in physics, a subtly wrong formula is worse than no answer at all.

The fix wasn't clever. I wrote a boundary-aware chunker that splits on paragraph breaks, blank lines, and structural keywords like "Theorem," "Definition," and "Example" — the signals that already exist in textbooks to mark where one complete thought ends and another begins. The chunk size became a secondary constraint rather than the primary one.

Retrieval precision on physics-specific queries went from roughly 60% to over 90% in my own testing. The lesson is uncomfortable but simple: chunk on meaning, not character count. For technical domains especially, the document's own structure tells you where to cut.

The silence of FAISS persistence

FAISS doesn't auto-save. You call save_local(path) when you want to persist the index to disk. I knew this. What I didn't account for was the incremental case — adding new documents to a running system.

After the initial build, I added new documents to the in-memory index without calling save_local again. The next session loaded the old index from disk and silently skipped everything added since the initial build. No error. No warning. The system returned results — just not from any document ingested after session one.

I spent two days debugging answer quality before I realized what was happening. The fix is boring: keep a hash registry of every document currently in the index alongside the FAISS files. At load time, compare the current document set against the registry. Any mismatch triggers a merge and a save. It's not interesting engineering — it's the kind of defensive bookkeeping that separates a weekend prototype from something that works reliably over time.

Small models hallucinate confidently

Cloud-scale models — GPT-4, Claude — are reasonably good at admitting they don't know something. Llama 3 8B is not. It will invent a plausible equation, state it with complete confidence, and cite nothing. In general domains this is annoying. In physics, where a slightly wrong formula can corrupt an entire problem, it's actively harmful.

Two interventions reduced this significantly. The first was explicit system prompt constraints — telling the model in plain terms not to use outside knowledge, not to infer values not stated in the context, and to say "I don't have enough information" rather than guess. Small models respond to explicit rules better than implied expectations. They follow instructions more literally than large models, which is a limitation in most cases and an advantage here.

The second was surfacing the FAISS retrieval distance score to the user. If the closest matching chunk in the vector store is still semantically far from the query — meaning the document probably doesn't cover the topic — the interface shows a low-confidence indicator. It doesn't prevent hallucination, but it puts the user in the right epistemic state: check the source material directly.

Making CPU inference feel fast

Llama 3 8B at FP16 on a mid-range CPU takes 45+ seconds to produce the first token. That's unusable. Two changes brought that to something acceptable.

Switching from FP16 to Q4_K_M quantization roughly halved generation time at almost no perceptible quality cost for Q&A tasks. Q4_K_M is the sweet spot for 7-8B models on consumer hardware — fast enough to be usable, accurate enough to be trustworthy for the specific domain you're working in.

Streaming the response token-by-token changed the experience more than the latency numbers suggest. The user sees text appearing within a second rather than waiting 20 seconds for a complete response. Total generation time is identical. Perceived latency is completely different. If you're building any local LLM interface, streaming isn't optional — it's the implementation detail that determines whether the product feels alive or broken.

What I'd tell myself at the start

Build document hash tracking into the FAISS wrapper from day one, not as a debugging fix six weeks in. Write your first serious tests using someone who actually needs the system — a student asking real exam questions — not with queries you've written yourself. I caught most of the hallucination problems because I knew the answers. The chunking problem only became obvious when someone else used it and got confused by a fragmented equation.

The gap between "RAG that works on demo data" and "RAG that works on the documents your actual users have" is mostly a chunking problem in disguise. Solve that first.