Why I Stopped Using Cloud LLMs and Built My Own

The decision to stop using cloud AI wasn't about money. The OpenAI API is cheap enough that API cost wasn't a meaningful constraint for most of my projects. The decision was about what happens to the data — specifically, the data belonging to the people I was building for.

When you send a document to GPT-4 and ask a question about it, that document leaves your machine. For my own projects that's a manageable trade-off. But as I started thinking seriously about who would actually use my tools — students in universities with data governance policies, researchers working with unpublished material, people in Bangladesh and similar countries with unreliable connectivity and tight budgets — the cloud model started to feel wrong rather than just suboptimal.

That's the origin of LocalMind AI: a fully private, self-hosted LLM platform. Multi-model support, real-time streaming, voice input and output, PDF ingestion over RAG, and a React Native mobile app that connects to the model running on your own laptop. Nothing leaves the device. This is what I learned building it.

Ollama changed what was possible

Before Ollama, running a local LLM meant managing model weights yourself, choosing between inference backends (llama.cpp, ctransformers, exllama), handling quantization files, and building an API wrapper on top. The setup complexity was the primary barrier to serious work — you spent more time on infrastructure than on the actual product.

Ollama removed most of that. It wraps quantized model serving into a clean, OpenAI-compatible REST API that runs on your machine. One command to pull a model. One command to start serving. The API surface is stable and familiar. It's the single most important piece of infrastructure the local AI ecosystem has produced, and LocalMind is built entirely on top of it.

Streaming is the product

Consumer CPU inference is slow. Llama 3 8B generates tokens at roughly 10–15 tokens per second on a mid-range laptop — not GPU-accelerated, no special hardware. If you wait for the complete response before showing anything, the user stares at a blank screen for 20–40 seconds. That's not a product anyone would choose to use.

Streaming — sending tokens to the client as they're generated — changes the experience completely. The user sees text appearing within about a second of submitting their question. Total generation time is identical. Perceived latency is entirely different. The interface feels alive rather than frozen.

The implementation runs through Server-Sent Events (SSE) piped from Ollama through the FastAPI backend to the React Native app. Getting that pipeline right — handling reconnects, flushing buffers, managing the async generators correctly — took longer than any other part of the API. But the user experience payoff is immediate and obvious.

Voice I/O: what I expected vs. what happened

I expected Whisper to be difficult. It wasn't. Even the base Whisper model handles English with a Bangladeshi accent reliably. The small model is effectively production-quality for conversational input. I had been using cloud STT for other projects and was genuinely surprised by how capable the local model was.

Text-to-speech was harder. Coqui TTS generates natural-sounding speech, but it's slow on CPU — typically 3 to 5 seconds to synthesize a 10-second audio clip. If you generate the full response before starting playback, the voice output lags the chat by 5+ seconds. Unusable.

The fix was to split the response into sentences and buffer the output: generate and begin playing the first sentence immediately, then synthesize the remaining sentences in a background thread while the user is listening to the beginning. First-audio latency went from around 5 seconds to just over 1 second. Still not instant — but close enough that the voice interface feels responsive rather than broken.

The mobile app discovery problem

The React Native app connects to the FastAPI server running on the user's laptop over the local network. This creates a setup friction problem: the user has to know their laptop's local IP address and type it into the app manually. Most non-technical users don't know what a local IP address is, let alone how to find it.

The correct solution is mDNS — the same protocol that makes printers and Bluetooth devices discoverable automatically without configuration. The app would broadcast a service query on the local network and the server would respond. Zero manual setup.

I know this is the right answer. I haven't built it yet. It's the most obvious UX gap in the product, and it's a concrete example of the distance between a working demo and something you'd confidently hand to a non-technical person and ask them to set up themselves.

How capable are small models, actually?

More than I expected, for the right tasks. Mistral 7B Q4 handles most conversational queries well. It summarizes documents, answers questions over retrieved context, and follows multi-turn conversations coherently. It's not GPT-4 — complex multi-step reasoning, ambiguous questions, and tasks requiring broad general knowledge all suffer. But for the core use cases LocalMind targets, it's often good enough.

The gap that still concerns me is context window size. Fitting a meaningful PDF into a 7B model's context requires careful RAG design — careful chunking, precise retrieval, disciplined prompt length management. Out of the box, local models struggle with long documents in ways that cloud models don't. That's the frontier I'm actively working on, and it ties directly back to what I wrote about in the RAG post.

The capability of small quantized models on consumer hardware is genuinely impressive. The constraint isn't quality — it's context. That's a solvable engineering problem, not a fundamental limitation.

LocalMind is open-source and linked from the project page. The streaming architecture and the voice buffering logic are the parts I'd suggest reading if you're building something similar.