Why Small Models Might Be All You Need

Mistral Small 3.1 landed in March 2025. 24 billion parameters, a single GPU, results that beat GPT-4o Mini on most evaluations. Alibaba's Qwen family keeps climbing in adoption. For a growing list of real-world use cases, compact LLMs match or beat the giants at a fraction of the price.

I've been deploying these in production for months. Here's what I've learned about when they're genuinely enough, and when they're not.

What counts as "small"?

The definition keeps shifting. A few years ago, 7 billion parameters was the standard. Today, "small" covers roughly the 3B to 30B range. The ones I'm talking about:

Mistral Small 3.1 (24B parameters): multilingual, 128K context window, runs on a single NVIDIA RTX 4090 or a Mac with 32GB RAM. Outperforms GPT-4o Mini and Claude Haiku 4.5 on several evaluations.

Qwen 3 series (0.6B to 235B MoE): Alibaba's latest open-weight family with hybrid reasoning and strong multilingual performance. Even the 8B variant has a thinking mode. Apache 2.0 license.

Gemma 3 (1B to 27B): Google's latest compact family, based on Gemini 2.0 technology. Multimodal, 128K context, 140+ languages, built for on-device and edge deployment.

These aren't toys. They handle production workloads with results that would have seemed impossible at this scale a year ago.

The cost math

Take a real scenario. You're processing 100,000 customer support conversations a month through an LLM for classification and routing.

With a frontier model like GPT-5 or Claude Opus 4 via API, you're looking at thousands of euros a month depending on conversation length and complexity.

With Mistral Small 3.1 on your own hardware, the expense is the GPU itself: maybe 1,800 € upfront for a 4090, or 180 €/month on a cloud instance. After that, inference is essentially free. Process a million conversations or ten million, same bill.

That's a different price structure, not a marginal improvement. For high-volume, moderate-complexity use cases, self-hosting economics crush the API-for-everything approach.

Where compact models match the giants

Through months of testing, I've found clear categories where compact LLMs hold their own.

Text classification and routing is the clearest case. Sorting emails by urgency, categorizing support tickets, detecting intent in user messages. A 7B fine-tuned on your data beats a generic frontier at these tasks. Not because it's smarter. Because it's specialized.

Structured data extraction works well too: pulling names and amounts from unstructured text is pattern matching at its core, and compact models do pattern matching well. Summarization of moderate-length content (blog posts, meeting notes, documentation pages) produces clean results, hard to distinguish from what larger models generate.

Translation for common language pairs (English-French, English-Spanish) holds up: the quality gap between a well-trained 24B and a 175B+ is negligible for standard business content. Same for common code generation patterns like CRUD operations and API endpoints. Nothing brilliant required. Just knowing the patterns is enough.

Where they fall short

These models have real limitations, and honesty matters more than hype.

Complex reasoning chains. Multi-step logic problems, mathematical proofs, anything requiring coherence over long sequences of analysis. This is where parameter count, and now extended thinking, actually matters.

Broad knowledge recall. If you need to surface obscure historical facts, niche technical specifications, or rarely-discussed topics, larger models simply have more knowledge baked in.

Subtle creative writing. There's a subtlety gap. Compact models produce serviceable text but struggle with voice, humor, the kind of prose that feels genuinely human.

Long-context analysis. Mistral Small 3.1 supports 128K tokens on paper. But real-world performance on questions requiring deep understanding of very long documents still favors larger models. The context window exists, attention quality degrades at the edges.

The privacy argument is underrated

Here's something that doesn't show up in benchmark comparisons but matters enormously in practice: data stays on your infrastructure.

When you self-host a compact model, your customer data never leaves your network. No API calls to third parties, no data processing agreements to negotiate, no risk of training data leakage. GDPR compliance by design rather than by contract.

For healthcare, finance, legal, or for European companies serious about data sovereignty, this alone can justify the choice even when a frontier would perform slightly better. The compliance overhead of external APIs often costs more than the marginal quality improvement is worth.

I've helped clients migrate from API-based solutions to self-hosting for exactly this reason. The performance was comparable. The compliance burden dropped.

The hybrid approach

The smartest strategy isn't "compact for everything" or "frontier for everything." It's matching size to difficulty.

Here's how I structure it for LLM integration projects.

Tier 1 is compact and self-hosted. It handles the bulk of the volume: classification, extraction, simple Q&A against a knowledge base, translation, code templates.

Tier 2 sits in the middle, mid-size models via API or self-hosted, covering general-purpose chat, content generation, moderate reasoning, document analysis.

Then tier 3: frontier via API, reserved for complex reasoning, creative work requiring nuance, novel problem-solving, and cases where accuracy is critical but volume is low.

A routing layer evaluates incoming requests and sends them to the right tier. Most, often 70-80%, land on tier 1. Your costs stay low. Quality stays high where it matters.

Getting started

If you haven't tried running a compact LLM locally, here's the minimum viable experiment.

Pick Mistral Small 3.1 if you want multilingual and strong general performance, or Qwen 3 8B if you want maximum speed on modest hardware. Ollama is the easiest runtime for local testing: download, pull a model, run it. Five minutes to first inference.

Then test against your real use cases, not generic benchmarks. Take 50 real inputs from your production workload and compare outputs side by side with whatever frontier you're currently using. Measure quality, but also latency, price per request, and privacy implications. The total picture often favors compact models more than a quality-only comparison suggests.

If the compact model handles 80% of your workload at 5% of the budget, you've found a significant optimization. Even if you keep the frontier for the remaining 20%.

Where this is heading

The trend is clear. Compact models are getting better faster than the giants are getting cheaper. Every few months, a new release pushes the capability floor upward. What required 70B parameters a year ago needs 24B today. What needs 24B today might need 7B a year from now.

Frontiers don't become irrelevant. The range of use cases where you actually need them keeps shrinking. Plan your architecture accordingly.

David and Goliath

David vs Goliath

For more on self-hosting, head over to the self-hosting guide. And if you're looking to integrate LLMs into your systems, I cover that in LLM integration.