Why Small Models Might Be All You Need

Mistral Small 3.1 landed in March 2025. 24 billion parameters, a single GPU, and results that beat GPT-4o Mini on most evaluations. Alibaba's Qwen 2.5 family keeps climbing in adoption. The pattern is impossible to ignore: for a growing list of real-world use cases, compact LLMs match or beat the giants at a fraction of the price.

I've been deploying these in production for months. Here's what I've learned about when they're genuinely enough and when they're not.

What counts as "small"?

The definition keeps shifting. Two years ago, 7 billion parameters was the standard. Today, "small" covers roughly the 3B to 30B range. The ones I'm talking about:

Mistral Small 3.1 (24B parameters): multilingual, 128K context window, runs on a single NVIDIA RTX 4090 or a Mac with 32GB RAM. Outperforms GPT-4o Mini and Claude Haiku 4.5 on several evaluations.

Qwen 3 series (0.6B to 235B MoE): Alibaba's latest open-weight family with hybrid reasoning and strong multilingual performance. Even the 8B variant has a thinking mode. Apache 2.0 license.

Gemma 3 (1B to 27B): Google's latest compact family, based on Gemini 2.0 technology. Multimodal, 128K context, 140+ languages, designed for on-device and edge deployment.

These aren't toys. They handle production workloads with results that would have seemed impossible at this scale a year ago.

The cost math is brutal

Let's look at a real scenario. Say you're processing 100,000 customer support conversations per month through an LLM for classification and routing.

With a frontier like GPT-5 or Claude Opus 4 via API, you're looking at thousands of euros per month depending on conversation length and complexity.

With Mistral Small 3.1 on your own hardware, the expense is the GPU itself: maybe 1,800 € upfront for a 4090, or 180 €/month on a cloud instance. After that, inference is essentially free. Process a million conversations or ten million, same bill.

That's not a marginal improvement. It's a different price structure entirely. For high-volume, moderate-complexity use cases, the economics of self-hosting are devastating to the API-for-everything approach.

Where compact LLMs match the giants

Through months of testing, I've found clear categories where compact LLMs hold their own.

Text classification and routing is the strongest case. Sorting emails by urgency, categorizing support tickets, detecting intent in user messages. A 7B fine-tuned on your data will beat a generic frontier at these tasks. Not because it's smarter but because it's specialized.

Structured data extraction works well too - pulling names, dates, amounts, and categories from unstructured text. This is pattern matching at its core, and compact LLMs do pattern matching well. Summarization of moderate-length content (blog posts, meeting notes, documentation pages) produces clean results that are hard to distinguish from those generated by larger models.

Translation for common language pairs (English-French, English-Spanish, English-German) is another area where the quality gap between a well-trained 24B and a 175B+ is negligible for standard business content. Same for common code generation patterns: CRUD operations, API endpoints, data transformations. Nothing brilliant required - just knowing the patterns is enough.

Where they fall short

These LLMs have real limitations, and honesty matters more than hype.

Complex reasoning chains are the biggest gap. Multi-step logic problems, mathematical proofs, anything requiring coherence over long sequences of analysis. This is where parameter count (and now extended thinking) actually matters.

They also struggle with broad knowledge recall. If you need to surface obscure historical facts, niche technical specifications, or rarely-discussed topics, larger models simply have more knowledge baked in. There's a subtlety gap in creative writing too - compact LLMs produce serviceable text but struggle with voice, humor, and prose that feels genuinely human.

And while Mistral Small 3.1 supports 128K tokens on paper, real-world performance on questions requiring deep understanding of very long documents still favors larger models. The context window exists, but attention quality degrades at the edges.

The privacy argument is underrated

Here's something that doesn't show up in benchmark comparisons but matters enormously in practice: data stays on your infrastructure.

When you self-host a compact LLM, your customer data never leaves your network. No API calls to third parties, no data processing agreements to negotiate, no risk of training data leakage. GDPR compliance by design rather than by contract.

For industries like healthcare, finance, and legal (or for European companies serious about data sovereignty), this alone can justify the choice even when a frontier would perform slightly better. The compliance overhead of external APIs often costs more than the marginal quality improvement is worth.

I've helped clients migrate from API-based solutions to self-hosting specifically for this reason. The performance was comparable. The compliance burden dropped dramatically.

The hybrid approach

The smartest strategy isn't "compact for everything" or "frontier for everything." It's matching size to difficulty.

Here's how I structure it for LLM integration projects:

At the bottom, compact self-hosted models handle the bulk: classification, extraction, simple Q&A against a knowledge base, translation, code templates. High volume, well-defined tasks. In the middle, mid-size models (API or self-hosted) cover general-purpose chat, content generation, moderate reasoning, document analysis. And at the top, frontier APIs handle complex reasoning, creative work requiring nuance, novel problem-solving, and cases where accuracy is critical but volume is low.

A routing layer evaluates incoming requests and sends them to the right level. Most (often 70-80%) go to the compact tier. Your costs stay low. Quality stays high where it matters.

Getting started

If you haven't tried running a compact LLM locally, here's the minimum viable experiment.

Start with Mistral Small 3.1 if you want multilingual and strong general performance, or Qwen 3 8B if you want maximum speed on modest hardware. Ollama is the easiest runtime for local testing - download, pull a model, run it. Five minutes to first inference.

Then test against your real use cases, not generic benchmarks. Take 50 real inputs from your production workload and compare outputs side by side with whatever frontier you're currently using. Measure quality, but also latency, price per request, and privacy implications. The total picture often favors compact LLMs more than the quality-only comparison suggests.

If the compact LLM handles 80% of your workload at 5% of the budget, you've just found a significant optimization, even if you keep the frontier for the remaining 20%.

Where this is heading

The trend is clear. Compact LLMs are getting better faster than the giants are getting cheaper. Every few months, a new release pushes the capability floor upward. What required 70B parameters a year ago needs 24B today. What needs 24B today might need 7B a year from now.

That doesn't mean frontiers become irrelevant. It means the range of use cases where you need them keeps shrinking. Plan your architecture accordingly.

David and Goliath

David vs Goliath

For more on self-hosting, head over to the self-hosting guide. And if you're looking to integrate LLMs into your systems, I cover that in LLM integration.