AI decisions Architecture and technology
Small vs large language models: which one should run your enterprise use case?
Use a small or specialised language model when the task is narrow, repetitive and well defined, runs at high volume, or must stay on your own infrastructure or device. Use a large frontier model when the task is open-ended, varied or needs broad knowledge and multi-step judgement. Most enterprise portfolios end up with both: a large model where breadth matters and smaller models for the high-volume tasks they handle well enough, chosen by evaluation on your own cases.
The options
Small or specialised model
A compact language model, often open-weight and sometimes fine-tuned or distilled, that is built or adapted to do a limited set of tasks well.
Large frontier model
A general-purpose model at the top of the capability range, usually consumed through a provider's API, that handles a wide variety of tasks without task-specific training.
Side by side
| Criterion | Small or specialised model | Large frontier model |
|---|---|---|
| Breadth of capability | Strong within its task; quality drops quickly outside what it was trained or tuned for. | Handles varied, unexpected and open-ended requests, including ones nobody planned for. |
| Cost per request | Usually much lower per call, which matters when volumes are high and the task is repetitive. | Higher per call; acceptable when volumes are moderate or each answer carries real value. |
| Latency | Fast responses, and can run close to the user or the data, including on devices or at the edge. | Slower and dependent on the provider's endpoint and network; fine for most interactive use, harder for real-time loops. |
| Data control and residency | Easy to deploy on your own infrastructure, so confidential data does not have to leave it. | Typically a managed service; data location and retention depend on the provider's terms and the region you contract. |
| Effort to get started | Needs a clear task definition, often a dataset for tuning or distillation, and serving infrastructure. | Low: a prompt and an API key give a working first version for many tasks. |
| Maintenance | You own serving, monitoring, updates and retraining when the task or the data changes. | The provider maintains the model, but versions change on its schedule and need re-evaluation on your side. |
| Typical failure mode | Confident but wrong answers on inputs that drift away from the training distribution. | Costs that grow unnoticed with volume, and variable behaviour across model versions. |
| Fit with agents and complex workflows | Good as a specialist step inside a workflow: classification, extraction, routing, checks. | Better as the planner or generalist that handles ambiguous instructions and multi-step tasks. |
Choose Small or specialised model when…
- The task is narrow and stable: classification, extraction into a schema, routing, redaction or a fixed writing format.
- Volume is high enough that cost or latency per request decides whether the use case pays off.
- Data must stay on your own infrastructure, a private cloud or a device.
- You have, or can build, representative examples and an evaluation set to prove the small model is good enough.
Choose Large frontier model when…
- Requests are varied and hard to predict, so no narrow training set would cover them.
- The task needs broad world knowledge, nuanced language or multi-step judgement.
- You are still discovering the use case and need the fastest route to a working version.
- Volume is moderate and the value of each answer outweighs the cost per call.
When to combine them
The strongest pattern is usually a portfolio behind a routing layer. Start with a large model to learn what the task really is and to build an evaluation set from real traffic. Then move the high-volume, well-understood parts to smaller models, sometimes distilled from the large one's outputs where licences allow, and keep the large model for the cases the small one flags as uncertain. Re-run the evaluation whenever either model changes.
Common mistakes
- Choosing by benchmark leaderboards instead of an evaluation set built from your own cases.
- Defaulting to the largest model for every task, then finding that cost grows with adoption faster than value.
- Moving to a small model to save money without measuring the quality drop on the hard cases.
- Forgetting that self-hosting a small model means owning serving, monitoring, security patches and upgrades.
- Treating the choice as permanent; models on both ends improve fast and the right split changes over time.
How Thinkia approaches it
We do not start from model size. We start from the task, its volume, the data it touches and the quality the business will accept, and we build an evaluation set from the client's own cases. Size, provider and deployment follow from that, and the answer often differs between tasks in the same process.
Architecturally, we put a model-agnostic layer between applications and models. Synapse is LLM-agnostic: applications talk to one gateway with corporate SSO, rate limiting and logging, routing rules decide which model handles each request, including local models first for confidential work, and the ROI dashboards show cost and usage per model. That makes moving a task from a large model to a smaller one, or back, a configuration change rather than a rewrite.
We also treat each approved model as something with an owner, permitted data classes and an evaluation on the client's tasks, reviewed when a new version appears. That keeps a portfolio of models from turning into fragmentation.
Thinkia products involved
- SynapseGoverned agentic platform: agents, models, costs and data in one place.
- Enterprise Knowledge AIGoverned knowledge layer: source-grounded answers with citations and audit trail.
Related AI solutions
Frequently asked questions
What counts as a small language model?
There is no fixed threshold. In practice it means a model compact enough to run on modest hardware, on your own infrastructure or on a device, and usually adapted to a limited set of tasks. What matters is not the parameter count but whether it meets your quality bar on your evaluation set.
Are small models always cheaper?
Per request, usually yes. In total cost, not always: self-hosting adds infrastructure, operations and people, and tuning or distillation has its own cost. Compare total cost at your real volume, including the cost of errors.
Can a small model match a frontier model?
On a narrow, well-defined task with good training data, it often gets close enough to be the better choice. On open-ended tasks that need broad knowledge or multi-step reasoning, the gap usually remains. Only an evaluation on your own cases tells you which situation you are in.
What is distillation and is it allowed?
Distillation trains a smaller model to imitate the outputs of a larger one on your task. Technically it is a well-established technique; whether you may use a given provider's outputs for it depends on that provider's terms of use, so check them before you start.
Do small models hallucinate less?
Not inherently. A small model tuned for a narrow task can be more consistent inside that task, but it can also be confidently wrong outside it. Grounding in your sources, constrained outputs and evaluation reduce unsupported answers whatever the model size.
Keep exploring
Related decisions
- Reasoning models vs standard LLMs: when is the extra thinking worth paying for?
- Open-source vs proprietary LLMs: how should an enterprise choose?
- Single AI vendor vs multi-model strategy: should you bet on one provider?
- Sovereign or on-prem AI vs cloud AI APIs: where should your models run?
- RAG vs fine-tuning: which one does your enterprise use case need?