Small Language Models: When a Small Model Is Enough

This article counts a language model as small if it has roughly 0.5 to 9 billion parameters. That size runs on a laptop with 8 to 16 GB of RAM, or on a recent phone. Models of this size now handle a large share of the work teams send to frontier APIs, at a small fraction of the price. They also fail in predictable places. Everything below is as of October 2026.

The table lists the small models worth testing first. "Vendor's stated purpose" is the vendor's own framing, not a benchmark result.

Rows marked "press" come from coverage of the vendor's launch. The vendor pages (Hugging Face model cards, Google's blog, Ollama's library) were not reachable when we checked this list on 30 October 2026. Check a row against the model card before quoting it elsewhere.

Two models sit just above the range. gpt-oss-20b (OpenAI, Apache 2.0) "fits into 16 GB of memory" according to TestingCatalog's launch report. Nemotron 3 Nano (NVIDIA) has 30B parameters in total with 3B active per token, according to The New Stack. Both run like a small model but need the RAM of a mid-sized one.

What small models do well

Each task below has a short input and a constrained output, and you can check the result.

• Classification. Intent, sentiment, language, spam or not, which of twelve queues a ticket belongs to. The label set is fixed, so the model cannot wander.
• Extraction. Names, dates, amounts and IDs from a receipt, an email or a form, returned as JSON against a schema.
• Routing. Deciding which model, tool or prompt a request should go to. The NVIDIA paper below argues for this split between small and large models. Routing also has the largest cost effect, because the small model decides how often the expensive one runs.
• Summarising short texts. A support thread, a meeting note, a product review. Not a 300-page report.
• On-device assistants. Autocomplete, dictation clean-up, reply suggestions, offline translation. Apple positions AFM 3 Core for lightweight on-device text work (and the larger Core Advanced for dictation and Siri). According to The Next Web, Gemma 4 E2B and E4B are the basis of Gemini Nano 4 on Android.
• Tool-call formatting. Turning a decision into a valid function call. Phi-4-mini supports function calling, according to frankx's analysis of the Phi-4 family.

The strongest written case for this split is a position paper, Small Language Models are the Future of Agentic AI (arXiv:2506.02153, version 2 dated 15 September 2025). Peter Belcak wrote it with seven co-authors from NVIDIA Research and the Georgia Institute of Technology. They argue that most calls inside an agent are narrow, repetitive and formatted. A model under about 10 billion parameters is therefore "sufficiently powerful, inherently more suitable, and necessarily more economical" for them. A large one should be called only when the task needs open-ended conversation. The authors define a small model as one that fits on a common consumer device with low enough latency for one user. By that definition, "as of 2025" most models under 10 billion parameters qualify. The paper is not a benchmark study, and no venue is stated. Treat its claim as a hypothesis to test on your own traffic.

Where they fail

The failures cluster around four things.

Multi-step reasoning. A small model that answers a one-step question correctly 95% of the time will not keep that rate across a chain. If each of ten steps succeeds independently with 95% probability, the whole chain succeeds in about 60% of runs. The same arithmetic applies to any model. Small models, though, start from a lower per-step rate on hard steps, so their curve drops sooner. The practical sign is an error rate that rises with task length rather than with task topic.

Long documents. Four of the eight rows in the table list a 128K context window. The window measures how much text the model can accept, which is more than it can use well. A small model given 80 pages tends to anchor on the beginning and the end and lose the middle. Chunk the document and ask one question per chunk, or send the whole thing to a larger model.

Obscure knowledge. A 3B model has less room to store facts. It will know what a VAT invoice is but not the clauses of a specific regional tax code. If a task needs recall of rare details rather than reading of supplied ones, put that information in the prompt or send the task to a larger model.

Instruction following under pressure. Give a small model a system prompt with fifteen rules and a tricky input, and it will often follow most of them but not all. The usual symptoms are invalid JSON, a missed "never do X" rule, or a reply in the wrong language. Constrained decoding (a JSON schema enforced by the runtime) fixes the format problem but leaves the judgement problem.

None of the model cards behind this article publish task-length error curves. The sign above is therefore a rule from practice rather than a vendor number. Measure it on your own examples with the protocol further down.

What they cost to run

You can run a small model in three ways. Which one is cheapest depends on your volume and your ops capacity.

Path A, a vendor's small model over its API

Each big vendor sells a small model at the bottom of its range. Vendors do not disclose parameter counts, so "small" here means the vendor's cheapest tier.

All of these models cost 50% less through a batch endpoint (OpenAI, Anthropic, Google and Mistral each state 50%). Ministral 3 is the odd one out. It is both a vendor API product and an open download, so you can start on path A and move to path C with the same model.

Path B, open weights on someone else's GPUs

Hosts run the public weights and charge per token. One model's price can vary several times over with host and variant. Llama 3.1 8B on Cloudflare alone runs from $0.045 to $0.282 input. The model name does not tell you the price.

OpenRouter also offers a free lane. Models with a :free suffix are limited to 20 requests per minute and 50 per day until you have bought at least 10 credits. After that the daily limit is 1,000. That covers the 200-example test below but not production.

Path C, your own hardware

We could verify memory figures for this path but no prices. The Ollama README gives the rule of thumb: "at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models." The same page lists download sizes at 4-bit quantisation.

DataNorth's coverage of Gemma 4 adds two data points. E4B "needs 12 to 16 GB" and runs well on Apple Silicon. E2B on a Raspberry Pi 5 decodes at 7.6 tokens per second in under 1.5 GB at 2-bit. No tokens-per-second benchmark for M-series Macs or consumer GPUs met the sourcing rule for this article, so we quote none.

Path C has no per-token charge. You pay instead for the machine, the electricity and the engineer who keeps the runtime patched. The local versus cloud comparison works through that arithmetic.

The spread between the cheapest and the most expensive bar is 25x, for models that all sit in the same "small" tier. Output price matters for generation tasks and barely at all for classification, where the answer is a few tokens.

How to decide

A short protocol settles the question better than opinion. It takes an afternoon.

1. Collect 200 real examples from production logs, not synthetic ones. Include the ugly ones: empty fields, mixed languages, the long tail.
2. Pick one metric you can compute without a human: exact match for classification, field-level F1 for extraction, schema validity plus a spot check for generation.
3. Run the small model and a frontier model on all 200 with the same prompt. Record the metric, the latency and the cost per call.
4. Look at the gap by example length. If the small model matches the frontier model on short inputs and falls off on long ones, you have found your routing threshold. That result is no verdict against small models.
5. Route. Send everything under the threshold to the small model and everything above it to the large one. Log both, so you can re-run the comparison next quarter when the models change.

The model routing guide shows how to build the router itself. The hybrid local-cloud playbook deals with the case where the small model runs on your own machines.

Further viewing

Google's Gemma 3n announcement is a clear short explanation of what a phone-sized model is built to do. It dates from May 2025. The current family, Gemma 4, shares its design goals.

Andrej Karpathy placed one of the most-quoted bets on the trend on the day GPT-4o mini launched, in July 2024.

What to do next

Pull 200 real requests from your logs this week. Run the five-step protocol above against GPT-6 Luna or Claude Haiku 5.5 first, since neither needs any setup. If the small model holds on short inputs, add the length threshold to your router and measure the bill for a month. If you want the test designed and run for you, we do this at Divero as a fixed-scope engagement.

Sources

• Small Language Models are the Future of Agentic AI (arXiv:2506.02153v2)
• Ollama README (mirror): memory requirements and model sizes
• OpenAI models and pricing
• Anthropic pricing
• Gemini API pricing
• Mistral inference pricing
• Cloudflare Workers AI pricing
• Groq models
• Together pricing
• Google releases Gemma 4 open models (DataNorth)