The first question I ask before reaching for a large model is simple: how much of this problem is actually language? A surprising amount of “intelligence” in a useful application comes from defining the task well, finding the right context, and checking the answer before it reaches a user.
Small language models make those boundaries easier to see. Their limits are visible early, which forces better decisions about data and product design. They are also cheaper to run, easier to specialize, and often fast enough to disappear into the interface.
Start with the shape of the task
Imagine a support system that has to classify a request, retrieve two relevant documents, and produce a short answer. The final prose is only one step. A compact model may handle the complete loop if each stage has a clear contract.
Data before cleverness
The best examples show both what the model should do and where it should stop. I like to begin with a modest set of carefully reviewed cases, then add examples only when an evaluation reveals a new failure mode.
One difficult, representative example is worth more than a hundred easy examples that all test the same behavior.
A simple training record might keep the instruction, relevant context, expected response, and an explicit refusal condition together:
{
"instruction": "Answer using the supplied policy",
"context": "...",
"expected": "...",
"must_abstain_when": "evidence is missing"
}Measure the system, not only the model
Model accuracy is useful, but users experience latency, retrieval quality, citations, and recovery from bad inputs. A practical evaluation should record each of those layers separately. When the answer is wrong, that separation tells us where to work next.
This is the part I’m still exploring: how small a model can become while the complete system remains useful, honest, and pleasant to use.