When a bigger model earns its cost, and when it doesn't

What the extra computation actually buys

Two mechanisms are at work, sometimes separately and usually together. One is size: more parameters and more arithmetic per token, which shows up not as a different kind of answer but as a lower failure rate on the same prompt. The other is deliberation: some tiers are allowed to generate a long chain of intermediate tokens before they say anything to you, drafting their reasoning before they answer. Whatever a vendor calls its tiers, the resource being sold is compute per answer.

That resource pays for a specific set of behaviours — weighing more than one reading of an ambiguous instruction, trying an approach and abandoning it, noticing that step six contradicts step two. Now take a question with one lookup and one sentence of output: convert a date format, restate a paragraph more plainly. Nothing to explore, no path to abandon, no contradiction to catch. The small tier and the large tier produce the same sentence. You paid more for it and waited longer — a premium on a policy that will never pay out.

Four properties that predict you need a bigger tier

The first is the number of steps that must hold together. A task that decomposes into a chain — read this, infer that, apply the rule, check against the constraint — fails whole if any link fails. The arithmetic is unkind: ten steps at 95% each, if their failures are independent, multiply out to about 60%. Real failures are correlated, and a model sometimes catches its own earlier mistake, so treat 60% as a direction rather than a forecast. The direction is that a small gain in per-step reliability matters far more than it looks once steps multiply.

The second is what a wrong answer costs. An error that costs you thirty seconds of re-reading is cheap; a wrong clause in a contract, a wrong dosage conversion, a migration script run against production is not. When the loss is that asymmetric, the average case is not the thing to optimise. For a single document of that kind, the price of the largest tier is small next to the tail you are protecting against — and it stays small even if the bigger tier removes only part of the risk.

The third is synthesis versus recall. "What is the invoice number" is recall: the answer sits in one place and the work is locating it. "Does clause 14 survive the amendment in annex C, and what does that do to the delivery date" requires building a representation of several things at once and reasoning from it. The volume of material is not what makes this hard; the number of pieces that must be true at the same moment is.

The fourth is whether you can check the answer cheaply yourself. If the check is mechanical — the tests pass, the figure appears verbatim in the source, the column sums — a cheap tier plus a check often beats an expensive tier alone, and stays cheaper even after counting the retries. If checking requires the very expertise you were buying, the quality of the first attempt is all you have.

Two things that feel like they predict it, and do not

The first is how important the subject feels. Medical, legal and financial material creates an instinct to reach for the highest tier available. The instinct is sometimes right, but for the reason above — the cost of being wrong — not because the words are medical. Transcribing lab values into a table is an easy task in a serious field. Building a fair rota for five people sounds trivial and can be genuinely hard. Judge the operation, not the subject matter.

The second is the length of the input. Pulling every renewal date out of a two-hundred-page contract is a long task made of many easy steps, not one hard step, and a tier that handles one paragraph correctly will usually get each instance right — but not necessarily find all of them, because recall on many-item extraction falls as the count of items and the distance between them grows. Models are also uneven at using material buried mid-input, and tiers differ in how much context they accept and how well they use it. That is a context problem rather than a depth problem, and it is more often solved by retrieving the right few pages — the subject of our article on answering from your own documents — than by moving up a tier.

The economics, plainly

Two costs rise between tiers: money per call, and time to an answer. The second is easy to underestimate — a tier that deliberates before replying may produce a great many tokens you never see, which in an interactive product shows up as a wait people notice and dislike, and in a batch pipeline as reduced throughput. Quality rises too, but not proportionally, and on any task the cheaper tier already handles it cannot rise at all: there is no headroom above correct. So the quality bought per unit of money falls as you climb, and it falls fastest on the easy work that usually makes up most of the volume.

This is why the calculus flips with scale. For one important document, take the best tier and stop thinking about it. For fifty thousand calls a day, the target is the cheapest tier that is adequate, where adequate means measured against a failure rate you chose deliberately, not "the best I can get". The common structure is to run cheap by default and escalate on a trigger: a failed check, a user pushing back, a case matching a pattern you already know is hard. That works only when the trigger is a real signal — a model's own statement that it is confident is a weak one.

Test it on your own hardest examples

Published comparisons between tiers are averages over somebody else's distribution of tasks, and the gap on a public leaderboard tells you little about the gap on your work — the argument of our article on reading benchmark scores. Your workload is narrower than any benchmark and probably clustered at one end of it.

The method is unglamorous and takes a few hours. Collect twenty to forty real examples from the harder end of the work you actually do, with the correct answers written down, or at minimum a written definition of what counts as wrong. Run the same prompt on both tiers. Grade blind: whoever marks the outputs should not know which tier produced which. Count failures, not preferences — preference tracks fluency and tone more than it tracks correctness.

Two cautions. The same prompt can return different answers on different runs, temperature 0 included, so run each case more than once; a difference of one case in thirty is noise. And read the failures instead of only counting them. If the cheaper tier fails on one recognisable kind of case, you have found a real boundary and can route that kind alone to the bigger tier. Scattered failures that look like misread instructions usually mean the prompt or the input data is the problem, and it will follow you up the tiers as a more expensive version of itself.

Re-run the test when you change the prompt substantially, and again when the vendor updates the model: a tier is a moving target, and an update can shift it in either direction on your particular cases.

Firas AI is arranged along this axis: Mini, Pro, Ultra and Max differ in how much reasoning goes into a single answer. Nothing above is specific to them; the test works the same way on any family with tiers.