The Local-Inference Bill Nobody Adds Up

The pitch for local inference is honest and specific: you stop paying per token. What it omits is that per-token pricing is not the only way a model costs money — it is just the only way that arrives as a bill with a number on it.

Move the model onto your own machine and the invoice disappears. Four costs do not. They are capacity, idle time, context, and your own hours, and because none of them is itemised, they are usually estimated at zero. The result is the same mistake people make with any “free” infrastructure: the spend does not vanish, it becomes invisible, and invisible spend does not get managed.

This is not an argument against local inference. It is an argument for pricing it correctly before you commit, because the honest version is often still a good deal — just for different reasons than the ones being advertised.

Line one: capacity, paid by the month regardless

The first cost is the machine, and the size of it is not a matter of taste. A model has a memory floor that you can read off the file: the weights must be resident before the first token, plus the KV cache for your context, plus runtime overhead.

That floor is much higher than the floor for the agent that calls it. The agent projects publish their own numbers, and they are small — Hermes lists a minimum of 1 GB of memory and a recommended 2–4 GB; OpenClaw’s minimum is 1 GB with 2 GB recommended. A quantised model in the 8–14 billion parameter range does not fit in either. The two budgets and how to compute them are covered in more detail separately; what matters here is the gap, because the gap is what you are buying.

And you buy it monthly, for as long as you want the capability. That is the structural difference from an API: an API bill scales to zero when you stop using it, and a machine does not. The plan you chose for the week you were experimenting is the plan you are still paying for in the quiet month.

Line two: idle, which is the whole problem

Agents have an awkward usage profile. They are bursty by nature: a message, a minute of work, then hours of nothing until a scheduled job fires or a human asks something. Peak capacity is set by the busiest minute; payment is for the whole month.

That mismatch is the actual cost of self-hosting, and it is invisible on a receipt because the machine is billed the same either way. You are not paying for tokens; you are paying for the ability to answer instantly at 3 a.m. during a week when nothing happens.

There is no way around this except accepting it. The mitigation is not a cheaper plan — it is a smaller model, and the price of a smaller model is quality. Which brings the third line.

Line three: the context multiplier

The number people budget is the model file. The number they pay is larger, and Ollama’s own documentation says why:

“Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.”

Two variables, both usually left at defaults, and both multiply. Context length is the obvious one — a long-context assistant costs more memory than a short-context one, per request and in total. OLLAMA_NUM_PARALLEL is the one people miss: it is the number of requests the server will handle simultaneously, and raising it to make the agent feel responsive multiplies the memory requirement rather than adding to it.

The same documentation describes the loading rule in terms of what is available, not what is installed:

“If your system has sufficient available memory (system memory when using CPU inference, or VRAM for GPU inference) then multiple models can be loaded at the same time.”

Available. On a box that also runs the agent, the database, and a web server, the difference between installed and available is where the out-of-memory kills come from — and the fix is a bigger machine, which is line one again, larger.

Line four: your hours

The line that never appears in any comparison is the time it takes to keep the thing working: choosing a quantisation and evaluating whether it is good enough, re-pulling models when a new generation lands, watching for the release that breaks a tool-calling convention, and rebuilding the part of your workflow that depended on a model that was retired.

None of this is hard. All of it is recurring. The honest accounting is to multiply the hours by what an hour of your time is worth to you, and to do it before the first download rather than after the third frustrating evening.

There is a specific trap here worth naming. A local model is a large one-time download and a monthly machine, so the cost per token falls the longer you keep the setup — the machine amortises and the model is already paid for. That is the argument for local inference, and it is a good one. But model generations turn over faster than most people expect, and each turnover resets the clock: a new model means new weights, a new quantisation evaluation, and often more memory. The amortisation curve keeps restarting, and nobody puts that on the chart.

The break-even, written as a formula

The comparison is only meaningful in one unit: cost per million tokens. Convert both sides to it and the decision becomes arithmetic.

For the API side:

cost per Mtok = (input tokens × input price + output tokens × output price) ÷ 1,000,000

For the local side:

cost per Mtok = (monthly machine cost + monthly time cost) ÷ (tokens per month ÷ 1,000,000)

So:

monthly time cost = hours per month × your hourly value

Nothing in those three lines contains a number, and that is deliberate — prices change, and a site that prints today’s figures is wrong in a quarter. What does not change is the shape, and the shape says something unfashionable: the local side has a large fixed numerator and a variable denominator. If your tokens per month is small, the local cost per million tokens is not high, it is absurd. If it is large, the fraction collapses and local wins decisively.

The crossover is entirely a function of volume. Which means the first question is not “can I run a model” but “how many tokens do I actually send in a month”. Almost nobody knows the answer, and it is the only number that matters.

Where local genuinely wins

Three cases, and none of them is about price:

The data cannot leave. If the content going into the model is not allowed to cross a network boundary, then the comparison is not between two prices — it is between local inference and not doing the task. This is the strongest reason, and it does not need a break-even at all.

Volume is high and steady. A pipeline that processes the same kind of input continuously, all day, every day, is the profile where owned capacity beats metered capacity. The machine’s idle time approaches zero, and the fixed numerator spreads over a large denominator.

You already own suitable hardware. A GPU that exists for another purpose has a marginal cost of nearly zero for inference. Buying one specifically for this is a different decision, and usually a worse one.

Outside those three, what you are buying is not cheaper inference. It is independence — and independence is a legitimate thing to want, as long as you are not calling it savings.

What to do this week

1. Measure your actual token volume for a normal week, including the scheduled jobs. Input and output separately, because they are priced differently on every API.
2. Read the model file size you would need, add the KV cache for your context length, and check what that leaves a machine. The arithmetic is short.
3. Price the machine, not the model. The model is a download; the machine is the recurring commitment.
4. Estimate the hours, honestly, per month. If the number embarrasses you, that is the useful signal.
5. Put all of it into the formula above and compare against the API spend for the same volume.

If the answer is “the API is cheaper”, that is not a defeat — it is a correct decision made before spending money, which is the entire point of doing the arithmetic. And if the answer is “local is cheaper by a wide margin”, you now have the number that justifies the machine, which is a much better position than hoping it works out.

One last note on the framing. The reason “no token fees” travels so well is that it is measurable and true, while the costs it displaces are diffuse and arguable. That asymmetry is not unique to this decision — it is the same one that makes a three-year hosting total look different from a first-year promo price — and the remedy is the same. Put both sides in the same unit, over the same period, and the argument reduces to a division.