What it takes to run a model on your own machine

Five words that do most of the work

The rest of this page uses these constantly, and none of them are obvious from the outside.

Runtime

The program that actually executes a model. A model file is a description of a calculation; the runtime is what performs it, and decides which parts run on the GPU and which fall back to the processor. Two runtimes can run the same file at very different speeds — which turned out to matter more than the model did.

Three names that are easy to confuse

Name	What it is
ONNX	An open file format for models. Not a runtime, and not a program at all — a way of writing a calculation down so more than one runtime can read it.
ONNX Runtime ORT	The general-purpose engine that executes such a file. Hand it a graph and it runs it, once.
onnxruntime-genai	A separate library that sits on top of ORT and adds the generative loop: decoding token by token, managing the KV cache, and applying the tokenizer and chat template. It does not replace ORT — it loads it and calls it.

So “ORT” below means the plain engine, and “genai” means the layer above it.

The distinction is the whole point of the fifth surprise. Running a graph once is what ORT is built for, and it is what speech synthesis and transcription need — a sentence in, audio out. A language model needs the same graph run once per token, thousands of times for one answer, handing its working notes back in at every step. Whatever a single run costs beyond the arithmetic is paid again on each of those thousands of runs. Closing that gap is what genai is for, and it is why the language model moved to it while speech and transcription stayed exactly where they were.

Context

Everything the model can see at once: your question, the conversation before it, and the answer as it is being written. It is measured in tokens — roughly three-quarters of a word each. When people say a model “forgets”, they usually mean something fell out of its context.

Attention head

Models read by comparing every word against every other word. Each comparison runs several times in parallel with different emphases — one head might track grammar, another the subject being discussed.Head dimension is how much room each one gets. More heads and a wider head mean a better reader and more memory.

KV cache

An answer is produced one token at a time, and each one is written in the light of everything before it. Rather than re-read the whole conversation for every token, the model keeps its working notes — the keys and values — from the tokens already seen. That cache is why the tenth word is as fast as the first, and it is usually the largest thing in memory after the model itself.

Quantization, and the export

The same model can be published at different numeric precisions: full precision, half, 8-bit, 4-bit. Each is an export — a separate file. They are not simply better or worse versions of each other, and choosing between them by file size is how this page began.

The 4-bit model is bigger than the 8-bit one

Reach for the smallest download and you would take the one labelled 4-bit. For Qwen2.5-0.5B-Instruct, every export published looks like this:

The 4-bit export is the second largest useful one, and the 8-bit is 262 MiB smaller. The reason is the embedding table. This model has a 151,936-token vocabulary against 896 hidden dimensions — 136 million parameters,28% of a 494M model, shared between the input embedding and the output head. The 4-bit export quantises the transformer and leaves that table at full precision: 519 MiB of its 750. The 8-bit export quantises everything.

So the one Amber ships is both smaller and higher precision on the weights that do the reasoning. Two axes, and they pointed the same way — which is not what the label suggested.

An export that exists is not an export that loads

The arithmetic above recommends model_q4f16.onnx: 461 MiB, the same 4-bit transformer weights, the embedding table at half precision instead of full. It is the obvious choice and it does not work. ONNX Runtime crashes loading it, inside a graph optimisation that inserts a precision cast and then cannot find it again.

The same export of a different model — Qwen3-4B — loads without complaint. There is no way to tell from the outside which will do which, so Amber has a tool whose entire job is to try:graph-probe creates a session, reports what the model declares, and treats a failed load as a result rather than an error. Ten minutes of running it replaces a week of reasoning about it.

The memory is in the geometry

A language model keeps a cache of every token it has already seen — two tensors per layer, per attention head. That cache is what makes the second question faster than the first, and its cost per token is a property of the model's shape rather than a number anyone chooses:

Six times the memory for the same conversation length. A cap that is generous for one model is punitive for the other, which is why the service reads these numbers out of the graph at load rather than having them compiled in — and why the larger model needed a different cache element type, not just larger numbers.

Reasoning models answer twice

Ask Qwen3 a one-sentence question and it replies with several hundred tokens of deliberation first — “Okay, the user is asking… let me check if there is a standard definition” — and then the answer. The answer is genuinely better than a smaller model's. It also arrives last.

On a surface that reads answers aloud, that is not a cosmetic problem: the assistant would speak its own monologue. The fix is to suppress the deliberation in the prompt rather than strip it from the output, because stripping it still pays for every token of it.

Six tokens a second is not a daily driver

Qwen3-4B on the CPU of the machine this was measured on generates aboutsix tokens per second. A three-hundred-token answer takes fifty seconds, and the deliberation above multiplies the count. A standard D-Bus call gives up at twenty-five seconds — the answer is still being written when the caller has stopped listening.

So Amber ships a small model that answers immediately on any machine, and treats it as a preview rather than the destination. The larger model is worth its download only with a GPU under it.

That was where this page originally stopped, with the hardware as the thing that decides. It is half the story: the two surprises below are about a file built for the wrong runtime, and a runtime that was leaving most of the card unused. Both were worth more than any upgrade.

A file can be built for a runtime you are not using

Order the exports by download size and the smallest useful one isq4f16 — 4-bit weights with half-precision arithmetic. Load it and it runs. Ask it a question and one model answers with pure whitespace, another with an exclamation mark repeated until it runs out of room.

An exclamation mark is token zero. It is what “pick the highest scoring token” returns when every score is not a number— the arithmetic overflowed half precision, and nothing anywhere reported an error, because an overflow becomes infinity, then not-a-number, then a perfectly ordinary token id.

Four runs found it, each removing one suspect. A Qwen2.5 model in the same format answers correctly, so the format alone is not the fault. A smaller Qwen3 fails identically to the larger one, so it is not one bad file. The same smaller model in an 8-bit export answers immediately. It was never the model: q4f16 exists for WebGPU, and outside that runtime this family overflows it.

The lesson is not about one format. An export is chosen for the runtime that will open it, and the arithmetic of file sizes will happily recommend one that cannot work.

The runtime mattered more than the model

A language model writes one token at a time, and each one is a full pass over the calculation. A general-purpose runtime pays its overhead on every one of those passes, and there are thousands per answer.

The scale of it is visible in one number. When a runtime meets a step it has no GPU implementation for, it moves the data back to the processor and returns it afterwards — and it reports how many such crossings a model needs. Speech synthesis needed 39, and runs once per sentence. The language model needed 449, and runs once per token: about 115,000 crossings for a single answer, against 39 for a spoken sentence.

Swapping to a runtime built for this shape of work — one that keeps the conversation’s working notes on the GPU instead of shuttling them — changed the numbers completely, on the same card:

The last row is the point. A 14-billion-parameter model on genai is faster than a 4-billion one on plain ORT, on a 16 GB card. The ceiling everyone assumes is the hardware was, here, the software reading the file.

Speech and transcription were left exactly as they were. They run once per sentence, not once per token, so they never paid this cost and had nothing to gain.

Where the numbers live

Every figure here is reproducible from the source: sizes come from the publishing API, geometry from the model file itself, and speeds from wall-clock runs. The engineering notes behind them are in the back end's own documentation, and the tool that reads a model's declared shape ships with it.

Amberlin is the surface these models answer through; the packages are how they are installed; Amberlin Settings is where these figures meet the machine in front of you; the stance is why any of it runs locally at all.