Running Claude Code on a local model, measured

Local inference for Claude Code rarely fails on memory. It fails on time per turn. I measured this on 2026-05-28 on an M5 Max with 128 GB: a dense 27B model in MLX 4-bit gives you 25 to 29 tokens per second, and it thinks before every answer, 332 to 965 tokens on the three bigger tasks. Three things decide it: dense or MoE, forced thinking, and tool reliability. The model names below are a snapshot from May 2026, the three criteria aren't. The first two decide your wall-clock time. The third decides whether the result is usable at all.

What I measured

Hardware: M5 Max, 40-core GPU, 128 GB of unified memory. Model: qwopus3.6-27b-v2-mlx, dense, 27 billion parameters, MLX 4-bit, 15.15 GB on disk and 14.11 GiB loaded. Engine: LM Studio, context window 16384, temperature 0.2, load time 3.73 seconds. On the M4 Pro before it, the same experiment with an FP4 quant collapsed to roughly 0.1 tokens per second on prefill because that chip doesn't have the neural accelerators Apple names for MLX acceleration.

Four tasks, all of them things I actually do: write a function, call a tool, emit clean JSON, find a bug.

| Task | Result | Time | Tokens (thinking) | |---|---|---|---| | Write code (Levenshtein sort) | truncated first, correct on the second run | 46 s | 1328 (965) | | Call a tool (find TODOs) | valid call | 4 s | 73 (37) | | Emit a JSON structure | correct | 17 s | 356 (332) | | Fix a bug (async inside forEach) | correct diagnosis plus fix | 39 s | 927 (850) |

Four out of four correct. The model can do the work. It just can't do it at the pace agentic work requires.

The first run of the first task is the one worth keeping. The token budget was capped at 1400, and the model spent 1399 tokens thinking without writing a single line of code. On the retry with 3072 it stopped on its own at 1328. So max_tokens is a ceiling, not a target: raising it doesn't produce more output, it only stops the truncation.

The setting that eats everything else

If you try this, set this before you touch anything else:

export CLAUDE_CODE_ATTRIBUTION_HEADER=0

Otherwise Claude Code attaches a header to every request that busts server-side prompt caching. The reported effect is around a 90 percent slowdown. That's from issue #50085, reported by wadabum:

"using any sort of custom ANTHROPIC_BASE_URL has a good chance to make practically anything uncacheable."

Without that line set, the model you pick barely matters.

Two more from the same category. The system prompt plus tool definitions take up roughly 20,000 to 25,000 tokens before you type anything, which makes a 32K context window a demo rather than a work environment. And if you use subagents, a separate variable decides which model they get. If you're also running a second session against the same working tree, you have a different problem, which I wrote up in two sessions, one working tree.

Criterion 1: dense or MoE

25 to 29 tokens per second isn't a bad run for a dense 27B model at 4-bit on this hardware. It's the normal case. A mixture-of-experts model in the same weight class, say 35 billion parameters with around 3 billion active, would run two to four times faster, according to the research I did alongside the test, because only a fraction of the weights runs per token.

This is the one criterion you fix purely by picking a different model. No flag, no setting and no faster engine gets that factor back once the architecture is dense.

Criterion 2: the thinking tax

The model I tested thought before every answer: 37 tokens on the bare tool call, 332 to 965 on everything else, including on questions a single line would have answered. /no_think was ignored, and it still produced 668 reasoning tokens afterwards.

Dense decode speed multiplied by the thinking tax gives you 4 to 46 seconds per turn. A real Claude Code task reads files, edits them, runs tests and fixes what broke, which is 10 to 30 turns. That's 10 to 25 minutes for something Opus finishes in one or two on the same tasks. That difference sounds like a rounding error on a single turn. It isn't.

Which means the question isn't "which local model is smartest". It's "which local model answers without a preamble". For agentic work, a model without a forced thinking mode is worth more than a few points of SWE-bench.

Criterion 3: tools, not code

The capability that gets checked least is the one where local models lose most clearly. They can write code. Calling tools in the right order, reading the output and deriving the next step from it is a different skill, and it's weaker.

These numbers come from the web research around the test rather than from my own measurement, so treat them as ballpark figures: locally runnable models sit at roughly 62 to 73 percent on SWE-bench Verified, against 87.6 percent for Opus and 79.6 percent for Sonnet. On pure tool orchestration the gap is wider. I'm not putting a number on it, because the figures I found don't sit on a benchmark I can name.

The failure mode is uncomfortably quiet. The model writes out in prose what it would do instead of emitting the tool call. No error, no crash, just a step that never happened. If you run local, you need the same verification stages you'd use anyway, applied more strictly. Mine are in six stages before write access.

What it actually saves

Realistically you save a factor of three to five, so roughly 15 to 50 dollars a month instead of 100 to 200. That's calculated, not measured, and it only holds if you split the work: local for routine tasks and subagents, the large model for the long runs. The 10x figure you see quoted assumes everything runs locally and counts none of the waiting.

Before you go down this road, run the other calculation: what you actually pay today, per month, with the receipts open. For most people the bigger lever isn't which provider answers, it's how much context gets burned per turn. That's a full lesson in the AI Builder Guide, with the measurement points, a per-task budget, and the places where cost appears without anyone noticing.

People who run local for privacy or air-gap reasons stay there. People who came only for the bill mostly go back, from what I see in the community threads, usually right after the first long task took twenty minutes.

Before you pull that model

The model in my test was distilled on traces from a large commercial model, which is precisely the case that model's terms of use rule out. I accepted that for a one-off benchmark. It disqualifies the model for daily use, and the model card won't tell you that in those words.

So check what a model was trained on before you check how it scores. On Hugging Face that's usually one line in the dataset list.

Common questions

Do I need 128 GB?

No, but the amount of memory decides which class of model is even on the table. The rule of thumb from my decision matrix: the model file should take up at most 60 to 70 percent of RAM, with the rest going to macOS, the KV cache and your apps. At 16 GB you're looking at 8B to 14B at 4-bit, at 24 GB the 24B models come into range, and from 32 GB up an MoE model with few active parameters starts to make sense. At 128 GB, memory stops being a criterion at all, and only quality, tool calls and speed are left.

Ollama or LM Studio?

Both speak the Anthropic endpoint natively now, so Claude Code connects without a router. The difference that shows up in long sessions is prompt caching. Without it you pay for the whole system prompt on every single turn, and as noted above that prompt is five figures. I ran this test through LM Studio.

What still holds in three months

This line has been sitting at the top of my own measurement note since the day I ran it:

RE-RESEARCH BEFORE REUSE. This snapshot is from 2026-05-28. Local models and inference engines change on a weekly cadence.

That note is translated from the German original. It applies to the model names and version numbers, which is why they appear here only as an example from one point in time. What survives is the order of the checks: architecture first (dense or MoE), then answering behavior (does it think unprompted), then tool fidelity. Answer those three before you download anything and you'll save an afternoon.

A second line from the same collection, also translated, written down after the M4 Pro failure:

If the original manufacturer (Apple here) names a hardware generation as a requirement, trust that more than tech blogs saying it "runs everywhere".

That was the expensive lesson from the previous attempt, and it has very little to do with local models specifically.

If this is deeper than what you came for, and what you actually want is to see how working with an agent looks in the first place, the Werkbank course is free and answers that question before the model choice comes up.