My own measurements
Which model for which task
I measure models on my own tasks. Here are the numbers, how I got them, and what they don't prove.
For structured extraction I use Sonnet 5.5; for the booking decision, an LLM rather than a cheap decision model. For the other tasks, my case counts don't support a verdict yet. You can rebuild the method below with your own cases.
As of:
How I measure
Public benchmarks tell you little about how a model does on your task. So every measurement here runs on my own labeled cases. You can copy the points below for your own tasks.
Your own cases with a gold answer.
Each task has labeled cases, either from real work or modeled on it. The gold answer is the correct answer, fixed before the run. On the decision tasks, the instructions to the model are word for word the ones used in production.
Fix the criterion before the first run.
I write down the threshold, metric, and selection rule before any model runs. Otherwise I'd bend the rule to fit the result without noticing. Any deviation goes into the log with a date.
Several runs per case where variance matters.
For extraction, every model ran three times over every case. On the decision tasks, Jev ran three times per case and the other models once.
Compare paired, with a margin.
All models run on the same cases. I compare the accuracy difference across those cases (paired Newcombe interval, 95%) against a margin: 5 percentage points, 2 for decisions that move money. A model only counts as a replacement if the lower bound of the interval stays above minus the margin. Overlapping individual intervals prove nothing. I only noticed that after the first decision run, so I set the margin there on September 19, 2026, when I already knew the results.
A second pass that tries to refute the numbers.
An independent recomputation checks every number, and for extraction a holdout labeled before any prompt work checks the choice. The decision tasks had no holdout. There, a paired re-analysis of the same data overturned my first verdicts.
Result 1: structured extraction
measured October 29, 2026
A product I'm building runs an interview and has to fill typed fields from free-form answers. Five models, 21 cases, three runs per case: 315 runs, all valid. The score is micro-F1 over the captured facts, threshold 0.90. Precision is how many returned facts are right, recall is how many of the right facts were found, and F1 combines the two.
| Model | Effort | Micro-F1 | Precision | Recall | USD per run (median) | USD per 1,000 runs |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | high | 0.9512 | 0.9468 | 0.9556 | 0.0149 | 14.90 |
| Claude Opus 5.5 | medium | 0.9490 | 0.9337 | 0.9648 | 0.0285 | 28.50 |
| GPT-6.1 Sol | medium | 0.9473 | 0.9304 | 0.9648 | no price source | no price source |
| GPT-6 Astra | medium | 0.9361 | 0.9227 | 0.9500 | no price source | no price source |
| Claude Haiku 5.5 | medium | 0.9348 | 0.9149 | 0.9556 | 0.0014 | 1.40 |
USD per 1,000 runs = median list price per run × 1,000.
Micro-F1 compared
Bars start at the 0.90 threshold and end at 1.00. The table above is the source.
Holdout: only Sonnet 5.5 and Haiku 5.5 ran on 20 synthetic cases, labeled before any prompt work. Sonnet reaches 0.9304, Haiku 0.9187. An independent recomputation confirms every value and the ranking.
My pick is Sonnet 5.5 at high effort: first on the gold set, ahead of Haiku 5.5 on the holdout, and half the price of Opus. Sonnet, Opus, GPT-6.1 Sol, and Haiku can't be separated statistically; GPT-6 Astra is measurably behind GPT-6.1 Sol. Haiku 5.5 costs about a tenth of Sonnet and passed, but it's last on the gold set. If cost matters more to you, Haiku is worth its own run on your cases.
What the first version taught me
In the first version (October 28, 2026), no model passed. The best reached 0.7398. The models weren't the problem. The gold set and the prompt described two different tasks: the labels included inferred facts, while the prompt asked for stated facts only. 64% of gold entries were inferred. If every model fails the same way, check first whether gold and prompt ask the same question.
Result 2: decisions and classification
measured September 17 to October 20, 2026
These are short, typed decisions: one option out of N, yes or no. Five labeled tasks from my own projects, eight models in the main run, with newer models added later on the same harness. The table below shows four models from the main run and three measured later. The linked article counts a sixth task that I leave out here.
Jev (from TypeSafe AI, pronounced like “Jeff”) isn't a chat model. It answers typed questions about a state and returns a probability for each option. You pay almost entirely for input tokens, so it's very cheap. The question was: is Jev enough where an LLM makes the call today?
Booking decision: book automatically, send to review, or abstain
This is the task that moves money (bookkeeping entries): 200 cases, margin 2 percentage points. A wrong automatic booking costs more here than an unnecessary review.
| Model | Accuracy | Wrong auto-bookings | USD per 1,000 calls | Measured |
|---|---|---|---|---|
| Gemini 3.8 Flash | 100.0% | 0 | 4.551 | |
| Gemini 4 Argon (low) | 100.0% | 0 | not reported | |
| GPT-5.6 Luna | 99.5% | 1 | 0.416 | |
| Claude Haiku 5.5 (low) | 99.0% | 0 | 0.377 | |
| Mercury Decide | 93.5% | 0 | 0.025 | |
| Jev | 83.5% | 0 | 0.067 | |
| Claude Haiku 4.5 | 63.5% | 12 | 1.922 |
Paired comparison per task
Accuracy difference, first model minus second, paired over the same cases, with a 95% interval. “Undecided” means the data neither proves the replacement nor rules it out. For a switch, that counts as a no: measure more cases first.
| Task | Cases | Comparison | Difference in points (95% interval) | Verdict | Measured |
|---|---|---|---|---|---|
| Booking decision | 200 | Jev vs. Gemini 3.8 Flash | -16.5 [-22.3; -11.6] | worse, confirmed | |
| Support request to the right help article (17 articles) | 67 | Jev vs. GPT-5.6 Luna | 0.0 [-6.6; +6.6] | undecided | |
| Job title to role family (32 families) | 96 | Jev vs. GPT-5.6 Luna | 0.0 [-7.2; +7.2] | undecided | |
| Fact check: claim unsupported? | 47 | Jev vs. Gemini 3.8 Flash | -2.1 [-11.1; +5.6] | undecided | |
| Does the application fit the job? (3 levels) | 18 | Jev vs. Gemini 3.8 Flash | -5.6 [-25.8; +12.6] | undecided | |
| Booking decision | 200 | Mercury Decide vs. Jev | +9.0 [+4.4; +14.1] | better, confirmed |
What held
- Jev didn't auto-book a single case wrongly: 0 of 200 (September 17, 2026). The tradeoff: at 83.5% correct decisions it sits clearly below the top. Where it got a case wrong, it sent it to review or abstained.
- Haiku 5.5 at low effort replaces Haiku 4.5 on the booking decision: 99.0% with 0 wrong auto-bookings (October 7, 2026) against 63.5% with 12 (September 17, 2026). Medium or high effort doesn't pull ahead beyond noise on any task.
- On the booking decision, Mercury Decide (a decision model with the same interface as Jev) beats Jev by 9.0 percentage points, paired (October 20, 2026). The comparison uses a Jev run from the same window (84.5%), not the 83.5% in the table. It doesn't come close to the LLMs at 99% and above.
- Gemini 4 Argon (low effort) scores 100% on the booking decision, level with Gemini 3.8 Flash (October 13, 2026). I have no cost figure for this task alone, so the table shows no price. GPT-5.6 Luna stays in use.
What didn't hold
- By the overlap rule, Jev would have counted as a replacement on four of these five tasks. After a paired re-analysis on September 19, 2026, none of those verdicts holds: four tasks are undecided, and the booking decision stays a clear no, 83.5% against 100% for Gemini 3.8 Flash.
- The cause is almost always the case count. To show a model is at most 5 points worse, support routing needs around 100 paired cases, role family and fact check 160 to 220. With 47, 67, or 96 cases, “undecided” was predictable. So size your case set to the margin before you run.
- Support routing is the textbook case: Jev and Luna both at 98.5%, and 67 cases still can't rule out a 6.6-point loss.
What the difference costs
On the booking decision, Jev costs $0.067 per 1,000 calls, GPT-5.6 Luna $0.416 (6.2 times as much), and Gemini 3.8 Flash $4.551 (67.9 times as much; as of September 17, 2026, billed through the Vercel AI Gateway). Even at 200 decisions a month, the most expensive of the three costs less than a dollar. So price isn't what decides here; wrong auto-bookings are.
For the booking decision I stay with an LLM, currently GPT-5.6 Luna. Haiku 5.5 at low effort is the candidate next to it: 0 wrong auto-bookings at about the same cost. I'll only switch after a shadow run. Jev and Mercury Decide are cheaper but clearly behind on accuracy. For the other four tasks, my case counts don't support a verdict.
What I don't know
- Case counts are small: 18 to 200 cases per decision task, 21 cases for extraction. Two tasks (47 and 18 cases) only work as a feasibility check.
- These are only my own tasks. They say nothing about coding, writing German text, or long context.
- Prices are list prices on the day of measurement. They change, and new tokenizers shift costs on top of that.
- All numbers apply to the model version and date named. In September, Jev couldn't be pinned to a version through the gateway. Since October there are versioned IDs.
- Reference values from different weeks can drift. Where it mattered, I re-measured in the same window.
- Some extraction prompt conventions were decided on the gold cases. Only the holdout is independent of that, and Sonnet 5.5 passes there too.
Keep reading
The method behind this is in two courses and one article.
- Course: Verify AI output reliably
How you check that agent work is right before you rely on it.
- Course: Multi-Agent Orchestration in Practice
How you split work across several agents and models.
- Reicht das billigere Modell? (Is the cheaper model enough?) (opens in new tab)
Article on gotzendorfer.at, October 21, 2026, German only.