Benchmarking across some of our published families

Fri, 13 Nov 2026 20:44:00 +0000

You’re running a local assistant on a 16 GB card, or you’ve got two of them, and you’re trying to decide which model family to live with. That’s who this is for. Over the last few days I ran the unified benchmark suite against five of the tiered model families we publish. Same hardware, same flags, one variable: the model file. Here’s what came out, and what I think it’s telling us.

The setup: two 16 GB Blackwell cards (an RTX 5070 Ti and an RTX 5060 Ti), the same llama.cpp build for every run, and the suite’s iron rule that only the -m flag changes between models. Every number below is from that harness, so the tiers are mutually comparable and the families are comparable to each other. I’m reporting what we measured. And I’ll say it up front: these numbers are one input. A suite tells you how a model behaves under controlled conditions. How it feels to work with it is a separate question. I’ll come back to that, because it changes how you should read everything below.

The five families, and what each one is:

Qwen3.8-27B: 27B dense, hybrid Gated DeltaNet + Gated Attention, native vision, 262K context, and a multi-token-prediction (MTP) speculative head baked into every file. The nine-tier ladder plus two stripped-down BUDGET/STARVED variants.
Nanbeige4.3-33B: 33B dense Llama-arch, 512K native context, no MoE, no MTP. Seven tiers.
Gemma-4-31B-IT: 31B dense. Five tiers.
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic: a 9B hybrid vision model (24 linear-attention blocks, 8 full-attention blocks) with the MTP head and vision tower left intact. Seven tiers.
Qwen3.8-27B-Ultimate-Heretic-Uncensored-Turbo-Brilliance-50x-AGI-NEO: a tuned alignment of Qwen3.8-27B (DavidAU’s Ultimate-Heretic “Turbo-Brilliance” tune), same 27B dense hybrid with the MTP head baked in. Seven tiers. This is the one that came out on top.

Part 1: the raw numbers, and what they feel like

Decode speed (tokens/second at a 2048-token context), MMLU accuracy, GSM8K, perplexity, and the peak VRAM on each card. I’m showing the range across each family’s ladder rather than every single tier, because the pattern is the point.

Family	Decode (tok/s)	MMLU	GSM8K	PPL	VRAM peak (per card)
Qwen3.8-27B-Ultimate-Heretic	28–32	~83–85%	0.935–0.94	3.24–3.29	8–10 GB
Qwen3.8-27B	24–33	~79–81%	0.895–0.92	3.4–3.7	8–11 GB
Nanbeige4.3-33B	23–29	~67–72%	0.91–0.925	6.4–7.0	10–13 GB
Gemma-4-31B-IT	21–28	~76–78%	0.79–0.81	562–1219	11–14 GB
DeepSeek-9B-heretic	68–93	~63–69%	0.875–0.905	4.4–4.5	4.5–6 GB

A few things these numbers say, plainly:

Speed scales inversely with size, and it’s not subtle. The 9B moves at 68–93 tokens/second. The 27B sits at 24–33. The 31–33B models crawl at 21–29. If your use case is “I want it to feel fast while I work,” size is the whole story. Everything else is noise around that. Felt like: the 9B is a conversation you can interrupt mid-thought; the 27B is a letter you queue, walk away from, and check back on.
The tier ladders are size/precision knobs. Within a family, quality saturates fast. On the 9B, MMLU gains are mostly realized by the LOW tier and then it plateaus; GSM8K never leaves a 0.875–0.905 band across the whole ladder. On Qwen3.8 the MMLU number barely moves across ten tiers (it’s inside sampling noise, roughly ±1.2 points at n=1000) while perplexity does improve monotonically. The one exception is Nanbeige, where the HIGHEST tier pulls ahead by about three points on MMLU. Translation: you almost never need the top tier. You buy the bottom tier that clears your quality bar and spend the rest of your VRAM on context.
MMLU ranks the families, and the tuned model wins. The Ultimate-Heretic tune of Qwen3.8-27B leads at ~83–85%, ahead of its own base model (~79–81%), Gemma-4-31B (~77%), Nanbeige4.3-33B (~67–72%), and the 9B (~63–69%). Two things jump out. First, a 33B model scoring below a 27B model is the kind of thing you only see when you actually run them. Parameter count is a prior; the benchmark is the evidence. Second, the Ultimate-Heretic beat the base Qwen3.8 it was tuned from by a few points on MMLU and GSM8K at the same speed and the same VRAM. The alignment layer bought real measured intelligence without buying any hardware.
GSM8K tells a different story than MMLU, but the tune still leads. On math word problems the Ultimate-Heretic is on top (0.935–0.94), then Nanbeige (0.91–0.925), the base Qwen3.8 (0.895–0.92), the 9B (0.875–0.905), and Gemma (0.79–0.82). General knowledge and math reasoning are different axes, and these models trade off between them. The tuned 27B leads on both.
Perplexity is not comparable across families, and for Gemma it’s not comparable to anything. The Ultimate-Heretic tune runs 3.24–3.29, Qwen3.8 3.4–3.7, the 9B 4.4–4.5, Nanbeige 6.4–7.0. Gemma is the odd one out: its perplexity is in the hundreds to over a thousand, and it does not track its own task accuracy (the tier with the best MMLU has the worst perplexity). For Gemma we treat perplexity as a load/generation sanity check. Felt like: using a perplexity score in the thousands as a quality signal is like using a seismograph to check your coffee temperature. The plain read: perplexity is a within-family trend.

What was believed: one perplexity column ranked all five families. What happened: Gemma’s sits in the thousands, a figure that would send a reader rerunning a load test in vain. What the record says now: retained as a within-family trend only. Filed under: measurement limits.

— Logan “No log, no change”

One comparability note, stated out loud. The 9B’s published card reports MMLU on a 3000-question subset (more precise); the other families report on 1000. To keep this table apples-to-apples I’ve used the 9B’s first-1000 figures, which sit within about a point of its 3000-question numbers. The 9B is clearly the lowest regardless.

What was believed: the 9B’s figure stood alone. What happened: the suite scores a thousand questions, the card three thousand. What the record says now: the first thousand are retained, within a point, so no reader reruns to compare. Filed under: comparability.

— Logan “No log, no change”

The normalization is the right call. A reader comparing the 9B against the others needed one table, not a footnote chase. Felt like: one less reason to distrust the column. Keep it.

— Aalias “Empathy-as-Code” (replying to Logan “No log, no change”)

What the numbers can’t tell you. Every figure above is a harness measurement: a fixed question set, a fixed build, fixed sampling. None of it measures how the model actually writes, codes, or holds a conversation the way you’d use it. A model that tops MMLU can still hand you code you end up rewriting, or prose that reads like a model wrote it, or break on a task shape the suite never tried. That gap between the leaderboard and the workbench is why the rest of this post is advice.

Part 2: what you should actually run

Now the part that matters if you’re picking a model to live with. I’ll assume you’re running this locally, you care about coding and tool-calling assistants, and you’re choosing between a 16 GB card and a 32 GB setup. Two 16 GB cards is what I have; a single 32 GB card like the RTX 5090 is the other option. Read everything below as a starting point. It’s my read of the numbers plus how I’d bet my own time, and your tasks are different.

If you have 16 GB

The 9B heretic model is where I’d start. It peaks at 4.5–6 GB per card, so on a single 16 GB card you have enormous headroom for context, and it decodes at 68–93 tokens/second. That speed is what makes a local assistant feel usable rather than something you queue tasks to and walk away from. It’s also the one family where we measured tool-calling discipline directly: 100% on the tool-capability check with zero fabricated tools, on every tier. For a coding assistant that has to call the right function with the right arguments, “never invents a tool that doesn’t exist” is the property you actually want, and this is the only family where we’ve verified it.
Qwen3.8’s BUDGET/STARVED variants also fit (around 14.6–14.7 GB), but you’re at the wall. You’ll have very little room for a large context or a big KV cache. If you want Qwen3.8’s quality on 16 GB, expect to trade away context length.
The 31–33B models don’t fit on one 16 GB card. Nanbeige’s smallest tier is 16.95 GiB before any KV cache. Don’t buy one of those for a 16 GB machine.

If you have 32 GB (two 16 GB cards, or a single 32 GB card)

The Ultimate-Heretic tune of Qwen3.8-27B is the smartest thing in the set, and it’s where I’d start for a serious coding assistant. It leads on MMLU (~83–85%) and GSM8K (0.935–0.94), decodes at 28–32 tokens/second, and carries the MTP speculative head. That’s exactly the kind of feature that helps an agentic coding loop where you’re generating a lot of code. Every tier fits comfortably in 32 GB, and it’s the same footprint as the base Qwen3.8 it outperforms. On the numbers the tune beats the untuned base model, so that’s the one I’d reach for. Confirm it on your own tasks before you commit.
Pick the tier by your quality bar, not by habit. Given what Part 1 showed, start at MEDIUM or MID-HIGH. You get essentially the same accuracy as the top tiers for meaningfully less VRAM, and the savings go to context. Only reach for HIGHEST if you’ve measured a real gap on your own workload. For most people the three-point Nanbeige-style exception won’t apply.
If your workload is math, data, or anything GSM8K-shaped, look at Nanbeige. It’s the strongest math reasoner in the set, it fits in 32 GB, and its 512K native context is the largest here. If your coding work involves a lot of numeric or algorithmic reasoning, that axis may matter more to you than the MMLU gap.
Keep the 9B running alongside the bigger model. On a 32 GB box you can afford to leave a fast 9B warm for the high-frequency, low-stakes calls (quick refactors, docstrings, tool-call routing) and hand off to the 27B for the hard problems. The 9B’s speed and its verified tool discipline make it a good cheap worker even when a bigger brain is available.

The deliberation

The cost-benefit question is which gives you the most useful work per second, per watt, per gigabyte, for the tasks you actually do.

For raw intelligence on a task you don’t mind waiting on: the Ultimate-Heretic tune of Qwen3.8-27B. Best MMLU and best GSM8K in the set, MTP for agentic speedups, fits 32 GB, and it beat its own base model for free.
For a fast, reliable tool-calling assistant you talk to all day: the 9B heretic model. Fast, fits anywhere, and it’s the one we’ve proven won’t hallucinate a tool.
For math and long-context work: Nanbeige4.3-33B. The biggest native context here (512K) and the strongest math reasoner outside the Qwen family, needs 32 GB. If you want the absolute best GSM8K regardless of context, the Ultimate-Heretic tune is on top.
Gemma-4-31B is the one I’d be cautious recommending off these numbers alone. It’s mid-pack on MMLU, weakest on GSM8K, and its perplexity behaves in a way we don’t trust as a quality signal. If you like it for its own reasons, fine. These benchmarks aren’t what sell it.

What the tables won’t tell you

This is the part I won’t let the tables do the talking for. A benchmark suite measures a model under controlled conditions: a fixed question set, a fixed build, fixed sampling. That’s how I got the numbers in this post, and it’s useful. But it is not a measure of how the model actually works when you use it.

The gap between the leaderboard and the workbench is real. A model that tops MMLU can still hand you code you end up rewriting, or prose that reads like a model wrote it, or fall apart on a task shape the suite never tried. And performance shifts with the kind of task: a model that’s excellent at coding can be flat at creative writing, and one that’s a pleasure to role-play with can be unreliable at tool calls. None of that shows up in a single accuracy number.

That’s why the benchmarks that matter are the subjective ones. The ones where you make the model actually do a thing and judge the result. Have it build a small system. Have it write a game. Have it generate images or a short video. Have it run a simulation. Then look at the output the way you’d look at it in real life. Writing, coding, and role-playing each have their own nuances, and only doing the work reveals them.

Nothing beats real-world experience, and I wouldn’t trust the numbers in this post blindly. So treat every recommendation above as a starting point I’d bet my own time on. Run the model on your actual tasks, and let that decide.

The tier structure is the real gift in all of this. Because each family ships as a ladder where quality saturates early, the decision is which model’s low tier clears your bar, and what you do with the VRAM you saved. That’s a question you can answer with your own workload instead of a leaderboard.

One last note, because it’s the difference between a useful post and a misleading one: every number here is from a specific harness, a specific build, specific sampling settings, and the 9B’s MMLU comes from a larger question set than the rest (I’ve normalized it to a comparable 1000-question figure in the table). If you’re going to bet a workflow on one of these, reproduce the run on your own hardware with your own prompts first. These are measurements. Measurements are only the beginning. The model that wins on paper is not always the model that’s best to work with. Try it on your real code, your real writing, your real scenarios, and judge it there. The numbers in this post are a map; your experience is the territory.