NVFP4 tiers, triaged: which file for your card, what it costs, what breaks first
Objective: you leave knowing which file to download for your card.
Our lab publishes tiered NVFP4 versions of Qwen3.8-27B, from VERY-LOW to HIGHEST, plus two tiny tiers for small cards. Developers keep filing the same tickets. Here is the fix that held up for each, the measurement behind it, and what it costs.
Symptom: fastest Qwen3.8 at home on a Blackwell card.
True fix: the LOW tier.
What does it cost? A few points of fidelity for a large multiple of speed.
→ A community member measured one hundred forty eight tokens per second on average with peaks near two hundred. The llm-bench aggregator recorded one run of our flagship file at one hundred thirty four tokens per second with quality near eighty seven. Perplexity testers measured the same files below plain Q4 at equal size. One wrote that every tested NVFP4 file was worse than his daily Q4 driver, and his follow-up landed at seven point four nine, still behind at equal speed. That is the whole deal.
Symptom: sixteen gigabytes of VRAM.
True fix: BUDGET or STARVED, the two tiers with the draft block stripped.
What does it cost? No speculative draft, smaller heads, and no MTP to save you.
→ Our own table peaks at fifteen point nine gibibytes with twenty seven tokens per second. One tester runs BUDGET at twenty tokens per second with one hundred twenty thousand context on a 5070 Ti. That is a full codebase in one window on a midrange card. What breaks first is the headroom. Keep contexts realistic and the KV cache quantized.
Symptom: maximum quality on a twenty four gigabyte laptop card.
True fix: HIGH, which keeps the language model head in full precision.
What does it cost? Seventeen and a half gigabytes of disk, and Blackwell silicon underneath.
→ An independent operator daily-drives it with one hundred eighty four thousand context and quantized KV cache, measuring near thirty eight tokens per second. That is a reply arriving while the question is still on screen, on a machine that fits in a bag. Off Blackwell there is no contest. We redirect those users to plain quants ourselves.
Symptom: the numbers look wrong on my machine.
True fix: check the engine before the file.
What does it cost to skip this check? You will benchmark your own plumbing and blame the weights.
→ One tester saw the draft head slow things down in LM Studio, then watched the penalty vanish on plain llama.cpp. Another measured twenty two tokens per second through Dockerized vLLM against one hundred one on native llama.cpp, and flagged his own setup as the suspect before retesting. The files did not change between those runs. The stack did.
Symptom: the model loops and overthinks.
True fix, with a warning label: our SSMFIX tier, or a lower reasoning effort.
What does it cost? It is an experiment, and experiments sometimes lose.
→ The base model overthinks on its own. A fixed variant exists because a user asked for one in our own thread. The card on that repo opens with a warning that bad results are not unlikely. We ship it labeled, not promised.
The abstraction strip. Nine names sit on one shared backbone of four hundred forty eight tensors. The tiers differ in ten tensors: head, embeddings, draft block. Download by head, not by name. HIGH keeps it in full precision. STARVED quantizes everything. The names add nothing once you know that.
Lineage. The backbone came from Unsloth’s NVFP4 quantization of Alibaba’s Qwen3.8-27B. Our part was the conversion and tiering into GGUF. Adoption so far: seven hundred fifty nine thousand downloads in thirty days. That counts downloads. It does not count happy users. We know the difference.
What was believed: seven hundred fifty nine thousand downloads meant seven hundred fifty nine thousand decisions. What happened: the same five tickets kept arriving. What the record says now: the archive holds the questions, not just the files. Filed under: provenance.
— Logan “No log, no change”
Verdict per card. Blackwell with room: LOW. Sixteen gigabytes: BUDGET. Twenty four gigabyte laptop, quality first: HIGH. Anything else: plain Q4. The true fix in each case costs exactly what is written above, no more. The fine print already wrote itself, so I just read the log.
Nobody asked whether the fastest file is the one you will keep.
esatapedico (Raff Martin)
I build software for the people whose hardware is almost good enough. Model conversions, speculative decoding, squeezing big models onto consumer GPUs. Everything I ship is free and Apache-2.0.