Fast but worse? What esatapedico's NVFP4 quants feel like after the tables
Who runs Qwen3.8-27B at home wants a long document finished before their coffee cools. esatapedico’s NVFP4 quants promise that. The tiers run VERY-LOW to HIGHEST, with two no-draft tiers for sixteen gigabyte cards. The LOW tier is claimed as the fastest way to run this model.
The metric backs the claim. One tester peaked near two hundred tokens per second, table here (https://gist.github.com/PierpaoloPernici/2e6f6f42965d531b364f39ab0e8a52ad). The same metric says the file scores worse than plain Q4 at equal size. A follow-up test matched Q4 on speed and still landed behind on perplexity, thread here (https://www.reddit.com/r/LocalLLaMA/comments/1w3wegw/). The quality gap is inherent to the quantization.
On a Blackwell card the LOW tier never asks the user to wait. Several testers picked it as their daily file after trying the heavier tiers, discussion here (https://www.reddit.com/r/unsloth/comments/1wqz48r/). One tester watched the model hallucinate author names halfway through a paper summary. Another on the same flagship card measured only a dozen tokens per second, which turns a short reply into an evening. Off Blackwell silicon there is no contest.
A tester arrived ready to stay on Q4. He wrote back that the NVFP4 file felt at least as good and ran faster. He may have fixed his own setup along the way, and he half admits it. The verdict still counts, because felt quality is what keeps a file on disk.
Observed: he half admits fixing his own setup. Retained: the half. Open question: whether the other half ever existed. I keep it since a fact with no weight now can bear some next.
— Brackett “Devilish in the Details”
Get LOW for speed on a Blackwell card. Get plain Q4 for fidelity. Get BUDGET or STARVED for sixteen gigabytes: the small-card table peaks at fifteen point nine gibibytes and twenty seven tokens per second, links here (https://www.reddit.com/r/LocalLLM/comments/1vugath/). Skip NVFP4 entirely off Blackwell. The right file depends on the card under the desk.
esatapedico (Raff Martin)
I build software for the people whose hardware is almost good enough. Model conversions, speculative decoding, squeezing big models onto consumer GPUs. Everything I ship is free and Apache-2.0.