I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.

If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.

I wonder if this would help, or if it solves different kinds of errors.

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

https://news.ycombinator.com/item?id=48192383

What setup are you using to do said private evaluation? Software wise I mean

Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts