Is it possible to make less than 1 bit quantization?
@RealBar you can prune the model
I have tried some older GLM models froms Cerebras REAP (https://huggingface.co/collections/cerebras/cerebras-reap). They were pruned by about 20% and then being quantized (e.g. by Unsloth). But that is still not another 10-20x on top of quantization. REAPed models work ok, but at that point you're probably just chasing shadows.
There are plenty of good enough smaller models out there if you don't have a few spare millions of $ in the bank to whip-up terabytes of VRAM.
That'll be hard - 1-bit is currently 86% smaller and retains around 76.2% accuracy
xD, man how much I want to see IQ0_XXXXXS but no, less then 1 bit quantization isn't possible with our today's compute. The tiniest unit in compute is a 1 or a 0 so... xD
unless if I have been lied to
Yes, it is possible to do below-1-bit quantization, but it's not trivial to do. Basically, you have to pack individual values in tensors into groups and then quantize the groups - so you basically quantize something like a [0.5, 0.3, 1.2, 0.9] quadruple into say [-2]. As long as the bit-budget for the aggregate is smaller than the number of aggregates, you get a below-1-bit quant.