VOOZH about

URL: https://huggingface.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF

⇱ bartowski/OLMo-2-1124-7B-Instruct-GGUF · Hugging Face

Llamacpp imatrix Quantizations of OLMo-2-1124-7B-Instruct

Using llama.cpp release b4191 for quantization.

Original model: https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct

All quants made using imatrix option with dataset from here

These were made by fixing the tokenizer.json pre_processor, using the one from the base model.

Prompt format 

<|endoftext|><|system|>
{system_prompt}
<|user|>
{prompt}
<|assistant|>

Download a file (not the whole branch) from below:

Filename Quant type File Size Split Description
OLMo-2-1124-7B-Instruct-f16.gguf f16 14.60GB false Full F16 weights.
OLMo-2-1124-7B-Instruct-Q8_0.gguf Q8_0 7.76GB false Extremely high quality, generally unneeded but max available quant.
OLMo-2-1124-7B-Instruct-Q6_K_L.gguf Q6_K_L 6.19GB false Uses Q8_0 for embed and output weights. Very high quality, near perfect, recommended.
OLMo-2-1124-7B-Instruct-Q6_K.gguf Q6_K 5.99GB false Very high quality, near perfect, recommended.
OLMo-2-1124-7B-Instruct-Q5_K_L.gguf Q5_K_L 5.46GB false Uses Q8_0 for embed and output weights. High quality, recommended.
OLMo-2-1124-7B-Instruct-Q5_K_M.gguf Q5_K_M 5.21GB false High quality, recommended.
OLMo-2-1124-7B-Instruct-Q5_K_S.gguf Q5_K_S 5.08GB false High quality, recommended.
OLMo-2-1124-7B-Instruct-Q4_K_L.gguf Q4_K_L 4.78GB false Uses Q8_0 for embed and output weights. Good quality, recommended.
OLMo-2-1124-7B-Instruct-Q4_K_M.gguf Q4_K_M 4.47GB false Good quality, default size for most use cases, recommended.
OLMo-2-1124-7B-Instruct-Q3_K_XL.gguf Q3_K_XL 4.31GB false Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
OLMo-2-1124-7B-Instruct-Q4_K_S.gguf Q4_K_S 4.25GB false Slightly lower quality with more space savings, recommended.
OLMo-2-1124-7B-Instruct-Q4_0.gguf Q4_0 4.23GB false Legacy format, generally not worth using over similarly sized formats
OLMo-2-1124-7B-Instruct-Q4_0_8_8.gguf Q4_0_8_8 4.22GB false Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac.
OLMo-2-1124-7B-Instruct-Q4_0_4_8.gguf Q4_0_4_8 4.22GB false Optimized for ARM inference. Requires 'i8mm' support (see details below). Don't use on Mac.
OLMo-2-1124-7B-Instruct-Q4_0_4_4.gguf Q4_0_4_4 4.22GB false Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. Don't use on Mac.
OLMo-2-1124-7B-Instruct-IQ4_XS.gguf IQ4_XS 4.00GB false Decent quality, smaller than Q4_K_S with similar performance, recommended.
OLMo-2-1124-7B-Instruct-Q3_K_L.gguf Q3_K_L 3.95GB false Lower quality but usable, good for low RAM availability.
OLMo-2-1124-7B-Instruct-Q3_K_M.gguf Q3_K_M 3.65GB false Low quality.
OLMo-2-1124-7B-Instruct-IQ3_M.gguf IQ3_M 3.47GB false Medium-low quality, new method with decent performance comparable to Q3_K_M.
OLMo-2-1124-7B-Instruct-Q3_K_S.gguf Q3_K_S 3.30GB false Low quality, not recommended.
OLMo-2-1124-7B-Instruct-Q2_K_L.gguf Q2_K_L 3.26GB false Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
OLMo-2-1124-7B-Instruct-IQ3_XS.gguf IQ3_XS 3.15GB false Lower quality, new method with decent performance, slightly better than Q3_K_S.
OLMo-2-1124-7B-Instruct-Q2_K.gguf Q2_K 2.86GB false Very low quality but surprisingly usable.
OLMo-2-1124-7B-Instruct-IQ2_M.gguf IQ2_M 2.68GB false Relatively low quality, uses SOTA techniques to be surprisingly usable.

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
1,501
GGUF
Model size
7B params
Architecture
olmo2
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for bartowski/OLMo-2-1124-7B-Instruct-GGUF

Dataset used to train bartowski/OLMo-2-1124-7B-Instruct-GGUF

Collection including bartowski/OLMo-2-1124-7B-Instruct-GGUF