VOOZH about

URL: https://huggingface.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF

⇱ bartowski/OLMo-2-1124-13B-Instruct-GGUF · Hugging Face

Llamacpp imatrix Quantizations of OLMo-2-1124-13B-Instruct

Using llama.cpp release b4191 for quantization.

Original model: https://huggingface.co/allenai/OLMo-2-1124-13B-Instruct

All quants made using imatrix option with dataset from here

These were made by fixing the tokenizer.json pre_processor, using the one from the base model.

Prompt format 

<|endoftext|><|system|>
{system_prompt}
<|user|>
{prompt}
<|assistant|>

Download a file (not the whole branch) from below:

Filename Quant type File Size Split Description
OLMo-2-1124-13B-Instruct-f16.gguf f16 27.44GB false Full F16 weights.
OLMo-2-1124-13B-Instruct-Q8_0.gguf Q8_0 14.58GB false Extremely high quality, generally unneeded but max available quant.
OLMo-2-1124-13B-Instruct-Q6_K_L.gguf Q6_K_L 11.51GB false Uses Q8_0 for embed and output weights. Very high quality, near perfect, recommended.
OLMo-2-1124-13B-Instruct-Q6_K.gguf Q6_K 11.26GB false Very high quality, near perfect, recommended.
OLMo-2-1124-13B-Instruct-Q5_K_L.gguf Q5_K_L 10.08GB false Uses Q8_0 for embed and output weights. High quality, recommended.
OLMo-2-1124-13B-Instruct-Q5_K_M.gguf Q5_K_M 9.76GB false High quality, recommended.
OLMo-2-1124-13B-Instruct-Q5_K_S.gguf Q5_K_S 9.50GB false High quality, recommended.
OLMo-2-1124-13B-Instruct-Q4_K_L.gguf Q4_K_L 8.74GB false Uses Q8_0 for embed and output weights. Good quality, recommended.
OLMo-2-1124-13B-Instruct-Q4_K_M.gguf Q4_K_M 8.35GB false Good quality, default size for most use cases, recommended.
OLMo-2-1124-13B-Instruct-Q4_K_S.gguf Q4_K_S 7.91GB false Slightly lower quality with more space savings, recommended.
OLMo-2-1124-13B-Instruct-Q4_0.gguf Q4_0 7.88GB false Legacy format, generally not worth using over similarly sized formats
OLMo-2-1124-13B-Instruct-Q4_0_8_8.gguf Q4_0_8_8 7.85GB false Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac.
OLMo-2-1124-13B-Instruct-Q4_0_4_8.gguf Q4_0_4_8 7.85GB false Optimized for ARM inference. Requires 'i8mm' support (see details below). Don't use on Mac.
OLMo-2-1124-13B-Instruct-Q4_0_4_4.gguf Q4_0_4_4 7.85GB false Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. Don't use on Mac.
OLMo-2-1124-13B-Instruct-Q3_K_XL.gguf Q3_K_XL 7.82GB false Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
OLMo-2-1124-13B-Instruct-IQ4_XS.gguf IQ4_XS 7.44GB false Decent quality, smaller than Q4_K_S with similar performance, recommended.
OLMo-2-1124-13B-Instruct-Q3_K_L.gguf Q3_K_L 7.37GB false Lower quality but usable, good for low RAM availability.
OLMo-2-1124-13B-Instruct-Q3_K_M.gguf Q3_K_M 6.78GB false Low quality.
OLMo-2-1124-13B-Instruct-IQ3_M.gguf IQ3_M 6.43GB false Medium-low quality, new method with decent performance comparable to Q3_K_M.
OLMo-2-1124-13B-Instruct-Q3_K_S.gguf Q3_K_S 6.10GB false Low quality, not recommended.
OLMo-2-1124-13B-Instruct-IQ3_XS.gguf IQ3_XS 5.80GB false Lower quality, new method with decent performance, slightly better than Q3_K_S.
OLMo-2-1124-13B-Instruct-Q2_K_L.gguf Q2_K_L 5.76GB false Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
OLMo-2-1124-13B-Instruct-Q2_K.gguf Q2_K 5.26GB false Very low quality but surprisingly usable.
OLMo-2-1124-13B-Instruct-IQ2_M.gguf IQ2_M 4.91GB false Relatively low quality, uses SOTA techniques to be surprisingly usable.
OLMo-2-1124-13B-Instruct-IQ2_S.gguf IQ2_S 4.59GB false Low quality, uses SOTA techniques to be usable.

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
1,199
GGUF
Model size
14B params
Architecture
olmo2
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for bartowski/OLMo-2-1124-13B-Instruct-GGUF

Dataset used to train bartowski/OLMo-2-1124-13B-Instruct-GGUF

Collection including bartowski/OLMo-2-1124-13B-Instruct-GGUF