VOOZH about

URL: https://huggingface.co/bartowski/Llama-3.1-Tulu-3-8B-DPO-GGUF

⇱ bartowski/Llama-3.1-Tulu-3-8B-DPO-GGUF · Hugging Face

Llamacpp imatrix Quantizations of Llama-3.1-Tulu-3-8B-DPO

Using llama.cpp release b4132 for quantization.

Original model: https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-DPO

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format 

<|system|>
{system_prompt}
<|user|>
{prompt}
<|assistant|>

Download a file (not the whole branch) from below:

Filename Quant type File Size Split Description
Llama-3.1-Tulu-3-8B-DPO-f16.gguf f16 16.07GB false Full F16 weights.
Llama-3.1-Tulu-3-8B-DPO-Q8_0.gguf Q8_0 8.54GB false Extremely high quality, generally unneeded but max available quant.
Llama-3.1-Tulu-3-8B-DPO-Q6_K_L.gguf Q6_K_L 6.85GB false Uses Q8_0 for embed and output weights. Very high quality, near perfect, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q6_K.gguf Q6_K 6.60GB false Very high quality, near perfect, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q5_K_L.gguf Q5_K_L 6.06GB false Uses Q8_0 for embed and output weights. High quality, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q5_K_M.gguf Q5_K_M 5.73GB false High quality, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q5_K_S.gguf Q5_K_S 5.60GB false High quality, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q4_K_L.gguf Q4_K_L 5.31GB false Uses Q8_0 for embed and output weights. Good quality, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q4_K_M.gguf Q4_K_M 4.92GB false Good quality, default size for most use cases, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q3_K_XL.gguf Q3_K_XL 4.78GB false Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
Llama-3.1-Tulu-3-8B-DPO-Q4_K_S.gguf Q4_K_S 4.69GB false Slightly lower quality with more space savings, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q4_0.gguf Q4_0 4.68GB false Legacy format, generally not worth using over similarly sized formats
Llama-3.1-Tulu-3-8B-DPO-Q4_0_8_8.gguf Q4_0_8_8 4.66GB false Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac.
Llama-3.1-Tulu-3-8B-DPO-Q4_0_4_8.gguf Q4_0_4_8 4.66GB false Optimized for ARM inference. Requires 'i8mm' support (see details below). Don't use on Mac.
Llama-3.1-Tulu-3-8B-DPO-Q4_0_4_4.gguf Q4_0_4_4 4.66GB false Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. Don't use on Mac.
Llama-3.1-Tulu-3-8B-DPO-IQ4_XS.gguf IQ4_XS 4.45GB false Decent quality, smaller than Q4_K_S with similar performance, recommended.
Llama-3.1-Tulu-3-8B-DPO-Q3_K_L.gguf Q3_K_L 4.32GB false Lower quality but usable, good for low RAM availability.
Llama-3.1-Tulu-3-8B-DPO-Q3_K_M.gguf Q3_K_M 4.02GB false Low quality.
Llama-3.1-Tulu-3-8B-DPO-IQ3_M.gguf IQ3_M 3.78GB false Medium-low quality, new method with decent performance comparable to Q3_K_M.
Llama-3.1-Tulu-3-8B-DPO-Q2_K_L.gguf Q2_K_L 3.69GB false Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
Llama-3.1-Tulu-3-8B-DPO-Q3_K_S.gguf Q3_K_S 3.66GB false Low quality, not recommended.
Llama-3.1-Tulu-3-8B-DPO-IQ3_XS.gguf IQ3_XS 3.52GB false Lower quality, new method with decent performance, slightly better than Q3_K_S.
Llama-3.1-Tulu-3-8B-DPO-Q2_K.gguf Q2_K 3.18GB false Very low quality but surprisingly usable.
Llama-3.1-Tulu-3-8B-DPO-IQ2_M.gguf IQ2_M 2.95GB false Relatively low quality, uses SOTA techniques to be surprisingly usable.

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
792
GGUF
Model size
8B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for bartowski/Llama-3.1-Tulu-3-8B-DPO-GGUF

Quantized
(5)
this model

Dataset used to train bartowski/Llama-3.1-Tulu-3-8B-DPO-GGUF