VOOZH about

URL: https://huggingface.co/bartowski/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-GGUF

⇱ bartowski/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-GGUF · Hugging Face

Llamacpp imatrix Quantizations of OneLLM-Doey-ChatQA-V1-Llama-3.2-1B

Using llama.cpp release b4132 for quantization.

Original model: https://huggingface.co/DoeyLLM/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format 

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Nov 2024

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Download a file (not the whole branch) from below:

Filename Quant type File Size Split Description
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-f16.gguf f16 2.48GB false Full F16 weights.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q8_0.gguf Q8_0 1.32GB false Extremely high quality, generally unneeded but max available quant.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q6_K_L.gguf Q6_K_L 1.09GB false Uses Q8_0 for embed and output weights. Very high quality, near perfect, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q6_K.gguf Q6_K 1.02GB false Very high quality, near perfect, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q5_K_L.gguf Q5_K_L 0.98GB false Uses Q8_0 for embed and output weights. High quality, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q5_K_M.gguf Q5_K_M 0.91GB false High quality, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q5_K_S.gguf Q5_K_S 0.89GB false High quality, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_K_L.gguf Q4_K_L 0.87GB false Uses Q8_0 for embed and output weights. Good quality, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_K_M.gguf Q4_K_M 0.81GB false Good quality, default size for most use cases, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q3_K_XL.gguf Q3_K_XL 0.80GB false Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_K_S.gguf Q4_K_S 0.78GB false Slightly lower quality with more space savings, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_0_8_8.gguf Q4_0_8_8 0.77GB false Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_0_4_8.gguf Q4_0_4_8 0.77GB false Optimized for ARM inference. Requires 'i8mm' support (see details below). Don't use on Mac.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_0_4_4.gguf Q4_0_4_4 0.77GB false Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. Don't use on Mac.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q4_0.gguf Q4_0 0.77GB false Legacy format, generally not worth using over similarly sized formats
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-IQ4_XS.gguf IQ4_XS 0.74GB false Decent quality, smaller than Q4_K_S with similar performance, recommended.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q3_K_L.gguf Q3_K_L 0.73GB false Lower quality but usable, good for low RAM availability.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-IQ3_M.gguf IQ3_M 0.66GB false Medium-low quality, new method with decent performance comparable to Q3_K_M.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q2_K_L.gguf Q2_K_L 0.64GB false Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-Q2_K.gguf Q2_K 0.58GB false Very low quality but surprisingly usable.

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
427
GGUF
Model size
1B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for bartowski/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-GGUF

Dataset used to train bartowski/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-GGUF

Space using bartowski/OneLLM-Doey-ChatQA-V1-Llama-3.2-1B-GGUF 1