Llamacpp imatrix Quantizations of Llama-Sentient-3.2-3B-Instruct
Using llama.cpp release b4132 for quantization.
Original model: https://huggingface.co/prithivMLmods/Llama-Sentient-3.2-3B-Instruct
All quants made using imatrix option with dataset from here
Run them in LM Studio
Prompt format
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 July 2024
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Split | Description |
|---|---|---|---|---|
| Llama-Sentient-3.2-3B-Instruct-f16.gguf | f16 | 6.43GB | false | Full F16 weights. |
| Llama-Sentient-3.2-3B-Instruct-Q8_0.gguf | Q8_0 | 3.42GB | false | Extremely high quality, generally unneeded but max available quant. |
| Llama-Sentient-3.2-3B-Instruct-Q6_K_L.gguf | Q6_K_L | 2.74GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q6_K.gguf | Q6_K | 2.64GB | false | Very high quality, near perfect, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q5_K_L.gguf | Q5_K_L | 2.42GB | false | Uses Q8_0 for embed and output weights. High quality, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q5_K_M.gguf | Q5_K_M | 2.32GB | false | High quality, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q5_K_S.gguf | Q5_K_S | 2.27GB | false | High quality, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q4_K_L.gguf | Q4_K_L | 2.11GB | false | Uses Q8_0 for embed and output weights. Good quality, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q4_K_M.gguf | Q4_K_M | 2.02GB | false | Good quality, default size for most use cases, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q4_K_S.gguf | Q4_K_S | 1.93GB | false | Slightly lower quality with more space savings, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q4_0_8_8.gguf | Q4_0_8_8 | 1.92GB | false | Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac. |
| Llama-Sentient-3.2-3B-Instruct-Q4_0_4_8.gguf | Q4_0_4_8 | 1.92GB | false | Optimized for ARM inference. Requires 'i8mm' support (see details below). Don't use on Mac. |
| Llama-Sentient-3.2-3B-Instruct-Q4_0_4_4.gguf | Q4_0_4_4 | 1.92GB | false | Optimized for ARM inference. Should work well on all ARM chips, not for use with GPUs. Don't use on Mac. |
| Llama-Sentient-3.2-3B-Instruct-Q4_0.gguf | Q4_0 | 1.92GB | false | Legacy format, generally not worth using over similarly sized formats |
| Llama-Sentient-3.2-3B-Instruct-Q3_K_XL.gguf | Q3_K_XL | 1.91GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
| Llama-Sentient-3.2-3B-Instruct-IQ4_XS.gguf | IQ4_XS | 1.83GB | false | Decent quality, smaller than Q4_K_S with similar performance, recommended. |
| Llama-Sentient-3.2-3B-Instruct-Q3_K_L.gguf | Q3_K_L | 1.82GB | false | Lower quality but usable, good for low RAM availability. |
| Llama-Sentient-3.2-3B-Instruct-Q3_K_M.gguf | Q3_K_M | 1.69GB | false | Low quality. |
| Llama-Sentient-3.2-3B-Instruct-IQ3_M.gguf | IQ3_M | 1.60GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| Llama-Sentient-3.2-3B-Instruct-Q3_K_S.gguf | Q3_K_S | 1.54GB | false | Low quality, not recommended. |
| Llama-Sentient-3.2-3B-Instruct-IQ3_XS.gguf | IQ3_XS | 1.48GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |
| Llama-Sentient-3.2-3B-Instruct-Q2_K_L.gguf | Q2_K_L | 1.46GB | false | Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable. |
| Llama-Sentient-3.2-3B-Instruct-Q2_K.gguf | Q2_K | 1.36GB | false | Very low quality but surprisingly usable. |
| Llama-Sentient-3.2-3B-Instruct-IQ2_M.gguf | IQ2_M | 1.23GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |
Embed/output weights
Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.
Downloading using huggingface-cli
Q4_0_X_X information
Which file should I choose?
Credits
Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
Thank you ZeroWw for the inspiration to experiment with embed/output.
Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
- Downloads last month
- 1,252
GGUF
Model size
3B params
Architecture
llama
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Model tree for bartowski/Llama-Sentient-3.2-3B-Instruct-GGUF
Base model
meta-llama/Llama-3.2-3B-Instruct