VOOZH about

URL: https://huggingface.co/bartowski/EVA-Qwen2.5-72B-v0.2-GGUF

⇱ bartowski/EVA-Qwen2.5-72B-v0.2-GGUF · Hugging Face

Llamacpp imatrix Quantizations of EVA-Qwen2.5-72B-v0.2

Using llama.cpp release b4132 for quantization.

Original model: https://huggingface.co/EVA-UNIT-01/EVA-Qwen2.5-72B-v0.2

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format 

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Download a file (not the whole branch) from below:

Filename Quant type File Size Split Description
EVA-Qwen2.5-72B-v0.2-Q8_0.gguf Q8_0 77.26GB true Extremely high quality, generally unneeded but max available quant.
EVA-Qwen2.5-72B-v0.2-Q6_K.gguf Q6_K 64.35GB true Very high quality, near perfect, recommended.
EVA-Qwen2.5-72B-v0.2-Q5_K_M.gguf Q5_K_M 54.45GB true High quality, recommended.
EVA-Qwen2.5-72B-v0.2-Q5_K_S.gguf Q5_K_S 51.38GB true High quality, recommended.
EVA-Qwen2.5-72B-v0.2-Q4_K_M.gguf Q4_K_M 47.42GB false Good quality, default size for most use cases, recommended.
EVA-Qwen2.5-72B-v0.2-Q4_K_S.gguf Q4_K_S 43.89GB false Slightly lower quality with more space savings, recommended.
EVA-Qwen2.5-72B-v0.2-Q4_0.gguf Q4_0 41.38GB false Legacy format, generally not worth using over similarly sized formats
EVA-Qwen2.5-72B-v0.2-Q4_0_8_8.gguf Q4_0_8_8 41.23GB false Optimized for ARM and AVX inference. Requires 'sve' support for ARM (see details below). Don't use on Mac.
EVA-Qwen2.5-72B-v0.2-Q3_K_XL.gguf Q3_K_XL 40.60GB false Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability.
EVA-Qwen2.5-72B-v0.2-IQ4_XS.gguf IQ4_XS 39.71GB false Decent quality, smaller than Q4_K_S with similar performance, recommended.
EVA-Qwen2.5-72B-v0.2-Q3_K_L.gguf Q3_K_L 39.51GB false Lower quality but usable, good for low RAM availability.
EVA-Qwen2.5-72B-v0.2-Q3_K_M.gguf Q3_K_M 37.70GB false Low quality.
EVA-Qwen2.5-72B-v0.2-IQ3_M.gguf IQ3_M 35.50GB false Medium-low quality, new method with decent performance comparable to Q3_K_M.
EVA-Qwen2.5-72B-v0.2-Q3_K_S.gguf Q3_K_S 34.49GB false Low quality, not recommended.
EVA-Qwen2.5-72B-v0.2-IQ3_XXS.gguf IQ3_XXS 31.85GB false Lower quality, new method with decent performance, comparable to Q3 quants.
EVA-Qwen2.5-72B-v0.2-Q2_K_L.gguf Q2_K_L 31.03GB false Uses Q8_0 for embed and output weights. Very low quality but surprisingly usable.
EVA-Qwen2.5-72B-v0.2-Q2_K.gguf Q2_K 29.81GB false Very low quality but surprisingly usable.
EVA-Qwen2.5-72B-v0.2-IQ2_M.gguf IQ2_M 29.34GB false Relatively low quality, uses SOTA techniques to be surprisingly usable.
EVA-Qwen2.5-72B-v0.2-IQ2_XS.gguf IQ2_XS 27.06GB false Low quality, uses SOTA techniques to be usable.
EVA-Qwen2.5-72B-v0.2-IQ2_XXS.gguf IQ2_XXS 25.49GB false Very low quality, uses SOTA techniques to be usable.
EVA-Qwen2.5-72B-v0.2-IQ1_M.gguf IQ1_M 23.74GB false Extremely low quality, not recommended.

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
1,412
GGUF
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for bartowski/EVA-Qwen2.5-72B-v0.2-GGUF

Base model

Qwen/Qwen2.5-72B
Quantized
(5)
this model

Datasets used to train bartowski/EVA-Qwen2.5-72B-v0.2-GGUF

Collection including bartowski/EVA-Qwen2.5-72B-v0.2-GGUF