Llamacpp imatrix Quantizations of EVA-LLaMA-3.33-70B-v0.0

Using llama.cpp release b4273 for quantization.

Original model: https://huggingface.co/EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.0

All quants made using imatrix option with dataset from here

Run them in LM Studio

Prompt format

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Download a file (not the whole branch) from below:

Embed/output weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

Downloading using huggingface-cli

Q4_0_X_X information

New: Thanks to efforts made to have online repacking of weights in this PR, you can now just use Q4_0 if your llama.cpp has been compiled for your ARM device.

Similarly, if you want to get slightly better performance, you can use IQ4_NL thanks to this PR which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.

Which file should I choose?

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Base model

meta-llama/Llama-3.1-70B

meta-llama/Llama-3.3-70B-Instruct

EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.0

(19)