21 items • Updated • 23
Llama-3.1-8B-Instruct-speculator.eagle3
Model Overview
- Verifier: meta-llama/Llama-3.1-8B-Instruct
- Speculative Decoding Algorithm: EAGLE-3
- Model Architecture: Eagle3Speculator
- Release Date: 07/27/2025
- Version: 1.0
- Model Developers: RedHat
This is a speculator model designed for use with meta-llama/Llama-3.1-8B-Instruct, based on the EAGLE-3 speculative decoding algorithm.
It was trained using the speculators library on a combination of the Aeala/ShareGPT_Vicuna_unfiltered and the HuggingFaceH4/ultrachat_200k datasets.
This model should be used with the meta-llama/Llama-3.1-8B-Instruct chat template, specifically through the /chat/completions endpoint.
Use with vLLM
vllm serve meta-llama/Llama-3.1-8B-Instruct \
-tp 1 \
--speculative-config '{
"model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3",
"num_speculative_tokens": 3,
"method": "eagle3"
}'
Evaluations
Use cases
| Use Case | Dataset | Number of Samples |
|---|---|---|
| Coding | HumanEval | 168 |
| Math Reasoning | gsm8k | 80 |
| Text Summarization | CNN/Daily Mail | 80 |
Acceptance lengths
| Use Case | k=1 | k=2 | k=3 | k=4 | k=5 | k=6 | k=7 |
|---|---|---|---|---|---|---|---|
| Coding | 1.84 | 2.50 | 3.02 | 3.36 | 3.61 | 3.83 | 3.89 |
| Math Reasoning | 1.80 | 2.40 | 2.83 | 3.13 | 3.27 | 3.40 | 3.83 |
| Text Summarization | 1.70 | 2.19 | 2.50 | 2.78 | 2.77 | 2.98 | 2.99 |
Performance benchmarking (1xA100)
- Downloads last month
- 15,469
Safetensors
Model size
1.0B params
Tensor type
I64
·
BF16 ·
BOOL ·
Model tree for RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3
Base model
meta-llama/Llama-3.1-8B Finetuned
meta-llama/Llama-3.1-8B-Instruct