datamix-9b Dolci-Translated A-75EN
Cross-architecture validation of the Translate, Replay, Mix recipe applied to
openeurollm/datamix-9b-80-20, a Llama-architecture, multilingually-pretrained
$9$B base with a Gemma tokenizer. Same $75/25$ English replay + Dolci-Translated
EU mixture as openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN, but trained with
a different framework and at a different context length because of the base's
architecture and pretraining limits.
Recipe
| Base checkpoint | openeurollm/datamix-9b-80-20 ($9$B Llama, Gemma tokenizer, vocab 262k, $2048$ context) |
| English half (Dolci replay) | allenai/Dolci-Instruct-SFT, 75% of the mixture |
| EU half (Dolci-Translated) | openeurollm/Dolci-Instruct-SFT-translated, 25%, 7 EU languages translated with gemma-3-27b-it |
| EU languages | cs, de, es, fi, fr, it, sv |
| Total samples | 2.87M (same mixture as OLMo A-75EN) |
| Final step | 41000 |
| Chat template | simple_chat (no built-in template on the base; use --chat_template_name simple_chat or apply manually) |
Training configuration
Departs from the OLMo runs in five places (see Section 3.3 of the paper):
- Llama-architecture $9$B base (vs OLMo)
- Maximum context length $2048$ (vs $32$k)
- Micro-batches accumulated to effective batch of $128$ (vs ${\sim}1$M tokens)
- Plain DDP (vs DeepSpeed ZeRO 2)
open-instruct/finetune.pyentry point (vs OLMo-core)
Peak LR $8\times10^{-5}$ matches the OLMo runs.
Evaluation
Per-language Bradley-Terry Elo at matched $1024/1024$ input/output truncation,
500 battles/language, against openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN
re-evaluated under the same truncation (paper Table 4, Figure 6):
| Model | en | cs | de | es | fi | fr | it | sv |
|---|---|---|---|---|---|---|---|---|
This repo (datamix-9b-Dolci-Translated-A-75EN) |
$879 \pm 16$ | $\mathbf{833 \pm 15}$ | $\mathbf{792 \pm 19}$ | $764 \pm 20$ | $\mathbf{815 \pm 35}$ | $790 \pm 18$ | $780 \pm 18$ | $\mathbf{820 \pm 32}$ |
| OLMo-3-7B A-75EN | $\mathbf{970 \pm 14}$ | $733 \pm 18$ | $742 \pm 20$ | $\mathbf{820 \pm 17}$ | $752 \pm 38$ | $\mathbf{804 \pm 16}$ | $\mathbf{821 \pm 15}$ | $786 \pm 32$ |
This checkpoint leads on cs ($+100$), de ($+50$), fi ($+63$), sv ($+34$); OLMo-3-7B A-75EN leads on en ($+91$), es ($+56$), it ($+41$), fr ($+14$). The crossover is consistent with a multilingually-pretrained base partially substituting for the multilingual SFT pool on lower-resource EU languages.
How to load
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN")
model = AutoModelForCausalLM.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN", torch_dtype="bfloat16")
# No built-in chat template; use 'simple_chat' from open-instruct or apply your own.
Citation
Please cite the paper if you use this checkpoint.
- Downloads last month
- 11