👁 Image

WARNING: This is a base language model that has not undergone instruction tuning or alignment with human preferences. As a result, it may generate outputs that are inappropriate, misleading, biased, or unsafe. These risks can be mitigated through additional post-training stages, which is strongly recommended before deployment in any production system, especially for high-stakes applications.

ALIA-40b Model Card

ALIA-40b is a highly multilingual model pre-trained from scratch that will come with its respective base and instruction-tuned variants. This model card corresponds to the 40b base version.

To visit the model cards of other model versions, please refer to the Model Index.

This model is released under a permissive Apache 2.0 license. Along with the open weights, all training scripts and configuration files are made publicly available in this GitHub repository.

---

Model Details

Description

Transformer-based decoder-only language model that has been pre-trained from scratch on 9.37 trillion tokens of highly curated data. The pre-training corpus contains text in 35 European languages and code.

Hyperparameters

The full list of hyperparameters can be found here.

Architecture

---

Intended Use

Direct Use

The models are intended for both research and commercial use in any of the languages included in the training data. The base models are intended either for language generation or to be further fine-tuned for specific use-cases. The instruction-tuned variants can be used as general-purpose assistants, as long as the user is fully aware of the model’s limitations.

Out-of-scope Use

The model is not intended for malicious activities, such as harming others or violating human rights. Any downstream application must comply with current laws and regulations. Irresponsible usage in production environments without proper risk assessment and mitigation is also discouraged.

---

Hardware and Software

Training Framework

Pre-training was conducted using NVIDIA’s NeMo Framework, which leverages PyTorch Lightning for efficient model training in highly distributed settings.

The instruction-tuned versions were produced with FastChat.

Compute Infrastructure

All models were trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.

The accelerated partition is composed of 1,120 nodes with the following specifications:

• 4x Nvidia Hopper GPUs with 64GB HBM2 memory
• 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
• 4x NDR200 (BW per node 800Gb/s)
• 512 GB of Main memory (DDR5)
• 460GB on NVMe storage

---

How to use

This section offers examples of how to perform inference using various methods.

Inference

You'll find different techniques for running inference, including Huggingface's Text Generation Pipeline, multi-GPU configurations, and vLLM for scalable and efficient generation.

Inference with Huggingface's Text Generation Pipeline

The Huggingface Text Generation Pipeline provides a straightforward way to run inference using the ALIA-40b model.

pip install transformers torch accelerate sentencepiece protobuf

Inference with single / multi GPU

This section provides a simple example of how to run inference using Huggingface's AutoModel class.

pip install transformers torch accelerate sentencepiece protobuf

Inference with vLLM

vLLM is an efficient library for inference that enables faster and more scalable text generation.

pip install vllm

---

Data

Pretraining Data

The pre-training corpus comprises data from 35 European languages and 92 programming languages, with detailed data sources provided below. The initial 1.6 training epochs used 2.4 trillion tokens, obtained by manually adjusting data proportion to balance the representation and give more importance to Spain’s co-official languages (Spanish, Catalan, Galician, and Basque). This way, we downsampled code and English data to half, Spanish co-official languages were oversampled by 2x, and the remaining languages were kept in their original proportions. During the following training, the Colossal OSCAR dataset was replaced with the FineWeb-Edu dataset. This adjustment resulted in a total of 2.68 trillion tokens used across 2 epochs, distributed as outlined below:

👁 lang distrib

The pretraining corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 53.05% of the total tokens. Following this, Starcoder provides 13.67%, and FineWeb-Edu (350B tokens subset) adds 10.24%. The next largest sources are HPLT at 4.21% and French-PD at 3.59%. Other notable contributions include MaCoCu, Legal-ES, and EurLex, each contributing around 1.72% to 1.41%. These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model. The remaining 10% comes from smaller sources in various languages.

Feel free to click the expand button below to see the full list of sources.

In the final pre-training phase, we used a high-quality subset of 160 billion tokens. Additionally, to expand the model's context window to 32K, 6.3 billion tokens were processed using the Llama 3.1 RoPE interpolation strategy.

We provide an extense Datasheet section following the best practices defined by (Gebru et al., 2021).

---

Evaluation

Gold-standard benchmarks

Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from SpanishBench, CatalanBench, BasqueBench and GalicianBench. We also use English tasks already available on the LM Evaluation Harness. These benchmarks include both new and existing tasks and datasets. In the tables below, we include the results in a selection of evaluation datasets that represent model's performance across a variety of tasks within these benchmarks.

We only use tasks that are either human generated, human translated, or with a strong human-in-the-loop (i.e., machine translation followed by professional revision or machine generation followed by human revision and annotation). This is the reason behind the variety in number of tasks reported across languages. As more tasks that fulfill these requirements are published, we will update the presented results. We also intend to expand the evaluation to other languages, as long as the datasets meet our quality standards.

During the implementation of the evaluation we observed a series of issues worth considering when replicating and interpreting the results presented. These issues include ≈1.5% variances in performance in some tasks depending on the version of the transformers library used, and depending on the use (or lack of use) of tensor parallelism when loading a model. When implementing existing tasks, we carry out a comprehensive quality evaluation of the dataset, the Harness task itself, and what kind of input models see during evaluation. Our implementation (see links above) addresses multiple existing problems such as errors in datasets and prompts, and lack of pre-processing. All this means that results will vary if using other Harness implementations, and may slightly vary depending on the replication setup.

It should be noted that these results are subject to all the drawbacks of every current gold-standard evaluation, and that the figures do not fully represent the model's capabilities and potential. We thus advise caution when reading and interpreting the results.

A full list of results compared to other baselines, a discussion of the model's performance across tasks and its implications, and details regarding problem-solving with task implementation will soon be available in the technical report.

All results reported below are on a 5-shot setting.

Spanish

Category	Task	Metric	Result
Commonsense Reasoning	xstorycloze_es	acc	79.5
NLI	wnli_es	acc	64.8
	xnli_es	acc	50.4
Paraphrasing	paws_es	acc	63.8
QA	xquad_es	acc	73.4
Translation	flores_es	bleu	25.9

Catalan

Category	Task	Metric	Result
Commonsense Reasoning	copa_ca	acc	86.0
	xstorycloze_ca	acc	80.0
NLI	wnli_ca	acc	70.0
	xnli_ca	acc	50.7
Paraphrasing	parafraseja	acc	67.8
	paws_ca	acc	67.5
QA	arc_ca_easy	acc	81.0
	arc_ca_challenge	acc	53.0
	openbookqa_ca	acc	41.6
	piqa_ca	acc	75.8
	siqa_ca	acc	53.9
Translation	flores_ca	bleu	33.7

Basque

Category	Task	Metric	Result
Commonsense Reasoning	xcopa_eu	acc	78.8
	xstorycloze_eu	acc	72.2
NLI	wnli_eu	acc	66.2
	xnli_eu	acc	45.9
QA	eus_exams	acc	61.5
	eus_proficiency	acc	60.4
	eus_trivia	acc	67.2
Reading Comprehension	eus_reading	acc	61.1
Translation	flores_eu	bleu	21.3

Galician

Category	Task	Metric	Result
Paraphrasing	parafrases_gl	acc	60.2
	paws_gl	acc	63.0
QA	openbookqa_gl	acc	36.6
Translation	flores_gl	bleu	31.2

English

Category	Task	Metric	Result
Commonsense Reasoning	copa	acc	94.0
	xstorycloze_en	acc	83.2
NLI	wnli	acc	67.6
	xnli_en	acc	57.0
Paraphrasing	paws *	acc	68.5
QA	arc_easy	acc	86.5
	arc_challenge	acc	59.4
	openbookqa	acc	38.4
	piqa	acc	81.7
	social_iqa	acc	53.8
	xquad_en	acc	80.7

* Current LM Evaluation Harness implementation is lacking correct pre-processing. These results are obtained with adequate pre-processing.

Long Context Evaluation

To assess the long-context capabilities of our model, we conduct a "needle in a haystack" test with the following configuration:

• Needle Phrase: "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day."
• Retrieval Question: "The best thing to do in San Francisco is"
• Evaluator: prometheus-8x7b-v2.0, used as the evaluation judge.

👁 Image

---

Ethical Considerations and Limitations

We examine the presence of undesired societal and cognitive biases present in this model using different benchmarks. For societal biases, we test performance using our Spanish version of the BBQ dataset (Parrish et al., 2022). We report that while accuracy in disambiguated settings is relatively high for a base model, the model performs very poorly in ambiguous settings. Further examination of the differences in accuracy scores as described in Jin et al. (2024) reveals a low-to-moderate alignment between the model's responses and societal biases. These largely vanish in disambiguated setting. Our analyses on societal biases show that while these biases are capable of interfering with model performance as expressed in the results on the BBQ dataset, their interference with task performance is somewhat limited given the results on the disambiguated dataset. We highlight that our analyses of these biases are by no means exhaustive and are limited by the relative scarcity of adequate resources in all languages present in the training data. We aim to gradually extend and expand our analyses in future work.

Our cognitive bias analysis focuses on positional effects in 0-shot settings, and majority class bias in few-shot settings. For positional effects, we leverage the ARC Multiple Choice Question dataset (Clark et al., 2018). We observe weak primacy effects, whereby the model shows a preference for answers towards the beginning of the list of provided answers. We measure the effects of majority class effects in few-shot settings using SST-2 (Socher et al., 2013). We detect significant effects, albeit extremely weak ones, implying that outputs are generally robust against variations in prompt format, and order.

We highlight that these results can be expected from a pretrained model that has not yet been instruction-tuned or aligned. These tests are performed in order to show the biases the model may contain. We urge developers to take them into account and perform safety testing and tuning tailored to their specific applications of the model.

---

Additional information

Author

The Language Technologies Lab from Barcelona Supercomputing Center.

Contact

For further information, please send an email to langtech@bsc.es.

Copyright

Copyright(c) 2025 by Language Technologies Lab, Barcelona Supercomputing Center.

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.

This work has been promoted and supported by the Government of Catalonia through the Aina Project.

Acknowledgements

This project has benefited from the contributions of numerous teams and institutions, mainly through data contributions, knowledge transfer or technical support.

We are especially grateful to our ILENIA project partners: CENID, HiTZ and CiTIUS for their participation. We also extend our genuine gratitude to the Spanish Senate and Congress, Fundación Dialnet, and the ‘Instituto Universitario de Sistemas Inteligentes y Aplicaciones Numéricas en Ingeniería (SIANI)’ of the University of Las Palmas de Gran Canaria. Many other institutions have been involved in the project. Our thanks to Òmnium Cultural, Parlament de Catalunya, Institut d'Estudis Aranesos, Racó Català, Vilaweb, ACN, Nació Digital, El món and Aquí Berguedà. We thank the Welsh government, DFKI, Occiglot project, especially Malte Ostendorff, and The Common Crawl Foundation, especially Pedro Ortiz, for their collaboration.

We would also like to give special thanks to the NVIDIA team, with whom we have met regularly, specially to: Ignacio Sarasua, Adam Henryk Grzywaczewski, Oleg Sudakov, Sergio Perez, Miguel Martinez, Felipes Soares and Meriem Bendris. Their constant support has been especially appreciated throughout the entire process.

Their valuable efforts have been instrumental in the development of this work.

Disclaimer

Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.

The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.

Citation

@misc{gonzalezagirre2025salamandratechnicalreport,
 title={Salamandra Technical Report}, 
 author={Aitor Gonzalez-Agirre and Marc Pàmies and Joan Llop and Irene Baucells and Severino Da Dalt and Daniel Tamayo and José Javier Saiz and Ferran Espuña and Jaume Prats and Javier Aula-Blasco and Mario Mina and Adrián Rubio and Alexander Shvets and Anna Sallés and Iñaki Lacunza and Iñigo Pikabea and Jorge Palomar and Júlia Falcão and Lucía Tormo and Luis Vasquez-Reina and Montserrat Marimon and Valle Ruíz-Fernández and Marta Villegas},
 year={2025},
 eprint={2502.08489},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2502.08489}, 
}

License

Apache License, Version 2.0

Model Index