HALvest

Dataset Summary

overview:

This is the unfiltered version of HALvest, comprising of fulltext from open papers found on Hyper Articles en Ligne (HAL) with extra fields for potential filtering. Our dump is mostly english/french but gather papers written in 56 languages across 13 domains.

You can download the dataset using Hugging Face datasets:

from datasets import load_dataset

ds = load_dataset("almanach/HALvest", "en")

Details

Building the dataset is a three steps process: data fetching from HAL, data merging and data enriching.

1. We first request HAL's API in order to gather open research papers and parse it -- effectively sorting papers by language. Then, we download the PDFs of the fetched data.
2. Using GROBID, we convert each PDF to an xml-tei format in order to have structured data. We convert each xml-tei file to a txt format before concatenating it with the paper's.
3. Finally, we compute some statistics about each document.

Languages

Please, note that the number of tokens is highly inflated in the raw version of the dataset because of badly encoded PDFs, translating to gibberish documents/texts.

Domains

Please, note that the number of tokens is highly inflated in the raw version of the dataset because of badly encoded PDFs, translating to gibberish documents/texts.

You can browse through every domains and sub-domains here: https://hal.science/browse/domain.

Considerations for Using the Data

The corpus is extracted from the HAL's open archive which distributes scientific publications following open access principles. The corpus is made up of both creative commons licensed and copyrighted documents (distribution authorized on HAL by the publisher). This must be considered prior to using this dataset for any purpose, other than training deep learning models, data mining etc. We do not own any of the text from which these data has been extracted.

Citation

@misc{kulumba2026halvestcontrastiveretrievallikeauthorshipattribution,
 title={HALvest-Contrastive: Retrieval-Like Authorship Attribution with Patch-Level Late Interaction}, 
 author={Francis Kulumba and Wissam Antoun and Guillaume Vimont and Laurent Romary and Florian Cafiero},
 year={2026},
 eprint={2407.20595},
 archivePrefix={arXiv},
 primaryClass={cs.DL},
 url={https://arxiv.org/abs/2407.20595}, 
}

Dataset Copyright

The licence terms for HALvest strictly follows the one from HAL. Please refer to the below license when using this dataset.

• HAL license