You need to agree to share your contact information to access this dataset

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this dataset content.

Dataset Card for "OSCAR 23.01"

IMPORTANT NOTE: THIS DATASET CARD IS STILL BEING WRITTEN, PLEASE BE PATIENT WHILE WE COMPLETE ALL THE INFORMATION ABOUT THE CORPUS

Dataset Summary

The OSCAR project (Open Super-large Crawled Aggregated coRpus) is an Open Source project aiming to provide web-based multilingual resources and datasets for Machine Learning (ML) and Artificial Intelligence (AI) applications. The project focuses specifically in providing large quantities of unannotated raw data that is commonly used in the pre-training of large deep learning models. The OSCAR project has developed high-performance data pipelines specifically conceived to classify and filter large amounts of web data. The project has also put special attention in improving the data quality of web-based corpora as well as providing data for low-resource languages, so that these new ML/AI technologies are accessible to as many communities as possible.

OSCAR 23.01 is the January 2023 version of the OSCAR Corpus based on the November/December 2022 dump of Common Crawl. While being quite similar to OSCAR 22.01, it contains several new features, including KenLM-based adult content detection, precomputed Locality-Sensitive Hashes for near deduplication, and blocklist-based categories. OSCAR 23.01 has also moved from gzip to Zstandard compression. You might already have zstd installed on your system, but if not, please check the Zstandard website for installation instructions.

Supported Tasks and Leaderboards

OSCAR is mainly intended to pretrain language models and word representations.

Languages

All the data is distributed by language, both the original and the deduplicated versions of the data are available. 151 different languages are available. The table in subsection Data Splits Sample Size provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.

Issues

OSCAR 23.01 may have quality issues on low size subcorpora, as it has been the case before.

Note that since the documents are identified as a whole, it is expected to have lines in other languages in a given language subcorpus. As an example, it is known and expected that the German subcorpus contains documents holding lines identified as Swiss German / Alemannic.

If you encounter something that is unexpected, please file an issue here: https://github.com/oscar-corpus/corpus/issues.

Dataset Structure

We show detailed information for all the configurations of the dataset.

Data Instances

TODO

Layout

{
 "content":"English sentence\nphrase en français\n????????????", // (1)
 "warc_headers":{ // (2)
 "warc-identified-content-language":"fra,eng",
 "warc-target-uri":"https://fr.wikipedia.org/wiki/...",
 "warc-record-id":"<urn:uuid:29eaa920-d299-4b1d-b687-c72bd8d68116>",
 "warc-type":"conversion",
 "content-length":"35298", // (3)
 "warc-refers-to":"<urn:uuid:39e42055-0d94-4e45-9c6c-9e7056635d64>",
 "warc-block-digest":"sha1:WFH2A5WHCS2H365GIAFYQPI7UOAMFGHB", // (3)
 "warc-date":"2022-11-26T09:45:47Z",
 "content-type":"text/plain"
 },
 "metadata":{
 "identification":{ // (4)
 "label":"fr",
 "prob":0.8938327
 },
 "harmful_pp":4063.1814, // (5)
 "tlsh":"tlsh:T125315FF2B6088901EEA097015DB39B4600B...", // (6)
 "quality_warnings":[ // (7)
 "short_sentences",
 "header",
 "footer"
 ],
 "categories":[ // (8)
 "examen_pix",
 "liste_bu"
 ],
 "sentence_identifications":[ // (9)
 {
 "label":"fr",
 "prob":0.99837273
 },
 {
 "label":"en",
 "prob":0.9992377
 },
 null
 ]
 }
}

Data Splits

Table

Dataset Creation

Curation Rationale

OSCAR was constructed using Ungoliant, a new pipeline derived from goclassy, itself being derived from fastText's one.

The pipeline works on documents rather than lines. Ungoliant is implemented in the Rust programming language, and uses rayon as its data parallelism strategy. Threading is done at shard, record and sentence level, making the whole generation process much more efficient.

Filtering will be explained in a future blog post at our website

Source Data

Initial Data Collection and Normalization

Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected nofollow and robots.txt policies.

Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.

To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR 22.01, the November/December 2021 snapshot was used. It is composed by 64 000 compressed text files containing documents and their headers.

Who are the source language producers?

The data comes from multiple web pages in a large variety of languages.

Annotations

The dataset does not contain any additional annotations.

Annotation process

N/A

Who are the annotators?

N/A

Personal and Sensitive Information

Being constructed from Common Crawl, Personal and sensitive information might be present. This must be considered before training deep learning models with OSCAR, specially in the case of text-generation models.

Considerations for Using the Data

Social Impact of Dataset

OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures.

Discussion of Biases

OSCAR is not properly filtered yet and this can be reflected on the models trained with it. Care is advised specially concerning biases of the resulting models.

Other Known Limitations

The fastText linear classifier is limed both in performance and the variety of languages it can recognize, so the quality of some OSCAR sub-corpora might be lower than expected, specially for the lowest-resource langiuages. Some audits have already been done by third parties.

Additional Information

Dataset Curators

This release of OSCAR was made possible by Julien Abadji, Pedro Ortiz Suarez, Rua Ismail, Sotaro Takeshita, Sebastian Nagel and Benoit Sagot.

Licensing Information

These data are released under this licensing scheme
We do not own any of the text from which these data has been extracted.
We license the actual packaging, the metadata and the annotations of these data under the Creative Commons CC0 license ("no rights reserved") http://creativecommons.org/publicdomain/zero/1.0/
To the extent possible under law, the OSCAR project, Inria, the Univertity of Mannheim and DFKI GmbH have waived all copyright and related or neighboring rights to OSCAR
This work is published from: France and Germany.

Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
* Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
* Clearly identify the copyrighted work claimed to be infringed.
* Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

We will comply to legitimate requests by removing the affected sources from the next release of the corpus.

Citation Information

@ARTICLE{2022arXiv221210440J,
 author = {{Jansen}, Tim and {Tong}, Yangling and {Zevallos}, Victoria and {Ortiz Suarez}, Pedro},
 title = "{Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data}",
 journal = {arXiv e-prints},
 keywords = {Computer Science - Computation and Language},
 year = 2022,
 month = dec,
 eid = {arXiv:2212.10440},
 pages = {arXiv:2212.10440},
 doi = {10.48550/arXiv.2212.10440},
archivePrefix = {arXiv},
 eprint = {2212.10440},
 primaryClass = {cs.CL},
 adsurl = {https://ui.adsabs.harvard.edu/abs/2022arXiv221210440J},
 adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@inproceedings{abadji-etal-2022-towards,
 title = "Towards a Cleaner Document-Oriented Multilingual Crawled Corpus",
 author = "Abadji, Julien and
 Ortiz Suarez, Pedro and
 Romary, Laurent and
 Sagot, Beno{\^\i}t",
 booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
 month = jun,
 year = "2022",
 address = "Marseille, France",
 publisher = "European Language Resources Association",
 url = "https://aclanthology.org/2022.lrec-1.463",
 pages = "4344--4355",
 abstract = "The need for large corpora raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.",
}

 
@inproceedings{AbadjiOrtizSuarezRomaryetal.2021,
 author = {Julien Abadji and Pedro Javier Ortiz Su{\'a}rez and Laurent Romary and Beno{\^i}t Sagot},
 title = {Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus},
 series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)},
 editor = {Harald L{\"u}ngen and Marc Kupietz and Piotr Bański and Adrien Barbaresi and Simon Clematide and Ines Pisetta},
 publisher = {Leibniz-Institut f{\"u}r Deutsche Sprache},
 address = {Mannheim},
 doi = {10.14618/ids-pub-10468},
 url = {https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688},
 pages = {1 -- 9},
 year = {2021},
 abstract = {Since the introduction of large language models in Natural Language Processing, large raw corpora have played a crucial role in Computational Linguistics. However, most of these large raw corpora are either available only for English or not available to the general public due to copyright issues. Nevertheless, there are some examples of freely available multilingual corpora for training Deep Learning NLP models, such as the OSCAR and Paracrawl corpora. However, they have quality issues, especially for low-resource languages. Moreover, recreating or updating these corpora is very complex. In this work, we try to reproduce and improve the goclassy pipeline used to create the OSCAR corpus. We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data. Also, unlike OSCAR, the metadata information is at the document level. We release our pipeline under an open source license and publish the corpus under a research-only license.},
 language = {en}
}

@article{kreutzer-etal-2022-quality,
 title = "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets",
 author = {Kreutzer, Julia and
 Caswell, Isaac and
 Wang, Lisa and
 Wahab, Ahsan and
 van Esch, Daan and
 Ulzii-Orshikh, Nasanbayar and
 Tapo, Allahsera and
 Subramani, Nishant and
 Sokolov, Artem and
 Sikasote, Claytone and
 Setyawan, Monang and
 Sarin, Supheakmungkol and
 Samb, Sokhar and
 Sagot, Beno{\^\i}t and
 Rivera, Clara and
 Rios, Annette and
 Papadimitriou, Isabel and
 Osei, Salomey and
 Suarez, Pedro Ortiz and
 Orife, Iroro and
 Ogueji, Kelechi and
 Rubungo, Andre Niyongabo and
 Nguyen, Toan Q. and
 M{\"u}ller, Mathias and
 M{\"u}ller, Andr{\'e} and
 Muhammad, Shamsuddeen Hassan and
 Muhammad, Nanda and
 Mnyakeni, Ayanda and
 Mirzakhalov, Jamshidbek and
 Matangira, Tapiwanashe and
 Leong, Colin and
 Lawson, Nze and
 Kudugunta, Sneha and
 Jernite, Yacine and
 Jenny, Mathias and
 Firat, Orhan and
 Dossou, Bonaventure F. P. and
 Dlamini, Sakhile and
 de Silva, Nisansa and
 {\c{C}}abuk Ball{\i}, Sakine and
 Biderman, Stella and
 Battisti, Alessia and
 Baruwa, Ahmed and
 Bapna, Ankur and
 Baljekar, Pallavi and
 Azime, Israel Abebe and
 Awokoya, Ayodele and
 Ataman, Duygu and
 Ahia, Orevaoghene and
 Ahia, Oghenefego and
 Agrawal, Sweta and
 Adeyemi, Mofetoluwa},
 journal = "Transactions of the Association for Computational Linguistics",
 volume = "10",
 year = "2022",
 address = "Cambridge, MA",
 publisher = "MIT Press",
 url = "https://aclanthology.org/2022.tacl-1.4",
 doi = "10.1162/tacl_a_00447",
 pages = "50--72",
 abstract = "With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50{\%} sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.",
}

@inproceedings{ortiz-suarez-etal-2020-monolingual,
 title = "A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages",
 author = "Ortiz Su{'a}rez, Pedro Javier and
 Romary, Laurent and
 Sagot, Benoit",
 booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
 month = jul,
 year = "2020",
 address = "Online",
 publisher = "Association for Computational Linguistics",
 url = "https://www.aclweb.org/anthology/2020.acl-main.156",
 pages = "1703--1714",
 abstract = "We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.",
}

@inproceedings{OrtizSuarezSagotRomary2019,
 author = {Pedro Javier {Ortiz Su{'a}rez} and Benoit Sagot and Laurent Romary},
 title = {Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures},
 series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019},
 editor = {Piotr Bański and Adrien Barbaresi and Hanno Biber and Evelyn Breiteneder and Simon Clematide and Marc Kupietz and Harald L{"u}ngen and Caroline Iliadi},
 publisher = {Leibniz-Institut f{"u}r Deutsche Sprache},
 address = {Mannheim},
 doi = {10.14618/ids-pub-9021},
 url = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},
 pages = {9 -- 16},
 year = {2019},
 abstract = {Common Crawl is a considerably large, heterogeneous multilingual corpus comprised of crawled documents from the internet, surpassing 20TB of data and distributed as a set of more than 50 thousand plain text files where each contains many documents written in a wide variety of languages. Even though each document has a metadata block associated to it, this data lacks any information about the language in which each document is written, making it extremely difficult to use Common Crawl for monolingual applications. We propose a general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language; we specifically design it so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint. We develop the pipeline so that it can be easily reapplied to any kind of heterogeneous corpus and so that it can be parameterised to a wide range of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered, classified by language, shuffled at line level in order to avoid copyright issues, and ready to be used for NLP applications.},
 language = {en}
}