RoBERTaCrawlPT-base

RoBERTaCrawlPT-base is a generic Portuguese Masked Language Model pretrained from scratch from the CrawlPT corpora, using the same architecture as RoBERTa-base. This model is part of the RoBERTaLexPT work.

• Language(s) (NLP): Portuguese (pt-BR mainly)
• License: Creative Commons Attribution 4.0 International Public License
• Repository: https://github.com/eduagarcia/roberta-legal-portuguese
• Paper: https://aclanthology.org/2024.propor-1.38/

Generic Evaluation

TO-DO...

Legal Evaluation

The model was evaluated on "PortuLex" benchmark, a four-task benchmark designed to evaluate the quality and performance of language models in the Portuguese legal domain.

Macro F1-Score (%) for multiple models evaluated on PortuLex benchmark test splits:

Training Details

RoBERTaCrawlPT is pretrained on:

• CrawlPT is a composition of three Portuguese general corpora: brWaC, CC100 PT subset, OSCAR-2301 PT subset.

Training Procedure

Our pretraining process was executed using the Fairseq library v0.10.2 on a DGX-A100 cluster, utilizing a total of 2 Nvidia A100 80 GB GPUs. The complete training of a single configuration takes approximately three days.

This computational cost is similar to the work of BERTimbau-base, exposing the model to approximately 65 billion tokens during training.

Preprocessing

We deduplicated all subsets of the CrawlPT Corpus using the a MinHash algorithm and Locality Sensitive Hashing implementation from the libary text-dedup to find clusters of duplicate documents.

To ensure that domain models are not constrained by a generic vocabulary, we utilized the HuggingFace Tokenizers -- BPE algorithm to train a vocabulary for each pre-training corpus used.

Training Hyperparameters

The pretraining process involved training the model for 62,500 steps, with a batch size of 2048 and a learning rate of 4e-4, each sequence containing a maximum of 512 tokens.
The weight initialization is random.
We employed the masked language modeling objective, where 15% of the input tokens were randomly masked.
The optimization was performed using the AdamW optimizer with a linear warmup and a linear decay learning rate schedule.

For other parameters we adopted the standard RoBERTa-base hyperparameters:

Citation

@inproceedings{garcia-etal-2024-robertalexpt,
 title = "{R}o{BERT}a{L}ex{PT}: A Legal {R}o{BERT}a Model pretrained with deduplication for {P}ortuguese",
 author = "Garcia, Eduardo A. S. and
 Silva, Nadia F. F. and
 Siqueira, Felipe and
 Albuquerque, Hidelberg O. and
 Gomes, Juliana R. S. and
 Souza, Ellen and
 Lima, Eliomar A.",
 editor = "Gamallo, Pablo and
 Claro, Daniela and
 Teixeira, Ant{\'o}nio and
 Real, Livy and
 Garcia, Marcos and
 Oliveira, Hugo Gon{\c{c}}alo and
 Amaro, Raquel",
 booktitle = "Proceedings of the 16th International Conference on Computational Processing of Portuguese",
 month = mar,
 year = "2024",
 address = "Santiago de Compostela, Galicia/Spain",
 publisher = "Association for Computational Lingustics",
 url = "https://aclanthology.org/2024.propor-1.38",
 pages = "374--383",
}

Acknowledgment

This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).

Related resources to the paper RoBERTaLexPT: A Legal RoBERTa Model pretrained with deduplication for Portuguese • 7 items • Updated Mar 2 • 4