ArTST (ASR task)

ArTST model finetuned for automatic speech recognition (speech-to-text) on MGB2.

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Speech Lab, MBZUAI
• Model type: SpeechT5
• Language: Arabic
• Finetuned from: (ArTST pretrained)[https://github.com/mbzuai-nlp/ArTST]

How to Get Started with the Model

import soundfile as sf
from transformers import (
 SpeechT5Config,
 SpeechT5FeatureExtractor,
 SpeechT5ForSpeechToText,
 SpeechT5Processor,
 SpeechT5Tokenizer,
)

from custom_tokenizer import CustomTextTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = SpeechT5Tokenizer.from_pretrained("mbzuai/artst_asr")
processor = SpeechT5Processor.from_pretrained("mbzuai/artst_asr" , tokenizer=tokenizer)
model = SpeechT5ForSpeechToText.from_pretrained("mbzuai/artst_asr").to(device)

audio, sr = sf.read("audio.wav")

inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
predicted_ids = model.generate(**inputs.to(device), max_length=150, num_beams=10)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription[0])

Model Sources [optional]

• Repository: github
• Paper : ArabicNLP2023

Citation [optional]

BibTeX:

@inproceedings{toyin-etal-2023-artst,
 title = "{A}r{TST}: {A}rabic Text and Speech Transformer",
 author = "Toyin, Hawau and
 Djanibekov, Amirbek and
 Kulkarni, Ajinkya and
 Aldarmaki, Hanan",
 booktitle = "Proceedings of ArabicNLP 2023",
 month = dec,
 year = "2023",
 address = "Singapore (Hybrid)",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2023.arabicnlp-1.5",
 doi = "10.18653/v1/2023.arabicnlp-1.5",
 pages = "41--51",
}

Open source project for Arabic Speech Recognition and Generation • 12 items • Updated Mar 2 • 13