Computer Science > Computation and Language
arXiv:2110.13900 (cs)
[Submitted on 26 Oct 2021 (v1), last revised 17 Jun 2022 (this version, v5)]
Title:WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Authors:Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
View a PDF of the paper titled WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing, by Sanyuan Chen and 18 other authors
View PDF
Abstract:Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at this https URL.
| Comments: | Submitted to the Journal of Selected Topics in Signal Processing (JSTSP) |
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2110.13900 [cs.CL] |
| (or arXiv:2110.13900v5 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2110.13900
arXiv-issued DOI via DataCite
|
|
| Related DOI: | https://doi.org/10.1109/JSTSP.2022.3188113
DOI(s) linking to related resources
|
Submission history
From: Sanyuan Chen [view email][v1] Tue, 26 Oct 2021 17:55:19 UTC (529 KB)
[v2] Fri, 29 Oct 2021 07:43:03 UTC (910 KB)
[v3] Wed, 22 Dec 2021 16:24:51 UTC (941 KB)
[v4] Mon, 24 Jan 2022 08:57:42 UTC (391 KB)
[v5] Fri, 17 Jun 2022 05:44:33 UTC (411 KB)
Full-text links:
Access Paper:
- View PDF
- TeX Source
View a PDF of the paper titled WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing, by Sanyuan Chen and 18 other authors
Current browse context:
cs.CL
References & Citations
Loading...
BibTeX formatted citation
×
Data provided by:
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.