Skip to content

This is a list of speech tasks and datasets, which can provide training data for Generative AI, AIGC, AI model training, intelligent speech tool development, and speech applications.

Notifications You must be signed in to change notification settings

WangHelin1997/SpeechTasks

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

55 Commits
 
 

Repository files navigation

SpeechTasks

This is a list of speech tasks and datasets, which can provide training data for Generative AI, AIGC, AI model training, intelligent speech tool development, and speech applications.

Continuously Updating!

I will add new tasks and datasets to this repo continously.

You are welcome to put an issue or email me at [email protected], to point out any unlisted tasks and datasets!

Table of Contents

  1. Tasks and DataSets
  2. Tasks with different input & output modes
  3. Tasks with different Level

Tasks and DataSets

Task DataSets Input Mode Output Mode Modeling Target Level Description
Accent Classification AccentDB Extended Dataset Audio Label Classification Acoustic, Language Accent classification involves the recognition and classification of specific speech accents.The task involves the recognition and classification of specific speech accents. The possible answers include American, Australian, Bangla, British, Indian, Malayalam, Odiya, Telugu, or Welsh. The objective is to correctly identify these accents based on the given speech samples, contributing to a system's ability to understand and interact with various speakers.
Accented Text-to-speech L2-ARCTIC Text, Audio Audio Generation Acoustic, Language Accented text-to-speech (TTS) synthesis aims to synthesize speech with a given foreign accent instead of native speech.
Acoustic Echo Cancellation AEC Challenge Audio Audio Regression Acoustic The Acoustic Echo Cancellation block is designed to remove echoes, reverberation, and unwanted added sounds from a signal that passes through an acoustic space.
Automatic Speech Recognition LibriSpeech
Common Voice
VoxPopuli
MLS
Libri-light
AISHELL
GigaSpeech
CoVoST
Libriheavy
TED-LIUM
TIMIT
WenetSpeech
Audio Text Classification Content Speech recognition, also known as automatic speech recognition (ASR), computer speech recognition, or speech-to-text, is a capability which enables a program to process human speech into a written format.
DeepFake Detection (Spoof Detection) ASVspoof 2015 Dataset
ASVspoof 2017 Dataset
ASVspoof2019
ASVspoof2021
ASVspoof5
ADD Challenge
In-the-Wild
WaveFake
SingFake
Audio Binary Label Binary Classification Acoustic Audio deepfake detection is a task that aims to distinguish genuine utterances from fake ones via machine learning techniques. <!--
Dialogue Act Classification DailyTalk Dataset Audio Label Classification Understanding Dialogue act classification aims to identify the primary purpose or function of an utterance within its dialogue context.The aim of this task is to identify the action in the audio. The possible answers could be question, inform, directive, or commissive. These identification tasks are important, as dialogue acts are central to understanding human conversation and dialogue-based AI system communication.
Dialogue Act Pairing DailyTalk Dataset Audio, Label Binary Label Binary Classification Understanding Dialogue act pairing involves assessing the congruence of dialogue acts—that is, whether a response dialogue act is appropriate given a query dialogue act. The objective is to determine whether a given dialogue act pairing is congruent or not. The answer could either be true or false. Being able to accurately judge the appropriateness of dialogue acts is key for a universal speech model to understand and participate in human conversations effectively.
Dialogue Emotion Classification DailyTalk Dataset Audio Label Classification Emotion Dialogue emotion classification is a task that assesses an AI model's ability to identify the most suitable emotion in a given dialogue extract. The main goal of this task is to correctly identify the communicated emotion in an audio clip. Possible answers include anger, disgust, fear, sadness, happiness, surprise, or no emotion. It is an evaluation of the model's capacity to interpret and distinguish emotions conveyed through speech, accounting both for linguistic content and paralinguistic indicators.
Dysarthric Speech Assessments UASpeech
TORGO
Audio Scalar Regression Acoustic Dysarthric speech assessments regarding speech intelligibility are conducted to check the patient’s status and track the effectiveness of treatments.
Dysarthric Speech Recognition UASpeech
TORGO
Audio Text Classification Content Dysarthric Speech Recognition is a task that aims to transcribe dysarthria speech which is a motor speech disorder caused by conditions like Parkinson’s disease or amyotrophic lateral sclerosis (ALS).
Emotion Recognition Multimodal EmotionLines Dataset
IEMOCAP
MELD
CREMA-D
MSP-Podcast
SAVEE
MESD
CMU-MOSEI
MEAD
Audio Label Classification Emotion Emotion recognition aims to identify the most appropriate emotional category for a given utterance.Recognizing the emotion expressed in an utterance can be quite challenging. While we can sometimes identify emotion from the linguistic content alone, the more important factors often lie in paralinguistic features — like pitch, rhythm, and other prosodic elements. For a universal speech model, understanding these paralinguistic features is crucial, as they distinguish speech from mere text in a significant manner.
Emotional TTS RAVDESS
EMOV-DB
LJSpeech Dataset
IEMOCAP
Text,Label Audio Generation Acoustic, Emotion Emotional text-to-speech (TTS) aims to synthesize speech with specfic emotional types.
Enhancement Detection LibriTTS-TestClean Audio Binary Label Binary Classification Acoustic Enhancement detection is a task focused on determining whether a given audio has been created or modified by a speech enhancement model. The objective of enhancement detection is to ascertain if an audio file has been created or altered by a speech enhancement model. The expected answer is either yes or no. The task poses a challenging problem because the speech model must not only process the content of the speech but also detect minute modifications that might indicate enhancement.
Expressive TTS Expresso Text, Label Audio Generation Acoustic, Understanding Expressive text-to-speech (TTS) aims to synthesize speech with specfic reading types or improvised styles.
HowFarAreYou 3DSpeaker Dataset
Spatial LibriSpeech
Audio Scalar Regression Acoustic The HowFarAreYou task aims to determine the distance of the speaker from the source of sound. The task's goal is to ascertain the approximate distance of a speaker, based on the provided audio or speech. The task's response could be an exact value, such as 0.4m, 2.0m, or 4.0m, indicating the speaker's distance from the sound source. Gauging the speaker's distance provides insights into the audio's spatial characteristics, which forms a crucial aspect of auditory scene analysis.
Instruct TTS None available Text Audio Generation Acoustic, Understanding Instruct text-to-speech (TTS) aims to synthesize speech with varying speaking styles to better reflect human speech patterns, given a certain instruction.
Intent Classification FluentSpeechCommands Dataset
SLURP
ATIS
Snips
Audio Label Classification Understanding Intent classification aims to identify the actionable item behind a spoken message. The objective of this task is to understand and categorize the intent performed by a spoken message. The recognized actions can vary, including activate, bring, change language, deactivate, decrease, or increase. Identifying the intent accurately is pivotal for building reliable speech-based applications and interfaces. We categorize this task into three types: Action, Location, and Object.
Keyword Spotting Google Speech Commands V1 Dataset
LibriPhrase
Audio, Text Binary Label Binary Classification Content Keyword spotting is a process that helps to detect keywords or phrases used in phone calls or audio recordings. These words and phrases can then be used to adjust the urgency of the call, train your employees, and gauge customer satisfaction.
Language Identification VoxForge Dataset
Common Voice
VoxLingua107
Audio Label Classification Language Language Identification task is aimed to determine the language spoken in a given speech recording. The main goal of this task is to identify the language spoken in a specific speech recording. This is an essential part of speech processing, as it facilitates the understanding and translations for different languages. The language spoken could be German, English, Spanish, Italian, Russian, or French.
Laughter Synthesis Laughterscape Audio, Audio Audio Generation Acoustic Laughter Synthesis task is aimed to generate sound of laughter of a given speaker.
Multilingual Speech Recognition Common Voice
VoxLingua107
MLS
FLEURS
CMU Wilderness
YODAS
Audio Text Classification Content, Language The task of Multilingual Speech Recognition (MSR) involves developing systems that can accurately transcribe speech data across multiple languages. Unlike traditional speech recognition systems that are designed for a specific language, MSR systems aim to handle diverse languages and dialects.
MultiSpeaker Detection LibriSpeech-TestClean Dataset
VCTK Dataset
Audio Binary Label Binary Classification Speaker MultiSpeaker Detection aims to analyze the speech audio to determine whether there is more than one speaker present in it. The core objective of this task is to analyze the speech audio for the presence of more than one speaker. It is crucial for a universal speech model to detect this as the presence of multiple speakers can alter the context and understanding of the spoken content.
Noise Detection LJSpeech dataset
VCTK Dataset
Musan Dataset
Audio Binary Label Binary Classification Acoustic Noise Detection aims to idenetify if the speech audio is clean or mixed with noises.The objective of noise detection is to ascertain if an audio file has been added the noise. The expected answer is either yes or no. There are many types of noises - like music, speech, gaussian or others. The task poses a challenging problem because the speech model must not only process the content of the speech but also understand the degradation of speech.
Noise SNR Level Prediction VCTK Dataset
Musan Dataset
Audio Scalar Regression Acoustic Noise SNR Level Prediction aims to predict the signal-to-noise ratio of the speech audio.The objective of noise SNR level prediction is to evaluate the noise SNR level of an audio file. The expected answer could be zero, five, ten, fifteen or zero. There are many types of noises - like music, speech, gaussian or others. The task poses a challenging problem because the speech model must not only process the content of the speech but also understand the degree of noise degradation.
Non-verbal Voice Recognition CNVVE Audio Label Classification Content Non-verbal Voice Recognition is to recognize non-verbal or non-lexical voice expressions, like humming.
Offensive Language Identification OLID Audio Label Classification Understanding Offensive Language Identification is to identify the type and the target of offensive texts in social media.
Overlapping Speech Detection AMI meeting corpora
DIHARD I Challenge Data
DIHARD II Challenge Data
VoxConverse
Audio Label, Timestamp Classification Content, Speaker Overlapped speech detection (OSD) is a task that estimates onsets and offsets of segments (i.e., a small part of an audio clip) within an audio clip (i.e., utterance, session, conversation as a whole) where more than one speaker is speaking simultaneously.
Reverberation Detection LJSpeech Dataset
VCTK Dataset
RIRs Noises Dataset
Audio Binary Label Binary Classification Acoustic Reverberation Detection aims to detect if the speech audio is clean or mixed with room impulse responses (RIRs) and noises, that is to say reverberation noises. The objective of reverberation detection is to ascertain if an audio file has been added the reverberation noises. The expected answer is either clean or noisy. The reverberation noises can be originated from large room, medium room or small room. The task poses a challenging problem because the speech model must not only process the content of the speech but also understand the degradation of speech in reververation cases.
Sarcasm Detection MUStARD Dataset Audio Binary Label Binary Classification Understanding Sarcasm Detection aims to detect if the sarcasm or the irony present in the speech audio. The objective of sarcasm detection is to recognize the presence of sarcasm or ironic expressions in the speech. The expected answer is either true or false. The task poses a challenging problem because the speech model should understand upper level of the semantic information.
Slot Filling SLURP
ATIS
Snips
Audio Text Classification Understanding The goal of Slot Filling is to identify from a running dialog different slots, which correspond to different parameters of the user’s query. For instance, when a user queries for nearby restaurants, key slots for location and preferred food are required for a dialog system to retrieve the appropriate information. Thus, the main challenge in the slot-filling task is to extract the target entity.
Speaker Counting MUStARD Dataset Audio Label Classification Speaker Speaker Counting aims to identify the total number of speaker in speech audio. The objective of speaker counting is to determine the number of speakers in the audio recording. The expected answer should be one, two, three, four, or five. The task poses a challenging problem because the speech model should undersdand the pattern of different speakers.
Speaker Diarization CHIME 5
CHIME 6
DIHARD II
LibriCSS
AISHELL-4
VoxConverse
Audio Label, Timestamp Classification Speaker Speaker diarisation is the process of partitioning an audio stream containing human speech into homogeneous segments according to the identity of each speaker.
Speaker Identification LibriSpeech-TestClean Dataset
VCTK Dataset
VoxCeleb1
VoxCeleb2
CN-Celeb
AVSpeech
VoxTube
Audio Label Classification Speaker Speaker recognition deals with the identification of the speaker in an audio stream.
Speaker Verification LibriSpeech-TestClean Dataset
VCTK Dataset
VoxCeleb1
VoxCeleb2
CN-Celerb
Audio, Audio Binary Label Binary Classification Speaker Speaker verification aims to verify whether the two given speech audios are from the same speaker. The objective of speaker verification is to exam if the patterns in the two audio recordings come from the same speaker. The expected answer is either yes or no. The task poses a challenging problem because the speech model should undersdand the pattern of different speakers.
Speech Edit LibriTTS
VCTK Dataset
LJSpeech Dataset
Audio, Text Audio Generation Acoustic, Content Speech edit allows the user to edit the recorded speech, e.g., insert missed words, replace mispronounced words, and/or remove unwanted speech or non-speech events, without degrading the quality and naturalness of the edited speech.
Speech Command Recognition Google Speech Commands V1 Dataset Audio Label Classification Content Speech Command Recognition aims to identify the spoken command. The objective of speech command recognition is to comprehend and grasp the command presented in the speech. The expected answer should be yes, no, up, down, left, right, on, off, stop, go, zero, one, two, three, four, five, six, seven, eight, nine, bed, bird, cat, dog, happy, house, marvin, sheila, tree, wow, or silence. The task poses a challenging problem because the speech model should understand the content information from the speech audios.
Speech Dereverberation Reverb-WSJ0
WHAMR!
CHIME 5
CHIME 6
Audio Audio Regression Acoustic Speech Dereverberation is the process by which the effects of reverberation are removed from sound, after such reverberant sound has been picked up by microphones.
Speech Detection LJSpeech dataset
LibriSpeech-TestClean Dataset
LibriSpeech-TestOther Dataset
InaGVAD
Audio Binary Label Binary Classification Content Speech Detection, also known as voice activity detection and speech activity detection, aims to identify whether the given audio clip contains real speech or not. The objective of speech detection is to analyze the audio and determine whether it consists of real speech or not. The expected answer is either yes or no. The task poses a challenging problem because the speech model should understand not only the content information from the speech audios but the pattern of human voice.
Speech Enhancement VoiceBank+DEMAND
DNS-Challenge
WHAM!
WHAMR!
Audio Audio Regression Acoustic Speech enhancement aims to improve speech quality by using various algorithms. The objective of enhancement is improvement in intelligibility and/or overall perceptual quality of degraded speech signal using audio signal processing techniques.
Speech Separation WSJ0-2mix
LibriMix
Real-M
WHAM!
WHAMR!
CHIME 5
CHIME 6
AISHELL-4
Audio Audio, Audio Regression Speaker Speech Separation is the extraction of multiple speech signals from a mixture.
Speech Text Matching LJSpeech dataset
LibriSpeech-TestClean Dataset
LibriSpeech-TestOther Dataset
Audio, Text Binary Label Binary Classification Content Speech Text Matching aims to determine if the speech and text are matched. The objective of speech text matching is to assess whether the speech and text share the same underlying message or not. The expected answer is either yes or no. The task poses a challenging problem because the speech model should understand the content information from the speech audios.
Speech-to-speech Translation CVSS
CoVoST 2
Audio Audio Generation Language, Content Speech-to-speech translation consists on translating speech from one language to speech in another language. This can be done with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which is text-centric.
Speech-to-text Translation MUST-C Audio Text Generation Language, Content Speech-to-text translation, consisting of automatic speech recognition (ASR) and machine translation (MT), refers to the process where spoken language is not only converted to text but also translated into another language.
Speech Quality Assessment VCC2018
BVCC
Audio Scalar Regression Acoustic Speech Quality Assessment is a task that estimates the quality of speech, like mean-opinion-score (MOS).
Spoken Question Answering Spoken-SQuAD
ODSQA
NMSQA
Audio Text Generation Understanding Spoken Question Answering (SQA) aims to find the answer from a spoken document given a question in either text or spoken form. SQA is crucial for personal assistants when replying to the questions from the user’s spoken queries.
Spoken Term Detection LJSpeech dataset
LibriSpeech-TestClean Dataset
LibriSpeech-TestOther Dataset
Audio, Text Binary Label Binary Classification Content Spoken Term Detection aims to check for the existence of the given word in the speech. The objective of spoken term detection is to analyze the speech and indicate whether the word is mentioned or not. The expected answer is either yes or no. The task poses a challenging problem because the speech model should understand the content information from the speech audios.
Stress Detection MIR-SD Dataset Audio Binary Label Binary Classification Acoustic Stress Detection aims to determine the stress placement in English vocabulary. The objective of stress detection is to analyze the stress patterns in English words. The expected answer should be zero, one, two, three, four, or five. For a universal speech model, understanding these paralinguistic features is crucial, as they distinguish speech from mere text in a significant manner.
Target Speaker Extraction WSJ0-2mix
LibriMix
Real-M
WHAM!
WHAMR!
CHIME 5
CHIME 6
Audio,Audio Audio Regression Speaker Target Speaker Extraction aims to segregate the speech of a target speaker from a mixture of interfering speakers with the help of auxiliary information.
Text-To-Speech Synthesis LJ Speech
LibriTTS
AISHELL 3
LibriTTS-R
YTTTS
Text Audio Generation Acoustic Text-to-speech (TTS) synthesis converts normal language text into speech.
Vocal Sound Classification VocalSound Audio Label Classification Acoustic Vocal Sound Classificatio aims at automatic human vocal sound recognition for laughter, sighs, coughs, throat clearing, sneezes, and sniffs.
Voice Conversion LibriTTS
VCTK Dataset
ESD
Audio, Audio Audio Generation Acoustic, Speaker Voice Conversion is a technology that modifies the speech of a source speaker and makes their speech sound like that of another target speaker without changing the linguistic information.

And more

Tasks with different input & output modes

Input Output Tasks
Audio Audio Acoustic Echo Cancellation, Speech Dereverberation, Speech Enhancement, Speech-to-speech Translation
Audio Audio, Audio Speech Separation, Voice Conversion
Audio Binary Label DeepFake Detection, Dialogue Act Classification, Enhancement Detection, MultiSpeaker Detection, Noise Detection, Reverberation Detection, Sarcasm Detection, Speech Detection, Spoof Detection, Stress Detection
Audio Label Accent Classification, Dialogue Act Classification, Dialogue Emotion Classification, Emotion Recognition, Intent Classification, Language Identification, Non-verbal Voice Recognition, Offensive Language Identification, Speaker Counting, Speaker Identification, Speech Command Recognition, Vocal Sound Classification
Audio Label, Timestamp Speaker Diarization, Overlapping Speech Detection
Audio Scalar Dysarthric Speech Assessments, HowFarAreYou, Noise SNR Level Prediction, Speech Quality Assessment
Audio Text Automatic Speech Recognition, Dysarthric Speech Recognition, Multilingual Speech Recognition, Slot Filling, Speech-to-text Translation, Spoken Question Answering
Audio, Audio Audio Laughter Synthesis, Target Speaker Extraction
Audio, Audio Binary Label Speaker Verification
Audio, Label Binary Label Dialogue Act Pairing, Keyword Spotting
Audio, Text Audio Accented Text-to-speech, Speech Edit
Audio, Text Binary Label Speech Text Matching, Spoken Term Detection
Text Audio Instruct TTS, Text-To-Speech Synthesis
Text, Label Audio Emotional TTS, Expressive TTS

Tasks with different Level

Level Tasks
Acoustic Accent Classification, Accented Text-to-speech, Acoustic Echo Cancellation, DeepFake Detection, Dysarthric Speech Assessments, Emotional TTS, Enhancement Detection, Expressive TTS, HowFarAreYou, Instruct TTS, Laughter Synthesis, Noise Detection, Noise SNR Level Prediction, Reverberation Detection, Speech Edit, Speech Dereverberation, Speech Enhancement, Speech Quality Assessment, Spoof Detection, Stress Detection, Text-To-Speech Synthesis, Vocal Sound Classification, Voice Conversion
Content Automatic Speech Recognition, Dysarthric Speech Recognition, Keyword Spotting, Multilingual Speech Recognition, Non-verbal Voice Recognition, Overlapping Speech Detection, Speech Edit, Speech Command Recognition, Speech Detection, Speech Text Matching, Speech-to-speech Translation, Speech-to-text Translation, Spoken Term Detection, Vocal Sound Classification
Emotion Dialogue Emotion Classification, Emotion Recognition, Emotional TTS
Language Accent Classification, Accented Text-to-speech, Language Identification, Multilingual Speech Recognition, Speech-to-speech Translation, Speech-to-text Translation
Speaker MultiSpeaker Detection, Overlapping Speech Detection, Speaker Counting, Speaker Diarization, Speaker Identification, Speaker Verification, Speech Separation, Target Speaker Extraction, Voice Conversion
Understanding Dialogue Act Classification, Dialogue Act Pairing, Expressive TTS, Instruct TTS, Intent Classification, Offensive Language Identification, Sarcasm Detection, Slot Filling, Spoken Question Answering

References

  1. Dynamic-SUPERB

  2. paperswithcode

  3. kaggle

  4. AI-ADL

  5. INTERSPEECH 2023

About

This is a list of speech tasks and datasets, which can provide training data for Generative AI, AIGC, AI model training, intelligent speech tool development, and speech applications.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published