Key Speech-to-Text Datasets: LibriSpeech, TED-LIUM, and CHiME Explained

When vendors cite accuracy benchmarks, they're usually testing against standard academic datasets. Understanding these datasets—what they contain and what they don't—helps you interpret benchmark claims and understand their relevance to your use case.

Why datasets matter

STT accuracy is always relative to test data. A model that scores 3% WER on one dataset might score 15% on another. The difference isn't the model changing—it's what's being tested.

Key questions for any benchmark:

• What kind of speech is in the dataset?
• How clean is the audio?
• Who are the speakers?
• Does this match my use case?

For a deeper dive into what WER actually measures, see our WER explainer.

LibriSpeech: the standard benchmark

What it is

Derived from LibriVox audiobook recordings. ~1,000 hours of read English speech from public domain audiobooks. The LibriSpeech corpus is freely available for research.

Characteristics

• Clean audio: Studio-quality recordings
• Read speech: Narrated audiobooks, not spontaneous conversation
• American English: Predominantly standard American accents
• Limited vocabulary: Literary language, not technical or conversational

What benchmark scores mean

LibriSpeech WER indicates performance on clear, carefully articulated English read aloud. It's a useful baseline but represents ideal conditions.

If a provider scores 2% WER on LibriSpeech:

• They handle clean English well
• They don't necessarily handle accents, noise, or conversational speech well
• Your real-world performance will almost certainly be worse

TED-LIUM: public speaking

What it is

Transcribed TED talks. Multiple versions exist (v1, v2, v3) with increasing size and coverage. Available from OpenSLR.

Characteristics

• Varied speakers: International speakers with diverse accents
• Prepared speech: Rehearsed talks, not spontaneous conversation
• Good audio: Professional recording but with audience ambience
• Diverse topics: Science, business, culture—varied vocabulary

What benchmark scores mean

TED-LIUM performance indicates handling of non-native accents and varied vocabulary, but still with prepared, clear speech.

Better than LibriSpeech for:

• Accent handling—see our guide on why accents matter in STT
• Topic diversity
• Realistic presentation scenarios

Still not representative of:

• Casual conversation
• Noisy environments
• Spontaneous speech with hesitations

CHiME: noise and real-world conditions

What it is

A series of CHiME Challenges focused on speech recognition in noisy environments. Multiple versions with increasing difficulty.

Characteristics

• Real noise: Traffic, cafes, buses, etc.
• Multi-microphone: Tests distant/array microphone scenarios
• Overlapping speech: Multiple simultaneous speakers
• Challenging conditions: Designed to push systems to failure

What benchmark scores mean

CHiME performance indicates robustness to noise and challenging acoustic conditions. This is where the gap between benchmark and reality shrinks. We explore what CHiME benchmarks teach us in detail separately.

Good CHiME performance suggests:

• Better noise handling
• More robust for real-world deployment
• Useful for applications in noisy environments

Caveats:

• Still may not match your specific noise characteristics
• Lab-curated noise differs from truly wild conditions

Other notable datasets

Common Voice (Mozilla)

Crowdsourced recordings from volunteers worldwide. Massive language coverage but variable recording quality. Good for language diversity, not for quality assessment.

VCTK

Multi-speaker dataset with varied British accents. Useful for accent testing within English.

Switchboard

Telephone conversations. Spontaneous speech but dated recording quality. Relevant for call center applications.

WSJ (Wall Street Journal)

Read news articles. Clean audio, formal language. Similar to LibriSpeech in character.

How to use benchmark information

Red flags

• Provider only cites LibriSpeech scores
• Benchmarks don't indicate dataset version or test conditions
• No breakdown by segment or condition
• Numbers seem too good compared to competition

Good signs

• Multiple dataset benchmarks provided
• Breakdown by accent, noise level, or other segments
• Honest about where performance degrades
• Willingness to test on your data

The right question

"How did you perform on [X] dataset?" is less useful than "Can you test on audio samples from my application?"

Benchmark scores are directionally useful but no substitute for testing with your actual content. See our benchmarking checklist for running your own evaluation.

For an overview of the major providers and how to compare them, see our STT API comparison guide.