Word error rate
Also: WER
Updated
Definition
Word error rate (WER) is the standard measure of speech recognition accuracy: the number of words a model got wrong, divided by the number of words actually spoken. A WER of 5% means about one word in twenty was substituted, dropped, or invented.
WER counts three kinds of mistake. A substitution is the wrong word in the right place ("their" for "there"). A deletion is a spoken word that never appeared. An insertion is a word that appeared and was never spoken. Add them up, divide by the number of words in the correct transcript, and you have the rate. It can exceed 100% on bad audio, because insertions are unbounded.
The number is only meaningful next to the audio it was measured on. A model that scores 3% on clean, read English from a podcast microphone may score 15% on the same speaker in a café, and worse still on a language it saw less of in training. Published WER figures are almost always measured on standard benchmark sets, which are cleaner than your desk.
What WER does not capture
Every error counts the same, and they are not the same. A dropped "um" is an improvement; a dropped "not" reverses the meaning. A misspelled surname costs you a correction; a misspelled "the" costs you nothing because you never see it. Formatting is invisible to WER entirely: "1030 am" and "10:30 AM" are the same words. So a tool with a slightly worse raw WER and a much better formatting pass can produce text that needs far fewer fixes.
That is why the number you actually care about is closer to "corrections per hundred words after formatting", which no benchmark reports and which you can only measure by dictating your own work for a week.
How to read a vendor’s accuracy claim
Ask which benchmark, which language, and which microphone. "99% accurate" with no dataset named is a marketing sentence. Ask whether the figure is before or after any correction step, and whether a personal dictionary was in play. And weigh it against the things WER cannot see: names, numbers, and whether the text arrives as writing or as a transcript.
Improving your own error rate
A better microphone, closer to your mouth, helps more than any model change. Finishing sentences before releasing the key gives the model context to disambiguate. And a personal dictionary for the names and terms you actually say fixes the errors that repeat, which are the ones that matter, because a mistake that happens once a month is a nuisance and one that happens every day is a reason to stop dictating.
How to measure your own word error rate on a Mac
You can measure WER on your own voice and your own microphone with a free open-source tool called jiwer. It computes the minimum edit distance between a correct transcript and what the recognizer produced, which is the same calculation benchmarks use.
The result describes your desk, your microphone, and the passage you read. It is a small sample, so treat it as a rough reading and repeat it with different text before you draw conclusions.
- Pick a passage of a few hundred words and save it as a plain text file named ref.txt, with one sentence per line. This is your reference.
- Dictate the same passage with the tool you want to test, and save the output as hyp.txt, with the same sentences on the same lines.
- Decide how you will treat capitals, punctuation, and numbers, then apply the same cleanup to both files. A simple choice is to lowercase everything and remove punctuation in both.
- Open Terminal and install the tool with: pip install jiwer (it needs Python 3.8 or later).
- Run: jiwer -r ref.txt -h hyp.txt to print the word error rate.
- Add -a to the command to print the alignment of each sentence, so you can see which words were substituted, dropped, or inserted.
- Add -c if you want the character error rate instead.
Why do two WER numbers for the same model disagree?
Usually because the text was cleaned up differently before scoring. WER is based on string edit distance, so it penalizes every difference between the two transcripts, including differences a person would call correct, such as a contraction written out in full or a number written as a digit.
The Whisper paper describes this problem and answers it with a text normalizer that is applied to both transcripts before the errors are counted. The authors report that for several datasets this step alone lowered WER by up to 50 percent. A WER figure with no description of the normalization behind it cannot be compared with another one.
- The Whisper normalizer removes the words hmm, mm, mhm, mmm, uh, and um before scoring.
- It removes any phrase inside brackets or parentheses.
- It expands contractions into their full form.
- It rewrites spelled-out numbers and currency amounts with digits, so that "ten thousand dollars" becomes "$10000".
- It converts British spellings to American spellings and strips most punctuation and symbols.
WER vs CER: words or characters
Character error rate (CER) is the same calculation done on characters instead of words. One wrong letter in a long word is one full error in WER and a small fraction of an error in CER, so CER is the gentler number and the two cannot be compared with each other.
Model publishers pick the unit per language. The Whisper project, for example, reports either WER or CER for each language on the Common Voice 15 and Fleurs datasets. When you read an accuracy table, check which of the two a row uses before you compare it with another row.
Questions people ask.
What is a good word error rate?
On clean English benchmark audio, the best current models score under 5%. In real use, with a laptop microphone and ordinary speech, 5 to 10% is typical and still very usable when the tool has a good formatting pass and a dictionary for your names.
How is word error rate calculated?
Substitutions plus deletions plus insertions, divided by the number of words in the correct transcript. A model that changes one word, drops one, and adds one in a hundred-word passage has a WER of 3%.
Is a lower WER always the better tool?
No. WER ignores formatting, punctuation, and which words were wrong. A tool that writes times, prices, and names correctly and removes fillers can need fewer corrections than one with a lower raw WER and no formatting pass.
How do I calculate WER myself?
Save the correct text and the recognizer output as two text files, install the open-source jiwer tool with pip install jiwer, and run jiwer -r ref.txt -h hyp.txt in Terminal. Apply the same cleanup of capitals and punctuation to both files first, or formatting differences will be counted as errors.
What is the difference between WER and CER?
WER counts wrong words and CER counts wrong characters, each divided by the length of the correct transcript. CER is lower for the same output because a word with one wrong letter is mostly right at the character level.
Does punctuation count in word error rate?
It depends on the text normalization used. Most benchmark setups strip punctuation and capitals from both transcripts before scoring, as the Whisper paper’s normalizer does, so the published number says nothing about how well a model punctuates.
