Speaker diarization
Also: speaker labels
Updated
Definition
Speaker diarization is the process of working out who spoke when in a recording, so a transcript can be labeled by speaker. It answers "who said this?" rather than "what was said?".
Diarization is a separate model from speech recognition. It listens for changes in voice, clusters the segments that sound like the same person, and assigns each a label, typically Speaker 1 and Speaker 2 until a name is attached. Combined with a transcript it produces the familiar meeting-notes layout, with each line attributed.
It is a transcription feature, not a dictation one. When you dictate, there is one speaker and it is you, so a dictation tool has no use for diarization and should not be judged for lacking it. When you transcribe a meeting, an interview, or a podcast, it is the difference between a transcript and a wall of text.
Why it is hard
Two people with similar voices, one person whose voice changes with a cold, overlapping speech, and a laptop microphone that flattens everyone into the same room tone all defeat the clustering. Speaker counts are usually guessed rather than known. The best systems are good on clean, turn-taking audio like a podcast and much worse on a real meeting where three people talk over each other. Expect to fix labels by hand.
Which tools have it
Meeting transcribers like Otter.ai build their product around it. File transcribers such as MacWhisper offer speaker labels on recordings. Dictation tools generally do not, because the job never has two speakers. If diarization is on your list of requirements, you are shopping for a transcriber.
How to add speaker labels to a recording on your Mac
The open-source toolkit pyannote.audio runs speaker diarization locally on your computer. It tells you who spoke when, with start and end times for each turn. It does not produce the words, so you pair its output with a transcript from a speech recognizer.
The steps below use the community pipeline, which is released under a CC-BY-4.0 license and needs a free Hugging Face account to download.
- Make sure ffmpeg is installed, for example with: brew install ffmpeg
- Install the toolkit in Terminal with: pip install pyannote.audio
- Sign in to Hugging Face and accept the user conditions on the pyannote/speaker-diarization-community-1 page.
- Create a Hugging Face access token at hf.co/settings/tokens.
- In Python, load the pipeline with Pipeline.from_pretrained, passing the pipeline name and your token, then call it on your audio file.
- Loop over the speaker_diarization field of the output to read each turn with its start time, end time, and speaker label.
- If you know how many people spoke, pass num_speakers when you call the pipeline. If you only know a range, pass min_speakers and max_speakers.
Can Whisper or Parakeet tell speakers apart?
No. Both are speech recognition models, and neither is built to label speakers. The Whisper model card says the models have not been robustly evaluated on speaker classification or speaker diarization. NVIDIA’s model card for Parakeet TDT 0.6B v2 does not list speaker diarization among its capabilities.
The Whisper paper records a related quirk. Early models tended to write plausible but almost always incorrect guesses for the names of speakers, because many training transcripts included the name of the person speaking. The authors removed the behavior by fine-tuning on transcripts without speaker annotations. If a transcript from a recognition model alone contains speaker names, do not trust them.
How diarization accuracy is measured
The standard metric is diarization error rate (DER), given as a percentage where lower is better. It adds up three kinds of mistake: false alarm, where speech is reported that was not there, missed detection, where real speech is not attributed to anyone, and speaker confusion, where speech is attributed to the wrong person.
As with word error rate, the scoring rules change the number. Some evaluations forgive a short collar of time around each speaker change, and some skip overlapping speech. The pyannote benchmarks state that they use no forgiveness collar and do not skip overlapping speech, which is the stricter setup. Check these two choices before you compare DER figures from different sources.
Questions people ask.
What does diarization mean in transcription?
Labeling a transcript by who was speaking. A diarization model detects voice changes and groups segments by speaker, so a meeting transcript reads as a conversation rather than a single block.
Does dictation software do speaker labeling?
Almost never, because dictation has one speaker. Speaker labels are a feature of meeting and file transcription tools. If you need them, choose a transcriber; if you need live dictation, a dictation app.
Does Whisper do speaker diarization?
No. Whisper transcribes words, and its model card says it has not been robustly evaluated on speaker diarization. To get speaker labels you run a separate diarization model, such as the open-source pyannote.audio pipeline, and combine its timestamps with the transcript.
What is diarization error rate?
Diarization error rate (DER) is the standard accuracy measure for speaker labeling, where lower is better. It combines false alarms, missed speech, and speech attributed to the wrong speaker.
Can speaker diarization run offline on a Mac?
Yes. The open-source pyannote.audio community pipeline runs locally on your computer, and once it has been downloaded it can be used without an internet connection.
