Whisper

Tiny and base are fast and rough, fine for a voice memo. Small is the usable floor for dictation. Medium and large are accurate enough for professional work and, on a modern Mac, still fast. Large-v3-turbo is the current sweet spot: a large model with a slimmed-down decoder that keeps most of the accuracy at several times the speed. Most Mac tools that offer a choice are offering these, and the right answer for dictation is the largest model that returns text before you get impatient.

Whisper reads a window of audio and then writes the transcript for it. That makes it accurate, because it sees the whole phrase before deciding, and it makes it batch: nothing appears until you stop. Tools built only on Whisper feel like a pause after every sentence. Tools that stream pair it with a transducer model for live words and keep Whisper for the languages that model cannot handle.

Whisper hallucinates on silence and on very short clips, producing plausible phrases nobody said, and it can repeat itself on long audio. Both are well documented and well mitigated: trim silence before the model runs, keep chunks short, and treat any output on non-speech audio as suspect. Numbers and proper nouns follow the usual ASR pattern of being most-probable spellings rather than yours.

OpenAI publishes Whisper as a Python package with a command-line tool. The project says the code is expected to work with Python 3.8 to 3.11, and it needs the ffmpeg tool to read audio files. These steps assume you already have Python and Homebrew.

1. 01 Open Terminal and install ffmpeg with: brew install ffmpeg
2. 02 Install Whisper with: pip install -U openai-whisper
3. 03 Transcribe a file with: whisper audio.mp3 --model turbo
4. 04 Wait while the model downloads on the first run. It is saved in the .cache/whisper folder in your home folder and reused after that.
5. 05 Find the transcript in the current folder. By default Whisper writes txt, vtt, srt, tsv, and json files.
6. 06 Add --language followed by a language name, for example --language Japanese, if you want to skip automatic detection.
7. 07 Run whisper --help to see every option.

OpenAI’s table lists the large model at 1550 million parameters, about 10 GB of video memory, and a relative speed of 1x. The turbo model is 809 million parameters, about 6 GB, and about 8x. OpenAI describes turbo as an optimized version of large-v3 with a minimal degradation in accuracy, and it is the default model in the command-line tool.

Two caveats come with those figures. The speeds were measured on an NVIDIA A100 graphics card transcribing English, and OpenAI says real-world speed may vary significantly with the language, the speaking speed, and the hardware. And turbo is not trained for translation, so it returns the original language even when you ask it to translate.

• tiny: 39 million parameters, about 1 GB, about 10x.
• base: 74 million parameters, about 1 GB, about 7x.
• small: 244 million parameters, about 2 GB, about 4x.
• medium: 769 million parameters, about 5 GB, about 2x.
• large: 1550 million parameters, about 10 GB, 1x.
• turbo: 809 million parameters, about 6 GB, about 8x.

Whisper was trained on two jobs: transcribing speech in its own language, and translating speech in other languages into English. Of its 680,000 training hours, about 438,000 are English audio with English transcripts, about 126,000 are non-English audio with English transcripts, and about 117,000 are non-English audio with a transcript in the same language, across 98 languages.

That is why translation only goes one way. Add --task translate to get English text from non-English speech, and use the medium or large model for it, as OpenAI recommends. Whisper does not translate English speech into other languages.

The tiny, base, small, and medium sizes also come in English-only versions with names ending in .en. OpenAI says these tend to perform better for English, especially tiny.en and base.en, and that the difference becomes less significant for small.en and medium.en.