Local LLMon-device language model
A local LLM is a language model that runs on your own computer rather than on a vendor’s server. In dictation tools it is the model behind the formatting pass: fixing punctuation, removing fillers, and matching tone, without the transcript ever leaving the machine.
The formatting pass is the step most likely to be outsourced to the cloud, because it is the step that wants a language model, and language models were until recently too big to run on a laptop. Small models changed that. A small model, run through the on-device frameworks Apple silicon provides, can clean up a sentence in a few hundred milliseconds.
Local is the right home for this step for the same reason it is the right home for recognition: the model sees your words, and your words should not become a vendor’s data on the way to being punctuated.
What the model is asked to do
Less than you would think, in a well-designed tool. Not to rewrite, not to improve, not to decide clock times or spellings, all of which are rules. It is asked for judgment: which fillers were filler, where the sentences break, what tone the destination wants. A narrow brief is what lets a small model do the job well, and it is also what makes the output checkable, because a model that was never allowed to add or remove content words can be tested for having done so.
Speed, memory, and heat
A local model competes with your other apps for memory and with your battery for power. The engineering that matters is keeping it small, keeping it resident so there is no cold start, capping how much memory its runtime may hold, and giving it a time budget so a slow answer is dropped rather than waited for. On an M-series Mac the whole pass fits inside a second with room to spare.
Cloud LLM formatting, for comparison
Sending the transcript to a hosted model buys a bigger model and costs a round trip, a subscription to pay for the inference, and a copy of every sentence you dictate on someone else’s server. For prose formatting the bigger model is not a meaningful advantage; the task is small. The round trip and the copy are real costs, every time.
In Flit
Flit’s formatting model runs on your Mac, with a strict brief, a content check on its output, and a time budget.
How Flit writes →Questions
Fair questions.
Does dictation software need an LLM?
For raw transcription, no. For turning a transcript into finished, tone-appropriate writing, a small language model helps a great deal, and it can run locally on Apple silicon. The best tools give it only the judgment calls and handle everything exact by rule.
Can a language model run on my Mac without the cloud?
Yes. Small models run on Apple silicon through on-device frameworks, and recent versions of macOS include models that apps can call locally. Both are fast enough to format a sentence in well under a second.
Will a local LLM slow down my Mac?
A well-behaved one will not. It should stay small, cap its memory, and run on the GPU while the rest of the machine carries on. Badly managed runtimes that let their cache grow without limit are the ones people notice.
Related terms
