# Local LLM Canonical page: https://flit.fyi/glossary/local-llm [Glossary](https://flit.fyi/glossary) / [Where it runs](https://flit.fyi/glossary#privacy) Also: on-device language model Updated October 4, 2026 Definition A local LLM is a language model that runs on your own computer rather than on a vendor’s server. In dictation tools it is the model behind the formatting pass: fixing punctuation, removing fillers, and matching tone, without the transcript ever leaving the machine. The formatting pass is the step most likely to be outsourced to the cloud, because it is the step that wants a language model, and language models were until recently too big to run on a laptop. Small models changed that. A small model, run through the on-device frameworks Apple silicon provides, can clean up a sentence in a few hundred milliseconds. Local is the right home for this step for the same reason it is the right home for recognition: the model sees your words, and your words should not become a vendor’s data on the way to being punctuated. ## What the model is asked to do Less than you would think, in a well-designed tool. Not to rewrite, not to improve, not to decide clock times or spellings, all of which are rules. It is asked for judgment: which fillers were filler, where the sentences break, what tone the destination wants. A narrow brief is what lets a small model do the job well, and it is also what makes the output checkable, because a model that was never allowed to add or remove content words can be tested for having done so. ## Speed, memory, and heat A local model competes with your other apps for memory and with your battery for power. The engineering that matters is keeping it small, keeping it resident so there is no cold start, capping how much memory its runtime may hold, and giving it a time budget so a slow answer is dropped rather than waited for. On an M-series Mac the whole pass fits inside a second with room to spare. ## Cloud LLM formatting, for comparison Sending the transcript to a hosted model buys a bigger model and costs a round trip, a subscription to pay for the inference, and a copy of every sentence you dictate on someone else’s server. For prose formatting the bigger model is not a meaningful advantage; the task is small. The round trip and the copy are real costs, every time. ## How to check whether your Mac can run a local model Two things decide it: the chip and the free memory. Both take a minute to check, and neither needs any extra software. 1. Choose Apple menu → About This Mac. 2. Look for an item labeled Chip. On a Mac with Apple silicon it is followed by the name of the chip. Note the Memory figure as well. 3. Open the Activity Monitor app and click Memory. 4. With your usual apps open, look at the Memory Pressure graph at the bottom of the window. It shows how efficiently memory is serving what you are running. 5. Check Swap Used. This is the space on your startup disk being used to swap unused files to and from RAM. 6. If pressure is already high and swap is in use before you add a model, a local model will compete with your apps for memory. ## Why does a local LLM use so much memory? A language model is a large file of numbers, and all of it has to be held in memory while the model runs. Nothing is fetched from a server on demand. The memory a model needs grows with the size of the model, which is why small models are the practical choice for a task that runs many times a day. Apple silicon has one feature that helps here. MLX, the machine learning framework from Apple machine learning research, describes a unified memory model in which arrays live in shared memory and operations can run on any supported device type without transferring data. In plain terms, the model does not have to be copied between separate pools of memory for the processor and the graphics hardware. ## Local LLM vs speech model: what is the difference? They do different jobs, and a dictation tool that formats text uses both. A speech model turns audio into words. A language model takes those words and decides how they should be written. Whisper and Parakeet are speech models, not LLMs. The sizes are public. OpenAI lists Whisper from tiny at 39 million parameters to large at 1550 million, with a turbo model at 809 million that it describes as an optimized version of large-v3. NVIDIA lists Parakeet TDT 0.6B v2 at 600 million parameters, for English. The two kinds of model overlap a little. The Parakeet model card includes automatic punctuation and capitalization in the transcript, so some formatting arrives before any language model sees the text. The language model is only needed for the judgment that is left. In Flit Flit’s formatting model runs on your Mac, with a strict brief, a content check on its output, and a time budget. [How Flit writes](https://flit.fyi/features#writes) - [The pass that runs on your Mac](https://flit.fyi/features#writes) - [What the model is allowed to decide](https://flit.fyi/#exact) ## Questions people ask. ### Does dictation software need an LLM? For raw transcription, no. For turning a transcript into finished, tone-appropriate writing, a small language model helps a great deal, and it can run locally on Apple silicon. The best tools give it only the judgment calls and handle everything exact by rule. ### Can a language model run on my Mac without the cloud? Yes. Small models run on Apple silicon through on-device frameworks, and recent versions of macOS include models that apps can call locally. Both are fast enough to format a sentence in well under a second. ### Will a local LLM slow down my Mac? A well-behaved one will not. It should stay small, cap its memory, and run on the GPU while the rest of the machine carries on. Badly managed runtimes that let their cache grow without limit are the ones people notice. ### How do I know if my Mac has Apple silicon? Choose Apple menu → About This Mac. A Mac with Apple silicon shows an item labeled Chip, followed by the name of the chip. ### Is Whisper an LLM? No. Whisper is a speech recognition model that turns audio into text. An LLM works on text that already exists, which in dictation means cleaning up the transcript. ### What is MLX on a Mac? MLX is an array framework for machine learning on Apple silicon, published by Apple machine learning research. It is one of the ways software runs models locally on a Mac. ## Guides that use it - [Superwhisper not working](https://flit.fyi/superwhisper-not-working) ## Related terms - [Formatting pass](https://flit.fyi/glossary/formatting-pass) - [On-device processing](https://flit.fyi/glossary/on-device-processing) - [Neural Engine](https://flit.fyi/glossary/neural-engine) - [Hallucination](https://flit.fyi/glossary/hallucination) ## Your voice was always faster. 7 days free, then $19 USD once. No account, no subscription, nothing uploaded. [Download for Mac](https://flit.fyi/download) [Buy now · $19 USD](https://flit.fyi/pricing#personal) macOS 14 or later · Apple silicon