Every time you paste a contract, a medical report or a company spreadsheet into an online chatbot, that data travels to someone else’s servers. For many uses that’s fine; for others it really isn’t. There’s a third way, much less complicated than people think: run the model directly on your computer. No account, no subscription, and the text never leaves your desk.
Here’s what you need, which programs to use and, above all, what to expect, because a local model is not ChatGPT or Claude in miniature.
The number that matters: memory
More than the processor, what decides which models you can run is memory: the graphics card’s memory (VRAM) if you have a dedicated GPU, otherwise system RAM. Macs with Apple Silicon share memory between CPU and GPU, which makes them surprisingly well suited to the job.
A model’s size is measured in billions of parameters (the “B” in names like 8B, 14B, 70B). The programs below usually download models quantised to 4 bits, a compressed form that saves a lot of space with a limited loss of quality. As a rough guide:
| Model size | Memory used (approx.) | What computer you need |
|---|---|---|
| 3-4B | 2-3 GB | almost any recent PC with 8 GB of RAM |
| 7-8B | 5 GB | 16 GB of RAM, or a graphics card with 8 GB |
| 12-14B | 8-9 GB | a 12 GB graphics card or a Mac with 16-24 GB |
| 30-32B | 18-20 GB | a 24 GB graphics card or a Mac with 32 GB or more |
| 70B | 40 GB or more | a workstation or a Mac with 64 GB or more |
Add some headroom for the operating system and for the “context”, the text of the conversation. A model that fits entirely in GPU memory answers quickly; if it spills into normal RAM it still works, but much more slowly. Without a GPU, on the processor alone, small models are usable and medium ones become slow.
The easy route: LM Studio
LM Studio is where I’d start if you don’t want to touch the terminal. It’s free for personal use and available for Windows, Mac and Linux.
- download it from the official site, lmstudio.ai, and install it;
- open the search section (the magnifying glass) and look for a model family, such as Qwen, Gemma, Llama or Mistral;
- next to each file LM Studio tells you whether it suits your computer: if possible, pick one that says it can fit entirely on the GPU;
- once downloaded, open the chat tab, select the model at the top and start typing.
A very handy feature is attaching documents to the chat: the program reads them locally and the model answers based on their content.
The tinkerer’s route: Ollama
Ollama is built for the command line and is perfect if you want to plug the model into other programs. After installing it from ollama.com, open a terminal and use a few commands:
ollama run model-namedownloads the model (the first time only) and opens a chat. For exampleollama run qwen3:8b; current names are in the library on the website;- type
/byeto leave the chat; ollama listshows downloaded models and how much space they take;ollama rm model-namedeletes them, useful because each one weighs several gigabytes.
The interesting part is that Ollama also exposes a local API on port 11434. Many programs, from coding extensions to web interfaces like Open WebUI and even Home Assistant for voice control, can connect to it and use your model instead of a cloud service.
What to expect (and what not to)
- An 8B model is not a frontier model. It summarises well, corrects text, translates decently and writes simple code. On long reasoning, maths and general knowledge it gets things wrong far more often than the big online services, and hallucinations are frequent.
- It doesn’t know what’s happening today. It only knows its training data and, out of the box, doesn’t browse the web.
- Quality in languages other than English varies a lot. Try two or three model families on the same task before choosing.
- Context is limited. Many programs set a cautious context window by default to save memory: if the model “forgets” the start of a long document, look for the context length setting and raise it, knowing it will use more memory.
Two security precautions
Only download models from the libraries built into these programs or from official Hugging Face pages, and prefer the GGUF format, which contains data only, not executable code. Also check the licence: not every “open” model can be freely used commercially.
My advice to get started: install LM Studio, download a 7-8B model and give it a real task you currently do with an online chatbot, such as summarising meeting minutes or rewriting an email. Within half an hour you’ll know whether a local AI is enough for you or whether, for now, you still need the cloud.