Run Your LLM

Yes. Open-weight models such as Llama, Mistral, Qwen and Gemma can be downloaded and run locally with no per-token fee, and tools like Ollama, LM Studio, Jan and GPT4All make it a one-click setup on Windows, macOS and Linux. The software is free; your real costs are the hardware you already own and the electricity. A modern laptop with 16 GB of memory can run small models of around 7 to 8 billion parameters comfortably, while larger models need more memory or a dedicated GPU. Quality of small local models is below the best cloud models, but it is good enough for drafting, summarizing, coding help and private document work. Browse the tools on this page to find one that fits.

The main constraint is memory. A model must fit in GPU memory (VRAM) or, on Apple Silicon, in unified memory, and quantized models need roughly 0.5 to 0.7 GB per billion parameters. That means an 8B model runs in about 5 to 6 GB, a 14B model in around 10 GB and a 70B model needs 40 GB or more. Macs with 16 to 64 GB of unified memory handle this well, and on PCs an NVIDIA GPU with 12 to 24 GB is the common choice. Models can also run on the CPU with ordinary RAM, only more slowly. Fast storage helps with loading. Start with a small model and move up only if the answers are not good enough.

Quantization shrinks a model by storing its numbers at lower precision, for example 4 or 8 bits instead of 16. The file becomes two to four times smaller, so a model that would not fit in your memory suddenly does, and it often runs faster too. The cost is a small loss of accuracy. At 8 bits the difference is usually negligible, and at 4 bits it is often barely noticeable for chat and writing, although it can show up in precise math, code and long reasoning chains. Below 4 bits, quality drops more quickly. Formats you will see include GGUF, GPTQ and AWQ. A larger model at 4 bits usually beats a smaller model at full precision in the same amount of memory, so prefer that trade-off.

It depends on volume. For occasional use, the local model costs nothing extra if you already own capable hardware, while an API charges for every token. At high, steady volume a dedicated GPU server can undercut API prices, but you also pay for the machine, power, maintenance and your own time. As a rough rule, light and private workloads favor local, and spiky or low-volume workloads favor an API because you pay nothing when idle. Cheap hosted open models have also narrowed the gap, since they cost only cents per million tokens. Run your own numbers with the price comparison, and read our guide to local LLM versus cloud API for the break-even math.

Running a model locally means your prompts and documents stay on your machine and are not sent to any provider, which is the main reason people choose it for contracts, medical notes, source code or internal files. The model itself does not phone home, and you can run it fully offline. It is not automatically secure, though: you still need to protect the computer, download models from trustworthy sources such as the publisher's official repository, and check the model license. Be careful with tools and plugins that give a model access to your files or network, since those can leak data too. For regulated data, confirm with your compliance team that your setup meets the requirements.