What Is a Local LLM? Understanding Quantization and GGUF, and Calculating Memory Before Running AI on Your Own Machine with Ollama or llama.cpp

Imagine you want AI to summarize a customer's contract, but company policy says those files must not leave the network. Or you simply want to experiment with code that calls an LLM without worrying about an API bill every time you hit run. The questions that follow are almost the same for everyone: can I run AI on my own machine, how powerful does the machine need to be, and what do I give up?

The answer is yes, and it is called a Local LLM. Today Superdev Academy walks through what a Local LLM is, what Quantization and GGUF mean in a model's file name, how to roughly calculate the memory a model needs, how Ollama and llama.cpp differ, and what security cautions apply. This is not a step-by-step installation guide, because we already have installation articles; the links are further down.

What is a Local LLM?

An LLM (Large Language Model) is the large language model behind the chatbots we use every day. Normally we call it over the internet from a provider (a cloud API). A Local LLM means downloading the model file onto your own machine and running the computation there, so what you type does not have to travel anywhere.

For example, the Ollama documentation, for a popular tool in this space, states: "Ollama runs locally. We don't see your prompts or data when you run locally." The same documentation also says Ollama offers cloud models, and that you can turn the cloud features off with the environment variable OLLAMA_NO_CLOUD=1 or by setting "disable_ollama_cloud": true in ~/.ollama/server.json. This matters for anyone who needs data to really stay on the machine: make sure you are running only local models.

What is Quantization and why should you know it?

A language model consists of an enormous number of values called weights, each originally stored at high precision such as 16 bits. A model with 7 billion weights (7B) therefore needs tens of gigabytes of memory. Quantization is a technique that reduces the number of bits used to store each weight, for example from 16 bits to 8 or 4, so the file is smaller and uses less memory, in exchange for a possible small loss of accuracy depending on how hard you compress.

The llama.cpp README says it supports integer quantization at 1.5, 2, 3, 4, 5, 6 and 8 bits for faster inference and lower memory use. The Hugging Face documentation that describes the quantization types in GGUF files lists the average bits per weight for some of them: Q4_K at 4.5 bits per weight, Q5_K at 5.5 bits and Q6_K at 6.5625 bits. These figures are slightly higher than the number in the name because each block also stores a scale value.

What is GGUF?

GGUF is a binary model file format designed for fast loading and saving for inference. According to the Hugging Face documentation, it was developed by ggerganov, who also develops llama.cpp, and unlike formats that store only tensors, such as safetensors, GGUF keeps both the tensors and standard metadata in one file. When you see a file name like ...Q4_K_M.gguf on a model page, the Q4_K part tells you the quantization type, and the .gguf extension tells you it can be opened by tools such as llama.cpp, LM Studio, GPT4All or Ollama, which is the list of tools the Hugging Face documentation names as supporting GGUF.

How much memory do you need? A simple formula you can run yourself

The most common question is whether your machine can run a model of a given size. The rough way our team estimates it is to multiply the number of weights by the bits per weight and divide by 8 to get bytes. The numbers below come from this formula, not from measurements on a real machine, and they count only the size of the weights. They do not include context memory (the KV cache) or program overhead, so real usage will be higher.

• 7B model at 16 bits: 7 billion times 16 divided by 8 gives about 14 GB

• 7B model at 8 bits: about 7 GB

• 7B model at Q4_K (4.5 bits per weight): 7 billion times 4.5 divided by 8 gives about 3.9 GB

• 13B model at Q4_K: about 7.3 GB

• 70B model at Q4_K: about 39.4 GB

One more thing to remember: according to the Ollama FAQ, the default context window is 4096 tokens, adjustable with the OLLAMA_CONTEXT_LENGTH variable or with /set parameter num_ctx inside ollama run. The longer the context you set, the more memory is used. So if your machine has exactly the memory in the figures above, it will not be enough. In practice, always leave more headroom than the calculated number.

On a Mac with Apple Silicon, where the CPU and GPU share one pool of memory, the llama.cpp README lists Metal as supported, so it runs on such machines directly. On a Windows or Linux machine with a discrete graphics card, the amount of VRAM is often the first ceiling you hit. The README lists several supported backends, such as CUDA, HIP, Vulkan and SYCL.

How do Ollama and llama.cpp differ?

Both are open source projects under the MIT license, according to their project pages, but they sit at different levels of convenience.

llama.cpp is an inference engine written in plain C/C++ with no dependencies. Its goal, per the README, is to "enable LLM (and VLM) inference with minimal setup and state-of-the-art performance" on a wide range of hardware, and it includes a server with an OpenAI-compatible API (llama-server). It suits people who want control over the details and want to pick the GGUF file type themselves.

Ollama wraps model download, running and an API into one easy command. The example in the project README is ollama run gemma4, which runs and opens a chat with that model, and the REST API example on the same page points to http://localhost:11434/api/chat. According to the FAQ, models are stored in ~/.ollama/models on macOS and stay in memory for 5 minutes by default. To free memory right away, use ollama stop followed by the model name.

Our team's view is that if you are just starting out, Ollama is the most direct entry point, and llama.cpp is the option when you want to dig deeper or tune more (the team's opinion, not a test result). If you want to install step by step, we already have Using Ollama to Run Private AI for Programmers and Calling Ollama from Go.

Security cautions

Running an LLM on your own machine is not automatically safe. There are three things to check before real use.

First, who can call your server. The Ollama FAQ says that by default Ollama binds to 127.0.0.1 on port 11434, accepting connections only from the same machine, and that setting OLLAMA_HOST="0.0.0.0:11434" makes it reachable from outside. The documentation we read explains how to open the port but does not mention authentication, so we should not assume there is any. If you must share it with a team, place it behind a firewall or a reverse proxy that controls access itself.

Second, where the model file came from. A GGUF file downloaded from anywhere is a multi-gigabyte file whose contents you cannot judge by eye. Download from publishers you trust, and read the license on each model's page, because a model's license is separate from the MIT license of Ollama or llama.cpp and differs from model to model, including on commercial use.

Third, installation. The curl ... | sh install script that the Ollama README suggests is a pattern common to many tools, but before running any script on a machine that holds important data, read what it will do, or use the direct-download installer the website offers. As for the code you write to call the model, follow the same principles as Secure Coding, just as with any other API call.

Is a Local LLM worth it compared with the cloud?

There is no single answer that fits every job. The clear advantages are that data stays on your machine, there is no per-call API fee, and it works offline. The limits are that models a personal machine can run are generally smaller than the frontier models providers run on clusters, so answer quality may not match on complex tasks (a general observation, not a result from tests in this article). You also take on the work of maintaining the machine, updating models and managing memory yourself.

Our advice is to decide from real work. Take ten sample documents or questions from your own job, try them on a local model and a cloud model side by side, and see whether the answers are usable. Results from your own work tell you more than anyone's comparison table. And remember that a Local LLM helps with where the data lives, but it does not decide for you whether processing customer data is permitted by law or by your organization's policy; that must be checked directly with whoever is responsible for it.

Summary: calculate memory before downloading a big file

A Local LLM means running a language model on your own machine. Quantization shrinks a model by lowering the bits per weight. GGUF is a file format that keeps tensors and metadata in one file. And Ollama and llama.cpp are two tools that open these files, the first focused on convenience and the second on control.

Before you click download on a multi-gigabyte file, try the formula, number of weights times bits per weight divided by eight, against your machine's memory, and add headroom for context. If the number is close to the ceiling, choose a smaller model or a heavier quantization instead of waiting for the machine to freeze halfway.

🎯 Follow dev knowledge from Superdev Academy:

• 🔵 Facebook: Superdev Academy Thailand

• 🎬 YouTube: Superdev Academy Channel

• 📸 Instagram: @superdevacademy

• 🎬 TikTok: @superdevacademy