Quick Overview
GGUF is the file format behind most open-weight AI models — Llama, Mistral, Qwen, DeepSeek, and thousands more. Running these models locally gives you complete privacy (nothing leaves your computer), works without internet, and zero API costs, forever. With one of the four free tools below, you can go from zero to chatting with your own local AI in under 10 minutes.
All four run the same inference engine under the hood, so the same .gguf file works in every one of them — pick whichever matches your comfort level. Never touched a terminal? Use GGUF Loader or LM Studio. Building an app or automating workflows? Ollama gives you a local OpenAI-style API. Want zero abstraction and day-one features? Go straight to llama.cpp, the engine everything else is built on.
- Computer with 8GB+ RAM (16GB recommended)
- 10GB free storage space
- Windows, Mac, or Linux
- No GPU required (but helps if you have one)
- No Python, no programming knowledge for the GUI tools
Choose Your Tool
All four tools are free and run GGUF models on your own hardware. The difference is the interface and the extras around it:
| Tool | Interface | Best For | Difficulty | Price | Local API | Platforms |
|---|---|---|---|---|---|---|
| GGUF Loader | Desktop GUI | Beginners & everyday chat | ⭐ Easy | Free & open source | — | Windows, macOS, Linux |
| Ollama | Command line + background server | Developers, automation, servers | ⭐⭐ Medium | Free & open source | REST + OpenAI-compatible :11434 |
Windows, macOS, Linux, Docker |
| LM Studio | Desktop GUI | Power users, model tinkering, RAG | ⭐ Easy | Free | OpenAI-compatible :1234 |
Windows, macOS, Linux |
| llama.cpp | Command line | Advanced users, custom builds, edge devices | ⭐⭐⭐ Advanced | Free & open source | OpenAI-compatible via llama-server |
Windows, macOS, Linux, and more |
GGUF Loader
Point, drag, chat — no terminal, no Python, no setup wizards.
GGUF Loader is a free, open-source desktop app that puts a simple point-and-click interface around local GGUF models. You install a small executable, drag in a .gguf file, and chat — there is no Python to install, no environment to configure, and no account to create. It supports any GGUF model that fits your PC's resources, from small 3B models to large 70B+ models.
This is the tool we recommend for anyone who has never run a local model before, or who just wants a private offline chat assistant without any friction.
1 Download & Install GGUF Loader
Pick the build that matches your system. The installer is a single file — run it and you're done. No dependencies, no extra runtime.
Looking for an older version or a build not listed here? Browse all releases on GitHub. And on Windows, if SmartScreen warns you about an unrecognized app (normal for new open-source releases), click More info → Run anyway.
2 Download a GGUF Model
GGUF Loader loads any standard .gguf file, so you can grab any of the thousands of models on HuggingFace. For your first model, keep it small:
-
Llama 3.2 1B — best first model (~1.5GB RAM)
📥 Download from HuggingFace -
Qwen 2.5 1.5B — great all-rounder (~2GB RAM)
📥 Download from HuggingFace -
Mistral 7B — stronger, needs 8GB+ RAM
📥 Download from HuggingFace
Q4_K_M.gguf — this quantization offers the best balance of quality and speed on typical hardware. See recommended models below for more options.
3 Load the Model & Start Chatting
- Open GGUF Loader
- Click "Load Model" and browse to your downloaded
.gguffile — or simply drag the file onto the app window - Wait for the first load (usually 10–30 seconds — the model is being copied from disk into RAM; later loads are faster)
- Type your first message and press Enter — your AI now runs 100% offline on your own computer
That's the entire setup. No servers, no accounts, no API keys — the model runs entirely on your hardware.
4 Tune Settings & Try Agentic Mode (Optional)
Once a model is loaded, a settings panel lets you shape how it behaves:
- Context size — how much of the conversation the model remembers. Bigger = smarter long chats, but uses more RAM. 4K–8K is a good default.
- Generation settings — creativity vs. predictability of responses.
- GPU offload (GPU build only) — how many model layers run on your graphics card.
For coding or multi-step tasks, switch on Agentic Mode: the model plans a step-by-step approach and executes it with sandboxed tools — reading, editing, and searching files, and running commands — turning your local model into an autonomous assistant.
✓ What we like
- Zero learning curve — install, drag, chat
- Tiny download (CPU build is ~145 MB)
- Drag & drop any GGUF file, no conversion
- Agentic Mode for autonomous coding tasks
- Open source and completely free
✗ Keep in mind
- No built-in model catalog — you download models from HuggingFace yourself
- Fewer power-user knobs than LM Studio
- GPU build is a larger download (~930 MB)
Having trouble? Jump to Troubleshooting → GGUF Loader.
Ollama
One command to download, run, and serve models — with a local API included.
Ollama is an open-source model manager that has become the standard way to run LLMs locally in professional setups. It downloads and stores GGUF-backed models for you, lets you chat with them in the terminal, and quietly runs a local API server on localhost:11434 so your own apps, scripts, and editors can talk to the model. If you're building anything that uses an AI model — a chatbot, a document summarizer, a coding helper — Ollama is the tool for you.
It uses simple one-line commands. You'll need a terminal: Windows (PowerShell or CMD), macOS (Terminal), or Linux (any shell). Every command below is copy-paste ready.
1 Install Ollama
Download the installer for your system from ollama.com/download, or use the one-liner on Linux:
$ curl -fsSL https://ollama.com/install.sh | sh- Windows: run
OllamaSetup.exe— installs in ~1 minute and starts the background server automatically - macOS: open the
.dmgand drag Ollama into Applications (works on both Apple Silicon and Intel) - Linux: run the command above, which installs the
ollamaCLI and registers it as a service - Docker (any OS):
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
Verify the installation:
$ ollama --version
ollama is running2 Run Your First Model
One command downloads the model and drops you into a chat — Llama 3.2 1B is about 1.3GB and loads instantly on most machines:
$ ollama run llama3.2:1b
pulling manifest... downloading... 100%
>>> Ask me anything — I'm running on your computer now!
>>> Explain quantization in one sentence
Quantization shrinks a model's numbers to fewer bits so it uses less
memory and runs faster, at a small cost in accuracy.
>>> /byeInside the chat, useful commands are /bye (exit), /clear (reset the conversation), /show info (model details), and /set parameter temperature 0.3 (tune generation). Browse thousands of ready-to-run models at ollama.com/library — Mistral, Qwen, DeepSeek, Phi, Gemma, and more.
3 Pull GGUF Models Straight from HuggingFace
Ollama can download GGUF models directly from any HuggingFace repository — no Modelfile needed. Point it at a repo and pick the quantization tag:
$ ollama pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
success
$ ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_MThe pattern is hf.co/<user>/<repo>:<quantization> — the same repos you'd browse manually on HuggingFace, served straight into your local model library. This works with virtually every popular GGUF upload.
4 Load Any Local .gguf File (Modelfile)
Have a GGUF file that isn't on a server — one you downloaded manually, quantized yourself, or received from someone? Import it with a tiny Modelfile. Create a folder, put your .gguf inside it, and create a plain-text file named Modelfile (no extension) with this content:
FROM ./qwen2.5-1.5b-instruct-q4_k_m.gguf
# optional settings
PARAMETER temperature 0.7
PARAMETER num_ctx 4096Then register it with Ollama and run it by name:
$ ollama create my-qwen -f Modelfile
transferring model data... success
$ ollama run my-qwenFROM line. If replies look oddly raw or unformatted, the template is missing — add a TEMPLATE block copied from the model's HuggingFace page. Run ollama show my-qwen to see exactly what Ollama detected.
Where Ollama stores models: ~/.ollama/models on macOS and Linux, C:\Users\<you>\.ollama\models on Windows. You can move this folder anywhere by setting the OLLAMA_MODELS environment variable — handy for keeping big models on an external SSD.
5 Use the Local API (The Best Part)
While Ollama is running, it serves a REST API on http://localhost:11434 — your apps can talk to your models with plain HTTP. On Windows and macOS the server starts automatically with the app; on Linux run ollama serve in a separate terminal. Loaded models unload themselves after ~5 minutes of inactivity.
$ curl http://localhost:11434/api/chat -d '{
"model": "llama3.2:1b",
"messages": [{"role": "user", "content": "Why is local AI private by design?"}]
}'Ollama also speaks the OpenAI API format at http://localhost:11434/v1 — so most OpenAI-based apps, SDKs, and tools work by changing a single URL. For example, in Python:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="llama3.2:1b",
messages=[{"role": "user", "content": "Hello, local world!"}]
)
print(response.choices[0].message.content)Command Cheat Sheet
| Command | What it does |
|---|---|
ollama run llama3.2:1b | Download (if needed) and start chatting |
ollama pull <model> | Download a model without running it |
ollama list | Show all downloaded models and sizes |
ollama ps | Show models currently loaded in memory |
ollama rm <model> | Delete a model and free disk space |
ollama show <model> | Inspect a model's template and parameters |
ollama create <name> -f Modelfile | Import a custom .gguf file |
ollama serve | Start the API server (Linux / headless) |
✓ What we like
- One command manages download, run, update, and delete
- Local REST + OpenAI-compatible API out of the box
- Huge curated library, direct HuggingFace pulls
- Tiny footprint, runs great on servers and Docker
- Works with popular desktop GUIs built on top of it
✗ Keep in mind
- Terminal-first — no official graphical interface
- Loading a raw .gguf requires writing a small Modelfile
- Server settings (host, port) are configured via environment variables
Server won't start? Jump to Troubleshooting → Ollama.
LM Studio
A full desktop app: search, download, chat, and serve models — all with mouse clicks.
LM Studio is a polished desktop application that turns the whole local-AI workflow into a GUI experience: you search HuggingFace from inside the app, download quantized models with a glance at how well they'll fit your RAM, chat with a clean ChatGPT-style interface, and — when you need it — flip on a local OpenAI-compatible server with one toggle. It's the middle ground between GGUF Loader's simplicity and Ollama's power.
It's especially loved by tinkerers: runtime settings like GPU offload, context length, and system prompts are all adjustable per conversation, and it keeps your chat history organized.
1 Download & Install LM Studio
Head to lmstudio.ai and grab the installer for your platform — Windows, macOS (Apple Silicon or Intel), or Linux. Install it like any desktop app and launch it; the first start takes a moment while it sets up its inference engine.
2 Find & Download a Model (No Browser Needed)
LM Studio has HuggingFace built in. Click the search icon (magnifying glass) in the left sidebar, then type a model name:
- Search for
Llama 3.2 1BorQwen 2.5 1.5B - Pick a well-maintained GGUF upload (uploaders like bartowski and lmstudio-community are reliable choices)
- Choose a quantization — LM Studio shows the file size for each and highlights which ones fit your RAM.
Q4_K_Mis the usual sweet spot - Click Download and watch the progress bar — models land in your local library automatically
3 Load the Model & Chat
- Click the chat icon in the left sidebar
- Select your freshly downloaded model from the dropdown at the top — this opens the model load settings
- Review the settings (you can just accept the defaults for your first chat):
- GPU offload — how many model layers run on your graphics card (Apple Silicon uses Metal automatically; NVIDIA uses CUDA)
- Context length — how much conversation memory the model keeps
- Click Load Model, wait for the progress bar, and start chatting
Chat history is saved automatically, and you can run several models side by side to compare their answers.
4 Load Your Own .gguf File
Already have GGUF files (for example, ones you downloaded for GGUF Loader)? Don't re-download — import them:
- Easiest: drag the
.gguffile from your file manager straight into the LM Studio window — it gets imported into your model library and appears under My Models - Manual: place files in LM Studio's models folder, organized as
<publisher>/<model-name>/<file>.gguf:- macOS / Linux:
~/.lmstudio/models/… - Windows:
C:\Users\<you>\.lmstudio\models\…
- macOS / Linux:
Once imported, the model loads exactly like one downloaded in-app — same chat interface, same settings.
5 Run a Local API Server (Optional)
LM Studio includes an OpenAI-compatible server — perfect when an app expects an OpenAI API key but you want to run everything locally:
- Open the Developer tab (terminal icon) in the left sidebar
- Load a model, then toggle Status: Running — the server starts on
http://localhost:1234 - Point any OpenAI SDK or app at it
$ curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.2-1b-instruct",
"messages": [{"role": "user", "content": "Hello from my own server!"}]
}'base_url to http://localhost:1234/v1 and any non-empty API key. Ports and model names are listed in the Developer tab, and you can require an access key for a bit of local security.
✓ What we like
- Beautiful, genuinely beginner-friendly interface
- Model search + RAM-fit guidance inside the app
- Fine-grained runtime settings per chat
- One-toggle OpenAI-compatible local server
- Imports existing GGUF files via drag & drop
✗ Keep in mind
- Heavier install (~1 GB with runtimes) than GGUF Loader
- Not open source (free, but closed)
- Loads models per-chat — a bit more clicking for quick one-off questions
Model not appearing? Jump to Troubleshooting → LM Studio.
llama.cpp
The engine everything else is built on — run GGUF straight from the source.
llama.cpp is the original open-source project that created the GGUF format — and the inference engine that GGUF Loader, Ollama, and LM Studio all run under the hood. Going straight to llama.cpp means zero abstraction: new models and features land here first, it runs on almost anything (from a Raspberry Pi to multi-GPU servers), and it ships with a surprisingly capable built-in web UI and API server. The trade-off: everything happens in the terminal, and model files are yours to manage.
You don't need to compile anything to get started — prebuilt binaries are attached to every GitHub release. Pick the route that matches your system below.
1 Get llama.cpp
Windows — prebuilt binaries (no compiling): open the GitHub releases page and download the latest bundle for your hardware:
…bin-win-cpu-x64.zip— runs on any modern PC…bin-win-cuda-…-x64.zip— for NVIDIA GPUs…bin-win-vulkan-x64.zip— for AMD/Intel GPUs
Extract the zip anywhere (a folder like C:\llama), then open a terminal in that folder — on Windows 11, right-click the folder and choose Open in Terminal.
macOS — Homebrew (one command):
$ brew install llama.cppLinux / macOS — build from source (~2 minutes):
$ git clone https://github.com/ggml-org/llama.cpp
$ cd llama.cpp
$ cmake -B build
$ cmake --build build --config ReleaseThe binaries land in the build/bin folder (inside build/bin/Release for Xcode or Visual Studio builds). On Apple Silicon, GPU acceleration via Metal is enabled automatically.
2 Chat With a Model (llama-cli)
Point llama-cli at any .gguf file and it drops you into an interactive chat. llama.cpp reads the chat template embedded in the GGUF file automatically, so instruct models work out of the box:
$ ./llama-cli -m ~/Downloads/Llama-3.2-1B-Instruct-Q4_K_M.ggufOn Windows the same call is .\llama-cli.exe -m C:\Users\you\Downloads\model.gguf — identical flags, identical behavior.
Flags worth knowing from day one:
-ngl 99— offload all model layers to your GPU (the single biggest speed boost)-c 8192— set the context window size (the default is small to save RAM)--temp 0.7— creativity of the responses-p "your prompt"— get a single completion instead of a conversation
3 Browser Chat + API (llama-server)
The best-kept secret of llama.cpp: llama-server spins up a built-in web chat interface and an OpenAI-compatible API with one command:
$ ./llama-server -m ~/Downloads/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080
server listening on http://127.0.0.1:8080 — web UI + OpenAI-compatible /v1 API- Open
http://localhost:8080in any browser — a full chat interface, ready to use - The same address serves OpenAI-compatible endpoints:
/v1/chat/completions,/v1/completions, and/v1/embeddings
$ curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "default",
"messages": [{"role": "user", "content": "Hello from llama.cpp!"}]
}'4 Bonus: Quantize Your Own GGUF Models
Here's the superpower no GUI tool offers: llama.cpp can convert any HuggingFace model into GGUF and shrink it to any quantization level, so you can build exactly the size/quality trade-off you want:
$ python3 convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf
$ ./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_MThe conversion script needs Python 3 with the project's requirements installed (pip install -r requirements.txt from the llama.cpp folder). Your fresh .gguf then loads in any tool — including GGUF Loader, Ollama, and LM Studio.
✓ What we like
- New models and features land here first
- Runs on almost anything — CPU, CUDA, Metal, Vulkan, even a Raspberry Pi
- Built-in web UI + OpenAI-compatible server
- Quantize your own custom GGUF files
- MIT licensed, tiny footprint, no dependencies
✗ Keep in mind
- Command line only — no graphical interface
- No model manager: you pass file paths yourself
- Fast-moving project — commands and builds change often
Binary not found or garbled output? Jump to Troubleshooting → llama.cpp.
Recommended Starter Models
These four models cover the whole range from "runs on anything" to "genuinely smart." All links go to trusted HuggingFace GGUF repositories, and every model works in all four tools. Grab the Q4_K_M file unless you know why you need another quantization.
Llama 3.2 1B — the starter
Meta's tiniest instruct model. Surprisingly coherent for its size, great for learning the ropes, drafting, and quick Q&A.
ollama run llama3.2:1bllama-cli -m Llama-3.2-1B-Instruct-Q4_K_M.ggufQwen 2.5 1.5B — the all-rounder
Alibaba's compact workhorse: strong reasoning and multilingual skills for its size, and noticeably sharper than most 1–2B models.
ollama run qwen2.5:1.5bllama-cli -m Qwen2.5-1.5B-Instruct-Q4_K_M.ggufMistral 7B — the classic
A fast, dependable generalist that still beats most newer small models on plain usefulness. The sweet spot if you have 16GB RAM.
ollama run mistral:7bllama-cli -m mistral-7b-instruct-v0.2.Q4_K_M.ggufLlama 3.1 8B — the brainy one
Where local models start feeling "real": strong reasoning, coding, and instruction-following. Needs 16GB RAM to breathe (or a GPU).
ollama run llama3.1:8bllama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.ggufQ2/Q3 = smallest files, noticeably degraded quality. Q4_K_M = the popular sweet spot. Q6/Q8 = near-original quality, double the size. Rule of thumb: your RAM should be roughly file size × 1.3 plus ~1.5GB for the OS and tooling.
Troubleshooting Common Issues
Every tool
Model loads slowly. First load takes longer as the model is copied from disk into RAM; later loads are faster. Keeping models on an SSD instead of a hard drive is the single biggest speed-up.
Out of memory errors. Try a smaller model or a lower quantization (Q4_K_M → Q3_K_M). For 8GB RAM, stay under 3B parameters; with 16GB, 7–8B models work well. Close browser tabs — they are usually the top RAM thief.
Slow response generation. Normal for CPU inference. Smaller models (1–3B) generate noticeably faster, and GPU offload helps if available.
Model won't load at all. Verify the download completed — the file size on your disk should match what HuggingFace lists. Partial downloads (often .gguf.part or an unusually small file) happen on flaky connections; just re-download.
GGUF Loader
Windows blocked the installer. SmartScreen often flags new open-source apps. Click "More info" → "Run anyway." The binaries are published on the official GitHub releases page.
GPU build runs slower than expected. Make sure your NVIDIA drivers are current, and that you actually downloaded the GPU build — the CPU build ignores your graphics card entirely.
Ollama
"connection refused" when calling the API. The background server isn't running. On Windows/macOS, start the Ollama app once; on Linux, run ollama serve. Check it's listening: curl http://localhost:11434 should reply "Ollama is running."
"model not found" (404). Download it first with ollama pull, and check the exact name with ollama list — the API is picky about tags.
Running out of disk. Ollama keeps every pulled model in ~/.ollama/models. Prune old ones with ollama rm <model> or move the folder via the OLLAMA_MODELS environment variable.
LM Studio
An imported .gguf doesn't show in My Models. The folder structure must be <publisher>/<model-name>/<file>.gguf inside ~/.lmstudio/models — a file dumped directly into the models folder is ignored. Drag & drop into the app avoids the problem entirely.
The server says no models are available. The server only serves models that are loaded. Load your model in the Developer tab first, then hit the endpoint again.
Crashes or freezes when loading big models. Lower the GPU offload value or context length in the load settings — both directly change how much memory the model claims.
llama.cpp
"command not found" when running llama-cli. The binary lives in the folder you extracted (or built). Run it with the ./ prefix from that folder — ./llama-cli — or add the folder to your PATH; Homebrew installs are on your PATH automatically.
Garbled or repetitive output. Make sure you're running an instruct model (the name contains "Instruct") and let llama.cpp use the chat template embedded in the file — conversational mode handles this by default. Also verify the download completed fully.
GPU isn't being used. GPU offload must be requested with the -ngl flag (e.g. -ngl 99 for all layers), and you need a matching build: CUDA for NVIDIA, Vulkan for AMD/Intel — while Apple Silicon gets Metal automatically.
Frequently Asked Questions
Which tool should I start with?
If you've never used a terminal: GGUF Loader (fastest setup) or LM Studio (nicest interface). If you're a developer or want to plug local models into your own apps: Ollama, because of its built-in API. If you want raw control, exotic hardware, or day-one model support: llama.cpp. All four are free — try two and keep the one that feels right.
Can I use the same .gguf file in all four tools?
Yes — GGUF is a standard format and all four tools run the same files. GGUF Loader opens the file directly, LM Studio imports it with drag & drop, Ollama wraps it in a one-line Modelfile, and llama.cpp takes the file path directly. One download, four tools.
How do I run a GGUF file with llama.cpp?
Install llama.cpp (prebuilt binaries from GitHub releases, Homebrew on macOS, or build from source), then run llama-cli -m your-model.gguf to chat in the terminal. For a browser chat UI and an OpenAI-compatible API, run llama-server -m your-model.gguf and open http://localhost:8080.
Do I need a GPU?
No. GGUF models are optimized for CPU inference, and models under 7B parameters run smoothly on most modern CPUs with 16GB RAM. A GPU (or Apple Silicon) makes bigger models much faster, but it's optional.
Is all this really free? What's the catch?
GGUF Loader and Ollama are free and open source. LM Studio is free (closed source). The models are open weights downloaded once from HuggingFace. Inference happens on your hardware, so there are no subscriptions, tokens, or API bills — the only cost is electricity.
Can I use these tools completely offline?
Yes, after the one-time downloads of the app and your models. Chatting, generation, and the local API servers all work with your internet disconnected.
Where is each tool storing my models?
GGUF Loader loads files from wherever you saved them. Ollama: ~/.ollama/models (Windows: C:\Users\<you>\.ollama\models). LM Studio: ~/.lmstudio/models (Windows: C:\Users\<you>\.lmstudio\models).
How much RAM do I actually need?
Rule of thumb: RAM ≈ model file size × 1.3, plus ~1.5GB for the system. A 0.8GB 1B model is comfortable on 4–8GB; a 4.9GB 8B model wants 16GB. When in doubt, choose the smaller quantization.
Can I get an OpenAI-style API locally?
Yes. Ollama serves OpenAI-compatible endpoints at http://localhost:11434/v1, LM Studio's built-in server at http://localhost:1234/v1, and llama.cpp's llama-server at http://localhost:8080/v1. Point any OpenAI SDK at one of those URLs and everything runs on your machine.