← Back to Blog

Running DeepSeek V3 Locally with GGUF

A realistic guide to the 671B model, the dynamic quants that make it possible, and the distill models that fit your PC today

The Honest Answer First

Yes, you can run DeepSeek V3 locally as a GGUF — but be clear-eyed about what it takes. DeepSeek V3 is a 671-billion-parameter Mixture-of-Experts model. Even aggressively quantized, it needs a workstation-class machine: a beefy CPU, a lot of system RAM, and ideally a GPU for the parts that don't fit in memory.

The good news: if you don't have a 128GB+ workstation, the DeepSeek-R1-Distill family gives you most of the reasoning magic in models that run on ordinary laptops. This guide covers both paths honestly.

Why Dynamic Quantization Changed Everything

Standard quantization treats every layer the same, which wrecks accuracy on huge MoE models. Unsloth's Dynamic GGUFs instead quantize the unimportant feed-forward expert layers down to 1–2 bits while keeping attention layers and critical MoE pathways at higher precision. That's how a 720GB FP16 model squeezes into files that fit on a single large SSD.

DeepSeek-V3.1 GGUF lineup (Unsloth)

Quantization        MoE bits    File size    Quality
TQ1_0              1.66-bit     170 GB       Minimal footprint
IQ1_S (Dynamic)    1.78-bit     185 GB       Compact
IQ1_M (Dynamic)    1.93-bit     200 GB       Fair
IQ2_XXS            2.42-bit     216 GB       Better
Q2_K_XL (Dynamic)  2.71-bit     251 GB       Good — recommended balance
IQ3_XXS            3.12-bit     273 GB       High
Q3_K_XL (Dynamic)  3.5-bit      296 GB       Great
Q4_K_XL (Dynamic)  4.5-bit      384 GB       Near-lossless

Files this size ship as multi-part splits (e.g. -00001-of-00006.gguf) — GGUF Loader and llama.cpp load all parts from the same folder automatically.

What Hardware Actually Runs It?

  • 1-bit tiers (TQ1_0 / IQ1_S, ~170–185 GB): at least 180–220 GB of combined RAM + VRAM. Realistic on a high-end workstation — e.g. a 24 GB GPU (RTX 3090/4090) plus 128–256 GB of system RAM with MoE layers offloaded to CPU.
  • Dynamic 2-bit (Q2_K_XL, ~251 GB): roughly 230–255 GB combined. Ideal for multi-GPU rigs or Apple Silicon unified memory (Mac Studio with 256 GB).
  • 4-bit near-lossless (Q4_K_XL, ~384 GB): 400 GB+ — server territory.

Bottom line: if your machine has less than ~128 GB of RAM, the full 671B model is out of reach. That's where the distills come in.

The Realistic Path: DeepSeek-R1-Distill

DeepSeek distilled its R1 reasoning into smaller open models that run on consumer hardware — these are the ones most GGUF Loader users should reach for:

Model                        Q4_K_M size    RAM/VRAM needed
DeepSeek-R1-Distill-Llama-8B   ~5 GB          8–12 GB
DeepSeek-R1-Distill-Qwen-14B   ~9 GB          12–16 GB
DeepSeek-R1-Distill-Qwen-32B   ~20 GB         20–24 GB (fits an RTX 4090)

The Qwen-32B distill is the standout — it punches at or above o1-mini on math and logic benchmarks, yet runs on a single 24 GB GPU at Q4_K_M. The 8B version runs comfortably on a budget laptop. All of them are plain GGUF files, so they load directly in GGUF Loader with zero special setup.

Running DeepSeek in GGUF Loader

GGUF Loader loads any GGUF file — including multi-part DeepSeek splits and every distill model. Here's the practical workflow:

  1. Download a GGUF. For the distills, grab DeepSeek-R1-Distill-Qwen-32B-GGUF (or 14B/8B) from Hugging Face — pick the Q4_K_M file for the best quality-per-GB balance.
  2. Open GGUF Loader → "Load Model" and select the .gguf file. For multi-part V3 splits, point it at the first part (...-00001-of-00006.gguf) — the parts are loaded as one model.
  3. Set the context size. GGUF Loader supports up to 32K tokens. For the big V3 tiers, keep it modest (8–16K) so the KV cache doesn't eat memory you need for weights.
  4. Choose your mode. "GPU Accelerated" offloads up to 35 layers to your graphics card — use it if you have a discrete GPU with 8 GB+. Otherwise "CPU Only" works fine, just slower.
  5. Chat. DeepSeek models follow the R1 reasoning style by default — they'll show their chain-of-thought before answering. For V3 specifically, generation settings around temperature 0.6 and top-p 0.95 give the cleanest, least repetitive output.

Memory-saving tips for the big V3 tiers

  • Offload the MoE expert layers to CPU so the GPU only holds attention layers — this is how a 24 GB card can drive a 170–185 GB model.
  • Keep the context window small (8–16K) and avoid long chat histories, which balloon the KV cache.
  • Close other heavy apps — at these sizes, system RAM is the bottleneck and every free gigabyte counts.

Summary

  • Full DeepSeek V3/V3.1 (671B): possible with dynamic 1–2 bit GGUFs, but you need 180 GB+ of RAM. Workstation or high-end Mac territory.
  • Best value for most people: DeepSeek-R1-Distill-Qwen-32B at Q4_K_M (~20 GB) — flagship-level reasoning on a 24 GB GPU, or the 14B/8B distills for smaller machines.
  • GGUF Loader handles both — plain GGUF files, multi-part splits included, with up to 32K context and optional GPU acceleration.

Looking for a place to start? The GGUF model download guide has direct links to safe sources, and the memory calculator tells you exactly what your hardware can run before you download 200 GB.

Next Steps