0:00 Running AI locally means picking from a thousand models, a dozen quant types, and a pile of settings. And everyone just tells you, "Use this one." But once you know what actually matters when you choose, what fits your card, what each setting really does, you stop guessing. Am I really stuck at 8 billion? Can my card even run something big? Which model for which job? What hardware actually matters? And what's the real difference between all these quants? And once it clicks, this is what a 6 GB card can do. Your two small GPU does way more than you think. By the end, you look at any model, any quant, any card, and know exactly what's right for you. No more guessing. Okay, so when you ask yourself, "Can I
0:46 run this model?" you're actually asking two questions. One, does it fit my system with all the VRAM and the system memory I have? Two, can I run it fast enough to do meaningful work? So, let's tackle the first part. Can I fit this model into my machine? And I'll be using a 3060 12 GB card as a baseline, but this should work the same on 8 GB, 6 GB, or even 24 GB. A model file is a bunch of numbers which decide how much to amplify or de-amplify the signals running inside the virtual brain of an AI. These are what we call parameters, like 8 billion, 13 billion, 35 billion, or 40
1:31 billion. So, that's how many connections this model has in its virtual brain. And these numbers represent how much you amplify or de-amplify each connection. Now, how do you know how much you can fit? You can do a very rough calculation. Number of parameters times bytes per parameter plus some extra for the context. So, a four-bit quantized model has almost half a byte per parameter, which means an 8 billion model is 8 billion times half a byte, roughly 4 GB of weights. And once you add the context on top, around 5 GB in your machine. With this rough calculation, an 8 to 13
2:17 billion parameter model is approximately where we are stuck, approximately the limit of a 12 GB graphics card. But there's a catch. We'll see later that we can go well past that and still get meaningful work done. Now, about the context. It can take anywhere from a little space in your VRAM to a whole lot. The weights are the neural connections in the AI's brain. The context is basically the AI's short-term memory. Every instruction it has, every conversation you both had, every action it took, all of that is captured in the short-term memory, which is the KV cache. Without this cache, the AI would have to reread everything that
3:03 happened so far again and again on every single token it generates. And the longer you talk, or the longer the AI thinks and performs actions, the more short-term memory it burns. And at long context, that short-term memory can end up bigger than the brain itself. Now, to keep it from growing out of control, you have one lever you can pull. KV cache quantization. It's the same thing we did for the model weights, quantizing the numbers in the KV cache down to lower bits to save space. Q8 roughly halves the space it needs with almost no quality loss. You can go lower, but it will start to
3:49 hurt your long context reasoning. Okay, now that we've got our fundamentals cleared, with this math, a 30 billion or even a 40 billion parameter model has no business fitting in a 12 GB card, right? Well, unless it's an MoE. So, hold that thought. We'll come back to it later. Now, the question is, how do you actually run a model? When we talk about running a model, what we're really talking about is running some deep learning kernels on top of these model weights. And there are several tools that do that. I'll divide them into two categories. First are the engines. These are the heavy lifters. They do the actual work of turning your input into
4:35 output. Most of the time, they're pretty bare-bones, not so user-friendly, but they give you the raw power and the flexibility to work at the very low level of model inference. And then we have the wrappers. A wrapper is exactly what it sounds like. It wraps an engine in a friendly tool or interface, so it's easy for a regular user to operate. The catch is, a wrapper adds a bit of overhead on generation speed sometimes. You're trading flexibility for user-friendliness. And here's the thing, this is where most of us get stuck. It's easy to install the wrappers, run the default model it hands you, and never go a level down. And that ends up becoming a wall for your local AI journey. But going down
5:21 one level is much easier than you think. But before going to the deepest level, I think everyone should start with a user-friendly wrapper. It creates an easy on-ramp and lets you understand what you can do with a local LLM running on your own system. A very good example is Ollama, one of the first wrappers around Llama.cpp, and it was hugely popular until they started pivoting to a cloud-first company. They closed source their UI, stopped crediting llama.cpp, and a lot of the community started calling the pivot a betrayal, a sellout. The key takeaway, don't get attached to a wrapper. Try to go beyond it. The open engine llama.cpp is yours forever. There's also LM Studio if you'd rather
6:08 have a GUI. Same deal, a wrapper over llama.cpp, proprietary and free for now. Now, let's come to llama.cpp, the engine that all the wrappers are wrapping. This is where the real control lies. We have the NGL knob, which lets us bluntly divide the model into two parts. One part on the GPU, one part on the CPU. Cool, but if we go one step deeper, we have override tensor. That lets you pick a specific tensor by name or with a regular expression to put the exact tensors or layers you want on to either the CPU or the GPU. And why does that matter? Because that precision is exactly where most of the optimization lies. We can put the heavy-duty but low
6:56 in memory layers on the GPU, and at the same time put the low compute but heavy in memory layers on the CPU. And if you're ever serving a whole team of a big card, that's when you jump to vLLM. But for solo use on a budget card, llama.cpp is all you need. Now that we have our inference engine all set, let's dive into the quantization part of local LLMs. Quantization is the reason we can fit a 20 GB model into a 12 GB VRAM card. A quick bit of computer science. Hang with me, it's easy. In computers, we store numbers in bits, ones and zeros. And a group of eight bits is one byte. Now,
7:41 when an AI lab trains a model, they usually train it at full precision called BF16, which is two bytes per parameter. So, an 8 billion parameter model needs 8 * 2 billion bytes to store, roughly 16 GB. And if you need a 16 GB graphics card just to run an 8 billion parameter model, the future of local AI looks pretty bleak. This is where quantization comes in. Instead of using all 16 bits to store one parameter, we cut the number of bits down to a lower, rougher precision. Going from 16-bit to 8-bit halves the model size. 16-bit to 4-bit is a quarter of the original. At about four times smaller, the model
8:27 barely loses any quality. But as you go lower, it starts to degrade. So, how do you read all these labels? You go to Hugging Face, filter for GGUF, the format Llama CPP uses, and the moment you open a model, you'll see a lot of different quantizations. Q4KM, Q5KS, IQ4, all this alphabet soup. So, let's decode it. The first part is the bits per weight. Q4, Q5, Q8 means roughly four, five, or eight bits per weight. The K stands for K-quant, a smarter way of quantizing where different weights get different precision. The weights that really matter get higher bits, the low-value weights get
9:13 lower bits. That's an upgrade from the older legacy method, which used the same bits for every weight in the model. Then there's S, M, and L, which tell you how many of those layers get bumped up to the higher bits. S is small, very few. M is medium, the balanced one. L is large. Then we have IQ-quant, which uses an importance matrix, the I matrix, plus a lookup table to squeeze the model even smaller at the same quality. It's like unzipping a file. It takes a little more time when you're generating, but if you have a powerful CPU, you'll still get good speed. So, if you're really tight on VRAM and willing to sacrifice a little speed, go for IQ. Good quality at about 10%
10:02 smaller than the equivalent K quant. And then there's Unsloth's own variation of I matrix quantization called UB, Unsloth dynamic. Similar idea with their own flavor on top. Okay, that's the whole alphabet soup decoded. So, out of all these, which one is the sweet default? Q4KM. Going from Q8 down to Q4 is almost free. You barely lose anything. And Q4 is the safe floor you can rely on. Go below it, down toward Q3 and Q2, and it starts to fall apart fast. Only after testing Q4, if you've got room to spare, should you go higher like Q5 or Q6. And if you're
10:47 tight on VRAM, drop to IQ or Unsloth dynamic. Now, after all this, we're clearly thinking that a 30 billion parameter model has no business fitting on a 6 GB, 8 GB, or even a 12 GB card, right? This is why we need to learn about different model types. There are two types of models. The first is a dense model, which fires all the neurons of its digital brain at once for every single token. And then there's a mixture of experts. Much like how a human brain works, it only uses a certain part of the digital brain to generate the next token. So, for an MoE, only part of the layers are activated, and that part is
11:33 very small compared to the total. That brings us to an optimization to move past the GPUs VRAM. Use our CPU and system RAM alongside the GPU. This has to be surgical. We pull out only the sections where the experts live. Memory heavy but low in compute and push those to the CPU. While we keep the compute heavy but low memory attention layers on the GPU. So the calculation changes for MOE. We don't need to hold the entire model in VRAM. Only the attention layers. Use the active parameters as your reference for how much VRAM you need. The rest of the weights stay in your system RAM. What that means is with
12:19 roughly 64 GB of system RAM and just 8 GB of GPU memory, you can run GPT-OSS 120 billion. A 117 billion parameter model that only really thinks with about 5 billion at a time at a totally usable speed. And because the active parameters are low, each expert the CPU executes is tiny. So it's easy for the CPU to compute that part while the GPU takes care of the big attention layers. And if you want to know more about how to do that, check out this video where I ran a 35 billion parameter model on a 10-year-old card with just 6 GB of VRAM. A big offloaded MOE can actually be
13:06 faster and smarter than a fully loaded dense model running entirely on your GPU. The speed comes from the active parameters, not the total parameter count. A 30 billion MOE has only 3 billion active and it can match and beat a dense 14 billion on your card. smarter and just as fast. But you have to watch the active parameters. There are lots of MOEs out there, but if the active parameters are very high, you hit diminishing returns. As long as the active parameter count is low, you really get the benefit. If pulling a 30 billion onto a cheap card is the kind of thing you're here for, that's basically this whole channel. Subscribe and stick
13:53 around. Okay, so now what hardware should you actually buy for this? There are two ways to run a model and each wants a different hardware priority. Either run a dense model fully loaded in VRAM or use MOE offloading where the experts go on the CPU and the attention layers stay on the GPU. So when you're buying for dense models, go for the highest VRAM your budget can afford. But for MOE offloading, more VRAM hits diminishing returns. So balance it. Enough VRAM for the attention, plenty of fast RAM for the big MOEs. So if you're going the dense route, VRAM is a cliff, not a slope. The moment you start offloading a dense model, your speed crawls. A
14:40 cheaper used card with more VRAM is much better than a powerful newer card with less. And even though Llama.cpp supports AMD, Intel, and Nvidia, let's be honest, Nvidia is the most optimized and not just in Llama.cpp, in pretty much any inference engine. So for my pick, I'd go with a 3060 12 GB. It's the sweet spot. It works well with a bigger 35 billion through MOE offloading and it handles dense models like Gemma 4 12B2. Want a bigger, better card without paying a lot? The 5060 Ti with 16 GB. There's also the Tesla P40 24 GB for $150 to $200, but it's an old Pascal card, no tensor cores, needs heavy cooling. Looks
15:28 great on paper, but it's kind of a trap. Now, for the dense path, your speed is just your GPU's bandwidth. But when you offload an MoE, and this is specific to how Llama.cpp does it, the experts actually run on the CPU. So, the weights never have to travel across PCI, only the tiny activations cross. So, your RAM bandwidth matters most, then the CPU, then PCI. Now, this isn't true for every engine. Something like vLLM streams the offloaded weights to the GPU over PCI. So, there PCI genuinely starts to matter. And it's also your bottleneck if you split a model across two GPUs. But for Llama.cpp style offloading, it's almost irrelevant. The data's tiny. The
16:16 key takeaway, for MoE offloading on Llama.cpp, maxing fast RAM with a decent CPU beats a bigger VRAM card on cost. For a bigger dense model, it's worth paying for the VRAM. So, does that mean you should go out and start shopping for GPUs or building a new rig? Honestly, I think it's better to first try what you already have. See how you can optimize it and what performance you can get out of that setup. A lot of the time, we ignore the hardware we already own and what it's capable of. And often, it has enough capability to cover 90% of your needs. So, go try with your current hardware first. If it doesn't cut it, then start shopping for an upgrade.
17:02 So, that brings us to what model should I choose for the task I have? There are two types to think about. The specialist for specific tasks like coding or vision, and the generalist for regular chat. You can pick a generalist you just talk to and get the basic things done and have specialist for specific use cases that you switch to when you want higher capability on that kind of work. And a small specialist can often beat a big generalist on its own task. A tiny OCR model can outread a 12 billion parameter vision model on PDFs. Also, do check the license because open weight doesn't always mean open source. For your own home lab or personal use,
17:48 it's mostly fine. But there are some catches if you're thinking of using it commercially. Once you've picked the kind of model you want, you go to Hugging Face to grab it. And there are a few things to check before you download. First, base versus instruct. If the model's name doesn't say instruct it or chat, it's a base model. And a base model won't actually answer you. It just auto completes your text. So, grab the instruct or chat version. This trips up a lot of people. Second, whose quant you're downloading. Stick to the people who do this well. Bartowski, Unsloth, Miss Rademaker, or the official repo. Third, grab the GGUF, not safe tensors
18:35 for a budget card. And last, don't trust the context length on the card. When it says 262,000 or a million tokens, that's a lab number. You can't fit that much KV cache on a 12 GB card. In real life, you can realistically get around 64,000, especially with that KV cache quantization we talked about. Now, the part you actually came for. Which models do I actually run? Your call, but these are my go-tos. For coding and agentic work, my go-to is Qwen 3.6 35 billion A3B, the same MOE we offloaded earlier. For agents, GLM 4.7 flash also runs
19:21 great locally. Though Gwen feels a little faster to me, I think that's down to its hybrid attention, but I'm not 100% sure. For writing, I lean on GPT OSS. For anything multimodal, charts, screenshots, visual data, Gemma 4. If you just want a solid general chat model, Gemma 4 12B. For vision specifically, Gemma 4 E2B or E4B. And for a rag setup, a small embedding model like Gwen 3 embedding. And look, this isn't just a benchmark toy. One of you watching is a teacher building study materials with a local model on a cheap card. That's the whole
20:06 point, real work on hardware you own. >> So, that's it. Everything that actually matters for running local AI on a cheap card. You can now look at any model and basically know, is it going to fit, dense, or do I offload? Which quant, and is it even right for what I'm doing? No more guessing. This is just the foundation. Agents, rag, fine-tuning, all come in their own videos. But here's the thing I really want you to take away. This was never really about saving a few gigs of VRAM. It's about owning the intelligence instead of renting it. The cloud providers want you to think you need their stack, but for 90% of what you do, you don't. You've got a perfectly good machine that can run real AI, and nobody can take it away from
20:51 you. Plus, your data stays on your machine. So, go run it yourself.
