Why LLMs get dumb (Context Windows Explained)

0:00 Sometimes when you're talking to an LLM like chat, GBT, it gets kind of dumb, right? You'll be deep into a conversation that you can't even scroll to the top of because it's so long and it starts to say weird things and hallucinate. It forgets what you're talking about. It makes stuff up and it's stinking slow. Why is this happening? Context windows. What is that? Did you know that L LLMs like chat, GBT, Gemini, Claude, even local models like llama or deep seek, they're kind of like us, you and me, and that they have memories. Short-term memory, which means they can remember things, that's awesome, but also sometimes they can forget stuff and it happens like this. Let's say you and me we're having some coffee and we're talking for about 15 minutes and that short amount of time, we remember pretty much everything. I remember that story you told. You remember that dumb joke? I said, but you're pretty fun to talk to. So we end up talking for an hour, two hours,

0:45 three hours, and at that point it's kind of hard to keep track of stuff. I forget the amazing point you made. Thankfully you forgot that dumb thing I said. And sometimes we forget the entire point of the conversation. We've talked so long. It kind of reminds me of how sometimes when I fight with my wife, we fight for so long that we forget what we're fighting about to begin with. As it happened to you, Chad GBT does the same thing as like you and your wife. The longer that conversation goes on, the more things you say, the more things it says back to you. It has to store all of that in its short-term memory. And that short-term memory has a limit. That limit is its context window. Lemme show you what that looks like. Here. I'm inside LM Studio, which is a great way to run local AI models on your computer. I'll select a local model to load. We'll do GEMMA three four B and we'll change our context link or our context window to 2048. Now what does that mean? What does 2048 what tokens? Tokens are,

1:31 how an AI counts the words you say to it. I'll show you. Let's load the model real quick. I'll click on load model. It's loading and we'll start a new chat. Now we might say this sentence is 133 characters or 26 words, but an LLM just cares about tokens. So this sentence would actually be 38 tokens. Notice our input token count 38, and we can have that broken down further. If we copy and paste this and go to a website like Open AI's tokenize and paste that text in and it actually sees that as only 34 tokens. Not every LLM will calculate tokens in the same way and notice it might do an entire word as a token. It might do a space and a word or just one comma as a token. And right now the model we loaded up, our GEMMA three four B has a context window of 2048 tokens, meaning that this is the maximum amount of tokens this model can pay attention

2:17 to at any one time. Let me show you, I'm going to make it forget something. So I'll start out by telling it something about me. Right now I'm reading a book called How to Take Smart Notes. Cool, that should be in its memory and down here we can track our usage. 14.6% full. Now let's load it up. Tell me a story about cows. The rain in Oak Caven Valley wasn't a gentle drizzle, it was a. 44.4% Fool. Amazing story. Give me the sequel. Okay, let's weave a sequel. Here it is, Bessie. In the Echoes. Here's 82%. Wow, just wow. Now I'll do a prequel. Alright, we're now at 118.4%. If I click on that, it'll show me the tokens. 2,400 tokens. Let's see if they can remember the book. I told it. What book am I reading right now? What does it talking?

3:03 Oh funny. It mentions memory loss. Oh, memory loss. Forget that extra token. No, I told you about a book I was reading. What is it? No, it doesn't remember at all. It forgot the first things I told it. Let's fix that. Let's increase its short-term memory. Man, I wish I could do this for myself. So notice right now it's set to 2048. I'm going to eject the model, but this time I'll change its context to 4,096 and I'll load the model. Let's see now what book am I reading right now? Bam. It got it. And even apologize for being stupid even though it wasn't his fault. But by the way, a lot of things can fill up the context window. The obvious things are things we say and the things it says back, but there also might be system prompts which are instructions for our LLMs LLMs. It's kind of hard to say after a while these you might explicitly give to it or

3:51 it's included by default. You never see it. You also might give it documents. You might paste A PDF or an Excel spreadsheet that'll take up some more tokens. And when you're doing some vibe coding the code, the code is ticking up tokens filling up that context window. Okay, there's our solution. Let's up the context Infinity. Let's do it right now. Load up our model. Well hold on. There are limits like right now, it tells us this model supports up to 131,000 tokens. Officially, you'll see that written as 128,000. That's still pretty big. Let's do that. Scoot the sucker on up. Whoa, hold on. Setting this high value for context length can significantly impact your memory usage. That's the kicker with running local LLMs. While GEMMA three four B has an astounding 128 K thousand token context window. It doesn't mean our GPU can support that full context specifically, it's going to be about VRA or video Ram. LM Studio is looking at my video car,

4:40 which is a 40 90 with 24 gigs of V ram and it's like, buddy, you're reaching your limits. Calm down. And that's something that may not be obvious to everyone. Yes, to run a powerful model, you'll want a powerful GPU, but we might overlook that to have long conversations, we'll need lots of vra. Can I show you what happens when I try to load up that full context window on my computer? Watch this. It might break my recording. So sky's the stinking limit. 120,000 tokens. I'm going to bring up my, oh no, it's going to kill my recording. I just know it. Here's my performance monitor. We'll load the model now and it's about to cry. Oh, look at it down here. Yeah, it just boom all the way up. It took out all my vra. Is it loaded yet? Oh man, my computer is just dying. Look at this. Oh no, you probably can't see it. Okay, the model's loaded, but it's going to be hard to talk to before it was so snappy.

5:27 Look how slow this is. And what you're seeing here is that bigger context windows require more compute power, more GPU resources and more ways than one. Look at this, I'm going to stop this and eject it. It's probably not even showing you right now, but with cloud models, that's a whole different story. You can use their full advertised context windows and they got some big ones. GPT-4 oh is rocking 128,000 tokens and he thinks he's better than everybody else. Oh three, many Clyde 3.7, 200,000 and the Gemini 2.5 from Google, 1 million tokens. Tell us your whole life story. It's going to remember. And what's crazy is they're saying that 2 million is right around the corner. What? Hey, number check from the future here. Meta just released Llama four Scout, which has a 10 million token context window. Are you serious? This is a local model. I just had to say that. That's crazy.

6:12 Now back to the video. But there is a catch. Even if you have a super large context window, it doesn't mean the LLM won't kind of freak out and forget stuff become less accurate or start to go extremely slow. You'll notice on those larger conversations, it'll have some trouble paying attention and that's a big problem. AI paying attention. Wait, what? Hey, real quick, it's time for a coffee break. This coffee break, and this tip I'm about to give you is sponsored by twinge. Now it's talking about context. What I like to do a lot is go to a website. For example, if I'm trying to learn about twinge, like how does it compare to A VPN, how does it compare? It's awesome. That's how it's, I might want to take the content of this webpage and put it into an LLM and ask you questions about it. How do we do that? Well, normally we'll just A control C and then paste it into our LLM. Boom. That's terrible. I hate that. It's ugly.

6:58 The format of all this information was not meant to be read by an LLM. It was meant to be read by humans and definitely not in this little chat box here. There's a better way. I found this cool tool. It's called Gina. All we have to do is this. Whatever webpage you're on, go to the address bar and at the very beginning just scroll over, type in R dot Gina with a J, do ai and then a slash watch this. Boom. It converts that entire webpage into markdown, that LLMs love. It's nice and clean. And as you're about to see LLMs do have a hard time paying attention. This definitely helps. Now I can just paste that sucker in there, say, summarize this and I'm off to the races. So R Gina ai, I just found this tool. I'm going to start using it all the time. And speaking of tools you should use all the time. Twin Gate should be one of those. It's a remote access solution, zero trust,

7:44 and it's what I use to stay connected to my home lab, my studio, my business when I'm traveling, when I'm out, when I have remote employees. I used it when I was in Japan. I used it on my road trip. It's fast, quick and more importantly, the most important thing nowadays, it's secure. If you're still using a VPN, you're crazy. And setting up twin Gate is easy. You can put it on a raspberry pie inside your network. I've shown that a few times here on my channel. Takes about five minutes. And for home laborers, it's free, even though it feels like it shouldn't be. There's too many features you're giving us, don't believe me. Check it out. Link below. Anyways, back to the video. There was a paper released called Lost in the Middle, and it showed us how LLMs are kind of like us with paying attention. And just like my wife, once she watches a long movie, she'll watch the first part, then fall asleep and then wake up at the end. And that's the context she has. I'll ask you to explain it to me in the same way, conversations with LLMs, with large context,

8:30 the models were more accurate with info at the beginning and even with info at the end. But in the middle, huge drop off. And across the board we saw this U shape. So what does that tell us? LLMs are falling asleep during our conversation. Kind of I'm telling you, LLMs have problems paying attention just like us. But that brings up a question. How do LLMs pay attention? What does that mean for a computer to do that? It's actually incredibly fascinating. It's called attention mechanisms, specifically with modern LLMs. It's referred to as a self attention mechanism. And honestly, I was going to do a deep dive on that and this video right here. But that would increase the context window of this video, making it harder for people to pay attention to. And given this is probably the middle of the video, you would forget it. So I'll have another video coming out on that, but I still want to talk about it. There will be another video somewhere around here soon, TLDR.

9:15 When you say something to an l and m like, Hey, I want coffee, but caffeine makes me jittery. What should I get? It will do something ri really similar to how we process and think. It'll use some fancy semantic math to decide which of these words is important, which is relevant both to the context of your entire conversation and to how the words relate to each other. It might look like this, and this is very simplified. They essentially assign attention scores saying, Hey, in this conversation, coffee is high, caffeine is high, jittery, but words like I or me, kind of low relevance to the context. And we kind of do the same thing. We only remember the things that are relevant. You said something about coffee, jittery, caffeine, and we'll use that context to process our responses. AI does the same thing. Now what's crazy is this process is pretty complex. All this math to assign attention scores, and it does this every time you send something to the LLM every time you add to

10:03 the conversation. So this conversation, no big deal chat, GBT can keep track of it easily. But if you start having a conversation where that scroll bar is hard to find, you know what I'm saying? You can't even find it to scroll up. Those larger contexts not only have insane V RAM requirements, they also have some pretty healthy computational requirements. Every time you add to that conversation, that math problem that has to run to figure out what's important gets bigger and it requires more GPU power. And that is why in those larger conversations, the LLM starts to hallucinate and it seems just a bit more slow. It's using a ton of memory and it's having to do some crazy math every time you talk to it. And not only that, think about this if you're saying a lot of different things, I know some people who just keep one conversation open with chat GBT and just shoot the breeze. They'll talk about coffee, the weather,

10:49 explain quadratic equations to me. And during the context of the conversation, it's trying to weigh all of these different words and how relevant they are to the entire context of the conversation. And that's crazy. And no wonder it starts to hallucinate. So a quick solution to that, and this is a rule I try to go by when I'm talking with LLMs. When you change an idea, when it's a significant shift from what you're currently talking about, start a new chat, the performance will be so much better. In fact, sometimes when you're talking with other LLMs like Claude, it'll even tell you at the bottom, Hey, you've been talking for a minute. Things are going to slow down. Why don't you go ahead and start a new chat so things can be better. But I'm hoping sometimes soon this won't be a problem anymore. Especially with Google's Gemini coming out with a 2 million context window. I want it to be able to pay attention to everything I'm saying. It's kind of like you having a notebook with a bunch of stuff you've written down. You've got the ability to have all that stuff with you.

11:37 That's your context. But the ability to quickly find what you need and that notebook might be kind of limited. You can't remember every page. Now, if you want to learn more about self attention mechanisms, that'll be its own video. It is fascinating. But real quick, I want to get back to the idea of context windows because that is the context of this video and it's actually good news for some of us who want to run these local AI models and they're full context. There are some optimizations that have come out to help us do that. The first one is called flash attention. Like hey, paying attention, kind of like that. Actually it's not really at all. I just wanted to do that, but now you're paying attention. It's actually an option right here when I want to load a model. So we'll choose GEMMA three, four B, and right here under experimental features we have flash attention. I'll go ahead and enable it. This will change how the model will compute its attention and assign those weights. Now it's doing the same kind of crazy semantic map that regular self attention

12:23 mechanisms are doing, but it is like that lazy kid in class who always got straight A's. It'll actually skip building the full table of token comparisons by processing tokens and chunks with optimized GPU routines. Essentially, it never stores the full matrix or all the complications in memory at the same time leading to significant improvements in not only memory but speed. So let's try it out real quick. I'm going to go full 131 to the limit, making sure I have flash attention, ready to go, pull up my poor GPU and we'll see what happens. It's slowly loading my model and it still did max out my memory, but we'll see how the speed is GPU spiking. It's thinking it's a lagging. Okay, still pretty slow, but there's actually two more things we can do. I'm going to stop it now. Eject, give me my memory back please. Oh, thank you. If I select the model once more, choose GEMMA three, four B, I'll select flash attention.

13:09 But we also have these KC and V cache optimizations. These two options will actually compress our data so it takes up less room in vra. So let's enable both of those suckers. Bam, bam. And we'll change the amount and the lower the quantization the better. So we'll go to quant four on both of these and we'll go to full 1 31. Let's load the model. Let's watch our memory. Look at that. It did not even max out our memory, which is amazing. We're using the full 128,000 contact window. Let's try and talk to it significantly faster. Keeping in mind, I'm recording right now using my GPU, I mean that's stinking great. Now, in addition to flashed attention and the compression of our data, there's also a thing called paged cache. Paged cache will actually move attention cache between your GPU, so your VAM and your system ram. So essentially it's sharing the RAM with your system.

13:56 Downside is that it's going to be significantly slower than the VRA sitting right there on your GPU. And yes, it is like your page file on your system. Your system will have system memory, your ram, and when you max out your ram, your computer might be configured to use a page file, AKA, your hard drive or your SSD, which will be significantly slower than your ram. So let's bring this video home. I'm a massive fan of larger context windows. I want to give an LLM everything I could possibly give it about me or a problem I'm trying to solve and help me figure it out. And if it's local, that's way better. But there's a couple of massive problems. A few we've already talked about. You're going to take up a ton of memory. GPU memory, your v ramm memory you may not have, and it's going to take a ton of power to process your conversation. The longer it gets resulting in slower conversations. But I think the scariest part, the scariest downside is the larger attack surface. Yes,

14:43 LLMs can be hacked and they're vulnerable to some creative prompting that can gel break out of their protection systems. The longer a conversation is, the more it can kind of forget what's in the middle as we saw in that little U curve. And the easier it will be for an attacker to hide some malicious stuff in there that could bypass its safety precautions. Anyways, our context window is way too big. We're going to end it right now. I'll catch you guys in the next one. Chat, GPI can't talk right now, which is bad for a guy making a video. Okay, don't put that. That's stupid.