Full transcript of “Stop Wasting Tokens - How Claude / ChatGPT Actually Reads Your Prompt | Tokens Explained Visually”. Click a timestamp to open the video at that moment. Need another video? Use the YouTube transcript generator.
0:00 Imagine you are working seriously with ChartGPT and my code almost complete and your prompts are crafted beautifully and ChartGPT is about to complete your task but boom! At the time low, ChartGPT stopped responding to you. And why is it not responding? Because of something called as rate limiting. You have reached the current usage cap for the ChartGPT 4.0. Try again in 4 hours. इद चाटजीपिटी मात्र में कादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु
0:47 गादु गादु गादु गादु गादु गादु गादु गादु गादु गादु गा Andhilo every single tool has these limits. And ye limits anni ala calculate chashtaru. There is no magic behind it. And the single term that defines these limits are called tokens. Asali tokens anti enthi. Tokens ala calculate chashtaru. Every single thing we will learn in this video. And literally chepnanu. There are a couple of developers who doesn't even understand tokens anti enthi en chephi. So I am making this video. to help you understand what tokens are visually. So by the end of this video, me reyden yes kundar ante, why some AI tools have limits and why long prompts cost you more and also why ChartGPT forgets your old context and why answers cost
1:38 you more than the prompts. And in all questions to part two, I will also explain two technical terms which are TPS, tokens per second and also the context window on Amata. You can tell us, in our previous video, when we are hitting the chartGPT API, in our output response, we have something called as prompt tokens, completion tokens and total tokens. In our last video, we did not cover this part. But by the end of this video, you will be able to understand these terms as well. So before we understand token and ANT, think about one question. When you type hello and say hello, So do you think Chargibit will be able to understand hello? H E L L O and yappi? Nah,
2:26 because it is a machine. Machine doesn't understand Tilgu gaani, Hindi gaani, English gaani. It is not a human. It doesn't have intelligence like human. But, machine ki hello and text adhan kaak poote, how is it able to interpret our meaning? Yeah? Think about it. So fundamentally, machines work based on mathematics. You need to have mathematical operations to be done. But if I can convert hello into something like numbers, I can do mathematical operations. So whenever you are
3:11 putting some text to any of these LLMs, ChartGPT gaani, Gemini gaani, Cloud gaani internal ga all of these text will be converted into numbers. Uppdu me brain lo question chun tathai, Uppdu e text nye numbers khan dala maarchaali. That's an interesting question. So if you are coming from a programming background, me kallari tell chun tathai which is called ASCII numbers. ASCII numbers अंटे, एंट अंटे, very simple thing. मनकोनन alphabets लो, पिरिथी alphabets की, we assign a dedicated number अनमाट. अंटे, capital A की 65, capital B की 66, small a की 97. इलागा, पिरिथी alphabets की, we will assign a dedicated number. उपड मनन initial question हेंटी,
3:59 hello गदा? So, we will split this into letters. H E L L O. And H your number in D let's assume 104 and E the 101 and L the 108 and L the 108 and O the 111. Open in hello on a word name I can convert into a vector of these numbers. So hello in a convert type in D. Now, chart GPT will be able to understand hello. Yes, absolutely. And I can convert high. I can convert by. I can convert any of these things by converting it into characters. Let us take a moment and think about it guys. Is it a good thing
4:45 or is it a bad thing? Think about it, I will wait like 10 seconds. So, there is one fundamental problem. A fundamental problem is when you convert into hi and hello. But one of the two output numbers, there is no link between them. That means, a machine will not be able to understand these numbers. Hi, hello, almost one meaning. So that is where the intelligence happens. We think about hi, hello, almost in the similar context. But a machine,
5:30 If you can convert these numbers in a specific standard format. So this is the fundamental problem that you will face when you are converting your text into numbers using ASCII number patterns. But ChartGPT is doing something more powerful. It understands patterns, meanings, relationships. So we need something that not only stores the numbers But we also need to retain the relationships, context and meaning. So, what is the other pattern? And that is where transformers and embeddings comes into picture. We will not go into transformers and embeddings in detail in this video. We will touch at a high level. So,
6:19 now let's understand as a token and aint aint aint aint. This is the important stuff. A token is a small piece of text the model reads before it turns that piece into a number. For the model to understand we need to convert our text into number. But, there is one more step and that step is called tokenization or a token. I will see a simple word as hello. Hello is a simple word. So, hello is just a one token and token We will convert into this number using something called as embeddings. Embeddings and T and T we will learn in the next section. So just wait a minute. And we will add one word which is hello space question mark.
7:06 Specifically hello space and question mark. And when we convert that into tokens, it is classified into three tokens. Hello one is token 1, space is token 2 and question mark is token 3. And let's say we will add one word. anti I mean I am not sure what this is but you like better for the words in a ppdu model will not be able to understand this in the country rape an arouju what if there are new words a engineering prompt engineering mundu learning quotes in a ppdu model will not be able to understand that so under can he they use something called as tokenization and they convert the word into the root word very small small chunks can the mata this entire word It
7:53 will be split into anti dish establishment Aryanism. So, ilaga vaka single word can be converted into 1, 2, 3, 4, 5, 6 tokens. Ithi one single word hai but it only takes one token. But ith vaka single word hai but it took 6 tokens. And let's take one more example. I love mangoes dot. even the punctuations will also take place in the tokenization. So, I anadoko token, love anadoko token, mango anadoko token. Look at this, mangoes is converting into a root word mango, play, played, plays. So, alaga, yes, is a separate token and dot is a separate token. So, वकका sentence नहीदी, you
8:39 are converting into tokens. So, tokens अंटे हैंटी, मन देगर उक प्रद्द sentence हुंदी, you are breaking that into smaller, smaller chunks. And that chunks are called tokens. And उपड़ मन देगे tokens हुंने. What is the logical next step? To convert each of these tokens into numbers. And आ numbers अला convert चेस्ता हुँजिंगे technique called embedding. Embedding sentence अंटे, very simple का चप्पालाँटे, it is a way, to convert your tokens into a vector or numerical representation अनुमाटा उपड माना कि मुंदध जेसना ASCII की तीन की difference जेन्टेंटे when you convert using embedding techniques you will find all of the similar products or similar text in a
9:24 similar region लो अनुमाटा let's say नेन mango, apple, banana अने three fruits की I converted into these kind of vectors and यी vectors अननी very similar space lo on tai anu maata. They are not closely related with car gaani, train gaani, bus gaani. Iwani okko close space lo on tai. Iwani inkko close space lo on tai. So here you are converting the text into vectors and that vectors are giving some meaning to it based on the meaning of the words. Indhuku they are closely related anu maata. And that is how model under stands, hello and hi are very similar because their vector representations are very close to each other. If you want detailed video on embeddings, please let me know in the comment section below. And
10:11 one more thing, there are different different types of embedding techniques. Me read a bit, why mango and word ki intapaddar vector representation on the, there are a couple of embedding techniques where for every word you get 1024 dimensions. And a word key you will get 1024 length vector. So it depends model, but the context and but the which kind of embedding model you need to use, you need to change all of those things. But that is for a later video. We will look at how OpenAI will convert the words into numbers. This is from the OpenAI website and tokenizer NGP. Now let's say I will put hello here. Now, if you want to change the text hello, it is just only one token and five
10:56 characters. And hello, it is a single color code. That means it is a single token. And when you change it to this token ID, it is converting the hello token into a number. So, what is the hello? It is 24912. What if I will put capital hello? When you even change caps lock, that token number is going to change. Token ID and this token ID will change based on different types of models that you use. GPT-4 or 3.5 is 9906 but GPT-5 is 13225. So every model follows their own type of embedding techniques. एक बादल की एटेक्निक
11:44 वाडाला नहीदि, we will learn in the later parts of the videos, not in this video. And GPT-3 जिसकुंटे 15496. उपट, let's take one more example. I love mangoes. एन्ट एक चुरोंटे? I love mangoes, if I put it here, I n अग टोकन आयंदि, love अन अग टोकन आयंदि, because, ने नेकड़ M small letter इच्ची मेता नी वेरे वेरे वेरे गि It is breaking into multiple tokens. I know mangoes because of the way that I write. It is occupying 6 tokens. But I will just put this in a single word. Mangoes. It is taking 5 tokens. So you are using more tokens. And I can get the token IDs every word. Maybe let's take this paragraph right
12:35 here. And let's put it here. And children, when I put text, if one is a token, you one is a token and programmatic word, it is broken down into two things. Program and matic. So, all of them are broken down into smaller words called tokens and pretty smaller word ki token ki, you are converting into a number using a technique called embedding and that id is called a token id. So, मिर रासे text न बट्टी, मिर इन्नी tokens वाड़तनारो, अने इद्धिएडवोतन नान माट, Even the spaces, punctuations, emojis, every single thing contains the tokens. वोक्वाल मिर तिलगु लोगानी, Hindi लोगानी, English लोगानी वाड़ते, you are going to consume more amount of tokens. I will
13:22 show it to you live in this right now. So, बड़ मन वें चाद्धा मन्टे, let's open translate, I will put English to Telugu. And, एक डेम If I take it here and if I put it here and it is only one token. Kaka po te if I copy the same hello ane Telugu word if I put it here and it is occupying nine tokens. Aan te meiru same communication English lo jas te you are going to consume one token but the same thing if you do it in Telugu you are going to consume nine tokens. Eindu koo? Because models are designed that way. And look at the token IDs. Chura ndi, hello ki nni token chuchne. But if I put it in English, it is just
14:11 one token ID. Isn't it interesting? That is why, meek tiliye kundane, you are consuming more amount of tokens and that is why you are hitting your limits. And there are different types of tokens guys. Input tokens and output tokens. But let's understand one simple thing. My chart GPT input picture and you get output. So what is the thing that is happening in the background? My input it is converted into tokens. And e-tokens are getting converted into embeddings. And e-embeddings will go into chart GPT. And chart GPT while your servers low. They will process all of these embeddings. and they will try to generate output embeddings and e embeddings ni they will generate it back into tokens and they will convert
15:01 these tokens back into the output mehroka promptic nappadu all of this entire process is going to happen in the back end so ilan tapadu you might notice a couple of things epudente okokka sari chargpt will give you the answers instantly konukkan sari it will take some time and give you the response so why That is where we learn about a concept called TPS which is called tokens per second. The meaning intente what is the capability of the LL model to generate how many number of tokens that it can generate in a second. One second law any tokens generate. Based on that you will see response fast fast slow.
15:51 If it generates more if it is fast if it generates less it is slow. And if fast slow based on your complexity of the question based on the servers busyness based on the prompt that you provide based on the thinking model reasoning capacity. So we turn it based on the output TPS depend on the output. So when you try to download free or open source models, you always check about what is TPS. If TPS is very bad, you will see that the model is lagging a lot. If TPS is very bad, you will feel that the model is working like blazingly fast. And now the interesting part which is the cost. Your pump each token is getting billed. That is for sure. So
16:42 all these LLM models, OpenAI, ChargeGPT, GROC, CLOD, all of these things will work based on the tokens. So let's say for example, ChargeGPT 5.5 model. So for every million tokens that you send as an input, it is going to charge you $5. That's a good amount. Million tokens and only $5. But that is on the input side. When you think about the output, it is going to charge you $30 per million tokens, which is almost 6x the price. In the input is basically me questioner maat. It is not
17:28 using any kind of processing or intelligence. But when you send a question to chat GPT, then youkka answer generate cheda na ki, it is going to use a lot of resources gada. And that is why output token cost is almost higher than the input token cost. And GPT 5.4 ki, 2.5 dollars per million tokens and 15 dollars per 1 million tokens for output. And cloud 4 plus 4.7 na, $5 for 1 million tokens in the input and $25 for 1 million tokens in the output animata. And obviously, prices change with a lot of time. Maybe Mir Chostanapadi video price might have already changed. So always check the official API documentation for the price list. This is just to give you an observation of how input and output
18:16 tokens will cost you and why output is always almost 4 to 5 times costlier than the input tokens. And obviously when you are using open source models like LAMA gani, LAMA 3.3 gani everything is free because you will download the whole model and you are not going to pay any price for these things. And the next important phase which is called the hidden tax anumata. Me ranu koan taro uppudu you got the understanding of what tokens are and I can calculate how many tokens that I am using. But nah. That is not the one that is going to work in the real time. In the content, now if you think about agents or LLMs, they have capabilities like tools, MCPs, skills,
19:03 markdown files, .md, memory, a lot of stuff and conversation history. So, when you are working in a codex or cloud, You are going to use a lot of prompts and Prithi prompt the part 2 you are not only just sending your questions. You are also sending what is your prompt, what is your context, what are your files and then tool definitions, MCPs, past conversations. they will calculate tokens. In the content you are not only sending your single question, you are sending all the information regarding all of the capabilities and a one need token can convert to the and a one need embedding local tie every time you
19:50 are sending a question. So when you say hi, it is not only just taking it as a high, it is taking with a lot of different different information guys. So that is why, it is going to consume a lot and I will also show you the breakdown on the screen itself right now. And the next and the last one which is called a context window. Context window ante ante ante, mana manushu lalaga LLMs ki, they have a limited capacity in the brain anu maata. They can only understand some part of text. Maybe let's say one book anu kundam. They can only understand one book at a time. That is what we call it as a context window. It has their own context
20:35 window. They have 1 million context window, 2 million context window. That meaning the number of tokens that you can send or memorize in a single session. That is what we call it as a context. For example, GPT 3.5, it can only hold maybe 32 pages of a book. And Gemini 1.5 Pro can capture 4000 pages of a book. So every model has its own context and you need to go to their API documentation phase to understand what are the limits and context for every model. Let's say you are sending a lot of text and you are sending and filling up the tokens and once you fill up these tokens, let's say like this at the end, once the
21:22 context is full usage limit and that is why Your AI tools will not remember the past context Anumata. In the context, you start filling up the memory brain. One point, these points, they will forget. So it is recommended to start new charts often and don't paste huge files unnecessarily because you are consuming a lot of context and then use cheap models for smaller tasks and tell the AI to be concise Anumata. And that's everything you need to know regarding tokens and if you have any questions about this video please let me know in the comment section below and if you know someone who might need this video please share it across. If you are a student please share it in the college group so that other people
22:07 can also learn from these videos. Thank you. Bye-bye.
