Cursor Team: Future of Programming with AI | Lex Fridman Podcast #447

0:00 the following is a conversation with the founding members of the cursor team Michael truell swall oif Arvid lunark and Aman Sanger cursor is a code editor based on VSS code that adds a lot of powerful features for AI assisted coding it has captivated the attention and excitement of the programming and AI communities so I thought this is an excellent opportunity to dive deep into the role of AI in programming this is a super technical conversation that is bigger than just about one code editor it's about the future of programming and in general the future of human AI collaboration in designing and Engineering complicated and Powerful

0:46 systems this is Le Freedman podcast to support it please check out our sponsors in the description and now dear friends here's Michael suale Arvid and Aman all right this is awesome we have Michael Aman suali Arvid here from the cursor team first up big ridiculous question what's the point of a code editor so the the code editor is largely the place where you build software and today or for a long time that's meant the place where you text edit uh a formal programming language and for people who aren't programmers the way to think of a code editor is like a really souped up word processor for programmers where the reason it's it's souped up is code has a lot of structure and so the

1:33 the quote unquote word processor the code editor can actually do a lot for you that word processors you know sort of in the writing space haven't been able to do for for people editing text there and so you know that's everything from giving you visual differentiation of like the actual tokens in the code to so you can like scan it quickly to letting you navigate around the code base sort of like you're navigating around the internet with like hyperlinks you're going to sort of definitions of things you're using to error checking um to you know to catch rudimentary B um and so traditionally that's what a code editor has meant and I think that what a code editor is is going to change a lot over the next 10 years um as what it means to build software maybe starts to look a bit different I I think also code edor should just be fun yes that is

2:20 very important that is very important and it's actually sort of an underated aspect of how we decide what to build like a lot of the things that we build and then we we try them out we do an experiment and then we actually throw them out because they're not fun and and so a big part of being fun is like being fast a lot of the time fast is fun yeah fast is uh yeah that should be a t-shirt like like fundamentally I think one of the things that draws a lot of people to to building stuff on computers is this like insane integration speed where you know in other disciplines you might be sort of gate capped by resources or or the ability even the ability you know to get a large group together and coding is just like amazing thing where it's you and the computer and uh that alone you

3:07 can you can build really cool stuff really quickly so for people don't know cursor is this super cool new editor that's a fork of vs code it would be interesting to get your kind of explanation of your own journey of editors how did you I think all of you are were big fans of vs code with co-pilot how did you arrive to VSS code and how did that lead to your journey with cursor yeah um so I think a lot of us well all of us originally Vim users pure pure VI pure Vim yeah no neo just pure Vim in a terminal and at Le at least for myself it was around the time that C- pilot came out so 2021 that I really wanted to try it so I

3:55 went into vs code the only platform the only code editor in which it was available and even though I you know really enjoyed using Vim just the experience of co-pilot with with vs code was more than good enough to convince me to switch and so that kind of was the default until we started working on cursor and uh maybe we should explain what copala does it's like a really nice autocomplete it suggests as you start writing a thing it suggests one or two or three lines how to complete the thing and there's a fun experience in that you know like when you have a close friendship and your friend completes your sentences like when it's done well there's an intimate feeling uh there's probably a better word than intimate but there's a there's a cool feeling of like

4:40 holy it gets me now and then there's an unpleasant feeling when it doesn't get you uh and so there's that that kind of friction but I would say for a lot of people the feeling that it gets me over powers that it doesn't and I think actually one of the underrated aspects of get up copet is that even when it's wrong is it's like a little bit annoying but it's not that bad because you just type another character and then maybe then it gets you or you type another character and then then it gets you so even when it's wrong it's not that bad yeah you you can sort of iterate iterate and fix it I mean the other underrated part of uh calot for me sort of was just the first real real AI product it's like the first language model consumer product so copile was kind of like the first killer app for LMS yeah and like the beta was

5:27 out in 2021 right okay mhm uh so what's the the origin story of cursor so around 2020 the scaling loss papers came out from from open Ai and that was a moment where this looked like clear predictable progress for the field where even if we didn't have any more ideas looked like you could make these models a lot better if you had more computer and more data uh by the way we'll probably talk uh for three to four hours on on the topic of scaling laws but just just to summarize it's a paper and a set of papers and set of ideas that say bigger might be better for model size and data size in the in the realm of machine learning it's bigger and better but predictively better okay this another topic of conversation but anyway yeah so around that time for some of us there were like

6:13 a lot of conceptual conversations about what's this going to look like what's the the story going to be for all these different knowledge worker Fields about how they're going to be um made better U by this technology getting better and then um I think there were a couple of moments where like the theoretical gains predicted in that paper uh started to feel really concrete and it started to feel like a moment where you could actually go and not you know do a PhD if you wanted to work on uh do useful work in AI actually felt like now there was this this whole set of systems one could built that were really useful and I think that the first moment we already talked about a little bit which was playing with the early bit of copell like that was awesome and magical um I think that the next big moment where everything kind of clicked together was actually getting early access to gbd4 so sort of end of 2022 was when we were um

7:00 tinkering with that model and the Step Up in capabilities felt enormous and previous to that we had been working on a couple of different projects we had been um because of co-pilot because of scaling laws because of our prior interest in the technology we had been uh tinkering around with tools for programmers but things that are like very specific so you know we were building tools for uh Financial professionals who have to work with in a juper notebook or like you know playing around with can you do static analysis with these models and then the Step Up in gbd4 felt like look that really made concrete the theoretical gains that um we had predicted before felt like you could build a lot more just immediately at that point in time and also if we were being consistent it really felt like um this wasn't just

7:46 going to be a point solution thing this was going to be all of programming was going to flow through these models it felt like that demanded a different type of programming environment to different type of programming and so we set off to build that that sort of larger Vision around then there's one that I distinctly remember so my roommate is an IMO gold winner and uh there's a competition in the US called of putam which is sort of the IMO for college people and it's it's this math competition is he's exceptionally good so Shang Tong and Aman I remember it sort of June June of 2022 had this bet on whether the mo like 2024 June or July you were going to win a gold medal in the Imo with the with like models IMO is international math

8:32 Olympiad uh yeah IMO is international math Olympiad and so Arvid and I are both of you know also competed in it so was sort of personal and uh and I I remember thinking Matt is just this is not going to happen this was like it un like even though I I sort of believed in progress I thought you know I'm a girl just like Aman is just delusional that was the that was the and and to be honest I mean I I was to be clear it very wrong but that was maybe the most preent bet in the group so the the new results from Deep Mind it turned out that you were correct that's what well it technically not technically incorrect but one point awayan was very enthusiastic about this stuff back then and before Aman had this like scaling

9:20 loss t-shirt that he would walk around with where it had like charts and like the formulas on it oh so you like felt the AI or you felt the scaling yeah I i l remember there was this one conversation uh I had with with Michael where before I hadn't thought super deeply and critically about scaling laws and he kind of posed the question why isn't scaling all you need or why isn't scaling going to result in massive gains in progress and I think I went through like the like the stages of grief there is anger denial and then finally at the end just thinking about it uh acceptance um and I think I've been quite hopeful and uh optimistic about progress since I think one thing I'll caveat is I think

10:05 it also depends on like which domains you're going to see progress like math is a great domain because especially like formal theor improving because you get this fantastic signal of actually verifying if the thing was correct and so this means something like RL can work really really well and I think like you could have systems that are perhaps very superhuman in math and still not technically have ai okay so can we take it off all the way to cursor mhm and what is cursor it's a fork of vs code and VSS code is one of the most popular editors for a long time like everybody fell in love with it everybody left Vim I left dmax for it sorry uh uh so it unified in some fun fundamental way the uh the developer

10:52 community and then that you look at the space of things you look at the scaling laws AI is becoming amazing and you decide decided okay it's not enough to just write an extension Fe vs code because there's a lot of limitations to that we we need if AI is going to keep getting better and better and better we need to really like rethink how the the AI is going to be part of the editing process and so you decided to Fork vs code and start to build a lot of the amazing features we'll be able to to to talk about but what was that decision like because there's a lot of extensions including co-pilot of vs code that are doing so AI type stuff what was the decision like to just Fork vs code so the decision to do an editor seemed kind of self-evident to

11:37 us for at least what we wanted to do and Achieve because when we started working on the editor the idea was these models are going to get much better their capabilities are going to improve and it's going to entirely change how you build software both in a you will have big productivity gains but also radical in how like the active building software is going to change a lot and so you're very limited in the control you have over a code editor if you're a plugin to an existing coding environment um and we didn't want to get locked in by those limitations we wanted to be able to um just build the most useful stuff okay well then the natural question is you know VSS code is kind of with copilot a competitor so how do you win is is it basically just the speed and the quality of the features yeah I mean I think this is a space that is quite

12:23 interesting perhaps quite unique where if you look at previous Tech waves maybe there's kind of one major thing that happened and unlocked a new wave of companies but every single year every single model capability uh or jump you get model capabilities you now unlock this new wave of features things that are possible especially in programming and so I think in AI programming being even just a few months ahead let alone a year ahead makes your product much much much more useful I think the cursor a year from now will need to make the cursor of today look Obsolete and I think you know Microsoft has' done a number of like fantastic things but I don't think they're in a great place to really keep innovating

13:10 and pushing on this in the way that a startup can just rapidly implementing features and and push yeah like and and kind of doing the research experimentation necessary um to really push the ceiling I don't I don't know if I think of it in terms of features as I think of it in terms of like capabilities for for programmers it's that like you know as you know the new one model came out and I'm sure there are going to be more more models of different types like longer context and maybe faster like there's all these crazy ideas that you can try and hopefully 10% of the crazy ideas will make it into something kind of cool and useful and uh we want people to have that sooner to rephrase it's like an

13:56 underrated fact is we're making it for oursel when we started cursor you really felt this frustration that you know models you could see models getting better uh but the coall experience had not changed it was like man these these guys like the steing is getting higher like why are they not making new things like they should be making new things they should be like you like like where's where's where's all the alpha features there there were no Alpha features it was like uh I I'm sure it it was selling well I'm sure it was a great business but it didn't feel I I'm I'm one of these people that really want to try and use new things and was just there's no new thing for like a very long while yeah it's interesting uh I don't know how you put that into words but when you compare a cursor with copilot copilot pretty quickly became

14:43 started to feel stale for some reason yeah I think one thing that I think uh helps us is that we're sort of doing it all in one where we're developing the the ux and the way you interact with the model and at the same time as we're developing like how we actually make the model give better answers so like how you build up the The Prompt or or like how do you find the context and for a cursor tab like how do you train the model um so I think that helps us to have all of it like sort of like the same people working on the entire experience on end yeah it's like the the person making the UI and the person training the model like sit to like 18 ft away so often the same person even yeah often often even the same person so you you can you create things that that

15:29 are sort of not possible if you're not you're not talking you're not experimenting and you're using like you said cursor to write cursor of course oh yeah yeah well let's talk about some of these features let's talk about the all- knowing the all powerful praise B to the tab so the you know autocomplete on steroids basically so what how does tab work what is tab to highlight and summarize it a high level I'd say that there are two things that curser is pretty good at right now there there are other things that it does um but two things it it helps programmers with one is this idea of looking over your shoulder and being like a really fast colleague who can kind of jump ahead of you and type and figure out what you're what you're going to do next and that

16:15 was the original idea behind that was kind kind of the kernel the idea behind a good autocomplete was predicting what you're going to do next you can make that concept even more ambitious by not just predicting the characters after cursor but actually predicting the next entire change you're going to make the next diff the next place you're going to jump to um and the second thing cursor is is pretty good at right now too is helping you sometimes jump ahead of the AI and tell it what to do and go from instructions to code and on both of those we've done a lot of work on making the editing experience for those things ergonomic um and also making those things smart and fast one of the things we really wanted was we wanted the model to be able to edit code for us uh that was kind of a wish and we had multiple

17:01 attempts at it before before we had a sort of a good model that could edit code for you U then after after we had a good model I think there there have been a lot of effort to you know make the inference fast for you know uh having having a good good experience and uh we've been starting to incorporate I mean Michael sort of mentioned this like ability to jump to different places and that jump to different places I think came from a feeling off you know once you once you accept an edit um was like man it should be just really obvious where to go next it's like it's like I I made this change the model should just know that like the next place to go to is like 18 lines down like uh if you're if you're a whim user you could press 18 JJ or

17:48 whatever but like why why even why am I doing this like the model the model should just know it and then so so the idea was you you just press tab it would go 18 lines down and then make it would show you show you the next edit and you would press tab so it's just you as long as you could keep pressing Tab and so the internal competition was how many tabs can we make them pressive once you have like the idea uh more more uh sort of abstractly the the thing to think about is sort of like once how how how are the edit sort of zero zero entropy so once You' sort of expressed your intent and the edit is there's no like new bits of information to finish your thought but you still have to type some characters to like make the computer understand what you're actually thinking then maybe the model should just sort of

18:34 read your mind and and all the zero entropy bits should just be like tabbed away yeah that was that was sort of the abstract there's this interesting thing where if you look at language model loss on on different domains um I believe the bits per bite which is kind of character normalized loss for code is lower than language which means in general there are a lot of tokens in code that are super predictable lot of characters that are super predictable um and this is I think even magnified when you're not just trying to autocomplete code but predicting what the user is going to do next in their editing of existing code and so you know the gold cursor tab is let's eliminate all the low entropy actions you take inside of the editor when the intent is effectively determined let's just jump you forward

19:20 in time skip you forward well well what's the intuition and what's the technical details of how to do next cursor prediction that jump that's not that's not so intuitive I think to people yeah I think I can speak to a few of the details on how how to make these things work they're incredibly low latency so you need to train small models on this on this task um in particular they're incredibly pre-fill token hungry what that means is they have these really really long prompts where they see a lot of your code and they're not actually generating that many tokens and so the perfect fit for that is using a sparse model meaning Ane model um so that was kind of one one break one breakthrough we made that substantially improved its performance at longer context the other being um a

20:08 variant of speculative decoding that we we kind of built out called speculative edits um these are two I think important pieces of what make it quite high quality um and very fast okay soe mixture of experts the input is huge the output is small yeah okay so like what what what else can you say about how to make it like caching play a role in this cashing plays a huge role M um because you're dealing with this many input tokens if every single keystroke that you're typing in a given line you had to rerun the model on all those tokens passed in you're just going to one significantly deg grade latency two you're going to kill your gpus with load so you need to you you need to design the actual prompts use for the model

20:54 such that they're cach caching aware and then yeah you need you need to re use the KV cach across request just so that you're spending less work less compute uh again what are the things that tab is supposed to be able to do kind of in the near term just to like sort of Linger on that generate code like fill empty space Also edit code across multiple lines yeah and then jump to different locations inside the same file yeah and then like hopefully jump to different files also so if you make an edit in one file and maybe maybe you have to go maybe you have to go to another file to finish your thought it should it should go to the second file also yeah and then the full generalization is like next

21:39 next action prediction like sometimes you need to run a command in the terminal and it should be able to suggest the command based on the code that you wrote too um or sometimes you actually need to like it suggest something but you you it's hard for you to know if it's correct because you actually need some more information to learn like you need to know the type to be able to verify that it's correct and so maybe it should actually take you to a place that's like the definition of something and then take you back so that you have all the requisite knowledge to be able to accept the next completion Al also providing the human the knowledge yes right yeah can you integrate like I just uh gotten to know a guy named Prime Jen who I believe has an SS you can

22:25 order coffee via SSH oh yeah oh we did that we did that uh so can that also the model do that like feed you and like yeah and provide you with caffeine okay so that's the general framework yeah and the the magic moment would be if it is programming is this weird discipline where um sometimes the next five minutes not always but sometimes the next five minutes of what you're going to do is actually predictable from the stuff you've done recently and so can you get to a world where that next 5 minutes either happens by you disengaging and it taking you through or maybe a little bit more of just you seeing Next Step what it's going to do and you're like okay that's good that's good that's good that's good and you can just sort of tap tap tap through these big changes as we're talking about this I should mention like one of the really

23:12 cool and noticeable things about cursor is that there's this whole diff interface situation going on so like the model suggests with uh with the red and the green of like here's how we're going to modify the code and in the chat window you can apply and it shows you the diff and you can accept the diff so maybe can you speak to whatever direction of that we'll probably have like four or five different kinds of diffs uh so we we have optimized the diff for for the autocomplete so that has a different diff interface than uh then when you're reviewing larger blocks of code and then we're trying to optimize uh another diff thing for when you're doing multiple different files uh and and sort of at a high level the difference is for

23:59 when you're doing autocomplete it should be really really fast to read uh actually it should be really fast to read in all situations but in autocomplete it sort of you're you're really like your eyes focused in one area you you can't be in too many you the humans can't look in too many different places so you're talking about on the interface side like on the interface side so it currently has this box on the side so we have the current box and if it tries to delete code in some place and tries to add other code it tries to show you a box on the you can maybe show it if we pull it up on cursor. comom this is what we're talking about so that it was like three or four different attempts at trying to make this this thing work where first the attempt was like these blue crossed out

24:45 line so before it was a box on the side it used to show you the code to delete by showing you like uh like Google doc style you would see like a line through it then you would see the the new code that was super distracting and then we tried many different you know there was there was sort of deletions there was trying to Red highlight then the next iteration of it which is sort of funny Would you would hold the on Mac the option button so it would it would sort of highlight a region of code to show you that there might be something coming uh so maybe in this example like the input and the value uh would get would all get blue and the blue would to highlight that the AI had a suggestion for you uh so instead of directly

25:32 showing you the thing it would show you that the AI it would just hint that the AI had a suggestion and if you really wanted to see it you would hold the option button and then you would see the new suggestion then if you release the option button you would then see your original code mhm so that's by the way that's pretty nice but you have to know to hold the option button yeah I by the way I'm not a Mac User but I got it it was it was it's a button I guess you people it's h you know it's again it's just it's just nonintuitive I think that's the that's the key thing and there's a chance this this is also not the final version of it I am personally very excited for um making a lot of improvements in this area like uh we we often talk about it as the verification problem where U

26:19 these diffs are great for small edits uh for large edits or like when it's multiple files or something it's um actually a little bit prohibitive to to review these diffs and uh uh so there are like a couple of different ideas here like one idea that we have is okay you know like parts of the diffs are important they have a lot of information and then parts of the diff um are just very low entropy they're like exam like the same thing over and over again and so maybe you can highlight the important pieces and then gray out the the not so important pieces or maybe you can have a model that uh looks at the the diff and and sees oh there's a likely bug here I will like Mark this with a little red squiggly and say like you should

27:05 probably like review this part of the diff um and ideas in in that vein I think are exciting yeah that's a really fascinating space of like ux design engineering so you're basically trying to guide the human programmer through all the things they need to read and nothing more yeah like optimally yeah and you want an intelligent model to do it like ly diffs Al diff algorithms are they're like Al like they're just like normal algorithms uh there's no intelligence uh there's like intelligence that went into designing the algorithm but then there there's no like you don't care if the if it's about this thing or this thing uh and so you want a model to to do this so I think the the the general question is like M

27:51 these models are going to get much smarter as the models get much smarter uh the the changes they will be able to propose are much bigger so as the changes gets bigger and bigger and bigger the humans have to do more and more and more verification work it gets more and more more hard like it's just you need you need to help them out it sort of I I don't want to spend all my time reviewing code uh can you say a little more across multiple files div yeah I mean so GitHub tries to solve this right with code review when you're doing code review you're reviewing multiple deaths cross multiple files but like Arvid said earlier I think you can do much better than code review you know code review kind of sucks like you spend a lot of

28:37 time trying to grock this code that's often quite unfamiliar to you and it often like doesn't even actually catch that many bugs and I think you can signific significantly improve that review experience using language models for example using the kinds of tricks that AR had described of maybe uh pointing you towards the regions that matter um I think also if the code is produced by these language models uh and it's not produced by someone else like the code review experience is designed for both the reviewer and the person that produced the code in the case where the person that produced the code is a language model you don't have to care that much about their experience and you can design the entire thing around the

29:24 reviewer such that the reviewer's job is as fun as easy as productive as possible um and I think that that feels like the issue with just kind of naively trying to make these things look like code review I think you can be a lot more creative and and push the boundary and what's possible just one one idea there is I think ordering matters generally when you review a PR you you have this list of files and you're reviewing them from top to bottom but actually like you actually want to understand this part first because that came like logically first and then you want understand the next part and um you don't want to have to figure out that yourself you want a model to guide you through the thing and is the step of creation going to be more and more natural language is the goal

30:10 versus with actual uh I think sometimes I don't think it's going to be the case that all of programming will be natural language and the reason for that is you know if I'm PR programming with swalla and swall is at the computer and the keyboard uh and sometimes if I'm like driving I want to say to swallet hey like implement this function and that that works and then sometimes it's just so annoying to explain to swalla what I want him to do and so I actually take over the keyboard and I show him I I write like part of the example and then it makes sense and that's the easiest way to communicate and so I think that's also the case for AI like sometimes the easiest way to communicate with the AI will be to show an example and then it goes and does the thing everywhere else or sometimes if you're making a website

30:56 for example the easiest way to show to the a what you want is not to tell it what to do but you know drag things around or draw things um and yeah and and like maybe eventually we will get to like brain machine interfaces or whatever and can of like understand what you're thinking and so I think natural language will have a place I think it will not definitely not be the way most people program most of the time I'm really feeling the AGI with this editor uh it feels like there's a lot of machine learning going on underneath tell tell me about some of the ml stuff that makes it all work recursor really works via this Ensemble of custom models that that that we've trained alongside you know the frontier models that are fantastic at the reasoning intense things and so cursor tab for example is

31:42 is a great example of where you can specialize this model to be even better than even Frontier models if you look at evls on on the on the task we set it at the other domain which it's kind of surprising that it requires custom models but but it's kind of necessary and works quite well is in apply um so I think these models are like the frontier models are quite good at sketching out plans for code and generating like rough sketches of like the change but actually creating diffs is quite hard um for Frontier models for your training models um like you try to do this with Sonet with 01 any Frontier Model and it it really messes up stupid things like counting line numbers um especially in super super large file

32:29 um and so what we've done to alleviate this is we let the model kind of sketch out this rough code block that indicates what the change will be and we train a model to then apply that change to the file and we should say that apply is the model looks at your code it gives you a really damn good suggestion of what new things to do and the seemingly for humans trivial step of combining the two you're saying is not so trivial contrary to popular perception it is not a deterministic algorithm yeah I I I think like you see shallow copies of apply um elsewhere and it just breaks like most of the time because you think you can kind of try to do some deterministic matching and then it fails you know at

33:16 least 40% of the time and that just results in a terrible product experience um I think in general this this regime of you are going to get smarter models and like so one other thing that apply lets you do is it lets you use fewer tokens with the most intelligent models uh this is both expensive in terms of latency for generating all these tokens um and cost so you can give this very very rough sketch and then have your smaller models go and implement it because it's a much easier task to implement this very very sketched out code and I think that this this regime will continue where you can use smarter and SM models to do the planning and then maybe the implementation details uh can be handled

34:02 by the less intelligent ones perhaps you'll have you know maybe 01 maybe it'll be even more cap capable models given an even higher level plan that is kind of recursively uh applied by Sonet and then the apply model maybe we should we should talk about how to how to make it fast yeah I feel like fast is always an interesting detail fast good yeah how do you make it fast yeah so one big component of making it it fast is speculative edits so speculative edits are a variant of speculative decoding and maybe be helpful to briefly describe speculative decoding um with speculative decoding what you do is you you can kind of take advantage of the fact that you know most of the time and I I'll add the caveat that it would be when you're

34:48 memory Bound in in language model Generation Um if you process multiple tokens at once um it is faster than generating one Tok at a time so this is like the same reason why if you look at tokens per second uh with prompt tokens versus generated tokens it's much much faster for prompt tokens um so what we do is instead of using what specul decoding normally does which is using a really small model to predict these draft tokens that your larger model would then go in and and verify um with code edits we have a very strong prior of what the existing code will look like and that prior is literally the same exact code so you can do is you can just feed chunks of the original code back into the into the model um and then the

35:35 model will just pretty much agree most of the time that okay I'm just going to spit this code back out and so you can process all of those lines in parallel and you just do this with sufficiently many chunks and then eventually you'll reach a point of disagreement where the model will now predict text that is different from the ground truth original code it'll generate those tokens and then we kind of will decide after enough tokens match uh the original code to re start speculating in chunks of code what this actually ends up looking like is just a much faster version of normal editing code so it's just like it looks like a much faster version of the model rewriting all the code so just we we can use the same exact interface that we use for for diffs but it will just stream

36:20 down a lot faster and then and then the advantage is that W wireless streaming you can just also be reviewing start reviewing the code exactly before before it's done so there's no no big loading screen uh so maybe that that is part of the part of the advantage so the human can start reading before the thing is done I think the interesting riff here is something like like speculation is a fairly common idea nowadays it's like not only in language models I mean there's obviously speculation in CPUs and there's there like speculation for databases and like speculation all over the place let me ask the sort of the ridiculous question of uh which llm is better at coding GPT Claude who wins in the context of programming and I'm sure the answer is much more Nuance because

37:06 it sounds like every single part of this involves a different model yeah I think they there's no model that poo dominates uh others meaning it is better in all categories that we think matter the categories being speed um ability to edit code ability to process lots of code long context you know a couple of other things and kind of coding capabilities the one that I'd say right now is just kind of net best is Sonet I think this is a consensus opinion our one's really interesting and it's really good at reasoning so if you give it really hard uh programming interview style problems or lead code problems it

37:51 can do quite quite well on them um but it doesn't feel like it kind of understands your rough intent as well as son it does like if you look at a lot of the other Frontier models um one qual I have is it feels like they're not necessarily over I'm not saying they they train in benchmarks um but they perform really well in benchmarks relative to kind of everything that's kind of in the middle so if you tried on all these benchmarks and things that are in the distribution of the benchmarks they're valuated on you know they'll do really well but when you push them a little bit outside of that son's I think the one that that kind of does best at at kind of maintaining that same capability like you kind of have the same capability in The Benchmark as when you try to instruct it to do anything with coding

38:37 what another ridiculous question is the difference between the normal programming experience versus what benchmarks represent like where do benchmarks fall short do you think when we're evaluating these models by the way that's like a really really hard it's like like critically important detail like how how different like benchmarks are versus where is like real coding where real coding it's not interview style coding it's you're you're doing these you know humans are saying like half broken English sometimes and sometimes you're saying like oh do what I did before sometimes you're saying uh you know go add this thing and then do this other thing for me and then make this UI element and then you know it's it's just

39:25 like a lot of things are sort of context dependent you really want to like understand the human and then do do what the human wants as opposed to sort of this maybe the the way to put it is sort of abstractly is uh the interview problems are very wellp specified they lean a lot on specification while the human stuff is less specified yeah I think that this this SP for question is both Complicated by what um Sol just mentioned and then also to what Aman was getting into is that even if you like you know there's this problem of like the skew between what can you actually model in a benchmark versus uh real programming and that can be sometimes hard to encapsulate because it's like real programming is like very messy and sometimes things aren't super

40:12 well specified what's correct or what isn't but then uh it's also doubly hard because of this public Benchmark problem and that's both because public benchmarks are sometimes kind of Hill climbed on then it's like really really hard to also get the data from the public benchmarks out of the models and so for instance like one of the most popular like agent benchmarks sweet bench um is really really contaminated in the training data of uh these Foundation models and so if you ask these Foundation models to do a sweet bench problem you actually don't give them the context of a codebase they can like hallucinate the right file pass they can hallucinate the right function names um and so the the it's it's also just the public aspect of these things is tricky yeah like in that case it could be trained on the literal issues or pool request themselves and and maybe

41:00 the lives will start to do a better job um or they've already done a good job at decontaminating those things but they're not going to emit the actual training data of the repository itself like these are all like some of the most popular python repositories like simpai is one example I don't think they're going to handicap their models on Senpai and all these popular P python repositories in order to get uh true evaluation scores in these benchmarks yeah I think that given the dirs and benchmarks um there have been like a few interesting crutches that uh places that build systems with these models or build these models actually use to get a sense of are they going in the right direction or not and uh in a lot of places uh people will actually just have humans play with the things and give qualitative feedback on these um like one or two of the foundation model

41:45 companies they they have people who that's that's a big part of their role and you know internally we also uh you know qualitatively assess these models and actually lean on that a lot in addition to like private evals that we have it's like the live the vibe yeah the vi the vibe Benchmark human Benchmark the hum you pull in the humans to do a Vibe check yeah okay I mean that's that's kind of what I do like just like reading online forums and Reddit and X just like well I don't know how to properly load in people's opinions because they'll say things like I feel like Claude or gpt's gotten Dumber or something they'll say I feel like and then I sometimes feel like that too but I wonder if it's the model's problem

42:32 or mine yeah with Claude there's an interesting take I heard where I think AWS has different chips um and I I suspect they've slightly different numerics than uh Nvidia gpus and someone speculated that claud's deg degraded performance had to do with maybe using the quantise version that existed on AWS Bedrock versus uh whatever was running on on anthropics gpus I interview a bunch of people that have conspiracy theories so I'm glad spoke spoke to this conspiracy well it's it's not not like conspiracy theory as much as they're just they're like they're you know humans humans are humans and there's there's these details and you know you're doing like these quzy amount of flops

43:19 and you know chips are messy and man you can just have bugs like bugs are it's it's hard to overstate how how hard bugs are to avoid what's uh the role of a good prompt in all this see you mention that benchmarks have really uh structured well formulated prompts what what should a human be doing to maximize success and what's the importance of what the humans you wrote a blog post on you called it prompt design yeah uh I think it depends on which model you're using and all of them are likly different and they respond differently to different prompts but um I think the original gp4 uh and the

44:05 original sort of bre of models last last year they were quite sensitive to the prompts and they also had a very small context window and so we have all of these pieces of information around the codebase that would maybe be relevant in the prompt like you have the docs you have the files that you add you have the conversation history and then there's a problem like how do you decide what you actually put in the prompt and when you have a a limited space and even for today's models even when you have long context filling out the entire context window means that it's slower it means that sometimes a model actually gets confused and some models get more confused than others and we have this one system internally that we call preum which helps us with that a little bit um and I think it was built for the era

44:55 before where we had 8,000 uh token context Windows uh and it's a little bit similar to when you're making a website you you sort of you you want it to work on mobile you want it to work on a desktop screen and you have this uh Dynamic information which you don't have for example if you're making like designing a print magazine you have like you know exactly where you can put stuff but when you have a website or when you have a prompt you have these inputs and then you need to format them will always work even if the input is really big then you might have to cut something down uh and and and so the idea was okay like let's take some inspiration what's the best way to design websites well um the thing that we really like is is react and the

45:41 declarative approach where you um you use jsx in in in JavaScript uh and then you declare this is what I want and I think this has higher priority or like this has higher Z index than something else um and then you have this rendering engine in web design it's it's like Chrome and uh in our case it's a pre renderer uh which then fits everything onto the page and and so you declaratively decide what you want and then it figures out what you want um and and so we have found that to be uh quite helpful and I think the role of it has has sort of shifted over time um where initially was to fit to these small context Windows now it's really useful because you know it helps us with splitting up the data that goes into the

46:28 prompt and the actual rendering of it and so um it's easier to debug because you can change the rendering of the prompt and then try it on Old prompts because you have the raw data that went into the prompt and then you can see did my change actually improve it for for like this entire evil set so do you literally prompt with jsx yes yes so it kind of looks like react there are components like we have one component that's a file component and it takes in like the cursor like usually there's like one line where the cursor is in your file and that's like probably the most important line because that's the one you're looking at and so then you can give priorities so like that line has the highest priority and then you subtract one for every line that uh is farther away and then eventually when it's render it to figure

47:13 out how many lines can I actually fit and it centers around that thing that's amazing yeah and you can do like other fancy things where if you have lots of code blocks from the entire code base you could use uh retrieval um and things like embedding and reranking scores to add priorities for each of these components so should humans when they ask questions also use try to use something like that like would it be beneficial to write jsx in the in the problem where the whole idea is should be loose and messy I I think our goal is kind of that you should just uh do whatever is the most natural thing for you and then we are job is to figure out how do we actually like retrieve the relative EV things so that your thing actually makes sense well this is sort of the discussion I had with uh Arvin of

48:00 perplexity is like his whole idea is like you should let the person be as lazy as he want but like yeah that's a beautiful thing but I feel like you're allowed to ask more of programmers right so like if you say just do what you want I mean humans are lazy there's a kind of tension between just being lazy versus like provide more is uh be prompted almost like the system pressuring you or inspiring you to be articulate not in terms of the grammar of the sentences but in terms of the depth of thoughts that you convey inside the uh the problems I think even as a system gets closer to some level of perfection often when you ask the model

48:45 for something you just are not not enough intent is conveyed to know what to do and there are like a few ways to resolve that intent one is the simple thing of having model just ask you I'm not sure how to do these parts based in your query could you clarify that um I think the other could be maybe if you there are five or six possible Generations given the uncertainty present in your query so far why don't we just actually show you all of those and let you pick them how hard is it to for the model to choose to speak talk back sort of versus gener that's a that's hard sort of like how to deal with the uncertainty do I do I choose to ask for

49:33 more information to reduce the ambiguity so I mean one of the things we we do is um it's like a recent addition is try to suggest files that you can add so and while you're typing uh one can guess what the uncertainty is and maybe suggest that like you know maybe maybe you're writing your API and uh we can guess using the commits uh that you've made previously in the same file that the client and the server is super useful and uh there's like a hard technical problem of how do you resolve it across all commits which files are the most important given your current prompt and we still sort of uh

50:21 initial version is ruled out and I'm sure we can make it much more accurate uh it's it's it's very experimental but then the ideaas we show you like do you just want to add this file this file this file also to tell you know the model to edit those files for you uh because if if you're maybe you're making the API like you should also edit the client and the server that is using the API and the other one resolving the API and so that would be kind of cool as both there's the phase where you're writing the prompt and there's before you even click enter maybe we can help resolve some of the uncertainty to what degree do you use uh agentic approaches how useful are agents we think agents are really really cool like I I I think agents is like uh it's like resembles sort of like a human it's

51:08 sort of like the like you can kind of feel that it like you're getting closer to AGI because you see a demo where um it acts as as a human would and and it's really really cool I think um agents are not yet super useful for many things they I think we're we're getting close to where they will actually be useful and so I think uh there are certain types of tasks where having an agent would be really nice like I would love to have an agent for example if like we have a bug where you sometimes can't command C and command V uh inside our chat input box and that's a task that's super well specified I just want to say like in two sentences this does not work

51:53 please fix it and then I would love to have an agent that just goes off does it and then uh a day later I I come back and I review the the thing you mean it goes finds the right file yeah it finds the right files it like tries to reproduce the bug it like fixes the bug and then it verifies that it's correct and this is could be a process that takes a long time um and so I think I would love to have that uh and then I think a lot of programming like there is often this belief that agents will take over all of programming um I don't think we think that that's the case because a lot of programming a lot of the value is in iterating or you don't actually want to specify something upfront because you don't really know what you want until youve seen an initial version and then

52:39 you want to iterate on that and then you provide more information and so for a lot of programming I think you actually want a system that's instant that gives you an initial version instantly back and then you can iterate super super quickly uh what about something like that recently came out rep agent that does also like setting up the development environment installing software packages configuring everything configuring the databases and actually deploying the app yeah is that also in the set of things you dream about I think so I think that would be really cool for for certain types of programming uh it it would be really cool is that within scope of cursor yeah we're aren't actively working on it right now um but it's definitely like we want to make the programmer's life

53:26 easier and more fun and some things are just really tedious and you need to go through a bunch of steps and you want to delegate that to an agent um and then some things you can actually have an agent in the background while you're working like let's say you have a PR that's both backend and front end and you're working in the front end and then you can have a background agent that doesn't work and figure out kind of what you're doing and then when you get to the backend part of your PR then you have some like initial piece of code that you can iterate on um and and so that that would also be really cool one of the things we already talked about is speed but I wonder if we can just uh Linger on that some more in the the various places that uh the technical details involved in making this thing

54:11 really fast so every single aspect of cursor most aspects of cursor feel really fast like I mentioned the apply is probably the slowest thing and for me from sorry the pain I know it's it's a pain it's a pain that we're feeling and we're working on fixing it uh yeah I mean it says something that something that feels I don't know what it is like 1 second or two seconds that feels slow that means that's actually shows that everything else is just really really fast um so is there some technical details about how to make some of these models so how to make the chat fast how to make the diffs fast is there something that just jumps to mind yeah I mean so we can go over a lot of the strategies that we use one interesting thing is Cash Waring um and so what you

54:56 can is if as the user is typing you can have yeah you're you're probably going to use uh some piece of context and you can know that before the user's done typing so you know as we discussed before reusing the KV cache results and lower latency lower cost uh cross requests so as a user starts type in you can immediately warm the cache with like let's say the current file contents and then when theyve pressed enter uh there's very few tokens it actually has to to prefill and compute before starting the generation this will significantly lower ttf can you explain how KV cach works yeah so the way Transformers work um I like it I mean like one one of the mechanisms that

55:42 allow Transformers to not just independently like the mechanism that allows Transformers to not just independently look at each token but see previous tokens are the keys and values to tension and generally the way tension works is you have at your current token some query and then you've all the keys and values of all your previous tokens which are some kind of representation that the model stores internally of all the previous tokens in the prompt and like by default when you're doing a chat the model has to for every single token do this forward pass through the entire uh model that's a lot of Matrix multiplies that happen and that is really really slow instead if you have already done that and you stored the

56:27 keys and values and you keep that in the GPU then when I'm let's say I have stored it for the last end tokens if I now want to compute the the output token for the N plus one token I don't need to pass those first end tokens through the entire model because I already have all those keys and values and so you just need to do the forward pass through that last token and then when you're doing attention uh you're reusing those keys and values that have been computed which is the only kind of sequential part um or sequentially dependent part of the Transformer is there like higher level caching of like caching of the prompts or that kind of stuff could help yeah that that there's other types of caching you can kind of do um one interesting

57:13 thing that you can do for cursor tab is you can basically predict ahead as if the user would have accepted the suggestion and then trigger another uh request and so then you've cashed you've done the speculative it's it's a mix of speculation and caching right because you're speculating what would happen if they accepted it and then you have this value that is cach this this uh suggestion and then when they press tab the next one would be waiting for them immediately it's a it's a kind of clever heuristic slash trick uh that uses a higher level caching and and can give uh the it feels fast despite there not actually being any changes in the in the model and if you can make the KV cach smaller one of the advantages you get is like maybe maybe you can speculate even

57:59 more maybe you can get seriously 10 things that you know could be useful I like uh like predict the next 10 and and then like it's possible the user hits the the one of the 10 it's like much higher chance than the user hits like the exact one that you show them uh maybe they typeing another character and and he sort of hits hits something else in the cache yeah so there's there's all these tricks where um the the general phenomena here is uh I think it's it's also super useful for RL is you know may maybe a single sample from the model isn't very good but if you predict like 10 different things uh turns out that one of the 10 uh that's right is the probability is much higher there's these passid key curves and you

58:44 know part of RL like what what RL does is you know you can you can exploit this passid K phenomena to to make many different predictions and and uh one one way to think about this the model sort of knows internally has like has some uncertainty over like which of the key things is correct or like which of the key things does the human want when we ARL our uh you know cursor Tab model one of the things we're doing is we're predicting which like which of the hundred different suggestions the model produces is more amendable for humans like which of them do humans more like than other things uh maybe maybe like there's something with the model can predict very far ahead versus like a little bit and maybe somewhere in the middle and and you just and then you can

59:33 give a reward to the things that humans would like more and and sort of punish the things that it would like and sort of then train the model to Output the suggestions that humans would like more you you have these like RL Loops that are very useful that exploit these passive K curves um Oman maybe can can go into even more detail yeah it's a little it is a little different than speed um but I mean like technically you tie it back in because you can get away with the smaller model if you are all your smaller model and it gets the same performance as the bigger one um that's like and SW I was mentioning stuff about KV about reducing the size of your KV cach there there are other techniques there as well that are really helpful for Speed um so kind of back in the day like all the way two years ago uh people mainly use multi-ad attention um and I

1:00:20 think there's been a migration towards more uh efficient attention schemes like group query um or multiquery attention and this is really helpful for then uh with larger batch sizes being able to generate the tokens much faster the interesting thing here is um this now has no effect on that uh time to First token pre-fill speed uh the thing this matters for is uh now generating tokens and and why is that because when you're generating tokens instead of uh being bottlenecked by doing the super realizable Matrix multiplies across all your tokens you're bottleneck by how quickly it's for long context um with large batch sizes by how quickly you can read those cache keys and values um and

1:01:08 so then how that that's memory bandwidth and how can we make this faster we can try to compress the size of these keys and values so multiquery attention is the most aggressive of these um where normally with multi-head attention you have some number of quote unquote attention heads um and some number of kind of query query heads U multiquery just preserves the query heads gets rid of all the key value heads um so there's only one kind of key value head and there's all the remaining uh query heads with group query um you instead you know preserve all the query heads and then your keys and values are kind of in there are fewer heads for the keys and

1:01:53 values but you're not reducing it to just one um but anyways like the whole point here is you're just reducing the size of your KV cache and then there is MLA yeah multi- latent um that's a little more complicated and the way that this works is it kind of turns the entirety of your keys and values across all your heads into this kind of one latent Vector that is then kind of expanded in frence time but MLA is from this company uh called Deep seek um it's it's quite an interesting algorithm uh maybe the key idea is sort of uh in both mqa uh and in other places what you're doing is sort of reducing the uh num like the number of KV heads the

1:02:38 advantage you get from that is is you know there's less of them but uh maybe the theory is that you actually want a lot of different uh like you want each of the the keys and values to actually be different so one way to reduce the size is you keep uh one big shared Vector for all the keys and values and then you have smaller vectors for every single token so that when you m you can you can store the only the smaller thing as some sort of like low rank reduction and the low rank reduction with that and at the end of the time when you when you eventually want to compute the final thing uh remember that like your memory bound which means that like you still have some some compute left that you can use for these things and so if you can expand the um the latent vector

…