The BEST AI Coding Assistants | July 2025

0:00 Well, this is always my favorite video. Again, this is only my second one, but I love evaluating all of the AI assistance that I can. So, then I can have that information for what I actually want to go and commit to for the following month. So, this month we've added a few things. So, we've added codeci, we've added Gemini CLI, we've added Kilo code, added open code, I added IDER, and I added warp.dev. So, the list has gotten a lot longer. I think if I'm counting right, I have 17 total that I ran through. And this took an enormous amount of time to get through. And I've changed a few things. So, as always, there's some that I haven't done. I'm sure you can name a bunch more. There's some that I even know, but I'm also not

0:46 testing things like Lovable or Bolt. Um, I'm trying to get as many as I can. If you're interested in any additional ones, let me know in the comments below or jump into my Discord. We've got an amazing group there. We love having new people come in. If you want to hang out with a bunch of professional engineers and people that are just enjoy coding, uh, join my Discord link in the description below. Now, what do what am I measuring here? I am measuring instruction following. And the way that I do that is I give a very very detailed prompt. And I'll show you an example of one that I'm not using right now. It was one that I had early on. I do not want these revealed fully because I'm worried that people that are building these AI coding tools or the LLMs are going to fine-tune their tools to my test and I

1:33 don't want that. But what I care about is if I tell you to do something, do you do it? Do you do it accurately? And then the quality of it. The way I do that is I have a series of unit tests that sit behind each eval. I work and iterate over and over and over and over again to fine-tune the instructions so that they will if you follow the instructions your unit test will pass. Then I use LLM as a judge giving pretty strict criteria about what is good and what is bad and then that is part of the weight factor as well. For this particular one I actually have moved to clawed code because now I'm able to automate a lot more of this. So I can run a bunch of tests and then I can use cloud code uh to actually grade everything. And I I did a bunch of testing to make sure the

2:18 way I've been doing it before was in a custom system that had set up. Uh cloud code makes this a lot easier because I can just do it all through the CLI. Now this is an example part of a prompt. So for example, the the eval that I'm doing are larger things. So it can be things like fixing bugs in existing projects but it mainly is things like implement some thing with more it's not just a single file it's just not it's not a single algorithm it's if I give it a really detailed spec and I tell it specifically what functions need to be in there what things need to be named the criteria for it the functional and non-functional criteria how much of that can it do by

3:04 itself. Like if I put it on auto mode and just want it to run, how much can it do? And then how high a quality is that code? Now, no LLM paired with an AI coding assistant can score across the point across the board 100% on everything. Uh so what I've done now is I've actually moved over to a ranking system. The ranking system is just going to be points based. These points, you don't have to worry about what the numbers mean. Just know that each of my evals have the v the same amount of points. The things that you accomplish get added into the point score. What's really most important is the variance between them. So month over month you're going to see these uh points change because I want to continually add more things to the test and get better at about it. And over time I may pull some

3:50 out. For example, I pulled a couple out this time because they were too easy. So now this month I am primarily testing clawed 4 because clawed 4 is just the one that seems to perform the most consistently across all of them. There were some from last month that actually had claw 3.7 that was pretty good. But Gemini 2.5 did not do very well across them. But I do have some exceptions which I'll talk about a little bit later because I am testing codeci with GPT4.1 and I am testing Gemini CLI with Gemini 2.5 Pro. Now, this is one of the biggest spreads. I'll talk about this again in a little bit, but number three for Claude 4, GitHub Copilot. What the crap? I think they fixed it. This is the biggest turnaround that I never expected. I

4:36 reran this several times because I thought there was some sort of anomaly, but no, GitHub Copilot, I am bringing my own key to this, so I don't know if that impacts the quality. It might. I'm just making that very known. In this particular one, I am using my own key, which GitHub copilot has also been recently open source. So that's also something very interesting. So from last month to this month, GitHub Copilot has went from being what I thought was basically unusable to actually pretty dang solid. I was impressed with that overall. Now, what do you think number two is? Um, this might actually surprise you, but we'll see. Open code. I am so thankful that people pushed me to actually bring Open Code in. Open code is phenomenal. It did an exceptional job and the cost of using it is really low

5:23 comparatively to some of the other AI coding tools. I gave it the full plan or I gave it the full uh prompt. It executes really really well. But number one is Claude Code. Claude Code has just come a long way. It crushed all the scores were higher in eval this month compared to last month and the new ones it crushed more than any other ones. It's the only AI coding assistant to actually hit over 17,000 the first um the first time I've kind of ran through this. Now the variance between them you know it seems kind of low on the GitHub copilot and open code stand side and that's because it is you know there there's margin of error there. There's about a 50 point swing between runs on each one. So it there is a likelihood that GitHub Copilot and Open Code could

6:10 be about even at some point, but Open Code did actually ek out the second place here. Now, a few things to just show you. Gemini CLI and Codeex CLI. My gosh, GBZ4.1 is painful. So, my thinking behind it was codec uh I know you can there's forked versions that you can actually bring in anthropic to it. I didn't do that. I probably will if there's interest into it next month. But I wanted to test where OpenAI is in all of this and how it compares to the other ones. It scored a 1,700. This is atrocious. It is unbelievably low. And Gemini CLI 2.5 Pro scored 8780. And this is also not that good. Uh, the

6:58 problem with these are I it it's hard to pin it down on if it's the AI coding agent or the model itself because I actually really like Gemini 2.5 Pro. I still use Gemini 2.5 and Root Code and other models. Like it is not a bad model, but I really do want Gemini CLI to get good. I'm I'm pulling for it. I'm rooting for it. And I want Gemini 2.5 Pro to be amazing in Gemini CLI. So I'm excited to see where this goes long term. This is going to be kind of the starting benchmark for kind of where this falls in the overall leaderboard of things. So Kilo code R code and Klein. Now uh I actually have

7:43 the dot dot dot there because it is the exact same score as R code 15,714. Klein 15,714 and kilo code at 15,414. That just shows you maybe how consistent my tests actually are at this point that three things that are basically identical kind of fall within the same scoring. Uh so it is kind of interesting. But I will I do want to talk a little bit about Kilo code here in a second because that one is an interesting one that I have a lot of thoughts about. So if you're interested in that, stay stay for a little bit past after the overall chart. Now the overall chart here you can see codec TLI GPT4.1 at 1700 significantly lower but look at how everything has kind of converged in

8:30 this particular test like everything primarily hovers between the 14,000 and 17,000 range. What's interesting is if you start at the top, Claw Code with Sonnet 4 number one, Open Code with Sonnet 4 number two, GitHub Copilot with Sonnet 4 number three, but barely behind Open Code. Then we have Trey, who is always does really good for me in any of my tests. This is a very solid agent. Um I I actually use this periodically myself. Uh it is just a super solid. It scored right behind GitHub Copilot and I was still very surprised. Get help co-pilot beat it. It kind of hurts my brain a little bit to tell you. Zed is fascinating. Uh but it is right in line with claw code opus. I

9:18 expected cloud code opus to do better but it very consistently did not. Um it very consistently scored a little bit lower on the evals. Now what I will say my personal opinion I do think cloud code opus with planning is far far superior. So if you are actually using cloud code, cloud code with opus plan and then cloud code donet to actually execute. And then we have a couple new entries here. So warp.dev came in right about the same level as root code about 130 points higher than root code. And then you can see our client 15,714 as I said on the previous one. Kilo code slightly behind uh which is weird to me but it was very consistently slightly behind. Uh so it you know it the

10:06 variants there they're basically identical to be totally honest with you but it was very consistently behind uh R code and Klein uh wind surf which I misspelled here is 14,824 which is basically even with cursor at 14,814 and then augment code which always blows me away because the thing that I learn is augment codes agentic capabilities where you give it a big thing to do or something to do. It's just not that good at it. Uh I shouldn't say that it is very good at it. It's just not as good as some of the other ones. I think there's some work to be done there on the augment code side, but that's not what I use augment code for. I use it for it context engine, and I'll talk about a little bit of that at the very end of this video. I also introduced

10:53 IDER. Now, IDER surprised me because previously when I had used IDER that uh and IDER, A I I D E R. I'm not sure exactly if that's pronounced correctly, but that one, what's fascinating to me is I had expected that one to be worse than it was because I had always found that one to be more of like a surgical tool, not so much of like a a big agentic tool. So, I'm actually very impressed with how Ader came through with that one. And void also kind of surprised me. That one did very bad uh with Sonic 4 and I was very uh very disappointed with that, but it was very consistently, you know, slightly better than Gemini CLI and and significantly worse than most everything else. And then the codec with GPT4.1. I do want to

11:40 test codec CLI a little bit more with different AI models. I did briefly try 03, but again, that even acted kind of weird. it was putting files in the wrong location. So, I didn't even score that one because it didn't even it didn't even meet the baseline principle for what I told it to do that. So, that was kind of disappointing. Again, I don't know if that's codeci or if that is in particular the models itself, but I will say I have used codeci as kind of like a surgical thing where I want it to do very particular things and I tell it to do that and it's fine. So, if you're going to use it, use it with GPT4.1, but you need to be very specific and it needs to be teeny tiny by bite-sized things that you want to be done. So, super unfortunate, but that just is the way it is. So, kilo code, I get these

12:26 ads all the time. Uh, in this particular one, it's like cursor plus windsurf plus klein plus rue, the best parts of each, and it's open source. I don't know what to make of this, honestly, because I've spent some time going through the source code. I spent some time kind of like digging into this as I was preparing for this video and you can kind of see stuff like this where they've got this features page on their website and they they basically they'll say like hey MCP marketplace rue doesn't have that. It should be a check there. It is a fairly new one so they just haven't updated their website yet. But now you see a PR stage uh switching MCP marketplace to the R code version. So rather than the kilo version or the client version whatever version was there before. So it's very fascinating to me uh how this is all kind of playing out. And I also

13:14 do not quite understand this one thing that they claim which is open router without the 5% markup. When you click on that it just tells you about bringing your own keys and the additional 5% of the original model blah blah blah. I don't quite understand how they're getting around that, nor do I find anywhere where they're explaining it. I am okay with every, you know, people forking things. Like, I think that's just the nature of it. But I'm a little worried about Kilo Code's like approach here because they're trying to take all of these things and try to cram it into a single entry, but it's not really bringing anything new to the table. Now, they did implement autocomplete. I tested it. It's, you know, there's nothing special about it. So, I have a hard time like thinking or recommending

14:00 Kilo Code at this uh point, but it is something I'm going to be monitoring. I'm curious if you guys are using Kilo Code or what your thoughts are on it, but I'm partially, you know, it's one of those things in the back of my head where it's like you see a company kind of like doing some weird marketing type things and it kind of has me a little worried. Um, anyway, just throwing that out there. I could probably go on and on about this. I just kind of am curious about where this is going. So anyway, back to the main topic here. GitHub Copilot, what the heck? They actually made this so much better. I'm very excited about this. I hope this isn't short-lived and I hope it sticks, but I tested it over two different days to make sure it wasn't something weird

14:46 going on. Again, I do want to be very specific. I did bring my own keys and I probably need to also test using their built-in kind of agentic stuff. And honestly, a lot of them like Zed for example, I bring my own keys as well. And the reason for that is is I don't have to pay the monthly fee for those. I can just use the bring your own key part of it. Zed is very expensive. It cost me a fortune to run the test in there because it just kept going over and over and over and over again. And I think that's probably a byproduct of uh bringing my own keys and there's no like built-in limit when that happens. So that cost me it it ended up being like $12 for a one eval to run and I'm like okay that's ridiculous cuz it it

15:33 does a excellent job of iterating and looping and iterating and looping. So again I think that is just something to call out there. and Augment Code still does not show well in these tests. And I think that's disappointing because it is an incredible incredible uh AI coding assistant. It's just not good at taking like a PRD or some sort of technical specification and converting that into something big. I'd say augment code is more about planning, ideulating, and then executing one off or two like things uh in a big codebase. Now, I will also say I've tried to bring augment code into the Chromium codebase and the file count was too large. There is a limit of 125,000 files. I think that was the number 125,000 or there's a large

16:20 number that the Chromium codebase hit. So, I was not able to actually test augment code in the Chromium codebase, which is one of the reasons why that test did not make it into this. So, I need to find an open source repo or some sort of thing that I can like snapshot that fits everywhere that I can run it and then I have some eval that I want to build off of that so I can see can it fix things in it? Can it give me the context for etc etc. Now, I do want to just talk about this a little bit. So, this is from June. We look at GitHub copilot. Again, don't pay attention to the numbers because the numbers are going to change month over month. Look at the difference between them. We had uh Trey, Windsurf, Zad, Cursor, Root Code, Client, etc. Augment code still at the bottom. Uh GitHub Copilot's at the very bottom. What's changed is GitHub

17:06 Copilot has like jumped up significantly. So very surprising uh with Claude 4 in particular. Uh but some of the other ones through code and client kind of stayed in the middle. That that's about where they were. Uh Windsurf and Cursor I think dropped slightly uh in the particular this particular month. So, let's go back to this here. And you can see Trey and Zed still relatively high performers. They do incredibly well. And then Warp.dev. I've actually becoming more and more attached to warp.dev's um uh terminal to be honest with you. It's so nice just to have that little bit of AI autocomplete and like niceness in there. Uh and its coding ability is not bad. So, I'm excited about that. I think I'm going to keep my subscription on that for a little bit and primarily uh just use

17:52 that as like a backup when I want to do something. So that's pretty cool. Now my subjective rankings again this is me and what I am going to be using primarily for the next coming month besides when I step off to go testing. This is where I place them in order of value for me. Now I have this little down arrow on augment code because I have dropped it from two to three and I still really really love augment code. That is not a anything negative on augment code. It's just it's been surpassed in the amount of usage that I'm using with other ones. Augment codes context engine is amazing. It's super fast. It's very very affordable. $50 a month. I'm grandfathered in at 30 which I'm very fortunate for. Augment co codes task list. It's just great across

18:39 the board. It's really great. I have it as number three because the amount of code that I'm creating with augment code has dropped. And what has it been surpassed with? Open code. And open code I have found over the last week or so. Um I'm going be totally honest here. I have not used this the entire month of June uh going into July, but I have used it significantly and I have come to really like CLI based coding tools. Open code allows me to switch to a bunch of different models. I would love to be able to do deep dive analysis to understand how models other than claude work on it, but it works exceptionally well with claude 4 and so much so that I think it goes neck andneck with claude

19:25 and it's something that I've actually highly thought about u moving some of my automated workflows into. But at the same time, my number one is still cloud code. It is just so good. And I I had someone call me um a claw an anthropic shill. I promise you there is no company and you can ask anybody that's reached out to me to offer me money. Like I was offered a pretty significant amount of money from Kilo Code to go and just mention them. I turn that down. I turn down everyone that offers me money to talk about a particular model or to talk about an AI coding assistant. These are my legitimate feelings for what I'm using on a day-to-day basis. This is a data point. This isn't the right data

20:11 point. This isn't totally correct, right? This is my opinion. These are my subjective ranking. Yours should look very different. I'm giving you this as something to kind of go off of and maybe a starting point. So, that that's where I am for going into the month of July. Clog code. You just cannot beat the $100 a month max plan. Uh, I've actually debated going back up to the $200 a month max plan and cancelling or downgrading my Open AI subscription because I found the claw code is just so freaking good. And I don't know if I use enough of Codeex, the remote agent, to justify $200 a month anymore. Although, I have kind of been liking 03 Pro and I I think I would lose access to that if I downgraded uh to the regular plan. So

20:59 anyway, this is what I'm going to be using. Uh, open code, clawed code, and augment. I will still have root code, and root code is still going to be like one. It's probably my fourth, but open code, what I'm really liking about it is that I'm going to start experimenting with uh using it in uh some of the stuff that I'm doing with like uh workflows and things like that. So I'm going to try to use it to automate some of the things that I want to do uh locally. For example, I have a bunch of tests that I run for marketing eval eval. I think I could probably just automate all of that using open code by over, you know, setting up the prompts correctly, having it switch models automatically, and I

21:45 can get out all and it removes so much of the tediousness of it. Right now I have a bunch of code that's kind of running and I do a bunch of stuff that's switching models and managing. I think I can do a lot of that stuff a lot easier. And from a coding perspective, it is just nice to have a different CLI based tool that's really solid to allow it to review the code that cloud code did and vice versa. So being able to have them work back and forth together, it's really nice. And I'm very excited about digging into open code a little bit more and understanding kind of how local models work with it because it doesn't seem to use a lot of context. I think the system prompt I actually want to dig in and go understand the system prop because it seems to work really well. But at the same time, I really do want to test things like Devstrol with it. I

22:30 think it could be pretty sweet if that is just a good local model development agent. So, I think that's going to wrap it up. I know this one I put a ton of effort into it, so I get pretty excited about it. Um, and that's why I haven't done a video for a few days because I've literally every break that I've had, I've been hammering on testing these. And I'm excited with the overall results. Uh, and I would love to know your thoughts in the comments below. You know, anything surprise you? I know I had two big surprises here. The main one being GitHub Copilots like Turnaround. Everything else, Open Code was also a big surprise to me with like how good it actually was. Was my second one. But anyway, let me know in the comments below. And please check out my Discord. Like I would love to have more people join that. I think we're over 600 people now. So, I mean, we're still a pretty

23:16 small group, but man, it's so fun hearing like some of the ideas and stuff that people have and asking questions and getting good conversations going. I'd love to get more of that happening. All right, until next time, everyone. I hope you have a wonderful, wonderful day. Uh, take care. Peace out.