Claude Code + Karpathy's Obsidian = New Meta

0:00 Imagine if Claude Code remembered everything and got smarter every time you spoke to it. In this video, I'm going to show you Andrew Kaparthi's Obsidian Rag system that gives Claude a self-updating memory. So the Claude Code never forgets, gets smarter in every conversation, actually knows who you are and how to fix its massive limitation that 99% of people aren't even talking about. So by the end of this video, you'll have a Claude Code memory system that will work for the long term. And if you don't know who I am, my name is Jack I built and saw my last tech startup for the 60,000 customers. And now I run several AI automation businesses and teach thousands of people stuff like this every day. So if you haven't already, grab that coffee and let's dive straight in. Andrey Karpathy, famous founder of OpenAI. So Andrey Karpathy

0:46 just gone viral by talking about Claude Code plus Obsidian to create what we call an Obsidian memory system. Now, the way that it works effectively is it will have a lot of files. that Claude will create or you'll download and Claude will organize and create the system for you. And the idea is that it's very easy for Claude to edit these if you want to update the information and you can even view them visually in Obsidian. It is essentially like your personal Wikipedia. So anything you want to remember, we can actually just do by having this kind of index that Claude can cycle through based on whatever information you want to, meaning that you don't lose important context information. whenever you're using Claude. Now, this went viral of 18 million views talking about Claude code and Obsidian because it solves a real problem. And that

1:32 is context loss. In other words, the fact that Claude has amnesia for all its wonderful perks, sometimes it just forgets it's a model and it will hallucinate. We expect at least 1%, sometimes up to 10% of the time, depending on how long the conversations happen for. And it can be confidently And the problem is we don't know when it is confidently incorrect and when it's actually just being accurate. So with the system, Claude can read and write its own memory and it's supplementary to and supports alongside Claude's basic context. So it was like this adds a fourth memory system, which I'll explain in this video. And it gets better, crucially, the more that you use it because we index more the right information and we build it out over time. However, 90% of the coverage that I've seen on this, I would describe

2:20 realistically as node porn. And it does actually, the system have one huge, actually several huge limitations. But don't worry, I got you. I can explain what those limitations are, which I haven't seen anyone else talk about in this kind of hype cycle and how you can actually fix it so you get a very long-term memory system that will work in any situation and give you the accurate data. So let's begin with the source of this, actually download it, show you a real use case of how it will work in your new system. So Andre Kapathi basically explained this in about four or five different points. Effectively, it's using a large language multiple personal knowledge bases for different topics of interest. The best way to visualize this is like you are essentially building a personal Wikipedia. When you're talking about an Obsidian memory system, it's your own personal wiki. So data ingestion happens at points of Obsidian

3:10 web clipping, or we can give it data. There is a best way to do this, I think, and a not great way to do this, which I'll show you. But effectively we quit this wonderful raw directory and the LLM will incrementally compile a wiki, which is just a collection of MD files and a directory structure. That's it. And then it has a directory. you ask it questions it goes through and it finds the relevant things clicks on them and then good things happen so in terms of ide this is basically what are you looking at to do this now obsidian itself is also a visual layer but you could use this with any project you're doing could be anti-gravity could be cursor you name it and he's put some additional things in here he's got this concept of linkedin to do health checks but i'll come on to that later in the video when we go through everything so it's actually super duper and i say dupy because it is super duper easy to set up so what we're going to do first of all is download obsidian to sharpen

3:57 our thinking cool story these guys not many people built a very successful company we're gonna go ahead and download it on the computer i don't have it on this one yet so we're gonna get the entire process cleanly together i also want to touch on this node view as well i think it's interesting i know a lot of people are look how i can visualize it in reality i haven't seen anyone make any informed decisions off it i think it's just something that looks but it's not really necessarily gonna help you do anything better. Although the relations between certain things on your memory system, are really important, which we can touch on later. So we're down to applications. Now we're just going to open this guy up. Obsidian, just think of it as a way to visualize files. That is literally it. It will show you and map your visualize files. So you can create a new vault, open folders vault, whatever you want to. I'm going to do a quick start and show what it looks like. We come in. If you click on this one here, this is

4:42 a graph view, which will show you basically the relation between all of the key things in your database. But first, let's go to dark mode, which you can do by clicking on Obsidian and then file and settings. This exists on appearance. I know you can flip between light mode and dark mode here. Dark mode is usually as you prefer. So now we are ready to rock and roll. Now we just need to add some important information. So the way we do this actually is we're gonna head back over to Andrej Kaparthi's tweet. Now, if I come over here, for example, one of the things he did is built this little GitHub, which is really simple, it's very vague. And on this LLM Wiki, he basically explains the core premise a bit. The idea is that instead of just retrieving raw documents at query time, the LLM incrementally builds and maintains a persistent wiki, a structured intellect collection of markdown files that sits between you and the raw sources. And you know what guys, all we're gonna do literally is come down, we're gonna do a classic Control A and C.

5:29 So we're going to come down and grab all of this stuff here. Beautiful. And then we're going to head over to our environment of choice. And we're going to be using anti-gravity to do this. Of course, this could be done in a terminal. It could be done in a cursor. It doesn't really matter. I'm just using this as an environment. Let me open up a brand new folder. So I've called this one Obsidian. Now I'm going to open up a terminal. I'm going to run Claude code just because I prefer that environment. I'm going to come up and let's run Claude. And one little hack, by the way, which I highly recommend you do is you can download these images locally. So if you go to Obsidian, Settings, File and Links, Set Attachment Folder Path to a Fixed Directory, e.g. Raw Assets, Then in settings, you can hotkeys and click download to find attachments for the current file, which means that when you grab information from a page, it will save it there, which is going to be super duper handy. So come back over then. I'm going to give it the final prompt, which is, hey there, I would like you to create a structure in line with the below, please. Ask me any questions if you wish to, but just follow this along and create it for me, please. Beautiful. And before we

6:16 actually run it, it's best to understand exactly what the Kaparthi Obsidian system actually is. So most drag is what we call re-derivation. In other words, we're constantly going and getting the same information every time we ask the question. But with the LLM wiki, the way it's gonna work is it will compound over time. So it's the same question every time you start from scratch with traditional rack, but this system, it gets smarter with every source that you add. So traditional rack, this is how it works. Query, chunks, answer. Claude finds chunks, stitches an answer, and then forgets it. Tomorrow, it's the same process. But with this system, the way that it works is one new source will update 10 to 15 different wiki pages. The knowledge stays in every future query reads from it. So it's self-building in that sense, which is cool. So this

7:02 has what we call the compounding effect. So one source, 15 pages are updated. You drop a new article into your folder, Claude reads it. It finds every page in the wiki it relates to. Say, for example, you and I want to grow a LinkedIn. And we say, hey, this is a great article on growing LinkedIn. When we add that to the system, this will then update every page that it's linked to, which is really cool. It flags anything. This one's really cool. That contradicts it. Is there any contradictions are getting ironed out in the process? Stuff that it knew. One source in, 15 files updated in a single pass, which is why it's so super cool. You're building a personal Wikipedia and it's only good if you can update it, right? So three layers, no database, everything lives in three places, a folder for the sources you feed into, a folder for the wiki that Claude writes and a single file that tells Claude the rules. The rules are the game. So we've got raw

7:49 sources, the wiki and the schema. You can see what that looks like. So the sources might be articles, PDFs, transcripts, web clips, you name it. We've got summaries. for the wiki called writes it and you kind of read it and then the rule book which is your conventions your workflows and it turns claude into a maintainer of your wiki now the way it works is you can ingest so we're going to drop in things these are the three kind of operations you're going to be able to query so one is like asking it questions and then this process called linting that carpathie mentioned which effectively is every couple of weeks find contradictions orphan pages uh stale claims the kind of maintenance that humans would naturally forget in the whole process awesome so these two files do a lot of heavy lifting the index.md and the log.md effectively though you're going to curate the lm does the bookkeeping and the wiki will compound and value so what kind of things could we build with this well

8:37 we could do a book companion we could do research deep dive a team or business wiki and personal life wiki how does it actually work four rules really simple i'm going to pick one domain so not everything like one book one project one research question it works better when you have it as a set project. So one project for one thing. So growing on LinkedIn or, I don't know, managing this merger and acquisition, that kind of thing. Think of them as individual mini Wikipedia bases that you can call upon and question for different topics. You're gonna invest in the first section. So Claude's gonna ask you questions, answer them properly. This is where your rule book gets written. And then you're gonna run two screens. So Claude code on one side, And then your Obsidian Graph view on the other, if you want to take a look at it, you can see the nodes lighting up as Claude does various different things. And then every couple of weeks, you're going to go ahead and purge

9:23 it. So at first session, you're going to paste that prompt I gave you. Claude's going to ask you questions. Hey, Jack, what's this working for? And then you're going to co-write the rules with Claude in that process. Then every session after that, the loop's going to begin again. You're going to drop a new source. Claude's going to integrate it, update the 10 to 15 related pages. And then you can see it expand and blow up in Obsidian. You create it, it maintains and the wiki compounds. So let's head back over now to anti-gravity and see how Claude is getting on with this. So it's got a few questions. Now it needs to know what the domain is about. We have everything now. And again, Claude.md, it's got all the different files and systems. It's really cool, really handy. So I think the first one we should be, should be a personal life assistant that we're going to use to leverage to make big decisions about our life. And we can just feed it that information. So let's come down and give Claude that information. So tell me the first domain

10:11 drop any file you want to into raw or ingest and do a real first run want me to get in it to get the version ready to rock and roll so i'm going to say something like hey the dude what i would like to do is do a file basically just on making better decisions and running my personal life for example there's an article i read on high agency and i would like it to be we're going to call this one a general productivity, life decision-making wiki. I'm going to drop in information about me, who I am, what I'm about, just relevant stuff. And then also things and mechanisms of thinking that I find that are valuable that I want you to integrate for different things. And what I'm first of all going to do is just drop in a URL to you. And I want you to go ahead and scrape everything in there, images and also text. But obviously in the future, you can use Web Clippers for that. Let me know when you are ready for the first URLs. Awesome. So I'm going to send this one

10:57 off and I'll show you that hack in just a second. But this is an example of an article in High Agency. It is a fantastic read. My buddy Nick shared this with me. Really, really highly recommend it. What I'm going to do is grab this link here. Come back over. I'm going to drop it in right there. It's ready for it. Life OS sounds like a great name. Let's start building up that personal wiki. Awesome. I'm going to drop that in there as well. And then I'll probably just get Claude as well to say, Oh, and by the way, just adding all the information you know about me to this life wiki, personal folders. If you go to my business folder, it has lots of information about me, business information, add all of that in there as well. Wonderful thing. If you've got code on your laptop and different folders that covered this, it can just go and grab that data. Now the hack, by the way, and it's something that Karl Parthi mentions good is that Opus itself has a web clipper. So what we're going to do for that is click on these dots at the top right, come down to extensions. And

11:43 in fact, you're looking for the web store itself. And basically we're going to search for Obsidian and this is your Obsidian web clipper like that. It can So you come down, add this to Brave, there we go, and just add that as an extension. And then personally, what I like to do, if you come in these dots, come down over to extensions, click on manage extensions, find your Obsidian web clicker, click on details, and just make sure you pin this to the toolbar, like so. And now you can use that web clicker when you want to. So for instance, if I was here, if I click on the little Obsidian, you can see I can now actually add this to Obsidian and you can basically specify everything that you want to. Now, one thing to call out with the Obsidian web clipper is if you click on the Obsidian logo here, for example, and then you click on this, this will then be the settings. So effectively what we can do is add all of the vaults here that we wanna save to. So I might have a personal operating

12:29 system. I might have a YouTube growth, LinkedIn growth, whatever it is. And what you can do is create a brand new template, give what you want to. So I could say personal operating system, whatever it is, you go through this whole thing, you can save it and you can add, this will then actually add it directly to your actual folder itself. You just name the location of the folder path here and then that's how you dynamically add everything over there. So now if I come back over to anti-gravity and now Cloud Code's done everything for me, I've got all the raw files. You can see it on the left hand side now. It's not only basically grabbed the thing that I wanted, but it's also added loads of specifics. So if I click on assets, for example, it's got images. These are all from the article. So I've got all these downloaded, all saved. Very, very cool. And in fact, if I come over now to Obsidian, which I have right there, you can see I've got the high agency George Mac. Now I can vectorize everything that I want

13:14 to. And if I want to see the whole node view, all I would simply do is come over here, check this out, open a graph, and I can actually do anything I want to. So with this in mind, I can now physically start asking it a question. And bear in mind, for me, this has got everything about me, decisions, mental models, people you literally know. Okay, these are mental models that it's kind of adapted. it to go ahead and use and do various different things it's got decisions here which are going to be all the decisions that i make about me loads of really cool things that we can then piece together and the more we use it the better it gets now i've come back over here now you can at this point build a search engine where you can ask it questions make it a beautiful beautiful interface do whatever you want to but honestly i don't think it's worth your time to get started you have a search engine his name is claude he lives here so you can have your master folder open up in your environment with all of your subfolders within it and just chat to it here. And

14:01 then once you exceed to a certain level and you want to add different bells and whistles, we can go ahead and we can do that. And you get to a point where with something like this, which is what I built for my community. If I ask, for example, what is anti-gravity? How do I set up a gravity claw? This is all trained on every YouTube video I've ever had or my school community, everything, you name it. So you can start building fancy stuff like this for yourself with your personal operating system, but I just wouldn't do it off the bat. Just interested in saving your time. You can just ask it to do that. You can give it API keys. You can do anything that you want to. So what we can do next out is just test that it works so i'm gonna say hey based on everything in this folder um i am considering how i can be more effective at my job how i can help my community members out and my clients and to do that i would love to know strategies of how to create more time in a day let me know your thoughts and in addition to that

14:46 any other files that you referenced on the left hand side in making that decision so i'm just asking a question based on its knowledge based on what's cool is the more information that we add to this the better and richer that it physically gets and if i come out and i say something like hey this thing isn't true anymore it will start to iron out those contradictions and you can see it's got a list of files of reference down there so now you start to build up this wiki and just follow that process for different areas so this system is awesome but it has limitations, limitations I haven't seen anyone else talk about. And if you don't understand where this falls down, you're not going to get the best out of this memory system. So let's go through the Obsidian RAG limitation that I've not seen anyone really cover and how you actually build this into a system that is more robust and give you better long-term memory. I've thought very long and hard about this and I put all the kind of data and research into it. So first

15:34 of all, if you're just using Obsidian RAG, cloud code breaks in five places or you need to be aware of the stress test number one is your cloud.md grows over time this is also true for your index file so if you think that every single thing you have in your index is like 75 tokens at 10 files 750 10 000 files you can see how your token window just for looking through the index page grows exponentially and so does your token cost for using this system so there's a tax you must pay when going through that that i think is just important to be aware of there's no semantic search it doesn't do semantically so it goes by the topic of the thing rather than the semantics around it summaries can go stale we call that drift um your million token window can fill

16:20 up a bit quicker this way and it's not really built for very large data sets this is built for more small to i'd say early medium-sized data sets um said another way it can feel magical at 100 but eat you alive at 10 000 well jack I don't think I have 10,000 files. Cool. Are you sure? I mean, with these things, if you're going to invest in a memory system, it's really important that you have something that can scale with you over time. And it's just worth bearing in mind on what your tokens per query might look like. These are just figurative examples, but you can see the kind of difference. There are files you add, the more expensive it can actually get. Obsidian scales linearly, pine cones stays flat. So these are rough order of magnitude estimates for this Kiparty pattern, not measured benchmarks. And the shape of the curve is what matters. So it's just worth bearing in mind and calling out just so you understand what's happening. So here's an interesting one.

17:07 If I had YouTube transcripts, this is something that is better off in my view, actually living in a pine cone database. So a good way to think about it is that obsidian is for the stuff that changes. And pine cone is for the stuff that doesn't. You actually don't have to use obsidian if you don't want to. I think one of the core benefits is the fact that you can clip and bring it in, but I'll explain all that in a second. So let's say for example, we have 184 YouTube transcripts. Giving that to obsidian, I think is not a good idea. That's way better off on pine cone. I'll explain why it's case. So think about this. If you're putting it through obsidian and using clawed up as 4.6, Claude has to read, even Apple's getting involved now, the entirety of the context window. First of all, to understand it, then it has to type it back out to the various different sections of summaries. Pinecone, for example, runs a script. It runs a

17:56 query where Claude's not even involved. And the embedding cost is like 100 times cheaper based on the embedded model that you're using. So your tokens at point of ingestion is way lower. So it's saving you a lot to put it into Pinecone rather than Obsidian in this case. So two systems, different physics. So Obsidian Rag, and this is just a great way to physically understand what happens when you're physically using it. And this might sound technical. This is so helpful to understand because when you understand how your systems work, you can build anything. So when we're using Obsidian, the session begins, Claude code loads the Claude.md automatically. And that's the schema rule book for how everything works. You can ask it a question. Claude reads the index.md Okay. And it's got a catalog of every page that you got one line summary. And it just says, look, well, Jack's asked me about the cloak of invisibility. Well, you know, Harry

18:45 Potter probably sounds like a great one to read. So does this, so does this, so does this. And it goes so on and so on and so forth. Claude uses the retool to load the specific files. The page is cross-referenced with other video Obsidian style Wiki links. And then Claude reasons over what it's loaded and writes back updating notes, adding new links and a pen into your log. So Obsidian lets Claude reason over the structure of how you've set this up. It's a graph. It can answer, you know, which hypothesis did I revise because of customer five. Pinecone doesn't do that. By contrast, Pinecone, Ingestion is one time. Vectors are stored in pinecones and index. You attach the exact original text. You can ask a question. You know, Claude just sends it over there. Pinecon runs an approximous, what we call nearest neighbor search, which finds the top K. Don't worry what that means. It's just like a technical gobbledygook for like what's most similar. Very, very quickly. And it returns the exact text. We

19:33 always discuss this as a librarian. And rather than pulling the entire text, Harry Potter, it just grabs a specific paragraph that you need to actually understand it. So pinecones gives Claude perfect recourse. cool of massive archives, exact text, flat cost, but it's similarity only, no graph, no reasoning over the relationships. So think of it like this, Claude.md is who I am, Pinecone is what I've said, Obsidian is how I think. Now there are three layers to this, not two, because you'll have your Claude.md file in your projects based on how you're physically using this, unless you're just querying it like a search engine. Now, what are they best for? Basically, Claw.md is your identity, voice rules, you know, read first instructions, obsidian rag, active projects, decision logs, idea gardens, anywhere that structure matters. And the pine cone is going to be transcripts, research archives,

20:18 books, anywhere exact recall physically matters. Said another way, the magic word for obsidian is reasoning, not memory. Claw.md is your identity. is you, your identity. Pinecone remembers you, it's how you recall, and Obsidian thinks with you, reasons over structure. So if you imagine, for example, we're going to work, one of the best ways to think about this is Claude is like your identity badge, your Claude.md. It's your name, your role, your rules, read before anything else, and it never changes mid-day. Your workshop is where you actively build, where you're doing hard work. You know, notes are linked by string, reasoning happens there. And then warehouse is Pinecone in this example, where we put all our big stuff, all our big files, all our video transcripts, our archive of every email we've sent would throw and let that live inside the warehouse so with this in mind what do you actually do what is the complete structure to run this sustainably

21:05 so that you have a system that never ever forgets well the way that you understand this relationally okay you have your identity in your claude.md files if you're using the claude app that's the about me section again you can have that based on projects obsidian can have all the little mini things the little projects that you're thinking about that is how you think Pinecone is your full archive of all your history. And then notebook LM is something we bring in to do deep dive and research. So for example, if we're in a project and we're learning about the manufacturing process of white bottles, I could say, hey, spin up a notebook with 200 resources explaining everything. Then we can bring that back over to our laptop and even throw that into our long-term memory. If we ever wanna recall it in anything that we're physically doing and understanding how these four connect relationally is really important. So for example, if

21:52 I was taking my entire YouTube transcripts, I'd saw that in Pinecone, my warehouse, my long-term storage, the place where the information is never, ever going to change. And the key thing to understand here is that you can connect Pinecone so easily now to Claude, it is ridiculous to the point where you can just ask it a question and Claude will reach over to Pinecone, find the answer and bring it back. And for every project that I've physically done, and I can even show you this now, I showed you earlier in my AI with Jack dashboard, like I find stuff like this where I can ask questions and I have processes here that will automatically on a daily basis, scrape the transcript for everything inside my YouTube channel. And it also works inside my community, right? Where people saying, Jack, I need to find something like, I don't know, let's say pricing AI services, whatever the thing is, this scrapes everything

22:39 daily and this all lives in Pancone and it's great. And you can click on this, come and check out the community. And again, it's like, there's a lot of different stuff that exists inside the community that you can come and grab. And all that is linked in Pancone. So I find it on a day-to-day basis, wonderful. But the thing about Obsidian and the Obsidian index system is it brings a completely different value set. You just need to understand what you're using it for. and where you realistically want to introduce Pinecone to support that. And so without a long-term memory system, we're leaving value on the table, which is why the next thing that we need to do is learn how to integrate Pinecone into our system so we can secure information forever, which we're going to do by watching this video right here.