0:00 Hello and welcome to the ultimate scraping guide for NADN. In today's video, I'm going to show you how to scrape any website on the internet using NAND. And the beauty of this is you don't have to be a developer and you don't have to have coding experience. So even if you're a total beginner right now, by the end of this video, you are going to be a scraping pro. As promised, I'm going to show you how to scrape any website using N8N. So let's go ahead and dive in. So before we get into that, the normal flow with NADN is normally you're just accessing your personal data and it's enriching your AI agents and it's powering your automations. These are things like your notion account, your air table, your Google drive, dropbox, maybe you have a superbase account with a bunch of data or you're using some other Postgress or you have one drive
0:47 running anything that you're normally accessing pretty easy to connect with on NAD and I have other videos on that as well. But here's the problem. the right data isn't always available. Sometimes these normal solutions just aren't going to be enough for the automation that you need. So that's where scraping can help. Now, normally we can't just connect to live internet data. That's kind of a hard thing to do. And scraping is going to be the key to doing that. It's going to be the key to accessing real estate details, leads, things like Amazon pricing, car prices, social media feeds, hotel prices, Google Maps, for instance. This is where scraping comes in handy. But there are some barriers to scraping that you'll need to be aware of. There are scraping rules that vary by website.
1:33 So, be sure to check the terms and conditions for the websites you're scraping. Websites may also block you from scraping. And setting up servers and having an experience with coding and being a developer. Not all of you are going to fit into that picture. So, here's the deal. Here's what we need to do. There's the main goal of scraping is to impersonate a human to look like a human using the internet. Grab all of that data with a robot and then put it into your automation or your AI agent. Now, there are kind of like two paths that you can take with this. The first route that a lot of people are going to take is the developer route where you need to be a developer. You you need to have experience with things like Selenium or Puppeteer. You need to write your own Python programs to do this sort of thing. But what I'm going to show you today is scraping that. so easy that my
2:19 grandmother can do it. So, in order to do that, we're going to be using a tool called Firecrawl. And what Firecrawl does is it allows you to turn any website data into LLM ready data. So, you can go grab that information off of the internet, bring it back into your automation, and it's ready to go for your AI agent just like that. Or you can just run a traditional automation. And in today's video, I'm going to be showing you two things. I'm going to show you how to set up scraping workflows using no code in NAD. And I'm also going to show you how to integrate firecraw with naden to scrape anything. So you don't have to be like these developers over here using these fancy tools like selenium and puppeteer. No shade to them. They can work as hard as they want setting all of this up. But the truth is you don't need to do that
3:05 anymore. There's tools out there that can actually perform these types of things for you. So these are the things that I'm going to be showing you in today's video. Now as you work your way through today's video, you're going to have less of this. You're going to have less of needing to hire prodevelopers to build crazy pipelines for you. Dealing with competitors that are beating you to important data and failing your scraping tasks, and you're going to have more of this. I'll show you how to build any scraper with no coding required, how to get the important data before your competitors do, and how to have time for more coffees and cocktails. So, with that being said, it's time to get into how to scrape any website using NADN. So stick around until the end of this video and enjoy this scraping master class. Now before we dive into the master class, I want you to comment below. What
3:50 got you interested in scraping? What made you click on this video? I'm genuinely curious to hear what motivated you to click on this video and to embark on this journey. Now, what we're going to do next is we're going to open up NADN. You're going to get signed up. They give you a 14-day free trial, so it's actually pretty generous. And now once you're on NAD, you're just going to open up your workspace. And once you open up your workspace, you might not have all of these workflows like I do here. But if you want to create your first workflow, you can just click this create workflow button. And we're just going to call this one the scraping workflow. So to change the name of the workflow, I can just click on this title up here. And I can call it the scraping workflow. And the first thing that I'm going to do here in NADN is I'm going to click on this add first step. And the nice thing about NADN is we can just
4:37 click on this trigger manually option. And this is going to allow us to start our automation just by hitting test workflow. It starts sending some data through here and it sends an item. So basically this is just saying we want to run through this automation one time. Now for a basic scraping workflow, all you need to do is click the plus right here and then connect one more node and that's the HTTP node. So I'll type in HTTP and I'll click on HTTP request. Now, all of this here might look a little bit confusing. And to be honest, it was confusing for me when I first got started with all of this, but I'm going to show you guys how to do all of this in today's video, click by click. So, don't worry about that. Don't be too concerned with all the new things here that you're not used to. We're definitely going to take care of that by the end of this video, and you're going
5:22 to learn a valuable new skill. But, in order to get all of this information, we're going to need to connect to our fire crawl tool that I was telling you about earlier in this video. So, in a separate tab, just go over and open up Firecrawl. So, you're just going to type in firecrawl.dev and then hit enter. And that's going to bring us to Firecrawl's website here. If you want to test it out, you can just type in a website here and you can start for free and see what kind of data it brings back for you. Now, I'm already signed in, so you can see my dashboard button up here. But if you're not signed in, you're just going to create a free account. And the nice thing about Firecrawl, guys, is they're going to give you 500 credits to begin with for completely free. This means that you can run 500 scraping requests. It's actually very generous for a free
6:08 plan. So, you might be able to do your scraping tasks without even having to worry about, you know, running out of a free plan. But the nice thing is is Firecrawl is actually super scalable. You can check out their pricing and you can see what plan will work for you. But for the sake of this video, just stick to the free plan, create an account, and then I'll see you over on your dashboard. So, I'm just going to click to the dashboard. And again, if you guys want to play around with this tool, if you want to see how it works, you can go to the playground, and it's a simple place where you can enter any URL, and then you can run it, and you can scrape any of those websites that you put in here. But for right now, what I want you to do is to go and click on this docs button up here. This is going to take us where we need to go in order to get the information that we need to fill out that HTTP request over in NAD. So, I'll
6:54 click on docs and then the feature that I'm interested in is the scrape feature. So, I'm going to go over to the sidebar here and click on scrape. And then if I just scroll down a little bit here to the tab where it says usage and then I switch to a C URL, then I can just hit copy. And all of this information is going to configure our scraping request. So, after copying this, I'm going to go back over to NADN, and all I'm going to do is hit import C URL. I'm going to paste in my command right here, and then I'm going to hit import. And what that did for me there is it just filled out my entire HTTP request. So, pretty much everything is good to go here. And now I can just change a few little things about this template that I that I want to basically switch out. Like, I can switch the URL
7:40 and the formats that this is returning. Now, normally when you're doing a scraping request, it's just going to return the raw HTML. This is what you can think of as the website's structure. So, it's just a language that defines how everything is going to be on the website, where things are positioned, how they're going to be picked up in terms of styling later on. So, it can look a little bit confusing to a beginner if you don't have experience with HTML. But the markdown one is just going to give you the raw text. And this is really what you want if you're going to be handing this off to a large language model or an AI agent. Now, the only other field that you're going to want to pay attention to is this value right here. You're going to need to pass in your API key in order to connect to your Firecrawl account. But the way that it brought this in in this header
8:26 parameters right here isn't quite how we want it. So, I'm going to show you how to set up your credentials in NADN so that you can keep this authorization in your account and it's a little bit more secure that way as well. So, for right now, what I'm going to do is I'm just going to delete this right here. And then I'm going to scroll up just a little bit to where it says authentication. And we can actually save this credential in our account if we hit this dropown right here. And if I hit generic credential type, and then you see this new one pop up here, generic O type. Click on the select again and then click on header off. Once you've clicked on header off, you can click on this next dropdown and you can click create new credential. And then we can put in the name. So the name was authorization and then here in the value we can type in the words bearer. So B E
9:12 A R E R and then a space. Just one click on your space bar and then we're going to go back over to firecrawl and we're going to go ahead and grab our API key. And this is basically like the password to access your fire crawl. So make sure that you don't leak this out to anyone. Make sure that you keep this private. So, we'll head back over to Firecrawl, and you're going to want to go over to the overview if you're still on the playground. And once you're on the overview, you're going to see your API key right here. And the nice thing is is they have this copy and paste button. So, you can just copy it to your clipboard, take it over, pop it into NAND, and you're good to go. So, I'll click on that copy API key. And then I'm going to head back over to NADN. And I'm just going to paste it right after my bearer space. And then controlV to paste
9:57 or commandV if you're on a Mac. And then this authorization is good to go. But if I wanted to, I could also rename this just to keep things nice and clean. So I'll just call this firecrawl tutorial. And I'll hit save. Now if I close this, I should be able to select the header off of firecrawl tutorial here. And I should be good to go. The only things we're going to need to change now are the URL that we want to scrape. And by the way, we can swap this out with any URL that you want. You can even automate a dynamic field here so that it can switch the URL based on the request. But just for the sake of this example, I'm just going to be doing a format of markdown. And I'm going to be scraping uh this firecrawl.dev docs. And I'm going to get rid of this HTML here.
10:44 So now the only format we're going to receive is markdown. And I'm going to be scraping this website for the firecrawl docs, which is the example they have in here. And I almost forgot, there is one header parameter that we need to send. And that one is content- type for the name. And then we're going to do application slashjson. And this is just telling it how we want to send and receive information for our requests. So now I'm going to go back to the canvas. And just like I said, if you wanted to test this workflow now, I could click test workflow right here or I could click it down here. And that's going to run this automation. and it's going to send an HTTP request over to Firecrawl. It's going to run all those tasks for us and then it's going to come back with that website data. So now I'll just hit test
11:29 workflow. And as you can see here, we watch it happen live and it's scraping that website for us. This might take a second, but then it should come back with all of that data. All right. And just like that, our scraping task here is complete. So now I can click into this HTTP request. And if I look on the right here where the output is, this is what's coming out of our automation. If I wanted to see what's going in, I can check out the input over here. And since there's no fields that are being passed in like the website that I want or something like that, we're not going to see any data here since we're just using the when clicked start this workflow. So, it's basically just telling the HTTP request to run one time. But if we shift our attention back over to this output here, you're going to see the status of this request was success true. And you're also going to see the data that
12:16 it pulled back. And within there, we're going to see the markdown. Now, this is the markdown format of the scraped website. So, this might look a little bit confusing, but really all it's showing us is the structure of this website, and it's actually a lot cleaner than if we were to run this with the HTML structure. So, all these links that you're seeing in here, these are things like the different logos and images that are on the website. So, this one right here is just the firecrawl logo. But if I keep scrolling through here, you're going to notice that we start seeing the title like welcome to firecrawl. And then it's going to tell us how this API works since the URL that we're scraping is the firecrawl docs. That's this page over here that we opened earlier. So we're actually just scraping this part of the website. So where we're seeing welcome to firecrawl here, that's what's
13:02 showing up over here in the scraping where it says welcome to firecrawl and then it goes on to include all the data and all the information from that website. So, if I keep scrolling through here, you're going to see that this is a long page of data with all of that information. And the beauty of this is it's up todate with what Firecrawl has just posted on their documentation. Next, I'm going to show you how to take that data and pass it to a large language model or an AI tool in order to summarize that information or get whatever information you want out of that scrape. And then I'm going to have it put that into a Google sheet so that it can tell me those specific bits of info that I need. And I'm even going to show you how to structure the data so that you can get consistent outputs every single time. Like let's say I put in a URL and I want to know the name of
13:49 the company for the website that I'm scraping. And I also want to know something else like a summary of the website. Maybe I want to try to find emails on that website or something like that. you can pull all of that information out of this data using an AI tool. So, in order to do that, we're going to go back to the canvas. And then from there, I'm going to hit this plus button right here, and we can type in Open AI over here. And then click on Open AI. Now, as you may know, OpenAI is the founder of Chat GPT. They created this model that we all know and love and you can actually access it in these automation tools like NAD and you can configure it to run the same sort of prompt over and over again as you run this automation in the future. So with whatever URL you put in there, you can have chat GPT look at that information
14:36 and then create a structured output with an AI response. So in order to do this, we're going to click on the message a model option right here. And what that's going to do is it's going to open up OpenAI, but I'm just going to go back to canvas for right now and show you what we're working with. So, we've got OpenAI message a model. This is another node here that we can use to access an AI. And we can also hook it up to tools so that it can use different tools or MCP servers. Like for example, let's say we wanted to give it access to a calculator tool like this one right here. If I go back to canvas now, you can see it has the calculator tool. So, it's kind of like giving your model different tools that it's allowed to use in its
15:22 response if it needs to use those. And this can be very useful because AI isn't the best at some tasks, such as math. If it's in the training data, it's going to remember that. But most of the time, you don't want to rely on AI for math. So, you give it a calculator tool to find the answers to your math problems. Next, let me show you how to configure this OpenAI message model. So, I'm going to click into this node and up at the top where it says credential to connect with, we're going to click in that empty space and click on create new credential. Here, we need our OpenAI API key. Just like we had an API key for Firecrawl, now we need to connect our OpenAI API key. And this is going to give our NADN automations and AI agents access to OpenAI's models like chat GPT.
16:09 So, it's actually really simple to get these credentials. All we have to do is head over to platform.openai.com. And then up here, you're just going to sign up, create an account, or log in if you already have one. And then once you've done that, you're going to see your little profile picture, and you're going to see this little uh cog icon. Click on the cog icon right there. And then over here on the left of your screen, you're going to see billing. You're going to click on billing and you're going to load in5 to$10 worth of credits. Now, some people ask me, Carter, I already have a ChatGpt account. I'm already paying ChatGpt 20 bucks a month. Does that allow for access to the API? Does that allow me to use this on other tools like we're doing right now? And the answer is no. You
16:55 have to load this in separately. But don't be too concerned with the price of OpenAI. You can set limits on how much it's actually charging you. And there's a usage tab to keep tabs on exactly how much you're spending with the different models. And it's going to cost you fractions of pennies per run. So you don't have to worry too much about the costs getting bloated up on that. And they aren't going to rise that fast. In fact, this month alone, I've only spent a couple dollars, like under $5 just running different automations. Now, once you've set up your billing using that tab right there, then you're going to go into API keys. So I'll click on API keys. And then what you're going to want to do next is hit create new secret key. And then you can put in your key name right here. For this I'm just going to put in scraper test. And you're going to
17:41 need to select a project. You can just select the default project. And then hit create secret key. And then you're going to copy your secret key. Make sure that you keep this private. This is your password to access your credits. And then we're going to head back over to NADN. And we're going to paste our key right here. I'm also going to rename this credential to scraper test. It doesn't really matter what you name it. This is just so you can recognize which credential that you're accessing. And then I'll hit save. And now I'm going to close this. And up here where it says credential to connect with, you're just going to make sure that this is on the new credential that you just added. And then you should be good to go. Now you can just select your model right here. And OpenAI just released a new model called 4.1. And I'm
18:26 actually going to test out 4.1 in this video to show an example. So they released it yesterday. At the time of recording this video, it's the 15th. So I'm going to click on the 0414 one. And this is going to be one of the newest models from OpenAI. Now here, this is looking a little bit more familiar. If you've used an AI tool before, you can enter a prompt. But the neat thing about this is I can drag in my markdown from the previous HTTP request and that's going to dynamically pull it into my prompt. So I can say something like here's the data for the website and then it's dynamically going to pull in whatever came from that HTTP request. And on the next line I can just say summarize the information. And now if I hit test step it's going to call
19:13 open AAI. It's going to run these AI tasks here. It's going to receive that markdown that I pulled from the previous HTTP request which was calling firecrawl. And it's going to come back with our information over here. All right. Boom. Now it says, "Here's a clear summary of the Firecrawl Docs homepage." And then it gives me all of that information right here. And it gives me a nice response. And if I go back to Canvas on the front end, this is what it's going to look like, right? We go from starting the automation to triggering the HTTP request. And then OpenAI gets called and it accesses the information from this previous node. Now, let's say we wanted to store this and structure this out. What I could do is I could hit this plus button right here. I could type in Google Sheets and we can just have this go straight into a Google sheet for us so that it's a nice
20:00 friendly user interface. So let's hit Google Sheets now and we're going to append row and sheet. Now I'm going to go back to canvas for now because there's a few other things we need to set up here in OpenAI. But this would be the basic flow for the automation. We start the automation. It scrapes a website, hands it to ChatGpt with a prepackaged prompt, and then ChatGpt outputs that information into our spreadsheet. Now, the other nice thing about this is not only can you pass the data from ChatGpt, but you can also pass that raw scraped data into there from your HTTP request or anything further back in this flow. Now, right now, we can just take that direct response and pop it into our spreadsheet. But what I want to do is I want chat GPT to respond with a structured output every time. I
20:46 want it to actually give me like the title of the website, maybe any emails that it found on the website if I'm looking for leads or I want it to pull a specific field from that website like prices or something like that. I can set up as many of these as I want to fill out this spreadsheet for me. So now we'll get into setting up a structured output in chat GBT. So if I click into OpenAI here, what you'll notice is by default our prompt here is using a user role. This means that it's just accepting the input as if a user is typing into the prompt bar and it's just giving a response. But what you can do is you can actually give it some system instructions. So if I change this role here to system and then I say let's just remove this prompt. I can say something
21:32 like your role is to extract information from the website data that I pass to you and output it in a structured format. Okay, so it's a very basic prompt, but this prompt right here is going to be so powerful because it tells the agent every single time exactly what it needs to do with the information. So now I could give it the different things that I want to have in the format. So I could say you are to deliver the following and I could just start creating a bullet list. So I could say summary and then I'll put a colon and I'm just going to explain what a summary is. So I'm going to say a summary is a basic 250 character short description of the content you receive. Great. Now I'm also
22:19 going to say title. That's another thing that I want to put in here. So, a title that you create for the info that you receive. So, here we're kind of just defining everything. The next thing I want it to do is to add emails. And I'll say list any emails that you find on the site in a commaepparated string. So, this might sound kind of confusing, but all this really means is that it's going to put commas between each of the emails that it finds when it delivers those in this format. And I'm also going to say here if you don't find any emails in the content just input n / a for not applicable. Now the beauty of this is you can set up your definitions how you want to. But there's one more step that
23:05 we have to do to get it to actually package that information up properly. And I'm just going to say finally you receive an input of scraped website. And I'm going to put this in some XML brackets. These are just these little chevron like brackets here. And this is how we format the different objects that are coming in. And then this instruction is super important. I'm going to say respond in JSON format. And then I'm going to give it the format of the things that we defined for it. Right? So I told it title, summary, and emails. So I'm just going to put in a curly bracket. And then it should create a second one. But if it doesn't, you can add that second one. So just these two curly brackets. And then between them, I'm just going to hit enter. And now, as
23:50 you can see, it just created a new line. We have a curly bracket on top, curly bracket on the bottom. And now I'm just going to space space. And I'll put a quote. And now these quotation marks are going to go around our different items. So the different things that we're passing through. So title, summary, and emails. So here I'm just going to type in title. And then we'll arrow over and I'll put a colon. and then I'll hit a space and we're going to put some more quotes and then arrow over again and comma. Okay, so that's how you enter an example of how it's supposed to respond. Now, if I hit enter again, I can just copy this line and I can paste it. And then we're just going to switch
24:35 out this word. So, this one will be summary. And then finally, we'll do emails as well. So, I'll just copy that, paste it, and emails. Now, the one thing you're going to have to do is on this last item here, you're going to want to get rid of the comma because it's the last item. It's just like when you're writing a sentence out and you're comma listing some items, you don't put a comma after the last item. So, that's just a way to think about this. This is basically the JSON structure that you need to pass. And that's again from these different things that I defined up here. So based on what you defined, you might have to change out the words in here. But this is what's going to allow it to pass those things in individual objects every single time. Finally, we're going to hit output content as JSON to make sure that
25:20 it gets this structure. All right, perfect. And now I'm going to add a message, and this is going to be the user message that contains that scraped website data like I was showing over here. So we'll copy that scraped website data tag right there. I'm going to enter down here and then I'm going to paste it in again and we're going to put a slash at the beginning of that to block it off. So, this is just a prompt structure. This is just to show it everything inside of this is our scraped website so that it knows where the content starts and where it stops. And it gets that same info every single time except in the middle of it, we're going to have the dynamic information from the website that you scraped. So, this is where we drag in our markdown in between those scraped website uh tags right
26:05 here. And if it doesn't pull in between, you can just copy that expression. Click and paste it in there. And now it's just surrounded by scraped website. So, that's perfect. That's ready to go. Now, if we test this again, it's going to take this user prompt and it's also going to take the prompt from the system as its instructions. And because we wrote it out the way that we did, it's now coming back with the title, a summary, and the emails. And in this case, there's not any emails on there, but if there were, it would pull those into a quick little list right here. So, we've got our title, we've got our summary, and the emails coming through. But how do we map those to different columns inside of Google Sheets? That's what I'm going to show you how to do next. So, I'm going to go back to Canvas. And now, I need to set up my Google sheet in another window so that I
26:52 can connect with it. I'm also going to delete this calculator tool because we're not actually using it. I was just showing you guys that as an example. Now, let's open up a new tab and type in Google Sheets and we're going to open that up. All right. And now here I've created a blank spreadsheet and I'm just going to give it a name. I'm going to call it scraped data. And we're going to add some headings here for the different data that we have coming in. So, as I said before, we can pull those specific bits of information that the AI puts out, like the title and the summary and the emails, but we can also pull things from further back in our automation, like the actual URL from the website. And maybe we want to pull in the actual scraped data to see what it pulled through. So, in this case, we're just going to type in URL. We're going to
27:39 type in scraped data. And then here we're going to start putting in the fields that chat GPT is going to fill for us. So title, summary, and emails. Yours might look a little bit different, but this example should be enough to teach you how you can kind of plug and play with this system, and you can do whatever you want with it later down the line. So now that we've configured our headers and we've given this a name, we can go back over to NADN, and I can click on Google Sheets right here. And then from there, we're just going to create a credential. So, I'm going to click on this dropown and hit create new credential. From here, we can switch to OOTH 2, which is recommended. And we can sign in with our Google account by clicking this button. Once you've signed in, it'll say connection successful. And it's just going to close that window
28:25 automatically. And then boom. Now, our account is connected. I'll go ahead and rename this credential to scraper test just for my organization. And I'll hit save. Great. Now, we can close this window. And next, we're going to want to go to this document dropdown. And here's where you want to search for the title of your spreadsheet. So I'm going to click on scraped data. And that's just the title that we set right here. And then the next thing that we're going to need is the sheet name, which this is called sheet one right now. So we'll head back to NADN. I already have the title for the document. And now we'll put in the sheet. So I'll hit choose and we're going to click sheet one. All right. Perfect. And now it's loaded in all of the columns that we set for the headers. Now this is why I love Naden, guys. From this point on, we can just drag and drop things in. Over here
29:11 on the left, you can see all of our data from OpenAI. You can see all of our data from the HTTP request. And you can see any data from that first node as well. Since it doesn't have any data at that point, we can just close that. And now, let's just go down this list and start filling these out with a nice drag and drop. So, I'll click into the HTTP request and I'll drag the markdown into the scraped data. And that's actually going to pull that into our spreadsheet. Now I'm going to scroll down a little bit here and see if they also pass in the URL. I'm not sure if they do, but we can add that. Okay. Yeah, source URL. Boom. And now I'm going to scroll up and for now I'll close the HTTP request. We have everything we needed out of there. Now I'm going to open up OpenAI. And because we set this up with the
29:57 structured output, it's now passing through title, summary, and emails. So I'll go ahead and grab the title and drag that in here. The summary will go ahead and pull into there. And then emails just to drag and drop into this one here. So now this is all set up and ready to go. So if I hit test step, it should pull all that information from the previous node. Then it's just going to pass it in over here. As you can see, it's loading. And now it's loaded in. So let's go back over to our spreadsheet. And if I scroll up a little bit here, you're going to notice that it made it super long because there's all that scraped data in there. But I can just drag this up and uh it should be able to hide that information right there. And just like that, we now have
30:43 the URL, we have the scraped data, we have the title, and if I double click on that, I can resize these columns to fit. We have the summary, and we can also have our emails in here. So now, this is where all that information can live so that we can use it for our other work that we're doing. So maybe you're using this to source leads or maybe you're using it to source competitor pricing. Whatever it might be, you can just pull it all into this nice Google sheet here. Now, if you've made it this far into the master class, I bet you're enjoying this content and maybe you're learning something new. But if you want to take it up a notch with me and the rest of my team over at AI Foundations, then I recommend joining our community. because it's not just me. It's not just courses, but we also have this amazing community feed where people are sharing their
31:30 projects. People are helping each other. They're working through automations and setting up AI systems. And if I scroll up here to our wins category, you're even going to see if you scroll through here, people are getting their first client. People are getting their second and third client. People are even landing jobs and building new automations. And we haven't even gotten to the best part yet. If we scroll up here and we go to the classroom, you're going to see that we have all of these different courses. We start with a start here course, kind of give you a road map for how to use the whole community to your full advantage. And then all these white courses here, these are the pillars. These are the different things that you learn when you're first getting into AI that are going to give you that solid base. These are the applicable skills that you're going to need in the workforce if you're trying to get a job in AI or if you're trying to start an
32:16 agency or grow your business. And then we've got a bunch of other courses in the pipeline. We have mastering agents with NAND. And I think this one's going to be relevant to you guys. If you're enjoying NADN and you want to step it up a notch and you want to learn from me and my brother here, we have an ever growing course portal in here that has a bunch of different foundational trainings on NAD so that you can learn everything that you need to know to build effective agents and systems in NAD. We even have these AI mastermind calls where you can come and you can talk with other members in more of a community setting where it's more of like a roundt mastermind. You don't have to be a pro. Some people are, some people aren't. We're at all different levels, but we're all excited about the advancements of AI. So, the only prerequisite I would say for this call
33:01 is that AI gets you excited. So, you do not want to go at this AI thing alone. Nobody really can keep up with everything. So, it's great to have a group of people around you that are both learning about it, coming back and teaching it, but also just people to network and talk and share your ideas with. If you want to join the community, I recommend you do it now because we are raising the price at 1500 members. And right now, we're creeping towards that number. There's less than 100 spots left until we raise that price. So, what I want you guys to do right now if you want to join is to use the link in the description or the top pinned comment. Go join, get integrated, and then come back to this video and we'll hit you with some more awesome free content here on YouTube as well. All right, so we're back here in NAND and the automation is pretty much
33:46 ready to go at this point. If I wanted to, I could simply come in and just change the URL on this every time that I wanted to pass it and I could continue adding to my spreadsheet. But we're going to polish this and add the finishing touches on it. I'm going to show you how to add an input form if you just want to put in a new website to scrape, but I'll also show you how to automate this entirely if you don't want to be putting those websites in manually by yourself. So, the nice thing about NAD is we can actually delete this first trigger node here and we can replace it. So, right here where it appeared add first step after I deleted that, I can click on that and I can type in form and you're going to see NAD form over here. So if I click NADN form and then I click on new NADN form event. Now I can define
34:32 my form. So I can give it a title. Let's just say scraping form. And then we'll give this form a description. So I'll just say enter your URL here to get a scraped output. And then we're going to add form element. So here I'm just going to type in URL for the field name. And this is where we're going to enter the URL. I can put in an example. We'll just do https colon/google.com and I'll put eg just for an example there. And for the element type, we'll keep it on text. And we're going to make sure that this field is required because it's going to need this to run the automation. You want to make sure respond when is on when the form is submitted. So that this actually triggers the automation when you hit submit on the form. And now I'll go back to canvas and I'll reconnect this on
35:18 form submission to our HTTP request. Now, if I hit test workflow, it's going to open up the form for me here and I can put in any URL. And then once we put in this URL, we can then pass it to our HTTP request. So, now you're seeing that we can start this automation simply by entering a URL in this form. And on the other end, we can have it in our Google sheet or you can have an email sent and packaged up that goes out to somebody who's using this product. This is where it gets powerful because you could sell this solution. You could package it up. You could sell it for thousands of dollars to some business owner that's looking for a tailored scraping solution. So, for this test here, I'm just going to paste in a test site, books.crape.com, and I'm going to hit submit. And then it says, "Your form response has been recorded." Now, we'll
36:03 go back over to our NAD tab, and we'll see that this thing just ran through. Now, what you'll see here is that it actually completed that once again. and it went through our entire automation after submitting that form, but we didn't pass in that URL yet. So, it just went through with that fire crawl link again. But if we click into our HTTP request here, we can then scroll down to the URL. We can remove that old URL and we can drag in the URL from our form event into that URL. And if I scroll down, you're going to see it didn't drag it in precisely. So, I'll just copy that without destroying anything here with the formatting. And then I'll just paste that into the URL. So, it just needs to be between those quotes on the URL
36:49 object here. And now, if I go back to Canvas and we test the workflow again, it'll open up our form. We'll try this again. books.crape.com. And I'll put in the https slash just to be safe and make sure that goes through. And now I'm going to hit submit. And now if we go back over to naden, we'll see that it's running through that automation with this custom URL. So it's going to that site right now, scraping it. It's going to then pass it over to OpenAI. And now it's passing it to OpenAI and then to our Google sheet. So, if I go over to our Google sheet here, you're going to see that it pulled in the result for books to scrape. So, if I drag this up
37:34 here, resize this column. Now, you can see that it gave us a title. So, book catalog demo, books to scrape. Boom. That's right. And then the summary, a demo website listing 10,000 books with prices and availability for web scraping practicing. Site data includes book titles, prices, and instock statuses with no real commercial purpose. That's right. And then there were no emails found, so it put the N/ A there. Now, this automation is going to work amazing if you just want a selfserve. Enter the URL into the form and then have the URL appear in Google Sheets along with all of that information. Ran through an AI tool, did all of that work. And you could even add a step in here for Gmail if you wanted to send an email. And the nice thing is is you can pass in an email on this field here. You can pass
38:21 in the URL. So somebody can come and actually use this and then receive a custom tailored email with all of the information that you want to give them. I'm going to make sure that I go and save this workflow. And then we're actually going to create an automated version of it. So this is working and this is good to go. But you might be wondering, what if I want to automate a scraping task? For instance, what if I wanted to grab news articles and scrape those web pages and then create content based on those news articles? This can be a great way to do marketing for your industry and you can kind of just do it in the background. So if I wanted to do something like this, I could just copy our existing workflow with control C and then if I hit Ctrl +V, it'll paste it and we can actually create a different version of this. You could also do this
39:06 in a separate NAD workflow, but in this case I'm going to be repurposing a lot of these different nodes that I've put in here. So I'm going to show you how to do it in this workflow right here. So what we can actually do is on this start here we can have it start with when a new article comes in and to do this you can go to rss.app. This is a website where you can actually track different categories or different websites and whenever they come out with a new article you can pass it over to your scraper and then from there you can do all kinds of content automations with it. So with rss.app app, you're just going to go to the login button or get started. And then once you're logged in, you're going to click on this my feeds right here. So I'll click on my feeds. Then I'm going
39:52 to create a new feed. So I'll hit new feed. And by default, we have all of these different templates that they offer. In this case, let's just go with a Yahoo newsfeed. This is just going to pull in general news. So if I hit the Yahoo newsfeed, now I'm just going to go with the news.yahoo.com. I'll hit try now. And here it's going to create a list of all the different articles that are coming through on Yahoo News. So this is basic news. There's a lot of politics. There's some sports. There's a little bit of everything here. But this is updating on a regular basis. So you can see here if I scroll all the way to the top, we have a post that came through 1 hour ago. So anytime Yahoo News posts, it's going to pass that through and then it's going to trigger our automation. So, this is where it gets really powerful, guys,
40:38 because you can have this running in the background, monitoring the news for you, and then creating real-time content using AI. So, once you find a feed that you like, I'm just going to hit save to my feeds. Then, you're going to notice that it pops up over here in the sidebar. So, if I click on it, it's just going to pull me right back here. And now I have this in my feed. If you hover over this title here, you're going to see your feed URL. So, what you're going to want to do is you're going to want to copy that feed URL. And then the beautiful thing is guys, we can go right back over to NADN. And here for this first trigger node, we can just have it trigger when a new article comes out on Yahoo News. So I'll go ahead and delete this for now. And we'll click the plus up here. And I'm just going to type in RSS and you should see the RSS feed
41:25 trigger. I'll click on that. And then we can paste in our feed URL right here. So I'll just paste that. And now it's going to be looking at that feed that I have. And it can pull every minute, every hour, or you can even do a custom pull. And you can do a chron expression here if you're a little bit more experienced with development. Or you can just have chat GPT help you write out a chron expression to have it do it by the second if you really wanted to. I don't think that's necessary. I think what we're going to do is we're just going to do uh every hour in this case because there's not articles coming out that need to be seen more than every hour in my case. But you could switch this up if you wanted to. Now I'm going to hit fetch test event. And as you can see here, it pulled through the latest article for
42:11 me. So it gives me the title here. It gives me the link, which is most important. That's what we're going to use to scrape. It gives me the publishing date. So we can see that it was today creator. And then it gives us a preview of the content as well. Beautiful. So now let's go back to the canvas. And as you can see, this one triggered. But if we connect it to our HTTP request, we can then click into that HTTP request. And if we scroll down to our URL here, we can just remove that old URL. And then we can drag in our link from the RSS feed trigger. Just like so. And now it's going to use that as the request. So if I hit test right here, now it's going to run that through firecrawl and go and get that data for us. Beautiful. So it came through here with the data. It gave us the full
42:56 article. And now if we wanted to, we could go back to canvas and we can readjust our AI model here to be able to uh make content based on the news. So I'll click into here and now it's just as simple as altering our system prompt. Okay. So I'm just going to say your role is to extract information from the article data that I pass you and output it in a structured format. You deliver the following. So, for the title here, we could just do a LinkedIn post or something like that. So, I'm just going to do content and then I could go and search, you know, how many characters a LinkedIn post is allowed to be. Perfect question for chat GPT. So, I'll say how many characters is a good LinkedIn
43:45 post. Okay. So, I'll go with 500 to 900 characters. So, I'll just copy that. We'll head back to NAND. And for the content, I'm just going to say content for a LinkedIn post with then I'll paste in the character count there, 500 to 900 characters. I'll delete the rest of these because all we need is the content. And then where it says you receive an input of scraped website, I'm just going to call it scraped article. And here we'll also rename this to article just to keep it consistent. Article. And the results that you get from this are going to come down to how well you can write this system prompt, the instructions for this agent. Oh, another thing we've got to do is we have to get rid of those other uh JSON formatting. So, I'll do content alone.
44:33 I'll get rid of those and we'll get rid of that last comma there. Okay, perfect. Just as a note, you're going to get better results if you know how to write a good system prompt here and you know how to pass in good examples of what good content looks like. So, if you really want to take your prompt engineering game up a notch, then that's another reason that I would recommend joining the community because this is a journey. As the models change and as they improve and as there's new nuances to learn, it's good to stay on top of this stuff with our course content and with a community of passionate AI enthusiasts who know how to get the best results when it comes to prompting. So again, if you really want to uplevel these prompts, then I recommend clicking the link in the description below, joining the community before the price goes up and really just getting integrated with us and really just immersing yourself in this world of AI
45:19 because it changes so fast. And like I said, you do not want to go at this thing alone. I'm just going to add a little something here to the system prompt to make it understand exactly what it's doing. And I'm just going to say, "Your content output should be an engaging repurposed unique version of the article that's passed. And I'm also just going to say don't copy it word for word, but also don't add any info that's not factual. Okay, so that was a really quick prompt. I didn't really think it through much, but I just wanted to give you guys the idea that you can really add a lot to these prompts. And you can even go in here and you can add examples of inputs and
46:06 outputs and I show you guys how to do that in other videos and we also go in-depth in the community. But for now, this will do. So if I hit test step now, it's going to look at this markdown that we've passed through because that variable hasn't changed. That stays the same. And it's going to come through here and it's just going to give me a piece of content. And this looks great. It's giving me a detailed breakdown of the article and all the information within it. So, this could be something that we could repurpose for X or we could repurpose for LinkedIn or whatever we want. And I'm just using general news that comes through from Yahoo News, but you could use any website that you want and you could run any kind of automation, whether it's internal or external for marketing. The sky is really the limit and you can get creative with this stuff. Now, I'm going
46:51 to go back to the canvas here and now there's just one final step of updating our Google sheet. So, in this case, it's going to our scraped websites Google sheet. I'm going to create a new Google sheet, though, because I like how this one's set up. I don't want to change anything. I'm just going to go back to our sheets home and click blank spreadsheet. And then from here, I'm just going to call this one LinkedIn posts. LinkedIn posts. And I'm just going to add a field here for content. That's all we need to pass for this one. Now, I'll go back over to NAND, and we're going to reconfigure this Google sheet here. As you can see, it has all of our old information in here. But if I go to document, I can then see that I have this LinkedIn posts one. So, I'll switch to that. We'll
47:36 change the sheet to sheet one because that's the default sheet. And then it's going to pull in our content column. And from there, we can just drag in our content from OpenAI. And it's good to go. Just like that, we have an automated content farm right here for us. Now, to keep these automations nice and separated, I'm just going to click, drag, and copy this second one. I'm not going to delete it yet. I'm just going to save this workflow. And we're going to hop over to an overview. Then we're going to create a new workflow. And I'm just going to rename this one to LinkedIn post generator. And we'll click into the page here. And now I'm just going to hit controlV to paste. And boom. It's that easy. Now we've got our LinkedIn post
48:22 generator here that I copied from the other automation. Now, if you want this to run in the background for you and you want this RSS feed to trigger every hour, then you're going to have to click activate up here. So, turn that automation on and then hit got it. And you can hit save. Now, let's go ahead and test this through one more time. But now, it should just be running in the background for us after activating that. So, let's go ahead and hit test workflow and see how this thing works. So, it's going all the way through the automation, hitting our HTTP request, going and getting that detailed content. And in a moment here, it's going to pass that over to OpenAI, and it's going to run that automation to generate our LinkedIn content. And then, boom, it's sending it over to our Google sheet. Now, let's go over to our LinkedIn posts. And as you can see here, we have
49:08 that LinkedIn post coming in. And this is actually a different uh article that it pulled in this time. So, there must have been something new that came up. But then from there, we can just copy and paste this, pull it over to LinkedIn, and we have a draft for a post. Of course, you want to be in the loop. You want to do your own research. But you also want to save time and start with some valuable ideas. We could totally automate this. We could pass it directly to LinkedIn if we wanted to. Like there is a LinkedIn node in here, I'm pretty sure. So, if I typed in LinkedIn, yeah, as you can see, I can just pull in LinkedIn and create a post if I wanted to. And boom, you hook up your credential, you have a post made, and it just goes out automatically. But I like this Google Sheets route because you can kind of add your own two cents and this is just a starting point. It's an idea and you're gonna have these coming in here all day as long as you
49:54 activate your workflow and make sure that it's saved. So, this has been how to scrape anything with NADN. I hope you thoroughly enjoyed this guide and if you did, please leave a like so YouTube knows to show you more of this content, this thorough amazing content that actually helps you learn, that actually gets to the point of how to do things and breaks it down for a beginner. And on top of that, if you really enjoyed my content and my presence, then I recommend subscribing to my channel and hitting the bell icon so that you get notified when I come out with a new video. If you really want to take it up a notch with me and my team, then you guys can join the AI Foundations community using the link below in the description or the top pinned comment. This is the future, guys. This is where things are headed. This is where work is going to be. And if this video didn't convince you of that, if the automations that you're seeing here didn't get you excited about it, then it probably isn't
50:40 for you. But if you see the future with this and you need a little bit of extra help or an amazing network to surround yourself with, then it's the right move. So, I hope to see you over there. If not, I hope to see you on future free videos here on YouTube. I'm happy to put them out. And with that guys, have a beautiful day or a beautiful evening, good morning or good night.
