Web Automation Tutorial #1 - Setup, Button Clicking, Web Scraping

0:00 welcome everyone in this series we'll be going over web automation from start to finish you do not need any prior coding experience for this series as the series is built for any level from beginner to Advanced so whether you have wrote zero lines of code or 1 million lines of code it will not make a difference you will learn how to fully control a browser using Code as well as making the browser click buttons going to Pages navigating across various websites scraping and a whole bunch of other web automation tasks this is the first video of the series we will have to set everything up so first things first you must download node.js you simply go to this website right here it will automatically detect whether you have a Mac windows and which form and all you must do is just click the download button uh I already got things started I already have nodejs downloaded but once it is downloaded all you have to do is just open your package

0:47 uh download whatever it asks you to download and boom you'll be good and then you're going to want to download Puppeteer but how exactly will you be able to do this we'll go over that very shortly but what is Puppeteer so Puppeteer is a no s library that will we will be utilizing to control our browser right this will be like The Mastermind behind the entire web automation task we will be doing this is a very important um framework that we'll be using there's a lot of documentation a lot of resources online about it which is why it is probably the most famous I know Selen is another big one but I always used Puppeteer and that's how I made most of my uh web automation programs so now what you must do is you just got to go to your desktop you can create a new folder say web Automation and you just got to open this

1:33 folder in Visual Studio code so so I opened this folder now I just make a new file let's call it index.js and you're going to want to open your terminal now this is how you check if you downloaded nodejs successfully your node.js -- version so if something comes up then that means you have node in your computer otherwise you get an error saying hey node is not found this command is not found in that case you do not download nodejs properly so you want to make sure you have it downloaded perfectly um and now you just got to do npm init and I just Spam enter at this point and you will have a package.json you got to make sure this is there and

2:19 now you got to do mpm install Puppeteer this is how you download Puppeteer and are able to use it in your program now I already have puppet you're downloaded but uh because this is a new folder it has nothing in it so you got to download all the node modules you want to download and while that's downloaded we can start looking at the documentation we're going to just want to copy this beginning part right here let's go to index.js and now we're going to do a curly brace parentheses this will automatically call it and then semicolon all right so puup tier is downloaded so now we do is const Puppeteer equals

3:05 require Puppeteer then now we'll be able to use Puppeteer uh successfully so you want to copy these two lines of code right here um one thing you want to do is you want to do headless false this way we will actually be able to see our browser okay and let's let's make this go to any website let's say google.com uh you want to format it properly so make sure the URL is a full URL all right so you save it and now we

3:50 run it doing no index.js and we're going to see the browser pop up very shortly and it will go to google.com the first time you run it might be a little slower but you'll see boom google.com and now you're like well this looks kind of funky in fact why is it like half the size well to fix that what we're going to do is we're going to first cancel this if you look at the documentation they have a set viewport which will basically set the dimensions of your page okay so by setting the viewport to this we can now do node index.js and we're going to see a sign significant difference in the size you see right here the browser looks much more normal but there's show some gray space right here what we're going to

4:36 want to do now is the following we're going to want to close this and we're going to want to install a better version of Puppeteer that already has all these things figured out for us in order to do that we're going to go to this right here puppeter extra plugin install it's a very important MPN package that basically masks your browser so no website is able to tell that you are a automated program they all think you're real a real user so you just want to paste that right here and now we can run puppet tier plugin steth so to do so we'll just copy these two these three lines right here you see all evasion techniques save

5:22 it take and take this viewport line out and we can now run the program if you get this module not found okay it's because I didn't install Puppeteer extra you want to make sure you install both all right now should be good uh puer do use not a function okay so to get past that error all you have to do is just change the way you're calling the code so if you just do async function run and you get this code right here copy it delete this paste it there and you can call run you'll be able to successfully run the program so let's run

6:09 it and you can see how Google Now fills up the entire screen and it looks like a real browser you cannot be able to tell that this is chromium not Chrome okay and now what we can do we can go to any website we want let's let's go to walmart.com let's find a product on Walmart.com okay so the iPad Air let's say I want my browser to click this button right here what I'm going to have to do is go to Dev tools let's bring it to the bottom click it right here so you want to look at through the HTML this is how

6:56 you look at the HTML the button okay and we want to copy we want to Target this button right so what we're going to have to do is find the selector so in this case the selector will be button this this is the object the the HTML element is a button and we're going to look at this data automation ID attribute and get the value so in other words if we just go to the console this is how you verify that you that you can through code detect a element on a page you want on a selector it's a query selector button remember is data automation ID equals single quotes ATC bracket double quotes enter so you

7:43 see right here how it's highlighting that button so all we have to do now is just copy that let's let selector equal oh so this is a full select right and let's make let's make sure we go to the correct website okay now we do a wait page. weit for selector now this will do is the program will wait for this selector to appear and once it does appear we can do await page. evaluate and this is how you inject

8:28 JavaScript directly into your browser so we do document. query selector do click right and that matches exactly what we did here right um the dot click makes the button actually be clicked and invokes a click property of the button and we see right here that we pass in a variable called selector which we use inside the actual JavaScript code we're going to be injecting to the browser and this selector is just like

9:14 the actual variable right here okay so if we were to name this s we could but this has to be selector because that's the variable we're passing in right but just to make it simple and easy I just keep the variable names the same and that's all we need to do so now if you were to rerun this we can see it successfully pre-order let's verify run the browser see right there it just clicked it by itself handsfree okay so it's added to cart and now you were just able to click a button on a website by running code like isn't that crazy you don't have to use your hands right and this is why web automation can be so powerful and now in the future videos we'll be going more in

10:00 depth into the many things you can do with Puppeteer JS and web Automation and we're create some very cool projects scrapers Bots agents uh chat Bots a whole bunch of things so if you're interested in watching the rest of this series be sure to like the video subscribe hit the notification Bell so you don't miss any upload thank you all for watching peace