Machine Learning for Everybody – Full Course

0:00 Kylie Ying has worked at many interesting places such as MIT, CERN, and Free Code Camp. She's a physicist, engineer, and basically a genius. And now she's going to teach you about machine learning in a way that is accessible to absolute beginners. What's up you guys? So welcome to Machine Learning for Everyone. If you are someone who is interested in machine learning and you think you are considered as everyone, then this video is for you. In this video, we'll talk about supervised and unsupervised learning models, we'll go through maybe a little bit of the logic or math behind them, and then we'll also see how we can program it on Google CoLab. If there are certain things that I have done, and you know,

0:46 you're somebody with more experience than me, please feel free to correct me in the comments and we can all as a community learn from this together. So with that, let's just dive right in. Without wasting any time, let's just dive straight into the code and I will be teaching you guys concepts as we go. So this here is the UCI machine learning repository. And basically, they just have a ton of data sets that we can access. And I found this really cool one called the magic gamma telescope data set. So in this data set, if you want to read all this information, to summarize what I what I think is going on, is there's this gamma telescope, and we have all these high energy particles hitting the telescope. Now there's a camera, there's a detector that

1:34 actually records certain patterns of you know, how this light hits the camera. And we can use properties of those patterns in order to predict what type of particle caused that radiation. So whether it was a gamma particle, or some other head, like hadron. Down here, these are all of the attributes of those patterns that we collect in the camera. So you can see that there's, you know, some length, width, size, asymmetry, etc. Now we're going to use all these properties to help us discriminate the patterns and whether or not they came from a gamma particle or hadron. So in order to do this, we're going to come up here, go to the data folder. And you're going

2:19 to click this magic zero for data, and we're going to download that. Now over here, I have a colab notebook open. So you go to colab dot research dot google.com, you start a new notebook. And I'm just going to call this the magic data set. So actually, I'm going to call this for code camp magic example. Okay. So with that, I'm going to first start with some imports. So I will import, you know, I always import NumPy, I always import pandas. And I always import matplotlib.

3:06 And then we'll import other things as we go. So yeah, we run that in order to run the cell, you can either click this play button here, or you can on my computer, it's just shift enter and that that will run the cell. And here, I'm just going to order I'm just going to, you know, let you guys know, okay, this is where I found the data set. So I've copied and pasted this actually, but this is just where I found the data set. And in order to import that downloaded file that we we got from the computer, we're going to go over here to this folder thing. And I am literally just going to drag and drop that file into here. Okay. So in order to take a look at, you know, what does this file consist of,

3:55 do we have the labels? Do we not? I mean, we could open it on our computer, but we can also just do pandas read CSV. And we can pass in the name of this file. And let's see what it returns. So it doesn't seem like we have the label. So let's go back to here. I'm just going to make the columns, the column labels, all of these attribute names over here. So I'm just going to take these values and make that the column names. All right, how do I do that? So basically, I will come back here, and I will create a list called calls. And I will type in all of those things. With f size, f conk. And we also have f conk one.

4:50 We have f symmetry, f m three long, f m three trans, f alpha. Let's see, we have f dist and class. Okay, great. Now in order to label those as these columns down here in our data frame. So basically, this command here just reads some CSV file that you pass in CSV has come about comma separated values, and turns that into a pandas data frame object. So now if I pass in a names here, then it basically assigns these labels to the columns of this data set. So I'm going to set

5:38 this data frame equal to DF. And then if we call the head is just like, give me the first five things, give me the first five things. Now you'll see that we have labels for all of these. Okay. All right, great. So one thing that you might notice is that over here, the class labels, we have G and H. So if I actually go down here, and I do data frame class unique, you'll see that I have either G's or H's, and these stand for gammas or hadrons. And our computer is not so good at understanding letters, right? Our computer is really good at understanding numbers. So what we're going to do is we're going to convert this to zero for G and one for H. So here, I'm going to set this equal to this, whether or not that equals G. And then

6:35 I'm just going to say as type int. So what this should do is convert this entire column, if it equals G, then this is true. So I guess that would be one. And then if it's H, it would be false. So that would be zero, but I'm just converting G and H to one and zero, it doesn't really matter. Like, if G is one and H is zero or vice versa. Let me just take a step back right now and talk about this data set. So here I have some data frame, and I have all of these different values for each entry. Now this is a you know, each of these is one sample, it's one example, it's one item in our data set, it's one data point, all of these things are kind of the same

7:23 thing when I mentioned, oh, this is one example, or this is one sample or whatever. Now, each of these samples, they have, you know, one quality for each or one value for each of these labels up here, and then it has the class. Now what we're going to do in this specific example is try to predict for future, you know, samples, whether the class is G for gamma or H for hadron. And that is something known as classification. Now, all of these up here, these are known as our features, and features are just things that we're going to pass into our model in order to help us predict the label, which in this case is the class column. So for you know, sample zero, I have

8:14 10 different features. So I have 10 different values that I can pass into some model. And I can spit out, you know, the class the label, and I know the true label here is G. So this is this is actually supervised learning. All right. So before I move on, let me just give you a quick little crash course on what I just said. This is machine learning for everyone. Well, the first question is, what is machine learning? Well, machine learning is a sub domain of computer science that focuses on certain algorithms, which might help a computer learn from data, without a programmer being there telling the computer exactly what to do. That's what we call explicit

9:01 programming. So you might have heard of AI and ML and data science, what is the difference between all of these. So AI is artificial intelligence. And that's an area of computer science, where the goal is to enable computers and machines to perform human like tasks and simulate human behavior. Now machine learning is a subset of AI that tries to solve one specific problem and make predictions using certain data. And data science is a field that attempts to find patterns and draw insights from data. And that might mean we're using machine learning. So all of these fields kind of overlap, and all of them might use machine learning. So there are a few types of machine learning.

9:52 The first one is supervised learning. And in supervised learning, we're using labeled inputs. So this means whatever input we get, we have a corresponding output label, in order to train models and to learn outputs of different new inputs that we might feed our model. So for example, I might have these pictures, okay, to a computer, all these pictures are are pixels, they're pixels with a certain color. Now in supervised learning, all of these inputs have a label associated with them, this is the output that we might want the computer to be able to predict. So for example, over here, this picture is a cat, this picture is a dog, and this picture is a lizard.

10:41 Now there's also unsupervised learning. And in unsupervised learning, we use unlabeled data to learn about patterns in the data. So here are here are my input data points. Again, they're just images, they're just pixels. Well, okay, let's say I have a bunch of these different pictures. And what I can do is I can feed all these to my computer. And I might not, you know, my computer is not going to be able to say, Oh, this is a cat, dog and lizard in terms of, you know, the output. But it might be able to cluster all these pictures, it might say, Hey, all of these have something in common. All of these have something in common. And then these down here have something in common, that's finding some sort of structure in our unlabeled data.

11:33 And finally, we have reinforcement learning. And reinforcement learning. Well, they usually there's an agent that is learning in some sort of interactive environment, based on rewards and penalties. So let's think of a dog, we can train our dog, but there's not necessarily, you know, any wrong or right output at any given moment, right? Well, let's pretend that dog is a computer. Essentially, what we're doing is we're giving rewards to our computer, and tell your computer, Hey, this is probably something good that you want to keep doing. Well, computer agent terminology. But in this class today, we'll be focusing on supervised learning and unsupervised learning

12:21 and learning different models for each of those. Alright, so let's talk about supervised learning first. So this is kind of what a machine learning model looks like you have a bunch of inputs that are going into some model. And then the model is spitting out an output, which is our prediction. So all these inputs, this is what we call the feature vector. Now there are different types of features that we can have, we might have qualitative features. And qualitative means categorical data, there's either a finite number of categories or groups. So one example of a qualitative feature might be gender. And in this case, there's only two here, it's for the sake of

13:07 the example, I know this might be a little bit outdated. Here we have a girl and a boy, there are two genders, there are two different categories. That's a piece of qualitative data. Another example might be okay, we have, you know, a bunch of different nationalities, maybe a nationality or a nation or a location, that might also be an example of categorical data. Now, in both of these, there's no inherent order. It's not like, you know, we can rate us one and France to Japan three, etc. Right? There's not really any inherent order built into either of these categorical data sets. That's why we call this nominal data. Now, for nominal data, the way that we want

14:00 to feed it into our computer is using something called one hot encoding. So let's say that, you know, I have a data set, some of the items in our data, some of the inputs might be from the US, some might be from India, then Canada, then France. Now, how do we get our computer to recognize that we have to do something called one hot encoding. And basically, one hot encoding is saying, okay, well, if it matches some category, make that a one. And if it doesn't just make that a zero. So for example, if your input were from the US, you would you might have 1000. India, you know, 0100. Canada, okay, well, the item representing Canada is one and then France, the item representing

14:46 France is one. And then you can see that the rest are zeros, that's one hot encoding. Now, there are also a different type of qualitative feature. So here on the left, there are different age groups, there's babies, toddlers, teenagers, young adults, adults, and so on, right. And on the right hand side, we might have different ratings. So maybe bad, not so good, mediocre, good, and then like, great. Now, these are known as ordinal pieces of data, because they have some sort of inherent order, right? Like, being a toddler is a lot closer to

15:33 being a baby than being an elderly person, right? Or good is closer to great than it is to really bad. So these have some sort of inherent ordering system. And so for these types of data sets, we can actually just mark them from, you know, one to five, or we can just say, hey, for each of these, let's give it a number. And this makes sense. Because, like, for example, the thing that I just said, how good is closer to great, then good is close to not good at all. Well, four is closer to five, then four is close to one. So this actually kind of makes sense. And it'll make sense for the computer as well. Alright, there are also quantitative pieces of data and quantitative

16:22 pieces of data are numerical valued pieces of data. So this could be discrete, which means, you know, they might be integers, or it could be continuous, which means all real numbers. So for example, the length of something is a quantitative piece of data, it's a quantitative feature, the temperature of something is a quantitative feature. And then maybe how many Easter eggs I collected in my basket, this Easter egg hunt, that is an example of discrete quantitative feature. Okay, so these are continuous. And this over here is the screen. So those are the things that go into our feature vector, those are our features that we're feeding this model, because

17:08 our computers are really, really good at understanding math, right at understanding numbers, they're not so good at understanding things that humans might be able to understand. Well, what are the types of predictions that our model can output? So in supervised learning, there are some different tasks, there's one classification, and basically classification, just saying, okay, predict discrete classes. And that might mean, you know, this is a hot dog, this is a pizza, and this is ice cream. Okay, so there are three distinct classes and any other pictures of hot dogs, pizza or ice cream, I can put under these labels. Hot dog, pizza, ice cream.

17:56 Hot dog, pizza, ice cream. This is something known as multi class classification. But there's also binary classification. And binary classification, you might have hot dog, or not hot dog. So there's only two categories that you're working with something that is something and something that's isn't binary classification. Okay, so yeah, other examples. So if something has positive or negative sentiment, that's binary classification. Maybe you're predicting your pictures of their cats or dogs. That's binary classification. Maybe, you know, you are writing an email filter, and you're trying to figure out if an email spam or not spam. So that's also binary classification.

18:41 Now for multi class classification, you might have, you know, cat, dog, lizard, dolphin, shark, rabbit, etc. We might have different types of fruits like orange, apple, pear, etc. And then maybe different plant species. But multi class classification just means more than two. Okay, and binary means we're predicting between two things. There's also something called regression when we talk about supervised learning. And this just means we're trying to predict continuous values. So instead of just trying to predict different categories, we're trying to come up with a number that you know, is on some sort of scale. So some examples. So some examples might be the price of aetherium tomorrow, or it might be okay, what is going to be the temperature?

19:31 Or it might be what is the price of this house? Right? So these things don't really fit into discrete classes. We're trying to predict a number that's as close to the true value as possible using different features of our data set. So that's exactly what our model looks like in supervised learning. Now let's talk about the model itself. How do we make this model learn? Or how can we tell whether or not it's even learning? So before we talk about the models, let's talk about how can we actually like evaluate these models? Or how can we tell whether something is a good model or bad model? So let's take a look at this data set. So this data

20:19 set has this is from a diabetes, a Pima Indian diabetes data set. And here we have different number of pregnancies, different glucose levels, blood pressure, skin thickness, insulin, BMI, age, and then the outcome whether or not they have diabetes one for they do zero for they don't. So here, all of these are quantitative features, right, because they're all on some scale. So each row is a different sample in the data. So it's a different example, it's one person's data, and each row represents one person in this data set. Now this column, each column represents a

21:04 different feature. So this one here is some measure of blood pressure levels. And this one over here, as we mentioned is the output label. So this one is whether or not they have diabetes. And as I mentioned, this is what we would call a feature vector, because these are all of our features in one sample. And this is what's known as the target, or the output for that feature vector. That's what we're trying to predict. And all of these together is our features matrix x. And over here, this is our labels or targets vector y. So I've condensed this to a chocolate

21:51 bar to kind of talk about some of the other concepts in machine learning. So over here, we have our x, our features matrix, and over here, this is our label y. So each row of this will be fed into our model, right. And our model will make some sort of prediction. And what we do is we compare that prediction to the actual value of y that we have in our label data set, because that's the whole point of supervised learning is we can compare what our model is outputting to, oh, what is the truth, actually, and then we can go back and we can adjust some things. So the next iteration, we get closer to what the true value is. So that whole process here, the tinkering that,

22:41 okay, what's the difference? Where did we go wrong? That's what's known as training the model. Alright, so take this whole, you know, chunk right here, do we want to really put our entire chocolate bar into the model to train our model? Not really, right? Because if we did that, then how do we know that our model can do well on new data that we haven't seen? Like, if I were to create a model to predict whether or not someone has diabetes, let's say that I just train all my data, and I see that all my training data does well, I go to some hospital, I'm like, here's my model. I think you can use this to predict if somebody has diabetes. Do we think that would

23:28 be effective or not? Probably not, right? Because we haven't assessed how well our model can generalize. Okay, it might do well after you know, our model has seen this data over and over and over again. But what about new data? Can our model handle new data? Well, how do we how do we get our model to assess that? So we actually break up our whole data set that we have into three different types of data sets, we call it the training data set, the validation data set and the testing data set. And you know, you might have 60% here 20% and 20% or 80 10 and 10. It really depends on how

24:15 many statistics you have, I think either of those would be acceptable. So what we do is then we feed the training data set into our model, we come up with, you know, this might be a vector of predictions corresponding with each sample that we put into our model, we figure out, okay, what's the difference between our prediction and the true values, this is something known as loss, losses, you know, what's the difference here, in some numerical quantity, of course. And then we make adjustments, and that's what we call training. Okay. So then, once you know, we've made a bunch of adjustments, we can put our validation set through this model. And the validation set is kind of used as a reality

25:06 check during or after training to ensure that the model can handle unseen data still. So every single time after we train one iteration, we might stick the validation set in and see, hey, what's the loss there. And then after our training is over, we can assess the validation set and ask, hey, what's the loss there. But one key difference here is that we don't have that training step, this loss never gets fed back into the model, right, that feedback loop is not closed. Alright, so let's talk about loss really quickly. So here, I have four different types of models, I have some sort of data that's being fed into the model, and then some output. Okay, so this output

25:52 here is pretty far from you know, this truth that we want. And so this loss is going to be high. In model B, again, this is pretty far from what we want. So this loss is also going to be high, let's give it 1.5. Now this one here, it's pretty close, I mean, maybe not almost, but pretty close to this one. So that might have a loss of 0.5. And then this one here is maybe further than this, but still better than these two. So that loss might be 0.9. Okay, so which of these model performs the best? Well, model C has a smallest loss, so it's probably model C. Okay, now let's

26:40 take model C. After you know, we've come up with these, all these models, and we've seen, okay, model C is probably the best model. We take model C, and we run our test set through this model. And this test set is used as a final check to see how generalizable that chosen model is. So if I, you know, finish training my diabetes data set, then I could run it through some chunk of the data and I can say, oh, like, this is how we perform on data that it's never seen before at any point during the training process. Okay. And that loss, that's the final reported performance of my test set, or this would be the final reported performance of my model. Okay.

27:29 So let's talk about this thing called loss, because I think I kind of just glossed over it, right? So loss is the difference between your prediction and the actual, like, label. So this would give a slightly higher loss than this. And this would even give a higher loss, because it's even more off. In computer science, we like formulas, right? We like formulaic ways of describing things. So here are some examples of loss functions and how we can actually come up with numbers. This here is known as L one loss. And basically, L one loss just takes the absolute value of whatever your you know, real value is, whatever the real output label is,

28:18 subtracts the predicted value, and takes the absolute value of that. Okay. So the absolute value is a function that looks something like this. So the further off you are, the greater your losses, right in either direction. So if your real value is off from your predicted value by 10, then your loss for that point would be 10. And then this sum here just means, hey, we're taking all the points in our data set. And we're trying to figure out the sum of how far everything is. Now, we also have something called L two loss. So this loss function is quadratic, which means that if it's close, the penalty is very minimal. And if it's off by a lot,

29:08 then the penalty is much, much higher. Okay. And this instead of the absolute value, we just square the the difference between the two. Now, there's also something called binary cross entropy loss. It looks something like this. And this is for binary classification, this this might be the loss that we use. So this loss, you know, I'm not going to really go through it too much. But you just need to know that loss decreases as the performance gets better. So there are some other measures of accurate or performance as well. So for example, accuracy, what is accuracy?

29:55 So let's say that these are pictures that I'm feeding my model, okay. And these predictions might be apple, orange, orange, apple, okay, but the actual is apple, orange, apple, apple. So three of them were correct. And one of them was incorrect. So the accuracy of this model is three quarters or 75%. Alright, coming back to our colab notebook, I'm going to close this a little bit. Again, we've imported stuff up here. And we've already created our data frame right here. And this is this is all of our data. This is what we're going to use to train our models. So down here,

30:40 again, if we now take a look at our data set, you'll see that our classes are now zeros and ones. So now this is all numerical, which is good, because our computer can now understand that. Okay. And you know, it would probably be a good idea to maybe kind of plot, hey, do these things have anything to do with the class. So here, I'm going to go through all the labels. So for label in the columns of this data frame. So this just gets me the list. Actually, we have the list, right? It's called so let's just use that might be less confusing of everything up to the last thing, which is the class. So I'm going to take all these 10 different features. And I'm going

31:26 to plot them as a histogram. So and now I'm going to plot them as a histogram. So basically, if I take that data frame, and I say, okay, for everything where the class is equal to one, so these are all of our gammas, remember, now, for that portion of the data frame, if I look at this label, so now these, okay, what this part here is saying is, inside the data frame, get me everything where the class is equal to one. So that's all all of these would fit into that category, right? And now let's just look at the label column. So the first label would be f length, which would

32:14 be this column. So this command here is getting me all the different values that belong to class one for this specific label. And that's exactly what I'm going to put into the histogram. And now I'm just going to tell you know, matplotlib make the color blue, make this label this as you know, gamma set alpha, why do I keep doing that, alpha equal to 0.7. So that's just like the transparency. And then I'm going to set density equal to true, so that when we compare it to the hadrons here, we'll have a baseline for comparing them. Okay, so the density being true just basically normalizes these distributions. So you know, if you have 200 in of one type,

33:05 and then 50 of another type, well, if you drew the histograms, it would be hard to compare because one of them would be a lot bigger than the other, right. But by normalizing them, we kind of are distributing them over how many samples there are. Alright, and then I'm just going to put a title on here and make that the label, the y label. So because it's density, the y label is probability. And the x label is just going to be the label. What is going on. And I'm going to include a legend and PLT dot show just means okay, display the plot. So if I run that, just be up to the last item. So we want a list, right, not just the last

33:54 item. And now we can see that we're plotting all of these. So here we have the length. Oh, and I made this gamma. So this should be hadron. Okay, so the gammas in blue, the hadrons are in red. So here we can already see that, you know, maybe if the length is smaller, it's probably more likely to be gamma, right. And we can kind of you know, these all look somewhat similar. But here, okay, clearly, if there's more asymmetry, or if you know, this asymmetry measure is larger, then it's probably hadron. Okay, oh, this one's a good one. So f alpha seems like hadrons are pretty evenly

34:44 distributed. Whereas if this is smaller, it looks like there's more gammas in that area. Okay, so this is kind of what the data that we're working with, we can kind of see what's going on. Okay, so the next thing that we're going to do here is we are going to create our train, our validation, and our test data sets. I'm going to set train valid and test to be equal to this. So NumPy dot split, I'm just splitting up the data frame. And if I do this sample, where I'm sampling everything, this will basically shuffle my data. Now, if I I want to pass in where exactly I'm splitting my data set, so the first split is going to be maybe at 60%. So I'm going

35:38 to say 0.6 times the length of this data frame. So and then cast that 10 integer, that's going to be the first place where you know, I cut it off, and that'll be my training data. Now, if I then go to 0.8, this basically means everything between 60% and 80% of the length of the data set will go towards validation. And then, like everything from 80 to 100, I'm going to pass my test data. So I can run that. And now, if we go up here, and we inspect this data, we'll see that these columns seem to have values in like the 100s, whereas this one is 0.03. Right? So the scale of all these numbers is way off. And sometimes that will affect our results. So I'm going to run this

36:28 is way off. And sometimes that will affect our results. So one thing that we would want to do is scale these so that they are, you know, so that it's now relative to maybe the mean and the standard deviation of that specific column. I'm going to create a function called scale data set. And I'm going to pass in the data frame. And that's what I'll do for now. Okay, so the x values are going to be, you know, I take the data frame. And let's assume that the columns are going to be,

37:14 you know, that the label will always be the last thing in the data frame. So what I can do is say data frame, dot columns all the way up to the last item, and get those values. Now for my y, well, it's the last column. So I can just do this, I can just index into that last column, and then get those values. Now, in, so I'm actually going to import something known as the standard scalar from sk learn. So if I come up here, I can go to sk learn dot pre processing. And I'm going to import standard scalar, I have to run that cell, I'm going to come back down here.

38:04 And now I'm going to create a scalar and use that skip or so standard scalar. And with the scalar, what I can do is actually just fit and transform x. So here, I can say x is equal to scalar dot fit, fit, transform x. So what that's doing is saying, okay, take x and fit the standard scalar to x, and then transform all those values. And what would it be? And that's going to be our new x. Alright. And then I'm also going to just create, you know, the whole data as one huge 2d NumPy array. And in order to do that, I'm going to call H stack. So H stack is saying,

38:53 okay, take an array, and another array and horizontally stack them together. That's what the H stands for. So by horizontally stacked them together, just like put them side by side, okay, not on top of each other. So what am I stacking? Well, I have to pass in something so that it can stack x and y. And now, okay, so NumPy is very particular about dimensions, right? So in this specific case, our x is a two dimensional object, but y is only a one dimensional thing, it's only a vector of values. So in order to now reshape it into a 2d item, we have to call NumPy dot reshape. And we can pass in the dimensions of its reshape. So if I pass in negative

39:45 one comma one, that just means okay, make this a 2d array, where the negative one just means infer what what this dimension value would be, which ends up being the length of y, this would be the same as literally doing this. But the negative one is easier because we're making the computer do the hard work. So if I stack that, I'm going to then return the data x and y. Okay. So one more thing is that if we go into our training data set, okay, again, this is our training data set. And we get the length of the training data set. But where the training data sets class is one, so remember that this is the gammas. And then if we print that, and we do the same thing, but zero,

40:39 we'll see that, you know, there's around 7000 of the gammas, but only around 4000 of the hadrons. So that might actually become an issue. And instead, what we want to do is we want to oversample our our training data set. So that means that we want to increase the number of these values, so that these kind of match better. And surprise, surprise, there is something that we can import that will help us do that. It's so I'm going to go to from in the learn dot oversampling. And I'm going to import this random oversampler, run that cell, and come back down here. So I will actually

41:31 add in this parameter called oversample, and set that to false for default. And if I do want to oversample, then what I'm going to do, and by oversample, so if I do want to oversample, then I'm going to create this ROS and set it equal to this random oversampler. And then for x and y, I'm just going to say, okay, just fit and resample x and y. And what that's doing is saying, okay, take more of the less class. So take take the less class and keep sampling from there to increase the size of our data set of that smaller class so that they now match. So if I do this, and I scale

42:24 data set, and I pass in the training data set where oversample is true. So this let's say this is train and then x train, y train. Oops, what's going on? These should be columns. So basically, what I'm doing now is I'm just saying, okay, what is the length of y train? Okay, now it's 14,800, whatever. And now let's take a look at how many of these are type one. So actually, we can just sum that up. And then we'll also see that if we instead switch the label and ask how

43:12 many of them are the other type, it's the same value. So now these have been evenly, you know, rebalanced. Okay, well, okay. So here, I'm just going to make this the validation data set. And then the next one, I'm going to make this the test data set. Alright, and we're actually going to switch oversample here to false. Now, the reason why I'm switching that to false is because my validation and my test sets are for the purpose of you know, if I have data that I haven't seen yet, how does my sample perform on those? And I don't want to oversample for that right now. Like,

43:59 I don't care about balancing those I'm, I want to know if I have a random set of data that's unlabeled, can I trust my model, right? So that's why I'm not oversampling. I run that. And again, what is going on? Oh, it's because we already have this train. So I have to go come up here and split that data frame again. And now let's run these. Okay. So now we have our data properly formatted. And we're going to move on to different models now. And I'm going to tell you guys a little bit about each of these models. And then I'm going to show you how we can do that in our code. So the first model that we're going to learn about is KNN or K nearest neighbors. Okay, so here, I've

44:49 already drawn a plot on the y axis, I have the number of kids that a family might have. And then on the x axis, I have their income in terms of 1000s per year. So, you know, if if someone's making 40,000 a year, that's where this would be. And if somebody making 320, that's where that would be somebody has zero kids, it'd be somewhere along this axis. Somebody has five, it'd be somewhere over here. Okay. And now I have these plus signs and these minus signs on here. So what I'm going to represent here is the plus sign means that they own a car. And the minus sign is going

45:42 to represent no car. Okay. So your initial thought should be okay, I think this is binary classification because all of our points all of our samples have labels. So this is a sample with the plus label. And this here is another sample with the minus label. This is an abbreviation for width that I'll use. Alright, so we have this entire data set. And maybe around half the people own a car and maybe around half the people don't own a car. Okay, well, what if I had some new

46:29 point, let me use choose a different color, I'll use this nice green. Well, what if I have a new point over here? So let's say that somebody makes 40,000 a year and has two kids. What do we think that would be? Well, just logically looking at this plot, you might think, okay, it seems like they wouldn't have a car, right? Because that kind of matches the pattern of everybody else around them. So that's a whole concept of this nearest neighbors is you look at, okay, what's around you. And then you're basically like, okay, I'm going to take the label of the majority that's around me. So the first thing that we have to do is we have to define a distance function. And a lot of times

47:17 in, you know, 2d plots like this, our distance function is something known as Euclidean distance. And Euclidean distance is basically just this straight line distance like this. Okay. So this would be the Euclidean distance, it seems like there's this point, there's this point, there's that point, etc. So the length of this line, this green line that I just drew, that is what's known as Euclidean distance. If we want to get technical with that, this exact formula is the distance here,

48:10 let me zoom in. The distance is equal to the square root of one point x minus the other points x squared plus extend that square root, the same thing for y. So y one of one minus y two of the other squared. Okay, so we're basically trying to find the length, the distances, the difference between x and y, and then square each of those sum it up and take the square root. Okay, so I'm going to erase this so it doesn't clutter my drawing. But anyways, now going back to this plot, so here in the nearest neighbor algorithm, we see that there is a K, right? And this K is basically

49:03 telling us, okay, how many neighbors do we use in order to judge what the label is? So usually, we use a K of maybe, you know, three or five, depends on how big our data set is. But here, I would say, maybe a logical number would be three or five. So let's say that we take K to be equal to three. Okay, well, of this data point that I drew over here, let me use green to highlight this. Okay, so of this data point that I drew over here, it looks like the three closest points are definitely this one, this one. And then this one has a length of four. And this one seems like it'd be a little

49:50 bit further than four. So actually, this would be these would be our three points. Well, all those points are blue. So chances are, my prediction for this point is going to be blue, it's going to be probably don't have a car. All right, now what if my point is somewhere? What if my point is somewhere over here, let's say that a couple has four kids, and they make 240,000 a year. All right, well, now my closest points are this one, probably a little bit over that one. And then this one, right? Okay, still all pluses. Well, this one is more than likely to be plus. Right? Now,

50:45 let me get rid of some of these just so that it looks a little bit more clear. All right, let's go through one more. What about a point that might be right here? Okay, let's see. Well, definitely this is the closest, right? This one's also closest. And then it's really close between the two of these. But if we actually do the mathematics, it seems like if we zoom in, this one is right here. And this one is in between these two. So this one here is actually shorter

51:30 than this one. And that means that that top one is the one that we're going to take. Now, what is the majority of the points that are close by? Well, we have one plus here, we have one plus here, and we have one minus here, which means that the pluses are the majority. And that means that this label is probably somebody with a car. Okay. So this is how K nearest neighbors would work. It's that simple. And this can be extrapolated to further dimensions to higher dimensions. You know, if you have here, we have two different features, we have the income, and then we have

52:19 the number of kids. But let's say we have 10 different features, we can expand our distance function so that it includes all 10 of those dimensions, we take the square root of everything, and then we figure out which one is the closest to the point that we desire to classify. Okay. So that's K nearest neighbors. So now we've learned about K nearest neighbors. Let's see how we would be able to do that within our code. So here, I'm going to label the section K nearest neighbors. And we're actually going to use a package from SK learn. So the reason why we, you know, use these packages and so that we don't have to manually code all these things ourselves, because it would

53:04 be really difficult. And chances are the way that we would code it, either would have bugs, or it'd be really slow, or I don't know a whole bunch of issues. So what we're going to do is hand it off to the pros. From here, I can say, okay, from SK learn, which is this package dot neighbors, I'm going to import K neighbors classifier, because we're classifying. Okay, so I run that. And our KNN model is going to be this K neighbors classifier. And we can pass in a parameter of how many neighbors, you know, we want to use. So first, let's see what happens if we just use one. So now if I do K, and then model dot fit, I can pass in my x training set and my

53:52 weight y train data. Okay. So that effectively fits this model. And let's get all the predictions. So why can and I guess yeah, let's do y predictions. And my y predictions are going to be cannon model dot predict. So let's use the test set x test. Okay. Alright, so if I call y predict, you'll see that we have those. But if I get my truth values for that test set, you'll see that this is what we actually do. So just looking at this, we got five out of six of them. Okay, great. So let's actually take a look at something called the classification report that's offered by SK learn.

54:39 So if I go to from SK learn dot metrics, import classification report, what I can actually do is say, hey, print out this classification report for me. And let's check, you know, I'm giving you the y test and the y prediction. We run this and we see we get this whole entire chart. So I'm going to tell you guys a few things on this chart. Alright, this accuracy is 82%, which is actually pretty good. That's just saying, hey, if we just look at, you know, what each of these new points, what it's closest to, then we actually get an 82% accuracy, which means how many do we get right versus how many total are there. Now, precision is saying, okay, you might see that we have it

55:29 for class one, or class zero and class one. What precision is saying was, let's go to this Wikipedia diagram over here, because I actually kind of like this diagram. So here, this is our entire data set. And on the left over here, we have everything that we know is positive. So everything that is actually truly positive, that we've labeled positive in our original data set. And over here, this is everything that's truly negative. Now in the circle, we have things that are positive that were labeled positive by our model. On the left here, we have things that are truly positive, because you know, this side is the positive side and the side is the negative side. So these are truly positive. Whereas all these ones out here, well, they should have been positive, but they

56:18 are labeled as negative. And in here, these are the ones that we've labeled positive, but they're actually negative. And out here, these are truly negative. So precision is saying, okay, out of all the ones we've labeled as positive, how many of them are true positives? And recall is saying, okay, out of all the ones that we know are truly positive, how many do we actually get right? Okay, so going back to this over here, our precision score, so again, precision, out of all the ones that we've labeled as the specific class, how many of them are actually that class, it's 7784%. Now,

57:03 recall how out of all the ones that are actually this class, how many of those that we get, this is 68% and 89%. Alright, so not too shabby, we can clearly see that this recall and precision for like this, the class zero is worse than class one. Right? So that means for hadron, it's worked for hadrons and for our gammas. This f1 score over here is kind of a combination of the precision and recall score. So we're actually going to mostly look at this one because we have an unbalanced test data set. So here we have a measure of 72 and 87 or point seven two and point eight seven, which is not too shabby. All right. Well, what if we, you know, made this three. So we actually see

57:55 that, okay, so what was it originally with one? We see that our f1 score, you know, is now it was point seven two and then point eight seven. And then our accuracy was 82%. So if I change that to three. Alright, so we've kind of increased zero at the cost of one and then our overall accuracy is 81. So let's actually just make this five. Alright, so you know, again, very similar numbers, we have 82% accuracy, which is pretty decent for a model that's relatively simple. Okay, the next type of model that we're going to talk about is something known as naive Bayes. Now,

58:42 in order to understand the concepts behind naive Bayes, we have to be able to understand conditional probability and Bayes rule. So let's say I have some sort of data set that's shown in this table right here. People who have COVID are over here in this red row. And people who do not have COVID are down here in this green row. Now, what about the COVID test? Well, people who have tested positive are over here in this column. And people who have tested negative are over here in this column. Okay. Yeah, so basically, our categories are people who have COVID and test positive, people who don't have COVID, but test positive, so a false false positive, people who have COVID

59:32 and test negative, which is a false negative, and people who don't have COVID and test negative, which good means you don't have COVID. Okay, so let's make this slightly more legible. And here, in the margins, I've written down the sums of whatever it's referring to. So this here is the sum of this entire row. And this here might be the sum of this column over here. Okay. So the first question that I have is, what is the probability of having COVID given that you have a positive test? And in probability, we write that out like this. So the probability of COVID given, so this

1:00:21 line, that vertical line means given that, you know, some condition, so given a positive test, okay, so what is the probability of having COVID given a positive test? So what this is asking is saying, okay, let's go into this condition. So the condition of having a positive test, that is this slice of the data, right? That means if you're in this slice of data, you have a positive test. So given that we have a positive test, given in this condition, in this circumstance, we have a positive test. So what's the probability that we have COVID? Well, if we're just using this data, the number of people that have COVID is 531. So I'm gonna say that there's 531 people that have COVID. And then

1:01:15 now we divide that by the total number of people that have a positive test, which is 551. Okay, so that's the probability and doing a quick division, we get that this is equal to around 96.4%. So according to this data set, which is data that I made up off the top of my head, so it's not actually real COVID data. But according to this data, the probability of having COVID given that you tested positive is 96.4%. Alright, now with that, let's talk about Bayes rule, which is

1:02:02 this section here. Let's ignore this bottom part for now. So Bayes rule is asking, okay, what is the probability of some event A happening, given that B happened. So this, we already know has happened. This is our condition, right? Well, what if we don't have data for that, right? Like, what if we don't know what the probability of A given B is? Well, Bayes rule is saying, okay, well, you can actually go and calculate it, as long as you have a probability of B given A, the probability of A and the probability of B. Okay. And this is just a mathematical formula for that. Alright, so here we have Bayes rule. And let's actually see Bayes rule in action. Let's use it on an example.

1:02:51 So here, let's say that we have some disease statistics, okay. So not COVID different disease. And we know that the probability of obtaining a false positive is 0.05 probability of obtaining a false negative is 0.01. And the probability of the disease is 0.1. Okay, what is the probability of the disease given that we got a positive test? Hmm, how do we even go about solving this? So what what do I mean by false positive? What's a different way to rewrite that? A false positive is when you test positive, but you don't actually have the disease. So this here is a probability that you have a positive test given no disease, right? And similarly for the false negative,

1:03:42 it's a probability that you test negative given that you actually have the disease. So if I put that into a chart, for example, and this might be my positive and negative tests, and this might be my diseases, disease and no disease. Well, the probability that I test positive, but actually have no disease, okay, that's 0.05 over here. And then the false negatives up here for 0.01. So I'm testing negative, but I don't actually have the disease. This so the probability that you test positive, and you don't have the disease, plus a probability that you test negative, given that you don't have the disease, that should sum up to one. Okay, because if you don't have the disease,

1:04:30 then you should have some probability that you're testing positive and some probability that you're testing negative. But that probability, in total should be one. So that means that the probability negative and no disease, this should be the reciprocal, this should be the opposite. So it should be 0.95 because it's one minus whatever this probability is. And then similarly, oops, up here, this should be 0.99 because the probability that we, you know, test negative and have the disease plus the probability that we test positive and have the disease should equal one. So this is our probability chart. And now, this probability of disease

1:05:16 being point 0.1 just means I have 10% probability of actually of having the disease, right? Like, in the general population, the probability that I have the disease is 0.1. Okay, so what is the probability that I have the disease given that I got a positive test? Well, remember that we can write this out in terms of Bayes rule, right? So if I use this rule up here, this is the probability of a positive test given that I have the disease times the probability of the disease divided by the probability of the evidence, which is my positive test. Alright, now let's plug in some numbers for that. The probability of having a positive test given

1:06:05 that I have the disease is 0.99. And then the probability that I have the disease is this value over here 0.1. Okay. And then the probability that I have a positive test at all should be okay, what is the probability that I have a positive test given that I actually have the disease and then having having the disease. And then the other case, where the probability of me having a negative test given or sorry, positive test giving no disease times the probability of not actually having a disease. Okay, so I can expand that probability of having a positive test out into

1:06:52 these two different cases, I have a disease, and then I don't. And then what's the probability of having positive tests in either one of those cases. So that expression would become 0.99 times 0.1 plus 0.05. So that's the probability that I'm testing positive, but don't have the disease. And the times the probability that I don't actually have the disease. So that's one minus 0.1 probability that the population doesn't have the disease is 90%. So 0.9. And let's do that multiplication. And I get an answer of 0.6875 or 68.75%. Okay. All right, so we can actually expand

1:07:48 that we can expand Bayes rule and apply it to classification. And this is what we call naive base. So first, a little terminology. So the posterior is this over here, because it's asking, Hey, what is the probability of some class CK? So by CK, I just mean, you know, the different categories, so C for category or class or whatever. So category one might be cats, category two, dogs, category three, lizards, all the way, we have k categories, k is just some number. Okay. So what is the probability of having of this specific sample x, so this is our feature vector

1:08:36 of this one sample. What is the probability of x fitting into category 123 for whatever, right, so that that's what this is asking, what is the probability that, you know, it's actually from this class, given all this evidence that we see the x's. So the likelihood is this quantity over here, it's saying, Okay, well, given that, you know, assume, assume we are, assume that this class is class CK, okay, assume that this is a category. Well, what is the likelihood of actually seeing x, all these different features from that category. And then this here is the

1:09:21 prior. So like in the entire population of things, what are the probabilities? What is the probability of this class in general? Like if I have, you know, in my entire data set, what is the percentage? What is the chance that this image is a cat? How many cats do I have? Right. And then this down here is called the evidence because what we're trying to do is we're changing our prior, we're creating this new posterior probability built upon the prior by using some sort of evidence, right? And that evidence is a probability of x. So that's some vocab. And this here is a rule for naive Bayes. Whoa, okay, let's digest that a little bit. Okay. So what is

1:10:15 let me use a different color. What is this side of the equation asking? It's asking, what is the probability that we are in some class K, CK, given that, you know, this is my first input, this is my second input, this is, you know, my third, fourth, this is my nth input. So let's say that our classification is, do we play soccer today or not? Okay, and let's say our x's are, okay, is it how much wind is there? How much rain is there? And what day of the week is it? So let's So let's say that it's raining, it's not windy, but it's Wednesday, do we play soccer? Do we not? So let's use Bayes rule on this. So this here

1:11:06 is equal to the probability of x one, x two, all these joint probabilities, given class K times the probability of that class, all over the probability of this evidence. Okay. So what is this fancy symbol over here, this means proportional to so how our equal sign means it's equal to this like little squiggly sign means that this is proportional to okay, and this denominator over here, you might notice that it has no impact on the class like this, that number doesn't depend on the class, right? So this is going to be constant

1:11:53 for all of our different classes. So what I'm going to do is make things simpler. So I'm just going to say that this probability x one, x two, all the way to x n, this is going to be proportional to the numerator, I don't care about the denominator, because it's the same for every single class. So this is proportional to x one, x two, x n given class K times the probability of that class. Okay. All right. So in naive Bayes, the point of it being naive, is that we're actually this joint probability, we're just assuming that all of these different things are all independent. So in my soccer example, you know, the probability that we're playing soccer,

1:12:44 or the probability that, you know, it's windy, and it's rainy, and, and it's Wednesday, all these things are independent, we're assuming that they're independent. So that means that I can actually write this part of the equation here as this. So each term in here, I can just multiply all of them together. So the probability of the first feature, given that it's class K, times the probability of the second feature and given this problem, like class K all the way up all the way up until, you know, the nth feature of given that it's class K. So this expands to

1:13:30 all of this. All right, which means that this here is now proportional to the thing that we just expanded times this. So I'm going to write that out. So the probability of that class. And I'm actually going to use this symbol. So what this means is it's a huge multiplication, it means multiply everything to the right of this. So this probability x, given some class K, but do it for all the i's. So I, what is I, okay, we're going to go from the first the first x i all the way to the nth. So that means for every single i, we're just multiplying

1:14:19 these probabilities together. And that's where this up here comes from. So to wrap this up, oops, this should be a line to wrap this up in plain English. Basically, what this is saying is a probability that you know, we're in some category, given that we have all these different features is proportional to the probability of that class in general, times the probability of each of those features, given that we're in this one class that we're testing. So the probability of it, you know, of us playing soccer today, given that it's rainy, not windy, and and it's Wednesday, is proportional to Okay, well, what is what is the probability that we play soccer

1:15:04 anyways, and then times the probability that it's rainy, given that we're playing soccer, times the probability that it's not windy, given that we're playing soccer. So how many times are we playing soccer when it's windy, how you know, and then how many times are what's the probability that's Wednesday, given that we're playing soccer. Okay. So how do we use this in order to make a classification. So that's where this comes in our y hat, our predicted y is going to be equal to something called the arg max. And then this expression over here, because we want to take the arg max. Well, we want. So okay, if I write out this, again, this means the probability of

1:15:55 being in some class CK given all of our evidence. Well, we're going to take the K that maximizes this expression on the right. That's what arc max means. So if K is in zero, oops, one through K, so this is how many categories are, we're going to go through each K. And we're going to solve this expression over here and find the K that makes that the largest. Okay. And remember that instead of writing this, we have now a formula, thanks to Bayes rule for helping us

1:16:40 approximate that right in something that maybe we can we maybe we have like the evidence for that, we have the answers for that based on our training set. So this principle of going through each of these and finding whatever class whatever category maximizes this expression on the right, this is something known as MAP for short, or maximum a posteriori. Pick the hypothesis. So pick the K that is the most probable so that we minimize the probability of misclassification. Right. So that is MAP. That is naive Bayes. Back to the notebook. So

1:17:31 just like how I imported k nearest neighbor, k neighbors classifier up here for naive Bayes, I can go to SK learn naive Bayes. And I can import Gaussian naive Bayes. Right. And here I'm going to say my naive Bayes model is equal. This is very similar to what we had above. And I'm just going to say with this model, we are going to fit x train and y train. All right, just like above. So this, I might actually, so I'm going to set that. And

1:18:19 exactly, just like above, I'm going to make my prediction. So here, I'm going to instead use my naive Bayes model. And of course, I'm going to run the classification report again. So I'm actually just going to put these in the same cell. But here we have the y the new y prediction and then y test is still our original test data set. So if I run this, you'll see that. Okay, what's going on here, we get worse scores, right? Our precision, for all of them, they look slightly worse. And our, you know, for our precision, our recall, our f1 score, they look slightly worse for all the different categories. And our total accuracy, I mean, it's still 72%, which is not too shabby. But it's still

1:19:11 72%. Okay. Which, you know, is not not that great. Okay, so let's move on to logistic regression. Here, I've drawn a plot, I have y. So this is my label on one axis. And then this is maybe one of my features. So let's just say I only have one feature in this case, text zero, right? Well, we see that, you know, I have a few of one class type down here. And we know it's one class type

…