Python for Data Science - Course for Beginners (3 Hours) | Learn Python, Pandas, NumPy, Matplotlib

0:05 in this video we are going to learn about some libraries of python that are very essential for data science so these are the seven major libraries that you should know while performing data science using python note that this list is not at all exhaustive and in practice you will actually need and use a couple of more libraries when doing data science with python but these seven libraries are the most common and I can say that they are the bare minimum that you need to know so let's understand what each of the library does one by one so the first library is numpy or numpy has very useful features like it lets you create n dimensional arrays in a very efficient manner so this is basically how you represent data in the computer's memory in Python apart from

0:52 that you can use numpy to perform linear algebra and numpy also has capabilities to Generate random numbers and random values the next library is scipy now as the name suggests scipy stands for Scientific Python so it has all the scientific Computing tools that you would need to perform certain tasks in Python so just like calculus signal processing fast Fourier transform Etc now when you talk about data science you definitely are going to talk about data from multiple files and from multiple sources like CSV Json Excel so on and when you are talking about taking these data what do you do you have to read this data which are in multiple formats you also have to perform certain

1:37 manipulations or make changes in the data set itself and you have to perform certain cleaning operations for example there might be some missing value there might be some random values so on so all of these tasks can be done using the pandas library in Python after pandas the next most important library is matplotlip now matplotlib is like a fundamental Library when you want to visualize data in Python so all the charts you know bar charts get the plots so on you can build all of them easily using matplotlib apart from matplotlif we have another Library which is c bond and C1 is a library that is essentially built on top of matplotlib what it basically does is it has already written smart functions that let you create very

2:24 beautiful visualizations very easily with less amount of code the next Library starts models as the name suggests this Library lets you create statistical models like for example regression and you can conduct statistical tests like t-test so on Anova and statistical data exploration last but not the least the next library is scikit-learn scikit learn is a library that is essentially used for performing machine learning modeling in Python apart from machine learning modeling it also has certain functions that let you perform data pre-processing so these are the seven important libraries that you need in order to perform any data science task in Python now that you know which libraries are important the next obvious question is

3:10 how can you install them on your system well the good news is because we are using Anaconda to install python in this course we already have some of these libraries in our system first of all as we know we are using Python 3 version in our course this is because it's the latest version of python and also python 2 is no longer supported next as you can see libraries like numpy sci-fi pandas foreign Etc are some of the libraries that we have already discussed that they are important for data science and these libraries already come with Anaconda installation so we actually don't need to install them explicitly let's try to see this in a live example

3:56 we'll try to install numpy on a system let's go to terminal so this is my terminal and as you know in order to install numpy we need to have a certain command let's see what is that command you remember we were talking about python package exchange a couple of videos ago so now that we know that we need to install numpy I'll just search numpy here so this is the version of numpy that is the latest and this is the command to install numpy so I'll just copy paste this command so I've copied it I'll go back to my terminal and I'll just paste this command here let's see what happens as you can see the output of the command is requirement is already satisfied that means number is already there in my

4:42 system and where is it it is inside the anacondas directory so as you know because Anaconda already has a couple of data science libraries like these installed into the system we don't need to install them explicitly in this video we are going to learn about numpy and this is the first library that we will see now let's quickly switch to the code window so numpy as we have already discussed is a very fundamental package when you want to do scientific Computing in Python it provides a bunch of features some of the useful ones are it provides an n-dimensional array object which is really efficient provides the operational broadcasting it lets you do linear algebra Fourier transform random

5:29 number capabilities and much more so we already know how to install a library in Python using the python package index so I'm not going to cover that in this video but suppose you want to check whether your system has a package installed or not the first thing that you can do is you can literally try to import the package if it works that means the system has the package installed if it does not or if it gives you an error that means you need to install that package so in this case I run import numpy as NP and I'm not getting any error that means this package is installed on my system now this way of importing by using as and giving an alias is actually very useful when you're working with a lot of libraries in Python suppose I have to use numpy again and again all over your program right so it would be difficult

6:16 to write the entire name numpy again and again and there are even packages that have names longer than this so using this as keyword you can give a Alias to any library that you import in Python so wherever you need to write numpy you can just write NP and you can call the functions using NP dot that's straightforward so now that we know that number is installed let's first of all check the version of numpy on our system so we have 1.17.1 installed make sure that you also have the same version so that there is no issues in the API of the library will give the command to install this version right below the video so first of all as we have seen again and again that numpy gives you an N dimensional array object so how can you create an array you can just do M NP

7:03 dot array function pass the list and if you run this you will get an array now you might be wondering that this data structure looks very similar to a list then what is the difference so this is a list let me just create it and I'll just try to create a similar array so I'll just copy this and I'll say in P dot array now do you see any difference between the data types of elements in this list and in this array so in this list we had numbers one four five we had a character alphanumeric character two and we had a Boolean false so finally when we created

7:48 the list all of the data types retain the original data types but in an array what happened was all of the data types was converted to string so basically one of the major differences between a list and an array is an Ariel has this limit that you can only have elements of the same data type if you have elements of different data type it will just try to convert them to the same data type so this is the first major difference between a list and an array this is what we are showing here also it basically uses typecasting and upcast every other data type to string type now there is another major difference so let's see that you have this list sample list let's just print it

8:34 so this is the list what will happen if you multiply the entire list by 2 let's see so it has basically duplicated the list and there are the entire list is once again appended to the original list so one two six one two six if we say do it by three it will do the same thing three times and so on you get the idea now we'll create a array so this is my sample numpy array in fact I'll just remove the print statement so so that you can see the array in its raw form so this is the array now what will happen if I do the same multiply by 2 here so do you see any difference between what happened here and what happened here basically here the entire list or

9:21 array is not duplicated it's actually the number two has been multiplied to every individual element of the array so this was the array one two six and the final array becomes 2 to 12. if you do three it will do the same thing so this is another major difference between a list and an array an array has a concept which is known as broadcasting what broadcasting essentially does is you do any operation on the array that operation will get implemented on each element of that array so instead of into three you can just say plus three so this is the number one two three four five six and each number has been added with three you can divide by three and you get the idea so these are the two major differences between a list and array and in the upcoming videos of the

10:08 course you will actually understand why this concept of broadcasting is very useful for data science the next thing that we are going to talk about is how to create a matrix in numpy so as you know numpa is very good for linear algebra so the fundamental instruments of linear algebra is Matrix so you can create a matrix in numpy first of all I have created this multi-dimensional list a right and if you want to create a matrix of this list you can just say NP dot array and pass the list and you get the Matrix which is nothing other than just a multi-dimensional array now you can access elements of this Matrix just like you would have done for a real Matrix suppose you want to access this number so this number is row 0 to row 0 and column one

10:56 so you get two if you do column two then you will get three and you can do it again and again now instead of explicitly creating a list and passing it to array function you can directly pass the structure in the array function itself in order to create a matrix so both of these Matrix are same we can also create a matrix that is filled up of random integers using np.random dot random function so 0 to 10 is actually the range in which you want the random numbers to be and 3 comma 3 is the shape of The Matrix so if I just increase the range to say 100 you will see that the numbers are two digit now let me do this back so notice that even though I'm running the same code with all the same

11:41 parameters both the times we are getting matrices with different random values and that's what the random number generating capabilities of numpy is right but sometimes in real life you need to be able to predict what kind of random numbers will be there in The Matrix so there is a way to fix this and that is known as seed so suppose I have given the seed 0 now if I call this function I'll get a bunch of numbers and if I call this function again with the same script I'll still get the same bunch of numbers so seed lets you fix the behavior of a random generator now because C is 0 anytime anytime you run this cell you will see that you will get the same Matrix and the minute you remove the seed or change it to some other value it becomes random so CD is a way of making sure that no matter how

12:27 many times you call the random function will get the same numbers and if you don't give the seed then you will have the original paver which is every time you call the function you will have a matrix of random numbers next thing in numpy is you can actually create an entire Matrix that is filled with nothing but zeros so this is the shape that you give 3 comma four zeros is the function and you can give the type to have this Matrix another Matrix four by five similar to 0 you can also create a matrix filled with ones so you can use the ones function give the shape of the Matrix and you get the matrices you also can create an identity Matrix so because identity Matrix is a square Matrix you have to give only one number and I have given five so it will create a five by five identity Matrix you can change this

13:14 value to 3 and you will get a 3 by 3 identity Matrix suppose you want to create a matrix that is filled with a specific number so this is the shape you want to fill it with 3.14 and you can use the NP dot full function and you get this Matrix similarly this time we have filled the Matrix with 7. the next concept that comes to mind is when you want to concatenate or combine two arrays or two matrices so this is the Matrix one and this is the Matrix two and first what we want to do is we want to combine them row wise so after four five six I want 789 here then 10 11 12 here so you can do that using NP dot concatenate you can pass both the matrices as a list and you have to give access equal to 0 so when we

14:00 will give access is equal to 0 you are telling number to do a row wise operation see these two matrices it combined in a row wise fashion suppose you want to do the same thing column wise then the same code you can use same concatenate function same way of passing the matrices but access should be one so you know you give access equal to 1 you tell numpy to do a column wise operation so let's see what happens so this is a first one two three four five six now after that we have added values in column fashion so 7 8 9 10 11 12. so this was basics of numpy a very useful library in Python and we will explore it as and when needed in the course let me show you the documentation of numpy to come to this page there are lot of useful things like tutorial Basics so on

14:46 you can click on numpy reference so these are the functions and possibilities that you can do with that you can do with numpy feel free to explore them and see what are the things that you can directly use in your work so there are things like Discrete Fourier transform and so on in this video we are going to learn about another useful library of python for data science that is PSI pi so let's go to code so as the name suggests PSI Pi stands for Scientific Python and it is a library that works very closely with numpy to give you scientific capabilities in Python so just like numpy first of all if you want to check if a library is present in python or not you try to import it two

15:34 in our case the import works that means IPI is there now let's check the version of scipy so it says that the version of Cyprus 1.3.1 but actually in this course we are looking for a version of scipy as version of sci-fi so we want 1.4.1 in this course to fix this issue what you can do is you can simply create a new cell so remember I told you initially that you can use exclamation to run commands on the terminal or command prompt and we also know and we also know that you can use command like say pip install

16:21 to install scipy right so this also a trick to this command which is if you want to install a specific version of scipy you can just say pip install scipy and you can do the syntax equal to equal to 1.4 Dot one let's run this command so now it will take some time because it's downloading this version of sci-fi so now it is uninstalling the old version and it has successfully installed the new version of sci-fi 1.4.1 so let's check the version again one minute because we have installed it recently we need to restart the kernel so let's restart the kernel

17:08 now let's import sipei again this time we don't need to execute this cell because it's already installed so I'll just comment it out and I'll check the version of sci-fi now as you can see we have the exact correct version of scipy so this trick is very useful that you can just use exclamation then use pip install then the library name equal to equal to the version you want and if you want to check what versions there are then you can always go to the documentation page of library and see what are the versions available gives you a couple of useful features some of the common and most useful features are for example you can compute the derivative of a function so you can do differentiation

17:53 so this is the function and the equation we have is x square plus X Plus 1 and you can use the derivative function from the sci Pi dot misc package and in this function you pass the function and you pass the value of x at which you want to differentiate this so at X naught equal to 2 you know by High School mathematics x square plus X Plus 1 actually becomes 2x plus 1 once you differentiate it so when you're passing the value of 2 so as you can see I got this error when I try to run this line of code it says name derivative is not defined can you guess why this is the case this

18:39 is because I did not run this cell before running this cell so if I have not imported derivative at all how can I call the derivative function so let's run this cell before and when I run this again I get the output so this is a silly mistake that you should also be careful of when you are actually programming in jupyter Notebook But first you need to import the right libraries only then you can use them and execute all the cells in the right order so we get the value of 5 and we know that this is the equation after derivative this is the F Dash X you put value 2 so it becomes 2 into 2 plus 1 5 so this works similarly you can calculate the second derivative and you can do a lot more so this was derivation using PSI Pi apart from that you can also calculate

19:25 permutations and combinations so 4c2 is 6 and 4 p 2 is 12. using PSI pi here we have used the com function and here we use the pump function then you can also do good amount of linear algebra from sci-fi also so you have to import the linear algebra module from PSI Pi which is LIN ALG we have also inputted the numpy so using numpy we created this Matrix and suppose you want to calculate the determinant of this Matrix so using Lin ALG module of PSI Pi you can use the debt function to calculate the determinant of this Matrix this is the

20:10 determinant similarly you can calculate the inverse of this Matrix and you get the idea so this was PSI Pi in a nutshell let's go to the documentation of psiping select the scipy reference guide so these many tasks you can perform using scipy things like signal processing integration optimization special functions interpolation Fourier transform and a lot of other stuff and if you click on one of these things you can actually go inside and see what are the functions that you have available for each of this tasks and they've also given pretty good tutorials so this was PSI pi so pandas is an open source library in

20:56 Python that we use a lot for reading data performing data manipulation and related tasks so it supports out of the box reading files from different formats like CSV Json Excel HTML and so on and these are actually the formats that are very common in the world of data science we can summarize the data we can filter and modify the data we can merge multiple files and so on so let's have a quick intro of this Library first of all you import it just like we have done in the case of numpy you use a Alias PD so wherever you say PD in the code python understand that PD means the pandas Library so we run this that means we are we already have pandas on a system because we do not get any error in case you get

21:42 any error and you want to install Panda you can simply use one of the two commands given here so if you have an account on a system with Python 3 just the way that we had told you to install it in the installation section of this course then you can just do pip install pandas but if you have Python 3 and python 2 multiple versions of python your system locally without Anaconda then you can do pip3 install pandas pip3 means Python 3 is PIP so I'll just use this command paper install pandas so we will learn this let's see what output we get so as expected it totally tells you that the requirement is already satisfied which is basically you already have pandas in your system

22:27 let's check the version of pandas so for this course we are using this version of pandas 0.25.1 so this is pandas now let's read a CSV file so we already have a CSV in our directory so this is how the CSV looks like so these are the column names they are separated by a comma then you have value separated by a comma right so let's read this CSV in Python remember that these were the column names and item identifier item where it's on so using PD dot create CSV function you can just give the path of the CSV file and you will be able to access it so now your CSV file has been stored in the variable data first of all let's see what are the first three rows of this data set so dot head function basically lets you

23:14 see the initial rows of the data set 3 means basically first three rows you can convert it to say 20 and you will get the first 20 rows so this is our data set these are the columns multiple columns and these are the values all the rows the next step is you want to understand how many rows and columns are present in the data so do that you can just use the dot shape property so data dot shift so we have 8523 rows and 12 columns now let's print the name of columns so data dot column gives you the name of columns so these are the columns we have identifier item weight item fat contained visibility outlet location type and so on the next interesting thing is you can also find the data type

24:01 of The Columns of a data set so as you can see these are real values 9.3 5.29 these are you know string values so let's see what are the data types of our columns so data.d types is the property that lets you view the data types of the columns so object essentially means where there is a string float 64 is again float or real data type so these column item weight and item visibility are decimals and item identifies a string so let's see atom interface the string atom weight and item visibility are decimals make sense right then you have Outlet establishment here which is int so let's see what is this so if you see the outlet establishment you will see there is no decimal because it's the year so this was how you can access the data types of columns in pandas the next

24:48 thing is what if you want to know what are the total number of missing values so you can use data then use the SNF function and then use the sum function this line basically says first of all count the total number of values that are missing in each column in data and sum them up so that you get a total number let's see so the item weight column has 1463 missing values and Outlet size as 2410 missing values apart from this no column has missing values so this is very useful when you want to actually understand that okay my data might be having missing values so this is a very useful step during pre-processing next thing is you can also calculate or find the summary of the data so summary is essentially column wise you know count mean standard deviation minimum maximum value this

25:34 just gives you a quick overview of the data itself notice that it only includes the columns that are numerical it doesn't include columns that have string type like item identifier because you cannot count the summary for them so this was basics of pandas pandas has a pretty good documentation and you have things like getting started user guide AP reference developer guide and so on you can choose whatever you need by default when you work with pandas you usually go to the reference guide so these are the for example file different file types that pandas accepts like SAS HTML Excel so on and you want to do some manipulation functions a lot of things are available here and you can select whichever function you need for example for plotting of a general utility functions you can select

26:20 that function and you will come to know details about how to call that function what values it returns some examples and this is how you can work with pandas in this video we are going to learn about two very important libraries of python that is matplotlif and C1 now these two libraries are used a lot when you're talking about data visualization so without waiting much let's move to code so first of all we want to import matlord lib to see if it's present in our system or not and as you know it comes with Anaconda so this line of code runs without any problem the version of matplotlib that we are going to use in this course is 3.1.1 in case your system does not have the same version you know how to install the

27:06 right version now we are going to use the pipelot module of matplotlab in order to make most of our visualizations so you can import this Pi plot module and give it the Alias PLT apart from that we also write this line which is percentage Metro lip inline this is basically a Jupiter shortcut which helps matplotlip to make visualizations in the notebook itself then we also have the same libraries numpy and pandas that we have seen before so let's import all of them so using map.lib you can create basic plots like line plot for example then you can also create other basic plots like for example bar chart so this is a stacked bar chart the orange portion represents the class 1 and the bluish shade represents the class 0. now

27:53 this was matplotlib apart from matplotlib we have another library that is C1 C1 is basically a simpler version of matplotlib if said in a very simplistic manner C1 is actually built on top of macro lip so a lot of things that you can do in my problem with say four or five lines of code you can actually do them with one or two lines of code in C1 apart from that a lot of good code on how to make beautiful visualizations are already in built in c bond let's see we have created some random data and so this is the density plot of the data this is how the distribution smoothly looks like let's have a histogram so this is a histogram now we can combine both of these and we can have both histogram and density plot in order to

28:39 give you a better idea of the data so as you can see matplotlip and C born are pretty similar but C1 has different types of plots that you can create by just using very less lines of code so here is another interesting default that c bond provides you let's see so this is known as pair plot and we have this data set where we have so many columns and then this is the target which is which flower type of which species thus these data points belong to if you use pair plot of C1 what it does is it plots the relationship between all of the variables with every other variable in the data set itself so multi-dimensional relationship like sepal with with simple length then with suppel width then with petal length then

29:25 petal width so on and as you can see all the data points or all the distributions are color coded based on their class so satosha versus so there's differently color coded so this was essentially matplotlip and C1 very useful libraries when you are working with data science in Python let's check out the documentations so I'll first go to matplotlib foreign there are lots of tutorials how to install and how to create interactive plots and so on you also have API overview where you can see what are the usage patterns you have books links and

30:10 you can just go through these examples and documentation and just see what are the things that are available in matplotlib and explore them more right you have so many examples of visualizations that you can actually create using matplotlib and you have gallery coming to C1 so this is a documentation of C1 C1 actually lets you access different types of visualizations that it provides based on the kind of relationship you want to plot so they are relational categorical distribution regression so on so you can click on the API and see what are the functions available you can also click on tutorials so these are the functions that are available and you can click on any one of these

30:56 functions like cat plot which is categorical plotting you can scroll down and you will see that they have given good amount of examples and you will be able to see really beautiful visualizations that are quite complex also that C1 is able to create by just using say three or four lines of code so this was my problem C1 in this video we are going to talk about the scikit-learning library of python now this library is used a lot for machine learning modeling let's go to code scikit-learn helps us do a couple of tasks the first task is pre-processing the data so pre-processing involves imputing the missing values encoding the categorical variables normalizing the data and so on then you can build different kinds of model like linear regression

31:41 classification models like decision tree and so on and finally you can also automate this entire process of taking the data pre-processing IT training different models and making a prediction on them let's see one by one there are six main steps to create a machine learning model in scikit-learn you can import the model then you need to prepare the data set then you need to separate the independent Target variables create an object of the model fit the model with the data which basically means train the model and use the model to predict the target so I have imported scikit-learn and it's already present when you install Anaconda so we are not getting any error in case you get any error with the version you can just run this piece of code in this course we need the version 22.1 so the first thing that we do is we just import a data set as you can see

32:26 there are certain values missing in these two columns item weight and Outlet size so this is where the simple imputer comes into picture second learn gives you a functionality using which you can impute these values so you can just give the column name and you can decide which strategy you want for example if it's a numerical column then you want to replace the missing values by the mean of the column if it's a categorical column then you want to replace by the most frequent category in the column so we just run this piece of code and finally after reading the above piece of code you realize that all the missing values of the data set has been dealt with next you can create regression and classification models in scikit-learn it

33:13 provides a lot of classification regression models for example linear regression decision reclassifier and you can use the pipeline feature of scikit-learn to create a pipeline that essentially helps you automate everything from the minute the data comes in pre-processing the data building the machine learning models and making prediction on them let's see the documentation of scikit-learn so this is a documentation of circuit learn there is classification you have algorithms like SPM nearest neighbors random first etc for regression you have multiple algorithms you have clustering model selection pre-processing feature extraction dimensional reduction so you have a lot of things and you can just click on one of them and see what classification algorithms you have so there are multiple such algorithms and

33:59 this Library performs really well so this was scikit-learn in a nutshell apart from the circuline also comes with a couple of data sets of its own that we'll see in the next video but machine learning modeling is something that will cover in Greater depth if you have part of the applied machine learning course in this video we are going to learn about stats model in Python let's move to code so stats model is a package in Python using which you can perform statistical modeling like linear regression and so on apart from that you can perform statistical testing you can also work with time series data and time series techniques and so on so this is a library that we'll be using further in the course but for now let's just have a quick intro of it

34:45 so we import it just to see if it's present or not it is and the version that we're using in this course is 0.10.1 we import a couple of other libraries and we load our data set using scikit lens data sets let's see how our data set looks like so we have a bunch of columns and our Target column is medv so this is the target column we just print the target column also this is the target column these are the predictive variables now we denote them as X and Y and we try to build a linear regression model using the OLS method

35:31 so worry not if you do not know what OLS is this is actually a statistical linear regression technique but for now you can take the predictions of the model and after you have to end the model and taken all the predictions we can just print the summary so what was the performance of the model what was the kind of data it had and so on so this was stats model in a nutshell as you can see you can do a lot of more things like regression linear models time series analysis a lot of statistical tools are easily available so let's go to the documentation so this is a documentation of stats models and you have a user guide here you have a API reference so you can see time series analysis other models statistical and tools data sets what kind of data sets

36:17 it has optimization so you have a lot of things that you can do using stats models now so far we have seen the basic introduction of all the python libraries that are commonly used for performing data science tasks we also know that there are five major tasks of data science that are done using Python and in this module we will learn how to do the first task that is reading data files so here are the types of data files that we will be working with CSV or comma separated values is one of the most common file types that you usually find when you're working in real world data science situations the next file type is text these are essentially any file that

37:03 have dot txt extension to it apart from these two we also have the MS Excel or spreadsheet format that is widely used in the industry now Json is a file type that is heavily involved when you are dealing with data from say websites and apis and so on if you notice you will see that out of all these different types of data files we have already dealt with text files in previous module so apart from text files in this module we will learn how can you work with CSV files Excel files and Json files in Python in this video we will work on CSV files we'll also understand the challenges

37:49 that occur while leading CSV files in Python and also how to tackle these challenges using python so once again you can install pandas using these two commands as you already know based on your system configuration so as you already know we'll be using the pandas library for this task now the first step that we usually do in order to check whether libraries present in python or not in your local system we try to import it so here we are saying import pandas as PD so this gave us no error and then we print the version of pandas in case pandas was not present in the system this would have given us an error and we could have used one of these commands to install pandas now the first thing that we are going to do is we'll read a simple CSV file let's see

38:35 how our CSV looks like so I'll go to the data sets folder I'll take the bigmart sales CSV so this is the CSK that we have these are the column names item identify item weight item fat contained these are the values of them everything is separated by a comma so this is actually a data set that we have taken as a sample data set and it is the sales data set from a store known as bigmart let's get back to our notebook so at this point we already know that if you want to import a CSV in pandas you just use PD dot create CSV so let's do that so the type of data that we have in this variable data is actually data frame type pandas.core dot frame Dot dataframe so you can imagine a data frame is a data structure in Python so

39:23 you can consider that a data frame is actually a type of data structure that pandas provides to you in Python now data frame has multiple features like it is two dimensional and you can have Columns of different data types it is very similar to spreadsheet or a table that you have in SQL and it is one of the most commonly used data structure in pandas let's see how our data looks like so this is the data that we read from the CSV and this structure is known as the data frame if you go back to the CSE you will realize the column names are same and the values are similar like you have FDA 15 drc01 so even here you have FDA 15 and drc01 so this is the same CSV data in the form of a data frame now when we read this data set in python

40:10 as you already know the first thing that we are curious about is to know how many rows and columns are present in our data so you can do that using data.shape so our data set has 8523 rows and we have 12 columns so this was a basic way of reading a CSV file in Python using pandas data frame and seeing how many rows and columns are there and what does the data look like this is something that we have already covered now let's go to some of the challenges that we face when we are reading csvs in practice so the first challenge that we'll have is in our CSV we'll actually have the first few rows which are like comment in Python and we want to avoid reading these rows because these are essentially comment and these do not form the part of the data set let me show you what I'm

40:57 talking about so I'll open this file big Mart sales toprower.csv so notice we have our data set starting from this line above this we have a bunch of comments written by the person who has either created this data set or curated this data set and this basically tells some general info about the data set so this is something that you find a lot when you're working with real world data or open source data people usually add comments describing the data in the file in the initial few lines of the file now if you try to read this file by the same way that we had read previously you'll actually get an error so we got a parser error and it's like it's not able to tokenize and so on so in order to prevent this error you

41:44 have to basically tell pandas that avoid reading these first couple of lines because this is not where the data set starts start reading from say line number six so let's see how can you do that so in order to do that there is a parameter in pandas read CSV function which is Skip Pros so you can just say how many rows you want to skip of the data set so we know that our actual data set starts at line number six or row number six so we'll just tell pandas to ignore the first five rows so that's what I have given F said skip rows five now if you run the same code you will see that the data set has been loaded without any error so this was challenge number one now the second challenge that we have when we are dealing with CSV is a lot of times when you're working with say open

42:30 source data or you have some data from your data engineering team you will notice that the data set is actually divided into multiple folders and directories in your system so you do not have one single CSV but you might have different csvs spread across different folders let's see what I mean so I go back to the data sets folder I have this folder known as multi directory if you open this folder you will see there are multiple subfolders each with an year as its name 1985 1987 1987 1998 and if when you open each subdirectory you will see there is a CSV this CSV is basically sales record for that year so what has happened is you have received the data in this format where you have sales record of the store

43:17 divided by year in different directories now how can you read this in Python let's see so the first thing that we're going to talk about is the globe Library so glob is essentially a library that you use when you want to read multiple files together in Python it kind of helps you to automate the process and make it easier so let's import glove then you write this code from directory in glob dot glove and you give the path of the directory which is the multi directory and then you give star symbol or stress symbol this basically says any folder inside the multi directory folder now let's see how many subfolders does glob detect in this multi directory folder so according to glob these are the subfolders which are correct as we

44:04 have already seen and now we also want glob to see how many files or csvs we have in each of these folders so initially we had run a single Loop for all the subfolders you can run the loop inside this Loop and then use the same method give the directory path add slash star so this is basically so initially we are going inside and taking all the folders and looping over them now within each folder that we have here in the directory as you know this is the directory value at each iteration within each folder we also again look for a slash star which basically says any files that are present inside these folders so let's see what are the files

44:49 so as you know we have one CSV each in each folder so now using law we have figured out how a data set looks like in terms of folder structure what's the next step so the next step is we'll read all of these files and we'll have to combine all of the CSV into one single CSV because at the end of the day all of these csvs talk about the same store the sales date of the same store but for different years so that's what we have done we have used the cm2 loops to iterate over all the subfolders of this multi directory and inside each subfolder go through all the files that are present then what we do is we read this CSV file which is basically if you first read the 1985 CSV file the 1997 and so on and we keep on appending these

45:36 csvs to this list which is data frame list so all of these CSV files are first of all LED as data frames in Python using pd.8 CSV and we have just made a single list of all of these data frames so let's run this once you run this you can actually try to access this list and see what it has so you will see there are multiple data frames in this list so when you take a length of this entire list you will get 9 and how many folders and files we add one two three four five six seven eight nine so we had nine files inside nine different folders and that's what this list has it has nine different data frames now now that you have all of these data framed as a list the next thing that you do is you want

46:22 to combine all of them to do combine these data frames we use the concat function of pandas it basically concatenates all of the data frames one below the another so let's see what happens when you concatenate so I'll run this piece of code and I check the head so as you can see all of the data frames have been concatenated and you can also see the shape of this data frame now so it has the same shape 8523 so this was Challenge number two when you're working in real world data setting and you have multiple files Each

47:07 of which are supposed to be combined to get your entire data set this is how you can achieve this using glove and the concat functions of pandas coming to challenge number three now before we go to challenge number three I also want to give you an example of how to save cscs in Python so you created this one big CSV of one big data frame and it's made up of combinations of all those small csvs and suppose you want to save this data frame as a CSV for further usage you can just use data frames name and then use dot 2 CSV function in pandas when you use this function you can give the path and if we give index is equal to false because we do not want this index to be stored inside the CSV

47:54 so when you run this line of code you will notice that this CSV is actually saved in your directory let's see so we have merged big Mart sales let's see I'll go back to the data sets I'll scroll down and this is where we have the merge big Mart sales CSV so this is the same CSV that we just saved here so this is a very useful concept of pandas which is basically any data manipulation or any changes that you perform on your csvs you can actually save them for future use so the next challenge that we have in front of ourselves is how can you read csvs that are separated by a specific delimiter so so far as you have already seen all the csvs the values were

48:41 separated by a comma right so this was the comma and you could see that this pattern was same but when you're dealing with real world data you also see that sometimes a CSV has apart from comma some other characters as delimiters for example there are files known as tab separated values or tsv in that case it is just a CSC but instead of having a comma you have tabs separating the values let me show you an example so I'll again go back to the data sets folder and I'll open this file big Mart sales delimiter so notice that you have tabs separating values now this is how Jupiter shows a tab when you are writing a text file this is essentially the same Tab Key that you have in your keyboard when you press that Tab Key to separate

49:26 data values in your file it's a CSV that is delimited or all the data values separated by a tab so let's see how can you solve this in Python so this is the same file with delimiter as tab if you try to read it using the same normal approach that we had seen before you will see that it just messes up the entire data like the columns are coming randomly slash T is coming and the values are getting like this so you can pass a parameter to the PD dot read CSU function in pandas saying delimiter is equal to backslash t so backslash T essentially means it's a shortcut or a shorthand for a tab space so when you do this and then print the same data set you will notice that in

50:12 this case the data set is actually coming very properly so compare this with this so apart from comma and slash T they can be other delimiters based on the kind of format the data you are using you can just pass that as a delimiter here and pandas will take care of the rest this was Challenge number three let's go to challenge number four so Challenge number four is actually a challenge that I have seen a lot in my day-to-day life which is a lot of times you're working only on your local system you're not using any big servers or large servers that have good amount of ram like 32 GB 64 GB in those situations you will realize that if you are working with data sets that are very large for example there's

50:58 a CSV file that has a hundred million rows in that case when you're trying to read that CSV it either takes a long time for pandas to read it or sometimes your program might even crash in those cases we what we try to do is we see the CSV in parts so what we tell is okay instead of seeing the entire 100 million rows I just want to see the first 100 rows then next thousand rows next thousand rows and so on and this is what the inrose parameter of the read CSA function helps us do so this is the same big Mart sales CSV that we have seen earlier we gives n rows equal to 100 as a parameter we read the file now you will see that the number of rows have been fixed to 100 it is no longer the same entire data set where we had like

51:44 8500 plus rows this is actually a very useful trick that you can apply when you are working with files that are too large to fit into your RAM if you just see the head of this data set it's the same data set nothing has changed now just like the fact that you can read first in rows in a data set you can also read specific columns when opening a file in pandas so this is very straightforward when you're opening a file using the read CSV function just use the parameter use calls and give all the column names that you want to read when you execute this piece of code you will see that you're only getting those columns which you have clearly mentioned so these were some ways in which you can deal with the

52:30 challenges that you face when you're working with CSV files in pandas in this video we will see how can you read Excel files in Python and what are some of the challenges that you face when you're working with Excel data in a real life setting and how can you overcome them so again we will be using pandas to read the Excel files you know how to install pandas by now the first thing that we do is we import the pandas Library as PD now let me show you the Excel that I'm talking about so this is the same Excel so this is the same data set that we saw in CSV form earlier about bigmart and this time we have it in Excel form so these are the columns item MRP

53:16 outlets on and you have these rows with values let's go back to code so if you want to read a Excel file in Python you can just use read Excel function of pandas and give the path to the file so when we execute this piece of code you'll see that we have got the same Excel file dot Excel SX format as a data frame now what are the challenges when you're working with Excel files so the first challenge is dealing with files that have multiple sheets now I know all of you who have worked with Excel or spreadsheets before you know that this is something that happens a lot a lot of times you have multiple sheets within the same spreadsheet or Excel file let's see so I'll first show you the sheet

54:02 that I'm talking about so this is the spreadsheet that I'm talking about you can see there are multiple sheets like 1985 97 and 1997. so basically these are sales data of the same store but each sheet is for each year and this is a very common way of organization data let's see how to load this in Python so if we use the same code of just using read Excel function and try to print the total data that we have in the spreadsheet you will realize that we only have data of this first spreadsheet or the 1985 year which is this it is the first spreadsheet so when you're reading Excel data in Python and suppose you may have a situation where there are multiple sheets within the same

54:47 spreadsheet you actually have to tell explicitly to python which sheet you want to access otherwise it will always access the first sheet let's see how to do that so suppose you want sheet 1985 you can just say sheet name equal to 1985 and just give the name here so this is a 1985 sheet you can see that the outlet establishment here is 1985. similarly you can do for 1987 you have the year right and then you can do 1997. so these are all the three sheets that you have in your spreadsheet and the next thing is as you would have already guessed we want to combine all of these data frames from these different sheets into a one big CSV or one big data frame to do that we again use the concat

55:33 function we first create a list of all these sheets or data frames and then we use the pandas concat function to create a single CSV this single CSV has all the three years so this was challenge number one now Challenge number two is just like a CSV even in a XL or spreadsheet you can have first few cells which just comments about the file and its description let me show you so this is a file and you can see the first few cells just have random comments usually they can be comments talking about what this file contains whether this file is talking about sales data so on now just like we did in CSV if you do not read this issue we'll have some random values and data that looks slightly Disturbed so we can

56:18 use the same skip rows equal to parameter in the read Excel function and this will help us to read the entire file correctly now here is an assignment for you remember that we were using glob to read CSV files from multiple directories so you can use the same glob code and instead of reading multiple CSV files from different directories you can actually read multiple Excel files from different directories and this is an assignment for you so there's this directory in your data sets folder known as multidirectory Excel go to that directory and try to use the same glob code that we saw in CSV and try to see if you can solve this I'm pretty sure with the knowledge that you already have you're going to do this very easily

57:06 in this video we are going to see how can you work with Json files in Python first of all we'll be seeing the pandas library of python so let's import that now this is how our Json looks like it's a pretty common straightforward Json and if you want to read this Json in pandas you can just use the read Json function and give the path and the name of the file so let's do this and you get the Json in the same data frame format that we have seen before for csvs and excels now this is a good example of a Json but sometimes when you're working with real world data you'll also see a new format in which Json is arranged and that format is this so the difference between these two files is this entire file represents a single Json while in this

57:53 file each line is actually a separate Json so when you are trying to import this file in pandas you need to be aware of that suppose we try to read this file in pandas like the same way we have done before it will give you an error the way to work with this kind of file is you use a parameter known as lines equal to true of the read Json function and then you won't get any error when you're trying to work with this kind of file apart from this another challenge that you see a lot when you're working with industrial data is sometimes you have nested jsons like this so inside one Json you actually have another Json so if you try to read a nested Json like this in pandas you will see that it

58:38 gives an error and this is where the Json module of the standard library of python comes into picture now this module is very focused on Json and provides you a lot of useful features using which you can work with Json files so let me import the Json module and if you want to read the Json file which is nested.json the basic syntax is same as you would have read a normal text file we use with open we give the file name we say as F inside though instead of using F dot read which is the normal function we use when we were working with text files here you will say Json dot load using the load function of the Json module you can see that you have successfully read the nested Json file

59:23 in Python let's pretty print this so pretty print is another module that comes with the python library now this module is very useful when you want to print the data in a structured format you can read more about pretty print and its documentation here but for now using pretty print we are able to see the structure of a Json so you have these different records and each in each record inside the details field you also have another Json which is consisting of ah gradient name so let's solve a small challenge or a task to understand how to work with nested jsons better first of all what we want to do is we have this nested Json where each record has a details section and within that detail section there are records of name of the

1:00:10 people their age and grade so what we essentially want to do is we want to go through all of these records in this Json and only select those records in which the age of the person is greater than 15 so this is one such record this is not that record the age is 11 which is less this is another such record but the age is 22 that is greater than 15. so let's see how can you write code for that first of all we know that we have read the entire Json file in my Json data variable and we also know that it's actually a list of multiple jsons or records so first of all let's try to access the first element of the list using the zero index so this is how a single element of the list looks like now the first thing that we need to do

1:00:55 when we are trying to access the age is to access the details part so let's say data 0 of details so this is another Json and if you want to access the age we can just say details each and we get the value now we just have to do this same thing in a loop so that we get all the ages of all the records in our data set so let's run this in a loop and these are the ages now we'll use the same logic and just shortlist the records in which the age is greater than 15. so this is a list that I have created filter data in this list I am storing each record in the form of a dictionary that has age greater than 15. let's run this code and these are all

1:01:42 the three records now now that you have this record you just need to write them to a Json file in order to write them to a Json file we use the same method that we use for writing text files we have given the right mode as w we have given the file name but what we do different here is instead of using the right function we say Json dot dump now we give the name of the variable in which our entire data set is present we give the name of the file let's read as F we give the indentation space when we execute this piece of code you will notice that the filter Json is promptly made in your disk very straightforward right all of these records have age greater than 15. so this is how you work with Json files in Python now you might have noticed that

1:02:29 when we read Json files inside python they look very similar to the default dictionaries that we have in Python and nested jsons look very similar to nested dictionaries in Python and that is actually correct when we are working with Json data inside python using the Json module we essentially get the data in the form of a dictionary and you can perform most of the functions on it which additionally has things like suppose I want to access filter data so this is my this is my list so the first element is a dictionary now I can just say dot keys so I'll get the keys I can run the DOT

1:03:16 items function I'll get the items I can even do the dot values function I'll get the values so what happens under the hood is this module of python which is Json essentially when reading a Json file inside python converts it into a form of a dictionary so that it's easier for you to work with so this is not something to worry about or get confused about this is straightforward and it gives you more power because as a programmer you already know how to work with dictionaries so this was dealing with Json data in Python now so far in the course we have seen how to read different types of data files in Python we also know that there are five major

1:04:03 tasks of data science that can be done using Python and in this module we will specifically learn how to do this task which is subsetting and modifying data so here are some topics that we will learn about first of all we have subsetting and under subsetting we have three major types of subsetting the data the first one is position based subsetting that is using row or column number to subset the data we have label based subsetting as next wherein we subset data based on the labels that are given to the row and we have value based subsetting wherein we subset data based on values that are present in rows or columns

1:04:49 to implement all of these techniques we will learn to do a couple of things first of all how to access stop or bottom rows of the data frame then how to set rows or column filters and much more after that we will learn about modifying the data in which we will cover things like how to fill missing values in your data frame manipulating values at either the row level or column level based on certain conditions and so on now under subsetting itself there are two types of indexing schemes in pandas that are used to subset data the first one is position based and the second one is label based first let's learn about the position based index as the name suggests it consists of basically an

1:05:34 integer denoting the row or column position the next is the label based index here we will have labels like our fcd01 for our rows and columns let's now have a couple of examples to understand both of these techniques first of all when we say position based index we essentially say suppose I have this given data frame now each row has a position number attached to it like the first two has index 0 the second row is index 1 and so on and this is essentially the same row number that you will find when you have this data set in a Excel file or a CSV file when we talk about label base index we are essentially talking about labels being given to each row of the data now this is a very high level difference we will

1:06:20 go into each of these techniques in more details in the upcoming videos so that you have a better understanding of them but for now here is another piece of information just like we had based index for the rows we also have position based index for the columns so this column item identifier as index 0 because it's the First Column then item weight has the decision based index 1 and item fat content has the position based index 2. so this was all about what we are going to study in this module in this video we will have an overview of different subsetting techniques that are used in pandas and by utilizing code examples we will cover all of these topics what is an index how to subset first and rows based on their position

1:07:06 index can we change the index and so on so by the end of this video and the next videos will have a very good understanding of how to work with subsetting in pandas so let's start with the first step that is simply importing the pandas Library and needing the data using the read CSV function so the data set that we are going to use for this video is the same bigmart sales data set that we have seen earlier now we already know that there

…