Python Advance Programming 2022 | Advanced Python | Advanced Python Tutorial | Simplilearn

0:12 hello everyone and welcome to this advanced python fill course by simply learn in this field course we will be covering some advanced python concepts that are important for developers in this video we will be discussing about iterators modules packages generators list comprehensions followed by that we will work on regular expressions serializations partial functions closures decorators map filter and reduce functions and towards the end we will be working on a small project based on our learnings we have anish with us who's been working with the industry for a very long time he will be helping us to learn the in-depth concepts in python and he will also explain us how to code efficiently over to you anish hello

0:58 everyone and welcome to another video on python programming in this video we will be covering the topic of iterators now an iterator in python is an object that contains a countable number of values or it's a larger object that contains a countable number of sub-objects an iterator is an object that can be iterated upon and what that means is we can traverse from each value and we can manipulate or perform certain functions or operations and tasks using each of the sub-objects or values within this larger collection called iterator now um when we're looking at it technically an iterator is an object which implements the iterator protocol in python which consists of the methods eter and the method next so to begin with let's look at a simple iterator we're going to see

1:43 the example of a list where we're going to traverse through the objects of a list using the for loop so we have objects one two three four and i'm going to print each of these values one by one as you can see here similarly we can also do the same thing for another iterable known as the string where we can go character by character again using the for loop as you can see here another example of an iterable in python is the dictionary so let's say i have a dictionary where i have two values two key value pairs uh key x which contains the value one and key y which corresponds to the value two now when i'm going to do a for loop through this by default i will be traversing through the keys x and y as you can see over here and of course the other types of weight troubles in

2:28 python are like sets and tuples as well all right now let's look at what this iterating protocol is in python so first of all we take an iterable it's an object that contains uh it's a collection of objects that contain sub objects and when we use the eater function it creates an iterator out of it so we can one by one we can traverse through um each of the objects and we can manipulate each of the objects that is contained in this and uh anytime we want to move to the next object we use the next function and when we have finally completed all of the or we've manipulated or traversed through each of the sub objects we encountered the stop iteration exception which tells us okay there are no more objects to be manipulated or

3:14 iterated upon so and as an example i will create a list where i will store the values uh some simple values integers one two three four one three four five okay um i'm going to create an iterator out of this where i will use it function and i'm going to say this is let's say this is called my my heater okay and i'm going to use iter function as i had mentioned and i'm going to pass my list in this now if you see what is my eter it should say it should be a list iterator at a particular memory location now let's use the next function to keep going through whatever is there in this so of course the first object should be one as you can see now when i look at the next it should be

4:00 3 then i should get 4 and now i should get 5 as i use the next successively now since i've reached the last of my objects in this collection now the next time i use this next function i should get a stop stop iteration exception that is raised as you can see here so this is the basic um iteration protocol in python now you might have uh already gone through the for loop video and in that i have spoken in quite a bit of detail as to how a for loop performs this this iterator protocol and how it creates an iterative iterable object out of say a list or a tuple and how we how it internally performs the next function and it breaks out of the for loop once it encounters the stop iteration exception

4:45 so um for loops are like they're very very uh similar to your standard iterator and they use pretty much the same protocol for its functioning now let's do a basic comparison of what is a a basic comparison between a generator and an iterator in python so in generator we use a function we use like we define it using the def keyword and we define it under one function to create a generator whereas in the case of iterators we use two separate functions the iter function and the next function the generator makes use of the yield keyword whereas the python iterator does not in generators we save the states of the local variables each time we use the yield keyword and we pause the loop temporarily whereas in iterators we do

5:31 not make use of any local variables all we need is any trouble like a list to iterate upon we may use any number of yield statements in generators and we do not use yield statements initiators generators are not memory efficient whereas iterators are very memory efficient so this is like basic uh comparison between generators and iterators in this video we will be covering the topic of modules in python now a module in python to put it simply is basically a python file that has functions and classes written into it and is saved with the dot py extension to specify that it is indeed a python file a python program can use one or more modules and modules can have one or more functions they basically help us organize our code and instead of having one like one long python file we can

6:17 have several files or several modules now modular programming is the process of breaking a large a large event or task into smaller and more manageable subtasks and we call these subtasks basically modules now individual modules can be grouped together like building blocks to create a larger application there are several advantages to modulizing code in a large application and we will be going through them right now first of all there is simplicity because instead of focusing on a big big problem a big task we can break down the problem into smaller smaller portions or modules now maintainability of code is much better through modules because they are designed to create these sort of boundaries or clear demarcations between different problem domains there is usability usability in the

7:04 sense that once we create a module we can use it in different parts of the code or even different different applications itself there is also scoping and what this means is modules typically define a separate namespace and this helps us avoid collisions between identifiers in different areas of a program functions modules and packages are basically all constructs in python that promote code module modularization you might have already come across a gordon intro into functions in this video we will be going through modules in another video we will also cover the topic of packages to import a module we use the import keyword and in this example what i'm going to show you is a simple function or a simple operation of importing the os module this is short for operating system and

7:51 as the name suggests we can perform system tasks through the functions and methods that are part of this os module so simply executing the statement will import this module onto our program for us to use now if we wanted to see all the functions that are present in a module what we have to do is use the dir function so if we print the dir os function it will print us basically a directory of all the functions that are present in the os module as you can see here like all of these are basically uh names to different functions that are part of the os module finally moving on another function commonly used with modules is the get cwd function and what this does is it will give us the current working uh directory that we are working upon right now so if i execute this it will

8:38 tell me what is the directory that i am currently working upon right now as you can see seed notebook dir slash 10. if we wanted to see all the files in a particular directory what we can do is use the list dir function followed by giving a path by default if we do not enter a path inside the parenthesis it will basically give us all the files in the current working directory which in our case is temp so if i were to execute this it gives basically gives me a list of all the files that are presently there in the directory that i am currently working upon now a basic step is um creating a python module the simplest way to create a python module is basically creating a python script with whatever functions and tasks that we want and methods that we want inside whatever module and saving it with a dot py extension um so in in my example what

9:26 i'm going to do is i have already created a script in notepad and i've stored it with a dot py extension that contains these four functions add subtract product and division um i have basically copied i basically use this code and i've stored it in a dot py script and i'm going to be using this as my uh as as a test module for this particular example now how do we import a module now once that i have created a script uh in my example will be called my calc dot py and we've stated stored it in our directory to import the module all we have to do is use the import statement so if i have to import my calc all i have to do is say import my underscore calc and now my calc has been imported um so we can use the import statement or if we want to import

10:11 something particular from mycalc we can do from my calc import now whatever like for example if we wanted the add uh the add function we can basically just say only add or if we wanted add and subtract we can say add and subtract for example so this is basically by using from we can say that we want something from something but only parts of it we don't want everything we can also use from mycalc import and we can say asterix to basically say uh import everything that exists in mycalc right now so this is another way of importing something from a module now to access contents in a module we have to use the dot operator and i will show you how right now so what i'm going to do is in my example i'm going to import my calc as calc so now what this since this

10:59 is the original folder or the original module but i'm going to give it a shorthand name so that i can easily use it in my code and i don't have to type out my underscore calc every time so now i'm going to import this as calc so anytime i have to refer to a particular method or function inside this script or this module i will just use this uh this placeholder this variable name so now i have two variables a and b and i'm storing 20 and 30 in them so my first func what i'm going to do is basically four operations i'm going to print the addition of a and b then the subtraction multiplication and the division of these two numbers and each time to do this operation i am going to use the add function that is present in the my calc module so to do that i have already imported my characters calc all i am going to do now is invoke these functions using the dot operator that

11:44 we're already familiar with so i'm going to say calc dot add and then i'm going to pass these two arguments or these two variables to say that i want to add these two and similarly i'm going to do it for subtraction um multiplication and division as you can see these are the results of my tasks now if i had to do a directory uh operation on the module calc it will basically give me a bunch of inbuilt or dunder methods that are already a part of any python script by default such as built-ins so if i execute this just to show you what happens when i use the dir function it will basically give me all the functions that are part of this python script so these double underscores that you see are basically uh denoting magic methods or dunder methods and these are like inbuilt methods as a part of the script and if you see here add division prod and sub

12:30 these are the methods that we have defined earlier on in our script ourselves now to go slightly deeper or in more detail into what happens when we do a module search um let's let's actually look at what happens so what happens is when for example we execute the statement import mycalc as calc as we did over here so the first the steps that actually take place are the interpreter executes the above statement and it searches for this python script in a list of directories assembled from these following sources that you see here so it takes the directory from which the input script was run or the current directory if the interpreter is being run interactively which is the case right now secondly it will list all the directories contained in the python path environment that you set up that you can set up or you can change later on as well and this format for python path is os dependent of course

13:18 but it should mimic the path environment variable that we see in windows systems and then the other source or the other directories is basically an installation dependent list of categories configured at the time python is installed now the resulting search path that is accessible in the python variable system.path this is basically obtained from a module named sys so if i imported this this and i did sys.path now these are all the directories that we are going to look for when we want to search for any particular module so if i'm doing uh import my kayak or import os it's not only going to look at the current directory that i'm working on it's also going to look at all of these directories that have been set up as you could say directories or paths in the system.path in this video

14:03 we will be covering the topic of packages now what if we developed a very large application in python that contains many different modules and we have so many different modules that have similar names and functionalities and they're all kept in like or dumped in the same location that it becomes very difficult to track these modules what if there was a way to organize and access these in a much simpler and more organized manner well packages allow us to do just that packages are a hierarchical structuring of the module namespace using dot notation and this is the same dot notation that we use when we want to access methods or functions that are part of other classes or modules so let's try to let's look at certain examples to understand what we mean so i have a directory called pkg this can be basically treated as what we call a

14:49 package now in our package we have two modules mod one dot py and mod two dot p by and what we're going to store in mod one and mod two dot p py are the follows so i'm going to define a function called foo and i'm going to print this particular statement mod 1 and then i'm going to basically say foo and then i'm going to say in class 4 i'm going to have an empty pass command similarly i'm going to define a function called bar in mod 2 dot py where i'm going to print this statement so basically mod 1 dot py has a simple function where i'm going to print this statement and mod 2 dot py has a simple function then i'm going to print this statement and i'm going to store them in mod 1 and mod 2 dot py respectively and these two modules i'm going to store in the directory pkg in my local folder now i've already done

15:34 this so we can already get to working with this now if i wanted to access these particular modules from the pkg folder i can simply use the import command and use the dot notation so i can say pkg.mod1 so this will inward this will import the mod1 module from pkg folder similarly i can do the same thing for the mod 2 module from the pkg folder in the same line so now that it's been imported just simply execute this and this imports whatever you needed now if we wanted to perform a function that was a part of mod1 or py we can just extend the dot notation and use the function as over here as you can see it performs this print command whatever it was supposed to do similarly i can invoke the

16:20 function that was part of the mod 2 module by extending the dot notation to it in the same way now another way that i can invoke i can import a package is by directly importing the package itself by using import pkg and not accessing any of the specific modules within it now let's see what happens when we do this so just for this example what i'm going to do is i'm going to restart the kernel so that the pkg modules that were already imported get cleaned away so we see what happens when we execute this on its own now if you see i haven't imported this now let's see if i can access my mod 1 and mod 2. now if you see here it's throwing me an error and this is because using simply using the import command on a particular package only imports

17:06 you could say the package namespace but it does not actually import any of the modules and files that were part of this package so if i wanted mod 1 from this package i would have to actually say import pkg.mod 1 directly and i cannot simply say import pkg and python will not import everything that is part of a package on its own i have to specify it myself specifically now i'm going to create and i'm gonna i mean i'm going to show you another type of importing from a package using the star operator so i have this package pkg once again now i'm going to store um three i mean four different modules within this package mod one one one dot p by mod two two two dot p by and i've already done this you don't need to bother about um seeing me doing it right now because

17:51 i've already created these files and what i've done is within mod one one one dot py i've defined i've written this statement and i've stored it in the python script similarly i've created another python script script called mod 222 dot py and i have basically written this in the code and i've simply done this for mod 333 dot py and i've done another for mod 4 dot py now once i have done this i've stored these python scripts into my uh into my package or pkg package so now what i'm going to do is i'm going to create i'm going to execute this command if you see this from pkg dot mod 333 import start so once i execute this now i'm going to say print directory and print directory basically will tell me all the functions that exist in my current directory if you see

18:37 here there's this bias function that was a part of mod 333 module now if i change this and if if i imported mod 111 and if i did the same thing you will see foo has now been added as a part of this direction directory and these are all uh functions that were part of these different modules where i'm importing from and that is what the star function does what the star function will do is it will basically tell python okay so the statement is sync from pkg.mod111 so it's saying that um okay look into this directory or this package and find this module and in this module import everything that exists inside it up until this point so this is another another way of basically importing uh files or functions from a particular package and a particular specific module

19:22 within that package so yeah this was a very brief video on packages in python uh the conclusion being that packages are basically a an organized way of storing many different modules that we might have created in a very large application and these modules can be accessed using the familiar.note notation that we have used in other examples such as methods when we were working with classes and objects in this video we will be taking an introduction into the concept of generators so python provides a generator which is like a function where we can create our own iterator in a generator function we use the yield statement instead of the return statement that we use for standard functions so let's look at a simple generator first of all so out here i

20:09 have defined a generator called my generator where i'm doing uh like there are three uh like three functions or three tasks that i'm doing uh first one is when i'm printing the first item and i'm yielding a value 10 then i'm printing second item yielding 20 then i'm printing last item using 30. so let's see how do we use this now since we know my generator is the function it uses the yield keyword every time we use the next keyword we will keep getting what is next in this function so let's store the generator in my gen so now we have stored the iterator in this function variable called gen or gen

20:56 now let's see when we do next and we use what happens so the first task is performed or the first iteration is performed where we get this the first yield command is done all right let's see what happens when we do next again gen now as you can see we get second item and we get 20. let's do it for the next let's do it once more since there is one more task there's one more iterable inside you can see this now a valid question is what will happen if we do next gen again since there is nothing after this well let's see what happens we should get an error as you can see stop iteration error this is another

21:43 built-in exception that happens when we have like reached the last object of an iterable so we get a stop iteration error so now a key difference between a generator and a function is that when we execute a function the moment we hit the return statement in a function or when we execute a function like whatever variables that were stored inside are lost and they are destroyed unless they were defined as global and the function is completely done executing and we leave the function completely whereas when it comes to a generator when we hit the yield function we don't actually completely leave this function so the first time the yield is found we

22:29 get 10 but the generator remembers its position inside this function so the next time we use the next we actually move on to the next iterable and then we move on to the next iterable so you can think of a generator as it sort of has memory if that makes sense it's a way to like create iterables in python or create iterators in python so to understand a what the difference between the yield command and the return command does let's take the same function all right and let's put a return over here after this particular the second yield okay let's do the same thing we say gen is equal to my generator

23:14 and let's start uh yeah let's start doing next gen and seeing is there so the first deal will be standard first item 10 the next one is again is going to say second item and then 20 but now if you notice there was a return i put over here so technically when i did the second next it's not only yielded 20 but it's also executed this return function or this return statement so let's see what happens if i do next again as you see i have gotten the stop iteration seemingly earlier than i did before and that's because i have used the return function and our in return statement and what that's done is that um if we've completely left this function

24:01 so as to speak and there is nothing next so as to speak so we get these top iteration and that's uh the difference between yield and return now in the next example i'm going to show you how we use a generator function in place of the for loop or for a for loop so let's take this whatever over here and i have defined a function called get sequence up to x and what this will do is basically it will keep printing the printing numbers up until whatever we have entered in place of x over here so if i said say sequence is equal to let's uh copy this all right and say i want um i want sequentially i

24:46 want every number up until 10 well not including 10 of course because python starts with zero right so let's do this and um now let's see so the first number will yield 0 and now if i did this again it should give me 1 2 and i can keep going until i will stop at the value 9 as you can see here so 8 9 now i should get the stop iteration error because that we have reached the end of our iteration so this is a way of using generator functions for for loops so another neat example where we're

25:32 going to use a generator is in the case of fibonacci numbers so let's use it let's see how we can generate a fibonacci series using um generator so i've defined a fibonacci function called fibonacci where i'm passing max so max is basically the max value that i want um yeah like up until where i want the sequence right so this is basically the code which will give you return me the subsequent values of my fibonacci sequence so let's pass this to fib and say i want all the fibonacci numbers until 10 so and i will pass 10 over here let's see what happens when we do next

26:18 fib well it should start with 0 then when we do this again we get one then we do zero plus one it should be one next number should be one plus one which is two then we should have two plus one three and then three plus two five then 3 plus 5 should be 8. now 8 plus 5 is 13 and since 13 is greater than what we passed for our max value here now when we try this we should get the stop iteration error as you can see in this video we will be learning about the concept of list comprehension now list comprehension is it's a neat way of creating a new list based on values that are already there in an existing list so you can think of it like

27:04 creating a new list using a much shorter and cleaner syntax compared to what you might traditionally be thinking so let's go through some examples and through those examples we will see how list comprehension makes creating these new this much simpler much easier for us so in our first example we will be creating a new list which will contain fruits that contain the letter a in their name so we already have a list of fruits over here as you can see it has apple it has banana cherry kiwi and mango now we want to create a new list which has only the fruits that have the letter a inside them so apple would be one of them banana would be one of them and mango would be one of them so traditionally if you look at this piece of code

27:50 this would do what we want to do but this contains a few extra lines but yeah it does it does do what we want to do now what if there was a way of doing the same thing by just writing one simple line of code and that exists as you can see here so i take the same list now this is where you need to follow so now my new list is equal to so i'm creating a list over here as you can see now within this list you have this line this is what we call a list comprehension so as you can see it says each fruit for each fruit in fruits if now it's checking for the letter or the character a inside each fruit and what this each fruit is that we're checking each

28:36 object in this list one by one so first in the first iteration each fruit will contain the string apple then in the next iteration it will contain banana and in each each iteration we are going to check if it contains the character a and if it does then we store that each fruit in the new list as you can see here so if i execute this line you see my my result is the same i have my whatever i wanted and this was a much cleaner way of writing or creating this new list let's look at some other examples now so my next example what i'm going to do is i am going to iterate through a string so i have a string it can be anything in this case i am using a string called human all right so i have a string called

29:22 human right now i want to store each character of this string into a list that we call etch underscore letters so if i use the for loop this is how we would do it if i execute this as you can see the list contains h and then u m a and then n now what if i wanted to do the same thing but uh by using list comprehension so what i would do is let's create a list over here let's underscore letters and um so as we know list com so a list comprehension will uh it's it's a so you just basically need one line in this case it'll just be like this now this is my string now let's see when i check what's in

30:09 edge letters let's see what is the output so as you can see i have the output as uh what i needed so this is a much cleaner way let's go through some more examples now now we can do more list comprehensions um we can even use so we have already seen uh one case where we checked for the character a we've seen a case where we went through where we we did a sort of a loop through a string we can also use this comprehensions in conditionals so let's uh look at an example where we use the uh this comprehension with an if statement so what if i wanted to store say the first 10 even numbers so this would so this is an example that i'll be using

30:55 so in my list i want to store the first 10 even numbers so this would be a way to do it so i have in range 20. so what this is going to do is x is going to take the value of 1 2 3 4 and so on until 98 either actually take it from 0 1 2 3 4 until 19. so when we do range 20 we're creating an iterate in iterable where uh we're going to loop through our iterate through the value of 0 to 19. so if i if i do this and this what this statement does is basically it checks if it's divisible by 2. so if this is equal to equal to 0 that means we have an even number and if this condition is met if this condition is met that means we store

31:42 the x into our number list so let's see what happens when we execute this it should contain uh 0 2 4 6 until 18 so this is the first and even numbers as you can see so this is a neat way of using if statement we can also use an if else statement so what if five i'm checking um each okay so i'm going to create another list and in this list i'm going to basically see if each of my uh num numbers either even or odd the first let's say 10 natural numbers so let's write this statement down so i want it to be even in case the i that i'm checking i being every number between so like say

32:28 the first 10 natural numbers as i said let's check if um if it is divisible by 2 well if it divides by 2 of course it's going to be even right else we say that it is odd and then we check for i in say the first let's say the first 20 okay so now when we do this let me check oh we check what's in the list so as you can see the list contains like even or even orders basically saying the number uh zero is even then the number one is odd and number two is even and so on and so forth so this is a an example where we used the list comprehension with uh if else statements

33:13 so another neat thing that we can do with the list comprehensions is that we can use it along with functions so let's look at this function in this example we have a function called double x where we define this double x uh this double function to take an input uh to take an argument x and what we return is the square of the argument so let's use this in uh in our lifts this comprehension so what we do over here is that we are we're checking what we're doing is for every x um bet in range 10 so we're taking the value of 0 1 2 3 until 9 and we're passing it to this function double x and we're going to get the square of them and we're going to store that in a in a list as you can see here let's call this list

33:59 uh x let's just call this list it's called y so now let's see what's in y and we check uh it should be 0 1 4 9 and so on so wait this let's check again does this square no this is actually 2 times so let's make it square as you can see so you can see 0 1 4 9 for the first 10 numbers 10 natural numbers now what i've shown you is obviously list comprehension but we can also do set and dictionary comprehensions in fact set comprehensions work exactly the same way as list comprehensions the only difference is the set comprehension will make sure that there are no duplicates inside the set so you can't have like duplicate items

34:45 which you can in the case of list comprehensions so let's uh let's just take a look at an example so here in this example what we're doing is we're going to check so we have this string that we're storing in the variable text and the string is uh something like life finds a way so whatever this is now what we're going to do is we're going to uh in iterate through this string character by character and we're going to check if uh there are we're going to check for wobbles essentially so uh we're going to see if the character a e i o or u is present in the string and if it is we will pass it into our set now if you if you look you will find there are multiple i's right and there are multiple a's as well so

35:32 if we did this through a list comprehension each time that we found find a vowel it would be added to our list but since this is a set and sets do not allow duplicates so you will only have one copy or one particular object for each of the vowels so uh of course now uh set comprehension is defined using curly braces so let's keep that in mind so now if i check what's in this i will see a e i and u even though there are multiple a's or multiple e's or i's and u's since this is a set we don't have duplicate copies of any copies of any of these uh so this was just a small example for showing you set comprehensions similarly we can do the same thing uh this this comprehension uh for dictionaries and we call the

36:19 dictionary comprehensions the only thing that we have to worry about when it comes to dictionary comprehensions is that we have to define a key when it comes to this so let's look at an example in this what we're doing is again dictionary comprehensions are also defined using curly braces just like sets except in this case we will define a key and that separates it from a set comprehension so in this case what we're doing is that i'm checking uh i'm going through the a list of numbers from zero to nine right so what the range then function does and what i'm going to do is i'm going to create a key which is going to be like 0 1 2 3 and the value of each key would be a square as you can see here

37:04 i'm doing i times i or just to make a little more clearer as you can see i'm doing i square over here for every value from 0 to 9 so what i should get is like a dictionary where the first key is 0 and the value of that key should be 0 as well because 0 square is 0. then for 1 it will be 1 and then for 2 4 and then 3 9 4 16. so let's see what happens as you can see here this is the the result that i wanted so just to summarize let's go through some key points uh to remember when it comes to list comprehensions um so it's obviously an elegant way uh to define and create lists based on existing lists

37:51 it's more compact and faster than normal functions and loops but we should also keep in mind that we shouldn't write like very complicated list comprehensions because the idea of a list comprehension is to be user friendly and also it's important to note that every list comprehension can be written in the form of a loop but every loop cannot be written in the form of a list comprehension and that is a probably a very key point to remember so yeah that's it uh this for this video on list comprehension set comprehension dictionary comprehensions i hope you learned a new a neat new technique or trick when it comes to creating your new list in this video we will be learning about the concept of regular expressions now a regular expression is simply a sequence of characters that defines a search

38:38 pattern for example if you see this example here you can see a symbol called the carrot followed by p and then five periods then an n and a dollar sign and what this regex pattern does is it's searching for a seven letter string that starts with p ends with n and has five characters in between let's look at this example now this is a regex pattern that is looking for a five character string that starts with a ends with s and has three characters in between now if we had to match this with this tree with these strings as an as i or test strings the first one would not match because it's a three letter string and we're looking for a five character string the second one would match because it's a five character string that ends with an

39:23 s starts with an a and has three characters in between this will not match because whatever pattern would do is it would find the first s and it would find that it the string itself starts with an a and it would say okay there are only two characters between this first s and this a and it would assume that this is actually just a four character string to actually identify two consecutive s's we would have to do something special and what we would have to do is what we will see later on so again this is the same example since this is case insensitive capital a is perfectly fine when we're searching for this and it will also match this is this is obviously two words and it's not a five character string so this will not be a match now the regex pattern or the regular

40:08 expression module in python can be imported using this statement import re now once we've done this we can actually use our regular expression tools and methods to do pattern recognition character recognitions and to identify different sequence of characters in our strings let's try to understand a bit more in depth about regular expressions now so what you're seeing here is one variable called pattern which stores something that looks like a regex and another variable that stores our test string called abyss now what we're going to do is we're going to use the match method of this module re and we're going to check whether this pattern or whatever we are trying to search using this regex is in fact does it

40:54 match with our test string and what we're trying to look for is if you see here very similar to what i just showed you a five character string that starts with a and ends with s so if you see our test string it indeed is a match and when we use this uh this command we should get a perfect match so we should be basically it should say that search is successful if i execute this and this is what it does now if i change the test string to this now you will see that now this is no longer what we're looking for a five character string with uh a and starting with a and ending with s and you will see that the search will be unsuccessful as you can see here now let's try to understand what exactly do we mean by uh when we use these

41:40 different certain these special symbols like for example this carrot or this dollar symbol um these are by the way known as meta characters and now we're going to look into all the uh or most of the different meta characters that are used in red x so to put it simply meta characters are simply characters that are interpreted in a special way by the regex search engine now here's the list of meta characters let's go through them one by one now the square brackets square bracket simply specify the set of characters that we wish to match so we want to match some combination of a b and c it can be one a two a's three b's four c's we are not looking at a specific pattern we just want to know if a b and c exists in any form in our string so if we used it

42:26 on a the string a simply we would find one match because the letter a is found here ac would match twice because we have a and c present here in this string hey jude we will have no matches because a b and c none of them are found here and in this weird string we will have actually one two three four and five matches so we would have five matches for this uh this bigger string because we have five instances of a b or c now let's use re.match to do to actually search for this and to see if we have our particular whatever patterns that we're looking for now in this input string this is just some example string i'm going to test this i'm going to

43:12 match whether my pattern that um any of a b and c is it actually found in this so let's see what happens when i use re.match for this so this is the format the syntax is simply the regex pattern and then comma whatever test string that we have so let's see what object gets created when we try to do this as you can see it's none even though you might be wondering like i can see an a over here i can see a c over here re.match actually gives us the result none now if i try to do it on this test string which is slightly different instead of the film titanic was at least 1998 what i'm saying here is all film titanic was at least in 1998 now let's see if the r match

43:57 works or not as you can see it actually does return when i execute this it returns us an object called re.match object with a span01 the span01 is basically telling us that we have found a match between the zeroth and the first character which is basically this character a which is what it has found and it's saying that the match is indeed the character a from amongst these options so what this is basically telling us and what this means actually is that the re.match method will only find matches if they occur at the start of the string being searched once there is a space in in our bigger string it will not search for whatever comes after it so what happened in the first instance was it was actually only searching for

44:43 matches in this particular word in my largest string and since it found no match it basically gave us the result none which was not the case in the second example because it found a match immediately in the first letter a now if we want to actually check the whole thing what we would be using is the re.search as you can see here the re.search method will actually check the whole string and uh it will return us the first instance where it finds a match now let's see this is the test string now and we're searching again for our uh we're searching for a b and c any any of these in our test string but we're going to use re.search and not match so now if you see it returns us when i execute this it returns us a match object again but this time it's saying the span of 9

45:28 and 10. now the spine panel 9 and 10 actually exists over here this is the ninth character and a itself is the tenth character over here so what this basically is telling us that the tenth character is where we found a match which was the letter a which was the first instance of either a or b or c in this whole string and this is basically how re.search works it will not only search the beginning of the string but it will search the whole string and return us the first instance wherever we find now what if we wanted to find out all the instances wherever a b and c were found and not just the first instance but in that case we would be using the method find all over here so when we use find all it will actually return us each and every case where it found a b or c and uh you might not it won't tell you the exact position

46:15 but it will tell you like um so if i execute this you can see it gives us a a list with a c a and a and what this basically means is it was looking for the characters a or b or c and it found a first and hence this for the first item the list and then it found a c and then it found two is successfully so this is exact this is actually how find all works where it will actually find all of the instances of our regex pattern now that you've seen how the square brackets work and how the re.search and re dot match and re.find all methods work now we're going to go through all the other meta characters that we saw above so the period now period matches any single character except the new line new line is basically slash n or a break now if you look at this picture

47:01 what it's going to do is um this expression basically saying that uh do we have um two characters like sets of two characters in our string so this is a single character string and obviously a single character string is not a double character string and this expression this double dot is basically asking us do we have two characters present in our string so if i had to match it with this a it will obviously be a no match because a is one character if we checked it with ac it is a match because a and c are two characters a c and d will also give us only one match because it will find a c and then it will leave d as a single character but when we check a c d and e or this string in particular it will find two matches because it will consider a and c as one of the matches

47:47 and d and e as one of the matches so below is basically what we are doing basically the same thing we're just using rd dot search for each of these different examples so if i if you see when i use find all on acde and when i'm searching for a two character match you will see that it will return me a c and d e as you can see here and similarly for something like this it will return me again whatever like two character patterns it finds successively so the next symbol the next meta character is the carrot symbol that you've seen earlier then the carrick symbol is basically used to check if a string starts with a certain character so if you look at these two this picture if you have two two expressions that we're going to check over here one is the uh one is checking if a is the beginning of a string and the other one will check if a

48:33 is the beginning of the string but is it followed by b or not so when we check this what you can see is the first string will be a match because it's only the character a and it will match here as well because this string starts with a but it will not match here because this string starts with b now if i look at this particular expression you can see that um this is going to search for if the string starts with a and is followed by a b so it will match in this case because a does start with a and is followed by a b but it will not match in this case even though it starts with an a because it's not followed by a b so out here is basically us using the ie.search to basically find these patterns the next symbol that we will come across is the dollar symbol so the dollar symbol that you can see here is used to check if a string ends with a certain

49:19 character and we've already seen how this is used in one of the uh one of the earlier examples so this is going to check basically if so this expression will check if the string ends with a so this this string is basically the character a so it will match this also ends with a so it will match and this particular string does not end with a so it will not match and we are going to do that using the re dot search over here so as you can see it finds a match in the first character itself here in this case it finds a match in the last character but in cab there is no match so it returns us a none another example is of the meta characters is star so this is the symbol for star and this what the star basically is used is for checking if there are zero or more occurrences of the pattern to the left of it so now if

50:05 you see here what this will check for is um is there any form of m and a that occurs before n and it's very important to note that the n has to follow the a in this case so let's look at these examples and what will happen so if i checked it with m and n this is actually a match because n is followed by zero occurrences of m and a which is if it might seem a little weird but it is technically the truth because uh the the star symbol checks for not doesn't check for the existence it just checks for if n occurs it must occur after an a or the a should not be there itself in this particular string so as you can see here the n occurs after an m and it's

50:50 fine because there is no a now you will see when it will not work so m a and n is obviously fine m a uh m triple a n is also fine because n follows an a and there's an m here this however main will not match now why even though we have m and a and we have an n you see the n is immediately followed by an i is n follows an i immediately or the n is preceded by an i immediately and this is a no match because our the strict condition in the search is that n must proceed in a if the a exists again in women it will match because n follows an a over here we're just going to use the id.search for these examples that we saw over here in terms of code using the re.search as you can see here

51:37 finally we have the next metacharacter which is the plus symbol yes so the plus symbol matches one or more occurrences of the pattern left to it now this is very similar to what we saw earlier in terms of the star except now in the plus the key difference is we're looking for one or more occurrences instead of zero or more occurrences that we were looking for in the case of star so when we say one or more occurrences now we need what is there to the left of the plus sign a zero no occurrence of it will not be allowed or will not cause a match so let's see now what we're going to do here we're going to use the same expression except we're going to replace the star with the plus sign so now what we're going to say is okay we need m and a at least one occurrence followed by an n the all of

52:24 these conditions have to be met so again if you took the first example m and n this time it will be a no match because there is no occurrence of an a over here for preceding the n so it's a no match m a and n is a match because well this is per this is perfectly fine m triple a and n is also perfectly fine but if you look at m a i and n much like earlier the n is preceded by an i and not an a so there is no match and again women is perfectly fine and we basically execute the same thing through the search function in this piece of code the next meta character is the question mark the question mark symbol matches 0 or 1 occurrences of the pattern to the left of it so let's see what this does so now we saw the star the star which was 0 or more occurrences

53:10 of the pattern to the left of it we saw the plus which is one or more occurrences of the pattern to the left of it now we will see the question mark which is zero or one only zero or only one and not zero or more of one or more which was the previous cases only zero and one or one occurrences of the pattern to the left of it again we're going to see the same expression except we're going to replace the meta character with the question mark so now let's see what what happens when we match so if you see here m and n is a match because there is zero occurrence of a and one occurrence of m preceding n so this is fine this is also fine because there is one way occurrence of a and m and then it's preserved followed by an n this will not be fine because now we have three occurrences of a which is

53:56 followed by an n we're only looking for zero or one this again main will be a no match for the same reasons that we have already gone through and women will be a match again for the same reasons and we'll read through the same things using the search function in this piece of code the next meta character is the braces character now the braces character is interesting because what we're going to do is we're going to see if there are so uh this is the typical syntax for the braces is and a brace followed by some integer n comma integer m and then closed braces so what this means is we're looking for at least n and at most m repetitions of the pattern left of the braces let's see through an example if you see this expression we're looking for at least two

54:41 and at most three occurrences of this the character a in our test strings so let's see what happens in the first case we have no matches in this case because this part of the string contains only one a and even this part of the string only contains one a we're looking for at least two and at most three this part will contain one match the second the second example because the second part of the string actually contains two a's for one uh two uh repetitions of ace con consecutively but this uh in the end and in the third example you can see both the strings actually contain matches so we get two matches because as you can see here there's two a's here followed by three a's here now when we look at this particular string let's look at what happens now so what we're going to find is that this

55:26 thing is perfectly fine so we have two occurrences of a but now we have four occurrences of a but what the search engine is going to recognize is these first three is and is going to see okay so there are three occurrences of a and i already have a match it's not going to consider this fourth a and the search it will just look at the first three and say okay there are uh three a's and my my match is fine so again there are two matches in this case again we're going to use the search function initially and you can see each in each of the cases there is a match except for the first case which is this case now let's see the find all option and see what happens when we do the same thing using find all so when we use find all it basically gives us each of the instances where the match has occurred in each of these lists now let's try to use a slightly more complicated example where we combine

56:13 different let's say different meta characters and we create a a more sophisticated expression that we want to search in our test strings so if you look at this expression what we're looking for is 0-9 now a dash symbol is valid in python using regex because what this basically tells it is that it's looking for either zero one two three four so on until nine basically any of the digits you're looking for any of the digits from zero to nine and we're going to uh check if at least two and at most four occurrences of digits if they are there or not in our particular code so let's see what are the matches that happens when we use it on test strings so if you use it on this string you can see it'll find one match this basically this particular part of the string where it says one two three

56:59 so this is uh three occurrences of digits so it's fine now when we look at this whole string 12 and whatever this is you will see this will not find a match here this obviously will not because these are characters but if you look at over this part it will find matches in the in terms of the first four and then seven and three the next two one and two again there are no matches because there's only single occurrences of digits we're looking for it at at least two and at most four so if i use the find all you will see like what all it will return so for the first test string what is going to return me is it's going to return me one two and three as you can see here but now if you look at this test string it's going to return me 12 it's going to return me 3 4 5 and 6 is going to return me 3 4 1 and 2 but not 7

57:44 because we're looking for at least 2 and then it's going to return me 4 5 6 and 7 followed by 8 and 8 as you can see in this list over here now if we use curly braces with only one digit it's basically telling the search engine that uh we want exactly so many repetitions of the preceding uh whatever kind of expression that we have so when we do this example for uh for example we're basically saying that we want uh x followed by a dash and the dash has to occur at least three times in our particular expression and then it should be followed by an x again which will not match in this case because there are only two dashes and it will match in this case because there are exactly three dashes and it will again not not match in this case because there are four dashes as you can see here none followed by an

58:31 object match followed by none next we move on to the alternation um the alternation meta character and as the word suggests alternation is basically looking for some alternate form of a and b so a should be followed by a b and uh basically this pattern has to keep going on so in the first string over here this pattern will not match in the second string over here uh we will have a match because we at least find an a even though there is no b but this is to consider match because we did find an a at least over here we'll have uh three matches because we have a followed by a b followed by an a again so if we did a reg expression for this we would basically find these many matches for each of these examples the next meta character is the group now a parenthesis is basically used to group sub patterns so let's look at an example

59:18 to understand what we mean now if you look at this complicated expression what this means is that i want some form of x and z the x and z have to occur together but before that what i want is some form of a following b following c some sort of alternation between a b and c so this part as you can see here is kept in a parenthesis because i want this whole condition to basically occur before i find in x and z so let's look at what happens here in the first case we don't have a match because uh there this basically this parenthesis condition is not being met the x and z is preceded by a space or basically the a and b is preceded is followed by a space so if the space didn't occur we would actually find a match as you can see in the next example

1:00:05 here you can see a and b occur basically this uh initial condition is met followed by x and z so there is a one match so it's fine for us now if you look at this slightly more a trickier one what you can see is we have two matches in this case first in a followed by x and z and then we have c a and b now this is perfectly fine because we we are looking for a b followed by an x and z we're looking for either of these things we're looking for some form of alternation of these things followed by an x and z and we can get that over here we get b followed by an x and z so you'll get two matches in this case now let's go through the concept of grouping constructs now that you've seen many meta characters you've seen how parenthesis works let's look at grouping constructs so grouping constructs break

1:00:51 up a regex in python into sub expressions or groups and this serves two purposes one is a grouping part which basically says that a group represents a single syntactic entity and any additional meta characters apply to the entire group as a unit and then there's a capturing part which basically tells us that some grouping constructs also capture the portion of the search string that matches the sub expression in the group you can retrieve captured matches later through several different mechanisms so let's look at basically what we mean by this example over here so if you see this this expression has a plus sign so we're looking for a as you know a plus looks for one or more occurrence so this will basically check for one or more occurrence of the character r only because there's no parenthesis as in this example so this plus is next to an r and it's going to check for

1:01:38 basically whether one or more occurrence of r happens uh preceding b and a but if we actually put b a and r in a parent thesis followed by plus it will actually check for the entire string bar and it will check one or more occurrences of bar together so if you look at these two examples when we use this uh on these examples you can see that these are fine like these matches are fine but if i used this parenthesis then these would be the acceptable matches and like ba triple r would not be a match if i use parenthesis uh as you can see here so this is how grouping constructs work and how parenthesis work so let's look at some other examples so what this will do is f-o-o has to be optionally followed by

1:02:25 b-a-r and b-a-r must exist together and this is because we have a question mark here this question mark symbol tells us that there should be zero or one occurrences similarly this particular complicated expression is checking for one or more occurrences of whatever we saw earlier basically this whole thing is followed by a plus so we're looking for this whole sequence for one or more occurrences of it c o w is simply just c o w we're looking for this pattern and this is going to check for zero or one occurrence of c or w because of the existence of the question mark for preceded by the parenthesis in this video we are going to continue our discussion on regular expressions now that you know the basics of regular expressions

1:03:11 let's continue our discussion with special sequences now in an earlier video on strings you might have encountered what is an escape sequence and how we use the backslash character to generate these escape sequences and an escape sequence is basically if you remember something that we use to either gives a certain special meaning to characters that are otherwise ordinary or it is used to remove the special meaning that are inherently present in certain characters that we call meta characters so let's look at what special sequences are in relation to regular expressions so the first one we're going to look at is the backslash capital a which is used to match if specified characters are at the present are present at the start of a string so out here if you can see we are checking if the word

1:03:57 d is present in these two strings so we would get a match in the first case over here because this starts with the word d but in the second case we won't get a match and this is how we're going to check it for the same thing using the find all function so if you see in the first case it gives us it returns us an object because we do find d in the beginning of the string in the second case we get us empties list because the string does not start with the word t now the next one that we'll look at is

…