Full transcript of “Lec 11 Fundamentals of Image representation and Image preprocessing and Data augmentation”. Click a timestamp to open the video at that moment. Need another video? Use the YouTube transcript generator.
0:00 Hello everyone, welcome to the lecture series on foundations of deep learning concepts and applications. So far in the last two weeks you have learned the fundamentals of a neural network including the concepts of single layer perceptron and multi-layer perceptron. You have explored activation functions. You have understood the role of optimizers and implemented multi-layer perceptron model through hands-on learning exercises. We have also discussed the various regularization techniques and saw their implementation. Now from week three onwards we will be moving into one of the most exciting and powerful areas
0:48 of deep learning convolutional neural network. Over the next two weeks that is week three and week four the following topics will be covered in detail. In uh uh week three since we'll be dealing with the images uh initially we will be learning the fundamentals of image representation then we will go into the architecture of uh CNN where we will learn in detail about the different layers in CNN and then we will also see the regularization techniques that we'll be using for CNN model and towards the end of week three we will be seeing the implementation of CNN using uh keras us. Coming to week four, once the uh
1:34 fundamentals of uh architecture of CNN is made clear, we will be moving into the various uh uh variants of CNN, various architectures of CNN where you'll be learning the alexnet, BGNet, you will be uh learning the Google net, restnet, inception, transfer learning. uh we will also be implementing all of these models uh without the transfer learning and we will also be implementing them with the transfer learning and towards the end of week three week four you will also be learning the ensemble model how to build an ensemble uh model so by the end of this uh segment you will have a very strong foundation um in CNN architecture
2:20 it's a working uh uh principle and practical implementation using Keras so that we'll be preparing you to the more advanced applications in CNN. So with this let's uh get started for uh today's uh session. The following are the uh topics that we'll be uh covering. We will first see the uh fundamentals of image representation and then we will also learn how many number of channels are there for each of these image types. Why not use a fully connected neural network when we are working on images? What exactly is the problem? We will understand that and towards the end of this session we will uh learn in detail uh about the various data augmentation uh techniques and how do you store the
3:07 data so that you can work um for you can work with the CNN. So uh since we are working uh uh since the topic is all about the CNN there are so many images that we'll be using. Therefore all the images that are used in this presentation are sourced from publicly um available academic and internet resources only and these are used solely for educational purposes uh so that we will have a better understanding of CNN's. The key references that are used for learning CNN are the following are the uh references. Um deep learning authored by Yon Goodfellow and um we have a course by Stanford uh University
3:55 that is a course on deep learning for computer vision. Uh most of the concepts are taken even from this particular and then we have a very good textbook that is deep learning with the Python. So the following are the three different uh references that are used for this particular uh uh series. Yeah. So when it comes to images, the following are uh the possible uh images that one can be working on. Uh you can work on RGB images uh that is a color images or you will be working on a grayscale image or it can be on a binary image. the following uh I'm not talking about the formats. The formats can be JPEG, PNG,
4:41 TIFF format. There are so many other formats. I'm saying the possible different types that you may be working will be either RGB, grayscale or the binary images. Okay. Now before we go into the details, let us understand how does the system actually read an image. When you look into uh this particular image and when we closely observe uh this image, this image will have uh the collections of small boxes. If you look into we have many such a small boxes. And what exactly is each of these box called? We call it as an pixel. Therefore, each pixel we have an pixel here. We have a pixel here. We have a pixel here. Each of uh these uh pixels
5:28 in an image is represented by a numerical value. Now these numerical values are called a pixel values. So how how does the system read an image? A system will read an image as a grid of uh pixels. Now when you look into an image, an image is having a dimension. we have an um x-axis and uh we have an yaxis. So what exactly is uh this dimension? The dimension of an image are defined by number of pixels along its height and um width. So how many number of rows are there and how many number of columns are there. So this is how we
6:16 define the dimension of an image. If uh this is the case, what is an image? Image can be represented as uh for a grayscale this is uh what this is the representation for a binary this is a representation for an RGB. This is the representation. If you look into each of this it looks like a matrix. It is a matrix. Why it is a matrix? Because we have a rows and we have the columns. So it is a matrix. Matrix of what? Matrix of a pixel values. So image can be represented as a matrix of a pixel value where each of these values. If you look into each of these uh values uh
7:02 actually denotes the intensity of the pixel. It denotes the intensity of a pixel. So for a grayscale what does it defines? It defines the brightness. when it talks when we talk about an RGB each of these pixel actually defines the color intensity. So uh with this uh image can be uh represented as an matrix of pixel values. Uh let us learn uh one of the most important concept and this is very much useful when we are doing the hands-on. So how many number of channels are there for each of these image type? So when uh this is an colored image, what example we are seeing here is a colored image. Now when you see this
7:49 color image, this is actually a combination of R, G and B component. So we will have for red if uh red this is one image, green this is one image, blue this is one image. So we have seen from the previous slide that image is represented as a matrix. So for a red we will have a matrix [clears throat] for green we will have a matrix and then a blue we will have a matrix. So how many such a matrix are there? The number of channels uh will be three. Why it is a three? Because we have uh three components that will define one pixel.
8:35 So if you take there is a pixel value here. There is a pixel value here. There's a pixel value here. Each of these pixel is actually a combination of three. So how do we represent a pixel here will be represented as a combination of three values where the first will be the R component, the second will be the J component and the third will be the blue component. So uh let me clear this screen. If you look into this is what has been explained. uh each pixel is actually a combination of three values. What are those three values uh uh we have an red component, green component and then the blue component. Therefore the number of
9:20 channels that are used for an RGB will be the three. The number of channels here are three. We will have three channels. Okay. Now how many number of channels are there when we are dealing with the grayscale? So this is an example for a grayscale. Now if you look into this image, this is uh left side is one example. This is one example and this is an second example just for an better understanding. If you look into an example one here, this is an example of eight. It is an handwritten 8. And if you look into here uh we have values zero and if you see we have a value 255. And then uh um we have all the values
10:07 between 0 to 255. Now uh how many number of channels are there for an uh gray scale? The number of channel here is one. Why exactly there is one. Now if you look into this very carefully uh we we don't have different colors. We we cannot say black and white are two different colors here. we have um we can we can uh see this as white is equal to 255 minus the black intensity. So uh if you take 255 when there is no black component so it is just the value is 255. So uh uh second example if you take it is 255 minus uh if you if you
10:55 take some uh uh value then you will have uh some other uh number which is uh not uh completely white but it is closer to white. Similar way in this place whatever the value that we substitute suppose we substitute 255 only so we get zero which is completely black. So we cannot say that there are two different uh colors. They are on on a same single uh channel they are at two extreme points. Therefore the number of channels that we'll be using for a grayscale is one. This is uh the number of channel. So for a grayscale we have one channel. So likewise when we are dealing with the
11:42 third type of image that is binary. So this is an example for an binary. It is an image which is represented in a binary format. Uh since uh the number of colors here are only two which is either zero or one. Therefore the number of channels that we'll be using for an binary image is one. So with this uh uh let's quickly look into a summary how many number of channels because this is one of the important concept when we are doing an uh hands-on implementation. So the three different types of images that one will be working will be either you'll be working on color you'll be working with color images or you'll be dealing with the grayscale or it can be an binary uh image. So if it is an uh
12:30 grayscale, if it is an RGB colored image, the number of channels um the value is three and um if it is an grayscale, the uh number of channel is only one and the value will range anywhere between 0 to 255. So 0 represents black, 255 represents white and then in between we'll have the different shades. So we will say it's a gray. And then the third type will be a binary. So number of channel here is only one and the value will be either zero or 255 or we can write it as 0 or 1. So this is all uh with respect to the number of uh channels. Now let's get
13:17 into the next important concept. uh in week one and uh two we have seen in detail uh about the multi-layer perceptron that is artificial neural network multi-layer perceptron. So multi-layer perceptron is used to solve a regression problem. It is used to solve a classification problem. So the model was performing very well. So why can't we use the same model when we are dealing with the images? So what exactly is the problem? If you look into this image very carefully, this we call it as an input and this is an output and in between we have an hidden layer. This is all hidden layers. So since we have a multiple layers we call it as an multi-layer uh perceptron MLP
14:03 multi-layer perceptron. Now uh uh what exactly is this input? Now if you remember we will have a data set. In the data set we'll have multiple uh features and the last one is the class label. So let's not bother how many number of features are there. So many number of neurons that you will put at an input layer when you are building an architecture. How many number of neurons how many number of features so many number of neurons at an input layer. Okay. So this is with respect to if you're taking an uh data set in the uh CSV format or a structured data. Now what is that we are dealing with? We are not dealing with um you know structured
14:48 data in the CSV um file but what we are dealing is an images. We are dealing with an images. Now sometime back we have seen how do an image is represented. Image is represented as a matrix and uh it will have uh the rows and it will have the columns and along with that it will also have uh it will also will it will also be having the channels. So if it is an grayscale the number of channel is one. If it is a binary the channel is one and if it is an RGB the channel is three. Okay. So uh uh for an neural network what is an input? Input is an image. Now in an
15:33 image we have a image is represented as the collection of pixels. So uh what is the dimension of an image? The dimension of an image is denoted as a number of such a pixels in each of its row and in its column. So for example if you consider an image of size 28 28 and three. What is the meaning of this? The uh height of this is 28. The width of this is 28 and the three indicates we are taking an uh colored image RGB. Therefore the number of channels here are. So uh internally how is that we are representing? We have 128 + 28. Um uh this is a second 28 + 28 and this is the
16:19 third 28 + 28. Now what is uh so the uh each of these pixel each of this image is represented by a pixel. So uh here a pixel is a feature for an uh image. So if you if you look into 28 28 into 3 the value is 2352 the uh value of this is 2352. So which means we will have 2352 as an input for this neural network. Now uh if 2352 number of neurons is passed as an uh input to um a hidden layer. Just imagine how many number of weights that we are going to have it. Every neuron is connected to every other neuron. So in
17:06 this particular place we have 2 3 52. So every neuron is connected to every other neuron. So you can imagine what will be the complexity of a model and if you look into this 28 + 28 this is such a low resolution image. So where you cannot be able to understand any feature which is present uh in the image. So if you increase the dimension of an image, say for example someone is working um with an image of a dimension 200 + 200 once again you're working with colored image therefore the number of channels here is three. So when you're working with this dimension the size of this will be almost you know the size of this
17:51 image is 1 lakh 20,000. So it is something which is very very complex architecture. Therefore we cannot go for for images we cannot use multi-layer perception. Little more let us get into the technical aspects why one cannot go for multi-layer perception when we are dealing with um images. Now uh um uh see here is an example two. Now when you look into this two the two is represented in the form of a matrix. So what is this? Uh it is represented in the form of a matrix where you can see there is uh no information which is present in the corners. So all this is
18:37 zero. So in the matrix because black is represented as zero. So all the values are uh zero and there is some information which is present uh here. So therefore you can see there are some values which are present here. So image this is an image for our human eye but for a system it is represented as a matrix of pixel values. Okay. Now what is the size of this? The size of this is 7 + 7. So since it is an grayscale. So the number of channel here is only one. Therefore the size of this image is 7 + 7 into 1 which is only 7 + 7 49 number of neurons. So when we convert this into
19:26 an single vector this two-dimensional data when you convert that into a single vector this is what is a this is what is a single vector representation of this matrix. Okay. Now what is the problem with this? Now since there are 49 such values we have 49 number of neur in the architectural diagram we have 49 neurons at an input layer. We have 49 number of neurons at an input. Now every neuron is connected to every other neuron at an hidden layer. This neuron is connected to this. This neuron is connected to this. This neuron is connected and so on so forth. This neuron is connected. One neuron is connected to if we have a 49
20:12 neurons at a hidden layer. One neuron is connected to 49 neurons. The second neuron is connected to 49 neurons. Third neuron is connected to third input is connected to 49. So we have 49 um features connected to 49 neurons at an hidden layer. So how many number of such connections are there? we have a 2 4 01 number of connections and on each of these connection we will have a weight. So next uh uh if you if you build the more complex architecture I leave it to your imagination how difficult the model architecture will be. So what exactly is the uh problem when we go with this uh
20:58 kind of an architecture. Now the minute we convert a two-dimensional into a single uh vector we are going to destroy something called as an spatial relationship. This is important point. Now what exactly is a spatial relationship? The spatial relationship here is how pixels are Now look into this matrix how exactly the pixels are positioned here. In the first row we have all zeros. In the second row we have 0 followed by 87 240 210 24 0 0 0 and in the third row we have 0 13 0 and so on so forth. Now whatever is the position of the pixel values in this
21:44 matrix representation the minute we convert that into a single vector we are going to uh lose its position. So this is what we are going to destroy the spatial relationship. So what is the spatial? How exactly the pixels are positioned and how exactly the pixels are related to each other. Now uh um if we destroy what exactly is going to happen here? Now uh if you if you observe this very carefully now u here when you look into this image we have something called as a curve here and uh here we have a straight line. something called as a straight line and again we have a curve. So and here we have a
22:30 curve we have so the minute you convert matrix into a single vector uh what exactly is happening um spatial relationship is lost along with that the local dependencies are also lost. What do you mean by local dependencies? Now u which is the edges corners will not be no more edges corners when it comes to an single vector. So this is the problem. Now what is the next problem? The next important problem is when it comes to an um multi-layer perceptron whatever the input that we pass to a neural network each of the input is treated as an independent uh uh feature. Now uh when you look into a
23:16 data set uh each of these features are uh treated independently for a neural network. Now uh what is uh the input that we are passing here? If you are passing an image when we are passing an image we are passing a pixels. Each of the pixel is passed as an input to this neural network. Pixels cannot be treated as an independent feature. Right? It cannot be treated. Why it cannot be treated? because every pixel is related to its neighboring pixel. There will be some correlation between uh the pixels. So this is one of the most important um thing that we will be losing the spatial relationship when we deal uh with when we pass an image as an input to a neural
24:02 network. So [snorts] uh if this is uh not very clear, let us take a simple example. Look look into look into this example. It is an example of 3 + 3 maybe a small portion of an image. Uh in in a bigger image we are just taking a small portion of 3 + 3. Now if you look into this here the horizontal um neighbors you if you take 1 2 and three are the horizontal neighbors. 4 5 6 are the horizontal 7 8 9 are the horizontal they are the horizontal neighbors. Similar way when we take vertical these are the vertical neighbors. They are the vertical neighbors. Now what exactly is the problem? The minute we convert this
24:49 into an sing uh you know the matrix should be converted into a single vector only then you can pass it to a neural network. Right? So the minute you convert this into a single vector just observe two and five observe three and four. So uh uh 2 and five here when we are in a matrix there is a relationship between two and five. What is the relationship between two and five? They are the vertical neighbors. But when you look into a single vector two is positioned and five is positioned in two different places. So we have lost the information. Now uh similar way when you take three and four here three and four are the neighbors when you have
25:35 take when you have converted that into a single vector. But when you look into an matrix um let me erase this so that it will be clear. When you look into three and four three and four are the neighbors in a single uh vector. But when you look into an matrix here there is no relationship between three and four. Three is an horizontal neighbor and four is an horizontal neighbor for the next one. So there is no direct relation here but here they are the neighbors. So this is one of the most important we'll be losing the spatial we are going to destroy the spatial relationship. So which we cannot afford when we are dealing with an images. Now
26:23 what is the uh next problem? Why can't we use multi-layer perceptron? The second problem here is when you look into parametric parameter exploration. What is a parameter exploration here? This is a simple uh input of 7 + 7. If you imagine an little good resolution image, little higher dimension image, then the uh the values will increase. So you will have an exploration. So with this the training will go slow and then there is a very high chances of increasing the overfitting uh problem. There is an high risk of overfeitting here. So uh this is the reason why we cannot use artificial neural network
27:10 that is a multi-layer perceptron for an images. So if that is the case what is the solution for this? The solution is we are introducing a concept called as convolutional neural network. We will see that uh in a while. Before that, let us uh uh uh get into the last topic for this particular session called a data augmentation. What exactly is a data augmentation? Why is that? It is so important when we are dealing with an images. Now look into this uh example. Here we have an X. Now when you look from your eye, from the human eye, this is an X. Now human I is human is so intelligent that even when you show this image human will still say it is an X.
27:58 This is also X for human. This is X. This even though it is not very clearly written. So even this is also X for a human. So for a human I uh it is uh for all these variants of X. This is an original uh image. This is original X and all these are different variants of X. But human I can able to say that all these are still X only. Similar way let's take one more example. Let's consider this as zero. When you consider this as zero all these are the different orientations or different variants of zero. Human I can able to understand this. But this is not the case when it
28:44 deals with an images. Why it is not the same? Let's uh uh look into the first uh two images so that we'll be able to understand. Now when you look here uh these two are not the same for a system. Why these two are not the same? To that matter all these images are not the same for the system. Here there is no information. Here there is no information. So it is fine. When it comes to this here there is no information but here there is some information. Similar way here there is no information but here there is some information. So uh for us all these are the different variants of X but for a system they are different but this should not be the case right. Uh so um how do the system
29:33 how do the model handles or CNN model handles this particular um scenario? Therefore we have a concept called as a data augmentation. What is the data augmentation? Data augmentation it is a technique or a tool which is used to this is an artificially you have increased we are artificially increasing the size and then the diversity of the training data set. Okay, we are doing two things here. We are artificially increasing the size and then the diversity of a training set. How are we increasing it? How are we increasing all of these? by modifying the existing original uh data. So we have an modified existing image from the existing image
30:19 from this existing image we are creating the different orientations of this. So uh this is called a data augmentation where we are artificially increasing the size and then the diversity of the training data set and how are we doing it? we are modifying the uh copies of the existing image. Why is that? We have to do this. This will help um the model uh generalization uh you know it will have a better generalization and thus reduces the overfitting. So let us go in detail uh about this data augmentation. In the data augmentation the very first uh step that we use is we rescale it. Now if you
31:05 remember in our multi-layer perceptron we do something called as an normalization. If some feature value is very small, if some feature value is very large in order to avoid the unintentional bias to a model, we are going to normalize so that all the values will fall into a particular range. So the same thing when you look into this particular image there is one pixel value which is very small. there is one uh uh pixel uh value which is very large. So uh in order to avoid an unintentional bias here also we are going to normalize it. How are we going to normalize it? Take each of the value and divide it with 255 because the maximum value pixel value will range
31:51 between 0 to 255 which is when you normalize it you want to bring down the values between 0 to 1. So if you want to bring the values between 0 to 1, it has to be divided with 255. Right? So take the first value 2.5 and then divide it with 255 we get this value. Likewise take the second number 119 divide it with the 255 we get this value and so on so forth. So this this is the very first um uh data you know data augmentation or pre-processing step that we use where we are going to normalize the values. How how are we going to normalize it? Just um bring down the values um between zero. This is the value between 0 to 255
32:39 and these are the values between 0 to 1. And why is that um you know just for an better understanding why is that we have not written it as 1 divided by 255 why is that we have written it as 1 dot 255 why is that we have written this dot here this is just for an Python in Python version two when we write 1 divided by 255 it is called integer division but from Python 3 onwards um it is an you floating uh uh division. So just to make sure explicitly it has to be a floating point division. We are writing it as one point divided by 255. This is just for
33:26 an Python um convenient uh sake. So uh let's look into how are we going to increase uh the variance of it? How are we going to do this? So we in Keras we have an important package called image data generator. We have an image data generator which is a part of keras and why is the and on what we are applying we are applying this on an image and this is a part of a pre-processing uh step and the kas is a part of tens of flow that is the reason we are writing it as from tensar flow go into keras into uh into pre-processing under we uh pre-processing we are
34:13 applying onto to images. So import image data generator. Now what is image data generator? Image data generator is a tool which is used now to artificially increase the samples. How is that we are increasing the samples? What is the package used? It is an image data generator. Okay. To this image data generator there are different parameters that we will be passing. What is the first parameter? The first parameter is whatever the uh image that we are going to pass to this that image should be normalized. How are we normalizing? We have just seen an example. Each of the pixel value will be divided it with with 255. And second uh important parameter
35:00 that we are passing is sheer range. Now what exactly is sheer range here? It is nothing but you are slanting the shape of an image. So if you look into this is the original image. What exactly the meaning of this shear is? Shear is slant. So how is that you can slant when you are dealing with the two-dimensional one? You can slant along with x-axis or you can even slant the image along with yaxis. So sheer range is a um slanting of an image. Now this is the original image by creating a slanting. Is it not we have created one version of uh an image by slanting along with y-axis is
35:47 it not we have created one more version of an original image. Yeah. So and if you look into there is one floating number. What is that floating number? This uh is an hyperparameter. You can change this uh value. Now whatever the example that we have taken here it is 0.2. Now what what is the math that goes behind this sheer range is this 0.2 will get multiplied with 180 divided by pi. So you will have 0.2 multiplied with 180 divided by 3.142. So we will get somewhere the value between 14 um I mean you will you will get a value u somewhere 11 something. So
36:34 now if you look into uh this when when we write just only one number 0.2 actually the value will range it is a range when we say a range you will have a two values between these two. So it is + 0.2 2 to -0.2 that is the range here. So you will you will be slanting uh uh from 11° positive 11° to negative 11 uh something when you do this calculation you will get 11 something. So uh since it is from + 0.2 to minus0.2 you will have a radians right? So it is a degree. So this is the degree with
37:21 which we are going to slant the image. So this is all about the uh shear range. Then so this is uh uh one example for a shear range. If you look into this is the original image and we want to create an orientation or different variant of this image. So how are we creating the very first parameter is the sheer range and by applying a shear range we can see that there is a slight uh slant of this uh original image. So this is one of the variant that we have created by using this shear range. The second is zoom range. So uh let me clear it so that you'll have an uh clear understanding. Now what we are discussing is a zoom range. Now zoom range here the value
38:08 that we have fixed is 0.3. So mathematically uh range always range will have uh uh two numbers. What are two values here? The mathematically it is 1 minus value is 0.3 and the second is 1 + 0.3. So uh the uh range here is between 0.7 to 1.3. So this is uh the value. So for now just understand independently each of these parameters. Later we'll be seeing with an example. So when you take a she zoom range. Zoom range you will have the value between 0.7 to 1.3. How did we get this 0.7? When we are passing 0.3 the
38:56 math is 1 minus value, 1 + value. 1 - 0.3 1 + 0.3 will be 0.7 to 1.3. Okay, how does this looks? Now can you uh look into you have this original image and these are different uh versions of different orientations of the original image. So uh now why is that what is an advantage of going for uh uh zoom is we can able to recognize the objects at a different scale. Uh this helps the model to recognize the objects at a different uh scale. So this [snorts] is one more example. If you see we have an original uh uh input uh here and then all of
39:43 these you you can able to see them with a different uh scale uh where there is a zoom in and a zoom out here. And next this is all about a zoom. Then we have a horizontal flip and we have vertical flip. What are the two values that we can give? Either you can make horizontal flip true uh vertical flip as true or you can make it as false. How does that particular uh example looks like? So if you see this is the original uh image. This is the original image for what we are now discussing is horizontal flip and vertical flip. This is the original image. When you do an horizontal flip, the original image will be in this. And if you do a vertical flip, it is in this
40:28 particular one. Now is it not uh with one uh original image we can able to see a different orientations of an original image. So is it not artificially we are increasing the um size of your training data set. And how are we doing? We are doing a slight modification. Now by slanting we have not lost the cat. It is still the cat only. If you uh uh look into uh uh you know by um uh doing a zoom we have not changed the animal the animal remained the same the input remained the same but we are looking into this image from a different orientation. So this is uh the different
41:13 um parameters. It is just not only these five different parameters. We have um different parameters also. We have more different parameters. Let's look into it. Before we go into different set, the next parameter is uh width shift range. So the next uh data augmentation parameter is a width shift range that we'll be passing to an image data generator. What is this width shift range? width in an image. This is the width, right? This is the width and this is the height and this is the width of an image. Now, what is this width shift range tells? We will be able uh to move this uh image
41:59 whatever the image we can able to move this image right or we can able to move this image left. So, uh we can move the image horizontally towards the left or towards the right. Now how much you want to move towards the right, how much you want to move towards the left is the value that we are going to fix it. So if you see here uh the value that we have fixed is 0.2. Okay. And what is the mathematics that goes behind is it is uh uh 0.2 WF. What is this W? WF is the width in in terms of number of pixels of an image. Suppose if you take the width size is uh 200 that is we have seen the dimension of an image is always represented as a number of pixels in the
42:47 column number of pixels in the row. So width uh number of pixels here are um 200. Now if you take this example 0.2 into 200 we get 40 pixels which means if you write 0.2 you have minus 0.2 2 to + 0.2 which means you will be moving minus is minus 40 pixels. The minute you write minus it will be moving towards the left it is left and then plus is plus 40 pixels which is you are moving towards 40 pixels towards the right side. So that's what has been written here. Um you know um left is negative and right
43:33 if it is a positive. Now uh um uh so how exactly uh uh it looks when you shift the image towards right side 40 pixels if you are moving towards right if you see the dimension of an image has not changed the dimension of an image has remained the same but the image which is the uh object which is there inside is being shifted the image is being shifted right now when image is shifted right by 40 pixels then um because it is uh moved towards the right the leftmost uh uh will be empty right when the leftmost is empty then you have to fill those leftmost how is that we are going to
44:18 fill that blank spaces the blank spaces are filled using an um parameter called fill mode so similar way when you take towards the left side this is shifting uh right and this is a shift left. Now how is how how much you are shifting? You are shifting 40 pixels towards the left side. When you are shifting 40 pixels towards the left, then right becomes empty. When right is empty, you have to fill those pixel values. How is that? We fill those pixel values. We fill them using fill mode. The function used is fill mode. Okay. So let's go in detail. I hope this is clear to you. So width shift range is mainly used to
45:06 shift the image horizontally either towards the left side or towards the right side. So whether to move left or right that depends on what is the value here. Whether if it is a negative value you are shifting towards the left. If it is a positive value you are shifting towards the right. How much to shift that depends on what is the width of the uh uh image. So uh you will multiply uh it with the pixel. You'll multiply W with the value here and whatever is the pixel values that we get so many pixels will be moved towards the left or right. Now let's get little more detailed into fill mode. The possible values that we get for fill underscore mode is you can write now what exactly is fill mode. Now
45:52 when we shifted the image to the right the leftmost portion of the image uh becomes empty. when you shift the image towards the left then the rightmost portion of the image is empty. Now how are we filling those uh left out portions? You can fill them using um either nearest uh so if you are filling it let's take this image in the pixel representation. This is just for you to understand. We have taken one image and this is the um matrix image represented in a matrix form. Now let us assume we are shifting this towards the right by one position. If you shift it right by one uh can you imagine if you shift everything towards the right by one we
46:37 are going to lose this entire okay once we lose then this portion becomes empty. If this portion is empty this uh will be empty. If this is empty, how is that you're going to fill that? Yeah. Um, so you will be filling it using a nearest. If you are using it nearest, what do you mean by nearest? Whatever are the nearest pixel values, the same pixel values will be repeated in this empty. So if you look into this uh example, so here all of these are the same. So it's like something is being dragged. So whatever is uh the latest let me erase and then uh show you. So whatever is these latest uh pixel
47:26 values these uh pixel uh values it is the same pixel values which is repeated in all of these places. That's the reason you're seeing that uh the values it it is all the same here. So this is called nearest. If you write fill mode equal to nearest then the nearest pixel values will be filled in that empty space. Second we have a constant. What is a constant? Now when you move towards the right by one the leftmost is empty. So in that left most empty we are going to fill one constant value. That constant value is zero. Then you will see that in all of this uh place you will be seeing a black patch. Okay. Next we have a
48:14 wrap. What is a wrap? Now what has come out? Uh when when when you shifted right by one position, what has uh come out? That will uh that is used to fill the empty space. So you see here this is the data which had come out and then we had an empty uh place um here. We had an empty here. Therefore we are going to fill what has come out. you are going to fill um in that particular. So this is a wrap and the last one is a reflect. What is a reflect? Reflect is an mirror one. Now let me erase to show you how exactly the uh reflect works. Now when we shift this right by one position then we have
49:02 lost um you know we have lost this particular and we have empty in this place. Okay. Now when you stand in that empty place and when you see what is the mirror reflection we get the mirror uh uh is the uh in an empty position when you stand and see what you see is these pixels right? So when you use a reflect it is uh those mirror um reflection is what is being filled. This is called uh width shift range. Similar way width shift range is moving the image horizontally. Similar way we have something called as height shift range. So uh in this place we have a height
49:50 shift range. What is height shift range? you are moving the image vertically upwards you you are moving the image either up or down. So but the working mechanism everything will remain the same. So when you see the width shift range how is that it looks you will see the image um moving um you know either towards the right or you will see the image to moving towards the left side. So can you see one single image but you have a different orientations. So similar way when you take an height shift range what is a height you are moving the image up or down. So but working mechanism will remain the math everything will remain same as the width shift range. So the value will be 0.5 h.
50:39 So u and minus0.5 h uh in width we have seen 0.2 2 WF minus and plus 0.2 W. W here is width. And for height shift range, we will take H. Whatever is the H value, you'll multiply with that. And so many pixel values will move uh in the uh up or in the down direction. Can you able to see how the images have moved in up and in down uh direction? And the last one we have is the rotation. What is uh uh rotation? when we write here 90°. So the minute you write 90° it is minus 90 to + 90°. So you can have the orientation in all
51:26 of these um you know for one image you can have 90° for one image you can have 87° for one you can have minus 45° 80. So this is the uh example for uh rotation and uh whenever we uh deal with the width shift right shift we should compulsorily use one parameter called fill mode. Here fill mode what we have defined is nearest. Now the most important step is whenever we go for data augmentation. Data augmentation is a technique that is applied always only on the training a data set. It is not applied on a tested data set. Therefore you see this entire is being given to a train data genen. But when it comes to
52:12 an uh testing images for test images we are only going for a rescale but none of these data augmentation parameters have been passed here. So with this what we are understanding data augmentation is something that has to be done only on training data set but not on a test data set. that is uh the uh an important step that you need to understand. Okay. So let's get into uh uh the concept here. Let us take an example and then uh uh understand how we are creating an different orientation. Let us assume I have a training uh data set and the uh size of this training data set we have 112 images. Now when we have 112 images, we have seen that optimizers will take
53:00 the data in batches. So we have a batch size also. What what is the batch size that we have defined? The batch size that we have defined here is 16. So when I uh uh so from 112 images when I'm taking only 16 images at a time then how many times I need to repeat so that I'll complete 112 images. So is it not 112 divided by 16 which will be uh um you know closer to value 7 which will be I think exactly seven right? Yeah. Or you may even get um here in this example you got the value seven. But um for some other example you may even get some value in terms of point then use a seal
53:47 and then convert it into a next integer upper integer higher integer value. Okay. Seven times you need to repeat so that you will complete this 112 images. Good. Now uh let us assume uh uh the very first uh out of this seven I'm taking the first uh um you know batch per uh steps. So this is the first 16 images I have taken it from the training data set. Okay. Now the first image what you have taken from this training will undergo all these transformation. Let me erase this. The first image what we are taking from the uh training will undergo all this transformation. What is this
54:32 transformation? Why we why it is going uh uh through this transformation only because that is the values that we have defined here for sheer range should be 0.2. The minute you write a 0.2 it is from minus0.2 to plus 0.2. So if you see all of these values will all of the values here will range between minus0.2 2 to plus 0.2. So the first image will between minus to plus randomly it has choosed this value and zoom range defined here is 0.4. So when it is 0.4 the value uh will be 1 - 0.4 is 0.6 and 1.4. So all of these values what you are seeing will be
55:17 between 0.6 to 1.4 only. you will not see any other value. Then we have horizontal flip either true or false. Vertical flip either true or false. Then width shift range here since the width value what we have defined is 0.2 the value will be between minus0.2 to plus 0.2 and similar way minus0.5 to plus 0.5 and the rotation what we have fixed is 90°. So it will be from plus 90° to you will have um between minus 90°. So uh whatever the values for the first image randomly it pick one of the uh value and that uh uh parameters are applied on this image. Then can you rec
56:04 can you imagine what would be the orientation of that image? Then you will take a second image. that the second image will go through this entire uh um image data generator. So there randomly it will pick again one more set of values and what values we have picked? Let us assume the values are these. When you apply these uh values onto the second image, can you imagine how this image orientation would have been? [snorts] Similar way third image will go through these orientation. Fourth image will undergo this um values have been applied and it will take one shape. Similar way all 16 images will go through this. So is it not a different orientations of an image we have uh
56:51 taken by doing this process. This is u from this uh seven this is the first one that we have done. Similar way you will take the second set of seven images from the training and these now will not again undergo these values only. Now it will go again a different set of values but it will uh restrict within whatever you have defined here. So with the third set of seven images will go through a different orientations. Fourth set, fifth set, sixth set, seventh set by end of this you are completing one epoch. What if you run this for 50 epochs? Can you imagine how many different orientations you are creating?
57:37 Variations we are creating. So this is uh what uh you know we are artificially increasing the size of the training uh uh data set. And how are we doing? by modifying the existing data only from the original uh data we are taking and then we are applying all of these uh parameters with this uh let's see the uh um continuation of the next u topic continuation of the previous topic how do you store the data for a CNN uh model now when you take a multi-layer perception you had a data in a CSV file and you were passing that data to read underscore uh CSV or read Excel if you are storing an data in Excel format read
58:24 and so you were passing into that particular function and the data was in the CSV file or in the Excel file but here we are dealing with images when we are dealing with the images how is that you should store the data first and how you can pass that data to an um you know model so here um if you are dealing with for example you are working on COVID data and you have a training data and then the testing data. So create one folder called COVID and then inside that create a subfolders and the first subfolder is a train second subfolder is test and again if it is an binary classification again under a training you will have a
59:11 COVID and non-COVID. Similar way for a test you will have COVID and you will have non-COVID. Now uh so how is that we have? We have created a folder under uh that we have a subfolders under each of these subfolders we have again a subfolder. What is these subfolder? Subfolders is where you are storing a class labels. Now under training you have COVID and non-COVID. COVID is one uh um class and non-covid is one class. So the uh with that class you will store the folder uh name. Since it is a binary classification you just have only two here. If it is a multiclass classification say for example benign border case and malignancy you will have
59:58 three such subfolders under train and test. So uh how is that um during hands-on implementation how is that we are uh storing or how is that we will be passing this there we had a read csv or read excel but here we will be using something called as flow from directory now flow from directory has a certain parameters and what are those parameters the very first one should have a path name and what is this path name the path name Um if you if you look into uh uh this flow from a directory uh the flow from a directory um expects a path and uh where each subfolder each of these
1:00:44 subfolders are treated as class labels and all the images which are stored under this COVID and all the images which are stored under non-COVID will be automatically labeled accordingly and these images the images which are stored in COVID will get the label zero. know all the images which are stored in non-covid will get the label one or vice versa or it can depends on alphabetical order and it is done by kas automatically uh internally this is a flow from uh directory now if you look into uh this flow from directory this is part of uh training data genen what is a training data genen if you see in the previous uh uh one uh training data
1:01:31 genen And is this can you recall this is a train data gen. You have defined all the data augmentation pre-processing um you know data augmentation parameters and then you have assigned it to a variable train data gen. So uh uh uh when you see all the images all the images which are there in a training set will go through this train data gen. So what exactly mean by this? Every image in the training will be going through all of these data augmentation uh parameters. That's what is written here. It handles the image pre-processing and data augmentation flow from directory. We have seen that it expects um as a subfolders and each we have something called as target size. What is a target
1:02:18 size? Now when you collect the uh images when you collect the data maybe some image will be of one dimension other image will be of a different dimension. So what we are going to do but a model expects all the images should be of same size. So we have here we are defining what should be the target size what should be the width and what should be the height. What should be the x and um y height and width. Uh so uh target size it can be either um 64 + 64 or it can be 32 32 512 cross 512. What is the target size that you want to fix it? Then we have a batch uh size. What is the batch size during the training? How many samples in batches you want to pass the
1:03:04 data for a model to get trained? It can be eight, it can be 16 or it can be 32 or it can be one of the um it can be a value here. And the last one we have a class underscore mode. When do we write it binary? When do we write it as categorical? Here the example what we have considered is a binary binary uh classification. because it is a binary classification where we are dealing with only two kinds of um uh labels that is COVID and non-COVID then class mode will be binary. Suppose one is working on multiclass classification problem. For example, somebody is working on cancer detection where you have benign, malignancy and border case. In that case, class mode will become categorical. So, I hope uh this is uh
1:03:51 clear to you. With this, we have come to an end of uh the uh first session um in week three where you are going to learn um CNN here. So so far if you quickly look into the summary we have started with the fundamentals of image representation where we have seen how many number of channels are there for each of we have considered three different types of images we have seen binary grayscale and then um colored image RGB. So and the number of channels for grayscale and binary the number of channel is one and for an RGB the number of channels here is three. And then we have immediately seen why not use a fully connected neural network for images. Ideally the most important is it
1:04:37 is losing the spatial relationship. And then we have uh finally we have seen what all different data augmentation uh techniques are there you know how is that we can able to perform this data augmentation using image data generator in kas and how do you load and store the images for um implementation this is done by using a flow from directory. So with this we have seen um you know the basic fundamental this will lay the strong foundation for understanding the CNN components and that we will be uh doing it in the upcoming session. So in the next session we will be going into the architecture of convolutional neural
1:05:25 networks where we will see what all the different key components what all different layers are there in CNN. We the different layers includes convolutional layer, we have pooling layer, activation function and we have fully connected. So in the uh upcoming session we will uh look into the uh introduction to convolutional neural networks and then we will understand the very first layer here called convolutional uh layer in a detail. So till then thank you so much.
