Mastering MLOps: From Model to Production

6:47 Hey everyone. Uh we will be starting in a few minutes.

28:44 Cool. Um, so I'm good to go whenever you guys want me to kick it kick it off.

29:44 Oh sir, I'm starting. Good evening everyone. I am Nishant, an associate at Consulting and Analytics Club, IIT Gujhati and I welcome you all to week 2 live session of summer analytics course. We are honored to welcome Mr. Abined Gupta as the keynote speaker for today's session on LNMS. Mr. Gupta is a seasoned professional with over 10 years of experience in data science, risk forecasting and statistical modeling. He currently works as an applied machine learning scientist at Shopify where he focuses on building robust, trustworthy and scalable machine learning systems that power real world applications. He is an IIT Karakpur alumni and he has completed his masters and PhD from Stanford University. His deep

30:30 understanding of datadriven systems and long-standing contributions to the ML space make him exceptionally well suited to guide us through today's session on LMS. Please join me in giving a warm welcome to Mr. Binit Gupta. Sir, you can take over. Hi everyone and uh thank you so much Nishant for that introduction. Um so first of all I'll just do a quick check. Can everyone hear me? Um so just mention in chat if you can and Nishant can you can you hear me? Yes. Yes. Perfect. All right. So I will kick it off. Um I think there is a little bit of a lag between

31:16 um when I'm speaking and when um YouTube is showing the live stream. So I'll just keep that in mind uh because this presentation is going to be interactive and I will be asking some questions and we'll be getting some feedback from the people on uh who are joining this live session. So thank you everyone for joining. As Nishant mentioned uh I'm a senior machine learning engineer at Shopify with background in machine learning and statistical modeling of over 15 years. Um during my PhD I was working on uh risk forecasting for disasters and more recently I've been working on uh fraud risk and today's session is going to be MLOps. I see some questions in the chat about uh

32:02 hackathons. Hopefully what I'll talk today will inspire you to think about uh your problems in a slightly different way when you're working on your hackathon project. Um I mentioned I want this session to be interactive. So there will be some questions throughout the presentation that I will be throwing at you to see what thoughts you have. So feel free to use the chat and as we are starting out um if you just want to try out chat mention where you're from uh what you're studying uh maybe what year are you or just say hi. All right so let us jump into the presentation.

32:48 So the first question what is MLOps? MLOps is ML operations and typically when we are developing uh taking a course on machine learning or training a new model we only tend to focus on the machine learning model itself which can be a prototype model or a proof of concept but it doesn't allow us to put that system into production. So MLOps is the set of tools that uh we put in place to operationalize ML models and make sure that they can be deployed in production and they are reliable uh and scalable and we can work on those ML models or training collaboratively as we are um uh thinking about taking an ML

33:34 model that we have trained and serving it to our users. If we look at what an ML system is, typically we f tend to focus on this part which is the ML code and the ML system comprises of a lot of other components that are that form the scaffolding around this ML code. So there's the configuration which is uh different machines that we are setting up uh different configuration of features that we are setting up uh maybe hyperparameter tuning configuration that we are setting up. Um then there is a lot of emphasis on data. Uh how we are

34:21 we collecting the data? What features can we extract from uh the data set that we have? How do we verify the data that we have uh is going to meet our objectives of solving the sample problem. And once we have trained the model then how do we serve that model? um what processes do we put in place to serve that model so users can access it uh within the requirements that they have and finally how do we monitor the performance of the model in order to determine whether it is time to uh retrain the model or change the model because it's not meeting our performance objectives. So what I want you to take from this slide is that as you're thinking about

35:07 the ML code part, ML code is just one small piece that fits into a very large puzzle uh of an ML system. And typically it tends to be the part that you that as a ML engineer I spend the least amount of time on. Most of my time actually is spent on working on the other parts that form the ML system. And as I'm going through this presentation, what I would like us as me and the audience to think about is this one problem. This will be our motivating problem that will serve as the through line for how to take ML code and build

35:54 an ML system around it. And the problem is something that I work on um in my day-to-day and I've been working on for the last few years predicting if a transaction credit card transaction is fraudulent or not. So think about when you go to a merchant and tap your credit card or debit card or you make an online purchase um as a merchant Shopify we want to know whether that transaction can be fraudulent or not and if it is fraudulent we want to decline that transaction and if it is not fraudulent we want to approve that transaction. So this is our motivating problem and as I'm going through the MLOps pipeline we will keep try to keep this at the back of our mind.

36:39 So to jump in uh let's first look at what is going to be our map throughout this presentation. I'll I've broken down the ML system into these four broad pillars. The first one is the data followed by training and experiment tracking then serving and monitoring and again bringing thinking back about what I showed with respect to the ML system. the ML code is just one small component which really fits into that training pillar and um this this kind of four pillars will serve as the map. So all my slides have this map in them and you'll see that so you can quickly know what part of the ML pipeline we are in at any given

37:25 point. So let's jump in and the first thing I'll start with is actually the ML code component because that's likely something that as the audience you are most familiar with right now as you're taking this course uh you have likely been looking at different ML models and um these models as you're thinking about this problem of predicting whether a credit card transaction is fraudulent or not you can probably think about which would be the right models to use. So I'll throw this question out to you the audience. Uh write in chat what is uh what which ML models would you use if you were given this problem to predict fraud in credit card transactions. So let's take a couple of minutes uh and write your answers in

38:16 chat. Um I I'm looking at the chat. Minina said hi. Hi Minaj. And uh let's see if other people want to chime in.

39:02 Hi everyone. Um Mayang joining from Chhattisgarh U Manage from Jhark. Um and Abib starting us off with some uh model suggestions. Um yeah so uh please give me some recommendations what you think would be the right ML models to use uh to predict fraud in credit card transactions and a logistic regression is a great answer with random forest again a really good option to use in this uh problem and let's take just one more minute uh so anyone else wants to chime in and provide suggestions neural networks uh really good uh option option we and neural networks is something for this particular problem that comes with

39:49 um some other considerations uh to think about. So if we have time towards the end I can talk about those uh where a pro neuronet networks could fit in into a problem like this. Um uh so let's see if we'll have time towards the end to talk about that. Um so dhar is it logistic regression? There is no one right answer. uh logistic regression is a good option and uh you are absolutely right that what we are trying to do is a classification problem where we are trying to predict whether some a transaction is fraudulent or not. Uh Minaj mentioned unsupervised learning. Um maybe in chat think write about how you think we would solve it using unsupervised learning. Um there are some

40:36 um some things I have tried with unsupervised learning. So I would love to learn more about what you have in mind. Um so a lot of logistic regression options there. Uh let's uh move to the next slide uh where I put in some options here as well. So everyone who posted is on the right track that when we are trying to predict fraud, it is a classification problem that we are trying to solve and typically for classification problems the first algorithm that comes to mind is logistic regression. Some other algorithms I've put down here support vector uh classification uh decision trees XG boost and someone uh Aush had mentioned random forest. Um so random forest would fit into the decision trees paradigm. It is a forest of decision trees. um uh

41:24 when it comes to random forest and XG boost if you haven't um come across it yet it's a faster random forest so there's some optimizations and parallelizations that have been done in XG Boost that make it a lot faster to train and typically gives better performance than random forest and this could be something that you think about u during your hackathon projects as well but here what I want to emphasize is when given a problem like this predicting fraud. It was actually fairly easy to think about what ML models we could apply here and that is the sort of emphasis of this talk that ML training or ML model tends to be the u maybe the

42:09 most accessible component of the problem. Uh but let's jump into the next part which is the data. So it's it's clear that we want to use a classification model. There are a lot of classification models out there uh that that have already been researched. Uh we can just plug and play different libraries. Logistic regression comes as part of scikitlearn. If you wanted to use lum forest, we could just u import that library. So it's it's fairly easy to just import those libraries, throw some data at it and let the model train. But what is the data? So again, let's take a couple of minutes and write in chat what you think the labels and the features for um this problem would be. Again, we're trying to predict fraud. Um

42:56 and we know that we want to apply a classification model, but there needs to be some data that we need to send to this classification model. So what would that data be? What would the labels be? And what would the features be? Whatever you have in mind, again, no wrong answers. Just write those in chat. And I love that the audience is participating actively. It's great when um the audience participates because it keeps me motivated to keep going and really brings up some nice conversations. Um like we mentioned neuronet networks and Minaj mentioned unsupervised learning and those are things that we can dig into um as we are thinking about this problem.

43:42 So more logistic regression um uh suggestions coming in into chat and let's change gears a little bit to what would you think the features and the labels are going to be for this problem. Uh what are how are we going to prepare our data to send to the machine learning model whichever we uh decide to use. Okay. So part mentioned amount uh location he also mentioned those. So good good choices. Amount would be the transaction amount and location there could be two locations actually. One the location where we are making that transaction. Um so that that would be the location of the merchant if we are there in person. But if we are making an online transaction then location can be

44:28 the location of the card holder or the buyer where they are sitting in front of their computer. And another location could be the merchant location where they are making this purchase. Um, Myntra, Amazon or a small uh merchant. Bank name is another uh good suggestion by Abi. Bank name would be the bank of the card holder. So let's say my card is issued by [Music] um Yes Bank or uh SBI and those would be features that can go in. Perhaps there are some banks where it's easy to do uh it it's easier uh to commit fraud and those can become features for our model that the model is ultimately going to

45:13 train on. Um the label uh Bria mentioned fraudulent and non fraudulent. I want you to dig a little deeper. How do we know that a transaction is fraudulent and not fraudulent? What is the ground truth? What is the signal that we are getting to know that the transaction is fraudulent? All I have is the transactions that have occurred. Um what additional signal am I receiving? Um that will tell me that a transaction that had occurred was fraud lent. We mentioned ID number. So that would be the card ID number. Uh information details maybe um information when uh we are mentioning you're talking about the name on the credit card. Um um let's see what other information do you

45:59 have in mind? uh yeah meant write in chat. Uh but those are also good features. The ID number we may not use the full ID number uh sometimes uh because there can be a lot of credit cards that are used and if you think about an ID number um it's not a numerical feature it is a categorical feature and we don't want to create a lot of categories. In this case, it could be millions of categories that we end up creating. But we what we do use is for credit cards, the first six digits are actually representative of uh the bank that issued them. And specifically within a bank, it can also be representative of a particular program that the bank has. So um there can be different levels of card. For

46:44 example, there can be a regular card and then there is like a platinum card sometimes. And the first six digits of a credit card number can tell you about exactly which card program um this card belongs to. And so we do use those first six digits sometimes as a feature. All right. So uh I I love seeing all the answers coming in. Uh keep posting uh and I'm reading all of those in the chat. Uh but I want to uh move on. So let's talk about what the labels are going to be. So the labels typically how we know whether a transaction is fraudulent or not are chargebacks. Um let's let what are chargebacks? So I

47:32 uh make a transaction and I don't like the product that I received. And sometimes I can do a charge back to say that I did not receive the product that was um uh that the merchant had advertised and is why I don't approve this transaction or I don't approve of this product. So I go to my bank and say I want to is uh charge back this this transaction and they would investigate that and they would tell me um they would either refund the money to me or tell me that no that was a right transaction and we don't want to give you a charge back but there can be other situations where so this is not an example of the transaction being fraudulent um because this is just

48:18 something that I did not like whereas another situation can that I open my bank statement or credit card statement and I see that there are some transactions that occurred which I did not authorize or I did not make and that is indicative of th those transactions being fraudulent. Maybe someone stole my credit card or someone was just trying out different credit cards and got lucky and hit my credit card and made those transactions. So in this case I will also issue a chargeback but tell the bank that the reason I'm issuing this charge back is because I did not authorize those transactions and then the bank would put in a reason code of this transaction being fraudulent and that is why I'm mentioning chargebacks

49:04 is the signal but specifically we want to dig into why a chargeback occurred and we look at these reason codes in that case and those reason codes tell us whether that was suspected fraud or not And that becomes our label. And if you think about um the process here um chargebacks tend to be delayed signals. So we have to wait for a chargeback to come in. A transaction let's say occurs today. I wouldn't see it posted for a few days. Uh maybe I don't even open my um recent transactions and I wait for my credit card statement to be issued next month. Then I look at my statement, notice these transactions and issue chargeback. So it could take 30 to 90 days for a

49:51 chargeback to be created. So I don't So I have to wait until that long to know whether a transaction was fraudulent or not. But this is the signal that we can use as our labels. For features, a lot of features that you guys were mentioning are raw features. So we are getting those features directly from the transaction itself. for example, amount. Um, other suggestions were bank account, ID number and so on. Um, but we can also add things like hour of the day when the transaction occurs. And this can be useful when um as a buyer maybe I am in IST but a transaction occurred at 2 am uh local time and that could be indicative of maybe it is someone uh who's making a fraudulent transaction because that doesn't align with typical

50:39 waking hours in that local time zone. Currency can be another u feature raw feature that can be used and then we engineer some features as well. So engineered features can be as simple as uh we just take the log of the amount because amount can range from uh very low values like $1 or 10 rupees to um you know 10,000 rupees and just use that as a raw feature the model may not learn very well. If we think about logistic regression, it's going to put a weight on this feature and it's not going to know how much weight to put when the range of amount is so large. And so taking the log helps us to shrink that

51:25 range and the feature and the weights that we get tend to be a lot more reliable. Then another example of feature that we can engineer location came up and instead of using the location directly what we can use is the distance between the merchant and the buyer. So we look at where the merchant is located and the buyer is located and calculate the distance between them uh maybe based on the uh postal code um and that distance if the distance is longer maybe that is indicative that a fraud uh that the transaction could be fraudulent. And then there are more complicated um engineering engineered features like the average amount that will be spent uh that has been spent by the card holder in the last 10 days. And I want to dig in a little uh and

52:13 then the last thing I want to talk about here is the scale of the data. So even after we have these labels and features, the scale of the data that will be going into training can be really large. So for a company like Shopify, we have uh a million or more daily transactions that are coming in. And for a company like Visa which uh you you'll see on your credit cards there can be 100 million or more trans daily transactions that they are seeing. And if you think about using the last one to two years worth of data for your training these can be like billions of transactions. So let's say 10 billion as as our number. 10 billion transactions a lot of features to train

52:58 this model. we will not be able to do this on our local machines. And this is the first example of where an ML model goes from a model to a system. Um when we are talking about scale this large, we really cannot use our local machine and just throw this data at a model and have the model be trained. We really need to think about how we are collecting this data, where we are storing this data, how we are processing this data because the scale we are working with is so large. So the next question I have uh for the audience is how do we quantify typical behavior in order to know that a transaction is anomalous. So what what makes a

53:45 transaction fraudulent? A transaction is fraudulent when it deviates from a typical behavior that a buyer or a merchant may have. So let's say as a buyer I spend a lot of time buying shoes or clothes and uh with a specific credit card and then suddenly I start buying furniture on that credit card and that could be indicative of that maybe I did not buy that furniture and someone else made that purchase my credit card and production. But to to know that I need to understand what is my typical behavior and that is a feature that I need to provide to the model. So it learns what my typical behavior is and then it can use that

54:32 typical behavior to identify that transaction I made is now an anomaly or an outlier. So and and could be indicative of fraud. So uh write in chat again how do we quantify the typical behavior um of of a buyer or a merchant uh basically that will how will we engineer these features that will go into model training. So again opening it up to the chat and uh what I would like you guys to think about is what features would you engineer that will be able to quantify typical behavior of a buyer or

55:17 a merchant. Um so I see some um uh messages coming in the chat. Um Shuy mentioning um lab features and data set. I actually did not understand. So Shui if you could elaborate that would be great. Minhaj mentioning IP address as a feature that is a really good feature. Um but how would IP address tell me about a typical behavior just just the value value of IP

56:05 address how would that tell me whether it's typical behavior or not because that is uh one single value mentioned a really good feature uh one lakh on grocery store that is likely something that no one is going to spend on at a grocery store and that could be indicative of fraud so we think about uh how would you engineer a feature like this like you cannot tell a model hey don't spend one lakh on a grocery store the model needs to learn itself and what you need is a feature that you're providing to the model such that the model will automatically be able to tell that no one spends one lakh on a grocery store so think about what are the features that you're going to provide the model in this case uh for

56:51 the model to learn this diar mentions um some pattern like amount should be significant um and hold some mathematical pattern. Yeah. So you're thinking along the right lines u and digging a little deeper into how would you provide this mathematical pattern to the model. [Music] Um that is exactly kind of what we are trying to now think about how what feature are we going to engineer that the model start learning starts learning about what is typical pattern and what is a different pattern isolation forest outliers uh I am not entirely sure about what an isolation forest is so if you can

57:37 elaborate Aush and Gamer uh mentioned outliers Again how would we measure this outlier? So even for outlier detection we do need some features if we typically make a small amount but it's certainly a huge amount that could be a pattern we mentioned and that is exactly uh right but I would ask the question u again what is a feature that will you will use v for the model to know that you typically do small amount purchases frequency of transactions I think higher you are now starting to get into some of the right features that I want you guys to think about. Uh Nishant mentioned amount spent in a day or the

58:24 average amount spent also completely right. So when we are trying to think about a typical behavior, we do need a historical component to it, right? Like we need to know the history of the behavior, history of those transactions and we need to then look at that history and quantify something from that history. And things that we can quantify from that history are uh things like how many transactions do I make daily. So that would be the frequency or how much amount do I spend in a day or how uh that could be the sum of the amount or what is the average amount I spend in a day. So those are now features that are starting to quantify the history of my my transactions as a buyer or a merchant's transaction as a seller that

59:10 is telling the model what typical behavior is and now sort of going into what are these features called. Most of these features are called window aggregation features because we are taking a time window of history and then engineering features on those time windows. And an example of a feature like that would be the average count of transactions at clothing stores in the last 30 days. And if you look at what this window aggregation feature is, it has four dimensions. The first is the aggregate. An aggregate in a feature like this is average. Other aggregate could be um the sum, the percentile, 10th percentile, 90th percentile, u standard

59:58 deviation, median, so on. The second component of the second dimension is entity. In this case, the entity was count of transactions. Uh and other suggestions that were that are in the chat are um the sum the amount of transactions. The third component would be a filter. So in this case the filter is uh clothing stores. Other example of filters can be instead of just counting the all the transactions I only count successful transactions or failed transactions because even if a transaction is not fraudulent sometimes let's say you entered the wrong CVE and that is a fa that will lead to a failed

1:00:43 transaction and if there are a lot of failed transactions you can imagine that could be typical of uh that that could be behavior that could be indicative of fraud. And the final dimension is window. In this case, we looked at the last 30 days of history and that forms our time window. But because we have these four dimensions, the point I want to emphasize here is that this feature space grows really really rapidly. So if I just think about 10 different aggregates, 10 different entities, 10 different filters and 10 different windows that I I'm going to use, we are looking at 10 ^ 4 or 10,000 features that can be created almost instantly. And if we go back to the scale of the pro data that we had, let's

1:01:30 say 1 billion rows or 10 billion rows, then the number of computations that we have to do is also growing exponentially. um n square in the number of transactions times the number of features that we are trying to create. So for 10,000 features we will have to do 10,000 * 1 billions* 1 billion computations and um why it's 1 billion * 1 billion because for each transaction one bill for each transaction in the 1 billion I when I look at the history I look at all the previous transactions so that my the tables that I'm joining I'm doing a self join and that creates n² computations and we were earlier We're talking about that our training data

1:02:17 will have 1 billion rows but just to create features the number of amount of compute is so incredible that we really really cannot do it on a local machine but we also cannot do it on any single machine even if I take a largest node on the cloud with one terabyte of memory I cannot do these computations because um I will need to store all of these transactions in memory and it's one terabyte is still not going to be enough for that. So now I have to look at distributed computing to be able to do these transactions. So now we are really getting into why MLOps becomes really important because MLOps is providing us the tools to be able to work at this scale of problem which is

1:03:02 typically what we would see in the real world. And the first example of MLOps tool uh or system I will talk about is what's called data ETL. Uh data ETL is the paradigm where we are extracting data from different sources first. In this problem, it could be our database which is processing those transactions. That is one data source. It could also be something like CSV files that Visa provides me to tell me that they have posted a transaction or that there was chargeback initiated on a particular transaction. Once we have loaded the data or once we have extracted the data from different data sources then we transform that data. Transforming is

1:03:48 processing the data uh modifying it into a consistent schema. For example, um I could have amounts coming in different local currencies or I could have timestamps coming in different local timestamps and I want to make them consistent. So all the amounts I convert to US dollars for example or all the time stamps I can convert to UTC and now I can uh work with them with with these fields consistently. But transformations are also about creating the features I mentioned. Uh we want to run these computations and these would be transformations that I would do on my data. And finally I want to load this data to be able to access this data later. So this loading can happen in storage like S3 u in in large tables in

1:04:36 data warehouses like snowflake where all this data is stored and uh as a user of this data as an ML engineer I can lo then use the data that is already present and already transformed and use this data for my model training. So ETL is something um as you'll go into work uh you will start hearing about as an important component of where how we ingest data from its sources transform data uh so it can be usable and then load that data into some storage so we can access it later. All right. So we have talked a lot about data. Let's jump into the next

1:05:21 component in this uh in this pipeline which is training. Again I want the audience to participate here. So when we are training for a classification model uh if you could suggest me what would be the right metrics that you would want to calculate during your training uh for this problem. So the problem is we want to identify fraud lend transactions or predict fraud in credit card transactions and we are we now have collected our data. We have engineered our features. We have the label which is chargeback uh or no chargeback which indicates that it is fraud. And now I want to train a model but for model training I need to be able I need to compute some metrics which will tell me

1:06:08 whether one model is better than another model. So, what are some metrics that you would use? Um, again, use the chat. We'll take a couple of minutes um to to gather feedback and then we'll talk about it. I love uh the audience participation. Um uh a lot of really good ideas coming in. Um and yeah, keep keep it going. Um again what I uh asked is what are some metrics that you want to measure uh for this problem of identifying fraudulent transactions um that will be relevant for this problem and that will tell you whether one model is performing better than

1:06:54 another. Yes asks a really good question whether the data is balanced or not. The data is heavily imbalanced in this case. So typically the fraud rates that we see are less than 1% and sometimes even less than 0.1%. So if we have one uh billion transactions we may have only um if my math is right 100,000 or 1 million fraudend transactions out of that. So heavily imbalanced data set. Uh yesesh mentioned apply PCA. PCA would be a good model for training but I'm not sure how we would generate metrics from that or how PCA would be able to tell us um whether one model is performing better than another. So if you could elaborate on that

1:07:43 Yash uh V is mentioning missing documents u excessive number of adjusting entries uh I think you're talking I I think those are maybe some suggestions uh for features that could be used but how would those tell me whether one model is performing better than another and it's totally possible that what I'm seeing is in chat is maybe some delay in uh some feature suggestions that you guys had provided me. So sorry if if that's what's happening here. Um but yeah I let's talk about what are some metrics that you are going to uh use to know during training that one model is performing better than another. All right. So, yes coming in hot with some uh list of metrics. ROC A

1:08:30 curve. Um so that will be ROC uh curve and area under that curve precision recall accuracy. Uh really good suggestions. Uh S mentioning some of the same features and added F1 score also a really good feature. So if you don't know what an F1 score is, an F1 score is the um I believe harmonic mean of precision and recall. Um so that is uh like a single scaler that combines both precision and recall into a single value. So instead of looking at precision separately and recall separately, you can just look at F1 score and that will quickly tell you whether one model is better than another. And AU is doing the same thing. You can look at an ROC curve from

1:09:18 different models but you can imagine that one model it's not necessary that the ROC curve of model 2 is consistently higher than the ROC curve of model one because if it is then it tells you ROC curve that model 2 is doing better. What typically happens in in training is ROC curve of model 2 would intersect at some point with the ROC curve of model one. And when something like that happens, it's not easy to know which model is performing better. So what we can do is take the area under that curve which then converts it into a scaler and tells us which model is performing better. The model with the higher area performs better. Other suggestions coming in to

1:10:03 use confusion matrix uh which will tell us all the uh four quadrants false positive false negative uh true positive and true negative and also a good idea um um as a metric then we mentioned cost based evaluations um that is an amazing suggestion because cost based evaluations I think what you're referring to would be a utility function here uh and a utility function here could be how much money do I make when I approve a transaction versus how much money do I lose when I approve a fraudulled transaction and I can use a utility function like that to determine whether one model is performing better than another. So great suggestions that came in and I want to um so let move forward and I'll mention sort of some of

1:10:51 the things that were already in the chat. So because we're working with imbalanced data sets accuracy is not a good metric. So, I mentioned we could be looking at 0.1% fraud. And if I build a trivial model that always predicts not fraud, then my model already has a 99.9% accuracy. But this model is not useful. And that is why whenever we are working with imbalanced data sets, accuracy should never be used. Like that's a rule of thumb that you can take uh with you through all your career. If your data is imbalanced and you're working on a classification problem, throw accuracy out the door. Other uh suggestions that came up

1:11:36 were precision and recall. Both are really good um metrics to use. Precision is measuring how many of the predicted fraud are in fact fraudulent transactions and recall is measuring um of all the fraud that you have in your training data, how much fraud was this model able to capture. But if you think about precision and recall t they are both conditional on selecting a particular probability threshold. So typically for a balanced data set that probability threshold is set at 0.5. So there's a 50% probability that it could be fraud and a 50% probability that it could not be fraud. And if the that probability is more than 50% then that is the label that you select. So if fraud is more than 50% you say that transaction is fraud. If it's less than

1:12:22 50% you say it's not fraud. But when you have an imbalanced data set, 50% is no longer a good measure. So what is that threshold that you would use? And we don't really know what that threshold should be. Um we can't make it equal to 99.9% because a model is never going to be perfect and that is why we have ROC curves. But in this case, an ROC curve is not a good measure because an ROC curve, if you think about it, its denominator for one of the fields is um true negatives. And because our data is so heavily imbalanced, when we take true negatives as the denominator, the numbers in the numerator stop mattering

1:13:08 because the denominator is so large that variations in the numerator are just going to be lost. So instead what we create is a precision recall curve. So instead of ROC we have precision on the y- axis, recall on the x-axis. Draw this curve and then take the area under this curve which becomes the scaler that we then use to be able to compare whether one model is performing better than another. And when we're thinking about uh training, the other uh thing we want to think about is what are the training resources that we are going to use. So I mentioned the data sets that we are using are really large. We cannot really train them on a uh on our local machines. So we may want to use very

1:13:54 large machines that cloud providers provide for example with a lot of storage and uh with a lot of memory like one terabyte. But sometimes even that is not sufficient. So we want to use distributed training. For example, there are some models XG boost offers distributed training. Uh like GBM offers distributed training. Um and uh there are different libraries that we can use. For example, Pispark uh allows us to do distributed training. And then there is a new framework called Ray which is really popular these days to for doing distributed training. And the final component I'll talk about is experiment tracking. Now let's say there are five people working on training uh and they're all trying out different things maybe different hyperparameters

1:14:40 different models and they're all doing their experiments they all have some results and they want you want to be able to compare those results so how do you collaborate uh on those experiments the easiest way where you would probably start is creating an excel sheet where you write you know what model you used what were some of the metrics that were generated from that model. Maybe you take some notes down about what you learned about this model. Uh and that's where you're going to start. But as the number of models increase, as the number of people increase, it starts getting hard to uh track this Excel sheet, some people may forget something they had done um and uh don't note it down or they copy the wrong metric and now

1:15:26 there's wrong information in this sheet. So instead what we can use are another set of MLOps tools which help us do experiment tracking uh and metrics tracking. So whenever we do an experiment run automatically it will track which are the parameters that we used to train this model what model did we use and what are the metrics that were generated from this model. Um so there are MLOps tools like MLflow which which do this for us. So another great example of where MLOps fits in because as the teams grow as the number of models increase we just are not able to track those experiments easily and we need something to be able to do that. All right so that wraps up the discussion on training and

1:16:12 experimentation. The next thing I want to jump into is model serving. So you trained a model let's say on your local machine or on a cluster. Now how would users access this train model? They cannot access your local machine and similarly uh they cannot access the cluster maybe where you train the model on and in fact you may have trained the model on a really large node or a really large cluster with a lot of machines but now when you're serving that model or when you're making predictions from a trained model you don't need that large of a machine. So how would users access? So the first thing you do is package your train model. And what

1:16:58 packaging means is you take in um all the components of this model. For example, what kind of model it is. What are its dependencies? For example, if you use scikitlearn or numpy, those will be dependencies of this model. And what are the train weights? So if you have a logistic regression model, what are the weights that you use? If you have a random forest model, what are the different decision trees in this model? And you'll you package them all together. You'll basically store them in in a single file. So these are can be docker files which are deployed. If you're not familiar with docker, docker basically will take in all the dependencies like numpy and scikitlearn and tell the machine that these are the dependencies you need to install to be able to make a prediction from this

1:17:43 model. when we are deploying this model or serving this model to the users, we want to consider the model size, latency, and the throughput. So model size is how large this model is. So if it's a logistic regression model, it will be really small. If you have 10,000 features, for example, you'll just have 10,000 weights that you need to use with 10,000 feature names. Very very small model. Uh but if it's a neural network model with a lot of hidden layers, that can be really large. So if any of you have used for example chat GPT you have to wait a long time to get get a response and long time so 5 seconds is considered can be considered a really long time uh when it you're serving your users and in a problem like this when we

1:18:29 are trying to predict fraud you know as soon as you tap your credit card you want to know the answer you don't want to be standing there waiting for 10 seconds to know whether that transaction is approved or not so the latency requirement ment is really strict in a few milliseconds that you need to be able to make this decision. And the final thing is throughput. Throughput is how many transactions can you make a prediction at any given time. If there are 10,000 people trying to tap their credit card all at the same time, the model needs to be able to process all those 10,000 transactions within the latency requirements that you have set up. So for a problem like this depending on your model size if we are using a decision tree model for example or logistic regression model we may want to package it in the memory of the same

1:19:16 node in which the service is running. So the service here is let's say checkout where as soon as you tap the credit card or make a purchase online you are seeing um some there is some node that is processing that checkout and so you package that model in the same node because it's a small size model and so no data needs to be sent somewhere else but typically as your model size grows the packaging is done at an endpoint where let's say you would set up a different pod a different node somewhere and this node which is processing checkout will say these are all the features of the transaction that is being made for example what is the amount what is the card holder ID uh

1:20:01 what is the merchant name um is it a grocery store or not etc and I'm sending all these features to this other node which is doing which has my mo model in it and this model is now going to make a prediction within this node and then through an API send the result of this back to this node. So there is network traffic that is now going in. I'm accessing the model prediction through an API and this network traffic adds to the latency. But the nice thing about this is when there's a lot of people trying to access this my model, all I need to do is replicate this model onto multiple nodes. So I can scale up and down this uh node to serve to to increase or decrease

1:20:50 my throughput. Whereas if my model was within this node itself then I depending on the throughput I may need to increase or decrease the entire service nodes because uh there's a lot of transactions that are now coming in and there could be more overhead here. All right. So um that is what model serving is. I think uh we are coming close to time. So there was another question I wanted to ask about when we think about model serving there are both online and offline features. So in uh if you have ideas about what those online and offline features would be uh write them in chat but I'm also going to move on and answer that for you um right away.

1:21:37 So the online features are when you tap your credit card what is the amount what is the hour of the day that uh this transaction is happening in etc. So these are online features because you only know about these features at the time that the credit card is tapped. Then there are offline features for example the features that we had engineered like average amount spent in the last 10 days. I cannot compute this on on fly because we looked at how compute heavy these computations can be and if we want because we want to maintain low latency if we wanted to calculate this on the fly we will never get a response from the machine whether this transaction is fraudent or not. So these are examples of offline features

1:22:23 that we actually store these features somewhere in our data set and then when this model wants to do its prediction, it will do a lookup in this data set and grab this feature that has already been premputed. So that's an example of an offline feature. And then there are online transformations. For example, if we want to take the log of an amount, then that is that is something that we are calculating on the fly because that feature is only available online. Similarly, if we want to calculate the distance between a buyer and the merchant that is conditional on the current transaction that is happening and we want to do this computation or this transformation online as well. So these are things that we have to think about when we are serving the

1:23:10 model to our users. how we how we are going to package the model, where the model is going to run, and how it's going to be able to access both the online and the offline features in a way that meets our latency and throughput requirements. The final component of this pipeline is performance monitoring. So, how do we measure the model's performance now on online data? So, we trained our model. It was doing really well on our test data set but we still want to know now that the model is provided to the users or served to the users how is it pro performing in the real world with data it has never seen. Uh so first of all just from the serving

1:23:56 infrastructure we may want to log the latency and the throughput and to make sure that it our serve model is meeting those requirements and if we need to scale up and down what are our machines that are serving the model we can do that based on latency and throughput of the model in the real world. Then we will look at chargebacks because those are our fraud indicators to know how well the model is performing with online data. But chargeback is a lagging indicator. I mentioned it can take 30 to 90 days to receive a chargeback. So we deploy our model and we may have to wait 30 days or 90 days to know how well the model is performing. And if we launched a model that was crappy for 30 days, the users will work

1:24:44 with a crappy model and there would be some fraud actor who would come in realize that we have a crappy model and they are going to commit a lot of fraud. So we need to figure out ways to gather some of those signals really quickly without having to wait that long. And what we could do in this case is just look at the chargebacks that are coming in more rapidly like within seven days and uh use that as an indicator for what how well our model would be performing in the future. And finally uh when we're talking about uh performance fraud actors change their patterns and fraud behaviors uh also evolve and so we expect that the model performance will deteriorate over time and we want to be able to measure that. So the way we do

1:25:31 it is through two different um uh they're called two different there are two different ways in which the performance can deteriorate. The first is called data drift where the distribution of the features themselves changed. For example, when COVID occurred and everyone was locked down maybe the number so the number of transactions happening online increased a lot more than the number of transactions that were happening in person. So the distribution of the features themselves are going to change. If we had a feature like the number of on typical online transactions by a buyer, those numbers are going to look very different. um before and after COVID. So we want to be able to track these differences because differences in the data distribution will tell us that

1:26:18 the model may not perform as well because it has learned on a different distribution than what we are seeing right now. And the other kind of drift is concept drift where instead of the data distribution itself changing the way that the data is indicative of fraud changes. So um in this case let's say we had a feature like number of credit cards uh per buyer. So we have buyer identified as an email address let's say and that person has two credit cards and we have that as a feature. Um and we know that if a buyer has a lot of credit cards then um that could be indicative of fraud and let's say one of the decision trees learned that if a

1:27:03 buyer has more than 10 credit cards then that's fraudland. But now with Apple Pay, Google Pay, Samsung Pay, each device essentially gets an its own credit card. So when you add your credit card on your phone or on your watch, your phone has a different credit card number and your watch has a different credit card number. And what the model sees is going to be different credit cards num numbers. So now the number of credit cards that people have has just increased so much that 10 is not a good number anymore to identify fraud. So the distribution of um what a feature value means for indicating fraud itself has changed. So that is called concept drift. And when we are doing performance is something that uh we want

1:27:50 to track both data drift and concept drift because that will tell us that it is time to retrain the model. And the final slide uh I have for this uh map is continuous integration and delivery. So you may have heard of these terms in DevOps and software engineering that what we want to be able to do is just add new data, add new features, uh change your change my model, uh serve that model to users and we want to do it continuously. We don't want to wait for someone to approve a feature or approve a model um because that adds time especially when we have could have thousands of models that are deployed. And so we want to add things like data schema testing. We don't want that someone had created a boolean

1:28:35 feature like 01 um for example chargeback and instead changed them to true false and now the data schema has broken and all our pipelines are going to break. Similarly, feature definitions can change. So when I looked at the last 30 days of transaction, I was looking at the current transaction time. And so if it happened at 2 p.m., I would go 30 days back from 2 p.m. Instead, someone changed the definition and say that I'm only going to look at calendar days. For example, even though the transaction happened at 2 p.m. on June 1st, I'm only going up to I'm going up to May 1st to May 31st or May 2nd to May 31st, which are my calendar days. And the definition changes and there's no

1:29:22 sometimes no way to track that the definition has changed. So, we want when we're doing CI/CD, we want to make sure that these things are not changing. And when we are adding or removing features, it can become impossible to then compare the performance of the new model with respect to the old model because maybe a feature that was that we were using to make predictions in the previous model or the old model doesn't exist anymore. And as a result, if I want to compare these two models, uh, one trained today, another train maybe three months ago, there I I I don't have access to the feature for this model. So I cannot do a comparison. and that can be a problem. So we have to think about that we don't retire features too early because we may need to have use those

1:30:08 features um in the future to do these performance comparisons. So that is we we do that you by using data versioning some data atl u

1:31:42 Maybe there is no sound. Um Nishant uh do you want to look into this? Oh yes sir. Just a second. Um while uh Nishant is looking into this uh can someone tell me what was the last thing you heard? Uh and I can quickly uh go through actually you cannot hear me so that's not possible. Oh sorry I think it's fixed right now. Okay. uh give me uh let me know in chat. All

1:32:30 right, it's fine. Now, what was the last thing you heard so I can quickly go through uh the rest of uh this slide? Uh Nishan, do you know what was the last thing? Uh sir, actually it was audit audible to me because I was in the Okay, fair enough. Uh sir the chat is saying you should restart from feature definition and data versioning. Okay so we talked about data versioning that we want to version our data. So uh we can go back revert back to the old

1:33:16 features that we might have created that are now removed from the uh new models because we want to be able to compare the performance of the new model that we have trained today versus a model that we may have trained 3 months ago. Uh similarly we want to do model versioning uh because we could have multiple models that are available to the users or we may want to run AB tests or control and treatment tests where we are saying that one set of users uh get model version X and another set of users get model version Y and we want to see which you are engaged the platform because if one model causes users to not do as many checkouts then that's a we want to be able to track. Similarly, when we are

1:34:01 publishing a new model, we may want to do tapered activation where we first serve the model to 1% of the users, then increase it to 2% and so on. Because if a model has higher latency um or poorer performance, it wouldn't affect 100% of the users if we were to activate that model immediately for everyone. And in these cases, we may want to revert back to a previous version. So if we didn't have any version definitions for our models, we wouldn't be able to do any of those things. So model versioning is also important. And finally, we talked about performance monitoring and in performance monitoring, if we notice that there is concept drift or data drift which has deteriorated model performance, we want to train the models. Initially that training could be manual. But as the number of model

1:34:47 increase to let's say thousands of models we could set up autotraining pipelines where the autotraining automatically occurs let's say weekly and this autotraining the metrics that we have established for our model performance are calculated automatically and if there is improvement in those metrics from autotrain models the model gets automatically deployed. So this is some CI/CD pipelines that we put in into our model training as the team grows as the number of models grow to be able to do this more efficiently. All right. So uh I talked a lot today about u why ML ops is important and that ML code is just a small portion of an ML systems pipeline.

1:35:33 And on this slide I'm just mentioning some of the tools that are commonly used in industry. to uh as part of this MLOps system. So when we talk about data ETL um there could be different data sources for example CSV files, databases or streaming events like when you open a web page um or and when you add something to the card those could be streaming events. Then we have to do feature engineering which can be really compute heavy and there could be a lot of storage requirements. So we use things like data warehouses or lakehouses which are provided by companies like snowflake or data bricks and then we want to be able to do this future computation every day or every few hours because we wanted as possible. So we would use

1:36:20 orchestration tool like airflow which will do these computations at a fixed uh frequency that we have established let's say every day. Then for experiment tracking, I mentioned we want to be able to run different experiments and to be able to track the metrics from each experiment. And some of the tools that we use are like ML flow, weights and biases, cube flow, all of these allow us to track our experiments. And some of these tools also help us uh do model packaging. So once the model is trained and it meets our criteria, then these we can package the model through within these tools themselves. Um if the model if the data size is really large or the model size is really large we want to do distributed training uh which can be

1:37:05 done through tools like Amazon SageMaker or raise another distributed training paradigm that I mentioned and another one would be pi spark for model serving I'd mentioned docker docker is a way to package all your dependencies into a single image and this image can then be deployed on any kind of machine that we want and it's really popular so if um you want to um you uplevel yourself uh for the job market. Definitely it's a really good idea to put Docker on your resume. ONX is a way to package model. It's typically used for neural networks. Kubernetes is u pod orchestration. So similar to docker, docker images only get deployed on a single machine whereas kubernetus deploys multiple docker

1:37:50 images on a set of machines all of which can talk to each other. So these are very popular tools for managing cloud infrastructure. Uh AWS Lambda is another way uh to deploy models. This makes it really easy uh instead of having to set up Kubernetes uh yourself which can be a little challenging. And fast API is a way to access your model prediction. So I mentioned if this is the service that is using your model results and this is where the model is deployed fast API is a way to uh get the results from the u model prediction part. And finally for monitoring you can use something as simple as a Jupyter lab notebook which where you can um gather the recent online data and calculate precision

1:38:36 recall or precision recall curve area under that curve and so on to know how well the online performance of the monitor compares with the offline performance. For latency and throughput, you may want to use logging tools like graphana or data dog and arise uh and some other companies provide model monitoring tools which help you measure concept and data drift. So these are just some examples of different ML ops tools. This this is a really really uh broad field where a lot of companies are providing trying to meet these uh requirements of data ETL experiment tracking serving and monitoring and some things like Amazon SageMaker or Google Vortex AI try to do all of these together in my personal opinion not well

1:39:23 uh but uh and and these dedicated tools tend to perform better but these are just some examples and as you're thinking about upskilling yourself for the job market some of These tools are uh what you can look into to add to your resume uh to make you more attractive to companies. And uh we talked a lot, we spent a lot of time today. So if uh I want to if you didn't learn anything else, just focus on this slide. Uh these are the three main uh takeaways. ML models. Um, that's what you're going to learn throughout this course, but they are just a small part of an ML system. Uh, a lot of the ML system is actually unseen and most of your time in industry is not going to be spent on the ML

1:40:08 model. It's actually going to be spent on the system that supports that ML model. MLOps is the infrastructure and the processes that helps you turn your prototype or proof of concept ML model into reliable production systems. And uh when you're starting to work in industry, asking questions early about what is the scale of this model uh what is the latency requirement or the throughput requirements will help you be successful and will really differentiate you from other people uh because without knowing the answers to this all you'll have is an ML model which no one will be able to use except you um and that's not going to be useful. So ask these

1:40:55 questions early and you'll you're going to be really successful as ML engineers in in your in your career and industry. All right, that's all I have. So uh we'll open it up to question and answers and thank you everyone in the audience for being um such great participants and answering all the questions and your messages. I really really appreciate it. So now opening it up to questions everyone. Um yeah so Nishant is there something else you would like me to talk about while we are taking questions or should we just wait for audience questions? Um I'll maybe hand it off to you for the

1:41:41 next steps. Uh sir Rashab is asking how features were integrated and uh why we can't compare old models with new models. So okay so both are good questions feature integration um are you dish asking about um once a model is served what I mentioned about online versus offline features. So can you elaborate on that a little bit? Um I'll uh answer your second question which is um why we couldn't compare previous model with the new model. So that is about data versioning that I had mentioned. Um so let's say I have a feature like number of credit card per buyer um that I was

1:42:29 using in my old model. Uh this model was trained three months ago. I was calculating this feature uh number of credit cards per buyer and my previous model was trained using this feature. Then when I started training my new model now this is 3 months later I realized because of the example I gave that number of credit cards is no longer an important feature and I removed this feature from this model. So because I have removed this feature from this model, uh I'm not calculating the value of this feature anymore. This model that was trained 3 months ago, its test data was also 3 months old. Right? This model that I'm

1:43:15 training today, its test data is now more recent. So I cannot compare the performance of this model and this model

…