Full transcript of “How Modern Data Teams Work (Engineer, Analyst, Scientist, Architect, ML)”. Click a timestamp to open the video at that moment. Need another video? Use the YouTube transcript generator.
0:00 All right. Now, first I would like you to understand the role of the data analyst. So what is a data analyst? Well, it is very simple. You answer the business questions using data. Think about yourself as a bridge between the raw data and the real business decisions. So now I can say that I have joined like around seven data teams and projects. And now I would like you to understand exactly what data analysts do in real companies. So now it is a story time. Let's go. Okay. So now in companies in business we have managers, stakeholders, project leads and their whole job is to make smart decisions by asking critical questions. So for example, which region is underperforming? Why are profits down and should we invest here? So there are many challenges in the business and they
0:46 need quick and smart decisions. So now without using data they are just guessing. They rely on opinions, gut feelings and many outdated informations left and right. And there is famous quote from William Deming I really like and use a lot without data you are just another person with an opinion. So that means if your decision process is based on opinions this has high chance to lead to confusion wasted resources and making bad decisions for the business. I think that pretty much sums it up. So now instead of guessing they have to decide based on data. So now in companies the data are scattered everywhere. Your data is stored in many different databases. There are a lot of spreadsheets and maybe as well APIs that
1:31 provides data. So they are everywhere and the managers of course don't have the time or the skills in order to dive into all those places. So that's why they go and hire an expert specialist. They hire the data analyst. So think about it like you are a detective and you start gathering data from different sources. So you go and query the different databases and as well pull the data from the spreadsheets and the different APIs. So you have to go and get the data wherever they lives. So of course all those data are messy and you have to go and clean and structure those data. So maybe you're going to go and put the data everything in one spreadsheet like in Excel and you start doing a lot of stuff on the data cleaning up the data organizing it doing a lot of calculations transformations to get the data ready to answer the business question. Now after you find
2:17 something meaningful from the data you start turning the data into insights into a visual reports because it is way easier to communicate the result to the stakeholders and managers using visuals. So that means once you are ready you go to the managers and you start presenting the result as a story using the report and visuals that you have prepared. Now the managers got facts and this time they have answers to their questions using data. So now they are more confident and they make better and smarter decisions for the business. So now you can see clearly what the data analyst does. He or she is the bridge between the raw data on the left side and the business on the right side and all what he is doing is answering the critical business questions using the raw data and as well it is very clear
3:03 which skills this analyst needs. So as we understood first you will be querying the databases in order to pull the data and in order to talk to the data and the databases you use SQL or sometimes we call it SQL and we have understood as well once you collect all the data you will be using spreadsheet like Excel in order to clean sort filter and do calculations on the data. So they have to master Excel and after analyzing the data using the spreadsheet you need data visualization skills using tools like maybe PowerBI or Tableau to create charts and clear visuals to be reported to the managers. And of course one very important skill and crucial. You must clearly understand the business questions from the managers and communicate the results and the findings very effectively. So you have to master
3:50 this skill. You have to tell a story behind the data. So you need really very good communication skills. So that means if you master those skills, you will be an amazing data analyst that's going to help the company and the managers making the smart decisions. And now my friends, we have an issue. This might work for a small company, but it will not work for modern big companies. And here's why. Now big companies generates big data and it's going to be really hard for the data analyst to manually extract the data for analyzers because you're going to need a lot of data and your request going to take like hours and even days to get the data and it might crash and it going to be sometimes impossible to get the data. So it's going to be for you as a data analyst really hard to answer any questions and your process going to take weeks until you give any
4:36 answers and of course managers cannot wait for weeks in order to get like an answer. So my friend this approach will not work. This is simply does not scale. So now we have to go and scale things and hire new people. Now a data architect it's like an architect of a building. You have to design the blueprint of a scalable data system and you have to go and organize the data into multiple layers. Like for example the medelian architecture you have a bronze layer for the raw data and a server layer for a clean data and a gold and very important layer for clean structured and optimized data. So all what the data architect is doing is designing this scalable data system. So now as you can see the data architect is designing this scalable data system but as usual they are not the one that is building it. For that we need another
5:22 experts and we call them the data engineers. So now as a data engineer you're going to go and build something called data pipelines. Well, it's all about to connect to multiple source systems and start automatically moving the data from the sources to the new data system. And as well, very important, the data must move quickly. So the pipeline is going to bring the data to the first layer to the pro layer as a raw data. Nothing fancy going to happen here. Then the data engineer going to move the data to the server layer and there the data get cleaned, structured and prepared for the final layer. Now in the final layer, the data engineer going to go and build a data model which is highly organized and optimized model that is perfectly made for quick analyzes. And we have with that a process that runs automatically
6:08 from the source systems until the final layer. And this runs every day very fast and in some scenarios as a stream like a real life data. So remember now our data analyst friend. So now, thanks to the data architect and the data engineer, analysts life now is way easier. No more nightmare of pulling the data manually from the source system. No need anymore for the spreadsheets and the Excel lists. Now everything is prepared for the data analyst to be fast. Hell yeah. Hey, come on, baby. Come on. Yes. Come on. So now again as a data analyst once you get a question from the business you can go quickly and straight away to the code layer where everything is prepared and now using SQL you will be querying the data model in the code layer in order to
6:54 explore and try to find answers for the question. So once you find it, you will go and build again like a visual reports and tell a story for the managers. And with this setup, you only focus on what matters when finding answers for the business. And this can speed the whole process for you and you will be ready to answer quickly the questions. Now I'm really sorry we are not done yet. There is still an issue in this story. Now all what you're doing here as a data analyst is that you are answering ad hoc questions and your answers are always like one-time reports. But what can happen for sure is that one or few reports going to deliver really an incredible value and it is not just useful for once and only for the managers. Many people across the company going to be interested in these reports.
7:39 So you're going to have a new user groups that going to ask you one question. I want your reports every day. No. God, please no. No. And the nightmare going to come back where you have every day to run the same query and generate the same reports and you're going to daily send this reports to the users and this going to be total waste of your time and going to build huge stress on you and that's why companies going to bring another person to the data team and this time we're going to have the business intelligence developer or BI developer. So once the BI developer joins you as a data analyst going to say hey go and automate this report and make it visible for many users. So first the BI developer going to build an environment like a server where the dashboards and reports going
8:26 to live. This must be secure and has an access management. Sometimes you cannot give all the data for everyone in the company. And as well this server must be accessible 24/7. So the report and dashboard must be always available. So once the reporting server is there the BI developer going to take everything the report the logic and build from it something called interactive dashboard and what is very important this dashboard going to be automatically connected to the gold layer to the data model so that every day and this is important the dashboard and the reports going to get fresh data from the data system from the gold layer and once everything is live the user is going to go and start requesting access to the reports and start accessing the dashboards and the data everything is highly automated and scalable. So the
9:12 data going to flow from the sources into the different layers of our data system and as well automatically flowing to the reporting server and refreshing the data there. So this is the big differences between the data analyst and the BI developer. The data analyst is the one that is bringing the first report and insights and once a visual or a report gets very important. The BI developer is the one that is responsible of making this report accessible, scalable and everyday available for many users. Now, so far we have held the business and the managers with questions that are about the past and the present. What happened? How much did we sell? Why did sale drop? But at some point the managers going to ask bigger questions. Questions about the future. What will happen next month?
9:59 Can we predict which customers might leave us? What can happen if we change our pricing strategy? So, they are advanced questions about the future. And to be honest, the data analysts don't have the tools in order to answer them. And for that, we need another expert called the data scientist. This is Sheldor the Conqueror. We are about to enter Axel's fortress. The data scientist is the one that going to use the data in order to build and train models. They're going to do a lot of experiments. So at the end the scientists going to use this model in order to give answers for the future. So with that the managers are not just informed they are as well prepared for the future. They are proactively making decisions. They stay ahead of the competition and as well they tackle challenges before they even appear. Now
10:44 this type of work is very important thing not only for the managers and projects leads. This is important for everyone else. Now this model this work from the data scientist must be visible for other users and this is the same thing that happens before right with the BI developer but now we need another type we have the machine learning developer ML developer. So now what they going to do they're going to as well prepare a server and environment to run and deploy the model from the data scientist. So they're going to go and put it in productive environments and everything going to be now automatically secure scalable and available 24/7. And the result of this model could be presented in like an application to show the result of the productions for everyone or as well sometimes they show it in dashboards and reports. So their
11:30 job is to make the result of this model available in many services in the company and accessible by other users and build by the ML engineer. And with that everyone from the managers to teams across the company benefits from advanced predictions every day and anytime. So now by looking to this everything is highly automated. The data kind of flow from the sources to the data system and then flows to the reports and dashboards and as well automatically flow to our model that is trained from the data analyst and built from the ML engineer. So this is what a modern big company build. They build scalable data system. Now if you look to this you can see there is a pattern. You can see the data architect is working closely to the data engineer to build the data system. The data analyst works as well closely to the BI developer in
12:17 order to build this reporting system and the data scientist together with the ML engineer. They work on building an advanced analytical system. And there is actually another pattern. You can see in those roles there is always somebody is like doing a design like the data architect is like designing the data system and the data analyst is like building the first version of the reports. Same thing for the data scientist. is the one that is experimenting and training the first version of the model. So those one are always like trying to discover, trying to put the blueprint. And now in the other side we have the other category. You have the persons that are building it, bringing it to a productive, scalable environment and making things live. Like the data engineer is the one that builds the data pipelines and this
13:03 data system. The BI developer is bringing the reports and dashboards into scalable and productive reporting system. And the machine learning engineer is the one that is bringing the model and deploy it into a productive system and offering the results in different services. So they are the engineers, they are the developers. So this is what a real modern data team looks like. All right my friends, one more last thing. If you really enjoy these type of tutorials where it's free and as well I sketch those complex concepts and you want more content like this then support the channel by subscribing, liking, commenting. This can really help with the algorithm and going to support the channel by reaching others like you and as well can help me to make more content like this. And of course, thank you so much for watching and I will see you in the next video.
