Azure Data Engineer Roadmap with Free Resources!!

0:00 why is everybody suddenly trying to become a data engineer a few years back there was no role called as data engineer companies used to hire different people for different kinds of role such as scafa developer spark developer Haro developer but as data grew bigger and bigger companies needed one person to handle everything that's how data engineering came into the existence and it is one of the most trending role right now and here is the crazy part job openings for data Engineers have significantly increased by 300% in the last few years companies are paying salaries between 20 LPA to 70 LPA to the experienced data professional and these salaries are expected to go even higher because companies need data to make better decisions this is solved by data engineer so if you are here for hyping job and futuristic job then this video is for you he everyone my name is sh wer and I a data engineer I have started my data engineering journey in 2021 but my life completely changed when

0:47 I started writing and sharing my knowledge on LinkedIn I have started my LinkedIn journey in January 2023 and within the one year I became one of the most followed data engineering professional with 200k followers now I want to take the journey to the YouTube share the best possible content for you to transition into Data engineering when we talk about data engineering there are basically two kinds of role one is a big data developer role and another one is the cloud data engineer role in Big Data developer role we basically focus on the old Technologies like Haru P EDB cavka airflow and Spark and in cloud data engineering role we basically focus on the cloud side which are AWS Azure or gcp in this video we are going to talk about azur data engineering so why are we going ahead with the cloud data engineering because most of the companies are trying to migrate their on premises data to the cloud side and

1:32 because of that reason they need a person who is proficient in the cloud side and one of the most used cloud in the market is azure now let's talk about how you can become aure data engineer for this I have created one PDF where I mention all the skills and best possible resources available on the internet for you I will be attaching the link for this PDF in the description section this is going to be my first ever YouTube video so please do subscribe to my YouTube channel and I will try to upload best possible content for data engineering so let's first start with the prerequisites if you want to get into Data engineering the first skill you need to master is SQL it is the foundation of everything data is stored in databases and data warehouses and SQL is the language to work with it so here is how you can Master the SQL step by step first step is to learn the basics

2:17 for Basics watch SQL tutorial for Everyone by Sumit s it has around 16 videos covering all the key Concepts second step is to learn the advanced SQL concepts for advanced SQL go through the playlist Again by Sumit sir it has around two videos but he have explained the topics in a very simple way now the third step is to practice the SQL questions for practicing the questions go through the anit bunel YouTube channel we have one playlist complex SQL questions for interview preparation and it has around 80 questions solved this question will help you in building the logic after going through this three playlist practice the SQL with real data sets I have compiled a list of 20 platforms where they have around 1,000 plus questions in total even if you cover 20 to 30 % of it trust me you will be able to crack any skq interview for

3:03 data engineering once you are comfortable with SQL the next thing that you should learn is a programming language you have three choices in programming language one is Java second one is Scala and third one is python you can easily ignore Java and Scala because most of the companies are moving from all of these languages to the python mainly because python is like a king of data engineering and it has so many packages and libraries available if you are just getting started I will urge you to learn the python so for python I would like to share only four resources first learn the python fundamentals this is an amazing playlist by Manish Kumar having 30 videos then you should learn data manipulation libraries those are pandas numai and matplot lib all of these playlist are covered by code Basics YouTube channel and you are done with

3:52 python now the third thing is data warehousing and data modeling a data warehouse is like a big storage room where a company keeps all data in an organized way this helps the businesses look at past Trends and make a better decisions for example imagine a company collecting data from different places like sales customers websites and apps if this data is scattered it's very hard to use a data warehouse brings everything together in one place and makes it easier to analyze now what exactly is data modeling data modeling is like creating a blueprint or how data will be stored and organized in a database or a data warehouse it helps in structuring the data so it's easy to access and use to learn this go through 20 video series by Manish Kumar secondly go through these five based articles on

4:37 cracking the data modeling interviews topics covered in both the resources are entities and attributes primary key and foreign key normalization denormalization star and snowflex schema data relationships indexing slowly changing Dimension fact and dimension tables so basically the fourth thing is Linux this video covers all the Linux command required for data engineering so just go through this video and Linux is done so after prerequisites we need to start with ppar it is one of the most important Big Data technology and it is a combination of SQL and python as a data engineer we need to process the terabytes of data efficiently and that is done by ppar very basic question we get is that what is the difference

5:23 between SQL and ppar so SQL runs on a traditional database and works well for small to medium data sets but it struggles with massive data on the other hand ppar distributes the competitions across multiple machines making it much faster for big data processing so this is the playlist that will cover the fundamental understanding of spark and for interview preparation go through this 40 videos so now let's talk about AER cloud in each Cloud we have thousands of services but we just need to focus on the services that we require as a data engineer the first one is Ober data Factory it is used for ETL and to orchestrate and automate data movement between different sources and destination second one is data bricks it is used for large scale data processing using spark the third one is AER synapse

6:10 analytics it is a data warehouse then fourth one is a data storage or AER blob storage they are used for scalable storage of structured and unstructured data then the fifth one is Aur stream analytics they are used for realtime analytics on streaming data then aure Cosmos DB it is a no SQL database then seventh one is HD insight a managed big data analytics service supporting Hardo Park Hau Kaa or large scale data processing Additionally you can also learn Azure logic apps Azure function and Microsoft fabric these many services are more than enough to go and start the Azure Cloud so now let's talk about the resources for theer cloud so for ADF we can go through this playlist by geek coder it has around 42 videos covering ADF from Basics to advance for data

6:55 breaks this Udu course is best for synapse analytics this playlist is good enough to cover all the concepts and for rest of the services again go through this U course which comes at a very minimal price of 500 rupees so this covers your resources for Azure Cloud so the next topic is git as a data engineer you just don't work with data but you also write lot of code for data pipelines transformation and automation but how do you manage that code track changes and ensure everything runs smoothly in production that's where git gitlab and CD comes in so for better understanding of git and CSD go through these two YouTube videos first one is complete git and GitHub tutorial for beginners and second one is gitlab cscd tutorial for

7:41 beginners so once you have completed all of the learnings as I told then the next thing I suggest you to do is understand how to combine all of the services and build a final project and for that I have compiled a list of three end to end projects available on YouTube these projects will give you an understanding of various services like data factory data brick apps analytics ADLs with that you will also get an understanding of SQL python P spark and this is how you will build a final project I will highly encourage you to do these projects after completing all of the playlist as we discussed so I always suggest not to just run behind the theories and the courses and all of the other things after having understanding of technical Concepts the first thing that you need to do is go on to the portal. a.com and create your free account AER basically

8:28 provides $200 credit for free and 12 month of free trial on some of the services so the first thing I want you to do is to create your account and you won't get charge for any of these things and once you do that you can easily access the Azure Cloud portal and start understanding about different Services provided by Azure this first step is very important because you will get used to the UF aure cloud and you will start understanding different services and how they look like and how they are integrated with each other so go to AER portal create your account start exploring and start doing projects at this stage you will be done with the theory learnings projects and practical exposure to the environment now the next thing is interview preparation for this I have prepared data engineering interview experiences for 100 plus companies to make this

9:14 product I have conducted thorough research across various platforms to gather authentic and reliable interview experiences I personally connected with the data Engineers who recently switched the companies to capture the genuine and firsthand insights about their interview processes so here I'm providing detail info on all the interview rounds for each company like screening round technical round system Design Round behavioral round HR round and the types of questions included are thetical scenario based coding and system Design This experiences will give you an idea of what is exact interview processes in those 100 plus companies and types of questions asked so the link for this product I will attach in the description section and you can purchase this product from from there only now final

10:00 step is doing the certifications after completing the learnings and projects you can easily apply for the certification exam if you want to validate your knowledge and stand out from the crowd when applying for jobs you can get the certification done from Microsoft that is aure data engineering associate dp23 having this certification on your resume makes you stand out and validates your knowledge as an Azure data engineer when a recruiter sees that you have obtained the certification the chances of getting the calls for interviews increases trust me if you follow these 12 steps you will become a full-fledged data engineer in next 4 to 5 month next I will be uploading a video on how to get 30 to 40 calls directly from the recruiter using no and Linkedin so please do subscribe to my YouTube channel for the same and bye-bye