0:00 Hey everyone! Today we're diving into the book AI Engineering by Chip Nguyen. 800 pages of really great content about this in-demand field that's offering salaries of $300,000 or more. In this video, I'm summarizing everything from the book to help you get a high-level overview of the field. We'll talk about foundation models, prompt engineering, RAG, fine-tuning, agents, how to build a system, improving inference, and more. I also want to mention, this is a super high-level overview of a very detailed technical book. Don't expect to learn all the details just from watching this video. I really recommend using this as a way to get an overview of what the field looks like and use it as a jumping off point for your own research and exploration. So what exactly is AI engineering and how is it different from traditional machine learning? Let's break it down. AI engineering has exploded recently for two simple reasons. AI models have gotten dramatically better at solving
0:47 real problems while the barrier to building with them has gotten much lower. this perfect storm has created one of the fastest growing engineering disciplines today. At its core, AI engineering is about building applications on top of foundation models, those massive AI systems trained by companies like OpenAI or Google. Unlike traditional machine learning engineers who build models from scratch, AI engineers leverage existing ones, focusing less on training and more on adaptation. These foundation models work through a process called self-supervision. Instead of requiring humans to painstakingly label data, these models can learn by predicting parts of their input data, This breakthrough solved the data labeling bottleneck that held back AI for years. As these models scaled up with more data and computing power, they evolved from simple language models to what we now call large language models or LLMs, and they didn't stop there. They've expanded to
1:33 handle multiple types of data, including images and video, often becoming large multimodal models. Nowadays, we're seeing foundation models power everything from coding assistants like GitHub Copilot to image generation tools, writing aids, customer support bots, and sophisticated data analysis systems. Now that we've covered what AI engineering is, is, let's dig deeper into foundation models themselves, how they're trained, how they work, and why understanding their architecture matters for AI engineers. Foundation models at their core can only know what they've been trained on. This might seem obvious, but it has profound implications. If a model hasn't seen examples of a specific language or concept during training, it simply won't have that knowledge. Most large foundation models are trained on web-crawled data, which brings some inherent problems. This data often contains clickbait, misinformation, toxic content, and fake news. To combat this, teams use various filtering techniques. For instance, OpenAI
2:22 only used Reddit links with at least three upvotes when training GPT-2. The language distribution in training data is also heavily skewed. About half of all crawl data is in English, which means languages with millions of speakers are often underrepresented. This is why specialized models for specific languages and domains are becoming increasingly important. Also, the distribution of domains in one of the main training datasets leans heavily towards business, tech, news, and art. In terms of model architecture, most foundation models use transformer architectures based on the attention mechanism. But to understand why transformers were such a breakthrough, we need to look at what came before. Transformers were invented to solve the problems of sequence-to-sequence models, which used recurrent neural networks for tasks like translation. These had two main components, an encoder that processes inputs and a decoder that generates outputs. Both worked sequentially, token by token.
3:11 The problem is that the decoder only has access to a compressed representation of the entire input. Imagine trying to answer detailed questions about a book when all you have is a brief summary. Also, input processing and output generation are done sequentially, so it's slow for long sequences. Transformers solved this with the attention mechanism, which allows the model to weigh the importance of different input tokens when generating each output token. It's like being able to reference any page in the book while answering questions. Plus, transformers can process input tokens in parallel, making them much faster. During inference, transformers work in two steps. First, pre-fill. Process all the input tokens in parallel to create the intermediate state. And second, decode. generate one output token at a time. The attention mechanism uses three types of vectors. First, query vectors. These represent what information the model is looking for.
4:00 Next, key vectors, like indices of previous tokens. And finally, value vectors, the actual content of the previous tokens. The model computes how much attention to give each input token by comparing the Q and K vectors. A high similarity score means that the token's content, V, will heavily influence the output. This is why longer context windows are computationally expensive. more tokens mean more K and V vectors to compute and store. Attention is almost always multi-headed, allowing the model to focus on different groups of tokens simultaneously. In LAMA 2.7b, there are 32 attention heads, for example. A complete transformer consists of multiple transformer blocks, each containing an attention module and a neural network module. The number of blocks is often called the number of layers. Before and after each block, there's an embedding module that converts tokens and their positions into vectors, and finally an un-embedding layer
4:50 that maps output vectors to token probabilities. So that's a super high look at this. I would really recommend either reading the book or check out StatQuest for an awesome overview of transformers and the attention mechanism. I'll link that in the description. That's really how I learned. While transformers dominate, they're not the only architecture. Models like RWKV, which combines RNN-based approaches with parallelization capabilities, are gaining traction for certain applications. In general, larger models with more parameters have greater capacity to learn and perform better. The number of parameters helps us estimate the compute resources needed for training and inference as well. However, note that parameter count can be misleading with sparse models, so those with many zeros, which can be more efficient. A large sparse model might require less compute than a smaller dense one. When designing models, compute is often the limiting factor. The Chinchilla Scaling Law helps calculate the optimal model size and data size for a given compute budget.
5:39 It suggests that the number of training tokens should be about 20 times the model size, so a 3 billion parameter model needs about 60 billion training tokens. While the cost for achieving the same model performance is decreasing over time, the cost for improvements remains high. Going from a 3% to a 2% error rate might require an order of magnitude more data, compute, or energy. But even small performance improvements can make a huge difference for downstream applications. As we keep scaling models, we're approaching two significant bottlenecks. First, training data. There's concern we'll run out of high-quality internet data in the next few years, forcing models to train on AI-generated content, potentially causing performance degradation. or requiring access to proprietary data like copyrighted books and medical records. Second, electricity. Data centers already consume 1-2% of global electricity, limiting how much larger they can grow without
6:27 significant energy breakthroughs. Pre-trained foundation models face two main issues. They're optimized for text completion, not conversation, and their outputs can be factually incorrect or ethically problematic. Post-training aims to address these issues through two main steps. First, supervised fine-tuning. Supervised fine-tuning optimizes the model for conversations instead of completion. This requires high-quality instruction data showing the kinds of requests the model should handle and how it should respond. It's essentially teaching the model what good responses look like. Second, preference fine-tuning. Preference fine-tuning aligns the model with human values using reinforcement learning, often called reinforcement learning from human feedback. This involves training a reward model that scores outputs based on human preferences, and optimizing the foundation model to generate responses that maximize these scores. While reinforcement learning from human feedback has been
7:16 the standard approach, newer methods like direct preference optimization, DPO, are gaining traction. Some companies even skip the reinforcement learning step entirely, instead generating multiple outputs and selecting those with high reward model scores. This is a strategy called best of n. Foundation models don't just produce a single definitive answer. They generate probabilities for possible outputs. How we sample from these probabilities dramatically affects the model's responses. The simplest approach is greedy sampling, always picking the highest probability token. But this leads to repetitive, predictable text. To introduce creativity, we use sampling techniques. Temperature controls how confident the model is in its predictions. Higher temperature values, like 0.7 to 1, make outputs more creative but potentially less accurate. but lower temperatures, close to zero, make outputs more deterministic and focused.
8:04 Top K sampling restricts the model to choosing from only the K most likely next tokens, typically between 50 and 500, depending on how diverse you want the responses to be. Top P sampling selects the smallest set of tokens whose cumulative probability exceeds a threshold P. A value of 0.9 means the model will only consider tokens that together make up 90% of the probability mass. This probabilistic nature explains many of the behaviors we see in foundation models, like inconsistency with minor input changes and hallucinations where models confidently state incorrect information. Now that we understand foundation models a little more, let's talk about one of the most crucial yet underappreciated aspects of
