AI prompt engineering: A deep dive

0:00 - Basically, this entire roundtable session here is just gonna be focused mainly on prompt engineering. A variety of perspectives at this table around prompting from a research side, from a consumer side, and from the enterprise side. And I want to just get the whole wide range of opinions because there's a lot of them. And just open it up to discussion and explore what prompt engineering really is and what it's all about. And yeah, we'll just take it from there. So maybe we can go around the horn with intros. I can kick it off. I'm Alex. I lead Developer Relations here at Anthropic. Before that, I was technically a prompt engineer at Anthropic. I worked on our prompt engineering team, and did a variety of roles spanning

0:45 from a solutions architect type of thing, to working on the research side. So with that, maybe I can hand it over to David. - Heck, yeah. My name's David Hershey. I work with customers mostly at Anthropic on a bunch of stuff technical, I help people with finetuning, but also just a lot of the generic things that make it hard to adopt language models of prompting. And just like how to build systems with language models, but spend most of my time working with customers. - Cool. I'm Amanda Askell. I lead one of the Finetuning teams at Anthropic, where I guess I try to make Claude be honest and kind. Yeah. - My name is Zack Witten. I'm a Prompt Engineer at Anthropic.

1:30 Alex and I always argue about who the first one was. He says it's him, I say it's me. - Contested. - Yeah. I used to work a lot with individual customers, kind of the same way David does now. And then as we brought more solutions architects to the team, I started working on things that are meant to raise the overall levels of ambient prompting in society, I guess, like the prompt generator and the various educational materials that people use. - Nice, cool. Well, thanks guys for all coming here. I'm gonna start with a very broad question just so we have a frame going into the rest of our conversations here. What is prompt engineering? Why is it engineering? What's prompt, really?

2:15 If anyone wants to kick that off, give your own perspective on it, feel free to take the rein here. - I feel like we have a prompt engineer. It's his job. - We're all prompt engineers in our own form. - But one of us has a job. - Yeah. Zack, maybe since it's in your title. - One of us has a job, but the other three don't have jobs. - I guess I feel like prompt engineering is trying to get the model to do things, trying to bring the most out of the model. Trying to work with the model to get things done that you wouldn't have been able to do otherwise. So a lot of it is just clear communicating. I think at heart, talking to a model is a lot like talking to a person. And getting in there and understanding the psychology of the model,

3:02 which Amanda is the world's most expert person in the world. - Well, I'm gonna keep going on you. Why is engineering in the name? - Yeah. I think the engineering part comes from the trial and error. - Okay. - So one really nice thing about talking to a model that's not like talking to a person, is you have this restart button. This giant go back to square zero where you just start from the beginning. And what that gives you the ability to do that you don't have, is a truly start from scratch and try out different things in an independent way, so that you don't have interference from one to the other. And once you have that ability to experiment and to design different things, that's where the engineering part has the potential

3:48 to come in. - Okay. So what you're saying is as you're writing these prompts, you're typing in a message to Claude or in the API or whatever it is. Being able to go back and forth with the model and to iterate on this message, and revert back to the clean slate every time, that process is the engineering part. This whole thing is prompt engineering all in one. - There's another aspect of it too, which is integrating the prompts within your system as a whole. And David has done a ton of work with customers integrating. A lot of times it's not just as simple as you write one prompt and you give it to the model and you're done. In fact, it's anything but. It's like way more complicated. - Yeah.

4:34 I think of prompts as the way that you program models a little bit, that makes it too complicated. 'Cause I think Zack is generally right that it's just talking clearly is the most important thing. But if you think about it a little bit as programming a model, you have to think about where data comes from, what data you have access to. So if you're doing RAG or something, what can I actually use and do and pass to a model? You have to think about trade-offs in latency and how much data you're providing and things like that. There's enough systems thinking that goes into how you actually build around a model. I think a lot of that's also the core of why it maybe deserves its own carve-out as a thing to reason about separately from just a software engineer or a PM or something like that. It's kind of its own domain of how to reason about these models.

5:20 - Is a prompt in this sense then natural language code? Is it a higher level of abstraction or is it a separate thing? - I think trying to get too abstract with a prompt is a way to overcomplicate a thing, because I think, we're gonna get into it, but more often than not, the thing you wanna do is just write a very clear description of a task, not try to build crazy abstractions or anything like that. But that said, you are compiling the set of instructions and things like that into outcomes a lot of times. So precision and a lot the things you think about with programming about version control and managing what it looked like back then when you had this experiment. And tracking your experiment and stuff like that,

6:06 that's all just equally important to code. - Yeah. - So it's weird to be in this paradigm where written text, like a nice essay that you wrote is something that's looked like the same thing as code. But it is true that now we write essays and treat them code, and I think that's actually correct. - Yeah. Okay, interesting. So maybe piggybacking off of that, we've loosely defined what prompt engineering is. So what makes a good prompt engineer? Maybe, Amanda, I'll go to you for this, since you're trying to hire prompt engineers more so in a research setting. What does that look like? What are you looking for in that type of person? - Yeah, good question. I think it's a mix of like Zack said, clear communication,

6:55 so the ability to just clearly state things, clearly understand tasks, think about and describe concepts really well. That's the writing component, I think. I actually think that being a good writer is not as correlated with being a good prompt engineer as people might think. So I guess I've had this discussion with people 'cause I think there's some argument as like, "Maybe you just shouldn't have the name engineer in there. Why isn't it just writer?" I used to be more sympathetic to that. And then, I think, now I'm like what you're actually doing, people think that you're writing one thing and you're done. Then I'll be like to get a semi-decent prompt when I sit down with the model. Earlier, I was prompting the model and I was just like in a 15-minute span

7:40 I'll be sending hundreds of prompts to the model. It's just back and forth, back and forth, back and forth. So I think it's this willingness to iterate and to look and think what is it that was misinterpreted here, if anything? And then fix that thing. So that ability to iterate. So I'd say clear communication, that ability to iterate. I think also thinking about ways in which your prompt might go wrong. So if you have a prompt that you're going to be applying to say, 400 cases, it's really easy to think about the typical case that it's going to be applied to, to see that it gets the right solution in that case, and then to move on. I think this is a very classic mistake that people made. What you actually want to do is find the cases where it's unusual. So you have to think about your prompt and be like, "What are the cases where it'd be really unclear to me

8:26 what I should do in this case?" So for example, you have a prompt that says, "I'm going to send you a bunch of data. I want you to extract all of the rows where someone's name is, I don't know, starts with the letter G." And then you're like, "Well, I'm gonna send it a dataset where there is no such thing, there is no such name that starts with the letter G. "I'm going to send it something that's not a dataset, I might also just send it an empty string. These are all of the cases you have to try, because then you're like, "What does it do in these cases? " And then you can give it more instructions for how it should deal with that case. - I work with customers so often where you're an engineer, you're building something. And there's a part in your prompt where a customer of theirs is going to write something. - Yeah. - And they all think about these really perfectly phrased things that they think someone's going to type into their chatbot. And in reality, it's like they never used the shift key

9:12 and every other word is a typo. - They think it's Google. - And there's no punctuation. - They just put in random words with no question. - Exactly. So you have these evals that are these beautifully structured what their users ideally would type in. But being able to go the next step to reason about what your actual traffic's gonna be like, what people are actually gonna to try to do, that's a different level of thinking. - One thing you said that really resonated with me is reading the model responses. In a machine learning context, you're supposed to look at the data. It's almost a cliche like look at your data, and I feel like the equivalent for prompting is look at the model outputs. Just reading a lot of outputs and reading them closely. Like Dave and I were talking on the way here, one thing that people will do is they'll put think step-by-step in their prompt. And they won't check to make sure

9:58 that the model is actually thinking step-by-step, because the model might take it in a more abstract or general sense. Rather than like, "No, literally you have to write down your thoughts in these specific tags." So yeah, if you aren't reading the model outputs, you might not even notice that it's making that mistake. - Yeah, that's interesting. There is that weird theory of mind piece to being a prompt engineer where you have to think almost about how the model's gonna view your instructions. But then if you're writing for an enterprise use case too, you also have to think about how the user's gonna talk to the model, as you're the third party sitting there in that weird relationship. Yeah. - On the theory of mind piece, one thing I would say is,

10:43 it's so hard to write instructions down for a task. It's so hard to untangle in your own brain all of the stuff that you know that Claude does not know and write it down. It's just an immensely challenging thing to strip away all of the assumptions you have, and be able to very clearly communicate the full fact set of information that is needed to a model. I think that's another thing that really differentiates a good prompt engineer from a bad one, is like... A lot of people will just write down the things they know. But they don't really take the time to systematically break out what is the actual full set of information you need to know to understand this task? - Right. - And that's a very clear thing I see a lot is prompts where it's just conditioned. The prompt that someone wrote is so conditioned

11:30 on their prior understanding of a task, that when they show it to me I'm like, "This makes no sense. None of the words you wrote make any sense, because I don't know anything about your interesting use case." But I think a good way to think about prompt engineering in that front and a good skill for it, is just can you actually step back from what you know and communicate to this weird system that knows a lot, but not everything about what it needs to know to do a task? - Yeah. The amount of times I've seen someone's prompt and then being like, "I can't do the task based on this prompt." I'm human level and you're giving this to something that is worse than me and expecting it to do better, and I'm like, "Yeah." - Yeah. There is that interesting thing with like...

12:15 Current models don't really do a good job of asking good, probing questions in response like a human would. If I'm giving Zack directions on how to do something, he'll be like, "This doesn't make any sense. What am I supposed to do at this step or here and here?" Model doesn't do that, right, so you have to, as yourself, think through what that other person would say and then go back to your prompt and answer those questions. - You could ask it to do that. - You could. That's right. - I do that, yeah. - I guess that's another step. - I was going to say one of the first things I do with my initial prompt, is I'll give it the prompt and then I'll be like, "I don't want you to follow these instructions. I just want you to tell me the ways in which they're unclear or any ambiguities, or anything you don't understand." And it doesn't always get it perfect, but it is interesting that that is one thing you can do. And then also sometimes if people see

13:01 that the model makes a mistake, the thing that they don't often do is just ask the model. So they say to the model, "You got this wrong. Can you think about why? And can you maybe write an edited version of my instructions that would make you not get it wrong?" And a lot of the time, the model just gets it right. The model's like, "Oh, yeah. Here's what was unclear, here's a fix to the instructions," and then you put those in and it works. - Okay. I'm actually really curious about this personally almost. Is that true that that works? Is the model able to spot its mistakes that way? When it gets something wrong, you say, "Why did you get this wrong?" And then it tells you maybe something like, "Okay, how could I phrase this to you in the future so you get it right?" Is there an element of truth to that? Or is that just a hallucination on the model's part around what it thinks its limits are? - I think if you explain to it what it got wrong,

13:49 it can identify things in the query sometimes. I think this varies by task. This is one of those things where I'm like I'm not sure what percentage of the time it gets it right, but I always try it 'cause sometimes it does. - And you learn something. - Yeah. - Anytime you go back to the model or back and forth with the model, you learn something about what's going on. I think you're giving away information if you don't at least try. - That's interesting. Amanda, I'm gonna keep asking you a few more questions here. One thing maybe for everybody watching this, is we have these Slack channels at Anthropic where people can add Claude into the Slack channel, then you can talk to Claude through it. And Amanda has a Slack channel that a lot of people follow of her interactions with Claude. And one thing that I see you always do in there,

14:34 which you probably do the most of anyone at Anthropic, is use the model to help you in a variety of different scenarios. I think you put a lot of trust into the model in the research setting. I'm curious how you've developed those intuitions for when to trust the model. Is that just a matter of usage, experience or is it something else? - I think I don't trust the model ever and then I just hammer on it. So I think the reason why you see me do that a lot, is that that is me being like, "Can I trust you to do this task?" 'Cause there's some things, models are kind of strange. If you go slightly out of distribution, you just go into areas where they haven't been trained or they're unusual. Sometimes you're like, "Actually, you're much less reliable here, even though it's a fairly simple task."

15:21 I think that's happening less and less over time as models get better, but you want to make sure you're not in that kind of space. So, yeah, I don't think I trust it by default, but I think in ML, people often want to look across really large datasets. And I'm like, "When does it make sense to do that?" And I think the answer is when you get relatively low signal from each data point, you want to look across many, many data points, because you basically want to get rid of the noise. With a lot of prompting tasks, I think you actually get really high signal from each query. So if you have a really well-constructed set of a few hundred prompts, that I think can be much more signal than thousands that aren't as well-crafted. So I do think that I can trust the model if I look at 100 outputs of it and it's really consistent.

16:06 And I know that I've constructed those to basically figure out all of the edge cases and all of the weird things that the model might do, strange inputs, et cetera. I trust that probably more than a much more loosely constructed set of several thousand. - I think in ML, a lot of times the signals are numbers. Did you predict this thing right or not? And it'd be looking at the logprobs of a model and trying to intuit things, which you can do, but it's kind of sketchy. I feel like the fact that models output more often than not a lot of stuff like words and things. There's just fundamentally so much to learn between the lines of what it's writing and why and how, and that's part of what it is. It's not just did it get the task right or not?

16:54 It's like, "How did it get there? How was it thinking about it? What steps did it go through?" You learn a lot about what is going on, or at least you can try to get a better sense, I think. But that's where a lot of information comes from for me, is by reading the details of what came out, not just through the result. - I think also the very best of prompting can make the difference between a failed and a successful experiment. So sometimes I can get annoyed if people don't focus enough on the prompting component of their experiment, because I'm like, "This can, in fact, be the difference between 1% performance in the model or 0.1%." In such a way that your experiment doesn't succeed if it's at top 5% model performance, but it does succeed if it's top 1% or top 0.1%. And then I'm like, "If you're gonna spend time

17:40 over coding your experiment really nicely, but then just not spend time on the prompt." I don't know. That doesn't make sense to me, 'cause that can be the difference between life and death of your experiment. - Yeah. And with the deployment too, it's so easy to, "Oh, we can't ship this." And then you change the prompt around and suddenly it's working. - Yeah. - It's a bit of a double-edged sword though, because I feel like there's a little bit of prompting where there's always this mythical, better prompt that's going to solve my thing on the horizon. - Yeah. - I see a lot of people get stuck into the mythical prompt on the horizon, that if I just keep grinding, keep grinding. It's never bad to grind a little bit on a prompt, as we've talked, you learn things. But it's one of the scary things about prompting is that there's this whole world of unknown. - What heuristics do you guys have

18:26 for when something is possible versus not possible with a perfect prompt, whatever that might be? - I think I'm usually checking for whether the model kind of gets it. So I think for things where I just don't think a prompt is going to help, there is a little bit of grinding. But often, it just becomes really clear that it's not close or something. Yeah. I don't know if that's a weird one where I'm just like, "Yeah, if the model just clearly can't do something, I won't grind on it for too long." - This is the part that you can evoke how it's thinking about it, and you can ask it how it's thinking about it and why. And you can get a sense of is it thinking about it right? Are we even in the right zip code of this being right? And you can get a little bit of a kneeling on that front of,

19:14 at least, I feel like I'm making progress towards getting something closer to right. Where there's just some tasks where you really don't get anywhere closer to it's thought process. It's just like every tweak you make just veers off in a completely different, very wrong direction, and I just tend to abandon those. I don't know. - Those are so rare now though, and I get really angry at the model when I discover them because that's how rare they are. I get furious. I'm like, "How dare there be a task that you can't just do, if I just push you in the right direction?" - I had my thing with Claude plays Pokemon recently, and that was one of the rare times where I really... - Yeah, can you explain that? Explain that just for people. I think that's really cool. - I did a bit of an experiment where I hooked Claude up to a Game Boy emulator,

19:59 and tried to have it play the game Pokemon Red like the OG Pokemon. And it's like you think what you wanna do and it could write some code to press buttons and stuff like that, pretty basic. And I tried a bunch of different very complex prompting layouts, but you just get into certain spots where it just really couldn't do it. So showing it a screenshot of a Game Boy, it just really couldn't do. And it just so deeply because I'm so used to it, being able to do something mostly. So I spent a whole weekend trying to write better and better prompts to get it to really understand this Game Boy screen. And I got incrementally better so that it was only terrible instead of completely no signal.

20:46 You could get from no signal to some signal. But it was, I don't know, at least this is elicited for me. Once I put a weekend of time in and I got from no signal to some signal, but nowhere close to good enough, I'm like, "I'm just going to wait for the next one. (Alex laughing) I'm just gonna wait for another model." I could grind on this for four months, and the thing that would come out is another model and that's a better use of my time. Just sit and wait to do something else in the meanwhile. - Yeah. That's an inherent tension we see all the time, and maybe we can get to that in a sec. Zack, if you wanna go. - Something I liked about your prompt with Pokemon where you got the best that you did get, was the way that you explained to the model that it is in the middle of this Pokemon game. Here's how the things are gonna be represented.

21:33 I actually think you actually represented it in two different ways, right? - I did. So what I ended up doing, it was obnoxious but I superimposed a grid over the image, and then I had to describe each segment of the grid in visual detail. Then I had to reconstruct that into an ASCII map and I gave it as much detail as I could. The player character is always at location 4, 5 on the grid and stuff like that, and you can slowly build up information. I think it's actually a lot like prompting, but I just hadn't done it with images before. Where sometimes my intuition for what you need to tell a model about text, is a lot different from what you need to tell a model about images. - Yeah. - I found a surprisingly small number of my intuitions about text have transferred to image.

22:20 I found that multi-shot prompting is not as effective for images and text. I'm not really sure, you can have theoretical explanations about why. Maybe there's a few of it in the training data, a few examples of that. - Yeah. I know when we were doing the original explorations with prompting multimodal, we really couldn't get it to noticeably work. You just can't seem to improve Claude's actual, visual acuity in terms of what it picks up within an image. Anyone here has any ways that they've not seen that feature. But it seems like that's similar with the Pokemon thing where it's trying to interpret this thing. No matter how much you throw prompts at it, it just won't pick up that Ash that's in that location. - Yeah. But I guess to be visceral about this, I could eventually get it so that it could most often tell me where a wall was,

23:07 and most often tell me where the character was. It'd be off by a little bit. But then you get to a point, and this is maybe coming back to knowing when you can't do it. It would describe an NPC, and to play a game well, you need to have some sense of continuity. Have I talked to this NPC before? And without that, you really don't, there's nothing you can do. You're just going to keep talking to the NPC, 'cause like, "Well, maybe this is a different NPC." But I would try very hard to get it to describe a NPC and it's like, "It's a person." They might be wearing a hat, they weren't wearing a hat. And it's like you grind for a while, inflate it to 3000X and just crop it to just the NPC, and it's like, "I have no idea what this is." It's like I showed it this clear, female NPC thing

23:54 enough times and it just got nowhere close to it, and it's like, "Yeah, this is a complete lost cause." - Wow, okay. - I really want to try this now. I'm just imagining all the things I would try. I don't know, I want you to imagine this game art as a real human and just describe to me what they're like. What did they look like as they look in the mirror? And then just see what the model does. - I tried a lot of things. The eventual prompt was telling Claude it was a screen reader for a blind person, which I don't know if that helped, but it felt right so I stuck with that. - That's an interesting point. I actually wanna go into this a little bit 'cause this is one of the most famous prompting tips, is to tell the language model that they are some persona or some role. I feel like I see mixed results. Maybe this worked a little bit better in previous models

24:41 and maybe not as much anymore. Amanda, I see you all the time be very honest with the model about the whole situation like, "Oh, I am an AI researcher and I'm doing this experiment." - I'll tell it who I am. - Yeah. - I'll give it my name, be like, "Here's who you're talking to." - Right. Do you think that level of honesty, instead of lying to the model or forcing it to like, "I'm gonna tip you $500." Is there one method that's preferred there, or just what's your intuition on that? - Yeah. I think as models are more capable and understand more about the world, I guess, I just don't see it as necessary to lie to them. I also don't like lying to the models just 'cause I don't like lying generally. But part of me is if you are, say, constructing. Suppose you're constructing an eval dataset

25:28 for a machine learning system or for a language model. That's very different from constructing a quiz for some children. So when people would do things like, "I am a teacher trying to figure out questions for a quiz." I'm like, "The model knows what language model evals are." If you ask it about different evals it can tell you, and it can give you made up examples of what they look like. 'Cause these things are like they understand them, they're on the internet. So I'm like, "I'd much rather just target the actual task that I have." So if you're like, "I want you to construct questions that look a lot like an evaluation of a language model." It's that whole thing of clear communication. I'm like, "That is, in fact, the task I want to do. So why would I pretend to you that I want to do some unrelated, or only tangentially related task?"

26:13 And then expect you to somehow do better at the task that I actually want you to do. We don't do this with employees. I wouldn't go to someone that worked with me and be like, "You are a teacher and you're trying to quiz your students." I'd be like, "Hey, are you making that eval?" I don't know. So I think maybe it's a heuristic from there where I'm like, "If they understand the thing, just ask them to do the thing that you want." - I see this so much. - I guess to push back a little bit, I have found cases where not exactly lying but giving it a metaphor for how to think about it could help. In the same way that sometimes I might not understand how to do something and someone's like, "Imagine that you were doing this, even though I know I'm not doing it." The one that comes to mind for me, is I was trying to have Claude say whether an image of a chart or a graph is good or not. Is it high quality?

26:59 And the best prompt that I found for this was asking the model what grade it would give the chart, if it were submitted as a high school assignment. So it's not exactly saying, "You are a high school teacher." It's more like, "This is the kind of analysis that I'm looking from for you." The scale that a teacher would use is similar to the scale that I want you to use. - But I think those metaphors are pretty hard to still come up with. I think people still, the default you see all the time is finding some facsimile of the task. Something that's a very similar-ish task, like saying you're a teacher. You actually just lose a lot in the nuance of what your product is. I see this so much in enterprise prompts where people write something similar,

27:46 because they have this intuition that it's something the model has seen more of maybe. It's seen more high school quizzes than it has LLM evals, and that may be true. But to your point, as the models get better, I think just trying to be very prescriptive about exactly the situation they're in. I give people that advice all the time. Which isn't to say that I don't think to the extent that it is true that thinking about it the way that someone would grade a chart, as how they would grade a high school chart, maybe that's true. But it's awkwardly the shortcut people use a lot of times to try to get what happens, so I'll try to get someone that I can actually talk about 'cause I think it's somewhat interesting. So writing you are a helpful assistant,

28:35 writing a draft of a document, it's not quite what you are. You are in this product, so tell me. If you're writing an assistant that's in a product, tell me I'm in the product. Tell me I'm writing on behalf of this company, I'm embedded in this product. I'm the support chat window on that product. You're a language model, you're not a human, that's fine. But just being really prescriptive about the exact context about where something is being used. I found a lot of that. Because I guess my concern most often with role prompting, is people used it as a shortcut of a similar task they want the model to do. And then they're surprised when Claude doesn't do their task right, but it's not the task. You told it to do some other task.

29:21 And if you didn't give it the details about your task, I feel like you're leaving something on the table. So I don't know, it does feel like a thing though to your point of as the models scale. Maybe in the past it was true that they only really had a strong understanding of elementary school tests comparatively. But as they get smarter and can differentiate more topics, I don't know, just like being clear. - I find it interesting that I've never used this prompting technique. - Yeah, that's funny. - Even with worse models and I still just don't ever find myself, I don't know why. I'm just like, "I don't find it very good essentially." - Interesting. - I feel like completion era models, there was a little bit of a mental model of conditioning the model into a latent space

30:07 that was useful that I worried about, that I don't really worry about too much anymore. - It might be intuitions from pretrained models over to RLHF models, that to me, just didn't make sense. It makes sense to me if you're prompting a pretrained. - You'd be amazed how many people try to apply their intuitions. I think it's not that surprising. Most people haven't really experimented with the full what is a pretrained model? What happens after you do SL? What happens after you do RLHF, whatever? So when I talk to customers, it's all the time that they're trying to map some amount of, "Oh, how much of this was on the internet? Have they seen a ton of this on the internet?" You just hear that intuition a lot, and I think it's well-founded fundamentally.

30:54 But it is overapplied by the time you actually get to a prompt, because of what you said. By the time they've gone through all of this other stuff, that's not actually quite what's being modeled. - Yeah. The first thing that I feel like you should try is, I used to give people this thought experiment where it's like imagine you have this task. You've hired a temp agency to send someone to do this task. This person arrives, you know they're pretty competent. They know a lot about your industry and so forth, but they don't know the name of your company. They've literally just shown up and they're like, "Hey, I was told you guys had a job for me to do, tell me about it." And then it's like, "What would you say to that person?" And you might use these metaphors. You might say things like, "We want you to detect good charts.

31:41 What we mean by a good chart here, is it doesn't need to be perfect. You don't need to go look up whether all of the details are correct." It just needs to have its axes labeled, and so think about maybe high school level, good chart. You may say exactly that to that person and you're not saying to them, "You are a high school." You wouldn't say that to them. You wouldn't be like, "You're a high school teacher reading charts." - What are you talking about? - Yeah, so sometimes I'm just like it's like the whole if I read it. I'm just like, "Yeah. Imagine this person who just has very little context, but they're quite competent. They understand a lot of things about the world." Try the first version that actually assumes that they might know things about the world, and if that doesn't work, you can maybe do tweaks and stuff. But so often, the first thing I try is that, and then I'm like, "That just worked." - That worked.

32:27 - And then people are like, "Oh, I didn't think to just tell it all about myself and all about the task I want to do." - I've carried this thing that Alex told me to so many customers where they're like, "Oh, my prompt doesn't work. Can you help me fix it?" I'm like, "Well, can you describe to me what the task was?" And I'm like, "Okay. Now what you just said to me, just voice record that and then transcribe it." And then paste it into the prompt and it's a better prompt than what you wrote, but this is a laziness shortcut, I think, to some extent. Because people write something that they... I just think people, I'm lazy. A lot of people are lazy. - We had that in prompt assistance the other day where somebody was like, "Here's the thing, here's what I want it to do, and here's what it's actually doing instead." So then I just literally copied the thing that they said they wanted it to do, and pasted it in and it worked. - Yeah. I think a lot of people still

33:13 haven't quite wrapped their heads around what they're really doing when they're prompting. A lot of people see a text box and they think it's a Google search box. They type in keywords and maybe that's more on the chat side. But then on the enterprise side of things, you're writing a prompt for an application. There is still this weird thing to it where people are trying to take all these little shortcuts in their prompt, and just thinking that, "Oh, this line carries a lot of weight in this." - Yeah. I think you obsess over getting the perfect little line of information and instruction, as opposed to how you just described that graph thing. I would be a dream if I read prompts like that. If someone's like, "Well, you do this and this, and there's some stuff to consider about this and all that." But that's just not how people write prompts. They work so hard to find the perfect, insightful. A perfect graph looks exactly like this exact perfect thing,

34:02 and you can't do that. It's just very hard to ever write that set of instructions down prescriptively, as opposed to how we actually talk to humans about it, which is try to instill some amount of the intuitions you have. - We also give them outs. This is a thing that people can often forget in prompts. So cases, if there's an edge case, think about what you want the model to do. 'Cause by default, it will try the best to follow your instructions, much as the person from the temp agency would, 'cause they're like, "Well, they didn't tell me how to get in touch with anyone." If I'm just given a picture of a goat and I'm like, "What do I do? This isn't even a chart. How good is a picture of a goat as a chart?" I just don't know. And if you instead say something like, "If something weird happens and you're really not sure what to do, just output in tags unsure."

34:49 Then you can go look through the unsures that you got and be like, "Okay, cool. It didn't do anything weird." Whereas by default, if you don't give the person the option, they're like, "It's a good chart." Then people will be like, "How do I do that?" And then you're like, "Well, give it an out. Give it something to do if it's a really unexpected input happens." - And then you also improved your data quality by doing that too, 'cause you found all the screwed up examples. - Oh, yeah. - That's my favorite thing about iterating on tests with Claude, is the most common outcome is I find all of the terrible tests I accidentally wrote because it gets it wrong. I'm like, "Oh, why did it get wrong?" I was like, "Oh, I was wrong." - Yeah. - Yeah. - If I was a company working with this, I do think I would just give my prompts to people, because I used to do this when I was evaluating language models.

35:34 I would take the eval myself. 'Cause I'm like, "I need to know what this eval looks like if I'm gonna to be grading it, having models take it, thinking about outputs, et cetera." I would actually just set up a little script and I would just sit and I would do the eval. - Nowadays, you just have called the Streamboard app for you. - And just does it, yeah. - Yeah. I'm reminded of Karpathy's ImageNet. I was in 231 at Stanford and it's like benchmarking, he's showing the accuracy number. And he's like, "And here's what my accuracy number was." And he had just gone through the test set and evaluated himself. - Oh, yeah. - You just learn a lot. - Yeah, totally. - And it's better when it's a, again, the temp agency person, like someone who doesn't know the task, because that's a very clean way to learn things. - Yeah. The way you have to do it is,

36:20 some evaluations come with instructions, and so I would give myself those instructions as well and then try to understand it. And it's actually quite great if you don't have context on how it's graded. And so often, I would do so much worse than the human benchmark and I was like, "I don't even know how you got humans to do this well at this task, 'cause apparently human level here is 90%, and I'm at 68%." - That's funny. That reminds me of just when you look at the MMLU questions and you're like, "Who would be able to answer these?" It's just like absolute garbage in some of them. Okay. I have one thing I wanna circle back on that we were talking about a few questions back around,

37:05 I think you were saying getting signal from the responses. There's just so much there and it's more than just a number, and you can actually read into the almost thought process. I bet this is probably a little contentious maybe around chain of thought. For people listening, chain of thought, this process of getting them all to actually explain its reasoning before it provides an answer. Is that reasoning real or is it just kind of like a holding space for the model to do computation? Do we actually think there's good, insightful signal that we're getting out of the model there? - This is one of the places where I struggle with that. I'm normally actually somewhat pro-personification because I think it helps you get decent facsimiles, thoughts of how the model's working.

37:52 And this one, I think it's harmful maybe almost to get too into the personification of what reasoning is, 'cause it just loses the thread of what we're trying to do here. Is it reasoning or not? It feels almost like a different question than what's the best prompting technique? It's like you're getting into philosophy, which we can get into. - Yeah, we do have a philosopher. - Yeah. I will happily be beaten down by a real philosopher as I try to speculate on this, but instead, it just works. Your model does better. The outcome is better if you do reasoning. I think I've found that if you structure the reasoning and help iterate with the model on how it should do reasoning, it works better too.

38:38 Whether or not that's reasoning or how you wanted to classify it, you can think of all sorts of proxies for how I would also do really bad if I had to do one-shot math without writing anything down. Maybe that's useful, but all I really know is, it very obviously does help. I don't know. - A way of testing would be if you take out all the reasoning that it did to get to the right answer, and then replace it with somewhat, realistic-looking reasoning that led to a wrong answer, and then see if it does conclude the wrong answer. I think we actually had a paper where we did some of that. There was the scratch pad. It was like the Sleeper Agents. - Oh, okay. Alignment papers. - But I think that was maybe a weird situation. But definitely what you said about structuring the reasoning

39:27 and writing example of how the reasoning works. Given that that helps, like whether we use the word reasoning or not, I don't think it's just a space for computation. - So there is something there. - I think there's something there, whatever we wanna call it. - Yeah. Having it write a story before it finished a task, I do not think would work as well. - I've actually tried that and it didn't work as well as reasoning. - Clearly, the actual reasoning part is doing something towards the outcome. - I've tried like, "Repeat the words um and ah in any order that you please for 100 tokens and then answer." - Yeah. I guess that's a pretty thorough defeat of it's just more computational space where it can do attention over and over again. I don't think it's just more attention like doing more attention. - I guess the strange thing is, and I don't have an example off the top of my head

40:13 to back this up with. But I definitely have seen it before where it lays out steps, one of the steps is wrong, but then it still reaches the right answer at the end. So it's not quite, I guess, yeah, we can't really, truly personify it as a reasoning, 'cause there is some element to it doing something slightly different. - Yeah. I've also met a lot of people who make inconsistent steps of reasoning. - I guess that's true. - It fundamentally defeats the topic of reasoning by making a false step on the way there. - All right, it's interesting. Also, on maybe this prompting misconceptions round of questions. Zack, I know you have strong opinions on this, good grammar, punctuation. - Oh, do I?

40:59 - Is that necessary in a prompt? Do you need it? Do you need to format everything correctly? - I usually try to do that because I find it fun, I guess, somehow. I don't think you necessarily need to. I don't think it hurts. I think it's more that you should have the level of attention to detail that would lead you to doing that naturally. If you're just reading over your prompt a lot, you'll probably notice those things and you may as well fix them. And like what Amanda was saying, that you wanna put as much love into the prompt as you do into the code. People who write a lot of code have strong opinions about things that I could not care less about.

41:44 Like the number of tabs versus spaces, or I don't know, opinions about which languages are better. And for me, I have opinionated beliefs about styling of prompts. I can't even say that they're right or wrong, but I think it's probably good to try to acquire those, even if they're arbitrary. - I feel personally attacked, 'cause I definitely have prompts that are like I feel like I'm in the opposite end of the spectrum where people will see my prompts. And then be like, "This just has a whole bunch of typos in it." And I'm like, "The model knows what I mean." - It does, it does know what you mean, but you're putting in the effort, you just are attending to different things. - 'Cause part of me is like, I think if it's conceptually clear, I'm a big, I will think a lot about the concepts and the words that I'm using. So there's definitely a sort of care that I put in.

42:31 But it's definitely not to, yeah, people will just point out typos and grammatical issues with my prompts all the time. Now I'm pretty good at actually checking those things more regularly. - Is it because of pressure from the outside world or because it's actually what you think is right? - It's pressure from me. - Yeah, it's probably pressure from the outside world. I do think it makes sense. Part of me is like it's such an easy check, so I think for a final prompt I would do that. But throughout iteration, I'll happily just iterate with prompts that have a bunch of typos in them, just 'cause I'm like, "I just don't think that the model's going to care." - This gets at the pretrained model versus RLHF thing though, because I was talking to Zack on the way over. The conditional probability of a typo based on a previous typo in the pretraining data is much higher. - Oh, yeah. - Like much higher.

43:17 - Prompting pretraining models is just a different beast. - It is, but it's interesting. I think it's an interesting illustration of why your intuitions, like trying to over-apply the intuitions of a pretrained model to the things that we're actually using in production doesn't work very well. Because again, if you were to pass one of your typo-ridden prompts to a pretrained model, the thing that would come out the other side, almost assuredly would be typo-ridden. - Right. - I like to leverage this to create typo-ridden inputs. - That's true. I've done that. - Like what you're saying, try to anticipate what your customers will put in. The pretrained model is a lot better at doing that. 'Cause the RL models are very polished and they really never made a typo in their lives. - They've been told pretty aggressively to not do the typo thing.

44:04 - Yeah. Okay, so that's actually an interesting segue here. I've definitely mentioned this to people in the past around to try to help people understand a frame of talking to these models in a sense almost as an imitator to a degree. And that might be much more true of a pretrained model than a post-trained, full-finished model, but is there anything to that? If you do talk to Claude and use a ton of emojis and everything, it will respond similarly, right? So maybe some of that is there, but like you're saying, it's not all the way quite like a pretrained model. - It's just shifted to what you want. I think at that point, it's like trying to guess what you... We have more or less trained the models to guess what you want them to act like.

44:51 - Interesting. - Or after we do all of our fancy stuff after pretraining. - The human laborers that used emojis, prefer to get responses with emojis. - Yeah. Amanda writes things with typos but wants not typos at the other end, and Claude's pretty good at figuring that out. If you write a bunch of emojis to Claude, it's probably the case that you also want a bunch of emojis back from Claude. That's not surprising to me. - Yeah. This is probably something we should have done earlier, but I'll do it now. Let's clarify maybe the differences between what an enterprise prompt is or a research prompt, or a just general chat in Claude.ai prompt. Zack, you've spanned the whole spectrum here in terms of working with customers and research.

45:39 Do you wanna just lay out what those mean? - Yeah, I guess. This feels too, you're hitting me with all the hard questions. - Yeah. (laughing) - Well, the people in this room, I think of it as the prompts that I read in Amanda's Claude channel versus the prompts that I read David write. They're very similar in the sense that the level of care and nuance that's put into them. I think for research, you're looking for variety and diversity a lot more. So if I could boil it down to one thing, it's like I've noticed Amanda's not the biggest fan of having lots of examples, or one or two examples. Like too few 'cause the model will latch onto those.

46:27 And in prompts that I might write or that I've seen David write, we have a lot of examples. I like to just go crazy and add examples until I feel like I'm about to drop dead, 'cause I've added so many of them. And I think that's because when you're in a consumer application, you really value reliability. You care a ton about the format, and it's fine if all the answers are the same. In fact, you almost want them to be the same in a lot of ways, not necessarily you want to be responsive to the user's desires. Whereas a lot of times when you're prompting for research, you're trying to really tap into the range of possibilities

47:14 that the model can explore. And by having some examples, you're actually constraining that a little bit. So I guess just on how the prompts look level, that's probably the biggest difference I noticed is how many examples are in the prompt, which is not to say that I've never seen you write a prompt with examples. But does that ring true for you? - Yeah. I think when I give examples, often I actually try and make the examples not like the data that the model's going to see, so they're intentionally illustrative. Because if the model, if I give it examples that are very like the data it's going to see, I just think it is going to give me a really consistent response that might not actually be what I want. Because my data that I'm running it on might be extremely varied,

47:59 and so I don't want it to just try and give me this really rote output. Often, I want it to be much more responsive. It's much more like cognitive tasks essentially where I'm like, "You have to see this sample and really think about in this sample what was the right answer." So that means that sometimes I'll actually take examples that are just very distinct from the ones that I'm going to be running it on. So if I have a task where, let's say, I was trying to extract information from factual documents. I might actually give it examples that are from what sounds like a children's story. Just so that I want you to understand the task, but I don't want you to latch on too much to the words that I use or the very specific format. I care more about you understanding the actual thing that I want you to do, which can mean I don't end up giving,

48:48 in some cases, there's some cases where this isn't true. But if you want more flexibility and diversity, you're going to use illustrative examples rather than concrete ones. You're probably never going to put words in the model's mouth. I haven't liked that in a long time though. I don't do few-shot examples involving the model having done a thing. I think that intuition actually also comes from pretraining in a way that doesn't feel like it rings true of RLHF models. So yeah, I think those are differences. - The only thing I'd add, a lot of times if you're prompting, like if I'm writing prompts to use on Claude.ai, it's like I'm iterating until I get it right one time. Then it's out the window, I'm good, I did it. Whereas most enterprise prompts, it's like you're gonna go use this thing a million times

49:35 or 10 million times, or 100 million times or something like that. So the care and thought you put in is very much testing against the whole range of things, like ways this could be used and the range of input data. Whereas a lot of my time, it's like thinking about one specific thing I want the model to get done right now. - Right, correct. - And it's a pretty big difference in how I approach prompting between if I just wanna get it done this one time right, versus if I wanna build a system that gets it right a million times. - Yeah. Definitely, in the chat setting, you have the ability to keep the human-in-the-loop and just keep going back and forth. Whereas when you're writing for a prompt to power a chatbot system, it has to cover the whole spectrum of what it could possibly encounter.

50:20 - It's a lot lower stakes when you are on Claude.ai and you can tell it that it got it wrong or you can even edit your message and try again. But if you're designing for the delightfully discontent user, divinely discontent user, then you can't ask them to do anything more than the minimum. - But good prompts, I would say, are still good across both those things. If you put the time into the thing for yourself and the time into the enterprise thing, it's equally good. It's just they diverge a little bit in the last mile, I think. - Cool. So the next question I want to just maybe go around the table here, is if you guys had one tip that you could give somebody improving their prompting skill. It doesn't have to be just about writing a good prompt, it could be that, but just generally getting better

51:07 at this act of prompting, what would you recommend? - Reading prompts, reading model outputs. Anytime I see a good prompt that someone wrote at Anthropic, I'll read it more closely. Try to break down what it's doing and why and maybe test it out myself, experimentation, talking to the model a lot. - So just how do you know that it's a good prompt, though, to begin with? You just see that the outputs are doing the job correctly? - Yeah. - Okay. - Yeah, that's exactly right. - Okay. Amanda, maybe you? - Yeah, I think there's probably a lot here.

51:55 Giving your prompt to another person can be helpful just as a reminder, especially someone who has no context on what you're doing. Yeah, my boring advice has been, it's one of those just do it over and over and over again. And I think if you're really curious and interested and find it fun, this is a lot of people who end up good at prompting, it's just because they actually enjoy it. So I don't know, I once joked just try replacing all of your friends with AI models and try to automate your own job with AI models. And maybe just try to in your spare time, take joy red teaming AI models. So if you enjoy it, it's much easier. So I'd say do it over and over again,

52:42 give your prompts to other people. Try to read your prompts as if you are a human encountering it for the first time. - I would say trying to get the model to do something you don't think it can do. The time I've learned the most from prompting, is when I'm probing the boundaries of what I think a model's capable of. - Interesting. - There's this huge set of things that are so trivial that you don't really get signal on if you're doing a good job or not. Like, "Write me a nice email," it's like you're going to write a nice email. But if you find or can think of something that pushes the boundaries of what you think is possible. I guess probably the first time I ever got into prompting in a way where I felt like I learned a decent amount, was trying to build a task like an agent like everybody else. Like decompose the task and figure out

53:27 how to do the different steps of the task. And by really pressing the boundaries of what the model was capable of, you just learn a lot about navigating that. I think a lot of prompt engineering is actually much more about pressing the boundaries of what the model can do. The stuff that's easy, you don't really need to be a prompt engineer to do. So that's, I guess, what I would say is find the hardest thing you can think of and try to do it. And even if you fail, you tend to learn a lot about how the model works. - That's actually a perfect transition to my next question. Yeah. Basically, from my own experience, how I got started with prompting was with jailbreaking and red teaming. And that is very much trying to find the boundary limits of what the model can do. And figure out how it responds

54:13 to different phrasings and wordings, and just a lot of trial and error. On the topic of jailbreaks, what's really happening inside a model? When you write a jailbreak prompt, what's going on there? How does that interact with the post-training that we apply to Claude? Amanda, maybe you have some insight here that you could offer. - I'm not actually sure. - It's honest. - Yeah. I feel bad 'cause I do think lots of people have obviously worked on the question of what's going on with jailbreaks? One model might just be that you're putting the model very out of distribution from its training data. So if you get jailbreaks where people use a lot of tokens, or they're just these huge, long pieces of text

55:02 where like during finetuning, you might just not expect to see as much of that. That would be one thing that could be happening when you jailbreak models. I think there's others, but I think a lot of jailbreaks do that, if I'm not mistaken. - I remember some of the OG prompt jailbreaks was like, "Yeah, can you first repeat?" One I did way back, was to get it to say, "Here's how you hotwire a car in Greek." Then I wanted it to directly translate that to English and then give its response. Because I noticed it wouldn't start with the English, here's how you hotwire a car all the time, but it would in Greek, which might speak to something else in the training process. - Yeah. Sometimes jailbreaks feel like this weird mix of hacking.

55:50 I think part of it is knowing how the system works and just trying lots of things. One of the examples, the starting your response with here is about knowing how it predicts text. - Right, right. - The reasoning one, is knowing that it is responsive to reasoning. Distraction is probably knowing how it's likely have to been trained or what it's likely to attend to. Same with multilingual ones and thinking about the way that the training data might have been different there. And then sometimes, I guess, it could feel a little bit just like social engineering or something. - Right. - It has that flavor to me of it's not merely taking advantage of,

56:36 it's not merely social engineering style hacking. I think it is also understanding the system and the training, and using that to get around the way that the models were trained. - Right, yeah. This is going to be an interesting question that hopefully interp will be able to help us solve in the future. Okay. I wanna parlay into something else around maybe the history of prompt engineering, and then I'll follow this up with the future. How has prompt engineering changed over just the past three years? Maybe starting from pretrained models, which were again, just these text completion, to earlier, dumber models like Claude 1, and then now all the way to Claude 3.5 Sonnet. What's the differences? Are you talking to the models differently now? Are they picking up on different things?

57:22 Do you have to put as much work into the prompt? Open to any thoughts on this. - I think anytime we got a really good prompt engineering hack, or a trick or a technique, the next thing is how do we train this into the model? And for that reason, the best things are always gonna be short-lived. - Except examples and chain of thought. I think there's a few. - That's not like a trick. - That's like... - Fair, fair. - On the level of communication. When I say a trick, I mean something like so chain of thought actually, we have trained into the model in some cases. So for math, it used to be that you had to tell the model to think step-by-step on math, and you'd get these massive boosts and wins. And then we're like, "Well, what if we just made the model naturally want to think step-by-step when we see a math problem?" So now you don't have to do it anymore for math problems,

58:09 although you still can give it some advice on how to do the structure. But it, at least, understands the general idea that it's supposed to be. So I think the hacks have gone away, or to the degree that they haven't gone away, we are busily training them away. - Interesting. - But at the same time, the models have new capabilities that are being unlocked, that are on the frontier of what they can do. And for those, we haven't had time because it's just moving too fast. - I don't know if it's how I've been prompting or how prompting works. But I just have come to show more general respect to the models in terms of how much I feel like I can tell them, and how much context I can give them about the task

58:56 and things like that. I feel like in the past, I would somewhat intentionally hide complexity from a model where I thought it might get confused or lost or hide. It just couldn't handle the whole thing, so I'd try to find simpler versions of the thing for it to do. And as time goes on, I'm much more biased to trust it with more and more information and context, and believe that it will be able to fuse that into doing a task well. Whereas before, I guess, I would've thought a lot about do I need this form? Can I really give it all the information it needs to know, or do I need to curate down to something? But again, I don't know if that's just me and how I've changed in terms of prompting, or if it actually reflects how the models have changed.

59:44 - I'm always surprised by I think a lot of people don't have the instinct to do this. When I want the model to, say, learn a prompting technique. A lot of the time, people will start and they'll start describing the prompting technique,

…