Brief Summary
The presentation explains the difference between prompt crafting and prompt engineering, emphasizing the importance of creating replicable and reliable outputs through structured frameworks. It explores various prompting techniques like zero-shot, multi-shot, and chain-of-thought prompting, highlighting their benefits and drawbacks using a practical task of extracting and classifying data. The presentation also covers prompt chaining for complex tasks and concludes with tips for effective prompt creation, such as simplicity, directness, and relevance.
- Prompt crafting is real-time interaction with a model, while prompt engineering focuses on replicable and reliable outputs.
- Techniques like multi-shot prompting and chain-of-thought prompting enhance the accuracy and nuance of model responses.
- Prompt chaining is useful for complex tasks by breaking them down into multiple steps.
- Effective prompts should be simple, direct, unambiguous, and relevant.
The presenter introduces the concept of smart prompting and its role in generating smart outputs, aiming to provide practical tips and techniques applicable to prompt creation. She works at autogen, where they assist organizations in crafting winning bids and proposals using large language models and linguistic engineering. She addresses the common questions about prompt engineering and clarifies her role in the field.
The discussion differentiates between prompt crafting and prompt engineering. Prompt crafting involves real-time interaction with a model, providing a prompt for a specific instance. While it can yield useful responses, the prompt's effectiveness isn't guaranteed across different texts or contexts. Prompt engineering, on the other hand, focuses on curating prompts that produce replicable and reliable outputs for a specific function. It involves continuous measurement and improvement, establishing frameworks that can scale effectively with any input.
The presenter introduces several popular prompting techniques, including zero-shot, multi-shot, and chain-of-thought prompting. These techniques are demonstrated through the task of extracting and classifying data using the autogen platform. Zero-shot prompting, which involves providing an instruction without examples, is a common starting point but may lack nuanced understanding. Multi-shot prompting offers examples to provide more context, while chain-of-thought prompting asks the model to explain its reasoning step by step.
The presenter demonstrates the prompting techniques with a classification task. In the zero-shot example, the model fails to capture the nuanced sentiment of a statement. Multi-shot prompting, by providing examples of positive, negative, and neutral statements, yields more accurate and nuanced outputs. However, it's important to be aware of potential biases in multi-shot prompting, ensuring that examples cover a wide range of interpretations. Chain of thought prompting aids in model debugging by revealing the model's thought process.
Prompt chaining, or multi-step prompting, is introduced as a method for handling complex reasoning tasks that cannot be instructed in one go. This technique involves breaking down a complex task into several steps to ensure the best piece of text is worked on at each stage, reducing model inconsistency and preventing conflicting instructions from interfering with each other. For example, analyzing sentiment on a large body of text can be broken down into classifying statements, extracting themes, and grouping those themes.
The presenter shows the output of the first prompt, which classifies customer feedback into sentiments using a multi-shot prompt on a larger dataset. The second prompt, a zero-shot prompt, identifies all the themes in the customer feedback, emphasizing the importance of repetition and providing context to the model. The third prompt classifies the themes into positive, negative, neutral, or other categories, providing a justification for each theme's sentiment. This multi-step approach yields more nuanced, accurate, and useful results than using a single technique.
The presenter concludes that simplicity is key, and while single-shot prompts may not always be nuanced enough, they are often the best choice. Once a satisfactory output is achieved, the possibilities are endless, including translating the output into JSON, adjusting the tone, or converting it into a PowerPoint presentation. Effective prompts should be direct, unambiguous, and relevant, with the techniques discussed serving to ensure these requirements are met.
In a Q&A segment, the presenter addresses how to refine prompts, noting that models can be prompted to refine prompts. While models can provide a good initial framework, it's important to consider the specific use case, target audience, and subjective opinion when writing prompts. Other models can be used to generate a first draft of a good prompt to improve one's own prompts.
Summarize any YouTube video in seconds
Get summaries like this one for any video, in your language, with the free Summ app.
Transcript
Full transcript of “An AI Prompt Engineer Shares Her Secrets”. Click a timestamp to open the video at that moment. Need another video? Use the YouTube transcript generator.
0:00 so in the short time I have with you all today I want to demonstrate how smart prompting leads to Smart outputs and hopefully give you a few practical tips and techniques that you can take away and use for your own prompt creation process so I work at autogen where we help organizations around the world write more winning bids tenders and proposals by leveraging large language models and linguistic engineering I get a lot of questions around what prompt engineering actually is and what I do in my role so I thought it' be useful to clarify that can I get a quick show of hands who here has used chat GPT or any other similar platform for their prompting wonderful everyone um perfect so what most of you have been doing in those instances is called prompt crafting prompt crafting is when you
0:47 interact Real Time with a model and give it a prompt for that individual instance you receive useful and relevant responses but you wouldn't necessarily expect that prompt to work on any other piece of text that anyone else would use so prompt engineering is curating prompts which produce replicable reliable outputs to fulfill a specific function whilst continuously objectively measuring and improving them it's about setting up Frameworks that scale well in the future with any unknown input all the time there are many prompting techniques out there but these are some of the most popular today I will demonstrate a few of these as we consider their benefits and their drawbacks and we'll do this by focusing on one task which is extracting and classifying things from a data
1:35 set I'll be using autogen eyes platform today to demonstrate these prompts so the feature that I'll use it offers multiple options um of outputs for each prompt so you'll see three options on the right hand side up here we have a zero shot prompt it's an instruction with no examples it's what everyone does the first time they uh interact with the larg language model it works well most of the time but it does have a few drawbacks sometimes it can lack a nuanced understanding of the task we're trying to achieve and we'll see that in practice today so in this example I've Ved to classify uh a piece of text and I've that piece of text has a debatable output so as a human I would expect the product arrived late but the quality is
2:22 excellent to generally be a positive statement although it has some positive and some negatives in it it's weighted heavier on the positive statement we can see that the model has uh given neutral for all three outputs so for this case zeros shot prompting doesn't encourage that nuanced understanding and I probably wouldn't trust it for even larger sets of data here the model lacks understanding of what I mean by each of those sentiments what does positive mean for me what does neutral mean for me in the previous example I didn't have enough context so one way of providing it with more context is offering examples of what you want this this is called multi-shot prompting a shot being an example Chain of Thought prompting means asking the model to think step by
3:07 step and show its reasoning so I've given it pretty much the same prompt but I've given it three examples of what I think a positive negative and neutral statement is you can see as we uh transform the text the outputs are far more Nuance the first one says positive the second one after explaining its uh uh chain of thought where I've instructed it to think step by step also concludes to be positive in the end it's far closer to what we're looking for one thing I will say is that you should be wary of bias with multi-shop prompting in this example the model might take it to mean that positive statement must always talk about product quality or it always must mention that the site was confusing or that it always has to have one positive and one
3:53 negative side to it if you're using multi-shot prompting for larger sets of data you have to make sure your examples cover all bases this can be quite difficult to do as you have to think of every way that something can be interpreted Chain of Thought also helps in this scenario because if I can see the model's thought process I can see where it went wrong and it can help with my model debugging multishot here was really great for improving this task and giving the model some more understanding of my intention but sometimes I want to do something a bit more complicated and I don't want it to give me just a one-word classification for something let's look at how we can introduce even more complexity with some prompting techniques prompt chaining or multi-step prompting is best for complex reasoning tasks that cannot be instructed in one go it
4:39 ensures you're working on the best piece of text at each stage and it doesn't leave room for model inconsistency it makes sure that the potentially conflicting instructions don't interfere with one another let's say I want a more complex analysis of some sentiment on a larger body of text as a human I might break this process down into a few different steps first classifying statements as a whole then extracting themes from the statement and then grouping those themes this type of breakdown also works great with prompting so for the sake of brevity I'm just showing the output of the first prompt that I gave the model the uh The Prompt instructed it to classify a list of customer feedback into sentiments using the same multishot prompt that I showed you guys before except on a larger set of data there's around 25 26
5:25 pieces of customer feedback here this is the first prompt in this chain the second prompt in the chain is a really good zero shot prompt up at the top here you can see we've asked it to identify all the themes in the following list of customer feedback your response should be an exhaustive list of accurate and relevant themes with a number beside them indicating how many times it appeared do not write the same theme out twice feedback colon so I think it's important to note uh some key parts of that prompt so I've asked it to identify all the themes and then I've repeated myself and said it should be an exhaustive list repetition is always good I've said that it's a list of customer feedback which provides
6:11 the model with context of what I'm talking about I've asked it to be accurate and relevant so it's clear that I want the themes only to do with customer feedback and then I've said how I want the structure to look you can see with all of the options the structure is exactly how I want it to look the answers are exactly what I'm looking for so I'm happy with this and we bring it over into the editor so as we move on to the third prompt in the chain you can see how with each step the output is a combination of everything that came before it context which is implicitly woven into the answer gets carried onto the next prompt The Prompt that I've given it here the third prompt in the chain is classify the following themes into positive negative neutral or other categories the response should have the
6:57 sentiment as a heading followed by the themes which fall under that sentiment under each theme Write a brief justification of why it falls under that sentiment and then I've given the list of themes as you can see on the right hand side this is far more nuanced accurate and useful than the first uh thing we looked at this is this output is actually a combination of all the techniques we've looked at so far and we could not have gotten this type of information which is this thorough and this accurate through only using one of the techniques once you you've got an output that you're happy with the possibilities really are endless you can translate it into Jason you can play around with a tone you can turn it into a PowerPoint
7:43 presentation and much much more so to conclude this short segment Simplicity is always best although a single shot prompt wasn't nuanced or accurate enough for our example today it is most often the best choice the last three examples I showed you of turning it into Jason or translating the tone all of those used a really good successful zero shot prompt a prompt should be direct unambiguous and relevant and each of the techniques we've gone through today are just different ways of ensuring that the prompt meets those requirements I really hope this offered some insight into what prompt engineering actually is and gave you some tools that you can take away and Implement into your prompt crafting I think we might have time for one quick question if there are any
8:29 questions in the audience yes we have someone here how do how do you think about it and do you use any tools to refine your prompts and how yeah yeah so that's that's actually a really interesting question um you can prompt models with how you want them to refine your prompts interestingly so you have to have the techniques in order to instruct the models and how you want your prompt to look anyway sometimes if I'm lacking inspiration on how to begin writing a prompt I will definitely use a
9:15 model I'll be direct unambiguous relevant clear about what my parameters are clear about what my instructions are and often it gives me really good uh framework the only problem with models are it's sometimes a bit more Nuance in that and you're looking to write prompt for a specific use case so I know my target audience I know my my customers and I know what they're looking for and I can try put that into prompting but ultimately it's your subjective opinion on whether you think it meets these metrics that you set out so yes absolutely you can use other models not only clae but any of the providers um to give you a first draft of a good prompt and to better your own prompts awesome I think that's all the time we have today but if you have any more questions I'll be around afterwards
10:00 and please feel free to add me on LinkedIn and ask me any questions there thank you so much
