Better Together: .NET Aspire and Semantic Kernel

0:00 [Music] okay uh welcome everyone to this session my name is Steven Sanderson I'm from the net team and today I'm joined by Matthew balanos from the semantic kernel team and we're going to be talking to you about some ways that net Aspire and semantic kernel can work together to give you a really good experience as a developer adding AI features to your application so let's start with what does that even mean what kind of AI features would you want to add to an application that is AI even useful for a typical business app so I think the answer is yes but let's go through an example of it and I'm going to try to give you just about the most

0:46 conventional and relatable example I can think of which is e-commerce so imagine there's an e-commerce site which uh as well as selling products to people also has a Customer Support Facility where customers can along and submit inquiries about products that they have bought or are thinking about buying and these inquiries get sent through to the staff working there and the staff have got some kind of web UI for working with that and it's probably got a big Grid in there in the middle that's perhaps how their workflow works so very typical business app how can we even use AI in a beneficial way here is it useful well yes uh for example we have got a search box so we might start by thinking can we upgrade that to be semantics search so that people can find stuff without

1:32 having to know exactly how it's phrased or without worrying about spelling mistakes we've also got these little titles here that can help staff remember what each of the different tickets is about and navigate around quickly and so it's nice to have an AI system automatically generate those for us we've got these different ticket types and we can use classification to set the right type automatically based on the text which allows you to trigger different workflows if you want to and then we've got these satisfaction rankings and it's very easy for a language model to give a sensible estimate of customer satisfaction so we can do that and that might help staff Focus their attention where it's most needed but that's not all we could also go a bit further and say when someone opens one of these tickets and they

2:17 start working on it they're going to have various Fields they can fill out and just like before we could use AI to automatically classify or enable semantic search we've then got perhaps a big conversation between the staff and the customer and that could take a lot of time to keep rereading each time you open so to save humans the trouble of reading all that stuff what about have ai produced some nice little summaries of that as well and then when it's time to actually answer these questions it's perhaps relevant to have some sort of chat assistant that does Q&A and that can find information from your business data in this case product manuals and of course we need it to be accurate and not hallucinate so we could force it to provide citation for any claims that it makes and then finally when it comes to

3:03 typing a response to the customer we could have a typ Ahad system that suggests phrases that fit in with what the users type typing and that matches all the information that we've already got so overall I think it's clear that there are lots of opportunities for making this sort of application more productive and more beneficial for your users with some AI features and it doesn't even have to be very difficult so let's have a look now at some ways that you can do that at the code level now obviously we can't go through every line of it because we haven't got time for that but I'm going to show you some of the highlights of how Aspire and semantic kernel can work together to enable that sort of thing okay so over here I've got visual studio and it's got my solution in it and this as you can guess is an Aspire application so here's

3:50 the Aspire dashboard and you can see I've got lots of different services like vector databases and other stuff in there the ones that we're going to be focused on just now are the back end where most of the app logic is and a couple of different web uis one for the customer one for the staff so let's start with this staff web UI and you can see it looks kind of like we had in those PowerPoint slides just then uh with all the stuff you expect so you know like semantic search to find all the things related to food and so on let's go into this ticket here you can see it's got quite a long conversation very long I definitely can't be bothered to read all that uh but I don't have to because the AI has produced a nice little summary of what that's all about so let's see in the code how that actually looks how do we generate a

4:36 summary like that is it difficult or not so I'm going to go into the bit of the backend API where messages get posted okay so this C method here runs and you can see this uses normal Entity framework code to save the new message to the database but also each time a message is posted we generate an updated summary how do that work well the main thing is that we're using a service called I chat completion service now that's an interface provided by semantic kernel and if you don't know it semantic kernel is a set of net packages and libraries that give you abstractions of a different AI services and implementations for them so we can have various different language model backends for this chat completion

5:22 service and then we can call any of them without having to change the rest of our code so in this case we get the chat completion service and I'll show you where that comes from in a minute and then we build a prompt that we're going to pass into the language model and we tell it you know what product this is about the brand we emit all of the messages in the the chat history to it and then we tell it what to do we're going to say okay write three summaries for me one uh a 30w summary that we'll use on the ticket Details page a short summary up to eight words that we use on the ticket list and then we'll also tell it to produce a customer score satisfaction and and uh it's quite easily able to do all that we tell it to reply as adjacent object uh which we can then pause and save all that stuff into

6:07 our database all right so it's not too difficult to do this kind of thing and just to show that that works when we do it live let's add a new uh support ticket to our system so I'll go in here I'll say okay I need some uh food for my cats so let's say uh protein energy bars maybe those are good for cats I don't know now because I'm a customer I'm not going to write a short succinct clear message I'm going to write an absolutely massive message that's really hard to understand it's got loads of information in it about my cats and their personality and it's got a poem about cats and all kinds of useless stuff like that and somewhere hidden amongst all that is my actual question so it's going to take ages for the support assistant to even work out what I'm asking them um but actually it doesn't take that long because you see we get this nice little

6:53 summary that's automatically generated energy bars for cats and uh specifically picks out the pertinent question or energy bar safe for feline consumption okay so you can see how that helps now in terms of where this IAT completion service comes from what is it is it open AI is it something else well one of the nice things about Aspire is that it can orchestrate this sort of thing for you and in this case I'm configuring it in my apphost so all the services in my application share the same language model service and I can easily switch it around in this case I've actually set up with a connection string that uh makes it use open AI uh but if I wanted to I could simply swap these two lines of code and now it would use Ola which is a

7:38 way of running language models locally on your development workstation uh so everything happens fully offline now I'm not actually doing that because it runs slower on my laptop but you can if you want to especially if you've got a more powerful GPU and you don't have to make any other code changes okay so that's Aspire and semantic kernel working together and in fact the integration between goes a little more deep than that uh in terms of for example Telemetry if I go into this tracers list here and we have a look at the activity that's been going on one of the things that we'll notice is that it's able to recognize when we've made calls to a chat completion service because semantic kernel will log the Telemetry and it's got all sorts of useful information in there like the number of tokens that were produced and the prompt and the

8:23 response things like that and if you're doing this in production you can aggregate all these statistics and keep track of your costs and the times and that sort of thing okay so that's an example of how we can do a bit of summarization with these Technologies now let's move on to a slightly more advanced thing now which is going to be this Q&A chatbot thing so let's see how that actually works and then we'll look at a bit of code so let's go into another support ticket now so the first problem I notic here is that this customer is trying to talk to me in German and I don't even speak German uh so that could be tricky but fortunately the some summary gives me a a good indication of what they're asking for they want to know the PIN for Bluetooth pairing uh now I don't know what the pin is uh which is why it's helpful to have

9:08 this sort of AI assistant that's capable of searching through business data and finding the relevant things so let's try what does the manual say about this and you can see it does a search for Bluetooth pairing pen and it works out that the default pin is 00000000 okay now do we know if that's really true well we make it give a citation and if I click this we can see it can take us into the actual product manual and highlight the exact basis for ex claim so we know it's actually true all right so then I could say something like write a reply to a reply to the customer saying that okay and it'll write a reply great but of course in this case I need it in German and so it writes a nice little reply to the customer and I can click this button us

9:54 as reply get rid of that that's a burgal fix later hit uh send and it sent the message back to the customer in German cool how did that work well let's go and have a look at a bit of the code for that here's my assistant API and you can see that um it's not super complicated the main thing that it's doing is it's using another one of these IAT completion services and just like before it builds up a prompt uh saying hey I I want you to answer customer support questions here's all the context with the product and so on here's the most recent message from the customer I want you to give give me an answer okay so it will provide an answer and then in terms of being able to search product manuals this is something that semantic kernel makes fairly straightforwards it's got this API where you can uh attach what it

10:41 calls plugins uh which look a bit like this a class that's got methods that are decorated with these Special attributes kernel function okay so in this case I'm saying there's a a call back that the language model can use if it needs more information called search manual and we need it to supply a search phrase and a product ID and then we can put whatever logic we want in there that goes off and gets you know search results from that and in this particular case I'm using a vector database and semantic search which is also provided by semantic kernel but you could do anything else you want you could use a traditional database full teex search it doesn't really matter what you supply some results and then the language model can use that to give an answer okay so that's pretty straightforwards um one

11:27 thing you might notice is the way that this button users reply appears sometimes but not always well that takes us on to this next matter which is classification it's very often that you want to uh keep to determine what sort of class something fits into in this case I want to classify messages as are they suitable for sending to the customer or not so I know whether to show this button and there are a couple of ways you can do it the first way I'll show you is just simply by using a language model and I chat completion service again so what I'm doing here is after we've sent the response we send a further message to the language model saying consider that answer you just gave and decide whether it's suitable to send to the customer or not and give me back ajacent object with a Boolean and

12:13 then based on that I'll decide whether or not to show that button so that's all that's going on there um but there are other cases where you might not want to use a language model for classification uh perhaps because you want to make it faster and cheaper still so language models are brilliant they're very powerful uh but also take a bit of time it might take you know hundreds of milliseconds or even seconds to answer your questions and if you've got a high performance highspeed workflow system you might want to classify things faster and cheaper than that so I'm going to show you a way of doing that um and slightly surprisingly I'm not even going to use net I'm actually going to use Python you might wonder why I'm doing that in a net event and the reason is not because I think you must use python in fact you can certainly do this with net the point is to show you that Aspire is very good for orchestration uh not

12:59 just with net not just with Docker containers but with other Technologies as well so if you want to bring some python or Java or JavaScript or whatever else into your aspire application you can do that and I'm going to do a bit of of classification with python here it's very straightforwards so in Visual Studio I've added this python project and you can see I'm constructing a fast API API server now if you don't know python don't worry about it this is a bit like minimal API for Python and I've got a couple of end points including this classifier and the classif is going to take in some text and a set of candidate labels and it will determine which is the most relevant uh label for the text okay and it's doing so using this uh local model called mini lm2 which it gets from hugging face which is a third party system and so

13:45 that's able to do this classification very fast locally just on my development machine or on your server and to show you that really working uh clearly I'm going to do it manually on the command line so if I get this text this is a python API and the candidate labels or animals programming or music you'll see it instantly classifies it as programming whereas if I change the string to this as a python it classifies it as animals all right so uh what I want to classify in this case is the incoming tickets question comment complaint or returns which is the most relevant label and you can see that when I did this thing about cats it was classified as a question because it is a question uh but if I go back as a customer and I say uh I need to return stuff or something else like that

14:32 then when we look at the list um you'll see that this has now automatically been classified as returns so we could trigger different workflows based on that and Aspire is fully capable of starting up things like python or Java or whatever else you want to use within your system all right so that's a couple of ways to do classification the last thing that I'm going to have time to show you is evaluation and in some ways this is kind of the most important one because this is the what makes the difference between just kind of like hoping that things work and actually measuring that they work and being an actual professional so I know that your professionals you want to be able to know that your stuff works and in fact how well it works so let's look at how we can quantify the quality of this AI assistant I want to reduce the whole thing to a single number that says how

15:19 good it is right now how can we do that well the main way that people do this sort of uh development time evaluation is by having a set of uh evaluation data a set of inputs and desired outputs and then we can test it against the inputs and see how well it matches the desired outputs all right so what how are we going to do that how would we get sample data well in fact all the data that you've seen it so far like all the product data the product manuals the customer support tickets all of that stuff has been generated through AI for the purposes of test data using this data generator project and if you get the code for this later you can see how that works and one of the things that it produces is a set of evaluation questions which in this case is a list

16:05 of hundreds and hundreds maybe thousands of questions and answers for example for product 35 what is included the answer is this stuff okay now we can take this as being the ground truth because this is the data that we generate first and based on this we uh we build all the product data and manuals and search out of it so for that reason we can consider this to be the truth so what we want to do now is ask our backend system all of these questions and test how well it answers them so what I'm going to do now is I'm just going to start up my uh application in the background here and that's running now and I'm going to use a little console tool to run an evaluation Loop now you could imagine

16:51 that we had a really nice UI for this and maybe at some point in the future someone will create one but right now we don't so just use your imagination because all I've got right now is a console app and this console app let me just start it running okay that is using some Aspire libraries to be able to call to discover and call into the backend service from this console application and it's looping through all those eval questions putting them into groups and then it's sending them in parallel to the back end to say you know here's a question here's another question here's another question and collecting all the answers and then it scores the quality of the answers and it does that by making another call to a language model to say how accurate is this and how relevant is this based on what the truth is and we get back a number and we can calculate the average

17:36 of that over time so we're going through right now and if you I I know the text is a bit small but at the moment it's averaging at about 75 um over time I happen to know from experience that in this setup it's going to average about 08 or something like that but the exact number doesn't really matter what matters is that you want to find ways of making the number go up by making changes so you could try changing the prompt or the assistant logic or the vector database or the chunking or any other thing that's going on in this system and then run the evaluator to see whether the quality goes up or down uh to give you a realistic example of that at an early stage when we were building this we noticed that it was hallucinating quite a lot and we would ask it questions and instead of looking at the product manual it just made up an

18:22 answer based on nothing just a pure hallucination and at that time the score was about 61 so we realized that we needed to look at the manuals more so we added to the the um prompt this one extra sentence if this is a question about the product we really want you to look at the product manual and just by adding that one sentence the score went up to 77 so clearly it's a good Improvement but it also made the time go up from two seconds to 2.6 seconds which you know also makes sense because spends more time looking at manuals and doing extra calls now but the point of this is that you the developer can now make tradeoffs in a principled manner you can make changes see whether the scores go up or down whether the times go up or down or the costs and decide whether these changes are good or bad okay so

19:08 that's all I'm going to have time to show you I'm going to hand over to Matthew in a second but just to summarize from me uh there are many different things you can do to add AI features to your app uh chat is one of them but I don't think I would recommend starting with that I think that if you're starting you should look at things like semantic search summarization translation classification all these kinds of things are much much easier to build and they can allow you to add some real value to your application over a fairly short period so I'd really recommend starting with some of those things although chat of course may be useful depending on your scenario uh I do think it's worth a bit of investment in having an evaluation system from the start so that you can check whether what you do is improving or getting worse over time I think as a net developer you've got some great

19:54 opportunities with a spire and semantic kernel uh and the way that they work really well together uh but also be open to other things you know it's an emerging field right now and all the recommendations we give are going to change over time people come up with new patterns for evaluation or testing or whatever else so be flexible and consider using whatever is best in your scenario now if you want the code that I've been using in this uh demo uh that should hopefully be available by the time you see this at uh the URL that I show right there uh so you can check that out and hopefully see how much of this stuff work works on the inside so that's it from me I'm going to hand you over from to Matthew now who's going to take you to the next level with your AI understanding thank you Steve I love

20:40 that demo because it shows how you can actually drive real productivity with your existing applications with AI um that's what it's all about right not just building shiny applications how do you actually make your own customers your own employees your own company more productive now what I'm going to be talking about is kind of what's coming next Steve kind of mentioned all this stuff is still emerging there's all these new things coming out what should you expect in the AI space and how are we going to help you within this Mantic kernel team in yournet language so what I'm here to kind of share with all of you today is what we are working on within semantic kernel to wrap that chat completion service that Steve was

21:25 talking about in an additional abstraction make it easier to actually talk back and forth with the chat completion service if you've been plugged into what's happening in the AI space you've probably heard this word agent pop up over and over and over again what we've done in semantic kernel just a a few days ago maybe a week ago we released our new agent framework for semantic konel so I I could talk on and on about what an agent is how it works but I always find it's easiest best to dive into actual code uh to see what this new abstraction does so what I have right now is a Jupiter notebook um which

22:12 has some very simple semantic kernel code basically I'm running a uh terminal application a console application and let's just go ahead and get started by loading up all of our different packages uh and importing these statements and this then we'll dive into what is an agent in semantic kernel and why it's valuable for you so once these uh packages are installed we'll go ahead and import all of our statements and as Steve showed you can create these we call them plugins inside a semantic kernel that give the AI the ability to either retrieve information to interact with the real world or in this case very

22:58 simply uh getting the current time now in Steve example he was using the chat completion service directly and making calls to it what we found though is most kind of those back and forth chats follow a similar pattern right you have a chat history you build it up you give it to the LM it gives you response back so what we believed and what the the industry has seen is if you can wrap this in another abstraction you one you make your code a little bit easier and you get a little bit more predictable way of working with these what we call agents so here what I'm doing we'll go ahead and click run is I'm building a kernel I'm adding in one of those chat completion services that uh you saw

23:45 earlier we're adding in that plugin so that the AI knows how to tell the current time but what's new what's new uh in within manac Colonel is we can now create what we call a chat completion agent we can now now create a chat completion agent where we can give it a name we can give it instructions uh so you don't have to Bubble or you don't have to combine all of that inside of a single uh prompt you can just tell it and then you can start a chat history and that instruction set will always be prepended to whatever to whatever that agent is working on we start our chat it's asking for our input we can ask simply what is the current time

24:30 and the AI assistant is able to give us an answer okay not a whole lot new over just using the chat completion interface uh by itself but what we're about to see is this also gives you the ability to use services that act as agents right so instead of having to build everything yourself with I trat completion you can talk to something like the Azure openai assistance API which if you're not familiar is a service that basically manages the thread on behalf of you so that you as a developer don't have to figure out where are you going to store that chat history how am I going to save that to something like Cosmos to be that's all managed for

25:17 you using the assistance API so we'll go ahead and play this one as well we're starting the same way we're creating our kernel uh this time we're creating an open AI assistant and what's new is instead of me creating a chat history object I can ask the service to go ahead and create a chat history for me or what they call a thread so that when I have this back and forth conversation I don't have to worry about saving that information so I can ask the same question of what is the current time and it'll work the same way but I'm actually not going to do that quite yet the other thing that's really valuable about the assistance API is it also comes with built-in tools or built-in plugins so I can comment out

26:03 this line that gives it the ability to check the time and I'll uh uncomment this one enable code interpreter what this is providing the AI is the ability to basically write python code basically do a bunch of operations in the side uh to answer any users's question and naturally python has some tools some coding that it can use to tell the current time so when I ask what is the current time what it's going to do behind the scenes is instead of using my plugin it'll actually use code interpreter to write some simple python code we actually see it right now it'll execute it and tell me what the time is okay we'll go ahead and say

26:49 bye now the last thing I want to show that's really demonstrates the value of this additional abstraction the value of having agents is with a common contract for how these agents work together it's now possible to put them in something like a group chat or a process and have them use the same interface to talk back and forth and we're going to see just that this was popularized and started with a research project called autogen and we've basically taken those best Concepts the group chat the agents into semantic kernel and made them Enterprise ready so what we're seeing here is we're going ahead and create two agents instead of just one an art director and a writer

27:35 you can see we have all these different instructions that we're providing it um what they can do what they can't do don't waste time and we ultimately want to be able to create a group chat where we include these two agents the writer agent and the reviewer but there needs to be some sort of way of controlling the chat who gets to speak next when is the chat over in this particular scenario we're going to be having the agent reviewer basically determine when is this chat over when is the writer done writing something that is of good enough quality right and once it has achieved that this chat will be over and the user will have that final answer so I'm going to go ahead and uh

28:20 invoke this and what I am asking these two AIS to do these two agents is to come up with some marketing copy for uh this concept a way for multiple AI agents to collaborate on a single task exactly what they're doing right now uh we can see that first message and it starts off with the copywriter hey let's let's try this AI conductor orchestrating brains uh the art director is like uh no I don't like this it's it's interesting but it's a bit abstract can we refine it right um and so uh it tries again um and then it finally is able to approve the this final message uh so that in this case a user can use it for their their marketing you can

29:05 imagine in the example that Steve showed that maybe the summaries aren't that great or maybe the responses back to the customer aren't uh high enough quality what we found through research is having multiple agents kind of battle it out right challenge each other uh bring with each other different uh perspectives or system instructions can actually make that final result even better right but but um this isn't the end all Beall you can imagine these things going back and forth getting in Loops um uh just talking is frankly not enough for these AIS to always complete work right and so the last thing I kind of want to

29:50 leave all of you with is what is coming even further with Samantha colel what else you should expect from the rest of the industry and that's really taking what we've learned through multiple agents working together these multi-agent patterns and making them a little bit more predictable more deterministic so that you always get a result that you expect from your multiple agents and you kind of saw this in uh Steve's demo uh when they had to do that classification logic when he had to show that reply button he basically wrote some code that says hey reply and then check if a reply button needs to be involved it's very simple it's like do step one and then do step

30:36 two you can imagine though as you build more complex business processes these things become unwieldy and so different Frameworks whether it's autogen or crew aai have started building easier ways of abstracting agents and allowing them to create business processes and follow them uh and and also include humans in the loop so get excited for what's coming next for semantic kernel. net Aspire um it's an exciting time to be a developer a ton of new things are on the horizon um and with that I'll kind of conclude my section on agents and I'll go ahead and pass it back to Steve and we can wrap things

31:22 up yeah I think that's that's all from the two of us so we hope that is of use to you and that you will give some of this a try and quite importantly let us know how you get on we want to learn from you as well so thank you very much enjoy the rest of this event