.NET AI Community Standup - AI in .NET: What’s New, What’s Next

0:24 Oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh. Heat. Hey. Hey. Hey. Heat. Heat. N.

1:38 Hey, hello everyone. Welcome. I have a little scary moment. Literally everything crashed when I was playing the intro video. So I close off the browser. I'm open everything here. I see the countdown. Three, two, one. But hey, here we are. Hello everyone. Welcome to another session from .NET and AI. My name is Bruno Capuano, part of the cloud advocate team. I'm here with Tanner. And hey, how are you Tanner? I'm doing pretty good, thanks. It's a pleasure to have you here to also talk about something super cool. We are going to get there later. Do you want to introduce yourself? I'm I know I'm the worst host ever. I need to invite people to talk about themsel introduce. Please, Tanner, tell you a little about yourself, please. Uh yeah, so uh I'm

2:23 Tanner Gooding. I'm on the net libraries team. Uh and I am one of the owners and maintainers for uh all things numeric uh math hardware intrinsics uh and tensor related innet right now. Hey, that's that's super cool. That's super cool. And this is what we are going to talk today a little about. Not a little, we are going to talk about literally that tensson and more. But before going there, let me let me talk a little about something super cool that should be live literally right now. And this is the this is a repo that we have with this called awesome GitHub compilers customization. There is go I think that the net blog is coming now. Literally now I need to refresh the net

3:09 blog and probably going to be here. Ctrl F5 not yet a little more but the whole idea is that we have a repo where we are going to start to collect and everybody's invited also to participate make your pull request and more uh to collect custom instruction reusable prompt custom chat models and more and it's super cool. I really really like it. There is a set of a specific instruction example if you are working with ASP.NET net that you can put in your repo or in your local VS code or visual studio and it's going to be part of the whole GitHub copilot experience where we you are chatting doing agent mode and more. So take a look at this. I am going to share the link in the in the chat now and it's part of what we've

3:55 been doing these days. I think it was on Monday where we announced that hey the GitHub copilot chat is open source also from the VS code. So everybody have access there and more. We really really want to give the people the chance to know these tools, extend these tools, learn more and do this and do more. So if someone is interested, hey, let me I'm going to share the link and take a look. There are a lot of these and I am not using them per se, but I pick up a little from the bicep one and a couple of others into a main big one that I create for the repo and it works and it really help Copilot to work better there. That said, as I said, I am taking a look at the comment right now. So, hello everyone. This is me. So, hello Radies. I'm going to go for the full

4:42 list. I also see some familiar names here. T probably if this is the T that I'm thinking probably coming soon with a super amazing topic and more. So, besides this, the second one that I want to share is remember this. We built this last time and I know kind of by coding creating this space battle game or whatever. So the code is already there. If someone wants to take a look at this, you have access to the full video and this is the main prompt that we created literally with one prompt did 80% of the game then the little iteration there. The final code is available in the Zenai in the generative AI for.net repo. So you can find everything here and it's also have the current uh version that it

5:30 can work with local models with a llama or with AI models on the cloud. So you can even run the game. Let me see if I can run the game. This is the game here. Let me open my GPU there. And if I start to play the game and I play the game with lama. Let me play the game with lama. There it is. It's playing with lama. We are going to start to see here how every frame, every movement and everything else is processed locally. I think I am using 54 mini for this which is very good, very nice. So if someone wants to take a look at this about also how you can play and do more, it's not very smart. It's kind of hiding in the bottom right corner. Come on, man. You need to

6:16 go and and kill the enemies and shoot the enemies. Okay. No, it's stuck there. It's fine. Which is it makes sense. It's not probably my main goal was stay alive when I create the prompt and this is what it's doing. But hey, if someone wants to take a look at this, it's literally uh again open source there and let me take a look at the lama. Yes, five for mini also a great chance if you want to implement this using AI foundry local which I have in my to-do this list or test with other models we have all of this so that I am going to go back to talk with Tanner about tensors numbers and more and the current

7:01 state of stuff that we have inn net what can you tell me can you tell ask about these Annette. Uh yeah. Uh so before that, I just wanted to say that I think both of the things you just showed off are pretty cool. Um uh I got into development originally because uh I wanted to make games and I know a lot of other people that got into development for this for similar reasons. So it's really nice to see new ways that people can get started on trying to build things like their first game or integrating new .NET features into it. Yes. And hey, you know what? talking with people like you which it came from the with kind of a game background. The way that when we created the game, it was very simple. We basically have a large language model. We make a

7:48 screenshot of the the current frame of the game. We give it to the model and say what I should do next. In this game is very easy. You can move right, move left, stop or fire. So there are no match options. But someone told me, hey, these models are great. These models can do more than that. You can not only pass the current frame, but if you pass the current frame and the two previews, the model will infer, okay, this is how the ships are moving. This is how the bullets are going up or down. This is how you are moving, you as the main player. So, it's going to give you a better reasoning, a better option, a better next action with some reasoning. And it really really make a difference when we have another video where we explain how to do that. And it's great.

8:33 And also initially when we started to do this I take a screenshot of the console literally and save an image a PNG image and I drop the image to the model. But someone told me that's heavy that's processing but that's heavy. You can store if you want the string because it's a console application and pass the string literally space space X space a space the the representation of the screen what I am drawing in the console to the model and oh my god it's starting to work probably 20 time faster and it's a learning process again with excuse to do a simple game I never did games I'm not smart enough and then you can do something learn and more it's kind of nice to do yeah it's I I think It's a lot like uh anytime you're trying to

9:19 learn or build something new, you have to start from somewhere and everyone needs help. And uh you know AI is just another tool in your toolbox that can be used to help get you started. You still have to augment it with your own reasoning. you still need to uh do trust and validation processes, but like using a calculator when you're learning math to help assert that you did the right steps and came to the right uh end result, it can just help accelerate and help provide additional validation and input and it's just another tool to help you succeed. Yes. Yes. And you you nailed it and that's the way that we are using AI. It's basically literally a copilot side by side helping us and then taking a

10:05 look and for me it's a learning experience. Oh, I can do this and I can do that. That said, I'm also aware that sometimes we have these crazy suggestions of stuff that doesn't make sense and more but also learning because the model have some context. They have been training with ABCs. They know what they know and they so it's a it's a full learning process. But if you told me I don't know last year that I was not probably last year but two years ago, three years ago that I in 10 minutes I will have a fully functional game like this ready to go with minor touches here and there. I say no. Now you have it and you can start to iterate and do and learn and more. I I like it. It's part of the process. I see someone asking a question here about the repo. So this is it. Sorry I hiding this one.

10:52 So, and the the repo if you want to take a look at the code and more, this is aka.msjiet. This is where we put everything that that we have there. So, that's that's the one that you can find the mega prompt to create the the game and also a final version with all of the included version to play locally with lama AI foundry more. And we use Microsoft extension for AI but feel free if you want to extend this with semantic kernel or whatever there it is and I think I using for the lama part I can't remember but yes that's cool and yes AI is here to help. So this is also an excuse to also to talk about how we are providing these new

11:39 features into the framework and more how is do do how how we are doing that. I I always like to see you and the other product team where they have this amazing discussion about the future of net how we are doing and more. So how do we get here? Yeah. So um as with anything it's kind of this iterative process and you can't just put everything in box. You can't ship everything with the product. There's there's limited resources. There's limited time and what ends up being crucial to the future of programming changes over time. Yes. Uh most often when you start with anything, it is this kind of niche domain uh where the people

12:27 who use it may have this broad ecosystem within themselves, but outside of that uh ecosystem, you don't really see it much. And that's kind of where we were with uh things like AI or science or or scientific computing or even deep math learning, you know, 15 years ago. You had some semblance of things, but it wasn't really core or crucial at that point. And so you had all of these ecosystems that were building up their own individual products. you had uh uh Python which really took off in in this ecosystem uh when they started with NumPy and expanded that into uh eventually PyTorch. Um but that was really modeled and designed

13:14 for and around Python and every language is different and unique. The way you want to work with something in Python is is a little bit different from how you want to work with it innet or how you might work with it in Rust or C or C++. Each language has its own nuances of what works well for that ecosystem in that language based on its own rules and paradigms and everything else. It's it's very similar to a spoken language and word ordering. Uh, like how in English everyone knows that it's the big green dragon, not the green big dragon. But most people, unless they're an English major, couldn't tell you the reason that ordering is the correct ordering. Um, and in other languages, uh, they have

14:01 slightly different orderings depending on their own semantics. And so, you know, it's very similar here. um when a language likenet starts working with that eco uh starts moving into one of those niche ecosystems uh typically that starts by providing bindings over the underlying library. So for years there's been projects like Torch Sharp which provides bindings for PyTorch uh and allows you to it to access and work with the PyTorch APIs in .NET. Uh but it's very much still oriented around Python. Uh a lot of the uh idiomisms and

14:47 everything else don't quite line up with the normal net look and feel. And so there's a little bit of uh give and take as people bring it up. Uh within the past few years, it's become fairly obvious that ML uh and AI is going to be a big part of programming moving forward. It is more and more important that everything provide a way to interact and access it. Uh and so we started looking at ways we could do that. And we can't just go and do that and shove out everything that's existed for the past 15 years. We can't just say, "Oh, you don't need to use uh Torch Sharp anymore." There's always going to be code bases that just can't move off of it. They need to keep supporting it. They need to keep working. So, you need

15:34 to build something that works with both the old and the new. Uh, and generally the way that works is the Net libraries team starts looking at can't we provide some kind of interchange type or contract that allows people to uh take the types from one library and use them more generally and start writing algorithms that are general purpose and work with multiple types. This is kind of similar to uh when we introduced span of t a few years ago. span of t was the interchange type between both managed data and unmanaged data. So you could now have one API that would work with unsafe pointers or that worked with regular arrays. And over

16:21 time we have since expanded that to allow getting spans from more types of data. Uh there are APIs uh hidden off because there are some nuances and dangers that you need to consider when using them, but there are APIs to do things like get a span from a list of t. There are APIs to go and get a span from a string or all these other types. And so span is really that interchange type that brings together all of the different collection types that we had over the 20 years prior. Um and so we start looking at how we can do the same thing for tensors. One of the obvious things you can look at is something like an interface that every type can go and implement. But there's some nuances with that. Some

17:06 people are using uh they're using they're using unmanaged allocations and so there's lifetime considerations. Some people are using uh strrus where they don't want to incur boxing overhead. Um and so while interfaces can be a good option, they they aren't necessarily the only option. Um and so what we ultimately ended up coming to is that tensors are fundamentally uh even though it's you know this this scary ML related word to many people, they're actually really just arrays. uh they're effectively this concept that says uh I have data and I have a shape to that data. So a

17:52 normal array has a shape where it's just a linear list of values. And then you've got multi-dimensional arrays where you can say, you know, I've got an image and it's got 1920x 1080 pixels. Um, and tensors are kind of this abstraction over that that says I can have my data and that data can have shape and that could be something like an image. It could be multiple images. It could be uh basically any kind of shape that you want it to be uh up to n dimensions you can define what n is. Uh and with that we started thinking well we already have span as a way for representing one-dimensional data abstractly and so we started building

18:39 our new type which is called tensor span of t um which is effectively span but it also tracks the shape of the data that you're working with. So now you can work with multi-dimensional data. And because span has worked so well as the general purpose interchange type, we expect that tensor span will be able to fill the same purpose for this multi-dimensional data that you're working with in AI, ML, image processing, games, and many other scenarios. We do also still have an interface that you're able to use and implement on your types for when you do have more concrete things. And we expect that various libraries will provide their own ways to get a tensor span or implement the interface over time.

19:26 And one question because I I've been doing net for a very very long time. I I always put my age on the table. People know I am I am old. The light is good because you can't see my white hair right now. So I will these lights are very good. But in in my 20 plus years of development and more I switched from C# to C++ then do did a lot of machine learning with Python and more and the tensor scenario that you mention I will say that are very specific are very I in the net community the whole net community where we build big enterprise applications and more I know that there are scenarios for tensors but I will say that not everybody's going to use

20:11 tensors alike like we use I don't know a string builder in example that is probably everywhere that said they're also very important because having those for those scenarios which are probably not very popular but they are there it's really really a key part of what we when we need to do some stuff there I really appreciate the work that the team always been doing trying to keep us to to to fill all of these gaps big gaps or small gaps but to fill the gaps there But do you know do do you have the sense do you have the the feeling and this is kind of a personal question that there is a market for this or we are going to put the span the tensor span and one in 10 people are going to use it and then it's going to be those stuff that are part of

20:57 the framework and are not very popular or where do you think I don't know maybe in gaming okay people who are doing gaming and net they may have some nice advantage now that we are starting to get there. Do you have any view about that? Yeah. So, yes. So, I think that the important thing to remember is like I just described it, tensors are just an abstraction over data. Yes. And so, while one of the primary use cases is in things like AI and machine learning, people have actually been using tensors for years, they've just called it something different. So, arrays Yeah. Arrays themselves are tensors. They're just limited to being one-dimensional tensors. Anytime you

21:45 open an image, you are working with a two-dimensional tensor. You are working with a 2D set of data. So, you've got data that that has a shape, both a width and a height, and so that makes it a tensor. Um, when you're working with things like text, you are, you know, you can view that text as a one-dimensional set of characters. And so you have this one-dimensional tensor, but you could also view it as a 2D set of data if you were to say, I have this sequence of characters and these separators such as whitespace that break apart the words. Yeah. And so you have now defined a shape to your data to help process it

22:33 as tokens to be able to better understand the intent or meaning behind that. And so compilers have fundamentally been working and doing tensor like work for years when they are doing tokenization, syntax tree formation, transformations, and everything else. even just opening your text editor and doing a search through it to find a match for a word or doing a red reg x find and replace you are fundamentally doing the types of operations that tensors underpin. It's just we've called it something different. And so, uh, really I think I I think while some things will continue to be very niche and specialized and not

23:20 everyone's going to go and immediately grab a tensor and start working with the data in the low-level foundations, you're going to find that tensors underpin everything in the code that we do. And because Tensor isn't oriented just towards AI and ML, it's really about just performing mathematical operations on data, you end up using it to accelerate all of your workloads. You can use it for things as simple as I want to compute the sum of all the values across this data set. So you know you're doing that when you do innumerable sum today in link or when you're saying find me the max value in this data set. All of those are things people already use with link today. They're just doing it with

24:06 one-dimensional data and tensor makes it easier to give shape to that data and make it a little bit more powerful. Hey makes sense. And yes, I haven't seen about that. But yes, probably when I am using some entity framework functions or whatever to make my life easier work easier working with my product database deep down there everything that's happen is using the latest libraries to make everything okay I get the idea and I the idea and I really really like it because five minutes ago you mentioned something that is key here that we've been doing this for a while but we've been using arise we've been using other stuff that we didn't realize Oh, I like it. And by the way, a couple of comments that we have here. Uh,

24:53 pocket say AI in net. Yes, we doing a lot of super cool things. AI inn net. Someone is also mentioning here the general relativity based on tensor. I will say strings, but string theories and more, but don't get I'm not going to get there. My my father is not here for them. And then we have a lot of people saying hi uh questions about to sharp and pytor that we literally talk and more. So hey it's super super cool and someone is asking what libraries are we using. We are not talking about a specific library today like we did in other episodes when we talk about extension for AI or lamasharp or others. But we are talking about mostly the tensors and numeric in the net. But I will share some links if someone is

25:38 interested how you can get some libraries to do AI that we have right now which is not to do AI. I don't like the the sentence. I will play my non-English speaker card there. But it's to maybe consume an AI model that we have in the cloud or locally or maybe do something like that. But hey, it's cool. And we have Palpatine here say in net. Good. Good. Yes, it's super good. Super cool that we are here. Sorry, Tanner. I was I wanted to give some love to the to the audience there. People are asking about this. So we get there to a point that with you and the team and the amazing people there we get to okay we need to do something we created the tensor we have the tensor span we have there and as far as I

26:23 remember when you explained this to me this is our current net 9 have an implementation we are working on that and then we are going to do more for net 10 this this is this is correct yes yes so int net uh we've actually been doing some incremental improvements to just general numeric and math support for years. So um you know back in net core 2.1 we introduced span uh we also introduced um math f which provided 32-bit floatingoint operations for uh uh uh for for available use whereas previously we only provided them for double. uh we then you know we continued

27:09 adding new APIs and started working towards compliance with various standards and specifications uh like the ITE 754 spec which defines how floating point is meant to work uh deterministically across machines um we we continued doing that through 3.1 5.0 No. Um and then in seven we actually introduced a new feature called generic math. So in the system numeric namespace there are a set of interfaces like I number of t and i binary integer of t. These are then implemented by all the foundational types like int32 single double uh even decimal. and they allow you to now write generic algorithms

27:55 where you constrain a t to one of those interfaces and you can just use operators. So you can now have something like a u sum of t method where you then say you know t.0 and you can do t plus t. Um you can now write all of that generically. You no longer have to write it for each type you want to support. And so that was a huge foundational step into allowing users to write more algorithms and do more with less code. Uh int net 8 we started looking at what we could use and provide for uh accelerating existing user workloads. Um, we'd been providing uh low-level functionality to have access to SIMD,

28:42 which is single instruction, multiple data instructions that every CPU has provided uh for the past, you know, 20 to 30 years. It's been a fairly foundational thing that helped launch and make games on hardware possible. Uh, particularly before we were so heavily reliant on GPUs. um and we provided access to that but it takes a lot of knowledge and work to make that possible. So we we thought what can we do to make it so that the user doesn't have to write all that code themselves. Uh and we started providing uh algorithms in box particularly via span that allowed users to not have to write that code. They could just do something like innumerable dossum or span.sort

29:27 sort or span index of and it would do the most optimal algorithm for you behind the scenes. Then we said, okay, we've done all of this, but we also need to be able to work with shaped data or data that isn't contiguous. You know, sometimes you're working with very large data and you can't load it all on your machine at once. You might have a uh you know a multimegabyte image that you've taken of the stars in the sky uh or that the Hubble Space Telescope you know takes pictures of. Those things are absolutely ginormous. And even GPUs have limits where they can only work with data that's so big. Most GPUs can only work with an image that's up to 16,000 pixels

30:14 wide and tall. anything bigger than that and you have to change how you work with the data. You have to break it apart into tiles and uh start processing it differently. That way you can fit it all into memory, work within the limits of the GPU and do the operations as needed. And when you just have a single continuous allocation via span, you can't do that. So we started looking at well we need a way to work with shape data. Everyone's starting working with tensors and we started providing these new things called tensor primitives. So we have this class in the system numeric tensors tensor uh namespace called tensor primitives that allows you to incrementally work with segmented or

31:01 chunked data and allows you to do all of those same operations but allows you to process it in a way uh that works for the for the data that you have. even if it's broken apart via multiple allocations, even if you're having to stream things off disk, uh it works with that data. You just have to write the algorithm yourself to deal with the segmentation of it. And from there, that was shipped in .NET 8 and it got improved a little bit more int. But once we had shipped in 8, we wanted to make a way so that users could not have to worry about dealing with the shape of the data themselves. And that's where we started working on tensor span of t. Okay. And so int net 9 we we

31:48 previewed as experimental the tensor span APIs. Now we we preview these as experimental because they are it is an absolutely massive amount of work to expose all of the support to make it efficient uh to make sure that it works with the various libraries and ecosystems that were already out there. um and generally just get and hear feedback from the community to make sure that we're doing it right. And so that's why we shipped as experimental int net 9. Uh the feedback that we got was that things are largely in the right shape. There's these these few little rough edges where we needed to work out usability, improve things, and generally make it better. And so we we've been working on responding to and addressing that feedback in the .NET 10 time frame. And as of a couple days ago,

32:36 uh I actually removed the experimental attribute from uh those APIs and we plan on shipping them as stable int net 10 uh this November. Hey, that's super cool. And a friendly reminder to everyone here, every year we have a new version of .NET. I mean, that's the plan and we've been doing the plan for the last five years or so. I can't remember. And the next one is going to be this November, .NET 10. But what I also like to remind everyone is that we can see right now the current state of these features. We can everybody can collaborate. Everybody can go take a look because this is open source. The net is open source. You can go to GitHub, take a look at this, create your issues. If you're not happy with something, ask your questions. We

33:22 are trying to get feedback from the community which is literally what you just said and everything that cames that will came in net 10 is going to be kind of the final cook version and hey I like when any of the product team removed the experimental label because that allows me to avoid to add my code the pragma suppress whatever exception to use and test something. So I know it's literally two clicks or two lines that I need to manually write but hey I like what is there but hey it also means that the mature level it's it's cool it's there and I hey thanks about that I mean I need and this is a question that I also have because I don't know if everybody is aware that when everything that you've been talking is not just

34:09 implement this forn net but also the framework works in GPUs CPUs npu and you have Intel, you have AMD, you have a lot of providers and more and the net framework runs literally everywhere. I want to give this a try example in my compilot PC when I have my NPU and see it's really using this and that and I really like the way that today and I only see this in a very high level as a developer. I literally don't know to know anything about that. It's super well abstracted, super well presented that I I am going to use it and under underlined is going to basically take advantage of the best resource to do that which I really really like when I am a developer my type of developer who

34:54 doesn't go deep for these those scenarios. So hey it's kind of nice to have it there and oh someone is asking that we need to provide Visual Studio 2026 with C++ 26 support. Okay, I am going to pass the question down to the Visual Studio team. This is not the C++ scenario, but I by the way I haven't done C++ in a while, but I didn't even know that we have that version there. But hey, happy to put there uh TER code in Captain America. Yes, probably because of the shield and more. It's I don't know maybe this is an AI screenshot analyzing what we are seeing right now but hey sorry small

35:44 pause there to to basically set that do do we want something to show that you want to show and yeah so let me make sure I've got the we only have 20 more minutes and I know that we start to talk and I really like you t and the super smart people that we have here at Microsoft when they start to talk how to solve this and that and more. It's It's super nice. Yes, not an AI, but maybe an AI man of peace here with with a nice smile. But hey, I really like to to hear about this, but I also know that people wants to see some some samples code and more. Yeah. Okay, let me bring you to the stage. There you are. Yeah. So, we ask you to zoom a little the code if you can. That

36:31 better? Yes, that's better. Yeah. So, uh, like I was talking about, uh, we've been working on building up these set of APIs for a while and because we've been working on building it up and providing these layers, we we provide access for developers to do everything from write the most naive code to literally picking uh to to almost literally picking the instructions that the CPU will execute by directly specifying you want this SIMD instruction. And because of that, we've got this layering system and we've got options for for developers. So, like I was talking about earlier, you might want to write something like a sum loop that takes all all the values in an input and computes what the sum of them

37:18 is. And the most naive version of that is you you define the result, you start at zero, you loop through all of the values, and you return the result. And that's all fine and good, but it's also not very fast. And if you've got a lot of values, you might need to make it faster. So you're going to do the next thing that you might do is you're going to go and do what's called unrolling your loop. And this takes advantage of modern hardware features where you can where the CPU even though you're seeing four operations here because they're next to each other, the CPU can actually execute multiple of them at the same time. Cool. So, uh, this even though you're doing four, most CPUs allow you to execute somewhere between two and

38:04 four ad operations at the same time. And so this will end up taking roughly the same time as one iteration of the previous loop, but since you're doing four of them, you're now looping four times less and you're getting faster faster execution overall. But you've had to write, you know, significantly more code to handle that. But it's not even all the perf you can get. You might then want to say, well, I want to directly use SIMD. I want to take those four instructions and make them one instruction, but still process all four at the same time. Makes it a little bit more explicit than rather rather than relying on the CPU to do it implicitly. So, you still have a lot of code you have to write. You might want to take advantage of even newer hardware. Um,

38:50 Vector 128 has been around since the '9s. Vector 256 was introduced around, you know, 2011 or so. Vector 512 is more recent around 2017 based hardware. Uh, but you're you're you're just adding more and more code to get more and more PF. That's where you want to say, well, what if I don't want to write all of that code or what if I don't know how to write all of that code? because not every developer is going to go and learn all of these intricacies on how to do it. Well, that's where Tensor Primitives comes in. You can take all of that code, all 306ish lines of it, and compress it down to this one line.

39:37 That's super cool. Okay, when you have the solution explorer covering this part, I was thinking what's happening there? How big is that? Or something. So, oh, hey, this is the type of thing that we've been talking and I really like that you put everything there. The team created all of this and we don't I don't I don't want to say that we don't care about that, but it's basically for us it's just tens of primitive. Zoom and that's it. You don't take advantage of everything. Well, and and and it's very easy to see the Perf difference between it. You know, if you've only got a few elements, you've got four of them, you're not really going to see any difference between the options. And in some cases the the the biggest options even come with this minor penalty. It's slightly slower but at the same time you're working you know

40:25 this is one and a half nanoseconds worth of difference. That's essentially two to three cycles instructions being executed different uh for a standard modern computer that's running at roughly 3 gigahertz because one nanocond is roughly one instruction cycle for a 1 ghahertz computer. You're going to execute a billion of those in a single second. That's that's essentially nothing. Now, as you go up in size, the the differences start showing. you start seeing well hey some scaler is taking 86 nconds but even though I'm executing 32 times more data when I start using some tensor it's only taking four times as much not

41:13 32 times as much and as you get even bigger and bigger these savings start getting more and more uh and that's because we've taken the care to do all of the tuning to do to handle all of these edge edge cases about well how big is the cache on your computer? Am I going to be reading so much data that I'm going to throw away everything else out of the cache and then I'm going to start having more cache misses? Am I best taking advantage of the hardware because it's got 512-bit support or 128 bit support? What if I'm on a machine that has 512-bit support, but it actually comes with a penalty to use it? like it downclocks your computer initially because it requires more thermal overhead. There's all these

41:58 nuances that most people don't want to know or care about. And so we simplify it down to that one line for you and it handles everything. And that reproduces across a wide range of machines including other hardware. So th this isn't just even though I've only shown numbers here for Intel and AMD. Uh you get the same performance optimizations on ARM. You get the same performance optimizations on uh web assembly. Uh okay. Hey, that's cool. You get uh you get some but not all of these optimizations on iOS and Android if you're using the mono runtime. Uh I need to run this in my Raspberry Pi. I need to I need to give it a try and see how I

42:46 mean not for the speed but just to to to learn to to do more and hey hey hey I I really like it and once again line 307 is the winner of today. Yeah, it's definitely the winner. Um but then then we start getting into well what if I don't have that contiguous data? What if I what if I want to work with data uh and not have to worry about about that extra stuff? This one line is great if all you have is a single readonly span. Yes. But if you start having shapes to your data, you needed something a little bit more powerful and that's where tensor comes in. And so the simplest thing is these two lines are equivalent to this one line. And the reason it

43:31 becomes two lines is because you're declaring a shape to the data. And this is most obviously visible in that you see that there's multiple overloads and that you have one that doesn't just take in the values, but you can also specify the length of the data. Okay. And so you can do this and you can specify what's known as the strides of the data. Yeah. Now lengths specify how big the data is per shape. So, uh, the the simplest thing to imagine is you've got a 1920x 1080 image. So, it's you go 1,920 columns and then 1,080 rows. Now, computers, and this is even true uh in other languages like C and C++, they

44:18 work in what's called row major order. So, that means that the thing that has the biggest impact to the address comes first. And so even though a lot of people think of coordinates as xy in practice computers index as yx because to compute the linear offset you have y * width plus x. Yeah. And so uh you end up having the data that way and that's why this ends up being 1080 then 1920. Now, if you're just working with a basic shape, that's all you have to worry about is just the lengths. But when you start working with slices of data,

45:03 you start having having to think of it differently. If you have a standard array where you've got 1 2 3 4, all of this is sequential. And if you want to slice and you just want to take two elements, well, you can do that at any point. You can decide to take uh 2, three. You can decide to take three 4. You could decide to take one, two. All of these are slices of the same data. You don't have to track anything special. You're still just sequential. When you have multi-dimensional memory like this where you've got rows and columns, it gets a little bit different because what if you wanted to say out of a

45:54 out of a 4x4 matrix, what if you only wanted to take a 2x two slice? So what if you wanted to take only the elements that I've bracketed here? Yes. Well, you'll you'll note that there's two pieces of sequential data and you've got these this gap in here. And computers describe the shape of this data using what's what's known as a stride. So the stride exists per dimension and it basically says here's how far apart the data is. With a normal array, the stride is always one because the data is one element apart from each other. So if you wanted just this and you wanted just a 2x two slice but you wanted it out of something like

46:41 a 4x4 matrix. Well then you have to actually describe what the underlying data allocation exists as. So you have to describe that column-wise elements are one apart. So you've got one element between each other. But for rows, you have to describe the whole thing that they're four elements apart. And so the stride says even though I've got a 2 by two here, what I'm doing is I'm taking the first two elements here and then the next two elements here by describing how the data exists in memory. And that's what the strides allows you to do. So given a 4x4, you can take any 2x two out of it without having to do extra things just by describing these.

47:29 Um, and you can do things like that given even the full image like a 1080 by 1920 by just doing tensor. Slice and slice works just like span where you specify what the start indexes are. So, I want to start at 0 0. And what this is going to do is it's going to take this. And if I ended up saying, I want to start at let's say two two instead, what's going to happen is it's going to say, well, here's zero, here's one, here's two. You have there. Yeah. And then I'm going to go zero, one, two. And so it's

48:15 instead going to take this section because that's where I said I wanted the slice to come from. And so you're able to it automatically does the stride calculations for you. It automatically determines how to track that data appropriately and allows you to work with it and it allows you to do that and still get all the efficiency and performance improvements that this gives you. What I like about this and by the way I'm going to go to the question. We have a couple of interesting questions in the chat. But what I like about this is that sometimes I I get this is kind of a natural way when you say slice. If you know this domain, you know what are you doing and I like to have this back on my nump and my pandas day the code and and I am not trash talking Python.

49:03 I'm a huge Python fan also. But the go sometimes get super tricky to understand because you have this huge whatever where okay what did I did now? It should be easier now I thinking because it's like reading a regular expressions. I am going to say copilot explain this for me. So today I probably go back to my Python day and say I can't remember what I did here how I filter this data. So explain me what I did there. But I like here that the concept are there and we have the slides and other and other functions and more to work with this because soon this is slides. So you know you have this and you are going to work here. I I I really like it and I hope that people are also getting if they know if they want to use it. There it is. Let me go back here to a qu a couple of questions. So

49:51 runner dude is asking about this. Are these implementations available online for reference? I will say yes. This is all part of the repo that we have. Yes. Uh yeah. So um uh there there's basically a few places you can find it. uh one I've got my the the benchmark that I wrote which isn't you know they're still fairly naive and they're not necessarily showing the best code for each size. They're more meant to be demonstrative to show the example of here's how complex code can get and it can actually get a lot more complex than that even um something that uh a lot of developers aren't aware of is we have a website called source.net. Now that's source dot then the word dot then

50:36 dot then net. Uh we need to share. Yes. Uh this is similar to the old reference source website that existed for .NET framework uh except it covers modern.NET and it covers the uh core libraries. It covers uh ASP.NET uh and you can just use it to search for anything. So if I just wanted to know how is int32 implemented, I could search in32 and easily see all the code for it. If I want to know how tensor span is implemented, I can do the same thing. Now beyond all of that, there is also the actual GitHub. And the nice thing about source.net is it lets you find the

51:21 code and then gives you a direct link to the code on GitHub. Uh and so it allows you to go and browse the code, see the implementation, see the functionality and everything else and allows you to actually go and um let's say you found a bug. Well, you can just go and log a new issue for us. Uh you can open a poll request if you want to make an improvement. You can start a discussion with us if you've got a question that you feel can only be answered by the experts on the net team. Um there's also of course other community resources available. There's the C# community discord which I'm an admin on. Uh there's the .NET evolution discord which is for people contributing to the core.NET repos and code bases.

52:07 Um and there's a number of communities out there that are hosted uh by various members of the .NET Foundation as well. And so there's lots of options to find these samples uh and to engage with us when you have additional feedback and nuance that needs to be given. Yes. And and I haven't I need to give some love to the to the discord community that you mentioned. I haven't been there a lot, but I am part of the AI foundry community discord channel community. We host office hours every week and more. And I like Discord. I spend my fair amount of time per day in Discord because it's kind of a natural conversation that you can have there and see what people are building and more. I am going to also switch to the C# but I also going to invite everyone like in

52:54 example we have a question right now from Philip we need war with custom stroke if we can't answer that because I know that we probably can't and we don't have time and more discord is usually a place that there are people from the team there are experts here and there that can help with these kind of things so hey I like it and yes source.net net is one of the trickiest one that I share the link in the chat for everybody for everyone who want to give the try but I remember the first time that I need to share this again playing my non-English speaker card need to share and spell this URL it was a fun moment people saying what are you saying man what is you mention three dots or no it's not three dots it's dot dot dot hey it's a fun one

53:39 sorry back to you D yeah um well to to answer the question on custom strus. Uh yeah, tensor is just generic. Uh it takes in a t. Uh and so it works with anything that is a t. You you can put in any type there. You could put it you can make it a tensor of object. You could make it a tensor of int. Uh you could make it a tensor of gooids if you wanted to. Um not sure why you would want to use a tensor of gooids, but I'm sure someone has a reason out there somewhere. uh you can use any t and it works with it uh strrus manage data etc. Now, one of the interesting things about this is we work closely with the language team, the C# language team. We

54:24 work with the F# language team. Um, and we frequently end up giving requests to provide better functionality. Uh, given that we want to support any T, but we also have operations we want to support for specific T. Uh because what you might want to do with a tensor span of int is not necessarily the same as what you might want to do with a tensor span of string. Yes. Uh that that kind of causes a problem because normally you can't expose API surface for one versus the other. Uh unless you start exposing extension methods, but extension methods have some limitations and everything else. And so

55:09 C# 14 uh is has a feature called extension members which is basically extensions leveled up. You can now define properties. You can have extensions that work for very specific constraints. So you can have something that differs for uh for a t that is a uh an integer versus a t that is a floating point. You can actually have that difference. Now and beyond that we also ended up asking for a feature called extension operators. So you can actually define operators for specific t. So we can support for example uh tensor plus

55:55 tensor on anything where the t is supports plus itself and we can do that via the generic math interface i edition operators. So we can end up saying we end up defining uh we don't have it in yet. Uh the the feature actually just the features we needed just landed last night. Um and they got uh they got posted on various social media by the by the language team. Uh but we we will be exposing here for the next preview uh things like uh extension operator plus that only works for uh things that are addable. So now you can do tensor plus tensor for int and float and double and everything else without having to worry about all that nuance and you

56:40 don't have to worry about oh it's not going to work or uh or it's going to be available but then crash for string or throw an exception because it just won't be available for string. Uh we we even have a new feature called userdefined compound assignment operators. So users could write uh you know with with normal integers a user can write int x= 5 and then they can do x +als 3 but you couldn't do this for your own struck type. Yes. Uh with userdefined compound assignment operators you can now define such uh compound assignment operators for your own types. uh which means that we can also then specialize tensor compound assignment to avoid extra

57:25 allocations, extra copying and other things because we're able to know hey I'm just overwriting the thing that's the input so I don't need to go and create this new allocation and copy of data and everything else and all of these features help make the not just tensor better but the general ecosystem and uh uh all of net better because it's available be outside of just the things we're going to be using it for. Yes. And hey, people are super happy with this extension operations or more and I know that we are near to close but I have Jose question started from the early beginning and more and I don't know if we have an answer for this but I don't know anyone in the team ever seem to

58:12 create kind of a binding for TensorFlow. Uh what one of the one of the big considerations is that uh these libraries come and go over time and even though TensorFlow was so big for many years, there's many features that have been removed or deprecated on it. It doesn't have the same support policies. Um and uh in many cases they're being replaced in newer code bases by other things. I believe Jax is one of the things that's replacing TensorFlow in many modern libraries and that's also provided by Google. Um uh whereas TensorFlow was a Google project originally be because of all of

58:58 this nuance is not necessarily appropriate or right for the .NET team to provide official bindings. But anyone in the community is able to go and create their own bindings for any library they want and if there are users then they will come they will use it and it will be adopted. Um, you know, not everything has to be or should be provided by the .NET team. Something some things are really best provided by the community. And while many things won't take off, if they don't take off, it probably wouldn't have taken off if we provided in the BCL either. Yes. And a great example of that is the OpenCV libraries that we have, which are not we don't create at Microsoft those, but it's super popular. I can't remember. Open CD sharp. I can't remember. I use I was using one last year and it's been there forever. It's nice mama at the

59:45 community and more. But again, that's a very specific domain that we need to do that. Hey Tanner, we are at the end of the hour. I like to be very very sensitive with the with the with the time and everything. Thank you very much. This was amazing. Also, time flies when I realize I want to see some code. It was literally 20 minutes ago. So sorry about that. And thanks everyone who are there. Tanner, thanks a lot. See you all next one in a couple of weeks. And hey, have a great day everyone. Bye bye.