Transformers.js: State-of-the-art Machine Learning for the web

0:00 [Music] good morning everyone my name is Joshua and I'm excited to talk about Transformers JS a library we're developing at hugging face uh that brings state-of-the-art machine learning to the web first up a small introduction starting with who am I well I suppose you may know me better by my ex logo profile picture or my Twitter or GitHub username zenova but in reality I'm a 24-year-old machine learning engineer from South Africa who's focused on bringing powerful AI models to the web after completing my computer science degree at the University of Cape Town last year I joined hugging face to continue developing a little side project I was working on at the time Transformers JS if I had to guess I'd

0:46 say I'd been working in open source for around 9 years now and I'm extremely grateful to the community for their support of the project if you know about hugging fa's Transformers Library you may already be able to guess what Transformers J is but for those unfamiliar Transformers JS is a library that provides high level abstractions for running set of-the-art pre-trained models in JavaScript it's designed to be functionally equivalent to the python Library meaning you can run the same pre-trained model using a very similar API as you'll soon see it's unbelievable unbelievable what our community full of people who truly see the potential of machine learning on the web have been able to build so far in the last 18 months we've already added support for 25 popular ml tasks

1:33 including text generation object detection speech recognition just to name a few in total we support 120 different model architectures covering a wide range of input modalities as shown here to make it really easy for users to get started we converted over 1,200 models to be compatible with Transformers JS and then uploaded the weights to the huging F Hub so how does it work first users convert their P torch tensorflow or Jax models to Onyx using our Optimum library of course if user wants to use one of the already converted models they can just skip the step and use the model ID from the huging phase Hub next the user writes their JavaScript code we recommend that new users utilize the pipeline function which provides a high level easyto usee

2:20 API that encapsulates pre-processing inference and postprocessing for advanced users we do expose the model tokenizer processor classes as well as other relevant functions and finally a user can open up their web browser and run their application it's really that simple it may also interest you to learn why I created the project in the first place well like all good superheroes or super villains I also have a backstory two years ago YouTube was dealing with a serious spam bot problem in the YouTube comments to me this sounded like a simple text classification problem so I collected and labeled a few million comments trained a small bird model using the Transformers Library and since I didn't want to uh pay for a server I thought it would be a good idea

3:05 to run this as a browser extension uh but that's when I ran into an issue there was no complete way to run a hugging face birt model in the browser so like Thanos at the end of Age of Ultron I decided to take the fine I'll do it myself approach and build a library to run Bert gpt2 and T5 models in the browser after posting about the project to the hugging face Discord there was a lot of interest from the the community encouraging me to add more tasks and models two weeks later the project blew up on Hacker News rising to number one trending and the in the process helping us grow to over 2,000 stars on GitHub fast forward to today and we're now working to achieve feature parity with the library uh by continu continuing to add new models and tasks as they're released and finally the

3:53 ultimate goal of Transformers JS is to help bridge the gap between web developers and machine learning by providing an easy way for people to add AI functionality to their web applications so what's the purpose of Transformers JS and why is hugging face exploring webml Technologies well the answer to that can be found directly in ambition statement to democratize good machine learning democratize being the key word here as you may know we maintain a large collection of Open Source libraries including Transformers diffusers data sets just to name a few however these are all implemented in Python which means they AR aren't as easily accessible to say web developers even though they make up a significant portion of Engineers Transformers J aims to solve this problem in addition to

4:40 Growing the community we aim to help developers build never-before seen web applications whether it be real-time inference while user draws something or processing of processing of sensor data Transformers JS serves as an easy way for developers to add AI to their applications whether it be distribution when it comes to distribution you simply cannot beat the web every time a new model comes out these days uh the first thing people ask for is a web demo the importance of a hassle-free zero install way for developers and users to get started cannot be overstated another benefit is that developers don't need to pay for a server to Showcase their applications models are hosted for free on the hugging face Hub and devs can deploy their applications using let's

5:26 say GitHub Pages or hugging face baces the last Point I've noted here aims to highlight what this means for the end user combining on device processing processing uh with the ability to access a plethora of browser apis uh empowers users to take control of their data providing personalized experiences in a sandboxed and secure EnV environment we've been extremely lucky to have received so much support from the community and now Transformers GS is one of the fastest growing JavaScript libraries out there on GitHub just recently we with a number one trending repo on the site uh allowing us to grow by an additional 2,000 stars in the span of a month to quote ATS ATS Infamous law any application that can be written by in JavaScript will eventually be written

6:12 in JavaScript and we're no exception we're also amazed to see the growth and Adoption of the library by web developers as of October this year we're averaging around 750,000 unique monthly users we host the uh JS and web assembly files using Js deliver for which we have received around 40 thou 40 million month requests in the past 12 months and in total we have over 2.4 million downloads on npm and currently we're averaging around 70,000 per week for the eagle-eyed viewers out there you may have noticed that the previous code Snippets use imports from huggingface Transformers instead of zenova Transformers and that's because today we finally released Transformers GS version 3 the main feature being web GPU support the

6:59 projected speedups depend on the model and device you're using of course but for simple bbased models we I was able to achieve a 64x speed up on a batch size of 32 on my Windows 10 PC some users have already posted uh of achieving over 100 x speed up on their devices next let's take a look at what you can build with Transformers JS to start I just wanted to take a step back and talk about the development philosophy philosophy that we follow first uh new mod releases for example llama 3.2 next we add support to the library typically this involves converting the model to Onyx and updating the library with the correct configuration values um but luckily since the Llama 3.2 architecture was the same as previous generations we could just reuse our conversion scripts if it

7:46 was a new architecture the process would be a bit more complicated first adding support to the python Transformers library then adding Onyx export support and then finally integrating it into Transformers JS lastly we create Visual and interactive web demos and publish the source code to GitHub not only does this help us increase the visibility of the project when posting through social media but also showcases uh develop showcases what is possible with the library by publishing the source code we hope that developers can learn from it and adapt the ml components into their own projects so now let's take a look at some applications that have been built with the library starting of course with privacy focused chatbots that run 100% locally in your browser the model shown here is 5 3.5 mini uh 4bit quantized 3.8

8:32 billion parameter llm from Microsoft uh that is optimized for inference on the web thanks thanks to a specialized multi-head attention web GPU kernel it can run up to 90 tokens a second on an RTX 490 we also added support for multimodal chat Bots meaning the model can process and respond to image inputs too the model shown here is Moon dream 2 uh a small Vision language model designed to run efficiently on edge devices we also add when we added support for these models we also added the ability to reuse Pass Key values uh meaning we don't need to recompute image embeddings uh for each follow message although the task of classification seems pretty simple such models can be used to create really powerful applications take for example the video shown on the left where we

9:18 perform zero shot classification to sort product reviews into classes defined at runtime similarly we can uh use a zero shot image classification model like clip uh to do the same but for images and videos the video on the right shows a model running at 20 frames per second on an RTX 2080 taking this idea further you may even you are even able to build games using the simple image classification models for example doodle Dash is a realtime ml powered web game that we built that can run on both desktop and mobile the game is inspired by Google's Quick Draw uh where you're given a word and you have uh and a neural network has to guess what you're drawing in our version you have one minute to draw as many items as you can one prompt at a time uh if the model predicts your label

10:06 correctly the canvas is cleared and you get a new word uh since the game runs locally you don't have to worry about server latency at all and the model is able to make real-time predictions to the tune of over 60 frames 60 times a second uh certainly our most viral demos have been for speech recognition like whisper web GPU where which enables multilingual transcription and translation across a 100 different languages using open AI set of whisper models on an M3 Max you can achieve a 10x realtime Factor being able to process around 100 well 120 seconds of audio in around 12 seconds this also means that you can transcribe as you speak as shown in the video on the right in browser image editing software

10:53 can also be enhanced using models like segment anything from meta for um this model show here is only around 20 megabytes in total and can decode Mouse inputs in real time in fact community members have already turned this into a figma plugin simplifying the integration into existing design workflows we also support image and video enhancement giving rise to an interesting set of use cases around video streaming and compression you can imagine how this can be used in browsers to lower bandwidth usage by streaming lower quality video and then enhancing it in real time on the client side we've also experimented with similar models to enable real-time background removal or blurring like those used for video conferencing conferencing

11:39 applications given a video of say an interview we can combine a variety of models and tools in a pipeline to create a simple video editing website first we extract the transcript using whisper then we generate Word level timestamps then we run a speaker diation model and finally the last step which is not included here here would involve FFM F fmeg JS to add captions or to slice the video as my University Professor would say the proof is left as an exercise for the reader uh depth estimation models like depth anything can be used to create immersive 3D demos with libraries like 3js a Community member even got it working with apple Vision Pro by building an application that brings your photos to

12:25 life and then music generation uh the audio clip shown here is uh 10c uh audio generated by music gen from meta uh the model can generate up to 30 to 30 seconds of audio uh we also in browser text to speech is now possible thanks to Transformers JS just in case the default voices are a bit too robotic for your use case and then we also support a multitude of vision and vision language tasks thanks to Florence 2 a 230 million parameter VIs Vision Foundation model that supports tasks like captioning object detection segmentation thanks to its small size the model runs extremely well in the browser and can be used for things like Altex generation a huge win

13:11 for accessibility finally and by pop by far our most popular multimodal use case embeddings the videos shown on here showcase demos that uh we built that are able to search image and audio databases with natural language returning the result in less than 50 milliseconds seconds uh let's hop into an Editor to see how you can get started in just a few lines of code for this example we'll showcase how to run whisper for automatic speech recognition the first step is to import the pipeline function from the Transformers JS library either from a CDN or from npm for Simplicity let's just stick to a CDN next we'll create an automatic speech recognition pipeline by specifying the model ID uh from the hugging face Hub as the second

13:57 parameter the last step is to pass the audio file to the pipeline and that's it just three lines of code to add state-ofthe-art AI to your web applications when running for the first time the model which in this case is around 40 megabytes will be downloaded and cached in the user's browser subsequent runs will then reuse this cached model you can also specify additional parameters like return timestamps which in this case by setting it to word will generate Word level timestamps for the transcript to find models that are compatible with Transformers JS you can navigate to hf.com select the Transformers J library from the option on the left and then filter by task to narrow down your search Once you found a suitable model

14:44 you can visit the model card which will provide you with additional information on its usage another way to get started is using visual blocks an amazing tool from our friends at Google that allow you to experiment with uh ml pipelines with a visual interface stay tuned for the talk later that's going to be giving you some more information and for those interested let's quickly pull the curtain back to talk about uh what's going on behind the scenes the first layer exists in Python world where we convert the models to Onyx Transformers JS lives in JavaScript land and that's where we do things like model loading caching pre-processing generation and post-processing uh finally Onyx runtime is then used to execute the model's forward pass on the desired execution provider for example web assembly or web

15:30 GPU in other words Onyx runtime web is to Transformers JS as pytorch is to the python Transformers Library another interesting aspect of the library is that we had to reimplement all the python tokenizers in JavaScript so that they can run in the browser as part of this effort we also created a minimalistic JavaScript implementation of the ginger processing engine uh templating engine which is designed to uh generate a prompt for a given set of chat messages and templates finally what's next for Transformers JS well first of all we'll continue adding new tasks and models aiming to achieve feature priority with the python Transformers Library unfortunately around 70% of web web users have browsers without webgpu support uh uh only 70% of users have

16:17 access to browsers with webgpu support and for browsers like things like Firefox and Safari webgpu Still Remains behind a flag meaning not all users can immediately uh improve their web dev development uh using web GPU acceleration that said the standards and processes that browser implementers are currently implementing um will become more stable and as uh during this process process Transformers GS will follow suit currently due to Pro debuff and web assembly limitations Transformers GS models are capped at 4 GB in size meaning we can't access some of the larger models um but we're working towards fixing this and soon you'll be able to run your 8bs l BS all those things um we also aim to improve integration with and perhaps help guide

17:04 standards for the next generation of web browsers where scientific Computing Computing and AI applications take center stage one can imagine a browser-based web store for models similar to the Chrome extension for Chrome web store for extensions from the users perspective they could search for web compatible models on the hug and phase Hub uh install it with a single click and then access it across multiple domains currently Transformers J is limited in this regard since models are cached on a per site or per extension basis maybe there are some Chrome engineers in the audience who are interested in turning this into a reality if so let's connect but anyway that's all for me today feel free to reach out if you have any questions or ideas and thanks so much

17:54 [Music]