Again, thank you for your patience.

We are super excited to be here and talking

to you about TensorFlow.js and how

to bring machine learning to JavaScript applications.

My name is Brijesh.

I am a technical program manager on the Google TensorFlow team.

And here is my colleague, Kangyi,

who is a software engineer on the TensorFlow team.

So here is an outline of what we are

going to cover in this talk.

We will start with the what and why of TensorFlow.js.

We will see some of the ready-to-use models

that TF.js supports.

We will do a code walk-through of the development workflow

both to use one of these pre-trained models,

as well as training a custom model.

We will delve a little deeper into the tech

stack and the roadmap.

We will show you what the community

is building with TensorFlow.js.

There are some very impactful applications

that are being built, and we'd love to show you some examples.

And finally, we will point to some resources

that you can start exploring.

All right, so, we have a lot of exciting content to cover,

so let's get started.

You may have heard at the TensorFlow.js keynote

an overview of the technology today morning.

We are going to build on that here.

But first, a quick recap of the basics of TensorFlow.js.

So in a nutshell, TensorFlow.js is

a library for machine learning in JavaScript.

It is built for JavaScript developers

to create and run machine learning models

with an intuitive JavaScript-friendly API.

This means you can use it to perform training and inference

in the browser, browser-based platforms, and in Node.js.

ML operations are GPU accelerated,

and the library is fully open source and anyone is

welcome to contribute.

OK, so TensorFlow.js provides multiple starting points

for your needs.

So, you can use the library, you can directly

use off-the-shelf models that the library provides,

and we will see a lot of these in a bit.

You could also use your existing Python TensorFlow models

with or without conversion, depending on the platform

that you're running on.

Or you can retrain an existing model with transfer learning

and then customize it to your data sets.

That's the second starting point.

Transfer learning typically needs a smaller data

set for retraining, so that might fit your needs better.

And the third starting point is you

can build your model entirely from scratch with a Keras-like

Layers API and train it.

You can train it either in the browser

or on server with Node.js.

So, we are going to delve much deeper into some

of these workflows today.

JavaScript, of course, is a ubiquitous language.

So by virtue of that, TensorFlow.js

works on a variety of platforms.

It lets you write ML code once and run it

on multiple surfaces.

As you can see, the library runs on any standard browser, so

regular web apps, progressive web apps are covered.

On mobile, TF.js is integrated with mini-app platforms

like WeChat.

We have just added first-class support

for the React Native framework, so apps can seamlessly

integrate with TensorFlow.js.

On server, TF.js runs on Node.

In addition, it can run on desktop applications

using the Electron framework.

All right, so, why TensorFlow.js?

Why run ML in the browser?

So we believe there are compelling reasons

to run ML on a browser client, especially

for model inferencing.

Let's look at some of these reasons.

Firstly, there are no drivers and nothing to install.

You include the TensorFlow.js library,

either at page load time by script sourcing it

into your HTML page, or by bundling with a package manager

into your client app, and you're good to go.

That's it.

The second advantage is you can utilize a variety of device

inputs and sensors using standard device

API, such as camera, microphone, GPS--

through standard web API, HTML API,

and through a simplified set of TF Data API,

and we are going to see some examples today.

TF.js lets you process data entirely

on the client, which means it's a great choice

for privacy-sensitive applications.

It avoids round trip latency to the server.

It is also WebGL accelerated.

So, these factors combine to make

for a more fluid and interactive user experience.

Also, running ML on the client helps reduce server-side costs

and simplify your serving infrastructure.

For example, no online ML serving that

needs to scale to increasing traffic

and so forth is needed because you're offloading

all your compute to the client.

You just host a ML model from a static file location

and that's it.

On the server, there are also benefits

to integrating TensorFlow.js into your Node.js environment.

If you are using a Node.js serving stack,

it lets you bring ML into the stack as opposed

to calling out to a Python-based stack.

So it lets you unify your serving stack all in Node.js

if you are using Node.js

You can also use your existing Core TensorFlow models to bring

them into Node.js, not just the pre-built,

off-the-shelf models, but rather your custom models that were

built with--

Python TensorFlow can be converted.

And in an upcoming release, you don't even

need the conversion process.

You can just use them directly in Node.

And finally, you can do all of this

without sacrificing performance.

You get CPU and GPU acceleration with the underlying TensorFlow

C library because that's what Node uses.

And we are also working on GPU acceleration via OpenGL,

so that removes the need for depending on CUDA drivers

as well.

So effectively, you get performance that's

similar to the Python library.

So these attributes of the library

enable a variety of use cases across

the client-server spectrum.

So let's take a look at some of those.

So, on the client side, it enables

you to build features that need high interactivity,

like augmented reality applications,

gesture-based interaction, speech recognition,

accessibility, and so forth.

On the server side of the spectrum,

it lets you have your more traditional ML

pipelines that solve enterprise-like use cases.

And in the middle, that can live either on the server

or on the client, are applications

that do sentiment analysis, toxicity and abuse reduction,

conversational AI, ML-assisted content authoring, and so

forth.

So you get the flexibility of choosing

where you want your ML to run--

on the client, on the server, either / or.

So whatever your use case is, TensorFlow.js

is production-ready-- ready to be leveraged.

So with that intro, I'd like to delve deeper

into the ready-to-use models available in TensorFlow.js.

Our collection of models has grown

and is growing to address the use cases that we just

mentioned, namely image classification for classifying

whole images, detecting and segmenting objects and object

boundaries, detecting the human body and estimating pose,

recognizing speech commands and common words from audio data,

and text models for text classification, toxicity

and abuse reduction.

You can explore all of these models today on GitHub.

You can use them by installing them with npm

or by directly including them from our hosted scripts.

So let's dive in to a few of these models

and see more details on these.

So this is the PoseNet model.

It performs pose estimation by detecting 17 landmark points

on the human body.

It supports both single person and multi-person

detection within an image.

Now there are multiple versions of this model.

Versions that are backed by MobileNet and those backed

by ResNet provide options for balancing accuracy

versus model size versus latency,

depending on your needs.

And it enables use cases like gesture-based interaction,

augmented reality animation, and so

on-- things that are well-served by running ML on the client.

By the way, you can explore a demo of this particular model

at our booth in the expo hall.

Another human model is the BodyPix model.

BodyPix enables person segmentation, again

both single and multiple persons in an image.

It identifies 24 body parts, such as left arm, right arm,

torso, left / right legs, and so forth.

It also provides Convenience API to segment and mask

each body part in a different color,

which is what you're seeing in this particular GIF.

And in terms of use cases, it can

be used for things like blurring faces in an image,

or blur the background, say, to protect privacy.

The COCO-SSD object detection model

enables object detection of multiple objects in an image.

It detects about 90 classes defined in the COCO data set.

And the nice thing is it takes as input

any browser-based image element, like an image or a video

or a canvas element, and returns an array of bounding boxes

with the detected class and the confidence level.

In this sample image, you can see

that the kite is detected with a high confidence score

and you get a bounding box.

There is also a DeepLab model which

offers semantic segmentation.

That's coming soon-- yet to be released.

So what I'd like to show you here

is how easy it is to use a model like that, like COCO-SSD,

in your JavaScript app without dealing with tensors,

transformations, or layers.

And this pattern really applies to any of our pre-built models.

So first, you script source the library

in your HTML file from the hosted CDN.

Alternatively, you can just npm install the library

and use a build tool like webpack

to bundle into a client.js file.

Then you load the model.

That's the .load method.

And you call the model .detect method on the image element

that you are trying to analyze.

And that's it.

What you get back is an array of objects

that have the bounding boxes and the classes that are detected.

So really, four or five lines of code to end up

leveraging a powerful ML model in the browser.

Text models are another useful set.

TF.js has a toxicity detection model

and a more general universal sentence encoder model.

So let's look at the toxicity model a little closer.

I want to show you a live example of this model in a web

app.

Give me a second.

OK.

There you go.

So here is a super-simple web app that simply loads the model

and passes a few sentences to it.

And what the model does is classify it on a few dimensions

like, does this signify an insult, is there toxicity,

is there a threat, and so forth.

So let's try an example here.

So, I'm going to say something toxic--

your-- if I can see it--

ignorance is pretty--

OK.

So, something that I would think is toxic.

So, that returns as toxic, as well as returns "is an insult."