how to dance on a mobile phone.

CHRIS MCCLANAHAN: TensorFlow flow

can take our smartphone camera and turn it

into a powerful tool for analyzing body posts.

ANDREW SELLE: We had a team at Google

that had developed an advanced model

for doing pose segmentation.

So we're able to take their implementation,

convert it into TensorFlow Lite.

Once we had it there, we could use it directly.

SHIKHAR AGARWAL: To run all the AI and machine learning models,

to detect body part, it's a very computationally expensive

process where we need to use the on device GPU,

TensorFlow Library made it possible so that we can

leverage all these resources-- the compute on the device--

and give a great user experience.

ANDREW SELLE: Teaching people to dance

is just the tip of the iceberg.

Anything that involves movement would be a great candidate.

TIM DAVIS: So that means people who have skills

can teach other people those skills.

And AI is just this layer that really just interfaces

between the two things.

When you empower people to teach people,

I think that's really when you have something

that is game changing.

NUPUR GARG: When Tim originally did this,

he did this in slow motion.

We use these models that are running on device in order

to speed up his dance performance to match

the professional dancer.

We also snapshotted a few motions

in order to understand what motions he was doing well

and what he needed to improve on applications like this can

be used on device for educational purposes

for not only dance but other applications as well.

New cutting edge models are also pushing the boundaries

of what's available on device.

BERT is a method of pre-training language representations,

which obtain state-of-the-art results on a wide array

of natural language processing tasks.

Today, we're launching MobileBERT.

BERT has been completely re-architected to not only

be smaller, but be faster without losing any accuracy.

Running MobileBERT with TensorFlow Lite

is 4.4 times faster on the CPU than BERT,

and 77% smaller while maintaining the same accuracy.

Let's take a look at a demo application.

So this is a question and answer demo application that

takes snippets from Wikipedia.

It has a user ask questions on a particular topic,

or it suggests a few preselected questions to ask.

And then searches a text corpus for the answers

to the questions all on device.

We encourage you to take a look at both of these demo

applications at our booth.

So we've worked hard to bring these features of Dance Like

and MobileBERT to your applications

by making it easy to run machine learning models on device.

In order to deploy on device, you first

need to get a TensorFlow Lite model.

Once you have the model, then you

can load it into your application,

transform the data in the way that the model requires it,

run the model, and use the resulting output.

In order to get the model, we've created a rich model

repository.

We've added many new models that can

be utilized in your applications in production right now.

These models include the basic models such as MobileNet

and inception.

It'll also include MobileBERT, Style Transfer, and DeepLab v3.

Once you have your models, you can use our TensorFlow Lite

Support Library that we're also launching this week.

It's a new library for processing and transforming

data.

Right now, it's available for Android for image models.

But we're working on adding support for iOS

as well as additional types of models.

The support library simplifies the pre-processing and

post-processing logic on Android.

This includes functions such as rotating the image 90 degrees

or cropping the image.

We're working on providing auto-generation APIs that

target your specific model and provide APIs

that are simple for your model.

With our initial launch, as I mentioned,

it'll be focused on image use cases.

However, we're working on expanding the use cases

to a broader range of models.

So let's take a look at how this looks in code.

So before a support library, in order

to add TensorFlow Lite in your application,

you needed to do all of this code,

mostly doing data pre-processing and post-processing.

However, with the use of the auto-generation support

libraries, all of this code is simplified

into five lines of code.

The first two lines are loading the model.

Then, you can load your image bit data into the model.

And it'll transform the image as required.

Next, you can run your model and it'll

output a map of the string labels

with the float probabilities.

This is how the code will look with auto-generation APIs

that we'll be launching later this year.

One of the biggest frustrations with using models

was not knowing the inputs and outputs of the models.

Now, model authors can include this metadata

with your model to have it available from the start.

This is an example of a JSON file

that the model author can package into the model.

This will be launched with the auto-generate APIs.

And all of our models in the model garden

will be updated to have this metadata.

In order to make it easy to use all of the models in our model

garden and leverage the TF support library,

we have added example applications for Android

and iOS for all of the models, and made

the applications use the TF Support Library

wherever possible.

We're also continuing to build out our demo applications

on both the Raspberry Pi and as CPU.

So now, what if your use case wasn't

covered, either by our model garden or the support library?

Revisiting all the use cases, there is a ton of use cases

that we haven't talked about in those specific models

that we listed.

So the first thing you need to do is either find a model

or generate a model yourself from TensorFlow APIs,

either keras APIs or estimator APIs.

Once you have a SavedModel, which is the unified file

format for 2.0, you can take the model,

pass it through the TensorFlow Lite Converter.

And then you'll get a TensorFlow Lite flapper for model output.

In code, it's actually very simple.

You can generate your model, save it with one line,

and use two lines of code to take in the same model

and convert it.

We also have APIs that directly convert keras models.

All the details of those are available on our website.

Over the last few months, we've worked really hard

on improving our converter.

We've added a new converter, which has better debug ability,

including source file location identification.

This means you can know exactly where in your code cannot be

converted to TF Lite.

We've also added support for Control Flow

v2, which is the default control flow in 2.0.

In addition, we're adding new operations, as well as

support for new models, including

Mask R-CNN, Faster R-CNN, MobileBert, and Deep Speech v2.

In order to enable this new feature, all you have to do

is set experimental new converter flag to true.

We encourage everyone to participate

in the testing process.

We've planned to set this new converter as a default back end

at some point in the future.

So let's look at the debug ability of this new converter.

So when running this model, it gives an error

that TF reciprocal op is neither a custom op nor a flex op.

Then it provides a stack trace, allowing

you to understand where in the code this operation is called.

And that way, you know exactly what line to address.

Once you have your TF Lite model,

it can be integrated into your application

the same way as before.

You have to load the model, pre-process it, run it,

and use the resulting output.

Let's take a look at a paradigm version of this code in Kotlin.

So the first two lines, you have to load the model.

And then you have to run it through our interpreter.

Once you have loaded the model, then you

need to initialize the input array and the output array.

The input should be a byte buffer.

And the output array needs to contain

all of the probabilities.

So it's a general float array.

Then you can run it through the interpreter

and do any post-processing as needed.

To summarize these concepts, you have the converter

to generate your model and the interpreter to run your model.

The interpreter calls into op kernels and delegates,

which I'll talk about in detail in a bit.

And guess what?

You can do all of this in a variety of language bindings.

We've released a number of new, first class language bindings,

including Swift and Objective C for iOS,

C# for Unity developers, and C for native developers on any

platform.

We've also seen a creation of a number

of community-owned language bindings

for Rust, Go, and Dart.

Now that we've discussed how TensorFlow Lite works

at a high level, let's take a closer look under the hood.

One of the first hurdles developers

face when deploying models on device is performance.

We've worked very hard, and we're

continuing to work hard on making

this easy out of the box.

We worked on improvements on the CPU, GPU, and many custom

hardwares, as well as adding tooling

to make it easy to improve your performance.

So this slide shows TF Lite's performance

at Google I/O in May.

Since then, we've had a significant performance

improvement across the board, from float models on the CPU

to models on the GPU.

Just to reemphasize how fast this is,

a floor model for MobileNet v1 takes 37 milliseconds

to run on the CPU.

If you quantize that model, it takes only 13 milliseconds

on the CPU.

On the GPU, a float model takes six milliseconds.

And on the Edge TPU, in quantized fixed point,

it takes two milliseconds.

Now let's discuss some common techniques to improve the model

performance.

There is five main approaches in order to do this--

use quantization, pruning, leverage hardware accelerators,

use mobile optimized architectures,

and per-op profiling.

The first way to improve performance

is use quantization.

Quantization is a technique used to reduce

the position of static parameters,

such as weights, and dynamic values, such as activations.

For most models, training at inference useful at 32.

However, in many use cases, using int 8 or float 16

instead of float 32 improves latency

without a significant decrease to accuracy.

Using quantization enables many hardware accelerators that only

support 8-bit computations.

In addition, it allows additional acceleration

on the GPU, which is able to do two float 16 computations

for one Float 32 computation.

We provide a variety of techniques

for performing quantization as part of the model optimization

toolkit.

Many of these techniques can be performed

after training for ease of use.

The second technique for improving model performance

is pruning.

During model pruning, we set unnecessary weight values

to zero.