I lead TensorFlow model optimization.

And today I will talk about our toolkit and, in particular,

about techniques around quantization

and neural connection pruning.

So first I'll introduce, what is model optimization?

What is our toolkit?

And then the reason about why we think it's important, why

we're investing in this area.

Then I'll cover the tools that we have available.

And at the end, I will give an quick overview

about the roadmap for the short term and the longer term.

And hopefully at the end of the presentation,

we still have some minutes to go over Q&A.

So our toolkit implements techniques

that should allow you to optimize machine learning

models for deployment and execution.

We think this is important because machine learning is

everywhere, right.

It's a very important field, and we

think that there is a lot of room to make it more efficient.

And this has some implications, both like economy--

like, you can make all these applications

much better, the quality or cheaper to execute.

But also we can enable some new models, and new deployments,

and new products that otherwise are not possible

even if you just tried to execute these machine learning

models on servers.

So currently machine learning runs either on the server

or on the Edge.

On the server, you may think that there

is a lot of capacity, that there's

a lot of compute and memories.

What is the benefit of optimizing these models?

Well, applications are still bound by latency.

There is still a very important metric

for a lot of applications, or you

want to improve the throughput-- how many tasks

can run on your server.

And these two are also directly correlated to money, right.

So everybody will want to save money,

and potentially, we're talking a lot of money.

Now on the Edge is a little bit more obvious

why we need optimization.

These are a very resource-constrained

environment even if you're talking

about applications in general.

We need to deal with reduced memory, computing,

power consumption is typically an issue, bandwidth,

both downloading, models from the Cloud.

Or even we've seen the chips to be able to transfer parameters

from memory to the processor, this

could be a problem if the model is too large.

Plus, we have a wide variety of hardware,

more than in the server, and we need

to make sure that these models run

efficiently on all these different types of hardware.

So it follows that if we are optimizing these models,

and we have better models, eventually,

it starts translating enabling new products that otherwise

couldn't exist if we just were running

these models on a server.

And these opportunities are larger than just

on smartphones.

Machine learning, it is trickling down

into more environments.

We have machine learning models, for example,

that are used to detect failures in machinery in factories,

or we use it in self-driving cars.

We use it in the office to scan documents and try

to understand them.

And just to give you some numbers,

right-- like, the size of the smartphone market

is really a fraction of the potential for the Edge devices,

in general.

So basically the two reasons is we want to make machine

learning model efficient.

It's already very important for the servers,

but it is pretty crucial for embedded devices.

So we started this toolkit about a year ago.

We initially launched post-training quantization

with this hybrid type of quantization,

and I'll go in more detail later in the presentation.

Then earlier this year, we launched the API

for neural connection pruning, then

we created this specification of quantized operations,

integer quantized operations for TensorFlow Lite.

And we launched also post-training quantizations

for targeting this specification.

More recently, we added support for reduced flow precision,

and hopefully soon we're going to be launching quantization

of our training API and also add in support to TF

Lite for Sparse computation.

So now I'll go into these techniques and these tools

in a little bit more detail.

So let's start with quantization.

But first I think it's important we

have some at least basic understanding of what

is quantization, and why it's hard,

and why we are approaching our tools the way

that we're doing it?

So let's start with a simple example.

You know, matrix multiply, it's a basic operation

for machine learning models.

You have two matrices, two tensors A and B,

and then you do some multiplication accumulation,

and then you get a third tensor, C. So each tensor

issues a bunch of values that produce the third tensor.

Then just a little reminder about how matrix

multiply works, each one of these results of the tensor C

are computed as multiplications and accumulations.

So if we look at one of them, and then

if we think how we're training these models, typically

in a higher precision--

let's say float 32--

then it follows that the operations

of the multiplications are float 32 in precision.

And then the product will be also a float 32, right.

And then the accumulation will be also float 32.

So this is fairly straightforward.

There is some loss in precision, but machine learning

is pretty good at dealing with it at least

at this level of precision.

So, no problem.

Now what does this have to do with quantization?

Well, let's go back to what are our goals for quantization.

We want to be able to address all these restrictions,

and also we want to be able to deploy

as much hardware as possible.

So a common thing that we do is, let's reduce the precision

that we operate in.

Let's say, for example, go from the 32-bit floats

to 8-bit integers, and then let's operate,

let's say, entirely within integer operations.

And this will be good because then we are going from 32 bits

to 8 bits, so we reduce the memory.

You know, the models are four times smaller.

Then integer operations are typically faster to execute.

They also consume less power.

And then because the parameters and also the dynamic values

activations are smaller, then we reduce bandwidth pressure.

It means that in the pipes in the chips,

there is more room for things to flow around, which

can also translate into faster compute and reduced power.

And then integer operations seem to be a fairly common

denominator across hardware.

CPUs, DSPs, different NPUs, they support the integer operations.

So OK, we are going to reduce the precision,

so how do we convert the 32-bit float to 8-bit integers?

Well, right now we do something very simple.

So we have this linear mapping, where we say,

OK, we take the values from a tensor.

We compute the minimum and the maximum value,

and then based on that, we spread them evenly

on the 8-bit range.

This basically is very simple, right.

So is that all that we need to do?

Well, I wish.

It's not that's simple, so let's go back to the example.

So we have the matrix multiply.

And now let's say that we quantized the values,

and they are already 8-bit integers.

So the operands and the multipliers are 8-bit integers.

And then the multiplication, the products, now you

need 16 bits to represent it, and then you

need to accumulate.

And you probably want 32 bits to accumulate on that, right.

So what is the problem?

Well, the problem is that now you have tensor received

a float of 32-bit values.

And that is not great when you want

to fit that into another matrix multiply,

that you really want to execute as 8-bit integers,

because we already taught they're

resource efficient, right.

So what do you do?

So you scale them back down, right.

So we just sort of quantize them on the fly now back down

to 8-bit integers.

So then you can feed them to your major 8-bit matrix

multiplier, and now it's all good.

But what are the implications of all this process?

Well, so it means that we're changing the static values,

the parameters, the weights.

We are changing also the dynamic values,

the activations, because we're, you know,

scaling them-- quantizing them on the fly.

And it also means that we are changing

the computation, right.

In this case, it's a very simple example.

We just added a scaling operation,

but it can be a bit more involved.

So you could say, OK, that doesn't seem that hard, right.

Like, we just added another number scaling operation,

and it's just easy, right.

Well, some math is a little bit more complicated than that.

This example actually is from layer normalization of an LSTM.

And this one, aside from looking a bit more complex,

it's an example of-- where if you just apply these naive

rules of operate, rescale, operate, rescale--

you actually end up in sort of a numeric black hole

where the scales cancel each other,

and basically just things don't work if you just

go naively about it.

And then it's more complicated because in your decisions

about how you're going to represent this computation

in integer form--

you know, quantization, we want to be efficient,

so lower precision is good.

But we also want to be accurate, which

means lower precision is bad.

So it's a lot of trade-offs that you have to do.

Then further complicating things,

we have heterogeneous hardware.

There is all different types of hardware

with different capabilities, very different operations

that each hardware supporters, and also different preferences.

Some hardware is, you know, better at executing operations

and produce floats, you know, with different bit widths

or different restrictions.

And we want to account for all the things

when we are creating our quantized recipe,

or a quantized program.

Then there is the fact that machine learning

is hard to interpret.

We don't understand it.

Like, we don't understand how it works--

not to the level that we can't have

good proofs to know that the transformation that we're

doing to this model, to this program,

will actually work or not result in a catastrophic error, right.

You don't want to take a model, you quantize it,

and then this model suddenly starts

giving you some weird results.

So it makes it much more complicated

to define these transformations.

And then finally, I will say that this

is a little more complicated, because the model is not

enough.

The program is not enough.

Depending on how the quantization is defined,

you might also need some extra data.

So an example of the matrix multiplier,

we need it to compute the minimum and maximum

values of the dynamic activations,

and that only can be done if we run inferences

through the model, which means that you need to provide

some representatives there.

So basically, it's just another hurdle