So hi, everyone, I'm Suharsh, and I'm here

to talk about the TensorFlow Model Optimization

Toolkit, which we have techniques

for quantization and pruning.

And feel free to ask questions or interrupt along the way.

I want this to be super interactive.

So what we're going to talk about today

is the high level of what quantization is, the challenge

it poses, and why it matters.

And then more of the specifics on,

in TensorFlow, what we are doing to work

on quantization and pruning.

So overall, quantization, the idea

is that you have your floating point network

with your inference graph, which is a floating point program.

And we're going to make modifications

to this program in the general sense

that we take these floating point calculations

and make them lower precision.

And the goal is to get as close in accuracy

as possible while providing some performance improvements.

So usually, this involves--

this is very general--

there's some function from the floating point to the integer

value.

There's a process to do the conversion

to make it valid for a particular hardware.

And then there's various algorithms

we have to get these parameters needed

for this function in the most efficient way.

So this is really general, and it may not make sense now,

but we'll make it more specific later.

Mhm?

AUDIENCE: Do the same conversion functions

work for mobile devices as well as specialized hardware?

SUHARSH SIVAKUMAR: No, and that's one of the challenges.

And we'll get to all the challenges.

That's a really good question [INAUDIBLE]..

Mhm?

AUDIENCE: I had another question.

Would do you also be [? motivating ?] soon why is

this not as simple as a [? downcast ?] from float

to int [INAUDIBLE]?

SUHARSH SIVAKUMAR: Yes, it'll all make sense, I hope.

AUDIENCE: We're obviously very interested.

SUHARSH SIVAKUMAR: So why does this matter?

So the first thing is that the ML programs have

lots of parameters, and by using lower precision,

we can instantly get these models a lot smaller, which

can help with memory bandwidth and network

costs of downloading models.

Second, if you have all your calculations in integers,

you could have lots of optimizations that

make the execution super fast.

Third, integers are super power efficient.

So on mobile, this is really important.

And then finally, this lets us explore a whole new avenue

of hardware design, where we can make custom chips like seastar

was the first, then Edge TPU, the new TPUs [? are ?]

having integer operations.

And this can get us cheap, power efficient, fast hardware.

[INAUDIBLE]

So--

AUDIENCE: I think it helps if instead

of saying integer operations, you

say fixed point [? fraction ?] operations.

[INAUDIBLE]

SUHARSH SIVAKUMAR: So I avoid it because it's

only kind of fixed point.

It's not like-- it is fixed point, but when I--

so I've said fixed point in the past,

and then folks always say it's not truly fixed point

because fixed point applies a rescale every time you combine

the two values, and sometimes I get pushback.

So I'm going to avoid--

because I used to say quantization,

and then people would say there's

a hundred steps of quantization.

So the integers are the key here because that's

what's providing the acceleration that's

specific to what we're doing in the TensorFlow stack.

And the specifics, I guess, will make sense

after we go into the equations.

So why is quantization hard?

And this was your point that we have different chips.

So each chip has its own specific tradeoffs

it chose to make.

Some may only support int8, some may support int16,

some way want power of 2 rescales.

All these really one-off decisions

to make the deployment story of how

do you take a general TensorFlow program,

put it on one of these chips-- really hard.

For float, we started to get to a world

where we can just say float can run anywhere.

But for these things, there's not

a lot of standardization on how to do this.

The second reason it's hard is it often requires custom tool

because you need extra metadata that often can only

be gathered by running inferences to know

how to quantize the values.

And we'll get more into that in detail.

So there's often an extra step in the process.

And then finally, for every specific ML problem,

we don't have a good answer for how

quantization will affect it.

You can use the same architecture,

but just do something else for your particular task

with the outputs of that architecture,

and quantization may help or hurt.

And it's pretty empirical right now,

where we just try it and see.

And we're still in the process of gathering a lot of examples.

But one of the goals we need to work on in ML research

is understand these models more to determine how quantization

error will impact things.

So now more into the detail.

So currently, what most hardwares implement

and what the TensorFlow and TensorFlow Lite stack

implement is affine quantization, which

this is like us milking y equals mx

plus b since seventh grade for the rest of our lives.

[LAUGHING]

So basically, you uniformly distribute your range

into fewer chunks than you had before,

and then bucketize them.

And this is effectively what all quantization is.

Currently, we have different ways of gathering statistics

to determine how to quantize.

So going back to this picture for a second,

we need some sort of min and max value to know how to quantize.

So this implies that we need tooling

to get this information.

And we have two types of tooling right now.

There's during training tooling, where

you can incorporate this as part of your training pipeline.

At the end of the day, you have a trained model

that has information on how to quantize it.

And you can also do this post-training,

and we'll talk about the trade-offs later.

AUDIENCE: I have a question.

So why are you [INAUDIBLE] not beyond the boundary

of your possible values?

Why do you choose-- or do you purposefully choose

to leave some values out?

SUHARSH SIVAKUMAR: So, yeah.

The min and max, it's kind of an open question

on what is the optimal min-max given a tensor, if I answered

the question right.

So you could choose to put your min-max much smaller

than your actual value seen, and you'll get some clipping.

And depending on the model and the problem,

we wouldn't really know if it's useful or not.

Because sometimes models don't care

about those extraneous values, and sometimes, they're

the most important thing the whole model.

AUDIENCE: The tricky thing is that when you set your min

and max, and if you're using int8,

you only have 255 values between the min and max.

Every [? activation ?] has to be cast

into one of those 255 values.

If you [INAUDIBLE] minus infinity [INAUDIBLE]

plus infinity, that's really useless.

But if your mean is 0 and your max is 0.01,

you can represent computations with a lot of precision,

so it's the trade-off.

SUHARSH SIVAKUMAR: Yeah.

And we do different types of these depending on the model.

And we've seen weird things where--

and it's always this battle between how much

does the network care about these extreme values versus how

much is it care about the average rounding

error along the way.

So it's always this rounding versus clipping-- that's all--

we just play with this lot.

AUDIENCE: You mentioned about min and max being primarily

influenced from training.

But [? I do ?] like to also do this [? add ?] [? infinite-- ?]

there's a constant feedback loop from--

SUHARSH SIVAKUMAR: So it's training or post-training.

So post-training might influence for like model compilation

time, and it doesn't stop.

AUDIENCE: Could I just clarify?

So the point is that it's no-- it wouldn't be considered

quantization if you just reduced float32

to float16, for example-- float8 or whatever.

So you still have a separate exponent

and you have just kind of fewer bits.

That's not considered quantization?

SUHARSH SIVAKUMAR: So technically, it is.

So the textbook term of quantization,

it is quantization.

But the quantization we're talking about here

is this integer quantization where

you have a shared min-max.

AUDIENCE: Where you really don't want it to have that--

SUHARSH SIVAKUMAR: [? We're ?] using that scale.

AUDIENCE: ----[INAUDIBLE] the exponent.

So that's the only thing that's useful for the hardware

[INAUDIBLE].

SUHARSH SIVAKUMAR: Exactly.

So in other like DSP literature, it's

sometimes called "block floating point," where

you have the exponent shared across all values of a tensor

rather than one exponent per element.

So in a way, float is just per element quantization.

Yeah.

So during training, the idea of during training quantization

is that you want to somehow get this network

to be robust to this error that quantization introduces.

So you emulate the effect of quantization

in the forward pass.

So if you ever see these TensorFlow fake_quant

operations, or the [? contra ?] [? quantized ?] rewriter tool,

this is its goal.

It's saying given a graph, we'll rewrite the forward pass

to emulate the error due to quantization,

and then in the backward pass, we'll

do some tricks to skip over those non-differential parts

that quantization introduces.

And then the goal is that that [? prompt ?] will magically

make the weights better for quantization.

And this can often get the best accuracy given

a particular schema of quantization,

but it's also really hard to train sometimes.

And machine learning, as we all know,

is the art of making as few changes

to your training to get it to converge.

And the second you do more, oftentimes, you

won't even converge if you go to too low of a precision.

And you just have to play around a lot

if you try the training route.

Additionally, the error introduced in training

is specific for a particular target.

So if you want the result of your training

to be portable and work across many different ships,

you're kind of in trouble now if they

have different characteristics.

AUDIENCE: So by emulating quantization,

does that mean on the forward pass after every op,

you just apply the quantization?

SUHARSH SIVAKUMAR: Yeah, and it's

a bit trickier than after every op,

because it's after every rescale that the hardware expects.

So a specific example is like, in TensorFlow,

you have con [? bias ?] [? that ?] [? value. ?] In most

of these inference backends, those are fused into one

[? fat ?] con [? bias ?] [? that ?] [? value. ?]

And your rescales are only at the inputs of the con

and the outputs of the [? value. ?] So only