I am Jeremiah and this is tensorflow in production.

I'm excited that you're all here because that means you're excited about production.

And that means you're building things that people actually use.

So our talk today has three parts.

I want to start by quickly drawing a thread that connects all of them in the first thread is the origin of these projects.

These projects really come from our teams that are on the front line, a machine learning.

So these are real problems that we've come across doing machine learning at Google Scale and these air the real solutions that let us do machine learning Google.

The second thing I want to talk about is this observation.

If we look at software engineering over the years, we see this growth as we discover new tools.

As we discover best practices were really getting Maur effective at doing machine doing software engineering and we're getting more efficient.

We're seeing the same kind of growth on the machine learning side, right?

We're discovering new best practices in new tools.

The catch is that this growth is maybe 10 or 15 years behind software engineering, and we're also rediscovering a lot of the same things that exist in software engineering but in a machine learning context.

So we're doing things like discovering version control for machine learning or continuous integration for machine learning.

So I think it's worth keeping that in mind as we move through the talks.

The 1st 1 up is gonna be tense, fearful hub, and this is something that lets you share reusable pieces of machine learning much the same way we share code.

Then we'll talk a little bit about deploying machine learning models with tensorflow serving, and we'll finish up with tensorflow extended, which wraps a lot of these things together in a platform to increase your velocity as a machine learning practitioner.

So with that, I'd like to hand it over to Andrew to talk about Tensorflow Hub.

Thanks, Jeremiah.

Hi, everybody.

I'm Andrew Gasparovic, and I'd like to talk to you a little bit about Tensorflow Hub, which is a new library that's designed to bring reusability to machine learning.

So software repositories have been real benefit to developer productivity over the past 10 or 15 years, and they're great, first of all, because when you're writing something new, uh, if you have a repositories.

You think?

Oh, maybe I'll check whether there is something that already exists and reuse that rather than starting from scratch.

Um, but a second thing that happens is you start thinking maybe I'll write my code in a way that specifically designed for reuse, which is great because it makes your code more modular.

But it also has the potential to benefit the whole community if you share that coat.

What we are doing with Tensorflow Hub is bringing that idea of a repositories to machine learning.

In this case, Tensorflow hub is designed so that you can create, share and reuse components of ML models.

And if you think about it, it's even more important to have a repositories for machine learning even more so than software development.

Because in the case of machine learning, not only are you re using the algorithm and the expertise, but you're also reusing potentially enormous amount of compute power that went into training the model and while of the training data as well.

So all four of those theat algorithm, the training data, the computer and the expertise all go into a module which is terrible.

Tensorflow hub and then you can import those into your model and north models.

Those modules are pre trained, so they have the weights and the tensor flow graph inside.

And unlike a model, they're designed to be compose herbal, which means that you can put them together like building blocks and add your own stuff on top there reusable, which means that they have common signatures so that you can swap one for another and you re trainable, which means that you can actually back propagate through a module that you've inserted into your graph.

So let's take a quick look at an example.

In this case, we'll do a little bit of image classifications and say that we want to make an app to classify rabbit breeds from photos.

But we only have a few 100 example photos, probably not enough to build a whole image classifier from scratch.

But what we could do is start from a general purpose model, and we could take the reusable part of it.

The architecture and waits there take off the classification, and then we could add our own classifier on top and train it with our own examples will keep that reusable part fixed, and we'll train our own classifier on top.

So if you're using Tensorflow hub, you started tensorflow dot org's slash job, where you can find a whole bunch of newly released state of the art research oriented and the well known image modules.

Some of them are include classification, and some of them chop off the classes, classifications, layers and just output feature vectors.

So that's what we want in our own case in this case, because we're going to add classifications on top.

So maybe we'll choose, um, now as net, which is a, uh, an image module that was created by a neural architecture search.

So we'll choose Nassan at a large version with the future vectors.

So we just paste the U R L for the module into our, um, TF hub code.

And then we're ready to use that module just like a function in between.

The module gets downloaded, and instead she ate it into your graph.

So all you have to do is get those feature vectors, add your own classification on top and output the, um, the new categories.

So specifically, what we're doing is training just declassification part while keeping all of the modules waits fixed.

But the great thing about reusing a module is that you get all of the, uh, training and compute that's gone into that reusable portion.

So in the case of NASA, it was over 62,000 GPU hours that went into finding the architecture and training the model, plus all of the expertise, the testing, um, and the research that went into nasnot You re using all of that in that one line of code.

And as I mentioned before, those modules are trainable.

So if you have enough data, you can do fine tuning with the module.

If you set that trainable parameter to true and you select that you want to use the training graph, what you'll end up doing is training the entire thing along with your classifications.

The caveat being that, of course, you have to lower the learning rate s so that you don't ruin awaits inside the module.

But if you have enough training data, it's something that you can do to get even better accuracy.

And in general, we have lots of image modules on t f.

We have ones that air straight out of research papers like NASA.

Net.

We have ones that are great for production, even once made for on device usage like mobile net plus all of the industry standard ones that people are familiar with, like inception and residents.

So let's look at one more example in this case, doing a little bit of text classification will do, uh, look at some restaurant reviews and decide whether they're positive or negative sentiment.

And one of the great things about TF hub is that all of those modules, because they're tensorflow graphs, you can include things like pre processing.

So the text modules that are available on TF hub take whole sentences and phrases not just individual words, because they have all of the token ization and pre processing stored in the graph itself.

So we'll use one of those and basically the same idea.

We're going to select a sentence.

Embedding module will add our own classification on top and will train it with our own data.

But we'll keep the module itself fixed and just like before, we'll start by Gohan in tensorflow dot org's slash hob and take a look at the text modules that are available in this case.

Maybe we'll choose the Universal Sentence Encoder, which is just recently released based on a research paper from last month.

The idea is that it was trained on a variety of tasks and is specifically meant to support using it with a variety of tasks.

And it also takes just a very small amount of training data.

Thio.

Use it in your model, which is perfect for our example Case.

So we'll use that universal sentencing coder and just like before, will pace the U.

R L into our code.

The difference here is we're using it with a text embedding column.

That way we can feed it into one of the high level tensorflow estimators in this case, the d n n classifier.

But you could also use that module it, like I showed any earlier example, calling it just a za function.

If your are using the Texan embedding column that also just like in the other example, can be trained as well and just like any other example.

It's something that you can do with a lower learning rate if you have a lot of training data and it may give you better accuracy and so um we have a lot of text modules available on T F.

We actually just added three new languages to the nlm modules Chinese, Korean and Indonesian.

Those air all trained on Jean use training data.

And we also have a really great module called Elmo from some recent research which understands words in context.

Ah, and of course, the universal sentencing coder, as I talked about.

So just to show you for a minute, some of those girls that we've been looking at, um maybe we'll take apart the pieces here.

Tfl dot dev is our new source for Google and selected partner published modules.

In this case, this is Google.

That's the publisher.

And the universal sentence encoder is the name of the module.

The one at the end is a version number.

So Tensorflow Hub considers, um, modules to be immutable.

And so the version number is there so that if you're, you know, doing one training run and then another, you don't have a situation where the module chain changes unexpectedly.

So all modules on TF abduct ever version that way, and one of the nice things about those you Earl's if you paste them into a browser.

You get the module documentation, the idea being that maybe you read a new paper, you see.

Oh, there's a U.

R L for T F up module and that you paste it into your browser.

You see the documentation, you paste it into some code, and in one line you're able to use that module and try out the new research.

And speaking of the universal encoder, the team just released a new light version, which is a much smaller size.

It's about 25 megabytes, and it's specifically designed for cases where the full text module wouldn't work for doing things like on device classifications.

Also today, we released a new module from Deep Mind, this one you can feed in video, and it will classify and detect the actions in that video.

So in this case, it correctly guesses the video is of people playing cricket.

And of course, we also have a number of other interesting modules.

There's a generative image module which is trained on Celebi.

It has a progressive gan inside and also the deep local features module, which I can identify the key points of landmark images.

Those air all available now on D F.

And last but not least, I wanted to mention that we just announced our support for Tensorflow.

J s.

So, using the Tensorflow Js converter, you can directly convert a T F of module into a format that can be used on the Web.

It's, ah, really simple integration to be able to take a module and use it in the Web browser with tensorflow dot Js, and we're really excited to see what you build with it.

So just to summarize, Tensorflow Hub is designed to be a starting point for reusable machine learning, and the idea is just like with a software repositories.

Before you start from scratch, check out what's available on Tensorflow Hub and you may find that it's better to start with the module and import that into your model rather than starting the task completely from scratch.

We have a lot of modules available, and we're adding more all the time, and we're really excited to see what you built.

So thanks.

Next up is Jeremiah to talk about T F servant.

All right.

Thank you, Andrew.

So next tensorflow serving.

This is gonna be how we deploy modules are deployed models to see get a sense for where this falls in the machine learning process, right?

We start with our data.

We use tensorflow to train a model in the output.

Our artifact.

There are these models, right?

These air saved models.

It's a graphical representation of the data flow.

And once we have those, we want to share them with world.

That's where Tensorflow serving comes in.

It's this big orange box.

This is something that takes our models and expose them to the world through a service so clients can make requests.

Tensorflow serving will take them, run the inference, run the model, come up with an answer and return that in a response.

So Tensorflow serving is actually the libraries and binaries.

You need to do this to do this production grade inference overtrained tensorflow models.

It's written in C++ and supports things like G.

R.

P.

C.

And plays nicely with kubernetes.

So to do this well, it has a couple of features.

The first and most important is it supports multiple models.

So on one tensorflow model server, you can load multiple models, right?

And just like most folks probably wouldn't push a new binary right to production.

You don't want to push a new model right to production, either.

So having these multiple models in memory lets you be serving one model on production traffic and load a new one and maybe send it some canary.

Request send.

It's, um, que es requests.

Make sure everything's all right and then move the traffic over to that new model.

And this supports doing things like reloading.

If you have a stream of models, you're producing tensorflow, serving well transparently.

Load the new ones and unload the old ones.

We've built in a lot of isolation.

If you have a model that serving a lot of traffic in one thread and it's time to load a new model, we make sure to do that in a separate thread.

That way, we don't cause any hiccups in the thread that serving production traffic.

And again this entire system has been built from the ground up.

To be very high throughput.

Things like selecting those different models based on the name or selecting different versions that's very, very efficient, similarly has some advanced matching right This way we can make use of accelerators.

We also see improvements on Standard CP use with dispatching and then lots of other enhancements.

Everything from protocol buffer, magic toe lots more and this is really what we use inside Google to serve tensorflow.

I think there's over 1500 projects that use it.

It serves somewhere in the neighborhood of 10 million Q P s, which ends up being about 100 million items predicted per second.