Placeholder Image

Subtitles section Play video

  • some examples.

  • The first thing you need is data.

  • You may want to validate results or test ideas on a common public

  • dataset.

  • It includes a large and rapidly growing collection of datasets

  • you can get

  • started with easily and combined with tf.data it is simple to

  • wrap your own data too. Here is a small sample of the datasets

  • available and all of these and many nor are included there.

  • Then with Keras, you can express the

  • models just like you are used to thinking about it.

  • Standard package is fit with model fit and evaluate as well.

  • Since deep learning models are often

  • commutationally expensive you way want

  • to try scaling this across more than one device.

  • Starting from a pre-trained model or component also works

  • well to reduce some

  • of this computational cost. TensorFlow helps provide a large

  • collection of pretained components you

  • can include in your model and Feinstein

  • -- fine tune for your dataset.

  • Keras comes with everything you might need for a typical

  • training job.

  • Sometimes you need a bit more control. For example when you

  • are exploring new kinds of algorithms.

  • Let's say you wanted to build a custom

  • encoder for machine translation, here is

  • how you could do this by subclassing the model.

  • You can even customize it training loop

  • to get full control over the gradients

  • and optimization process.

  • While training models, whether packaged with Keras or more

  • complex ones, it is often valuable to understand the

  • progress and even analyze the muddle in detail.

  • TensorFlow board provides a lot of visualization to help with

  • chis and

  • comes full integration with Colab and

  • other Jupyter notebooks allowing you to

  • see the same visuals. All of these features are available in

  • TensorFlow 2.0 and I am really excited to announce

  • our alpha release is available for you as of today.

  • [Applause] Many of you in the room and

  • across the world really helped with lots of work to make this

  • possible. I would really like to take this moment to thank you

  • you all. Please give yourself a round of applause.

  • We really couldn't have done this

  • without you.

  • In addition to all the great improvements we talked about,

  • this

  • release comes with a converter script

  • and compatibility module to give you

  • access to the 1.X APIs. We are working for a full release over

  • the next quarter.

  • There is a lot of work going on to make TensorFlow 2.0 work well

  • for you. You can track the progress and provide

  • feedback on the TensorFlow GitHub projects page.

  • You asked for better documentation, and

  • we have worked to streamline our docs

  • for APIs, guides and tutorials. All of this material will be

  • available

  • today on the newly redesigned TensorFlow.org website where you

  • will find examples, documentation and tools to get

  • started. We are very excited about these changes and what's

  • to come. To tell you more about improvements in TensorFlow for

  • research and production, I would like to welcome Megan Kacholia

  • on stage. Thank you.

  • Thank you. Thanks Rajat. TensorFlow has always been a

  • platform for research to production.

  • We just saw how TensorFlow high-level APIs make it easy at

  • a get started and build models and now let's talk about

  • how it improves experimentation for research and let's you take

  • models from

  • research and production all the way through. We can see this in

  • paper publications

  • which are shown over the past few years in this chart.

  • Powerful experimentation begins and

  • really needs flexibility and this begins with eager execution

  • and TensorFlow 2.0 every Python command is immediately executed.

  • This means you can write your code in

  • the style you are used it without having to use session

  • realm. This makes a big difference in the realm of

  • debugging.

  • As you iterate through, you will want

  • to distribute your code on to GPUs and

  • TPUs and we have provided tf. function turning your eager code

  • into a graph function-by-function. You get

  • Python control flow, asserts and even print but can convert to a

  • graph any time you need to, including when you are ready to

  • move your model into production. Even with this, you will

  • continue to get great debugging.

  • Debugability is great not just in eager

  • but we have made improves in tf. function and graph. Because of

  • the mismatch inputs you get an error.

  • As you can see, we give information to

  • user about the file and line number where the error occurs.

  • We have made the error messages concise, easy to understand and

  • actionable. We hope you enjoy the changes and they make it

  • easier to progress with the models. Performance is another

  • area we know researchers as well as all users for that matter

  • care about. We have continued improving core performance in

  • TensorFlow. Since last year, we have sped up

  • training on eight Nvidia TeslaV100s by LLGS double.

  • With Intel and MKL acceleration we have gotten inference speed

  • up by almost three times. Performance will continue to be

  • a focus of TensorFlow 2.

  • 0 and a core part of our progress to final release.

  • TensorFlow also provides flexibility

  • and many add on libraries that expand and extend TensorFlow.

  • Some are extensions to make certain

  • problems easier like tf.text with Unicode.

  • It helps us explore how we can make machine learning model

  • safer by a tf.privacy. You will hear new announcements on

  • reinforcement learning and tomorrow we

  • will discuss the new tf. federated library.

  • It is being applied to real world applications as well.

  • Here are a few examples from researchers at Google where we

  • see it applied to areas like data centers and making them

  • more efficient. Our apps like Google Maps, the one in

  • the middle, which has a new feature

  • called global localization and combines street service.

  • And devices like the Google Pixel that use machine learning

  • to improve depth estimation to create better portrait mode

  • photos like the ones shown here. In order to make these real

  • world applications a reality, you must be able

  • to take models from research and prototyping to launch and

  • production. This has been a core strength and focus for

  • TensorFlow. Using TensorFlow you can deploy models on a number of

  • platforms shown here and models end up in a lot of places so we

  • want to make sure TensorFlow works across all these servers,

  • Cloud, mobile,

  • edge devices and Java and number of platforms.

  • We have products for these.

  • TensorFlow Extended is the end the end platform.

  • In orange, shown here, you can see the

  • libraries we have Open SourceSourced so far. We are

  • taking a step further and providing components built from

  • these libraries that make up an end-to-end platform. These are

  • the same components used internally in thousands of

  • production systems powering Google's most important

  • products. Components are only part of there story.

  • 2019 is the year we are putting it together and providing you

  • with an integrated end-to-end platform.

  • You can bring your own orchestrator.

  • Here is airflow or raw Kubernetes even.

  • Not matter what orchestrate you

  • chose, the items integrate with the metadata store.

  • This enables experiments, experimentation, experiment

  • tracking and model comparison and things I am sure you will be

  • excited about and will help you as you iterate through. We have

  • an end-to-end talk coming up from Clemens and his team and

  • they will

  • take you on a complete tour of

  • TensorFlow Extended to solve a real problem. TensorFlow Lite is

  • our solution for

  • running models on a mobile and IOt hardware.

  • On device models can be

  • reore responsive and keep users on device for privacy.

  • Google and partners like

  • iqiyi provide all sorts of things. TensorFlow Lite is about

  • performance.

  • You can deploy models to CPU, GPU, and even EdgeTPU.

  • By using the latest techniques and adding support for OpenGL

  • and metal on

  • GPUs and tuning performance on EdgeTPUs we are constantly

  • pushing the limits of what is possible. You should expect

  • greater enhancements in the year ahead.

  • We will hear details from Raziel and colleaguess coming up later.

  • JavaScript is the number one

  • programming laj language in the world

  • and until recently hasn't benefited from

  • all the machine learning tools. We released TensorFlow.js last

  • year.

  • Since then we have seen huge adoption in the JavaScript

  • community with more

  • than 300,000 downloads and a 100 contributors. We are just at the

  • beginning given how

  • big the JavaScript ecosystem is.

  • Today we are excited to announce TensorFlow.js 1.0.

  • We have a library of off the shell models for common ML

  • problems and we

  • have adding support for more platforms

  • where jafs runs and a huge focus in TensorFlow.js is on

  • performance improvements. Compared to last year, mobile

  • net

  • inference and browser is now nine times faster. You will

  • learn more about these advances in our talk later in the day.

  • Another language we are excited about is Swift.

  • It is reexamining what it means for performance and usability.

  • With a new programming model, it intends to bring further

  • usability. We are announcing Swift for TensorFlow is now at

  • 0.2 and ready for you to experiment with,

  • try out and we are really excited to be bringing this to

  • come community. In addition to telling you about

  • version 0.

  • 2 we are excited to announce Jeremy Howard of fast.ai is

  • writing a new course and Chris and Brennan will tell you a lot

  • more about this later today.

  • To recap everything we have shown you,

  • TensorFlow has grown to a full Eco

  • system from research to production from server to

  • mobile. This growth has been fuelled by there community and

  • wouldn't be possible without the community.

  • To talk about what we are planning for you and with you in

  • 2019, I will hand it over to Kemal.

  • [Applause] Thank you, Megan.

  • Hi, may name is Kemal and I am the product director. We are

  • celebrating the most important part of what we are building and

  • that is the community.

  • I love building developer platforms.

  • I used to be a developer and now I

  • enable developers by building a better platplatform. We turned

  • to the community and consulted with all of you on important

  • product decisions.

  • We received valuable feedback and couldn't have built 2.0

  • without you.

  • We created special interest groups, or

  • SIGs, like networking and TensorBoard to name a few.

  • SIGs are a great way to build the pieces of TensorFlow they

  • care the most about. We wanted to hear more about what you

  • were building and launched the power by TensorFlow campaign.

  • We were amazed by the creativity of the

  • products. After three years, our community is really thriving.

  • There are 70 machine learningGD Es

  • around the world, 1800 contributors and countless more

  • doing amazing work to help make TensorFlow successful.

  • On behalf of the whole TensorFlow team

  • we want to say a huge thank you. [Applause]

  • We have big plans for 2019 and I would like to make a few

  • announcements.

  • First, as our community grows, we welcome people who are new to

  • machine

  • learning and it is really important to provide them with

  • the best educational material. We are excited to announce two

  • new online courses. One is with deeplearning.ai and published in

  • the Coursera

  • platform and the other is with Udacity. The first batch of the

  • lessons is available right now and provide an awesome

  • introduction to TensorFlow for developers. They require no

  • prior knowledge to machine learning.

  • I highly encourage you to check them out.

  • Next, if you are a student for the very first time, you can

  • apply to the Google summer of code program and get to work

  • with the TensorFlow engineering team to

  • help build a part of TensorFlow.

  • I also talked about the power by TensorFlow campaign.

  • We are excited and decided to launch a 2.0 hackathon on

  • DevPost to let you share

  • your latest and greatest and win cool prizes. We are really

  • excited to see what you are going to build.

  • Finally, as our ecosystem grows, we are having a second day of

  • the summit, but we really wanted to do something more. We wanted

  • a place where you can share what you have been building on

  • TensorFlow.

  • So we are excited to announce TensorFlow World.

  • A week long conference dedicated to open source collaboration.

  • This conference will be co-presented by O'Rielly media

  • and TensorFlow and held in Santa Clara at the end of October. Our

  • vision is to bring together the awesome TensorFlow World and

  • give a place for folks to connect with each other.

  • I would like to invite on stage Gina Blaber to say a few words

  • about the conference.

  • [Applause]

  • Thank you, Kemal.

  • O'Rielly is a learning company with a focus on technology and

  • business. We have strong ties with the open source community

  • as many of you know,

  • and we have a history of bringing big ideas to life.

  • That's why we are excited about partnering with TensorFlow to

  • create this new event that brings machine

  • learning and AI to the community.

  • The event is TensorFlow happening October 28th-31 in

  • Santa Clara. When I say community, I mean everyone.

  • We want to bring together the entire TensorFlow community of

  • individuals and teams and enterprises. This is it place

  • where you will meet experts from around the world, the team that

  • actually creates TensorFlow, and the companies and enterprises

  • that will help you deploy it.

  • We have an open CFP on the TensorFlow World site. I invite

  • you all to check it out and send in your proposals soon so your

  • voice is heard at that event. We look forward to seeing you at

  • TensorFlow World in October. Thank you.

  • [Applause] Thank you, Gina. This is going

  • to be great.

  • Are you guys excited?

  • So we have a few calls to action for you.

  • Take a course, submit a talk to TF World. The grand prize for

  • hackathon on DevPost will include free tickets to

  • TensorFlow World. You know, one thing I love is to hear about

  • these amazing stories of people building awesome stuff on top of

  • TensorFlow.

  • As a team, we really believe that AI advances faster when

  • people have access to our tools and can then apply them to

  • the problems they care about in ways that we never really

  • dreamed of. And when people can really do that, some special

  • things happen. I would like to share when you something really

  • special.

  • Looking at this book, page by page, and

  • trying to decipher, read and transcribe whatever is there

  • takes an enormous amount of time.

  • It would require an army army polyographer.

  • Machine learning enabled us to solve

  • problems that up to 10-15 years ago we

  • thought were unsolvable.

  • You have thousands of images of dogs and cats in the internet

  • but very little

  • images of ancient manuscripts. We involved high school students

  • to collect the data. I didn't know much about machine

  • learning in general, but I found it easy to create a TensorFlow

  • environment. When we were trying to figure

  • out which model worked best for us, Keras was the best solution.

  • The production model runs on TensorFlow

  • layers and estimator interface. We experimented with binary

  • classification with fully connected

  • networks and finally we moved to

  • convolutional neural networks and classification.

  • When it comes to recognizing single

  • characters, we can get 95% average accuracy.

  • This will have an enormous impact.

  • We will have a massive quantity of historic information

  • available. I think solving problems is fun.

  • It is a game against myself and how good I can do.

  • That is -- [Applause]

  • This is such a great story.

  • I think about the scholars who wrote

  • these manuscripts, they cannot have

  • imagined centuries later people bringing their work to life.

  • We are thrilled to have

  • Elena. The team and I will be around. Come and say hi. With

  • that, I am going to hand it over

  • to Martin who will talk about TensorFlow 2.0. Thank you.

  • Thank you, Kemal. I am Martin Wicke.

  • I am the engineer lead for TensorFlow 2.

  • 0 and I will unsurprisingly talk about TensorFlow 2.0. TensorFlow

  • has been extremly successful and inside of Google and outside of

  • Google it has grown into a vibrant community that when we

  • started it we were not able to imagine.

  • Our users are building these engenous,

  • clever, elegant things every day from

  • art to music to medieval manuscripts, science and

  • medicine. The things people created were

  • unexpected and beautiful. We learned a lot from that.

  • We learned a lot from how people did it.

  • TensorFlow has enabled all this

  • creativity and jump started this whole

  • AI democraization which is great.

  • It has been hard to use sometimes. Sometimes a little

  • bit painful.

  • So, you know, using sessions wasn't the

  • most natural thing to do when coming from regular Python. The

  • TensorFlow API has grown over time. It got a little bit

  • cluttered and confusing. In the end, you can do everything with

  • TensorFlow, but it wasn't maybe always clear what was the best

  • way.

  • We have learned a lot since we started this and we reallyized

  • that you need rapid prototyping and easier debugging. There is a

  • lot of clutter. So, we are fixing this, addressing these

  • issues with a new version of TensorFlow.

  • Today, we are releasing an alpha of TensorFlow 2.0. Many of you

  • have participated in the

  • design reviews that went into this for all the features we

  • have blended. Those of you living on the bleeding edge of

  • development have played it with, installed it, tested it, found

  • issues and fixed issues. Thank you very much for your help.

  • You can now install the alpha release of TensorFlow 2.

  • 0 using just pip install pre-TensorFlow and you can go

  • and play with it. What has changed?

  • When we created TensorFlow 2.

  • 0, the biggest flow was flexibility. We have integrated

  • Keras with TensorFlow. It gives you a path of how to develop

  • and deploy models and has a well established API that makes

  • people productive and we know it gets you started quickly.

  • TensorFlow also includes a complete implementation of

  • Keras, of course, but we went further. We have extended Keras

  • so that you can

  • use all of the advanced features in

  • TensorFlow directly from tf.Keras.

  • The other change is eager execution.

  • It used declarative style and was distance from running

  • Python. In TensorFlow 2.

  • 0, TensorFlow

  • behaves just like

  • the Python style. All of that stuff is not going away.

  • It is just getting easier. TensorFlow 2.0 is a major

  • release which means we have

  • the ability to clean it up and we did, like a lot. We had a lot

  • of duplicate functionality we have removed.

  • We have consolidated the API and organized the API. This is not

  • only with TensorFlow itself

  • but the whole ecosystem of tools that has grown around

  • TensorFlow. We have spent a lot of effort to align these

  • interfaces and defined exchange

  • formats that will allow you to move all throughout this

  • ecosystem without hitting any barriers. We have removed a lot

  • of stuff from TensorFlow but it doesn't mean -- it

  • means that it has gotten more flexible not less. It retains

  • and expands on the flexibility and we have created a more

  • complete low level API that exposes all

  • of the internal used ops to the user.

  • We provide inheritable interfaces for

  • the crucial concepts like variables and checkpoints.

  • This allows framework authors to build

  • on top of TensorFlow and maintain

  • interoperability with the rest of the ecosystem.

  • There will be a talk about tf.

  • agents later and Sonnet by DeepMind tomorrow. You will see

  • how this works in practice. The real question is of course what

  • do

  • I have to do be apart of this great new era of TensorFlow and

  • we know it is

  • always hard to upgrade to anything and that is especially

  • true for major up grades. We will start the process of

  • converting one of the largest codebases in the world to

  • TensorFlow 2.0 and we will be writing a lot of migration

  • guides. We have started already and some are

  • online and will provide a lot of best practices for you. If we

  • can do it at Google, you can probably do it too. We are

  • giving you, and us, a lot of

  • tools to make this transition easier.

  • We are shipping a module called tf.contrib which contains all of

  • the complex API. If you rely on a specific function you

  • can find it there except for tf. contrib which isn't going to be

  • included but we have created a community supported alternative

  • that support your use case if you are relying on something in

  • there.

  • We are also publishing a script which automatically updates your

  • code so it runs on TensorFlow 2.0. Let me show you how that

  • works. Let's say I want to convert this

  • program which is a simple language model chain of

  • Shakespeare.

  • What where do is run TF upgrade 2. There is just a handful of

  • things it changed in this case. You can see, if you look at what

  • it changed, there is some functions it renamed.

  • A lot of the reorganization of the API.

  • It renamed multi nomial to random

  • categorical. A lot had functions changed or added and deleted.

  • The script will make those

  • changes.

  • If all else fails, there is not an equivalent and we will use

  • the compatibility module I talked about earlier.

  • If there is no perfect replacement, we will use that to

  • make sure your code still works as expected after the

  • conversion. It is pretty conservative.

  • For instance, the atom optimizer is a subtle behavior change that

  • will mostly not affect you but we will convert that just in

  • case it might. Once the conversion is complete, and

  • there are no errors, then you can run the new program against

  • TensorFlow 2.0 and it will work. That's the idea. It should be

  • pretty easy for you. Hopefully you won't get trouble. Note that

  • this automatic conversion

  • will fix it so it WOSH works but won't fix the style to the new

  • TensorFlow 2.0 style.

  • That can not be done automatically and

  • there is a lot more to know.

  • If you want to know more, check out the

  • talk at 10:30 tomorrow. Let me give you an idea about the

  • timeline of all this. We have cut 2.0 alpha today or

  • yesterday.

  • Took some time building it. We are now working on implementing

  • some missing features that we know about, converting

  • libraries, converting Google.

  • There is lots of testing and optimization that will happen

  • and we are targeting to have a release candidate in spring.

  • Spring is a flexible concept but in spring.

  • After we have this release candidate, it still has to go

  • through release testing and integration testing.

  • I expect that to take longer than regular releases but that

  • is one you can see an RC and eventually a final. So, if you

  • want to follow along with the process, please, do so.

  • All of this is tracked online on the GitHub TensorFlow 2.0

  • tracker. If you file any bugs, those are

  • up there

  • as well so everybody knows what is going on.

  • TensorFlow 2.0 really wouldn't be possible without you. That

  • has been the case already but in the future even more so. We will

  • need you to test this, we will

  • need you to tell us what works and doesn't. Please install the

  • alpha today. We are extremely excited to see what you can

  • create. Check out the docs.

  • They are at tensorflow.org/r2.0. With that, I think you should

  • hear about what it is like to work with TensorFlow 2.0. We

  • will have plenty of content over the next couple days to tell you

  • exactly about all the different aspects of it

  • but we will start with Karmel who will speak about high-level

  • APIs in

  • TensorFlow 2.0.thank you.

  • Hi, my name is Karmel Allison.

  • Martin just told you about TensorFlow 2.0 and how to get

  • there and I will tell you a little bit more about the

  • high-level APIs.

  • What do I mean when I say high-level APIs? With our

  • high-level APIs, we bring you

  • tools to help you easily build your models and scale them.

  • As Martin mentioned, a couple years ago, TensorFlow adopted

  • Keras as one of the high level APIs we offered. Keras is a

  • specification for model bidding. It works with multiple machine

  • learning frameworks and it provides a shared

  • language for defining layers, models,

  • losses, optimizers and so on.

  • We

  • implemented tf.Keras. Raise your hand if you are used this? Keras

  • is just that. Simple.

  • It was built from the ground up to be

  • Pythonic and easy to learn and has been instrumental in

  • inviting people into the machine learningworld.

  • It is easy for you to design and modify your model architectures

  • without needing

  • to write tons of boilerplate. At the same time, by replying on

  • inheritance and interfaces, Keras is

  • extremely flexible and customizeable.

  • I can define architectures, modify what happens and even

  • change the whole training loop, if I want to.

  • Which is to say Keras is simple and effective.

  • Anyone can figure out how to use it but we had a problem. Tf.

  • Keras was built for smaller problems and many machine

  • learning need to operate at a larger scale. We have

  • estimators.

  • They were built from the ground up to

  • distribute and scale with fault tolerance across hundreds of

  • machines no questioned asked.

  • This is the much love wide and deep model.

  • A work horse that took mean years to

  • define but is a built-in for training and deployment.

  • Estimators are powerful machines but you told us there is a steep

  • learning curve and it is not always easy to figure out which

  • parts to connect where. We have learned a lot building estimator

  • and tf.Keras and reached the point we don't

  • think you should have to chose between a simple API and

  • scalable API.

  • We want a higher level API taking you from endness to plan

  • and scale, no questions asked. In TensorFlow 2.0, we are

  • standardizing on the Keras API

  • for building layers and models but

  • bringing the power estimators into tf.Keras so you can move

  • from prototype, distributes training and production serving

  • in one go.

  • OK. How are we doing this? This is a tf.

  • Keras model definition in TensorFlow 1.13.

  • This is the same model definition in 2.0.

  • The code is the same but what is different? In 1.13 this built a

  • graph-base model that ran a session under the hood. In 2.

  • 0, the same model runs in eager mode without any modification.

  • This let's you take advantage of everything eager can do for us.

  • We see our dataset pipeline

  • behaves like a num

  • py array. We are able to achieve this performance with datasets

  • and Keras by taking advantage of graphs even in an eager context.

  • Eager makes debugging and prototyping

  • easy and all the while Keras builds an eager-friendly

  • function under the hood that tracks your model and the

  • TensorFlow Runtime takes care of optimizing for performance and

  • scaling. You can explicitly run your model in eager mode with

  • the flag run eagerly even though for this example it might be

  • overkill, you can see run eagerly allows you to use Python

  • control flow for

  • debugging while prototyping your model.

  • Another big change in TensorFlow 2.

  • 0 is we have consolidated many headings

  • and maing it easier to know what you should use and when.

  • We have one set of optimizers that work

  • across TensorFlow in and out of eager

  • mode on one machine or distributed.

  • You can set opt amam -- optimizers.

  • We have one set of metrics that encompasses all the former tf.

  • metrics and Keras metrics and allow for

  • easy subclassing if you want even more.

  • Losses, similarly, have been consolidated into a single set

  • with a number of frequency used built-ins and an easily

  • customizeable interface if you chose. We have one set of layers

  • built in the style of Keras layers which means they

  • are class based and

  • fully configureconfiguable. One set of layers that we took

  • particularly care with were the RNN layers in TensorFlow.

  • Raise your hand if you are used RNN layers in TensorFlow?

  • These slides are for you. In TensorFlow version 1 we had

  • several

  • versions of

  • LSTMs and GRUs. In 2.0, there is one version of the LSTM

  • layer and one version of the

  • gru level. This runs the kernel but falls

  • back to

  • CPU ops if you don't have GPUs available. In addition to all of

  • those, if you

  • need layers that are not included in the

  • built in set, tf.

  • Keras provides a subclass that is easy to customize. This is

  • how TensorFlow addons operate

  • providing specialized and complex layers, metrics, and so

  • on by building

  • on top of the Keras base classes.

  • We streamlined APIs. That is a start.

  • The next thing we did was integrate.

  • We found easily configurable data parsing. In TensorFlow 2.

  • 0, you can use Feature Columns to parse data and feed directly

  • to downstream layers.

  • This works with Keras and estimators so you can mix and

  • match to create reusable data input pipelines. OK. You have

  • data in your model and you are ready to train.

  • One of the most loved tools we have is TensorBoard and I am

  • pleased to see TensorBoard integration with Keras is as

  • simple as one line. We have the TensorBoard callback

  • to our model when training here and this gets us both our

  • training progress, here we

  • see accuracy and loss, and a conceptual graph representing

  • the model we built layer by layer.

  • As an add in bonus, it includes full callback so you can better

  • understand

  • the placement and find ways to minimize bottlenecks. Speaking

  • of performance, one of the

  • exciting pieces we are bringing TensorFlow 2.

  • 0 is the -- these APIs have been

  • designed to be easy to use, out of the box performance and be

  • versetle. How do they work?

  • With Keras, you can add and switch distribution patterns

  • with a to a lines of code.

  • We are distributing this across multiple GPUs.

  • Because we have integrated distribution strategies

  • throughout Keras and TensorFlow you get a lot with these few

  • lines of code.

  • Data is prefetched batch by batch and mirrored in sync using

  • all reduce and we see greater than 90% scaling

  • efficiencies meaning you can scale your model up without

  • changinging code and without loosing any of the conveniences

  • of Keras. Once you have trained your model, you are likely going

  • to want to package it for use with production systems, mobile

  • phones or other programming laj -- language.

  • This functionality is experimental but today and going

  • into 2.0 you can easily export your models for use with tf.

  • serving, tf.lite and more.

  • You can reload the models back into that

  • you be Python to continue training and using in your

  • normal workflow. That is where we are today in the alpha.

  • We are obviously not done yet. Let me tell you a little bit

  • about what is coming up in the next few months.

  • We talked about distributing over multiple GPU but that is

  • just the start.

  • We can swap out strategy and scale up to multiple nodes.

  • Here we cake the same Keras model and distribute it across

  • multiple machines

  • working in sync using collective ops.

  • The multi-worker mirror strategy supports Ring and allows great

  • performance. You can try it out today. If you

  • are interested in a TensorFlow

  • native solution to synchronous multi working training. We are

  • excited to say using Keras and distribute strategies you will

  • be able

  • to use the same code to distribute models on TPUs. When

  • released, this strategy will work

  • on Google Cloud

  • TPUs and Colab which can access TPUs directly. As we approach

  • the final 2.0 release we will continue to bring scalability

  • into Keras and allow parameter service strategy, the multi

  • node asynchronous training that estimator does today.

  • For those that want a higher level API, we will be adding

  • support for partition variables across many machines for some

  • of the largest scale models like those we run at Google.

  • And that's just a sneak peek so please check out the 2.

  • 0 preview and check out Keras if you haven't already.

  • You will hear more about Keras and high-level APIs throughout

  • the day and

  • we have a workshop tomorrow that Martin mentioned.

  • With that, let me introduce Raziel who

  • will tell you more about TFLite. Thanks, Karmel. Hi, everyone.

  • My name is Raziel and I am an engineering lead.

  • Today I will be covering at -- a lot of the work the team has

  • done

  • over the past year that you can take advantage of right away. I

  • am cover a lot of work we have for this year. Before we jump

  • into that, let me give

  • you a brief introduction of what TensorFlow Lite is.

  • TensorFlow Lite is our infrastructure framework to run

  • machine learning on device.

  • The reason why we decided to built TensorFlow Lite was

  • because machine learning you think of it as running on the

  • server, but more and more it is

  • moving to edge devices like mobile

  • devices, cars, Wearables, you name it. There is more machine

  • learning moving to different kinds of devices. There is good

  • reason for that.

  • One, you have access to a lot more data

  • because you have access to the algorithm, camera, and don't

  • have to

  • stream it all the way to the server so you can do a lot of

  • stuff.

  • It means you can build faster and close-knit interactions.

  • Another advantage of running machine

  • learning on devices is it has a strong privacy preserving

  • component because the data stays on the device and doesn't

  • have to go to the server. However, that means basically by

  • building machine learning on devices we can build products

  • that otherwise

  • wouldn't be able to really work if you

  • really rely only on server-side execution. However, you know, it

  • is hard. Perhaps one obvious thing is you don't have the same

  • compute power you have on the server. If you have a new shiny

  • machine learning model that consumes too much compute power,

  • it is not like in the server where you can throw perhaps more

  • servers at it and be done with it. You have to be mindful of

  • the resources that you have on the device.

  • Another of those resources is the memory. You have restricted

  • memory so you have to be careful about how you craft your

  • model, perhaps try some compression techniques.

  • Last but not least, many of these devices are battery

  • powered and even if you have a model that fits in memory and

  • can execute, well, you don't want to consume all the battery

  • in a battery powered device.

  • For all these reasons, we built TensorFlow Lite and we want to

  • make deploying machine learning on edge devices as easily as

  • possible. What can you do with it right now? You can do a lot

  • of things. These are different tasks that are supported by

  • TensorFlow Lite already. If you look at it when you hear machine

  • learning you typically fall into one of these tasks. You know,

  • text-to-speech, image

  • processing, audio processing, a lot of

  • content generation so really, like, everything is already

  • there for you to

  • take advantage of and because we are already able to support

  • these tasks, it

  • means it has a lot of usage already. TensorFlow Lite is

  • currently deployed

  • in over 2 billion devices in production.

  • This not only means Google properties, some of which are

  • core to Google like

  • the assistant and Google photos but come from other large

  • companies and those are frameworks.

  • A lot of our infrastructure is power

  • and frameworks.

  • I can tell you many reasons why it is a good idea to use our

  • infrastructure but we thought it would be best if we bring

  • two of our component clients to tell you

  • how they use TensorFlow Lite.

  • Our first presenter is Alex coming from the Google Assistant

  • which has a wide variety of devices it is running. Let's

  • hear why they are using

  • TensorFlow Lite and how they are doing.

  • [Applause] Thanks, Raziel. I am Alex. I am

  • an engineering lead on our embedded speech recognition team

  • at Google. One of the key applications for speech

  • recognition at Google is the Google Assistant.

  • Google recently announced the Google Assistant is on about a

  • billion devices ranging from phones, speakers, smart

  • displays, cars, TVs, laptops, Wearables,

  • you name it we are trying to put the assistant into it. That

  • means high end devices, low end

  • devices, battery powered, plugged in,

  • and a large range of operating systems. The neural networks I

  • build have to be able to run anywhere.

  • For assistant, we have a couple key speech on-device

  • capabilities. The first is the hot word.

  • You need to be able to say

  • hey, Google to activate the device and we need to make sure

  • it is your voice.

  • We have to have a tiny memory and

  • computational input and we are extremely latency sensitive.

  • Our other major application is on device speech recognition. We

  • use this when you are offline or have a bad network connection.

  • Instead of using the server, we run speech recognition on the

  • device.

  • Here we are trying to run large models as fast as possible to

  • get the best probably accuracy. It is an a high load returning

  • into short burst typically. Over the years, we have been working

  • on

  • this for a long time, before there was TensorFlow Lite,

  • before there was TensorFlow, so we have been investing a

  • lot for a long time in building our own on-device neural net

  • libraries and shipping them to production in Google product for

  • a long time.

  • As a team, we have invested in keeping

  • the code size tiny, memory small, and optimizing for all

  • different platform and all of our code after many years

  • production use is hardened and as bug-free as you can get it,

  • hopefully.

  • Over the last year, we decided to

  • migrate to TFLite.

  • This wasn't an easy decision for us and

  • as we switched over we checked

  • carefully to make sure TFLite would

  • beat our existing models and we have moved all over in the last

  • year. We are excited because this lays the

  • ground work for us moving to a new standard that is being

  • widely adopted

  • and we think TFLite will help us

  • accelerate models on GPUs and EdgeTPUs and new accelerators

  • coming. Thanks.

  • I will turn it back over to Raziel.

  • [Applause] Thanks, Alex. So, Alex analyzed

  • the key goals for TensorFlow Lite.

  • We want to be able to support your current-use-cases, but

  • perhaps equally important or more important, is we want

  • to cater to your needs that you will have 1-2 years in the

  • future.

  • Our next presenter is Huiije from a China-based company that

  • has several applications and hundreds of millions of users.

  • Let's welcome him.

  • Thanks, Raziel.

  • I am Huiije from NetEase in Youdao.

  • We have built a lot of dictionary translation and note

  • taking applications in China.

  • We have over 800 million users in total.

  • We also have 22 million daily active users.

  • In the past years, we have built

  • several dictionary translation and note taking applications. We

  • have the most popular dictionary

  • application in China, for example, which

  • is Youdao dictionaries and we have the

  • most popular translator in China and the most popular dictionary

  • and language

  • learning application in India which is Youdao dictionary.

  • We provide features for users in these applications so they can

  • conveniently look up words in the scenario such as,

  • for example, you can use your camera to

  • recognize the words from the images and

  • do the OCR translation on your devices.

  • We use the TensorFlow Lite to use

  • theOCR and translation here. We have provided the photo

  • translation in your applications.

  • For example, you can use camera to take

  • a photo from the many scenario and it

  • will do the whole image of OCR and

  • translate the text into another language.

  • Then it will erase the text on the

  • image and replace the original text to the translated text to

  • present users with the translation.

  • In this scenario, we also use the

  • TensorFlow Lite to accelerate our OCR

  • and translation services here.

  • The OCR and translation is very sensitive to the binary size and

  • also

  • computational resources and responding time.

  • So we chose the TensorFlow Lite to

  • accelerate our abilities on device to provide more efficient

  • on device inference here. That is why we chose the TensorFlow

  • Lite to do this. Thanks.

  • [Applause] Really exciting to see how our

  • >> INSTRUCTOR: -- infrastructure is used

  • by hundreds of users around the world. TensorFlow Lite is geared

  • not only towards Google but everybody should be able to take

  • advantage of it.

  • For the rest of the presentation, I will cover what

  • I said at the start.

  • A lot

  • of features that are available over the past year and roadmap

  • for this year.

  • We organized our engineering in four things.

  • First is usability, overall it means we want to be able to get

  • you up and running as easily and fast as possible.

  • Now once you have your model basically we want to get you

  • executing that model as fast and possible in the hardware that

  • you care about.

  • Then with optimization we look at certain things we can do to

  • make it faster and smaller. Finally, with our documentation

  • efforts,

  • we want to give you all the resources you need. Let's start

  • with usability. There is different ways of getting up and

  • running in TensorFlow Lite but at

  • the end of the day you will most likely

  • train the model in TensorFlow and convert it to TensorFlow

  • Lite. The flow is very simple.

  • You train in TensorFlow, you get a

  • model and use the TensorFlow Lite convertert and get a model

  • you can

  • execute on different kinds of devices.

  • However, there is points of failure. TensorFlow Lite made

  • the decision to

  • take a subset of TensorFlow and optimize those. That means

  • basically we don't have all the same hundreds of operations that

  • TensorFlow has.

  • The other reason is some semantics.

  • We don't support jet on TensorFlow like

  • control flow which is typically used for reoccurring neural

  • networks.

  • We have made already like strides

  • trying to basically make this easier and better for you.

  • Last year we launched TensorFlow select

  • and this basically means you can execute TensorFlow ops in

  • TensorFlow Lite. In the pipeline, we are making

  • improvements to TensorFlow Select by enabling selective

  • registration and I will explain what all this means in the

  • following slides.

  • We are working on contraflow support as well.

  • Select, like I said, means you can execute TensorFlow and

  • TensorFlow Lite. It comes at the cost of some binary size

  • increase because right now it is basically all or nothing, but

  • that is why we are building selective registration.

  • Selective registration is something you can take advantage

  • of already on

  • TensorFlow Lite for our building ops.

  • You include in the binary only the ops you are using. We are

  • bringing that to TensorFlow Select.

  • We are trying to blur the line of what the TensorFlow and

  • TensorFlow Lite ops

  • are and achieve one of the key points is

  • blur the performance between one and the other.

  • Now, contraflow is something we are working on.

  • This is one of fully lighter pain points

  • we are trying to convert occurring neural networks to

  • TensorFlow Lite. We are adding supports and loops and

  • conditionals.

  • All of this is part of a bigger effort

  • to revamp our converter and we took your feedback very

  • seriously and we wanted this converter to answer these three

  • questions.

  • Some

  • Something fails and what went wrong and what can you do to fix

  • it. OK. Jumping to the next theme is performance.

  • We want your models to execute as fast

  • as possible in the hardware you have available. If you saw in

  • Megan's slides, we have made huge slides in these areas.

  • This is an example of a model.

  • The execution of the CPU, floating

  • point compared to the fast version in CPU. With the

  • recently developing preview we launched, we get huge gains.

  • Like over 7X improvements.

  • By using the HTP we get crazy speed ups. 62X.

  • Again, this is an area we are working really hard.

  • Again, some things that are already available.

  • We have a very nice benchmark and

  • profiling tool and a few delegates. For those not

  • familiar with the concept

  • of TensorFlow delegates, that is our abstraction layer that

  • allows executing

  • models on the CPU.

  • In the pipeline we are working on CPU improvements because we

  • know CPU is important. I am not listing here in the pipeline

  • anything around newer delegates because it tends to be a

  • partnership with other companies and I don't have anything to

  • announce yet, but we are working on that.

  • Our benchmarking tool is basically a profiler. We added

  • some features. Personal threading.

  • You can see statistics for opposite execution and it

  • supports the neural networks API. If you execute the profiler

  • you get a lot of information. This is just a snippet of all

  • the stuff that you get.

  • In this case, we are showing you here information for each op in

  • your graph and how long it took to execute. Then you also get a

  • nice summary of the

  • archives in your graph and many useful statistics so you can

  • decide maybe there

  • is a way that you can, you know, change one for another type that

  • is more efficient or tweak your graph in some way.

  • Now let's talk about the

  • delegates which is an abtraction layer for extracting graphs.

  • The way it works is very nice. In the TensorFlow Lite you

  • prepare a model, right? You are not necessarily saying this

  • model is for this hardware. You just prepare your model.

  • Then inpreter will take it and for those delegates you raised

  • it will go one by

  • one and it will say how much of these graph can you execute.

  • What the delegate cannot execute, it

  • will be executed in CPU.

  • So, by doing this basically you will always be able to execute,

  • right?

  • And if anything, you will get better performance if the

  • delegate can consume more of your graph.

  • We have already an

  • edge-TPU delegate. It has a small power footprint and

  • later on we will have more information

  • about the edge-TPU and a very cool demo.

  • The next delegate is the GPU delegate. These are available

  • right now.

  • We see very nice performance

  • improvements at the cost of relatively small binary size

  • increase.

  • How do you enable the GPU delegate?

  • Just like any other delegate, this is an example of the

  • interpreter and executing one inference.

  • The only thing you need to do is share

  • and pass to interpreter and that is it. So, stuff that we are

  • working on for

  • the

  • GPU deldelegate, we want to add more ops.

  • We want to improve performance even further.

  • We want to finalize APIs so this is generally available.

  • The final delegate I want to mention is the neural network

  • API.

  • This will allow you to execute any

  • other hardware that the API supports.

  • It should be just transparent for you.

  • Finally, CPU is still very important for many use-cases.

  • Most devices have some form of a CPU.

  • We are targeting further improvements

  • on the arm and X86 architectures.

  • The

  • next theme is optimization.

  • Maybe we can do 2X model to make it faster and smaller. We

  • already did some work here.

  • Last year we released this quantization tool that gives

  • nice performance on CPU. This is the first launch that we

  • have

  • while we are calling model optimization toolkit.

  • It will be available for use for everybody.

  • In the pipeline some of these new tech

  • techniques we will put in the toolkit is more quantization

  • work.

  • It gives further improvement on CPU and allows you to execute

  • more hardware.

  • A lot of the hardware like neural

  • processing units are fixed point base

  • with no floating base support.

  • We are working on a technique called production pruning. So

  • quantization means changing the precision of your model and how

  • it is executed on the device.

  • We launch post training quantization

  • tool where you take your save pod model

  • and when you convert to TensorFlow Lite

  • you turn on the flag and it updates.

  • Trying to get the most performance and small Keras

  • loss.

  • We see nice improvements this way.

  • If the model is is fully quantized and you get 4X.

  • For convolutional, we see 10-15% performance improvements but for

  • fully

  • connected RNNs we see up to 3X. This is simple to try. If you

  • are using TensorFlow Lite or plan to use it, give this a try.

  • You have to validate how the accuracy is

  • affected in your particular model but so far we have seen

  • very good numbers. Many of the two billion devices

  • we

  • mentioned before are running this way.

  • In the pipeline, again, we are making

  • further improvements in the quantization approaches that we

  • have.

  • We are working on this for quantization.

  • We want to make it powerful by enabling quanterize straining

  • and easy to use. We are bringing more post-training quantization

  • support. This is an example of Keras model.

  • It is very easy with two dense layers.

  • Let's say you want to train with

  • quantization those two dense layers. Well, soon enough, you

  • will be able to

  • just import your API and say quantatize those two layers.

  • The post-training quantization for fixed point will work very

  • much like the one we already support where you have the

  • conversion path and the only thing

  • you need to parse extra is some

  • calibration data which is the typical model your sees as

  • input. If you have an image processing model, you just need

  • to pass some images.

  • You don't really need to pass a ton of images. In preliminary

  • numbers, model Nets, we

  • see with 25 images we do well. These are preliminary numbers we

  • see with this tool.

  • We have the float baseline and then if

  • you do quantization training you get almost the same accuracy

  • with all the benefits.

  • Then even further if you really, for some reason don't want to

  • invest in this training or you have a model, you know,

  • that you got somewhere, you can still

  • quantize it with post training and you

  • see the accuracy is almost the same.

  • The other technique we are heavily

  • investing in is connection pruning meaning we prune

  • connections in the

  • network and those connections become

  • Keras and that creates a sparse tensor.

  • The benefits of a parse tensor is you can compress them and

  • create potentially faster models if

  • you have less computations to do.

  • Coming very soon, we are going to add a training pruning as a

  • Keras-base API and

  • in the pipeline we are working on first

  • class support of this parse execution in TensorFlow Lite.

  • Again, in our example from Keras you

  • have two dense layers, want to prune connection from those

  • layers, and the

  • only thing you need to do is say prune. It is very, very simple.

  • When you convert the model to

  • TensorFlow Lite format, basically by

  • just even seeding the file you will get nice compressions.

  • These are some, again, very preliminary numbers. A mobile

  • net which is pretty hard to prune compared to other perhaps

  • larger

  • models, we see various models of 50% and even 75%.

  • Finally, documentation, again, we want to give you all the

  • resources you need to get the most out of TensorFlow Lite.

  • We have already done pretty nice improvements.

  • We have a site with more content.

  • We currently have a model

  • representation with

  • five models and representations you can try. We are working on

  • new models and looking to expand our repository.

  • The goal is not only that it is pretty but that it answers your

  • questions. Please give us more feedback if there

  • is things that are still lacking there.

  • Yeah, we revamp the documentation and

  • very important, we also have the roadmap for the foreseeable

  • future. A lot of the stuff I am talking about

  • now you can see there in more detail perhaps.

  • And then, again, we have our repository with all these

  • applications and tutorials and we are expanding it. We want to

  • make the repository into

  • something that, you know, if you don't want to go, or before you

  • start trying to train something, perhaps you have a

  • model you can reuse.

  • By the way, TensorFlow mobile is

  • aggregated especially using TensorFlow select allowing you

  • to use TensorFlow

  • Lite and there is no need for TensorFlow mobile except if you

  • are doing training on the device. That is the reason we

  • are keeping that there. Yeah, training.

  • So, if you see we are building a lot of the basic infrastructure

  • that you will need when you are trying to prepare a

  • training pipeline to execute on device. I don't have anything to

  • announce now but I just wanted to let you know that

  • we are working on it and thinking a lot about it.

  • What about TensorFlow 2.0?

  • Well, it should work.

  • You should be able to convert it to TensorFlow Lite. Before we go

  • two last things. One is there was a lot of content in this

  • talk.

  • Tomorrow we have breakout sessions and will cover

  • TensorFlow Lite in general and optimization in particular.

  • And I want to introduce Pete Warden who

  • is going to talk about some very cool products.

  • [Applause] Awesome.

  • Thanks so much, Raziel.

  • I am really excited to be here to talk

  • about a new project that I think is pretty cool. So, TensorFlow

  • Lite for microcontrollers. What is that all about? This all

  • comes back to when I actually

  • first joined Google back in 2014. There were a whole bunch

  • of internal projects I didn't know about as a member

  • of the public that sort of blew my mind

  • but one in particular came about when I actually spoke to Raziel

  • for the first time.

  • He was on the speech team at the time working with Alex who you

  • just saw. He explained that they use neural

  • networks models of only 13 kilobytes in size. At that time,

  • I only really had experience with image networks and the very

  • smallest of them was still like multiple megabytes so this idea

  • of

  • having a 13 KB model was amazing for me.

  • What amazed me even more was when when he told me why they

  • had to be so small.

  • They needed to run them on dsp and other embedded chips in

  • smartphones so Android

  • could listen for wake words like hey Google while the main CPU

  • was powered off to save the battery.

  • These microcontrollers often only had 10

  • kilobytes of RAM and flash storage.

  • They couldn't rely on Cloud

  • connectivity because the amount of power

  • that would be drained to send data over

  • would have been prohibitive. So, that really struck me. That

  • conversation and the continued

  • work that we did with the speech team because they had so much

  • experience doing all sorts of different approaches with

  • speech. They had spent a lot of time and a lot

  • of energy experimenting and even within the tough constraints of

  • embedded devices, neural networks were better than any of

  • the traditional method they used. I was left wondering if

  • they would be

  • really useful for other embedded sensor applications as well and

  • it left me really wanting to see if we could actually build

  • support for these kind of

  • devices into TensorFlow itself, so that more people could

  • actually get access to it. At the time, only people in the

  • speech community really knew about the

  • groundbreaking work that was being done

  • and I wanted to share it a lot more widely. So, today I am

  • pleased to announce we are releasing the first experimental

  • support for embedded platforms in TensorFlow Lite.

  • To show you what I mean, here is a

  • demonstration board that I actually have in my pocket.

  • This is a prototype of a development

  • board built by SparkFun and it has a

  • cortex M4 processor with 384 kilobytes

  • of RAM and a whole MB of flash storage

  • and it was built by to be extremely low power -- Ambiq.

  • It is able to run on a single core battery like this for many

  • days potentially.

  • I am going to take my life in my hands

  • now by trying a live demo.

  • Let us see if this is actually

  • -- it's going to be extremely hard to

  • see unless we dim the lights. There we go.

  • What I am going to be doing here is by

  • saying a particular word and see if it actually lights up the

  • little yellow light. You can see the blue LED is flashing and

  • that is telling me it is running inference.

  • If I try saying

  • "yessaying "yes", "yes".

  • I knew I was taking my own life into my own hands. There we go.

  • As you can see, it still is far from perfect but it is managing

  • to do a job of recognizing when I say the word and

  • not lighting up when there is unrelated conversations. So why

  • is this useful? Well, first this is running entirely

  • locally on the embedded chip, so we

  • don't need to have any internet connection.

  • It is a good useful first component of a voice interface

  • system.

  • The model isn't quite 13 kilobytes but it is down to 20

  • kilobytes so it only

  • takes up 20 kilobytes of flash storage on this device.

  • The footprint of the TensorFlow Lite

  • for microcontrollers is only another 25 kilobytes and it only

  • needs about 30 kilobytes of RAM to operate. It is within the

  • capabilities of a lot

  • of different embedded devices.

  • Secondly, this is all open source, so you can actually grab

  • the code yourself

  • and build it yourself, and you can modify it.

  • I am showing you here on this particular platform but it

  • actually works on a

  • whole bunch of different embeded chips.

  • We really want to see lots more supports so we are keen to work

  • with the

  • community on collaboratoring to get more devices supported. --

  • collaborating.

  • You can also train our own model. Just something that

  • recognizes yes isn't all that useful, but the key thing is

  • that this comes with a toil you can use to actually train your

  • own models.

  • It always comes with a dataset of

  • 100,000 utterances of about 20,000 common words.

  • The first link there, if you could go to that link, and

  • contribute your voice

  • to the open dataset, it should actually increase the size and

  • the quality of the dataset that we can actually make available.

  • That would be awesome. You can actually use the same approach

  • to do a lot of different audio recognition to recognize

  • different kinds

  • of sounds and even start to use it for similar signal processing

  • problems like,

  • you know, things like predictive maintenance. So, how can you try

  • this out for yourself?

  • If you are in the audience here, at the end of today, you will

  • find that you get

  • a gift box, and you actually have one of these in there.

  • [Applause]

  • All you should need to do is remove the little tab between

  • the battery and it

  • should automatically boot up pre-flash with this yes example.

  • You can try it out for yourself and let me know how it goes. Say

  • yes to TensorFlow Lite. We also include all the cables so you

  • should be able to program it yourself through there serial

  • port.

  • These are the first 700 boards ever built so there is a wiring

  • issue. It will drain the battery.

  • It will last more like hours than days but that will

  • actually, knock on wood, be fixed in the final product that

  • is shipping.

  • And you should be able to develop with these in the exact

  • same way you will with the final shipping product.

  • If you are watching at home, you can pre-order one of these from

  • SparkFun

  • right now for, I think, it is $15.

  • And you will also find lots of other

  • instructions for other platforms in the documentation.

  • We are trying to support all of the -- or as many of the modern

  • microcontrollers that are out there that

  • people are using as possible.

  • And we welcome collaboration with everybody across the

  • community to help unlock all of the creativity that I know is

  • out there. I am really hoping I am going to be spending a lot of

  • time over the next few

  • months reviewing pull requests. And finally, this was my first

  • hardware

  • project so I needed a lot of help from a lot of people

  • including to TensorFlow

  • Lite team especially Raziel, Tim, and

  • Andy, Allisster, Nathan, owen and Jim at SparkFun were life

  • savers.

  • We got these in our hands in the middle of the day yesterday.

  • And also those at Ambiq who actually designed this process

  • and helped us get the software going.

  • A lot of people are arms as well

  • including a big shout out to Neal and Zach. This is a very

  • early experiment but I can't wait to see what people build

  • with this. One final note, I will be around to

  • talk about M CUs with anyone interested on day two. I am

  • really looking forward to chatting with everyone. Thank

  • you. [Applause]

  • Thanks, Pete. We really hope that you try this.

  • It is the early stages but you see a huge help to make this

  • happen. We think it will be really

  • impactful for everybody. Now, before we go again, and I

  • promise

  • this is the last thing you hear from me,

  • I want to welcome June, who is going to

  • talk about how by use

  • TensorFlow Lite with the edge-TPU

  • delegate are able to train these usable machines.

  • [Applause] Thanks, Raziel. Hi, my name is

  • June Tate-Gans and I am one of the lead software engineers

  • inside Google's new Coral group and I

  • have been asked to give a talk about the

  • edge-TPU based teachable demo.

  • Coral is a platformplatformplatform for

  • products with on demand learning.

  • Our first product is a single board

  • computer and a USB stick. What is the edge-TPU?

  • It is a Google designed asic. It doesn't require a network

  • connection to run and allows for a whole new range

  • of applications and machine learning.

  • The first product we built is the Coral Dev Board.

  • This is a single board computer with a removable som. It runs

  • Linux and Android and the som

  • has a

  • gigabyte of ram, Wi-Fi and Bluetooth and the EdgeTPU.

  • The second is the corilatored board connected to whatever

  • you need be it Raspberry Pi or a Linux work station.

  • This teachable machine shows off a form of edge training.

  • Traditionally, there is three ways to do this.

  • There is K-nearest neighbors, weight imprinting and last layer

  • retraining.

  • For this demo, we are using the K-nearest neighbors approach.

  • In this animated GIF, you can see the TPU enables very high

  • classification rates. The frame rate is actually the rate at

  • which the TPU is classifying the images that I am showing it. In

  • this case, you can see we are getting about 30 frames per

  • second.

  • It is essentially real-time classification.

  • With that, I actually have one of our teachable machine demos

  • here.

  • If we can turn this on. There we go. OK.

  • So, on this board we have our EdgeTPU development board

  • assembled with a camera and a series of buttons. Each button

  • corresponds with a class and lights up when the model

  • identifies

  • an object from the camera.

  • First we have to plug this in.

  • Every time I take a picture, by pressing one of these buttons it

  • associates that picture with that particular class and

  • because it is

  • running

  • inference on the EdgeTPU it lights up immediately. The first

  • thing I have to do is train it on the background. I will press

  • this blue button and where you can see it immediately turns on

  • and

  • this is because it is doing inference in realtime.

  • Now, if I train one of the other buzz

  • buttons using something like a tangerine, you can see it can

  • classify

  • between this tangerine and the background. I can grab other

  • objects, such as this TensorFlow Lite sticker. It looks similar.

  • It is the same color. Let's see what was the class I used?

  • Yellow. OK. Sorry. Now, even though it is a similar

  • color,

  • it can still discern the TensorFlow Lite

  • logo from the tangerine.

  • [Applause]

  • you can imagine in a manufacturing content, your

  • operators are no knowledge of machine learning or training,

  • can adapt your system easily and quickly using this exact

  • technique.

  • That is about it for the demo but before I go, I should grab

  • the clicker

  • and also I should say, we are also

  • giving away some EdgeTPU accelerators.

  • For those here today, we will have one available and for those

  • on the livestream

  • you can purchase one at Coral.Google.com. We also have

  • links for you up here. Please take a look at the TensorFlow

  • Lite and our new hardware and thank you very much.

  • [Applause] Thank you, June. That was

  • awesome.

  • So, that is the end of our first block of talks.

  • We are now going to have a break that's about 45 minutes.

  • We want to be back here by noon. Sorry we ran a little long.

  • Thank you for coming up. After the break, we will be talking

  • about building models, Swift, and the community. It is going

  • to be a lot of great talks.

  • Go grab something to eat and come on back here by noon.

  • Thank you very much.

  • Tf.

  • tf.tf.tf.tf.

  • TF an

  • TensorFlow 2.

  • flight 370

  • TensorFlow Lite

  • TensorFlow Lite

  • Swift Keras

  • tf.

  • touchdown

  • megabytes

  • CPU GPU TPU

  • SIGs

  • tf.

  • function function

  • function TUNGZ

  • nction

  • function function tf.function tf.

  • function

  • autograph Alexandre Passos

  • SIGs SIGs Edd

  • Swift TensorFlow 2.

  • FASZ

  • test test test test

  • Hello, everyone.

  • Typhl TIFL TIFL

  • te

  • TFLite TFLite

  • K-nearest

  • Youdao

  • Youdao Youdao Youdao

  • Youdao Youdao

  • test test test test test

  • Hi, everyone.

  • My name is Gal Oshri and I am here to talk about TensorBoard.

  • How many have used TensorBoard before? It helps you visualize a

  • lot of data coming out of TensorFlow.

  • It has a wide variety of features but

  • today we will talk about a couple new additions.

  • We will actually switch over to rademo.

  • This is Colab

  • .

  • I made sure we installed TensorFlow 2.

  • 0 alpha and a couple other sets so we don't have to install it

  • now. We are going to use the dataset

  • and

  • train a really simple keras sequintial model.

  • We will train it with the fit API and give it the TensorFlow

  • callback. If you have used Colab before, you will

  • know we will need to download the logs

  • to our local machine, make sure TensorBoard and set-up and point

  • at it and go back to Colab.

  • We are enabled showing TensorBoard directly within

  • Colab. You will notice the way we start

  • TensorBoard here is exactly the same as in the command line.

  • It is the same command just as the magic cell and magic

  • function in front of it.

  • The same thing will also work in Jupyter notebooks.

  • Let's see what is different. When using the keras callback we

  • have trained validation showing up on the same chart to make it

  • easier to compare them in accuracy, loss and other

  • metrics. This makes it easier to detect things like overfitting.

  • In the graph's dashboard, while seeing

  • the ops and exillerary information is

  • useful for many scenario, sometimes you want to see the

  • model you created in keras. So you can switch to the keras tag

  • and just few that model.

  • We can expand this to see the actual layers we added.

  • There is several other APIs for using TensorBoard within

  • notebooks that let you change the height of the cell as well

  • as list the active instances within your Colab notebook but

  • we are not going

  • to look at that because I want to switch

  • gears and talk about hyperparameter training. We

  • picked the dropout rate, number of units in the dense layer and

  • the optimizer.

  • If we really care about the modeled's performance we will

  • want to try out different values and experiment. The way this

  • looks in TensorBoard is you might include all that

  • information about what the values are in the run names.

  • As you can see here

  • in the bottlebottle -- bottom left. It is not the best

  • experience so let's

  • see if we can do something better.

  • What I'm going to show next is something that is going to

  • change in terms of the

  • API and the UI but it is available in

  • the TF 2.alpha today. We will do several different imports and

  • define which values of the hyperparameters we want to try.

  • We will start out with just a few so we don't take up too much

  • time during the demo. We are going to log a summary that

  • tells TensorBoard what were the hyperparameters we care about

  • and what

  • were the metrics. We then wrap the existing training code that

  • we had, just to make sure we logged the accuracy at the end

  • on the validation set, and also tell

  • TensorBoard that the experiment has started and finished.

  • This time we are going to start TensorBoard before doing our

  • training because in most cases your training will take longer

  • than one minute and you want to view the TensorBoard while your

  • model

  • is training to understand its progress. We started it.

  • It has no data but once a couple epox

  • of training have finished we can refresh it and start to see

  • something.

  • You will notice in the top we have the h-parameters dashboard

  • which show a

  • table where each run is represented by a row and we have

  • columns.

  • As the runs finish, the table becomes populated with them. On

  • the left, we have ability to filter and sort. We can say we

  • don't actually care about

  • the number of units or only want to see experiments where the

  • accuracy is at least 85. Before we proceed further, I want to

  • actually cheat a little bit and log and access some completed

  • experiments where we have run a wider range of combinations of

  • values for the hyperparameters. Now, while this is loading, I

  • want to

  • point out that I am pointing TensorBoard

  • directly at a folder in my Google Drive. I had all my

  • TensorFlow logs on another

  • machine, uploaded them to my Google Drive and I can access

  • them directly within my Colab notebook. This takes a moment to

  • load but, hopefully, when I refresh we can see it.

  • I can switch over to the H-parameters dashboard and see

  • the experience. I can switch over to this view that

  • shows the visualization of access for each hyperparameter

  • and each metric. Each run is represented by a

  • line that

  • passes through all these axis at the points corresponding to

  • hyperparameters and values. I can click and grab to select a

  • range. I have selected the experiments with the relatively

  • high accuracy and they become highlighted in this

  • visualization.

  • I can immediately see that all these experiments used the

  • add-in optimizer as

  • asupposed to SGD and had a relatively high number of units

  • in the dense layer.

  • This gives great ideas about what we can experiment with

  • next. I can skew the scatter plot view which shows the

  • correlation between the

  • different hyperparameters and metrics.

  • I can, again, select a region here to

  • view those points across the other charts.

  • Just to summarize, we have looked at TensorBoard in Colab.

  • An easier way to compare the trend validation runs,

  • visualizing the keras conceptual graph, and better hype parameter

  • tuning with the H-program dashboard.

  • All this information is available as documentation at

  • tensorflow.org/TensorBoard.

  • We also have a demo station upstairs and would love to hear

  • from you.

  • Next up is Alex who will talk about tf.function and autograph.

  • Thank you very much.

  • Hello. I am Alex. I am here to tell you about how you are

  • going to build graphs in TensorFlow 2.0. This might make

  • you a little uncomfortable because we already spent time

  • early telling you in TensorFlow 2.0 we use eager execution by

  • default. So why am I taking that away from you?

  • And I am not. Graphs are useful for quite a few things.

  • The two ones I care the most about

  • personally are that some hardware like

  • our TPUs really benefit from the full

  • program optimization if we have graphs and if you have graphs

  • you can take your model and deploy it on mobile and server

  • devices and make it available to as many people as you can think

  • of.

  • At this point you are probably -- I remember TensorFlow 1.0 and

  • the kind of code I had to write to use graphs.

  • Where -- is he going to tell me I have to keep doing that? No,

  • with TensorFlow 2.0 we are changing the programming model

  • which builds the graphs in TensorFlow.

  • We are removing that model where you

  • add a bunch of nodes and rely on session.run to prune things out

  • and figure out

  • the precise things you want to run and

  • replacing it with a simpler model based on the idea of the

  • function. We are calling it tf. function because that is the

  • main API entry point. With tf. function, many things you are

  • used to

  • are going to go away and I dearly hope you won't miss them.

  • The first thing going away is you will never have to use

  • session.run anymore. [Applause]

  • If you have used TensorFlow execution, you know how it work.

  • You have

  • tensors and operations. Tf.function is just like an

  • operation

  • except one that you get to define using a composition of

  • the other operations in TensorFlow however you wish.

  • Once you have your tf.function, you can call it inside another

  • function, you can take its gradient, you can run it on the

  • GPU, on the TPU, on

  • the CPU, on distributed things just like

  • you would do with any other TensorFlow operation.

  • We are letting you define oprations in Python and making

  • it as easy as possible and trying to preserve as many of

  • the semantics of the Python programming language you know

  • and love and when you execute these functions in a graph.

  • So, obviously the first thing you would think is that is it

  • actually faster?

  • If you look at models that are large

  • conconvolutions it is not any faster

  • because they execute plenty fast. Res models get small and

  • you can measure the difference in performance.

  • I show for this tiny LSTM cell there is a 10-fold speed up if

  • you use tf.function verse if you don't use tf.function to execute

  • it.

  • As I was saying, we really try to preserve as much as the

  • Python semantics as we can to make this code easy at a use.

  • If you have seen TensorFlow graphs you

  • know they are not polymorphic polymorphic.

  • Python code tends to be free in the things it accepts. With tf.

  • function we do the same.

  • Under the hood, when you call a tf.

  • function, we look at the

  • tensors you are passing and see if you made a graph that is

  • compatible, if not we make a new one. We hide them so you can use

  • your tf.function as you would use a normal TensorFlow

  • operation and eventually you will get all the graphs you need

  • built

  • up and your code will run BLAGZ blazingly fast. If you want

  • accease the graphs you are generating you can get them. We

  • expose them to you so you have need

  • to manipulate the grap or do weird things I don't approve you

  • can still do it. The main reason we changed this model, really,

  • is not to replace session.run with tf.function but it is by

  • changing the promise for what we do to your code, we can do so

  • much more for you than we could do before.

  • With the model where you add nodes to a graph and prune them

  • it is hard for it TensorFlow Runtime to know what order you

  • want the operations to be executed in.

  • Almost every TensorFlow operation is

  • stateless so it doesn't matter but for where it does you

  • probably had to use control dependencies other things to

  • make it work. I am here to tell you you will never

  • have to use control dependencies again if you are using

  • tf.function. The promise behind tf.function is you write code

  • you would like to run eagerly, we take it and make it run fast.

  • As we trace your Python code to generate a graph, we look at the

  • operations you

  • run and every time we see a stateful

  • operation, we add the minimal necessary set of control

  • dependencies to ensure the resources accessed are accessed

  • in the order you want them to be. If you have two variables

  • you are updating we will do that in parallel. If you have one

  • variable updating many times we will order the updates so you

  • are not surprised by them happening out of order or

  • something like that.

  • There is really no crazy surprises and weird identifying

  • behavior.

  • You should never need to explicitly add control

  • dependencies to your code but you will still get the ability

  • of knowing what order things execute.

  • If you want to check something before something else, just put

  • that line of

  • code before that line of code like how

  • you do normal programming.

  • We can simplify how you use variables also. You know

  • variables are useful and let you persist and checkpoint and all

  • those

  • things but they can be a little finically.Thifinicy. Another

  • thing we are removing from TensorFlow is the need to manual

  • initialize variables yourself.

  • The story for variables is a little complicated because as we

  • try to make

  • code compatible with eager execution and graph semantics

  • you will find examples where it is unclear what we should do. My

  • favorite one is this one.

  • If you run this in TensorFlow 1. X and session run repeatedly you

  • will get a series of numbers that goes up.

  • But if you run the code eagerly, every

  • time you run it you will get the same number back. If I want to

  • wrap this code of tf.function, which one should it do?

  • Follow the 1.X behavior or the eager behavior?

  • I think if we took a poll I don't agree with myself. This is

  • an area.

  • Non ambiguously created variables is perfectly OK. You

  • can create a variable and capture it by closure in a

  • function.

  • That is a way of a lot of TensorFlow is written. You can

  • write the function so it only creates variables the first time

  • it is called. This is what most libraries in TensorFlow do under

  • the hood.

  • This is how keras is implemented, sonnet, and all

  • sorts of other libraries that use TensorFlow variables.

  • They try to take care to not create variables every time they

  • are called otherwise you are creating too many variables and

  • not training anything. Code that behaves well gets thrown into

  • function and it is fine. If you have seen this, I didn't need to

  • call the initializer for this variable I am creating. It is

  • even better. I can make the initializer depend on

  • the value of there argument of the

  • FUNGSZFUNGSZ -- function in arbitarily

  • ways and we add the dependencies to make sure the state updates

  • happen in the way we want, so there is no need to worry about

  • this.

  • You can create variables like how you would use in a normal

  • programming

  • language and things behave the way you want.

  • I am really happy about the autograph integration as well.

  • If anyone has used TensorFlow with

  • autograph it can be awkward but we are

  • finally breaking up with the prior forms

  • and you can write code that looks like this.

  • I have a wild loop here and this is

  • probably the worst way to make a tensor. If you put this in a tf.

  • function it will put it in a graph and execute it. This is

  • nice and great. How does it work?

  • Under the hood things like tf.

  • file are still there but we have tf.

  • autograph that rewrites control expressions into something that

  • looks like this which is not how you would want to write code.

  • This then can be taken by TensorFlow and turned into fast

  • dynamic graph code. How does this work? To explain

  • that I would like to take a step back and think about how does

  • anything in TensorFlow work.

  • You can have a tensor and do tensor

  • plus tensor times other tensors. Just use a tensor like you would

  • a floating point number. Python has a thing called operator

  • overloading that let's us change the behavior of standard Python

  • operators

  • when applied on our custom data types

  • like tensors. We can add these underscores and

  • change

  • how TensorFlow does addition and subtraction.

  • Python doesn't let us write underscore underscore. It makes

  • me very sad.

  • If you this can -- if you think about this for a few second, you

  • can come up with rewrite rules that will let us

  • lower to byte code with underscore-underscore if that is

  • under writeable.

  • If code looks like this, you can write this as condition.if a and

  • B.

  • You need to do fiddling of the scopes because I am sure you

  • know Python's lexical scoping is not as lexical as you think.

  • Names can leak out of scopes and it is

  • a little messy but that is also a mechanical transformation.

  • If this is potentially a mechanical transformation let's

  • do this.

  • We wrote this compiler called autograph that takes your Python

  • code and rewrites

  • it in a form that let's us call

  • underscore underscore if and etc on tensors. This is all does but

  • this unlocks a lot

  • of the power of native Python control into your tensor graphs.

  • On this function, I have two loops.

  • One it is a static Python loop because where write four I in

  • range. I is an integer. Autograph sees this and leaves

  • it untouched.

  • You still get to use Python control flow to chose how many

  • layers your

  • network will have or iterate over a sequential but when your

  • control flow does depend on the properties of tensors

  • like in the second loop for intf.range then autograph sees

  • it and turns

  • it into a dynamic tf.wildloop. This means you can implement

  • this in 10 lines of code just like how you would

  • use in a normal language which is pretty nice. And anything

  • that you can do in a TensorFlow graph you can make happen

  • dynamically.

  • You can make prints and assertions happen. Notice here

  • that I don't need to add control dependencies to make sure they

  • happen in the right order because of the thing we were

  • talking about earlier. We already do. We control the

  • dependencies automatically for you to try to really

  • make your code look and

  • behave the same as Python code.

  • The TensorFlow Runtime is not powerful enough to support

  • everything that Python can do. So, for example, if you are

  • manipulating list of tenors at run-time you should still use a

  • tensor array.

  • It works very well and compiles to eefficient TensorFlow code.

  • You no longer need to write the boilerplate code associated with

  • it. This is how you stack a bunch together in a loop.

  • Wrapping up, I think we have

  • changed a lot in TF 2.

  • I hope you agree the changes are worth it. I will quickly walk

  • through you a dif of what your code will look look before and

  • after. Session run goes away.

  • Control dependencies go away.

  • Variable initialization goes away.

  • Cond and

  • wildloop go away and you just use functions like a normal

  • language.

  • Thank you and welcome to TF 2.

  • All the examples are on tensorflow.

  • org/alpha and dig you will find a Colab

  • notebook with these and more and play

  • around with the tf.function for auto.

  • Now I would like to introduce Ryan who

  • is going to tell you about all the ways we have of feeding data

  • into TensorFlow.

  • I am here to tell you about TensorFlow datasets.

  • It is on GitHub and pipi. First a little set-up.

  • This is the partnership care about. TensorFlow 2.0 makes it

  • look nice.

  • You get predictions, your loss,

  • gradient, update, it all looks fantastic. And they are good

  • together. Data and models. Really, I just want to focus in

  • on this one piece. This one line.

  • Iterating through your training dataset.

  • In TF 2, with the training data API,

  • you have this expressive simple way of

  • expressing high performance but flexible in put pipelines.

  • We might start with a TF dataset and we

  • can shuffle and repeat and build complex

  • pipelines and we know this will be performance.

  • This map call might be doing image preprocessing on each

  • record in the dataset. It is happening in parallel you can

  • see here. TensorFlow will be spinning up a bunch of threads

  • so the paping is high performance. At the end, you

  • have the ability to pre-fetch. As your tight model training

  • loop is

  • going, you can be preparing batches ready to feed into the

  • iteration.

  • This is fantastic bit but unfortunately

  • the source data is a little out of step.

  • We want this tight iteration with four iputs and targets.

  • What is hidden is how did you get the data into a format that

  • you can feed into the model in the first place?

  • Every source dataset when is all a little bit different. This

  • makes sense when you distribute data at rest there is some

  • natural format you would like to distribute it in. We are not

  • interest in data at rest. We are interested in having data in a

  • format that is ready to move into a TF data pipeline.

  • What we really want is something that is good together.

  • We want these two pups, data and model,

  • to really be simpatico. It is about getting data into a format

  • that is ready to move into a tf. data pipeline. Here is

  • TensorFlow datasets.

  • You import TensorFlow datasets and this

  • load API for

  • mnist, and we are grabbing the training split of the dataset.

  • We are asking for supervised tuples.

  • We will get input and target tuples. We get a tf.

  • data dataset and we can build a complex pipeline. We are adding

  • shuffling and batching in this case.

  • We are off and we can iterate through our inputs and do all

  • the modeling things we need. Load hides a lot of the

  • complexity. What is happening here with load is that mnist is

  • being fetched from the source, it is being pre-processed, put

  • into a standard format, and it is being documented with

  • statistics and everything which I will show you in a second, and

  • then you are building the sort of front end where you have

  • preprocessed data on disks ready to move.

  • Typically you would only do this one and tf.ds will tash the

  • results for you and you are getting a tf.data pipeline out

  • of it.

  • We don't just for mnist. We have imagenet.

  • The first step is running this batch script, muck with the

  • files over here, make sure you filter out these records and

  • things like that and of course this

  • is all taken care of for you.

  • ImageNet is gated with a password and

  • username and so currently TensorFlow datasets ask you to

  • download the data and put it in a certain place and TensorFlow

  • datasets takes it from there.

  • We are working with cagal to get

  • ImageNet to be downloaded without any user interaction.

  • We are imdb reviews in this case.

  • Note the API doesn't change.

  • This is a tfts data call.

  • Of course, text is a notoriously difficult format to work with

  • when you are interested in doing machine learning on it you have

  • to do encoding. We are shipping with three really

  • powerful

  • encodings one is byte level, character loading encoding and

  • another is subwords.

  • We have a great subword text encoder.

  • We will get imdb reviews already

  • encoded using a vocabulary with about 8,000 tokens.

  • Each of these datasets is packaged together as a dataset

  • builder. If you want to add a dataset to

  • TensorFlow datasets, which I hope every single one of you do,

  • there should be

  • about a thousand new datasets in a week, dataset builder is

  • simple.

  • It does those exact three things lat load hiding. It has a method

  • called download and prepare taking source data on the local

  • web or local directory and produces preprocessed files. It

  • takes data at rest and put it into a format of data ready to

  • move. The second is as dataset and

  • that takes the preprocessed files on disk and

  • produces a tf.data.dataset.

  • It third is metadata about the dataset.

  • Dataset.info has the features in the dataset and each feature

  • documents the name, shape and type.

  • So in this case, the image for mnist is

  • 28, 28 by 1, type int 8. The number of classes is documented

  • in the class label feature and you get

  • statistics on the dataset and what are the splits exposed,

  • train and set, and how many records in each one.

  • Mnist has 60,000 in the training and 10,000 for it testing. If

  • you want to grab a dataset from TensorFlow datasets and filter

  • for all

  • the ones that are trivially supervised datasets, you can

  • look for supervised keys and pick up the features you need.

  • For text, again, one of these things

  • that is a bit annoying to work with

  • often, so this is imdB reviews.

  • The text feature contains the encoder.

  • We see it is a texting encoder with a vocabulary size of 8,

  • 185.

  • We support numpy usage as well. We want to make TensorFlow

  • datasets really portable.

  • You can call this tfts as

  • umpy arrays. Coming back to our initial example, you know, we

  • have this beautiful, you know,

  • model and data, but of course trained dataset -- where did

  • this come from and how did you get the data? With TensorFlow

  • datasets a simple change and you have something that works

  • end-to-end, out of the box, which is something we are really

  • happy about and hope you are too.

  • You can come find us on pypi and GitHub.

  • The community on GitHub has gotten surprisingly active

  • surprisingly quickly. Come join us. We could use your data if

  • you are

  • distributing a dataset and would like to

  • make your dataset famous, please add it. Come help us on GitHub.

  • There are dataset requests and you can

  • help implement a dataset that has been requested or help us

  • develop. We develop out in the open and develop straight out on

  • GitHub. Thanks very much.

  • Hopefully you guys find this useful and

  • come help us out. Thank you.

  • Next is something that I think is really great.

  • Chris Lattner

  • and Brendon telling you about Swift for TensorFlowTensorFlow.

  • Thanks, Brian. Fantastic. Hi, I am Chris and this is Brennan and

  • we are super excited to tell you about a new approach to machine

  • learning. Here in the TensorFlow team, it is our jobs to push the

  • state-of-the-art of machine learning forward. We have

  • learned a lot over the last few years with deep learning. We

  • have incorporated most of that all into TensorFlow 2.

  • 0 and we are really excited about it.

  • Here we are looking further beyond than TensorFlow 2.0.

  • Eager mode makes it easy to train dynamic model, but

  • deploying it still requires you take that and write a bunch

  • of C++ code to help drive it. That could be better. Some

  • researchers are interested in taking machine learning models

  • and integrating them into larger applications.

  • That often requires writing C++ code also.

  • We want more expressive and differential

  • mechanisms and we are excited to define reusable types that can

  • be put into new

  • places and used with automatic differentiate. We want to make

  • your more productive by taking errors in your code and

  • improving your iteration time. Now, what we are really trying

  • to do here is lift TensorFlow to entirely new heights. We need to

  • innovate at all levels of the stack and this includes the

  • compiler and the language. That is what's Swift for TensorFlow

  • is about all.

  • We think that applying new solutions to old problems with

  • help push machine learning even further than before. Let's jump

  • into some code. First, what is Swift?

  • Swift is a modern and class platform language designed to be

  • easy to learn and use. Swift uses types which are great

  • because they can help you catch errors earlier and encourage

  • good API design.

  • Swift uses type inference so it is easy to use and elegant but

  • it is also open

  • source and has an open evolution

  • language processing allowing us to change the language and make

  • it better.

  • This is how you define a model in Swift for TensorFlow. We are

  • laying out our layers here and we can define a forward function

  • which

  • com poses them together in a linear sequence.

  • You probably noticed this looks

  • like keras and that is no error.

  • All we have to do to train is

  • substantiate the model, pick an optimizer and random data and a

  • training loop. We will write it by hand because that gives you

  • the maximum flexibility to play with constructs and you can do

  • whatever you want.

  • Some of the major advantages of Swift for TensorFlow is the

  • workflow.

  • Instead of telling you about it, what do you think Brian?

  • -- Brennan.

  • What could be easier than opening up a browser tab?

  • This is Google Colab hosted Jupyter notebooks and comes with

  • Swift for TensorFlow built right in.

  • Let's see it in action.

  • Here is the layer model or the model that Chris just showed you

  • a couple slides ago.

  • We are going to run it using some

  • random training data right here in the browser. We will

  • instantiate the data and trained it using Swift for TensorFlow in

  • our browser on training data right here. We can see the

  • training losses decreasing over time so that is great,

  • but if you are ever like me when I try to use machine learning in

  • any application, I start with the simple model and I have to

  • iterate. I have to tweak the model to make it fit better to

  • the task at hand. So, since we are trying to show you the

  • workflow, let's edit this model, let's make it more accurate.

  • Here we are. Now, let's think a little for a moment.

  • What changes do we want to make to our model?

  • This is deep learning

  • after all so the answer is always to go deeper.

  • Not just sequential layer but skip connections are a good idea

  • to make sure your model continues to train effectively.

  • Let's go through and actually add an

  • extra layer to our model, let's add skip

  • connections, and we will do it all right now in under 90

  • seconds. Are you ready? Here we go. We need to define

  • our additional layer first.

  • We will fill in this dense layer.

  • One thing you can see is that we are using tab auto complete to

  • help fill in code as we are trying to develop and modify our

  • model.

  • Now, we are going to fix up the shapes right here really quick

  • so that the residual connections will all work. If I can type

  • properly that would go better. Great.

  • We have now defined our model with the additional layers. All

  • we need to do is modify the forward

  • pass so that we add those skip connections. Here we go. The

  • first thing we need to do is store

  • in a temporary variable the output of the flattened layer.

  • Then we will feed that output to our first dense layer.

  • Dense.

  • applied to tmp in context.

  • For the cuda graphs here is our dense applied.

  • Tmp plus tmp2 in context. Run that.

  • Yes, it worksism -- works. Let's see how it does.

  • We will reinstantiate our model and run the training loop.

  • This loss is now substantially lower.

  • This is an example of what it is like to use Swift for TensorFlow

  • as you apply

  • and develop models for applications and challenges.

  • [Applause] Thank you.

  • But Swift for TensorFlow was designed for researchers and

  • researchers often

  • need to do more than just change models

  • and change the way architecture fits together.

  • They need to define entirely new abstractions and layers. Let's

  • see that live right now.

  • Let's define a new custom layer.

  • Let's say we had the brilliant idea we wanted to modify the

  • standard dense layer that takes weights and biases and

  • add an additional bias set of parameters.

  • We will define this double bias dense layer right here. I am

  • going to type this quickly. Stand by. 15 seconds. Here we

  • go! [Laughter]

  • That was great. Let's walk through the code so you can see

  • what is going on. The first thing that we have is we define

  • our parameters.

  • These are w, like our weights for our

  • neurons and B1 and B2.

  • We define an initializer that takes an input and output size

  • just like dense

  • does and we use that to initialize our parameters. The

  • forward pass is very simple to write.

  • Here is just aplied to and we take the matrix multiplication

  • of input by our

  • weights and add in our bias terms. We have just defined a

  • custom layer in Colab in just a few lines of code. Here is model

  • two and we will use our

  • double bias dense layer and we are going

  • to instantiate and use our

  • custom handwritten training loop. Because TensorFlow can

  • statically analyze your code, it can be really helpful to you. I

  • don't know about you but I regularly

  • put typos in my code and Swift for TensorFlow is helping you

  • out.

  • It is saying look, you mistyped softcross-in.

  • This is not the right idea but this is an example of how easy

  • it is for researchers to experiment with new ideas

  • really easily in Swift for TensorFlow but let's go deeper.

  • Swift for TensorFlow is, again, designed for researchers and

  • researchers need to be able to customize everything. That is

  • the whole point of research.

  • So, let's show an example of how to customize something other

  • than just a model or layer.

  • You may have heard that large GPU

  • clusters or TPU super pods are, like, delivering massive

  • breakthroughs in

  • research and advancing the state-of-the-art in certain

  • applications and domains.

  • You may have always heard as you scale up you need to increase

  • your batch size. Let's say you are a researcher

  • and you want to try and figure out what are the

  • best ways to train deep neural networks

  • at larger batch sizesment if you are a researcher, you probably

  • can't guy a whole GPU cluster or went a whole TPU

  • super pod all the time for your experiments but you often have a

  • GPU under your desk.

  • Let's see how we can simulate that on a single machine.

  • We will do it all in a few lines of code here. Here is our custom

  • training loop and the standard part. This is our 1-10 training

  • epics. What we will do is instead of just

  • applying our model forward once, we have an additional interloop,

  • right?

  • We will run our forward pass, we will run our model four times,

  • and we are

  • going to take the gradients for each

  • step and aggregate them in this grads variable.

  • This simulates running on four independent accelerates in a

  • data parallel fashion on a batch that is four

  • times as large as what we actually run.

  • We will then use our

  • optimizer. That is all there is to it. We

  • are really excited by this flexibility and capabilities

  • that Swift for TensorFlow brings to researchers. Back over to

  • you, Chris. Thanks, Brennan.

  • [Applause] So I think the focus on catching

  • errors early and also productivity enhancements like

  • code completion can help you in a lot of ways. It's not just

  • about automating typing

  • of code, but it can be about discovery of APIs. So another

  • thing that is really cool

  • about Swift as a language is it has good interoperability with C

  • code.

  • You can import header files and call directly from C without

  • wrappers or boilerplate or anything involved. It just

  • works. We have taken this approach in the TensorFlow and

  • brought it to the world of Python. One of the cool things

  • about this is that allows you to combine the power of Swift for

  • TensorFlow with all the advantages of the Python Eco

  • system. How about we take a look?

  • Thanks, Chris.

  • The Python data science ecosystem is incredibly powerful

  • and vibrant.

  • We wanted to make sure you didn't miss

  • your favorite libraries and utilitiesio

  • you are -- utilities you were used to.

  • Let's see how this works in

  • context to

  • numpy.

  • We import py plot from the library and

  • numpy and assign it to np.

  • After that we can use np just as if we were in Python.

  • We will call sine and cosine and pass them to py plot.

  • When we run the cell, it just works

  • exactly as you would expect. [Applause]

  • Thank you. Now, this sort of kind of looks like

  • the Python code you are used to writing but this is actually

  • pure Swift. It just works seamlessly. This is maybe a bit

  • of a toy example.

  • Let's see this a little bit more more in context.

  • OpenAI has done a lot of work in the area of reinforcement

  • learning and

  • developed a Python library called OpenAI gym which contains

  • a collection of environments that are very useful when you

  • are trying to train a reinforcement agent across a

  • variety of different challenges.

  • Let's use OpenAI gym to train a reinforcement agent in

  • TensorFlow right in our browsers. The first thing we

  • need to do is import

  • gym and define a few hyperparameters and now define

  • our neural networks. We will pick a simple two layer

  • dense network in this case and it is just a sequential model.

  • After that we have helper code to filter

  • out bad or short episodes and whatnot. Here is the real meat

  • of it.

  • We will use gym to instantiate the cart pole environment and we

  • will instantiate our network in the optimizer and here is our

  • training loop. We are going to get a bunch of

  • episodes, we are going to run our model, get the

  • gradientgradients and apply those to the optimizer and

  • record the mean rewards as we train. Very simple,

  • straightforward Swift. And here you go see us training

  • a Swift

  • for TensorFlow in an OpenAI gym environment using the Python

  • bridge. Totally seamless. Afterwards, you can keep track

  • of the parameters of the rewards. In this case we will

  • plot the mean

  • rewards as the model trained using Python Numpy. Totally

  • seamless. You can get started using Swift for TensorFlow using

  • all the libraries you know and love and take advantage of what

  • Swift for TensorFlow brings to the table.

  • Back over to you, Chris. Thanks, Brennan.

  • One thing I love about this is it is not just about leveraging

  • big important libraries.

  • We are working on the ability to integrate Swift for TensorFlow

  • and Swift for Python together which we think will provide a

  • nice path to help you

  • incrementally move code from one worth to the other.

  • I think it is fair to say calculus is

  • an intergral part of machine learning. We think

  • differentiable programming is so important we built it right into

  • the

  • language enabling more flexible and custom work with

  • derivatives. We think this is really cool. I would like to

  • take a look.

  • We have been using Swift for TensorFlow's differential

  • programming

  • throughout the demo but let's really break dedown.

  • Here we define my function that takes two doubles and returns a

  • double based

  • on some products and sums and quotients. If we want Swift for

  • TensorFlow to automatically compute the derivative for

  • us, I would annotate it at differential.

  • Swift for TensorFlow will then derive the derivative for this

  • function right when we run the cell.

  • To use this auto generated derivative, use gradient which

  • takes a closure to evaluate and a point you want to evaluate

  • your closure at. Here we go. This is what it is to take the

  • derivative of a function at a particular point. We can change

  • this around. This one is my favorite. Tasty number. That

  • works nicely.

  • One thing to note is we have been taking the partial

  • derivatives of my function with respect to A. Of course you can

  • take the partial

  • derivative and get a full gradient of my function like so.

  • Often with neural networks, however, you

  • want to get not just the gragradients, you often want

  • what the network predicted? This is useful to compute

  • accuracy or other debugging sort of information.

  • For that, you can use value with gradient.

  • That returns a tuple containing the value and gradient

  • shockingly enough.

  • One thing to note, in Swift tuples can have named

  • parameters. You can see it prints out nicely and you can

  • access values. We think this is another nice little thing that

  • helps make writing and debugging code and more importantly

  • reading it and understanding it later a little bit easier. The

  • one thing I want to call out is throughout this we have been

  • using normal types.

  • These are not tensor of something. It is just plain old

  • double.

  • This is because automatic differentiateation is built

  • right in and

  • it makes it easy to express your thoughts very clearly. But even

  • though it is built into the

  • language, we have worked very hard to

  • make sure automatic differiation is customizeable. Let's show

  • you. Let's say you want to define an

  • algebra in 2d space.

  • You are certainly going to need a point data type.

  • Here we define a point instruct with X

  • and Y and mark it differentiable. We can define

  • helper functions on it like dot or other helper functions. Swift

  • for TensorFlow when you try and use your code it will often

  • automatically infer when you need

  • gradients to be automatically computed for you by the

  • compiler. It is a good idea to document your intentions so you

  • can annotate your helping functions as at differentiable.

  • The other reason we recommend doing this is because it helps

  • catch errors. Swift for TensorFlow is telling

  • you that hey, you can only differentiate functions that

  • return values that

  • conform to differentiable but int doesn't conform to

  • differentiable.

  • This is telling you my helper function

  • returns an int and int is about taking small steps and integers

  • are very discreet.

  • Swift for TensorFlow is helping to catch errors right when you

  • write the code and tell you what is going on.

  • The solution, of course, is to just not

  • mark it as a differentiable and the cell runs just fine. Let's

  • say we also wanted to go beyond just defining the dot product.

  • Let's say we wanted to define the magnitude helper function.

  • That is the magnitude of the vector

  • defined by the origin to the point in question.

  • To do that, we can use the distance formula and we can

  • define the extension

  • on point that does this.

  • We will pretend for a moment that Swift doesn't include a

  • square root function because I want a good excuse for you to

  • see the interopera

  • interoperability with C.

  • We will use C's square root function. We can define the

  • magnitude and it -- no, it doesn't quite work.

  • Let's see what is going on. We wanted magnitude to be

  • differentiable and it is saying you can't differentiate the

  • square root

  • function because this is an external function that hasn't

  • been marked as differentiable.

  • The square root is a C function compiled by the C compiler and

  • that

  • can't automatically compute derivatives for you. This is

  • saying it will not work and

  • this is excellent because it gives me a great excuse to show

  • you how to write custom gradientss.

  • We define a wrapper function here that calls to the C square

  • root function. The backwards pass, we take our double,

  • and we return to a tuple of two values. Rather the first element

  • in the tuple is the normal value and the forward pass.

  • The second a pullback closure and this is where you define the

  • backwards pass capturing whatever values you need from

  • the forward pass. We are going to run that.

  • We are going to go back up to the

  • definition of magnitude and change it

  • from square root to my square root, rerun the cell and it

  • works.

  • We defined point, dot and magnitude. Here I defined the

  • silly function and marked it as differentiable. We are going to

  • take two points, we are also going to take a double, right?

  • You can mix and match differentiable

  • data types totally fluidly.

  • We will do magnitude and dot products. It is a silly

  • function.

  • We can use it, compute the gradient of

  • this function at arbitrary data points. You can get the value of

  • there function

  • and full gradients in partial derivatives with respect to

  • individual values.

  • That has been a quick run through of

  • custom gradients J custom data types

  • with the differentiate built in. Let's put all this together and

  • show how you can write your own debuggers as

  • an example of how this power is all in your hands.

  • Often when you are developing models, you want to -- or

  • debugging models, you want to be to see the gradients at

  • different points within your model.

  • We can define in regular Swift code a gradient debugger.

  • It will take as input, a double, and it is going to return it

  • just like normal for the forward pass, right?

  • It is an identity function. We will get the gradient, print it

  • and return it. We are just passing it through printing it

  • out. We have defined this gradient debugger and we can use

  • it in our silly function

  • to see what is going on as we take derivatives.

  • Gradientdebugger, here we go. We can rerun that.

  • And when we take the gradients, we can see that for that point

  • in the silly function, of a.

  • b, the gradient is 3.80.

  • That's been a brief through of how

  • automatic differiation works and how you

  • can extract the power to build whatever systems you need. Back

  • over to you, Chris.

  • The algorithms we are defining were built in the 1970s.

  • There is a tremendous amount of depth

  • and I am excited to see what you all can do it about.

  • If you are interested in learning more, we are detail

  • documents online.

  • The language of Swift has low level performperformance and

  • there is no gill to get in the way of concurrency.

  • They have advanced compiler techniques to identify and

  • extract graphs for you. The consequence of this together is

  • we

  • think Swift has the world's most advanced eager mode. You may

  • wonder why do we care about this stuff?

  • We are seeing trends in the industry

  • where people are defining neural networks. This requires you to

  • export graphs and write a bunch of C++ code to load and

  • orchestrate them in various ways.

  • Let's look at an example of this.

  • Alpha go-

  • zero is amazing work.

  • You have deep learning and it drives through it Monte Carlo.

  • It is the combination of all three

  • things that make AlphaGo Zero possible.

  • This is possible today.

  • If you are an advanced team like DeepMind you can do this. We

  • think breaking down barriers like this can lead to new

  • breakthroughs in science and drive progress forward. Instead

  • of talking about it, again, let's take a look.

  • MiniGo is an Open Source Go player

  • inspired by DeepMind's AlphaGo Zero.

  • The miniGo project were writing everything in normal TensorFlow

  • and it was working great until tay started trying to run at

  • scale on large clusters of TPUs.

  • They ran into performance problems and

  • had to rewrite things into C++ to effectively utilize the

  • modern accelerators.

  • We have reimplemented Monte Carlo

  • research and the rest of MiniGo player in Swift. We define a

  • helper function, these are

  • our white and black players, we are going to run basically play

  • the game until we have a winner or looser. So, let's actually

  • run this.

  • We define a game configuration and we

  • will play between a Monte Carlo tree search versus a random

  • player to see how

  • easy it is to flip back and forth or mix and match between

  • deep learning and other arbitrary machine learning

  • algorithms right here in Swift. Here you go.

  • You can see them playing white, black, playing different moves

  • back and forth and it just goes.

  • We think that Swift for TensorFlow is going to unlock

  • whole new classes of algorithms and research because of how

  • easy it is to do everything in one language with no barriers

  • and not rewriting everything into the C++. Back to you,

  • Brennan. The cool thing about this is you

  • can do

  • something like this in a work workbook. We have seen families

  • of techniques that can be bound together and fused and bringing

  • this to more people we think will lead to new kinds of

  • research. Our work on usability and design is not just about

  • high-end researchers.

  • We love them but Swift is widely used

  • to teach new programmers to learn how to code.

  • I am excited to announce a new journey

  • we are embarking on with Jeremy Howard.

  • At fast.ai we are always looking to push the boundaries of what

  • is possible especially pushing to make recent

  • advances more accessibility. We have been involved with this and

  • building the world's best document classifier.

  • Hundreds of thousands have become deep

  • lun learners.

  • We

  • think with Swift for TensorFlow we can go even further.

  • We are announcing a new program caught by someone that knows

  • Swift well. Chris, I think he means you.

  • I am super excited to help teach the

  • next generation of learners and excited

  • Jeremy is bringing his expertise in

  • Swift and helping us shift the higher level APIs. The most

  • important part is Swift for TensorFlow is really TensorFlow

  • at its core and we think this is super important.

  • We have worked hard to make sure it integrates with all things

  • going on in the big TensorFlow family and we are very excited

  • about that. You may be wondering where you

  • can get this. Swift for TensorFlow is Open Source. You

  • can find it on GitHub now and you can join our community.

  • It also works great in Colab as you have seen today. We have

  • tutorials and examples and the demos you saw today are

  • available in Colab. We have released our 0.2 release

  • including all the basic infrastructure and underlying

  • technology to power these demos and examples and we are actively

  • working on high-level APIs right now.

  • This is not ready for production yet as you can guess but we are

  • excited about

  • shaping the future, building this out, exploring this new

  • program model, and

  • this is a great example for advanced researchers to get

  • involved and help shape the platform. We would love for you

  • to try it out and let us know what you think.

  • Next up, I would like to invite Edd to

  • talk about the TensorFlow community.

  • Thanks, Chris.

  • Hey, everybody. I work with TensorFlow helping build

  • community and collaboration around the open source project.

  • Usually I put the thank you slide at the end for listening

  • to me but actually I want to thank you for contributing and

  • being part of the TensorFlow project. Whether you are here in

  • this room or on the livestream, there has been some amazing talk

  • on YouTube from people in China, and India, and Japan, and all

  • over the world joining us virtually today. Thank you for

  • your contributions.

  • The project is where it is today because of you.

  • Of course, in core TensorFlow alone we have had this many

  • commits.

  • This figures out over 50,000 from over 18,000 contributors

  • and much more than just code commits.

  • 39,000 stackoverflow questions about TensorFlow, we have 66

  • machine learning

  • Google developer experts, many of whom who are here with us

  • today.

  • Welcome and thank you guys. It is really great to have you with

  • us and thank you for everything you do helping teach people

  • about TensorFlow. We have had 14 guest posts to the

  • TensorFlow blog and that keeps going up.

  • There are so many ways people are contributing. Thank you.

  • You are really helping build out the TensorFlow ecosystem. I want

  • to discuss how we are growing the Eco system and report back

  • on some of the changes we have made over the last year. I am

  • going to cover how we are making

  • it easier to get involved in TensorFlow and how we are trying

  • to consult betwer

  • the users in the community and be more transparent about our

  • development. Going to cover how we are empowering everybody to

  • get involved and to do more

  • and increasing in the number of contact points.

  • I will go a bit more into depth about the conference announced

  • today. Let's talk about how we are

  • making

  • contributions to TensorFlow easier.

  • One of there most important thing is

  • increasing its modularity.

  • We are trying to make it less of a mono lift.

  • When you come and want to

  • contribute to an open source project it helps to know where

  • to go.

  • We are creating more surface area. Our special interest

  • groups play a big

  • part in this and I will talk a bit more about them later. It is

  • not just code.

  • There is so many more places to

  • contribute this year compared to last year. I will talk about the

  • documentation groups, groups involved in testing, groups who

  • are blogging and on YouTube and more.

  • I was super excited to see we published

  • TensorFlow tutorial in Korean. That isn't something we did on

  • our team. That is part of the community.

  • We were similarly able to publish it in

  • Russian thank you to Andrew Stepin.

  • I am also really excited about the TensorFlow 2.0 testing group

  • led my Paige Bailey. This is a bunch of contributors and

  • Google developer experts who are working to get TensorFlow 2.0 a

  • test. You see an example of friction load and

  • folks are going through ML loads and documenting what they find

  • awesome and also things that could be a little better.

  • This group meets weekly and often has

  • guest talks from maintainers and SIGs leaders and it helping

  • bring TensorFlow 2.0 from cutting edge to something

  • thoroughly tested and ready to use.

  • We mentioned we have over 14 posts from guests on the

  • TensorFlow blog.

  • This is from a great post about real-time segmentation with

  • TensorFlow.js that comes from a grad student and researcher at

  • itp. Whether it is testing, whether it is documentation,

  • whether it is blogs or conference talks, thank you.

  • Now, I want to talk a little bit about TensorFlow RFCs.

  • As you probably know, RFC means request for comments. This time

  • last year, we were not that organized about how we evolved

  • TensorFlow's design in terms of

  • communicating and I stood on this stage and told you about

  • launching the RFC process.

  • We have accepted 21 of them over the period of the last year.

  • This is our way to communicate design. We post an RFC about a

  • design and consult widely.

  • This isn't just about code coming outwards, it can be

  • created on and commented on by anyone.

  • We have several RFCs brought by the broader community and I

  • expect to see several more. One of the things I am most proud

  • about

  • is how the RFC process is underpinning the 2.0 transition.

  • All the major changes have been proposed and consulted with in

  • RFCs. This isn't just a great way of consulting and getting

  • feedback, going

  • forward you have a big repository of documentation

  • about why design choices were made a certain way in TensorFlow

  • and great educational results for people who are coming on and

  • want to get involved in contributing to the project. I

  • really want to give a big thanks to

  • anyone who authored or reviewed an RFC.

  • You played a vital role in making TensorFlow better. Let's

  • talk a bit about the social structure of TensorFlow. You

  • know, last year I talked about how coming to a large project

  • can be a little bit daunting. You don't know where people are,

  • whether people have your interest in common. We created

  • the special interest groups or SIGs as a way of organizing our

  • work. There are so many uses of TensorFlow, so many environments

  • and architectures.

  • Many of them are outside of the scope

  • of the core team and we want to enable TensorFlow to grow and be

  • more

  • sustainable by creating a way for like-minded people to

  • collaborate.

  • This is why SIGs exist. They are groups of people working

  • together for a define project focus.

  • We started last year and we have six up and running now.

  • I am going to give you a quick state of

  • the SIGs and many, in fact most of the leaders, are here with

  • us. I will give a shout out to them and hopefully you will be

  • able to talk to

  • them in the launch and tomorrow.

  • SIG addons first.

  • Martin mentioned at the beginning of the

  • day that TF contrib is no longer a part

  • and SIG addons are where that is going.

  • They conform to these well defined APIs.

  • Most there is already an RFC posted

  • about where you can find things that you

  • used to find in contrib. How can you get involved if you have a

  • favorite and want to step up and be a maintainer for everybody

  • and how you can join in the project.

  • I encourage you to take a look at that. SIG build is where

  • TensorFlow meets the outside world. It is not always the most

  • glamorous

  • piece of work but building it and

  • packaging and distributing it is tough.

  • Thank you to

  • Jason and Austin who lead that SIG.

  • It is home for third party builds for architectures we

  • don't ship.

  • IBM power and SIG helps us be a better neighbor in the Python

  • ecosystem.

  • Machine learning generates a lot of extreme situations and

  • changes in the

  • way we package and distribute software.

  • SIG-IO helps connect other systems and your data NLT real

  • world exists using systems in other formats.

  • This group is led my yang tan and antoine. If you are using

  • any of these in the

  • Apache Eco system or the environments

  • you can use this addaddon. This group has already dropped four

  • releases and shipped our integration

  • with their module too.

  • SIG networking is where a lot are going to.

  • If you are using gdr verbs this is where you can find this.

  • If you are interested in this, or any

  • of the other SIGs, please talk to the leaders. They want help

  • and are up and running

  • and in a great place to bring people on.

  • If you are looking for a way to get involved in TensorFlow, this

  • is an ideal one.

  • Let me talk about SIG TensorBoard.

  • This is a great time to be involved as

  • the TensorBoard team is trying to figure out how we can best

  • enable people using TensorBoard for creating plugins or at

  • scale. If you go to the demo area above and go

  • to the TensorBoard stand you will find

  • Gal and Moni there who would be happy to talk to you. This is

  • the URL for anything you want to do joining the TensorFlow

  • community from docs to testing to SIGs and all the other ways

  • to be involved. The developer mailing list. Please head there.

  • If you are here with us, rather than on the livestream, we are

  • doing a

  • contributor

  • luncheon tomorrow where many of the core

  • team and SIGs will be there to talk to you. Finally, let's move

  • on and talk about TensorFlow World. I am so excited about

  • this. It is our vision to bring together the amazing people who

  • are part of our community and give space for everyone to

  • connect with each other. There is so much we can learn from how

  • we are all working with TensorFlow. Working with

  • O'Rielly media, we will have this event at the end of October

  • here in Santa Clara.

  • It will be four days that really celebrate the TensorFlow

  • ecosystem.

  • We will have content, talks, tutorials, there will be an expo

  • and place for vendors to present. We understand

  • TensorFlow gets out into the real world there is a large

  • ecosystem beyond folks in this room. We are really excited and

  • that means we can bring everyone together. The main point of

  • doing something like this is to connect all the amazing users

  • and everyone with experience to share. As you heard the call for

  • proposals is now open.

  • If you have anything to share with your work, your product,

  • your company, head to TensorFlow World and put in a proposal. You

  • have about 4-5 weeks while the call for proposals is open and

  • we will be selecting talks a few weeks after that.

  • I really hope you will

  • lend your voice to this event and I am so excited to see you

  • in October. Once again, thank you. We appreciate how much you

  • are part of our community.

  • In my job, the best thing every day is

  • when I get to talk to folks developing or using TensorFlow.

  • It is so exciting to work with everybody here.

  • This is the way you can meet me. Please do.

  • If you have any issue or devire to contribute, reach out. We are

  • really happy to talk to you. Thank you very much for your

  • attention. -- desire. [Applause]

  • Hey, everybody. Thank you, Edd. That was wonderful. We are at

  • another break. Come back here at 2:25. We will be talking about

  • TensorFlow in production. Then the next block of talks is about

  • probability and so forth. Don't forget to go upstairs.

  • We have deem demos of all kinds of stuff and Googlers galore.

  • Check it out. Thanks a lot.

  • . >>PAIGE BAILEY: Eager execution

  • and all

  • of the things that we have loved to come to know and love through

  • the last several iterations of TensorFlow development.

  • We have a lot of tutorials and documentation that have been

  • recently released, it is about making you

  • productive as quick

  • as possible, and giving you scalable,

  • reproducible learn

  • ing. >>LAURENCE MORONEY: Yes, and

  • with it being open source, of course, is the

  • community is just a huge part of what we do.

  • >>PAIGE BAILEY: We could not do it without the community.

  • >>LAURENCE MORONEY: And that is why we are doing the live

  • stream.

  • >>PAIGE BAILEY: If you have questions, #askTensorFlow is the

  • hashtag. >>LAURENCE MORONEY: And one

  • question

  • that we are going to cover, you will get

  • today, it is great to do my training for

  • a fixed number of epochs, but when I reached the desired

  • accuracy metric, how do I continue training?

  • >>PAIGE BAILEY: It is not easy to compute if you have gotten to

  • a point where your boss says, okay, cool, 99 percent accuracy,

  • that is fine for us today. >>LAURENCE MORONEY: Should we

  • take a look at how we are doing that?

  • >>PAIGE BAILEY: Absolutely. >>LAURENCE MORONEY: I opened up

  • a codelab where you can see where I'm

  • causing call backs, and call backs are the ways that you

  • would achieve this. And at the top, you can see the class, the

  • call back.

  • And then when an epoch ends in training, I can look at the

  • logs.

  • If the accuracy log is greater than 60

  • percent, he is really happy that I'm on the training, 60 percent

  • accuracy, then I would cancel the training.

  • And then to be able to set it up, I

  • would create an object, call backs, an

  • instance of the class, and I love the

  • click-in, the little things that I love, and call backs equal

  • call backs, and

  • when I do the training, I will show off

  • and make my run time tied to the GPU, so it is really fast. So

  • let's do a little bit of training with this one, so I'm

  • doing this live

  • and I'm connecting to the VM, it is training, getting ready to --

  • >>PAIGE BAILEY: The RAM and disk utilization.

  • >>LAURENCE MORONEY: We are on the first epoch, that is

  • progressing away, 60,000

  • images being trained

  • , and I hit accuracy of 83, which is really good. But it

  • reached 60 percent accuracy, so I canceled the training.

  • Call backs are helpful, before I

  • learned about call backs, it is like, I

  • would set something up to train for 100 epochs, go to sleep,

  • wait up the next morning, and after three epochs, it didn't do

  • its job and I wasted the time. >>PAIGE BAILEY: Yes, call backs

  • are incredibly invaluable, it does not just apply to accuracy

  • either. There's a bunch of additional metrics that can be

  • useful for your particular workflow, and this code works in

  • TensorFlow 2.0. >>LAURENCE MORONEY: Absolutely.

  • >>PAIGE BAILEY: So it is Caras, I love Caras.

  • >>LAURENCE MORONEY: It is a love affair, and one neat thing about

  • Caras that you

  • might not realize, is the same code for TensorFlow 1.x is the

  • same for 2.0, so what is going on behind the scenes is it is

  • equally in 2.0 instead of a graph mode. So this codelab, 1.

  • 13, this code will run in 2.0 without you modifying the code.

  • >>PAIGE BAILEY: Absolutely. >>LAURENCE MORONEY: Should we

  • take the next question? >>PAIGE BAILEY: Cool. So the

  • next question is from Twitter it looks like. And what about all

  • the web developers who are new to AI ?

  • Does the version 2.0 help them get started?

  • >>LAURENCE MORONEY: Oh web developers.

  • >>PAIGE BAILEY: I just finished talking to two folks from the

  • TensorFlow JS team about the cool new things they have seen

  • from the community, a vibrant ecosystem of artists and

  • creators that are using browser-based and server-based

  • tools to create these machine learning models, training and

  • running them. >>LAURENCE MORONEY: Yep. And

  • for web developers, there's a

  • whole bunch of ways you can get started with this. So you

  • mentioned TensorFlow JS, we will start with that, but it is a

  • JavaScript library.

  • And this will allow you to train models in the browser, as well

  • as executing them. And that blew my mind.

  • >>PAIGE BAILEY: And the node bindings, being able to use the

  • GPU inside your

  • laptop with, you know, Google Chrome or your favorite flavor

  • of browser, to

  • train a model

  • . That is absurd. >>LAURENCE MORONEY: And node

  • bindings as well, with node.JS. It is not only in JavaScript,

  • but server-side JavaScript with node. And am I supposed to say

  • Node.js? I will say node. And, of course, by the fact that it

  • is a node, one of my personal favorites, are you familiar with

  • Cloud Functions for Firebase? >>PAIGE BAILEY: I'm not, I'm

  • intrigued. >>LAURENCE MORONEY: I used to

  • work on the Firebase team, so shout out to my friends on

  • Firebase. >>PAIGE BAILEY: I have heard so

  • many good things about Firebase. >>LAURENCE MORONEY: It is for

  • mobile and web developers, and one of the things

  • that Firebase gives you are these things

  • called Cloud Functions for

  • Firebase, I called up the web page, and you can execute the

  • functions on the back end without maintaining a server

  • infrastructure and execute them in response to a trigger, such

  • as an

  • analytic event, or a sign-in event.

  • >>PAIGE BAILEY: Or you get new data and

  • you have to process it. >>LAURENCE MORONEY: There you

  • go. And the fact that they run JavaScript code on the back end,

  • and now you can train models in a cloud function.

  • >>PAIGE BAILEY: That's amazing. >>LAURENCE MORONEY: So web

  • developers, a lot of great options for you.

  • However it is you want to do it, in

  • the browser, the back end, you can get started with it.

  • >>PAIGE BAILEY: And TensorFlow 2.0, if it gives additional

  • tools for application developers, I think it would

  • mostly be in terms of those codes and tutorial that we are

  • mentioning before. And we also released some courses.

  • >>LAURENCE MORONEY: Yep. >>PAIGE BAILEY: So it is easier

  • than ever to get started.

  • And, um, the models that you create can

  • be deployed to TS lite. >>LAURENCE MORONEY: And we have

  • been talking about Caras, and the layers are supported in

  • TensorFlow.JS. It is not just for Python developers, if you

  • are a JS developer, you can define your layers.

  • >>PAIGE BAILEY: And an R developer, they

  • have Caras for R, which was created by

  • JJ Layer and Francois Chalet. >>LAURENCE MORONEY: A lot of

  • options for developers. And next question, from Twitter, are

  • there any JS learning examples from object detection?

  • >>PAIGE BAILEY: Node.JS is popular, we have learned.

  • >>LAURENCE MORONEY: So object detection.

  • How do we answer this? So I'm -- it depends on what you mean

  • by object detection because, in Google, we talk about object

  • detection and use that term for, in an image, you have a

  • lot of objects and bounding boxes for them, there are

  • samples for that. >>PAIGE BAILEY: And a lovely

  • thing is

  • the community is adept at creating TensorFlow.

  • JS examples, for example, code pens, and

  • Victor Dibia, a

  • machine learning expert, for tracking hand movements. So if

  • you want to create -- >>LAURENCE MORONEY: You have an

  • incentive.

  • >>PAIGE BAILEY: I have a limited

  • edition collection of TensorFlow socks, and I would love to send

  • you a bright,

  • shiny pair of limited edition socks.

  • >>LAURENCE MORONEY: They are classics, they have the old

  • logo. >>PAIGE BAILEY: Not the new

  • logo, but the old box style ones.

  • >>LAURENCE MORONEY: I love them both.

  • And, like the question was about

  • transfer learning, and we did not have a demo of object

  • detection, I would love to show Transfer Learning and being able

  • to detect a single item in the frame. To do that, we call

  • image classification. So can I roll the demo?

  • >>PAIGE BAILEY: Please do. Are you going to do your favorite --

  • >>LAURENCE MORONEY: I am a child of the '80s, I love Pac-Man. So

  • you will notice that it says loading mobile net now, so it

  • downloaded

  • the mobile net model. So I'm going to add some new classes to

  • the mobile net model and then use transfer --

  • >>PAIGE BAILEY: There it goes. No, no, go for it.

  • >>LAURENCE MORONEY: So Pac-Man, old arcade game, you try to move

  • away from the ghosts.

  • I will train it to move up when I'm pointing up, I will gather

  • samples, about 50, when I go right like this, and turning

  • left is going to be hard. I didn't think it through. Bear

  • with me, so maybe I will do left like this, get my head out of

  • the way, a few samples like that, and then down will look

  • like this, and hopefully these aren't weird gestures in some

  • country. And something like that, okay.

  • So I have now picked 50 samples of these, I will retrain these

  • in the browser with transfer learning.

  • So if I look at the rate, I will start training and I will see --

  • it starts going quickly, you can see the loss

  • started at 4, it went town down down down, all to zero, and

  • probably a digit beyond the 6 digits,

  • never at zero, it is a very low loss, now we will give it a try.

  • I will stop playing the game, move left, you can see the

  • bounding box

  • around it, kind of shows that I'm

  • up, right, watching the screen.

  • And now you can see that I have trained it, going right this

  • time. Right, right, and up.

  • And it thinks it is down, up, and right.

  • And I'm not good at the game. >>PAIGE BAILEY: You are using it

  • as an excuse to play Pac-Man all day.

  • >>LAURENCE MORONEY: It is a great example of transfer

  • learning, and you can see in JavaScript how easy it is to

  • extract the features from mobile net and then retrain it, and it

  • is moving as I'm talking.

  • And so enough fun with Pac-Man

  • -- >>PAIGE BAILEY: Transfer

  • learning can be used for a variety of cases. So be sure to

  • look at the examples listed on the website.

  • And this is on Twitter, which is very popular. Hi, dear ask

  • TensorFlow. >>LAURENCE MORONEY: We have to

  • respond to that, they said dear. >>PAIGE BAILEY: Yes, are you

  • going to publish the updated version of

  • TensorFlow, the poet's tutorial from Pete Warden on TS2.

  • 0, TS lite, and a lot of other shenanigans.

  • >>LAURENCE MORONEY: And the network API.

  • >>PAIGE BAILEY: I love Pete Warden's codelab on TensorFlow

  • for poets, and he had a great talk today.

  • >>LAURENCE MORONEY: I didn't get to see it, I was preparing.

  • >>PAIGE BAILEY: It is about creating

  • tiny models and putting them on computers on mobile and embedded

  • devices. >>LAURENCE MORONEY: So at some

  • point, we will update it. I don't think there is an updated

  • version available, but one of the things that I liked about

  • the TensorFlow for

  • poets codelab is it got me building a model quickly for the

  • mobile device. And the draw back of that is there are

  • a bunch of scripts that I ran, I didn't

  • know what I was going on with them.

  • And I decided a bunch of TensorFlow

  • lite examples, image classification, object

  • detection, image recognition, and speech detection.

  • They are all open source, Android and

  • iOS, and they have full instructions on how to do it

  • yourself. The image classification is fun, and I

  • will try and run that in my, um, Android

  • Emulator, so we can see it running in an emulated

  • environment. So let me get that started. You can see it being

  • cached.

  • For example, here, it is doing a classification of what is in the

  • background, if I hold up a water bottle, this way, it detects it

  • is a water bottle. This is running in the Android Emulator,

  • okay? And this is using TensorFlow lite, and this is the

  • sample on there that does the same thing that you would seen

  • in TensorFlow for poets, it is using mobile net and building an

  • application around mobile net. If you look, running in the

  • emulator,

  • I'm getting inference times in the 100 and 170 milliseconds.

  • >>PAIGE BAILEY: The ability to take large scale models and sort

  • of pull them done to a manageable side on a mobile, or

  • an embedded device, is huge, and I'm excited to see what

  • TensorFlow lite does this year. >>LAURENCE MORONEY: So we are

  • working on a bunch of tutorials, both samples are out there, if

  • you look at the GitHub page, you will see that there are --

  • there are details on how it is built. Let me go back on here.

  • And there are details how it is built, how you can put it all

  • together and compile it.

  • You don't need to use

  • build TensorFlow to use TensorFlow lite, so you can go

  • and start kicking the tires on these applications for yourself.

  • >>PAIGE BAILEY: Excellent. >>LAURENCE MORONEY: All right.

  • We will have more codelabs, I would

  • love to get Pete's TensorFlow for poets

  • updated, hopefully for I/O. >>PAIGE BAILEY: And watch Pete's

  • talk. It was so good, people were talking

  • about putting models on comadors. I was nerding out a

  • little bit. >>LAURENCE MORONEY: And that's

  • why we're here. Cool! So -- do we have time for one

  • more question before break? So we are heading to break now, so

  • Paige and I will be back after the break with the rest of your

  • questions. Please keep asking them on the live stream and on

  • social media. Like I said, anything that we cannot get to,

  • we will try to get to offline and we will reach out back to

  • you. >>PAIGE BAILEY: Absolutely. So

  • super excited to have you here,

  • super excited to be at the TensorFlow Developer Summit.

  • >>LAURENCE MORONEY: And we will see you back here in a few

  • minutes. >>PAIGE BAILEY: Yep.

  • [Music].

  • PAIGE BAILEY: Welcome back. I'm Paige Bailey.

  • LAWRENCE MORONEY: I'm Laurence Moroney.

  • PAIGE BAILEY: We're here to answer your TensorFlow questions

  • with #askTensorFlow. LAWRENCE MORONEY: And we will

  • try to get back to you today, if not, we will reach out to you

  • later to answer them.

  • PAIGE BAILEY: We will go to the first question.

  • LAWRENCE MORONEY: This is on Twitter,

  • once I installed TensorFlow GPU, it did not work at first, failed

  • to run native TensorFlow run time.

  • I remember seeing this one, it was on YouTube in response to a

  • pip install

  • TensorFlow GPU in codelab, because once

  • upon a time, you had to pip install TensorFlow GPU to use

  • it, and now you are running into issues, and the reason is good,

  • it is good news, you don't have to run it anymore.

  • So if we switched to colab, this is a notebook that I was showing

  • you earlier on.

  • If you wanted to use GPU in colab, pick the run time type,

  • pick GPU as the hardware accelerator and now you don't

  • need to pip install TensorFlow GPU, it

  • does it under the hood behind the scenes, it is really cool

  • and earlier

  • why I trained this so quickly, I used the GPU and there is no pip

  • install GPU on here. PAIGE BAILEY: When we were

  • installing TensorFlow 2.0, we had an issue with the GPU

  • install,

  • and you needed specific

  • Kuta drivers.

  • Kolob is a great tip if you -- LAWRENCE MORONEY:

  • >>AND: GPU stuff, this is what I ran into a number of times

  • when using the GPU, you have to carefully take a look at the

  • version of Kuda and see the DNN that you are using, because I

  • made the mistake that I just went to the vendor's website, I

  • downloaded the latest version, I installed them, and then I saw

  • TensorFlow was actually supporting a slightly earlier

  • version.

  • So if you do get an error when you are using GPU, take a look

  • at the version of the driver it is looking to support and

  • then from the VEPD er vendor's website, download that version.

  • PAIGE BAILEY: Driver issues, they are always a treat.

  • LAWRENCE MORONEY: Exactly. And it is like -- it is one of the

  • things that makes our job interesting.

  • PAIGE BAILEY: Absolutely. LAWRENCE MORONEY: Haha, all

  • right. Should we look at the next one?

  • PAIGE BAILEY: Oh, let's go.

  • Adi Kumar had at least eight excellent questions on Twitter.

  • LAWRENCE MORONEY: He wins the volume award.

  • PAIGE BAILEY: He absolutely does, we

  • will get to all of them, not in this Ask TensorFlow segment, but

  • we will focus on one for today and then answer the rest

  • offline. LAWRENCE MORONEY: And I think a

  • lot of them were about file formats and how do I use

  • different file formats. Should I drill into that?

  • PAIGE BAILEY: Absolutely.

  • The question is, which is the preferred format for saving the

  • model going forward, saved model, or something else?

  • If we look at the laptop, we can take a gander at one of the

  • slides from the

  • keynote this

  • morning, that Keras is a driver of 2.0 and this is at the heart

  • of the deployment.

  • You can see it for TensorFlow serving, lite, and JS, and a lot

  • of other language bindings.

  • We are pushing for the saved model.

  • LAWRENCE MORONEY: And you cannot go wrong.

  • PAIGE BAILEY: It is easier to use than the other deployment

  • options we have seen before. LAWRENCE MORONEY: So I think the

  • guidance is in the recommendation, not just for AD,

  • but for everybody else. When you are thinking about saving

  • your models, take a look at save model, consider using it, and

  • not only is the

  • advantage of the file format, but just how it is supported

  • across all of these. PAIGE BAILEY: And an excellent

  • point of TensorFlow 2.

  • 0, I will keep selling it, we have a

  • number of code samples and tutorials today on saved model.

  • LAWRENCE MORONEY: I was playing with the

  • lite stuff, saving as a saved model, and the TensorFlow lite

  • process was a lot easier for me. So it is really refined, we are

  • iterating on that, and I think it looks really cool.

  • PAIGE BAILEY: Yep! Excellent. LAWRENCE MORONEY: So thank you

  • for all

  • of those questions, Adi, we will try to answer the rest of them,

  • most of them were focused around the file format and hopefully

  • saved model will help you. PAIGE BAILEY: Let's go to the

  • next one.

  • LAWRENCE MORONEY: This is from Eli

  • Gaguba, is it possible to run TensorBoard on codelabs?

  • PAIGE BAILEY: You are going to be so delighted.

  • LAWRENCE MORONEY: Because before if was running on codelabs, we

  • were talking about, we really want to run codelabs.

  • PAIGE BAILEY: It is so painful!

  • If you wanted to get it working in a

  • collab notebook, it was not really approved by our bosses or

  • in general. But, yes, so the good news is that you

  • can run TensorBoard in codelab. LAWRENCE MORONEY: And before it

  • was publicly announced, when it was announced

  • internally, we got an email from Paige with a lot of smily faces.

  • So thank you for your question, it made Paige's day.

  • PAIGE BAILEY: And I'm downloading files in hopes to

  • play with it a little bit, but here you can see it working.

  • You should be able to do different

  • operations like smoothing

  • , and changing the values and using the

  • embedding visualizer from the Collab

  • notebook to improve your accuracy and model debugging.

  • Another thing that we are working

  • very, very hard on is you don't have to specify ports. You

  • don't need to remember if you have multiple TensorBoard

  • instances running,

  • that you are using 6006, or whatever, for another.

  • It automatically selects one that's a good candidate, and it

  • would create a good fit for you. So the team is phenomenal, if

  • you have

  • any interest whatsoever in TensorBoard at all, I recommend

  • stalking the POs, like I do. That's how I found out that

  • TensorBoard was added to Jupiter notebooks and also to codelab.

  • So it is so exciting, we will have this link in the

  • documentation as well, the little notes underneath for you

  • to play it. LAWRENCE MORONEY: It is so great

  • to have a PR stalker [ laughter ].

  • PAIGE BAILEY: I get push notifications in my phone, it is

  • a problem. But they are doing such great work.

  • LAWRENCE MORONEY: So yes, TensorBoard is working in

  • Collab.

  • PAIGE BAILEY: And also project Jupyter notebooks! TensorBoard

  • everywhere.

  • LAWRENCE MORONEY: And we all love it. I have not played with

  • it that much, do the plugins work?

  • PAIGE BAILEY: So TensorBoard is a collection of different

  • visualization, you can see scalers, accuracy, histograms,

  • you can see embedding

  • visualizers which allows you to do a

  • great Mnist example from TensorFlow a

  • couple years ago, it is beautiful, and beholder, created

  • by a community member.

  • And the plugins, a couple of the Google Summer of Code projects

  • this summer are focused on getting additional visualization

  • plugins added to TensorBoard. And so in addition to the what

  • if tool, and in addition to beholder, you

  • can make your own TensorFlow plugin.

  • LAWRENCE MORONEY: And what

  • while I'm geeking out about it all day, I'm sure we need to

  • move to another question. So the next question that came in:

  • How

  • would you use future columns with Keras?

  • PAIGE BAILEY: I know you know the answer to this question.

  • LAWRENCE MORONEY: And it is a passion of yours. Feature

  • columns are a part of estimators and are a way of getting your

  • data efficiently into estimators, and with people

  • saying, like, hey, it is all there in estimates, what about

  • Keras?

  • You have been working on great stuff around that.

  • PAIGE BAILEY: And I know that the

  • YouTube channel has a series from Karmel.

  • LAWRENCE MORONEY: Karmel is the engineering director for

  • high-level

  • APIs, she has a great series around high level APIs for

  • TensorFlow, and is teaching about that and her and her team

  • are working on parity for columns on TensorFlow 2. I have

  • not checked the alpha yet, but it is on the way, if it is not

  • there already. So you should be able to use them in Keras.

  • PAIGE BAILEY: Yes, we are in the process

  • of building out, if you wanted to migrate your model, using

  • estimators to be more of a TensorFlow 2.

  • 0 with Keras, I'm in the process of building a migration guide.

  • If you have any interest around that, please feel free to reach

  • out and we're excited to get that released pretty soon.

  • LAWRENCE MORONEY: I'm excited to see that, personally, because

  • the very first

  • stuff I did in TensorFlow was with estimateers before I

  • learned Keras, and I wanted to change them to use Keras without

  • re-writing them. PAIGE BAILEY: Absolutely, Keras

  • is just friendlier to work with, I feel.

  • LAWRENCE MORONEY: They are both greatest estimators, and Keras

  • is great for beginners, so hopefully we will get the best

  • of both worlds.

  • So next question, Jeff Thomas, looking for data sets for

  • testing and comparing different training methods.

  • Looking for new data sets, like MNIST

  • is great, but after a while, people want something new. So

  • what could you tell Jeff? PAIGE BAILEY: We can tell him

  • about TensorFlow data sets!

  • LAWRENCE MORONEY: And there's a blog post about it.

  • PAIGE BAILEY: It is about creating the data ingestion

  • pipelines for you to be able to easily use a variety of data

  • sets with all of your deep learning and machine learning

  • models with just a few lines of code.

  • So if you are familiar with skikit

  • learn, and all of this, it is nifty data ingestion practices,

  • this feels very similar. It is easy to do, training and

  • testing, and verifications.

  • And we have an LOT of data sets readily available for you right

  • now to go ahead and explore. And another thing that I would

  • especially like to call out is, um, a member of our community --

  • so anybody can make your data famous, right? If you have a

  • data set that you are using on your research lab, if you have,

  • like, a bright and shiny CSV, that you think would be a cool

  • add to this section. LAWRENCE MORONEY: I have never

  • used a

  • CSV called bright and shiny. PAIGE BAILEY: Everybody uses

  • them, a

  • community manager, Andrew, an undergraduate researcher at

  • Stanford,

  • added 200,000 of these to the Stanford ML group, in less than

  • a day.

  • It is as simple as take the template format for images or

  • for audio or

  • whatever you are using, add some additional metadata for the data

  • set, change the types for the features you are using with

  • that, and voila. LAWRENCE MORONEY: You are off to

  • the races. PAIGE BAILEY: And you can make

  • your data famous. LAWRENCE MORONEY: A wonderful

  • thing about it, if you are getting started or

  • if you are learning, if you look at the samples, pre-TensorFlow

  • data sets, you get a lot of code about downloading the data,

  • unzip it here, relabel it like this, put the files in these

  • folders, take a CSV and make these features, but it is one or

  • two lines of code and the training is set into data and

  • test sets for you, that type of thing.

  • So for learners, I found it exciting because I didn't have

  • to go through 100 lines of code before I get to the neural

  • network. PAIGE BAILEY: And also data

  • sciency

  • people, creating and training a model is understanding the

  • shape, format, and the statistical distributions, and

  • this is really helpful. LAWRENCE MORONEY: Yep. So

  • that's all we have time for. We're going to take a break, we

  • will be taking some more of your questions during the break.

  • And then we will be back in just a few minutes to start answering

  • them. Thanks very much. [Music].

  • . LAWRENCE MORONEY: Welcome back

  • everybody to the live stream where Paige and I are answering

  • questions that you posted on

  • social media, on the live chat, Stack Overflow, and anywhere

  • else you can find them.

  • If you have questions, please hashtag #AskTensorFlow.

  • PAIGE BAILEY: And if we do not answer your questions today, we

  • will in further sessions, so keep asking, even if we

  • don't necessarily get to them.

  • LAWRENCE MORONEY: There are so many that we cannot get to them

  • there today, a great problem to have. We have a show, Ask

  • TensorFlow. So we will put these questions into

  • the show, so please asking and this is

  • what keeps the community engaged.

  • So the first question, the pre-built binary for GPU on 16?

  • PAIGE BAILEY: That is very specific.

  • LAWRENCE MORONEY: I like specific

  • questions, even if I can't answer them. So the pre-built

  • binaries for

  • TensorFlow tend to be associated with a

  • specific driver for

  • NVIDIA. So if you are looking at any of the pre-built

  • binaries, look at what version

  • of the driver you have supported.

  • I'm not an expert on NVIDIA cards, and if you go over here,

  • like on my laptop, I have -- I have called up, like, some

  • of the what NVIDIAs say about the

  • TensorFlow system requirements, and the

  • specific versions of the drivers they support.

  • And one gotcha that I found in working with GPUs is it is

  • easier to go to the driver vendor and download the latest

  • version, but that is not one that TensorFlow is built for

  • that it supports.

  • So make sure that they actually match each other, you should be

  • good to go even with that particular card.

  • PAIGE BAILEY: If you have warm feelings and excitement for

  • builds for

  • TensorFlow, we have an interest group, Sync build. So please go

  • to the community section

  • of our GitHub and check up the listserv and joining it in and

  • joining our weekly stand-up. LAWRENCE MORONEY: Right. So

  • thanks, Arthur, for that question.

  • And then the next question is a really funny one. How many

  • times have you been asked this today?

  • PAIGE BAILEY: At least 12. At least, and then the other flavor

  • of

  • it is, is this symbol that I use all the time, is this supported

  • in TensorFlow 2.0, and if not, what has changed?

  • LAWRENCE MORONEY: Yep.

  • And I -- I mean, people have invested so much time in 1.

  • x, they don't want it to be deprecated or go away.

  • So do scripts for 1 work with 2? PAIGE BAILEY: The sad fact is

  • probably not. They would not work with TensorFlow 2.0 out of

  • the box.

  • But, we have created an upgrade utility to use, it is

  • automatically downloaded with TensorFlow 2.0 when you download

  • it, and you can check out what it is doing in this

  • Medium blog post that I and my colleague, Ana, created, as well

  • as this TensorFlow 2.0 video, and the gifs, the best

  • communication medium possible, shows you how you can use the

  • upgrade script on an N file.

  • So any, sort of, arbitrary Python file

  • or Jupyter notebooks, a machine learning IDE extension that

  • allows you to do that as well.

  • And it will give you an export.

  • text file that shows you the symbol

  • renames and the manual changes if you have to add them. And

  • usually you do not. So to see it in action, you can take a

  • look at this particular text generation

  • example that we have running a

  • Shakespeare, well, take all of the corpus of Shakespeare text,

  • trains it

  • against the Shakespeare text, and generates something that he

  • could have potentially written, should he have had

  • access to deep learning resources.

  • LAWRENCE MORONEY: I know you all, and I

  • will up hold the wildest humor of your idle dream.

  • PAIGE BAILEY: I didn't know you knew Shakespeare.

  • LAWRENCE MORONEY: I played Henry the IV.

  • PAIGE BAILEY: I was in much ado about nothing.

  • LAWRENCE MORONEY: So while we are on

  • much ado about nothing, the Jupyter

  • notebook. PAIGE BAILEY: Export the Python

  • file, and to reupgrade it. LAWRENCE MORONEY: You have to

  • reconnect

  • the run. PAIGE BAILEY: So starting it.

  • And the requirements, we can check

  • that we are using TensorFlow alpha, and like I mentioned

  • before, all you have to

  • do is preface this was a bang, the name of the Python file is

  • text generation, and I wanted to create an upgrade, shift

  • enter, and it does its upgrading magic and, very quickly, it

  • tells me all of the things that we need to change in order to

  • make it 2.

  • 0 compatible and create that file for me off to the side. So

  • if I wanted to run this model, it should be able to -- it

  • should be able to train as it would.

  • So let's just check to make sure that that would be the case.

  • LAWRENCE MORONEY: I think a lot of the errors you are seeing

  • here, it is renamed APIs rather than breaking changes within the

  • APIs as well. PAIGE BAILEY: This is true.

  • You have re-names and additional key words. So for more

  • information about that,

  • and to see it running

  • in a video, you can go and take a look there.

  • LAWRENCE MORONEY: Sounds good, and you

  • have some gifs in there? PAIGE BAILEY: Absolutely.

  • LAWRENCE MORONEY: And any jifes for those that don't say gifs?

  • PAIGE BAILEY: It is in the works.

  • LAWRENCE MORONEY: [ Laughter ], so when it comes to upgrade,

  • there's a few little gotchas, so hopefully the blog

  • post, the videos, will help you get around them.

  • PAIGE BAILEY: And the community you are mentioning before, we

  • had such an interest in testing TensorFlow 2.0 and trying it out

  • against historic models that we've formed a weekly testing

  • stand-up, and also we have a

  • migration support hour that is being implemented with the

  • internal support hour.

  • So if you have an external group that

  • is interested in upgrading your model, please join the testing

  • group and we can get you situated.

  • LAWRENCE MORONEY: And a lot of the stuff

  • that we have seen in Keras, you have the great slide where she

  • is training -- PAIGE BAILEY: The code is

  • exactly the same. LAWRENCE MORONEY: Exactly the

  • same. So while there is stuff changing under the hood, a lot

  • of the surface-level code that you are writing in Keras is not

  • changing. PAIGE BAILEY: If you are used

  • Keras, you are probably not going to have any problems.

  • LAWRENCE MORONEY: So good stuff. Should we move on to the next

  • question? So I know we can talk about 2.0 all day, but we just

  • mentioned Keras, and it appears.

  • PAIGE BAILEY: So I guess I can ask you this question. What is

  • the purpose of keeping

  • estimators in Keras as separate APIs, is there something native

  • to Keras models that allows for distributed training, or train

  • and evaluate? LAWRENCE MORONEY: The purpose of

  • keeping them, there are many purposes. So for me, the main

  • purpose that I would like to think of is, because a lot of

  • people are using them. PAIGE BAILEY: Including internal

  • Google

  • teams that would tar and feather us if we removed them [ laughter

  • ]. LAWRENCE MORONEY: And so, when

  • it comes to estimators, estimators are really great for

  • large-scale training. PAIGE BAILEY: Yes.

  • LAWRENCE MORONEY: And it is like a lot of times, you are doing

  • the large scale training, keep going with estimators, they are

  • great. When I first started with TensorFlow, I started with

  • estimators, I could not figure out what a node was in a neural

  • network and all of these concepts that I had to learn,

  • where I had a simple estimator that I can use to do a DNN or

  • something like that. And so, you know, they are there for a

  • reason and they are the same for the reason. Keras is one of the

  • things that, from the point of view of making life easier for

  • developers that we have been

  • doubling down on TensorFlow 2, and the code is the same between

  • 1 and 2, and if -- the layers API makes it super simple for

  • you to design a neural network, and the fact that you can go low

  • level

  • beyond that, you know, to define your

  • own layers allows you to drive stick rather than automatic.

  • PAIGE BAILEY: And what is beautiful about Keras 2.0 is you

  • have Keras the way you are used to using it, and if you want

  • customizations, there's a sub-component, if you wanted to

  • go lower, we have something called TS module and we have

  • even exposed some of the basic, most core ops in TensorFlow as

  • well. So, at any sort of level you want to interact with the

  • API, you can. LAWRENCE MORONEY: Yep, and I

  • think there is another part of the question that is around

  • distributed training. So you can scroll down there, and there

  • is something called distributed strategy.

  • PAIGE BAILEY: Yep. LAWRENCE MORONEY: With Keras and

  • with TensorFlow 2, and the whole idea behind that is you can

  • distribute your training, maybe across multiple GPUs on the same

  • machine, maybe across multiple GPUs on different machines, and

  • maybe

  • across CPUs all over the place. So distribution strategy is a

  • part of that, to help you with that. So estimators in Keras,

  • we love them both, they are still there, and hopefully this

  • is something that will help you with a question. I think we

  • have time for just one more. PAIGE BAILEY: Absolutely.

  • LAWRENCE MORONEY: Oh, this is a Paige question.

  • PAIGE BAILEY: This is totally a me question. I'm the Python

  • person. So ask TensorFlow, when will

  • TensorFlow be supported in Python 3.

  • 7 and be accessed in Anaconda3? I can answer Python 3.7, and I

  • would love to speak a little bit more about support for Python

  • going forward. LAWRENCE MORONEY: Please,

  • please. PAIGE BAILEY: So, to answer the

  • 3.7 question, I'm going to bounce over to the TensorFlow 2.

  • 0 project tracker, and these are all of the standing issues that

  • we have when doing development for TensorFlow 2.0, it is

  • transparent -- LAWRENCE MORONEY: You can see

  • your avatar.

  • PAIGE BAILEY: I have filed many issues and they are transparent

  • to the public. If you want a context on where we stand

  • currently and what we have yet to

  • do, this project tracker is a great way to understand that.

  • And please take a look at 3.7 and, there we go.

  • In process of releasing binaries for Python 3.5 and 3.

  • 7, we issued 25, 420, it is going off the screen, and 4.29.

  • You can take a look at that issue and

  • it is currently in progress, there is not really an eta, but

  • something that we want complete by the time that the Alpha RC is

  • released. So that is wonderful to see. And there is also, um,

  • a website

  • called Python three statements, I think it is Python3statements.

  • com, maybe it is .org. There we go!

  • LAWRENCE MORONEY: Yay. PAIGE BAILEY: Cool. TensorFlow

  • made the commitment as of

  • January 1, 2020, we no longer support Python 2 and we have

  • done that with a plethora of our Python community.

  • So TensorFlow, Scikit-learn, etc.

  • , we are committed to Python 3 and Python 3 support.

  • So you will -- you will be getting your Python 3 support

  • and we are firmly committed to having that.

  • LAWRENCE MORONEY: And a nice thing about

  • the issue tracker is it is not a big,

  • hey, we have it, coming to a random point in the future, it

  • is transparent and you can see what we are doing.

  • PAIGE BAILEY: And you can see our people

  • commenting and the engineers commenting back, yeah, man, I

  • was running it last

  • night, one more test and it is it.

  • LAWRENCE MORONEY: That is all we have time for, and remember,

  • #AskTensorFlow

  • on Twitter, Stack Overflow, YouTube.

  • PAIGE BAILEY: Send a carrier pigeon if that's your preferred

  • method of communication.

  • LAWRENCE MORONEY: I prefer owls. PAIGE BAILEY: Owls are awesome.

  • LAWRENCE MORONEY: If you have questions, we will do the best

  • to answer them, we will put a lot on YouTube.com/TensorFlow.

  • So don't forget to subscribe, thank you so much for being

  • engaged. PAIGE BAILEY: Thank you.

  • [Music].

  • Often I hear sometimes from researchers,

  • I really only do research and care about training the machine

  • learning model and I don't mean these instruments.

  • What I would argue is research often leads to production and

  • where we want to

  • avoid is researchers having to reimplement their hard work in

  • the model

  • they have work when they want to put the model into the

  • production.

  • We really want the research community

  • to build the models and the framework in the community.

  • A second comment that I hear often is

  • well, I only -- and all are built to scale up. I don't

  • really need all of these heavy tools. What we have seen time

  • and time again

  • at Google, is small data today becomes large data tomorrow.

  • There is really no reason why you would

  • have to reimplement your entire stack because just your dataset

  • grew. We want to make sure you can use the same tools early on

  • in your journey so the tools can grow with you and your product

  • with the data so that you can scale the exact same code to

  • hundreds of machines.

  • We are built TensorFlow Extended and it

  • has had a profound impact in how we do

  • production and becoming an AI first company.

  • TFX powers some of the most important alphabet companies.

  • It is used at six different alphabet companies and within

  • Google it is used with all of the major products and all

  • of the products that don't have billions of users. I said before

  • we want to make TFX available to all of you because we have seen

  • the profound impact it has had on our business and we are

  • really excited to see what you can do with the same tools in

  • your companies. A year ago, we talked about the libraries that

  • we had open sourced at that point in time.

  • We talked about TensorFlow transform,

  • the training libraries, estimators, and keras, and

  • TensorFlow Serving. I made the point that back then, as today,

  • all of these are just libraries. They are low-level libraries you

  • have to use independently and stitch together

  • to make use and trend for your own

  • use-casesuse-cases.

  • We added TensorFlow validation later that year. It was valuable

  • to release these

  • libraries because some of our most

  • important partners externally have had a huge impack. Airbnb

  • uses TensorFlow server and our friends at Twitter published a

  • fascinating blog post of how they use

  • TensorFlow to rank tweets on their home

  • timeline and used it to analyze data and

  • used TensorFlow Hub to share the word embeddings they used for

  • these models.

  • Coming back to this picture, for those

  • who saw my talk last year, I promised everyone there would be

  • more.

  • This is just a set of libraries right now. Today for the very

  • first time, we are sharing the horizontal layers that

  • integrate all of these libraries into

  • one end platform into one end program called TensorFlow

  • Extended.

  • First, we have to build components.

  • At the top in orange are libraries from the past and in

  • blue is current libraries.

  • Libraries are low level and flexible.

  • We can build many components part of the machine learning

  • pipeline.

  • I will go into detail in each one of

  • these components later. A component is no longer just a

  • library.

  • It is a package binary or container that can be run as

  • part of a pipeline.

  • It is well defined input and outputs. In the case of model

  • validation, it is

  • the last validated model.

  • A new candidate model and validation. It has a well

  • defined configuration and most importantly it is one

  • configuration model for the entire pipeline.

  • You can figure a TFX pipeline end-to-end. Some of you may have

  • noticed because

  • model validation needs to last, it needs context and needs to

  • know what was the

  • last model validated. We need to add a metadata store that keeps

  • a record of previous runs so SH of

  • the more advanced capabilities can be enabled.

  • How does this context get created? The trainer produces

  • new model, the

  • model validator knows about the last validated model and

  • candidate model and downstream we take the new candidate

  • model and validation outcome and if the out' come is positive we

  • push to the server and if it isn't we don't because

  • we tonight --

  • don't want to push a model that is not neter. The metadata store

  • is new. When most people talk about machine learning learn

  • floes and pipelines they think about task dependency and think

  • about one component and when that is finished there is

  • another component that runs. However, all of you who do

  • machine learning in production know we need data dependencies

  • because all components consume and create artifacts and as the

  • example of model validation has showed,

  • it is incredibly important to know these dependency. We need a

  • system that is task and data aware so each component has a

  • history of all the previous runs and knows about all of the

  • artifacts. What is in this metadata store? Most

  • importantly, type definitions of artifact and their properties.

  • For TFX, it contains the definition of all artifacts that

  • have been consumeed and produced by components and all of their

  • properties.

  • It is an extensible type system and you can add new properties

  • to the artifacts if you need to track more properties of those.

  • Secondly, we keep a record of all of the execution of the

  • components.

  • With that execution, we store all of the

  • imported

  • artifact and the output artifacts that were produced and

  • the Runtime components.

  • If you want to track the code snapshot, you can store it in

  • the metadata store as well.

  • Putting these things together allows us to do something called

  • linear tracking across all executions. If you know every

  • execution, all inputs

  • and outputs, you can piece together a

  • story of how an artifact was created.

  • We can say what were the upstreams that went into

  • producing the artifact and what were the downstream runs

  • artifacts

  • that were produced using the artifact as an input. That is a

  • powerful capability and let

  • me talk you through some of the examples of what this enables.

  • The first is pretty straightforward. Let's say I

  • want to list all of the training runs I are done in the past. I

  • am interested in the trainer and I want to see all the training

  • runs that were recorded.

  • In this case I had two training run and see all the properties.

  • This is pretty straightforward and nothing new to see here. We

  • can visualize the lineage and all the information we have. The

  • first comment on this slide is we are working on a better UI.

  • This is just for demonstration purposes.

  • If you look at the end of the graph, you see the model expert

  • path.

  • This is the specific instance of the model that was created and

  • we see the model was created by the trainer and the

  • trainer created this model by consuming a schema, transform

  • and examples. These are specific instances so the IDs are not

  • just number.

  • They are schema of ID number 4 and for

  • each of those artifact WEEZsee they were created upstream.

  • This allows forward and backward in the

  • artifact and the narrative I used was walking back from the

  • model but you

  • could look at training data and say were what the artifacts

  • produced using that training data.

  • This slide shows a visualization of the data distribution that

  • went into the model.

  • At first glance, this may not be something like that that is

  • earth shattering because we have done this before. We can compute

  • and visualize statistics but if you look at the code snippet, we

  • are fought referring data or statistics.

  • We are referring to a model.

  • We say show me the distribution of data the model was trained on

  • and we can do this because we have a track record of all the

  • data and statistics that went into this model. We can do a

  • similar thing in the other direction saying for a specific

  • model show me the sliced metrics that were produced downstream by

  • TensorFlow model

  • analysis and we can get this visualization again just by

  • looking at a model and not specifically pointing to the

  • output of TensorFlow analysis. We know all the models that were

  • trained

  • and where the checkpoints lie so we can

  • start support to the historic runs and look at the tensor port

  • for the models you have trained in the past.

  • Because we have a track record, you can

  • launch tensor point and compare two models in the same instance.

  • This is really model tracking and experiment comparison after

  • the fact. We enabled this by keeping a track record of all of

  • this. If you have multiple models, you

  • can look at the data distribution for multiple models

  • and this usually helps with debugging a model. If you train

  • the same model twice, or

  • on data and it behaves differently, sometimes it can

  • pay off. We are overlaying two distributions of the statistics

  • for one model and the

  • other and you would see if there is a considerable drift between

  • those two.

  • All of these are enabled by this lineage tracking.

  • Another set of use-cases is visualizing previous runs over

  • time. So if you train the same model over time over new data we

  • can give you a time series graph of all metrics over time and you

  • can see if your model improves or gets worse over time as you

  • re-train it. Another powerful use case is carrying over state

  • from previous models because we know you have trained the model

  • in the past we can do something called warm starting. We can

  • reinitialize the model with weights from a previous run.

  • Sometimes you want to reinitialize the entire model or

  • just an embedding and in this way we can continue training

  • from

  • where we left off with a new dataset.

  • And another very powerful application

  • of this is being able to reuse previously outputed outputs.

  • If you have a pipeline that ingest data, applies

  • transformation to data and

  • then you train the model, every time you make a small change to

  • the model you don't want to recompute every upstream. There

  • is no reason just because you changed something in the model.

  • Because we have a track record of all the previous steps, we

  • can make a decision of saying your data hasn't changed, your

  • transfer code hasn't changed and we will reuse the artifacts that

  • produced the upstream and you can iterate much, much faster.

  • This improves iteration speeds and saves

  • compute because you are not reexot -- recomputing things

  • you already computed in the past.

  • We talked about components and how do we orchestrate TFX

  • pipelines? Every component has something we call a driver and

  • publisher. The driver's responsibility is to basically

  • retrieve state from the metadata store to inform what work needs

  • to be done. In the example of model validation, the

  • driver looks into the metadata store to

  • find the last validated model. The publish keeps the record of

  • everything going in the component, produced and Runtime

  • configuration so we

  • can do the lineage tracking I mentioned earlier.

  • In between is the executer who is unaware of the metadata

  • because it is important to make that piece relatively simple

  • because if you want to change the training code in a component

  • you should not have to worry about drivers and publishes,

  • just the executer. It makes it easier to write new

  • components for the system, also. We have one shared configuration

  • model

  • on top that configures end-to-end pipeline.

  • As you can see this is a Python dsl,

  • you see it has an object for each component from top to

  • bottom. The training component you can

  • see basically receives configuration and

  • says your input come from the transer outoutput and the schema

  • that was inferred.

  • Let's see what is inside the of the trainer and that is really

  • just TensorFlow code.

  • We use an estimator and an estimator train and evaluate

  • method to train this model.

  • It takes an estimator and we just use one of our pre-made

  • estimators.

  • This is a wide and deep model you can instantiate and return.

  • We don't have an opinion on what this code looks like. It is just

  • TensorFlow.

  • Anything that produces a safe model as output is fair game.

  • You can use a keras model that produces an inference graph or

  • if you chose to you can use lower level APIs in TensorFlow

  • as long as it produces a safe

  • model in the right format that it can be useded in TensorFlow

  • or for model analysis, you can write any type of TensorFlow

  • code you want.

  • So, if you have noticed, we still haven't talked about

  • orchestration.

  • We are a configuration, components and metadata store. I

  • know some of you might be thinking is he going to announce

  • a new orchestration system and the good news is no. At least

  • not today.

  • Instead we talked to a lot of users and

  • unsurprisingly found out that there is a significant install

  • base of

  • orchestration systems in your companies. We just heard from

  • Airbnb, of course they developed airflow and there is a

  • lot of companies that use Kubeflow and there is a number

  • of other orchestration systems. We made the choice to select any

  • number

  • of orchestration systems because we don't want to make you adopt

  • a new

  • orchestration system to use the TFX pipelines.

  • Reason number two is we want you to extend TFX pipelines.

  • We published our version of what a TFX pipeline looks like and

  • the components we use at Google, but we want to make it easy for

  • you to add components before, after and in parallel to

  • customize the pipeline to your own use-cases.

  • All these orchestrations are made to be able to express

  • arbitrary workflows and if you are familiar withing of those

  • orchestration systems you should be able to use them for a use

  • case. Here are two examples of what it looks like with airflow

  • and Kubeflow pipelines.

  • On the left you see the same TFX

  • pipeline configured that is executed on AirFlow.

  • In the Chicago taxi cab example we used 10,000 records.

  • On the right side you see the same

  • pipeline executed on Kubeflow on data Cloud so you can scale it

  • up to the 100 million rows in that dataset.

  • It is the same configuration and components running in both

  • environments and you can chose how you want to orchestrate them

  • in your own favorite orchestration system.

  • This is what this looks like if it is put together.

  • TFX goes all the way from raw data to your deployment

  • environment. We discussed a shared configuration model at

  • the top.

  • The metadata system that keeps track of the runs no matter how

  • you orchestrator

  • the component and two ways to publish them with airflow and

  • Kubeflow but you can chose to orchestrate the TFX pipeline in

  • any way you want. All of this is available now so

  • you can

  • go on

  • GitHub/TensorFlow/TFX.

  • Tomorrow we have a workshop where you

  • can get hands on experience with

  • TensorFlow Extended and there is no

  • prerecks and you don't to bring your own laptop.

  • We will jump into an example with the Chicago taxicab

  • dataset.

  • This is a record of cab rides in Chicago for some period of time.

  • It contains everything you would expect. It contains when the

  • trip started, when it ended, where they started and ended,

  • how much was paid for it and how it was paid.

  • Some of the features need

  • transformation so longitude and latitude

  • features need to be packaged.

  • We treat them as categorical features.

  • Strings need to be integerized and some

  • of the flows need to be normalized.

  • We feed the dense features into the deep part and all the others

  • use the wide part.

  • The label we are trying to predict is a bullion which is if

  • the tip is larger than 20% of the fare. Really, what we are

  • doing is building a

  • high tip predictor just in case if there

  • is any cab drivers in the audience or online, come find me

  • later. I think this will be beneficial to you if you can

  • predict if a cab ride give as high tip or not. Let's jump in

  • and start with data validation transformation. The first part

  • of the TFX pipeline is ingesting data, validating that data if

  • it is OK, and then transforming so it

  • can be fed into a TensorFlow graph.

  • We start with the

  • examplexamplegenwhich takeexamplegen. This is

  • extensible. You can ingest any type of data

  • in the pipelines. What is important is after the step the

  • data is in a well defined place where we

  • can find it, in a well defined format because all downstream

  • components standardize and it is split between training and

  • develop. You have seen the configuration

  • of these components before.

  • It is very minimal configuration in most of the cases.

  • Next we move on to data analysis and validation. I think a lot of

  • you have a good intuition of why it is important. Machine

  • learning is the process of

  • taking data and learning models that predict some field in your

  • data and you are always aware if you feed garbage in you get

  • garbage out. There is no hope in learning a

  • good machine learning if the data is wrong or has errors in

  • them.

  • This is reinforced if you have continuous pipelines produced on

  • data from a bad model and you are reinforcing the same

  • problem. Data understanding is critical

  • for model understanding.

  • There is no hope in understanding why a model isn't

  • predicting something if the data isn't right.

  • The question you may ask as a cab

  • driver is why are trip predictions bad

  • in the morning hours and I will try to

  • answers these with the TFX toolsets we V. There is a lot of

  • care taking with code.

  • It is previewed, checked into repository, version controlled

  • and data needs to be first-class citizens in these systems. With

  • the question what are expected values from payment types? That

  • is a question about the schema of your data and what we would

  • argue is that the schema needs to be treated with

  • the same care as you treat your code. And catching errors early

  • is absolutely critical because I am sure as all of you

  • know errors propagate thew the system.

  • If the data is not OK, every downstream goes wrong.

  • These errors are hard to fix if you catch them late in the

  • process. Really catching those problems as early as possible is

  • absolutely critical.

  • In the taxicab example, you would ask a question is this new

  • company I have in

  • my dataset a typo or a real company

  • which is a

  • natchal evolution of my dataset.

  • The statistics training program takes in the data and it can be

  • training or

  • serving logs in which case you can look at the skew between

  • serving and training data. Statistics really capture the

  • shape of your data and the visualization components draw

  • your attention to things that need your attention such as if a

  • feature is missing most of the time it is highlighted in red.

  • The configuration for this component is minimal as well.

  • Let me zoom into some of these visualizations. One question I

  • posted earlier was why

  • are my predictions bad in the morning hours. You could look at

  • the dataset and see for trip start hour in the morning hours

  • before 2:00-6:00 A.M. you don't have much data because there

  • is not that much taxi trips at that time

  • and not having a lot of data in a specific area can mean your

  • model is not robust or has higher variance and this

  • could lead to worse predictions.

  • Next schemagen.

  • It takes the output and infers a schema for you.

  • There is very few features in this case so you could handwrite

  • the schema. If you have thousands of features, it

  • is hard to handwrite that expectation so we infer the

  • schema for you and it represents what you expect from your data

  • and what good data looks like and what values your string

  • features can take on and so on.

  • Again, very minimal configuration. The question we

  • can answer is what are expected values for payment types. If you

  • look at the bottom, you see the field payment type can cake ton

  • cash,

  • credit card, dispute, no charge, p card and unknown.

  • Next time I run this, and this field takes on a different

  • value, I will get

  • an anomaly which comes from the example validator. The example

  • validator takes the statistics and schema as input and produces

  • on anomaly report. That anomaly report tell you if

  • the data

  • is missing feature, has the wrong relevancey, if the

  • distribution has shifted and deis important to highlight the

  • anomaly report is human readable so you can look at it and

  • understand what

  • is going on but it is always machine

  • readable so you can automatically make

  • decisions on the

  • anomalanomaly. Here you can see the field company has

  • taken on unexpected string values.

  • That just means these string values were not in your schema

  • before and that

  • can be a natural evolution of data.

  • The first time you run it maybe you

  • didn't see trips from those taxicab companies but when you

  • look you see they are normal so you can update or if you saw a

  • lot of scrambled text you would know there is a problem in your

  • data that you would have to go and fix.

  • Moving on, we actually get to transform. Let me just recap the

  • types of transformations we want to do.

  • I highlighted them in blue.

  • We want to convert the strings to ints

  • and for the tense features we want to normalize them. All of

  • these transformations require you to do a full pass.

  • To pact packatize you need to figure out the boundaries. For a

  • string to int you need to see all

  • the string values.

  • This is exactly what we build TensorFlow transform for. It

  • allows you to express the preprocessing function of your

  • data that contains some of these transformations that require a

  • full pass of your data and enable automatically to run the

  • data processing graph to compute those statistics. In this case,

  • you can see the orange

  • boxes are statistics that we require,

  • for minimalization we require a mean and

  • standard deviation and TensorFlow

  • transform says scale to C store and it will create a data

  • processing graph for you that will compute the standard mean

  • and data and return the results.

  • That graph is a hematic graph that contains all the

  • information you need to apply your transformations. That graph

  • can then be used in training and surveying guaranteeing there is

  • no drift between them.

  • This

  • eliminates the chances of skew and at

  • serving time you need to feed in the raw data and all the

  • transformations are done as part of a TensorFlow graph. The

  • transfer component takes in data, the schema allows you to

  • parse the data,

  • and code in this case the system

  • provided preprocessing function, and it

  • produces a hematic graph that gets attached to your training.

  • It can materialize the transform data

  • and that is a performance

  • optimization you need.

  • It can pay off to serialize some of the training before the next

  • step. The component takes in a module

  • file, a file where you configure preprocessing function.

  • In this code snippet, the actual code isn't important but I want

  • to highlight the last line in this code snippet and that is

  • how we transform our label because there label is a logic

  • expression of saying it is greater than 20% of my fare. I

  • want to highlight this because you don't need analyze phase for

  • all of your transformations.

  • In cases where you don't need analysis phases, the

  • transformation is just a regular TensorFlow graph.

  • To square something to C square and integerize schemes you need

  • these phases.

  • This is the user code you would write

  • and transfer creates the graph and returns the transfer graph

  • you need to

  • apply these transformations.

  • Now we are done with this we haven't trained a machine

  • learning model right but we have made sure we know our data is in

  • a place where we can find it, it is in a format we can understand

  • and split between training and deval and we know our data is

  • good because we

  • validated them and we are applying

  • transforms consistently between training and serving.

  • This bringstuse the training step.

  • This is where the magic happens. The training steps is the

  • TensorFlow graph and step. The training component takes in

  • the output of transform as mentioned which is the transform

  • graph and optionally

  • the materialized data, schema and

  • training codes that you provide.

  • It creates output to TensorFlow models. Those models are in the

  • safe model

  • format the standard serialized model in TensorFlow which you

  • heard quite a bit about this morning. We produced two of

  • them. One is the inference graph and another is the evaluate

  • graph which contains the metrics and necessary annotations to

  • perform TensorFlow model analysis. This is the

  • configuration you have seen earlier and again the trainer

  • takes in a module file and the code that is actually in that

  • module file, again, I am going to show you the same slides to

  • reiterate the point is just TensorFlow. In this case it is

  • the train and

  • evaluate method from estimators and the estimator being returned

  • here. Again, just to make sure you are

  • aware of this, any TensorFlow code here that produces the safe

  • model in the right format is fair game. This works really

  • well. With this, we have now trained the TensorFlow model and

  • now I am going to hand it off to my colleague Christina

  • who will talk about model evaluation analysis.

  • -- model validation. Thank you, Clemens. Hi, my name

  • is Christina and I am a software engineer on the Google brain

  • team.

  • I am here today to tell you about some tools my team and I

  • have built to help make the end-to-end lifecycle of the

  • machine learning pipeline easier. I am going to start by

  • talking about

  • model analysis and salad validation. These are two

  • different components in

  • TFX but they are similar in how they are executed. The main

  • difference is how you, as an end user, will use them.

  • I will start by talking about

  • the evaluator. Why is this important? We have gathered

  • data, cleaned that data, trained a model, but we really want to

  • make sure that model works and some model evaluation can help

  • you

  • assess the overall quality of your model.

  • You also may want to analyze how your model is performing on

  • specific Slices of the data. So, in this case with the Chicago

  • taxi example that Clemens started us off with, why are my

  • tip predictions sometimes wrong? Slicing the data and looking at

  • where

  • you are doing poorly can be a real benefit because it

  • identifies low-hanging fruit where you can get

  • gains in accuracy by adding more data,

  • or making some other changes to make some of these segments

  • improve. You also want to track your performance over time.

  • You are going to be continuously training models and updating

  • them with fresh data so your models don't get stale and you

  • want to make sure that your metrics are improving over time

  • and

  • not regressing and model evaluation with help with this.

  • The component of TFX that supports this

  • is called

  • evaluator.

  • You need to specify the splits in data

  • you find more interesting so the evaluateator

  • evaluator. A process is run to generate metric for the overall

  • slice and the Slices you have specified.

  • The output of the evaluator is evaluation metrics. This is a

  • structured data format that has your data, the splits you

  • specified,

  • and the metrics that correspond to each split.

  • The TensorFlow model analysis library also has a visualization

  • tool that allows you to load up these metrics and

  • dig around in your data in a user-friendly way.

  • So going back to our Chicago taxi example, you can see how

  • the model

  • evaluator can help you look at your top-line objective and how

  • well can you

  • predict trips that result in large tips.

  • The

  • TFMA visualization shows the overall data.

  • Maybe you want to stay 95% accuracy.

  • That is a lot better than 94.7. Maybe you want to bump that up.

  • Then you can dig into why your tip predictions are sometimes

  • wrong. We sliced the data by the hour of the day the trip starts

  • on and sorted by poor performance.

  • When I look at this data, I see trips that start at 2-3:00 a.m.

  • were performing quite poorly in these times. Because of the

  • statistics generation

  • tool Clemens talked about I know the

  • data is sparse here but if where didn't

  • I would think maybe there is something

  • that people that get taxies during that

  • time that causes erratic tipping behavior.

  • You also want to know if you can get

  • better at predicting trips over time. You are continuously

  • training the model for new data and hoping you get better. The

  • TensorFlow model analysis tool that powers the evaluator in TFX

  • can show you

  • the trends of your metrics over time. Here you see three

  • different models and

  • the performance over each with accuracy and AU C.

  • Now I will move on to talking about the model validator

  • component.

  • With the evaluator, you were an active

  • user, you generated the metrics, you loaded them up into the UI,

  • you dug around in your data, you looked for issues you could fix

  • to improve your

  • model, but eventually you are going to iterate, your data is

  • going to get better and model improve and you will be ready to

  • launch. You will have a pipeline continuously

  • feeding new data into this model. Every time you generate a

  • new model with new data you don't want to have to do a

  • manual process of pushing this to a server somewhere.

  • The model validator component of TFX acts as a gate that keeps

  • you from pushing bad versions of your model while

  • allowing you to auto

  • automate pushing.

  • We want to avoid pushing models with degrated quality

  • specifically in an automated fashion. If you train a model

  • with new data and the performance drops but say it

  • increases in certain segments of the data you care about, maybe

  • you make the judgment call this is an improvement call overall

  • so we will launch it.

  • But you don't want to do this automatically you want to have

  • say before you do this. This acts as your gatekeeper.

  • You want to avoid breaking downstream

  • components if your model started outputing something your server

  • binary couldn't handle you would want to know that before you

  • push. The TFX component that supports

  • this is called the model validator taking similar input

  • and output to the model

  • evaluator and the libraries that compute

  • the metrics are pretty much the same

  • under the hood, however, you provide two models.

  • It then runs on your eval split data and

  • compares the metrics on the same data between the two models. If

  • your metrics have stayed the same or improved, you go ahead

  • and bless the model. If the metrics that you care about have

  • degraded, you will not bless the model, get some information

  • about which metrics failed so you can do further analysis.

  • The outcome of this is a valididation outcome.

  • It just says blessed if everything went right.

  • Another thing to mote about the model

  • validator is it allows next day eval of your next day push.

  • Maybe the last model you blessed, it was trained with old

  • data. With the model evaluator, it evaluates it on the new data.

  • And finally, I am going to talk about the pusher.

  • It pusher is probably the simplest

  • component in the entire TFX pipeline but it does serve quite

  • a useful purpose.

  • It has one input which is that blessing

  • that you got from the model validator.

  • Then the output is if you passed your

  • validation then the pusher will copy

  • your saved model into a file system destination you have

  • specified. Now you are ready to serve your

  • model and make it useful to the world at-large.

  • I am going to talk about model deployment next. So this is

  • where we are. We have a trained saved model.

  • A saved model is a universal serialization format for

  • TensorFlow models.

  • It contains your graph, your learned

  • variable weights, your assets like embeddings and vocabulary,

  • but to you this is just an implementation detail. Where you

  • really want to be is you have

  • an API, you have a server that you can query to get answers in

  • real-time or

  • provide those answers to your users. We provide several

  • deployment options and many of them are going to be

  • discussed at other talks in this session.

  • TensorFlow.

  • js is optimized for serving in the browser.

  • TensorFlow Lite is optimized for mobile devices. We heard a talk

  • about how Google

  • Assistant is using TensorFlow Lite to support model inference

  • on their Google Home devices. TensorFlow Hub is something new.

  • André is going to come on in five minutes and tell you about

  • that so I am

  • not going to step on his toes. I am going at a talk about

  • TensorFlow Serving.

  • If you want to put up a resting API you would want to use

  • TensorFlow Serving. Why would you want to use this?

  • For one thing, TensorFlow Serving has a lot of

  • flexibility.

  • It supports multi tenancy.

  • You can run multiple models on a single server instance and run

  • multiple versions of the same model. This can be useful when

  • you are trying to canary a new model.

  • Say you have a tried and tested version

  • of your model, you have created a new

  • one, it past your evaluator and validation but you want to do

  • testing with real users before you completely switch over.

  • TensorFlow Serving supports this.

  • We also support optimization with GPU and TensorFlow RT.

  • You can expose a gRPC or REST API. TensorFlow Serving is

  • optimized for

  • high performance.

  • It provides low let latency and request

  • batching and traffic isolation so if you are serving multiple

  • models on a single server, a traffic spike in one of those

  • models won't affect the serving of the other.

  • And TensorFlow Serving is production ready, finally. This

  • is what we use to serve many of our models inside of Google.

  • We have served millions of QPS with it.

  • You can scale in minutes particularly if you use the

  • Docker image and scale up on Kubernetes.

  • We support dynamic version refresh so you can specify a

  • version refresh policy to take the latest version of your model

  • or you can pin to a specific version.

  • This can be useful for rollbacks if you find a problem with the

  • latest version

  • after you have already pushed. I am going to go into a little bit

  • more detail about how you might deploy a REST API for your

  • model.

  • We have two different options for doing this.

  • The first, the top command, is using Docker which we really

  • recommend. It requires a little bit of ramp up at the beginning,

  • but you will really save time in the long run by not having to

  • manage your environment and not having to manage your own

  • dependencies.

  • You can run locally on your own host but you have to do all that

  • stuff long term.

  • I will go into more detail on the Docker run command.

  • You start with Docker run, chose a port

  • you want to bind your API to, provide

  • the path to the saved model generated by your trainer and

  • hopefully it was pushed by the pusher, you provide the model

  • name, and you tell Docker to run the TensorFlow Serving binary.

  • Another advantage of using Docker is

  • that you can easily enable hardware

  • acceleration if you are running on a

  • host with a GPU and Nvidia host

  • installed you can run on an accelerated hardware.

  • If you need further optimization we

  • support tensor RT which is optimized for deep learning

  • inference. Your Chicago taxi example that we have been using

  • here probably wouldn't benefit from this but if you had an

  • image recognition model, ResNet, you

  • could get performance boosting and cost

  • savings by using TensorRT.

  • We provide a command line allowing you to convert this

  • model and a simple change to the command line and you are

  • running on accelerated GPU hardware TensorRT optimization.

  • To put it all together again, we introduced TensorFlow Extended

  • or TFX and showed you how the different components that TFX

  • consist of can work together to help you manage the

  • end-to-end lifecycle of your machine learning pipeline.

  • First, you have your data and we have tools to help you make

  • sense of that, process it, and prepare it for training. We then

  • support training your model and after you train your model we

  • provide tools that allow you to make sense of what you are

  • seeing, of what your model is doing and to make improvements,

  • also to make sure you don't regress. Then we have the pusher

  • that allows you to push to various deployment options

  • and make sure model available to serve users in the real world.

  • To get started with TensorFlow Extended, please visit us on

  • GitHub.

  • There is more documentation at tensorflow.org/TFX and some of

  • my teammates are running a workshop tomorrow and they would

  • love to see you there. You don't need to bring a laptop. We have

  • machines setup and ready to go

  • and you can get hands-on experience

  • using TensorFlow Extended. Thank you. Please welcome André who

  • will talk about TensorFlow Hub.

  • Thank you, Christina.

  • Could I get the notes on the presentation? Thank you. Hello.

  • My name is André and I am a software engineer in Zurich and

  • I am the

  • technical lead for the TensorFlow Hub project. It is a

  • library and project for using machine learning.

  • We Open Sourced it last year. Today I will give you an update

  • on the project. Let's start by talking about when you want to

  • use it. If you have problems collecting

  • enough

  • data to train models from scratch than this is for you.

  • Additionally, since it is so easy to reuse computations and

  • weights, it becomes possible to leverage features

  • without having to learn how to fit them into neural networks.

  • Images, text and videos are features you can use with a

  • single line of code. You might also have encountered

  • problems where codebases become really coupled

  • and experimentation becomes slower over time.

  • By defining an artifact that does not depend on code, hub

  • allows for more

  • maintainable systems similar to how libraries help software

  • engineers.

  • The main concept is this pre-trained

  • building blocks that we call modules. Typically we start by

  • training a large model with the right algorithm and data. From

  • this large model, we are interested in just a part which

  • is reusable.

  • This is typically a bottleneck layer or disrooted

  • representation.

  • Then we can package them into a saved

  • model that defines the commutation and

  • no longer depends on the original code.

  • You can share with web servers, cloud and artifact system. You

  • can bring the module back as a piece of a new model to solve a

  • new task.

  • Since the model is defined by

  • TensorFlow primitives you can fine tune

  • the waits weights and adjust them. For the last few months,

  • we have been making it easier to use. This concept of saving a

  • part of the model and loading it back is getting integrated right

  • in the core of TensorFlow.

  • We have added saveModel features that make it possible to share

  • more than just

  • signatures and with eager execution it

  • becomes even easier to select the part the network that gets

  • exports. Let's look at a few examples. In TensorFlow 2.

  • 0, we can load a module with hub.load. We can use tf.saved

  • model.

  • load if it is on the system. Once loaded, we can call it

  • right away.

  • Due to the new capabilites you can share any object composed of

  • TensorFlow primitives. Text features has two members. Call

  • which is a TFX function and embedding which is a TF

  • variable. We are really excited that we added

  • support for polymorphic functions on saved model. This

  • will provide a more natural

  • interface than we had before with signatures. Here we see an

  • image representation

  • module being loaded and being used in inference mode or used

  • during training mode where batch mode is on. When it is on, we

  • can control some of its parameters.

  • This is just backed by TF graphs so we no longer need to be

  • selecting things here and there.

  • We just get the API that looks very like Python.

  • Additionally, we have added the new symbol. Hub.keraslayer. This

  • makes integrating hub models into a keras model easier.

  • In this example, we see how easy it is

  • to build a text sentence -- a sentence classification model.

  • We have three layers.

  • The top model is this keras layer, is a

  • layer that receives inputs and outputs representation. Then we

  • have a dense layer and a classification layer.

  • Since the keras layer with this MLM layer, we can just feed

  • sentences straight into the model. We never have to define

  • the logic.

  • If we wanted to try other text models we

  • could just change that string.

  • That is the easiest way to try the new research.

  • We have released 0.3 version of TensorFlow Hub. We have hub.load

  • and hub.keraslayer and they are usable in TensorFlow 2.0 both in

  • eager and in graph mode.

  • To let you preview this

  • functionality, we have published modules in this format and the

  • next step for us is to backport existing models.

  • A bit more practical now.

  • Google AI and DeepMind teams have been

  • sharing information on tfhub.dev.

  • This was launched last year.

  • One of the most popular was the universal sentence encoder.

  • This encoded short sentences into a vector that could be used

  • for many natural language tasks.

  • The team added a cross-lingual version of these. Sentences with

  • similar meaning,

  • independent of the language, end up in points close together. You

  • can learn the classifier using English data and you could run

  • it on other languages.

  • We have also added image augmentation models.

  • They have been shown to transfer to new tasks. An interesting

  • thing is you could grab

  • this module and one of the image representation modules and you

  • could string them together. The image augmentation model would

  • reduce the amount of data by data augmentation and the image

  • feature

  • vector by transfer learning. There are many more.

  • We have BERT model for tasks, we have biggan models and more.

  • Some were designed especially for low resources.

  • Additionally, we have been working to make the modules more

  • integrated.

  • We have a common line utility to

  • convert the hub model into TF model.

  • It can be used with a library for ML

  • and they can also be used inside TF transform.

  • If you want to try it you can go to

  • tfhub.dev. Most of them include the link where you can see them

  • in action. Thank you.

  • Thank you very much. OK.

  • Now you have all learned hopefully training neural

  • networks that if I show up there is food.

  • So, we are going to come back here at 4:45 to do a little wrap

  • up. The wrap up session is going to be amazing not that this

  • wasn't also amazing. We will be talking about probablyability,

  • reinforcement learning, we will be talking about -- Elena is

  • coming back to tell us about her stuff. It is going to be great.

  • After that there is a party. Please stay with us. Come back.

  • Don't forget to go upstairs and check out the demos. Thank you

  • so much.

  • Aida net

  • et

  • AdaNet

  • da net

  • auto M L

  • test test test test

  • Extreme weather is changing.

  • There is more extreme rainfall,

  • heavy flooding, forest fires.

  • Being able to predict these extreme

  • events more accurately is the biggest changes we are facing.

  • There is 100 Tarabytes every day. Climate data is a big

  • problem. We need things that are fast and

  • can sift through the data accurately and rapidly.

  • Deep learning is poised for problems in climate science.

  • A lot of

  • NERSC are using TensorFlow. We use TensorFlow to layer and to

  • create the deep learning model, we started from segmentation

  • models which

  • have proven to be successful and we use

  • TensorFlow to enhance the models until

  • we found a set of models that perform well enough for the

  • specific task.

  • The network required 14 teraflops. If you want to do

  • this on your work

  • station, it would take months to train.

  • It takes the largest computational resources on the

  • planet. Two tennis courts in size, this computer is

  • state-of-the-art and a million times faster than your common

  • laptop. 3.3 exaflops.

  • Imagine what you do at your work station and imagine having

  • 27,000 times that power. We can do that now.

  • We were surprised how good it scales.

  • 2,000 nodes, 5,000 nodes.

  • This was the first time anyone has ran

  • an AI application at this scale.

  • The climate scientists could express

  • things in Python in TensorFlow and get all the code people are

  • used to.

  • We are entering a state where AI can

  • contribute to the predictions of these

  • extreme weather events. We can tackle things we never

  • thought. Understanding diseases like Alzheimer's and cancer.

  • That is like incredible. We have shown with a PRAM work

  • like

  • TensorFlow you can get to massive scale and accomplish

  • goals.

  • Genetics, neuroscience, physics -- that

  • is immensely exciting for me.

  • -- framework --

  • [Applause] Hello, everyone.

  • I am Josh Dillon and I am a lead on the TensorFlow Probability

  • team and today I will talk about probability stuff and how it

  • relates to TensorFlow stuff. Let's find out what that means.

  • OK.

  • These days machine learning often means specifying deep

  • model architectures and

  • fitting them under loss and keras makes specifying this

  • architecture easily but what about the loss?

  • Choosing the right loss is tough.

  • Improving even a reasonable one it be tougher.

  • Once you fit the model, how do you know it is good?

  • Why not use entropy or mode? Wouldn't it be nice if there was

  • a plug and play with keras and the rest of TF model?

  • This would make comparing models easier

  • by maximizing likelihood and having as

  • it SKT -- rapidly available statistics we can reject the bad

  • ones and accept the good ones.

  • Wouldn't it be great to say I want to maximize the log

  • likelihood and summarize what I learned easily and in a unified

  • way. Let's play with that idea. Here we have a dataset.

  • These blue dots and our task is to

  • predict, pretend task, is to predict the

  • Y coordinate from the X coordinate.

  • You might do this by specifying a deep

  • model and choosing the mean squared error as the loss

  • function.

  • But our wish here is to think probalistically and that means

  • maximizing the log likelihood like here with this Lambda

  • function. We want to get back a distribution, a thing that has

  • attached to it statistics we can use to evaluate what we just

  • learned.

  • If only such a thing were probably.

  • Of course, it is and you can do this now. Using TensorFlow

  • Probability

  • distribution layers you can specify the

  • model as part of your deep net. The loss now is actually part of

  • the model, sort of the way it used to be,

  • the way it is meant to be. We have two dense layers.

  • The second one outputs one float that

  • is parameterizing a normal distrubution

  • mean through this distribution Lambda layer. Being we are able

  • to find this line,

  • that looks great, and the best part is once we instantite this

  • model with test points we have back a distribution for

  • which you get not just the mean but intropy variance and

  • standard deviation and all these things.

  • You can compare between this and other

  • distributions as we will see later -- entropy. If we look at

  • the data, something is fishy here, right? Notice as the

  • magnitude of X increases,

  • the

  • variance of my increases also. What can we do to fix this?

  • Answer? Learn the variance too.

  • If we are fitting a normal why on earth

  • do we think the variance would be one? That is what you are

  • doing when you use mean squared error.

  • To achieve this, all I had to do was make my previous layer

  • output two floats. Passing one to the mean as normal, one

  • in as the standard deviation of the normal, and press no change

  • and now I

  • have learned the standard deviation from the data itself.

  • That is what the green lines are.

  • This is cool because if you are a

  • statistician, it means you are learning known unknowns.

  • The data had variance and you learned

  • it and it cost you basically nothing to do but a few key

  • strokes. The way you saw to do this was self evident from the

  • fact you were using a normal distribution for a curious

  • constant sitting there.

  • This is good but I don't know is there enough data for which we

  • can claim this

  • red line is the mean and the green lines

  • are the standard deviation? How would we know? Is there anything

  • else we can do? Of course there is. A keras dense layer has two

  • components. A kernel matrix and a bias vector.

  • What makes you think those point estimates are the best

  • especially given your dataset itself is random and

  • possibly inadequate to meaningfully and reliable learn

  • those estimates. You can learn a distribution over

  • weight, this is the same as learning an

  • ensemble that is

  • infinitely large.

  • As you can see, all I had to do was replace keras.

  • dense with with dense variation.

  • The keyword here -- I like to think of the unknowns unknowns.

  • As a consequence, of course,this means

  • any model or instantiation is a random variable because the

  • weight is a random available. That is why you see all the

  • lines, there

  • are many lines, we have an ensemble of them but if you

  • averaged them and took the standard deviation that would

  • give you an estimate of credible intervals over your prediction

  • and you can go to

  • your customer and say here is what I think will happen and

  • this is how much you can trust me.

  • We lost the head

  • -- I just output two floats and feed that into my output layer

  • and press no change. I am learning both known and unknown

  • knowns and all it cost me was a few key strokes.

  • So what you see here is an ensemble of

  • standard deviations associated with the known-unknown parts,

  • the variances observable in the Y axis, as well as a

  • number or ensemble of these mean regressions. OK. That's cool. I

  • like where this is going but I have to ask what makes you think

  • a line is even the right thing to fit here? Is there another

  • distribution we could chose? A richer distribution that would

  • actually find the right form of the data?

  • Of course the answer is yes.

  • It is a

  • Galsian process. The

  • Gaussian process can see this. How can you do that if you are

  • specifying mean squared error as your loss? You can't.

  • It has to be part of the model.

  • When you bake these ideas into one model

  • you get to move it around fluidly between weight

  • uncertainty and variance in the data and even uncertainty in the

  • loss function you are fitting itself.

  • So the question is, how can this all be so easy?

  • How did it fit together? With TensorFlow Probability.

  • TensorFlow Probability is a collection

  • of tools designed to make probalistic reasoning in

  • TensorFlow easier. It is not going to make your job easy. It

  • is just going to give you the tools you need to express the

  • ideas you have. You still have to have domain knowledge

  • and expertise but you can encode that in a

  • probalistic

  • formulism and TensorFlow Hub has the way to do that.

  • Gone are the days of hacking a model in

  • R and importing it to a faster language. If you predict the

  • light is green, you

  • better be confident you should go.

  • You can do that with probalistic modeling and TensorFlow

  • Probability.

  • We saw one small part of TFP. The tools are broken into two

  • components.

  • Those tools use FWL -- for building

  • models and those useful for do analyzing.

  • It is summary statistics like it is in other libraries.

  • Our distributions support batch shape but for the most part they

  • should be pretty natural and easy to use. We also have

  • something we call bijectors which is a library for

  • transforming random variables. This can be like taking the X of

  • a normal and now you have a log normal.

  • In comply

  • -- complicated cases it is like

  • mascots, regressive flows, MVPs and

  • other

  • sophisticated probalistic models. Edward2 helps you

  • combine different variables as one.

  • On the inference side, no library would be compete without

  • Monte Carlo tools.

  • We have transitions components one is

  • called Markov chain Monte Carlo and variational inference tools.

  • And of course we have our own optimizers that come up.

  • The point is this toolbox has maybe not everything but

  • certainly it has most of

  • what you might need to do fancier modeling to get more. It

  • doesn't have to be hard.

  • You saw the keras examples were sequences of one-line changes.

  • Of course, TensorFlow Probability is used widely

  • around alphabet. DeepMind and Google brain and Google

  • accelerated sciences, product areas, infrastructure areas use

  • it for planning

  • purposes, but it is also used outside of Google.

  • Baker

  • Hughes uses this to predict anomalies.

  • For example, anyone who flew out here

  • would be happy to know that bhge uses their prediction to predict

  • the lifespan of jet engines.

  • If we had a dataset using a failing engine that would be a

  • tragedy. Using math they get around this

  • by

  • modeling models and then trying to in

  • the abstract figure out if they are going to be good models.

  • The orange boxes use TensorFlow Probability extensively.

  • The orange border box is where they use TensorFlow.

  • The basic flow is to try to treat the model itself as a

  • random available and determine if it will be a good model on an

  • otherwise incomplete dataset.

  • From this they get remarkable results

  • and dramatic increases in false positives and false negatives

  • over large

  • datasets in complicated systems. So the question is who will be

  • the next success story? Try it out. It is an open source Python

  • package built using TensorFlow that makes it

  • easy to combine deep learning with probalistic models.

  • You can pip install it. If you are interested in

  • learning more

  • about the

  • bayessian methods, check out thus this book.

  • If you are not a bayesian, that is OK too. If you are doing

  • linear aggression, it could solve the problem on the order

  • of 30 iterations which can't be said of standard gradient

  • dissent.

  • If you want to find out more, you can

  • check out our GitHub repository where you can find several

  • Jupyter notebooks. Thanks. [Applause]

  • And with that, I would like to hand it

  • off to Eugene and Sergio who will be

  • talking about reinforcement learning.

  • Thank you.

  • Hello, everyone. How is it going? Very packed day? Lots of

  • good stuff? Are you ready for more? Or not? OK. Good. So my

  • name is Sergio Guadarrama and I am a software engineer at Google

  • brain

  • and I work in the TF agent steam.

  • And I am Eugene Brevdo working on reinforcement learning.

  • How many of you remember how you learned how to walk?

  • You stumble a little bit, you try one step, doesn't work, you

  • lose your balance, you try again.

  • When you try to learn something hard practice you need to try

  • many times. This is moving the legs and trying to

  • learn how to walk.

  • After learning to try multiple times, in

  • this case is 1,000 times, it falls. It will train a little

  • longer and then

  • it is able to walk around and go from one place to another and

  • find their way around the room.

  • You have probably heard about all the

  • applications including recommended

  • assistance, data --

  • real robots like that, chemistry and math and always alpha go

  • that play Go better than any human. Now, I have a question

  • for you.

  • How many of you have tried to auto implement an algorithm? I

  • see quite a bit of hands. Very good. It is hard.

  • Yeah. We went through that pain too. You know, many people who

  • try get the first prototype right away. It seems to be

  • working but then you miss a lot of different pieces.

  • All the details have to be right, everything.

  • There is a lot of pieces and a lot of things you need to do. We

  • suffer through the same problem at

  • Google and decided to implement a library that many people can

  • use and we are happy to announce it is available online. You can

  • go to GitHub and pip install it

  • and start using it right away. Hopefully, you will provide

  • feedback contributions so we can make it better over time.

  • Now, where is

  • TF age NLT

  • -- agents? For people new, we have Colabs

  • and things with documentation and samples so you can learn

  • about it. For people that want to solve the

  • complex problems, we have already ways to take their

  • algorithm and apply it. For people who are researching

  • and want

  • to develop new algorithm

  • they will not need to build all the pieces. They can build on

  • top of it.

  • You can start doing experiments right away.

  • We build on top of the goodies in TensorFlow 2.0 you saw today.

  • Tf.keras to build the networks and tf.functions when you want

  • thinks to go faster and we make it mobile and

  • extensible so you can cherrypick the pieces you need. For those

  • who are not ready for the

  • change yet, we make it compatible with TensorFlow 1.14.

  • We go back to the sample of the little robot trying to work.

  • This is how the call looks like. You have it define networks.

  • An actor distribution network and a criteriaic and actor.

  • Assuming we have data collected we can just train through it.

  • TF-agents provide a lot of environments already.

  • We also provide a state-of-the-art

  • algorithms including dqn, and many others.

  • There are more coming soon. They are fully tested with quality

  • version tests and a speed test to make things keep working. As

  • an overview of the system, it looks like that. On the left

  • side you have all the collection aspects and we will have policy

  • that will interact with environments and collects on

  • experience. We probably put in some replay buffer to work later

  • and on the right side we have all the training pipeline. We

  • are going to write from this experience and our agent is

  • going to

  • learn to improve the policy by training a neural network. Let's

  • focus for a little bit on this environment.

  • How we define a problem and new task. Let's take another

  • example. In this case it is breakout. The idea is you have

  • to play this game move the power left and right and try to break

  • the bricks on the top. You break the bricks you get rewards and

  • the points go up.

  • You let the ball drop and the points go down.

  • The agent refers multiple frames and it will decide which action

  • to take and based on that it will get reward and just loop

  • again. How this look into the code is something like this.

  • You define the specific specification.

  • What kind of observation this environment provides. Any other

  • information. Multiple things.

  • Then the action speckspecs. In this case left and right but in

  • many environments we have multiple actions. And a reset

  • method because we will play the game a lot and have to reset the

  • environment and a step method that taking an action will

  • provide a new observation and give us a reward.

  • Given that we define the policy, create the environment, reset

  • the environment and start looking over it and start

  • playing the game. Your policy is very good you will get a good

  • score.

  • To make the learning even faster, we make these parallel

  • environments and you can run these games in parallel multiple

  • times and wrap in TensorFlow so it will

  • go even faster and then do the same loop again. What happened

  • in general is like we

  • don't want to define this policy by hand.

  • Let me hand it over to Eugene who is going to explain how to

  • learn those policies. Thank you, Sergio.

  • Yeah, as Sergio said, I am giving an example of how you

  • would interact with the environment by a policy. We are

  • going to go into the a little bit more detail and talk about

  • how to

  • train policies to maximize the rewards.

  • So kind of going over it again,

  • policies take observations and emit parameters of a -- a

  • distribution over the actions.

  • In this case, the

  • observations are an image, there is an underlying neural network

  • that converts those images to the parameter of the

  • distribution and the policy emits that distribution or you

  • might sample from it to actually take actions.

  • So, let's talk about networks.

  • I think you have seen some variation of this slide over and

  • over again today.

  • A network in this case network used for deep Q learning is

  • essentially a container for a bunch of keras layers.

  • In this case, your inputs go through a convolutional layer

  • and so on and so

  • forth and the final layer emits logits over the actions you

  • might take. The core method of the network is the call. It

  • takes observations in a state,

  • possibly an RNN state, and emits the logits and you update the

  • state. OK. So let's talk about policies. First of all, we

  • provide a large number of policies. Some of them

  • specifically tailored to

  • particular algorithms and particular

  • agents but you can also implement your own. It is useful

  • to go through that.

  • A policy takes one or more networks.

  • The fundamental method on a policy is the distribution

  • method.

  • This takes the time that is essentially

  • containing the observations, passes that through one or more

  • networks and emits the parameters of the output

  • distribution in this case logits.

  • It then returns a tuple of three things.

  • The first is an actual distribution object. Josh spoke

  • about the probability and here is this low probability category

  • called distribution built from the logits, it emits the next

  • state possibly containing RNN state information, and it also

  • emits site information.

  • Site information is useful -- side information is useful

  • perhaps you want to emit information you want to login

  • your metrics and that is not the action, or maybe you want to log

  • some information that is necessarily for training later

  • on so the agent is going

  • to use that information to actually train. OK.

  • So now let's actually talk about training.

  • The agent class encompasses the main ML algorithm and that

  • includes the training

  • and reading batches of data and trajectries to update the neural

  • network. Here is a simple example. First, you create a

  • deep queue learning

  • agent, you give it a network, you can access a policy

  • specifically a collection policy from that agent, that policy

  • uses the underlying network that

  • you passed in, and maybe performs some

  • additional work like maybe performs exploration. It also

  • logs site information that is going to be necessary to be able

  • to train the agent.

  • The main method on the agent is called train.

  • It takes experience in the form of

  • batch

  • trajectories from a replay buffer. Assuming you have

  • trained the data and

  • performed well you might want to take a policy that provides more

  • greedy actions and just exploits and takes the best actions it

  • thinks are the best and

  • doesn't log any side information. That is the plan

  • and policy.

  • You can save this to a saved model and put it into the plain.

  • For a complete example, here we have a

  • deep Q learning network.

  • It has other arguments describing what kind of keras

  • layers to combine. You build the agent with that

  • network. Then you get a tf.data dataset.

  • In this case, you get it from a replay buffer object, but you

  • can get it from

  • any other dataset that emits the correct batch trajectory

  • information and you

  • iterate over that dataset calling agent.train to update

  • the underlying neural networks that are updated in the

  • reflected policies. Let's talk a little bit about collection.

  • Given a collection policy, it doesn't

  • have to be trained, it can have random parameters, you want to

  • be able to collect data and we provide a number of tools for

  • that, again, if your environment is something that is in Python

  • you can wrap it.

  • The core tool for this is the driver. Going through that first

  • you create your batched environments at the top. Then

  • you create a replay buffer. In this case we have a tf.uniform

  • replay buffer. This is backed by TensorFlow variables. Then you

  • create the driver.

  • The driver accepts the environment and the collect

  • policy from the agent and a number of callbacks. When you

  • call driver.run what it will do is it will iterate,

  • in this case it will take 100 steps

  • between a policy and environment, it

  • will create trajecto trajectories.

  • After the buffer finished, you have 100 more frames of data.

  • Here is kind of the complete picture.

  • Again, you create your environment, you interact with

  • that environment to the

  • driver given a policy, those

  • interactions get stored in the replay buffer, the replay buffer

  • you read from with a tf.

  • data dataset and then the agent trains

  • with batches from that dataset and

  • updates the network underlying the policy.

  • Here is kind of the set of commands to do that.

  • If you look at the bottom, here is that

  • loop, you call driver run to collect data, it stores data in

  • the replay buffer, and then you read from the dataset generated

  • from that replay

  • buffer and train the agent and you can iterate this over and

  • over again. We have a lot of exciting things

  • coming up. We have a number of new agents we are going to

  • release. C51 and so on.

  • We are adding support for contextual bandits, we are going

  • to release a

  • number of baselines and as well as a number of new replay

  • buffers.

  • In particular, we are going to be

  • releasing some distributed replay buffers, in the next

  • couple of quarters, and those will be used for distributed

  • collection. Distributed collection allows

  • you to parallelize your data collection across many machines

  • and be able to maximize

  • the throughput of your training URL algorithm that way.

  • We are working on distributed training also using TensorFlow

  • new distribution

  • strategy API allowing you to train at a massive scale on many

  • GPUs and TPUs and

  • we are adding support for more environments.

  • Please check out TF-agents on GitHub.

  • We have a number of Colabs, I think 8-9,

  • exploring different parts of the system.

  • As Sergio said, TF-agents is built to

  • solve many real-world problems and in particular we are

  • interested in seeing your problems. For example, we

  • welcome contributions

  • for new environments, new RL algorithms

  • for those of you out there who are RL experts. Please come chat

  • with me or Sergio

  • after the talks or file an issue on the GitHub issue tracker.

  • And let us know what you think. Thank you very much.

  • Thank you. [Applause]

  • And now I would like to pass it off to

  • Nick and Yannick who will talk about TensorFlow.js.

  • Thanks, guys.

  • Hey, everybody. My name is Nick and this is my colleague

  • Yannick. We are going to talk about JavaScript and ML.

  • TensorFlow.js is a library we launched last year. It is a

  • library for training and deploying ML models in the

  • browser and on node.js. We want to showcase what you can do

  • today with a platform and where we are going. One of the great

  • parts about the library is there is really no drivers to install.

  • If you run it in the browser you can get out of the box GPU

  • acceleration.

  • The browser itself tends to be interactive and builds great

  • applications and demos for using ML.

  • Privacy is a very important part to the library. You can run

  • inference and training locally on the client which works around

  • all sorts of privacy issues you might

  • have with doing server side inference or training.

  • And what can you do today with the library? We have a

  • collection of, like, pre-trained off the shelf models you can use

  • without any knowledge of ML. We also have the ability to take

  • existing Python models and convert them and run them in

  • TensorFlow.js.

  • We also have a full-stack for training, inference and low

  • level linear algebra

  • and that runs in a browser in node.js.

  • We also have a bunch of platforms that JavaScript can

  • run on outside of just the browser.

  • The first thing I want to showcase is

  • some of our new off the shelf models we have launched.

  • The first one is a bunch of pre-trained -- or off the shelf

  • models are a bunch of pre-trained models.

  • They are image, audio and text classification models. The APIs

  • are all user-friendly.

  • You don't have to worry about

  • converting images to tensors.

  • They are available on MPM and we have precompiled hosted scripts

  • a well. We are working on this a lot in the upcoming year and we

  • will have pore and more models as we go forward.

  • The first model was body pics. I want to show you how easy it is

  • to

  • use this model.

  • First include the library and body pics. It is two simple

  • imports.

  • The next step is to create an image

  • DOM. This is Frank. He is trying to do something on the couch but

  • I want to actually load the model and find body parts on

  • Frank. The first thing to do is to load

  • the body pics model. Just a simple one-line call. The next

  • step is to call one of the methods we expose which is

  • estimate

  • person segmentation and I can pass in a

  • DOM element. There is also a bunch of really

  • easy to use methods for doing filtering on the image. So I can

  • take the results of that and

  • render it directly on there DOM. It shows head, body, arm and so

  • on. Another model we launched a couple

  • weeks ago is the toxicity model.

  • To use this model, we will use the prehosted scripts.

  • Two lines of code.

  • I will load the toxicity model and ask

  • the model to classify just some really

  • lovely text, pretty pg, you suck, and I will get a result

  • back as a JavaScript

  • object that has seven labels we identify

  • different types of toxic-type text and the probabilities and

  • if it matches. We also have the ability to take

  • pre-trained Python models for run them directly in browsers.

  • If you have a pre-trained model trained in Python world we have

  • a command line tool that makes it easy to serialize the model

  • as a JSON object and the weights and distribute them in a web

  • format.

  • We support saveModel, TensorFlow Hub and keras models.

  • It supports over 170 and counting ops and we will be

  • TensorFlow 2.0 compatible. I want to walkthrough how simple

  • it is to use this. I is a Python model and run it through the

  • command line tool and I can easily load that in my

  • JavaScript application. Very simple.

  • I want to hand it off to Yannick to

  • walk through the pre-trained model.

  • TensorFlow.js allows you to train in the browser and in

  • node. The primary tool for this is the layers

  • API which is a keras-compatible API for authoring models.

  • We are going to take a quick look at

  • what TF.js code looks like.

  • The first step to is import the library and when working in

  • node.js you can also use the node.js bindings which execute

  • the TensorFlow

  • operations using native compile C++ code.

  • If you are an a system that supports

  • kuda, you can import -- cuda --

  • you can -- this is what creating a convolutional model for a

  • classification task looks like.

  • It is similar to keras code in Python.

  • We start by instantiating a model and add a flat operation

  • and a dense layer with output classes.

  • Similar to Python we use model. compile and specify loss

  • function and optimizer and model.fit is the function that

  • drives the train loop.

  • In JavaScript it is an a sync function

  • so we want to wait for the result or to be done.

  • Once it is done training we can save the model.

  • We support saving to a number of targets both on the client and

  • server. And you can use model.

  • predict to get a result from the model, finally. Over the past

  • year, we have also heard feedback from the community on

  • ways we

  • can improve the experience of training with TensorFlow.js. We

  • would like to show you progress we have made.

  • First up it is tf.data.

  • It is an an API for managing data training. It provides a

  • whole set of utility functions for dataset transformation.

  • Finally, it works with streams and the lazy evaluation allows

  • you to work with

  • data that doesn't fit in memory which can be quite important.

  • Let's look at a simple example.

  • We load up a

  • CSV file and use the is label attribute.

  • For example, in this map

  • transformation, the price data has been separated out into that

  • Y object and the rest of the features are in the X object.

  • Once we flattened the data we can apply typical ML

  • transformations including

  • things like shuffling, an ML best practice, and batting which

  • will do the

  • work of creating properly sized mini batches for training and

  • know what to pull into memory when the training loop is

  • running.

  • Other things include things like normalization.

  • Finally we run the train loop.

  • Model.fitdataset supports models and pulls the right stuff into

  • memory as needed. And that's tf.data.

  • So the other area we have been

  • responding to community feedback is visualization.

  • I want to talk about tfjs-vis which is

  • model of internals as well as evaluation

  • metricsism you can view internals and evaluation

  • metrics. First we import the library and you

  • should node we provide tfjs-vis as a separate package.

  • To visualize training behavior we can

  • use this showshow.fitscallbacks function.

  • In one line, fit callbacks will plot the selected metrics.

  • In this case loss and accuracy metrics

  • on the batch end and at the end of each epoch.

  • This allows us to adjust hyperparameters as usual.

  • You can look at model internals with show.

  • model summary and show.layer summary. Here we see the

  • architecture of the

  • model including output shapes of SARSAR -- various layers.

  • We see the distribution of layers in the first

  • convolutional layers including

  • statistics like nans and zero counts which are useful for

  • debugging models.

  • We want to finally announce TensorBoard support in node.js.

  • You can monitor when using the TensorBoard layer in node.js. A

  • single line will generate the

  • necessary callbacks to write the methods

  • to a TensorBoard log file using this command. Then you can open

  • in TensorBoard and look at how your training is going just like

  • you may be used to. With that, I am going to hand it back to Nick

  • to talk about some of the platforms we execute on.

  • JavaScript is an interesting language because it runs in a

  • lot more places than you think. There is the traditional browser

  • front for running JavaScript in the browser. We all know about

  • that. Node.

  • js is a big server-side solution. Very popular. But

  • there is also a growing trend with JavaScript in more places.

  • One of them is desktop applications and

  • electron is a very popular platform for developing

  • applications.

  • Those who have used Spotify or visual code those are good

  • examples of electron.

  • JavaScript is moving into the mobile space also. I want to

  • highlight a couple examples

  • we have seen in the industry on all four platforms.

  • First is the browser.

  • Our friends at Google creative labs

  • have built experiments on how creative tools could be more

  • accessible to everyone. There is going to be a great lightning

  • talk on this and they will talk about everything they have built

  • with this project.

  • Uber has built an in-browser tool for

  • model agnostic visualization of ML performance. They use

  • TensorFlow.

  • js for acceleration of their linear algebra. K means

  • clustering and so on. They are also giving a great lightning

  • talk about how they use TensorFlow.

  • js to solve this problem for their platform. This is all in

  • browser, again.

  • Another cool industry example is Airbnb. Airbnb built an identity

  • document

  • detection model that they use as a full TensorFlow ecosystem

  • solution. On our Airbnb profile, if you were to

  • upload a government-issued ID it is a very big trust issue. They

  • built a TensorFlow model to detect if a profile picture that

  • you are trying to upload directly in the client contains

  • government-issued IDs. They use this in the browser using

  • TensorFlow.

  • js as well as on their mobile devices for TFLite.

  • On node.js, a good example of this being used is clinic

  • doctor. This is a node.js performance analysis tool. They

  • use our node.

  • js bindings to filter out gc spikes on

  • node processes that are running to give a true accurate CPU

  • performance benchmark.

  • On the desktop, our team here at Google

  • with magenta and their music generation

  • models, have built a series of desktop plugins for the Ableton

  • studio. These are full desktop applications that use the

  • magenta models and

  • accelerate them all into the GPU and desktop applications.

  • We have a demo at your booth for how this works.

  • The cool part is all JavaScript and GPU

  • acceleration on the desktop with no cuda drivers all through the

  • findings.

  • Mobile is another interesting space.

  • WeChat, for example, is one of the most popular apps in the

  • world with over 1

  • billion total users and a subapplication platform called

  • the mini programs. They are great because it is no-need

  • install app in advance used on demand

  • and has over 1 million apps and 1.5 million developers.

  • The mini program is built using JavaScript and makes it easy for

  • developers to create, deploy and share on the WeChat platform.

  • I want to show a demo of a tfjs models running on WeChat.

  • I have just a regular iOS device and I will open WeChat.

  • Someone shared this tfjs example and I can load up the

  • application.

  • It is running our post net model and if

  • I aim it at Yannick here, there we go,

  • this is just purely done in JavaScript

  • and it is running our off the shelf mobile model doing about

  • 30 frames a second.

  • This is all done with the WeChat JavaScript platform.

  • platform.

  • Thank you.

  • So, all this work over the past year and the fantastic projects

  • created by the

  • community, makes us very excited to announce TensorFlow.js 1.0

  • today. It is available now and we are super excited to see what

  • the community builds

  • with it and hopes the API stability makes it easier going

  • forward.

  • We are focused on providing a stable API you can build

  • applications on and manage upgrade easier and bringing

  • market improvements in performance particularly on

  • mobile devices. We will look at that in a bit more detail. To

  • look at the update closer, since announcng TensorFlow.js last

  • year at the Dev Summit we have been working on improvements

  • across a number of platforms.

  • We see increases of about 1.

  • 4X to 9X in some extreme cases.

  • This chart shows inference performance

  • with a batch size of 1 on mobile net in Chrome.

  • It is a mobile image classification and

  • we see inference times going from about

  • 15 milliseconds to about 150 milliseconds on the pixel 2. We

  • have done performance on the iOS devices as well so really

  • excited for you to try this. What is next for us? Well, we

  • want to enable you to execute saved models on our node.js back

  • end without going through the conversion process and this will

  • open up

  • many more models to serve using the js.

  • node stack.

  • We are always keeping an eye on browser

  • acceleration proposals like simd and WASM.

  • The browser is only going to get faster and so will we.

  • We want to expand the platforms on which TensorFlow.js can run.

  • We are working on platforms like the

  • Raspberry Pi and other hybrid mobile platforms that run

  • JavaScript. So thanks and for more

  • information on

  • the things we talked about, you can

  • visit any one of these links. Thank you.

  • And last up is Elena to talk about her

  • work using machine translation in the Vatican secret archive.

  • Thank you, Nick and Yanni.

  • It is great to be here. Just one second.

  • I am excited to be here and to talk to

  • you about the In Codice Ratio project that is going on at Roma

  • university. To talk to you about TensorFlow help us

  • build a model that is able to

  • transcribe ancient manuscripts.

  • You can see the polyographers and archivist and on the left is

  • the data science team. That is why I think the name, In Codice

  • Ratio, reflects us well.

  • In Latin, it means a known edge for

  • transcript

  • but in Italian it means software code so it is knowledge through

  • software.

  • You might ask yourselvesyourselves,

  • what brings polyographer and archivist

  • and data scientists together.

  • They both want to

  • discover knowledge from big data.

  • Historical archives are an endless

  • source of important cultural and historical information. Just to

  • give you a scale of how large this information can be, let's

  • compare

  • for a second the sides of the Vatican

  • secret archive to the height of Mt. Everest. If you were to take

  • each shelve of the

  • Vatican secret archive, you would get to 85 kilometers tall.

  • That is about 10 times the size of Mt. Everest and the content

  • spans centuries and continents. There you have examples of

  • letters

  • coming from China, from Europe, from

  • Africa, and of course from the Americas.

  • What is our goal?

  • To build tools and technology that enables in general to

  • perform large scale analysis on historical archives because

  • right now the process is entirely manually. You have to

  • go there, consult the documents manually and be able to read the

  • challenging handwriting and if you find information that may be

  • linked to

  • another collection, you have to do it all by yourself.

  • But first, we have to face the very first challenge and that is

  • when you are dealing with web content, for example, if you

  • want to extract data from the internet, that is already text.

  • We said we are dealing with historical documents and that is

  • often scans. Traditionally OCR is fine for

  • printed text but then you get to this.

  • This is a medieval handwriting and a language nobody uses

  • anymore.

  • It is a handwriting nobody is able to write or read anymore

  • for that matter.

  • It is heavily aggregated and you want to get text out of it. You

  • might want to train a machine learning model but then we come

  • to the second challenge and that is scalability in the dataset

  • collection process.

  • The graph you see there is an algorithmic scale and it might

  • show you something you know that it is known as

  • the zip flow that tells you that there

  • is very few words and most of the words do not occur that

  • often. What does that mean for us?

  • If we want to collect data at vocabulary or word level, we

  • have to

  • annotate thousands of lines of text

  • which means thousands of -- hundreds of pages. Similar

  • systems do exist. There are state-of-the-art systems but

  • most of the polyographer even when they know of the tools get

  • discouraged using them because they say it is not

  • cost-effective because it can

  • take up to months or even years on these

  • documents to get a transcription service they will use once or

  • twice whereas they would like to do it faster. We asked ourselves

  • how can we scale on this task?

  • We decided to go by easier steps, simpler steps.

  • The very first things we did was to select data for single

  • characters.

  • This enabled us not to involve polyographer but people are less

  • experience.

  • We built a custom crowd sourcing

  • platform that worked like capcha solving.

  • You see an actual screen there. The workers were presented with

  • an image and target and had to match the target and select the

  • areas inside the image.

  • In this way, we were able to involve more than 500 high

  • school students and in about two week's work we made more than

  • 40,000 annotations.

  • Now we had the data and we wanted to build a model.

  • When I started working at the project, I

  • was pretty much a beginner in machine learning and so

  • TensorFlow helped me put

  • in practice what I was studying in thereytheory. It was a great

  • help that I could rely

  • on tutorials and the community and where everything else failed

  • even the source code.

  • So, we started experimenting and we decided to start small first.

  • We didn't want to overkill.

  • We wanted the model to fit exactly our data.

  • We started small and proceeded incrementally and in a constant

  • cycle of tuning hyperparameters and the best optimizer and

  • initializers and the number of layer and the type of layers

  • and then evaluaing and training again. Then we used keras which

  • allowed us to keep the code spall and readable. This is what

  • we settled for.

  • It might look trivial but it allowed us

  • to get up to 94% average accuracy.

  • Where does this fit in the whole scheme

  • of the

  • transcription center? We have the input image.

  • We are relying on over segmentation and that is a bit

  • old school but it allows us to feed the single characters or

  • combinations of characters inside of the

  • classifier which then produces different transcription

  • according to a latent

  • language model which we also built from publically available

  • sources. How good do we get?

  • We get about 65% exact transcription

  • and we can get up to 80% if we consider minor spelling errors

  • or if the segmentation is perfect.

  • If we had perfect segmentation we could get up to 80%.

  • We can see it can be more challenging. What are plans for

  • the future? We are very excited about the

  • integration

  • integration of TensorFlow and keras.

  • What we found out was that sometimes some were lagging

  • behind and sometimes

  • we wanted to get to one part of the features or from keras or

  • TensorFlow and we found ourselves doing lots of -- I

  • don't know if that is your experience as well but we found

  • ourselves doing back and forth between TensorFlow and keras

  • and now we get the best of the two worlds and are very excited

  • about that. How do we plan to expand our machine learning

  • system?

  • First thing first, we are trying

  • units that achieved good results on medical imaging.

  • We are planning to use them to get rid

  • of the segmentation and achieve the result of classifying

  • together. Those examples worked particularly well.

  • Of course, since there could be ambiguitywe could do error

  • correction

  • and

  • transcripttranscription. We want to evolve and move to word

  • level and even sentence level annotated

  • characters, but still our focus is

  • calbility and we want to involve polyographers as little as

  • possible.

  • This is generated input from GAN but we plan to use an encoder to

  • evolve the

  • dataset with little human interaction, like the less we

  • can.

  • And in the end this brings us to use sequence model that will

  • take full

  • advantage of the sentence level context, for example, and could

  • be able to solve

  • things that we could not be able to

  • solve with single, let's say, with single character

  • classification, for example abbreviation.

  • Many words occur aggregated in this

  • text just like you would text saying me

  • too and use 2 the number or for you and

  • that is the same with this kind of manuscripts. That is one

  • application you could have.

  • We are planning to use sequence models to get to a neural model

  • because so far we have only experimented with statistics.

  • And one last thing, I mentioned the people in the team but there

  • is so many people I would like to thank that were not in those

  • slides.

  • First of all Simone who should have

  • been here but he couldn't make it.

  • And then py school of AI and Luke for

  • their amazing mentoring and a high school teacher that

  • actually allowed us

  • to involve those students that work for the platform.

  • And of course, all the graduate and undergraduate students that

  • worked with

  • us and helped us achieve what we have achieved and what we plan

  • to achieve in the future.

  • And of course, that you can -- thank you for your attention.

  • [Applause] Thank you, Elena. All right.

  • This is the end of day 1. We hope you had a great time.

  • A big thank you to our speakers, our staff, our crew, and to the

  • audience

  • here in Sunnyvale and all around the world. With that, let's take

  • the livestream

  • out with a look at some highlights.

  • Hello, everybody.

  • Right now, right now, before you leave, there is a survey.

  • Please, it has been emailed to you right now. Please do the

  • survey. This is machine learning. We need data. No,

  • seriously, like we can't bring

  • these events if you can't find out what you think of them.

  • Please, please fill out the survey. Next we have a social

  • night. Tomorrow we have talks, breakout

  • sessions, we have coding, we have workshops.

  • Doors open at 8:00 A.M.

  • and lightning talks start at 9:50.

  • We will

  • have breakfast and start lightning talks. We have

  • breakout on performance, mobile and 2.0 and we have an amazing

  • series of research talks.

  • Please, please, we will see you tomorrow.

  • Go have a drink.

  • [Applause]

some examples.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it