Subtitles section Play video Print subtitles some examples. The first thing you need is data. You may want to validate results or test ideas on a common public dataset. It includes a large and rapidly growing collection of datasets you can get started with easily and combined with tf.data it is simple to wrap your own data too. Here is a small sample of the datasets available and all of these and many nor are included there. Then with Keras, you can express the models just like you are used to thinking about it. Standard package is fit with model fit and evaluate as well. Since deep learning models are often commutationally expensive you way want to try scaling this across more than one device. Starting from a pre-trained model or component also works well to reduce some of this computational cost. TensorFlow helps provide a large collection of pretained components you can include in your model and Feinstein -- fine tune for your dataset. Keras comes with everything you might need for a typical training job. Sometimes you need a bit more control. For example when you are exploring new kinds of algorithms. Let's say you wanted to build a custom encoder for machine translation, here is how you could do this by subclassing the model. You can even customize it training loop to get full control over the gradients and optimization process. While training models, whether packaged with Keras or more complex ones, it is often valuable to understand the progress and even analyze the muddle in detail. TensorFlow board provides a lot of visualization to help with chis and comes full integration with Colab and other Jupyter notebooks allowing you to see the same visuals. All of these features are available in TensorFlow 2.0 and I am really excited to announce our alpha release is available for you as of today. [Applause] Many of you in the room and across the world really helped with lots of work to make this possible. I would really like to take this moment to thank you you all. Please give yourself a round of applause. We really couldn't have done this without you. In addition to all the great improvements we talked about, this release comes with a converter script and compatibility module to give you access to the 1.X APIs. We are working for a full release over the next quarter. There is a lot of work going on to make TensorFlow 2.0 work well for you. You can track the progress and provide feedback on the TensorFlow GitHub projects page. You asked for better documentation, and we have worked to streamline our docs for APIs, guides and tutorials. All of this material will be available today on the newly redesigned TensorFlow.org website where you will find examples, documentation and tools to get started. We are very excited about these changes and what's to come. To tell you more about improvements in TensorFlow for research and production, I would like to welcome Megan Kacholia on stage. Thank you. Thank you. Thanks Rajat. TensorFlow has always been a platform for research to production. We just saw how TensorFlow high-level APIs make it easy at a get started and build models and now let's talk about how it improves experimentation for research and let's you take models from research and production all the way through. We can see this in paper publications which are shown over the past few years in this chart. Powerful experimentation begins and really needs flexibility and this begins with eager execution and TensorFlow 2.0 every Python command is immediately executed. This means you can write your code in the style you are used it without having to use session realm. This makes a big difference in the realm of debugging. As you iterate through, you will want to distribute your code on to GPUs and TPUs and we have provided tf. function turning your eager code into a graph function-by-function. You get Python control flow, asserts and even print but can convert to a graph any time you need to, including when you are ready to move your model into production. Even with this, you will continue to get great debugging. Debugability is great not just in eager but we have made improves in tf. function and graph. Because of the mismatch inputs you get an error. As you can see, we give information to user about the file and line number where the error occurs. We have made the error messages concise, easy to understand and actionable. We hope you enjoy the changes and they make it easier to progress with the models. Performance is another area we know researchers as well as all users for that matter care about. We have continued improving core performance in TensorFlow. Since last year, we have sped up training on eight Nvidia TeslaV100s by LLGS double. With Intel and MKL acceleration we have gotten inference speed up by almost three times. Performance will continue to be a focus of TensorFlow 2. 0 and a core part of our progress to final release. TensorFlow also provides flexibility and many add on libraries that expand and extend TensorFlow. Some are extensions to make certain problems easier like tf.text with Unicode. It helps us explore how we can make machine learning model safer by a tf.privacy. You will hear new announcements on reinforcement learning and tomorrow we will discuss the new tf. federated library. It is being applied to real world applications as well. Here are a few examples from researchers at Google where we see it applied to areas like data centers and making them more efficient. Our apps like Google Maps, the one in the middle, which has a new feature called global localization and combines street service. And devices like the Google Pixel that use machine learning to improve depth estimation to create better portrait mode photos like the ones shown here. In order to make these real world applications a reality, you must be able to take models from research and prototyping to launch and production. This has been a core strength and focus for TensorFlow. Using TensorFlow you can deploy models on a number of platforms shown here and models end up in a lot of places so we want to make sure TensorFlow works across all these servers, Cloud, mobile, edge devices and Java and number of platforms. We have products for these. TensorFlow Extended is the end the end platform. In orange, shown here, you can see the libraries we have Open SourceSourced so far. We are taking a step further and providing components built from these libraries that make up an end-to-end platform. These are the same components used internally in thousands of production systems powering Google's most important products. Components are only part of there story. 2019 is the year we are putting it together and providing you with an integrated end-to-end platform. You can bring your own orchestrator. Here is airflow or raw Kubernetes even. Not matter what orchestrate you chose, the items integrate with the metadata store. This enables experiments, experimentation, experiment tracking and model comparison and things I am sure you will be excited about and will help you as you iterate through. We have an end-to-end talk coming up from Clemens and his team and they will take you on a complete tour of TensorFlow Extended to solve a real problem. TensorFlow Lite is our solution for running models on a mobile and IOt hardware. On device models can be reore responsive and keep users on device for privacy. Google and partners like iqiyi provide all sorts of things. TensorFlow Lite is about performance. You can deploy models to CPU, GPU, and even EdgeTPU. By using the latest techniques and adding support for OpenGL and metal on GPUs and tuning performance on EdgeTPUs we are constantly pushing the limits of what is possible. You should expect greater enhancements in the year ahead. We will hear details from Raziel and colleaguess coming up later. JavaScript is the number one programming laj language in the world and until recently hasn't benefited from all the machine learning tools. We released TensorFlow.js last year. Since then we have seen huge adoption in the JavaScript community with more than 300,000 downloads and a 100 contributors. We are just at the beginning given how big the JavaScript ecosystem is. Today we are excited to announce TensorFlow.js 1.0. We have a library of off the shell models for common ML problems and we have adding support for more platforms where jafs runs and a huge focus in TensorFlow.js is on performance improvements. Compared to last year, mobile net inference and browser is now nine times faster. You will learn more about these advances in our talk later in the day. Another language we are excited about is Swift. It is reexamining what it means for performance and usability. With a new programming model, it intends to bring further usability. We are announcing Swift for TensorFlow is now at 0.2 and ready for you to experiment with, try out and we are really excited to be bringing this to come community. In addition to telling you about version 0. 2 we are excited to announce Jeremy Howard of fast.ai is writing a new course and Chris and Brennan will tell you a lot more about this later today. To recap everything we have shown you, TensorFlow has grown to a full Eco system from research to production from server to mobile. This growth has been fuelled by there community and wouldn't be possible without the community. To talk about what we are planning for you and with you in 2019, I will hand it over to Kemal. [Applause] Thank you, Megan. Hi, may name is Kemal and I am the product director. We are celebrating the most important part of what we are building and that is the community. I love building developer platforms. I used to be a developer and now I enable developers by building a better platplatform. We turned to the community and consulted with all of you on important product decisions. We received valuable feedback and couldn't have built 2.0 without you. We created special interest groups, or SIGs, like networking and TensorBoard to name a few. SIGs are a great way to build the pieces of TensorFlow they care the most about. We wanted to hear more about what you were building and launched the power by TensorFlow campaign. We were amazed by the creativity of the products. After three years, our community is really thriving. There are 70 machine learningGD Es around the world, 1800 contributors and countless more doing amazing work to help make TensorFlow successful. On behalf of the whole TensorFlow team we want to say a huge thank you. [Applause] We have big plans for 2019 and I would like to make a few announcements. First, as our community grows, we welcome people who are new to machine learning and it is really important to provide them with the best educational material. We are excited to announce two new online courses. One is with deeplearning.ai and published in the Coursera platform and the other is with Udacity. The first batch of the lessons is available right now and provide an awesome introduction to TensorFlow for developers. They require no prior knowledge to machine learning. I highly encourage you to check them out. Next, if you are a student for the very first time, you can apply to the Google summer of code program and get to work with the TensorFlow engineering team to help build a part of TensorFlow. I also talked about the power by TensorFlow campaign. We are excited and decided to launch a 2.0 hackathon on DevPost to let you share your latest and greatest and win cool prizes. We are really excited to see what you are going to build. Finally, as our ecosystem grows, we are having a second day of the summit, but we really wanted to do something more. We wanted a place where you can share what you have been building on TensorFlow. So we are excited to announce TensorFlow World. A week long conference dedicated to open source collaboration. This conference will be co-presented by O'Rielly media and TensorFlow and held in Santa Clara at the end of October. Our vision is to bring together the awesome TensorFlow World and give a place for folks to connect with each other. I would like to invite on stage Gina Blaber to say a few words about the conference. [Applause] Thank you, Kemal. O'Rielly is a learning company with a focus on technology and business. We have strong ties with the open source community as many of you know, and we have a history of bringing big ideas to life. That's why we are excited about partnering with TensorFlow to create this new event that brings machine learning and AI to the community. The event is TensorFlow happening October 28th-31 in Santa Clara. When I say community, I mean everyone. We want to bring together the entire TensorFlow community of individuals and teams and enterprises. This is it place where you will meet experts from around the world, the team that actually creates TensorFlow, and the companies and enterprises that will help you deploy it. We have an open CFP on the TensorFlow World site. I invite you all to check it out and send in your proposals soon so your voice is heard at that event. We look forward to seeing you at TensorFlow World in October. Thank you. [Applause] Thank you, Gina. This is going to be great. Are you guys excited? So we have a few calls to action for you. Take a course, submit a talk to TF World. The grand prize for hackathon on DevPost will include free tickets to TensorFlow World. You know, one thing I love is to hear about these amazing stories of people building awesome stuff on top of TensorFlow. As a team, we really believe that AI advances faster when people have access to our tools and can then apply them to the problems they care about in ways that we never really dreamed of. And when people can really do that, some special things happen. I would like to share when you something really special. Looking at this book, page by page, and trying to decipher, read and transcribe whatever is there takes an enormous amount of time. It would require an army army polyographer. Machine learning enabled us to solve problems that up to 10-15 years ago we thought were unsolvable. You have thousands of images of dogs and cats in the internet but very little images of ancient manuscripts. We involved high school students to collect the data. I didn't know much about machine learning in general, but I found it easy to create a TensorFlow environment. When we were trying to figure out which model worked best for us, Keras was the best solution. The production model runs on TensorFlow layers and estimator interface. We experimented with binary classification with fully connected networks and finally we moved to convolutional neural networks and classification. When it comes to recognizing single characters, we can get 95% average accuracy. This will have an enormous impact. We will have a massive quantity of historic information available. I think solving problems is fun. It is a game against myself and how good I can do. That is -- [Applause] This is such a great story. I think about the scholars who wrote these manuscripts, they cannot have imagined centuries later people bringing their work to life. We are thrilled to have Elena. The team and I will be around. Come and say hi. With that, I am going to hand it over to Martin who will talk about TensorFlow 2.0. Thank you. Thank you, Kemal. I am Martin Wicke. I am the engineer lead for TensorFlow 2. 0 and I will unsurprisingly talk about TensorFlow 2.0. TensorFlow has been extremly successful and inside of Google and outside of Google it has grown into a vibrant community that when we started it we were not able to imagine. Our users are building these engenous, clever, elegant things every day from art to music to medieval manuscripts, science and medicine. The things people created were unexpected and beautiful. We learned a lot from that. We learned a lot from how people did it. TensorFlow has enabled all this creativity and jump started this whole AI democraization which is great. It has been hard to use sometimes. Sometimes a little bit painful. So, you know, using sessions wasn't the most natural thing to do when coming from regular Python. The TensorFlow API has grown over time. It got a little bit cluttered and confusing. In the end, you can do everything with TensorFlow, but it wasn't maybe always clear what was the best way. We have learned a lot since we started this and we reallyized that you need rapid prototyping and easier debugging. There is a lot of clutter. So, we are fixing this, addressing these issues with a new version of TensorFlow. Today, we are releasing an alpha of TensorFlow 2.0. Many of you have participated in the design reviews that went into this for all the features we have blended. Those of you living on the bleeding edge of development have played it with, installed it, tested it, found issues and fixed issues. Thank you very much for your help. You can now install the alpha release of TensorFlow 2. 0 using just pip install pre-TensorFlow and you can go and play with it. What has changed? When we created TensorFlow 2. 0, the biggest flow was flexibility. We have integrated Keras with TensorFlow. It gives you a path of how to develop and deploy models and has a well established API that makes people productive and we know it gets you started quickly. TensorFlow also includes a complete implementation of Keras, of course, but we went further. We have extended Keras so that you can use all of the advanced features in TensorFlow directly from tf.Keras. The other change is eager execution. It used declarative style and was distance from running Python. In TensorFlow 2. 0, TensorFlow behaves just like the Python style. All of that stuff is not going away. It is just getting easier. TensorFlow 2.0 is a major release which means we have the ability to clean it up and we did, like a lot. We had a lot of duplicate functionality we have removed. We have consolidated the API and organized the API. This is not only with TensorFlow itself but the whole ecosystem of tools that has grown around TensorFlow. We have spent a lot of effort to align these interfaces and defined exchange formats that will allow you to move all throughout this ecosystem without hitting any barriers. We have removed a lot of stuff from TensorFlow but it doesn't mean -- it means that it has gotten more flexible not less. It retains and expands on the flexibility and we have created a more complete low level API that exposes all of the internal used ops to the user. We provide inheritable interfaces for the crucial concepts like variables and checkpoints. This allows framework authors to build on top of TensorFlow and maintain interoperability with the rest of the ecosystem. There will be a talk about tf. agents later and Sonnet by DeepMind tomorrow. You will see how this works in practice. The real question is of course what do I have to do be apart of this great new era of TensorFlow and we know it is always hard to upgrade to anything and that is especially true for major up grades. We will start the process of converting one of the largest codebases in the world to TensorFlow 2.0 and we will be writing a lot of migration guides. We have started already and some are online and will provide a lot of best practices for you. If we can do it at Google, you can probably do it too. We are giving you, and us, a lot of tools to make this transition easier. We are shipping a module called tf.contrib which contains all of the complex API. If you rely on a specific function you can find it there except for tf. contrib which isn't going to be included but we have created a community supported alternative that support your use case if you are relying on something in there. We are also publishing a script which automatically updates your code so it runs on TensorFlow 2.0. Let me show you how that works. Let's say I want to convert this program which is a simple language model chain of Shakespeare. What where do is run TF upgrade 2. There is just a handful of things it changed in this case. You can see, if you look at what it changed, there is some functions it renamed. A lot of the reorganization of the API. It renamed multi nomial to random categorical. A lot had functions changed or added and deleted. The script will make those changes. If all else fails, there is not an equivalent and we will use the compatibility module I talked about earlier. If there is no perfect replacement, we will use that to make sure your code still works as expected after the conversion. It is pretty conservative. For instance, the atom optimizer is a subtle behavior change that will mostly not affect you but we will convert that just in case it might. Once the conversion is complete, and there are no errors, then you can run the new program against TensorFlow 2.0 and it will work. That's the idea. It should be pretty easy for you. Hopefully you won't get trouble. Note that this automatic conversion will fix it so it WOSH works but won't fix the style to the new TensorFlow 2.0 style. That can not be done automatically and there is a lot more to know. If you want to know more, check out the talk at 10:30 tomorrow. Let me give you an idea about the timeline of all this. We have cut 2.0 alpha today or yesterday. Took some time building it. We are now working on implementing some missing features that we know about, converting libraries, converting Google. There is lots of testing and optimization that will happen and we are targeting to have a release candidate in spring. Spring is a flexible concept but in spring. After we have this release candidate, it still has to go through release testing and integration testing. I expect that to take longer than regular releases but that is one you can see an RC and eventually a final. So, if you want to follow along with the process, please, do so. All of this is tracked online on the GitHub TensorFlow 2.0 tracker. If you file any bugs, those are up there as well so everybody knows what is going on. TensorFlow 2.0 really wouldn't be possible without you. That has been the case already but in the future even more so. We will need you to test this, we will need you to tell us what works and doesn't. Please install the alpha today. We are extremely excited to see what you can create. Check out the docs. They are at tensorflow.org/r2.0. With that, I think you should hear about what it is like to work with TensorFlow 2.0. We will have plenty of content over the next couple days to tell you exactly about all the different aspects of it but we will start with Karmel who will speak about high-level APIs in TensorFlow 2.0.thank you. Hi, my name is Karmel Allison. Martin just told you about TensorFlow 2.0 and how to get there and I will tell you a little bit more about the high-level APIs. What do I mean when I say high-level APIs? With our high-level APIs, we bring you tools to help you easily build your models and scale them. As Martin mentioned, a couple years ago, TensorFlow adopted Keras as one of the high level APIs we offered. Keras is a specification for model bidding. It works with multiple machine learning frameworks and it provides a shared language for defining layers, models, losses, optimizers and so on. We implemented tf.Keras. Raise your hand if you are used this? Keras is just that. Simple. It was built from the ground up to be Pythonic and easy to learn and has been instrumental in inviting people into the machine learningworld. It is easy for you to design and modify your model architectures without needing to write tons of boilerplate. At the same time, by replying on inheritance and interfaces, Keras is extremely flexible and customizeable. I can define architectures, modify what happens and even change the whole training loop, if I want to. Which is to say Keras is simple and effective. Anyone can figure out how to use it but we had a problem. Tf. Keras was built for smaller problems and many machine learning need to operate at a larger scale. We have estimators. They were built from the ground up to distribute and scale with fault tolerance across hundreds of machines no questioned asked. This is the much love wide and deep model. A work horse that took mean years to define but is a built-in for training and deployment. Estimators are powerful machines but you told us there is a steep learning curve and it is not always easy to figure out which parts to connect where. We have learned a lot building estimator and tf.Keras and reached the point we don't think you should have to chose between a simple API and scalable API. We want a higher level API taking you from endness to plan and scale, no questions asked. In TensorFlow 2.0, we are standardizing on the Keras API for building layers and models but bringing the power estimators into tf.Keras so you can move from prototype, distributes training and production serving in one go. OK. How are we doing this? This is a tf. Keras model definition in TensorFlow 1.13. This is the same model definition in 2.0. The code is the same but what is different? In 1.13 this built a graph-base model that ran a session under the hood. In 2. 0, the same model runs in eager mode without any modification. This let's you take advantage of everything eager can do for us. We see our dataset pipeline behaves like a num py array. We are able to achieve this performance with datasets and Keras by taking advantage of graphs even in an eager context. Eager makes debugging and prototyping easy and all the while Keras builds an eager-friendly function under the hood that tracks your model and the TensorFlow Runtime takes care of optimizing for performance and scaling. You can explicitly run your model in eager mode with the flag run eagerly even though for this example it might be overkill, you can see run eagerly allows you to use Python control flow for debugging while prototyping your model. Another big change in TensorFlow 2. 0 is we have consolidated many headings and maing it easier to know what you should use and when. We have one set of optimizers that work across TensorFlow in and out of eager mode on one machine or distributed. You can set opt amam -- optimizers. We have one set of metrics that encompasses all the former tf. metrics and Keras metrics and allow for easy subclassing if you want even more. Losses, similarly, have been consolidated into a single set with a number of frequency used built-ins and an easily customizeable interface if you chose. We have one set of layers built in the style of Keras layers which means they are class based and fully configureconfiguable. One set of layers that we took particularly care with were the RNN layers in TensorFlow. Raise your hand if you are used RNN layers in TensorFlow? These slides are for you. In TensorFlow version 1 we had several versions of LSTMs and GRUs. In 2.0, there is one version of the LSTM layer and one version of the gru level. This runs the kernel but falls back to CPU ops if you don't have GPUs available. In addition to all of those, if you need layers that are not included in the built in set, tf. Keras provides a subclass that is easy to customize. This is how TensorFlow addons operate providing specialized and complex layers, metrics, and so on by building on top of the Keras base classes. We streamlined APIs. That is a start. The next thing we did was integrate. We found easily configurable data parsing. In TensorFlow 2. 0, you can use Feature Columns to parse data and feed directly to downstream layers. This works with Keras and estimators so you can mix and match to create reusable data input pipelines. OK. You have data in your model and you are ready to train. One of the most loved tools we have is TensorBoard and I am pleased to see TensorBoard integration with Keras is as simple as one line. We have the TensorBoard callback to our model when training here and this gets us both our training progress, here we see accuracy and loss, and a conceptual graph representing the model we built layer by layer. As an add in bonus, it includes full callback so you can better understand the placement and find ways to minimize bottlenecks. Speaking of performance, one of the exciting pieces we are bringing TensorFlow 2. 0 is the -- these APIs have been designed to be easy to use, out of the box performance and be versetle. How do they work? With Keras, you can add and switch distribution patterns with a to a lines of code. We are distributing this across multiple GPUs. Because we have integrated distribution strategies throughout Keras and TensorFlow you get a lot with these few lines of code. Data is prefetched batch by batch and mirrored in sync using all reduce and we see greater than 90% scaling efficiencies meaning you can scale your model up without changinging code and without loosing any of the conveniences of Keras. Once you have trained your model, you are likely going to want to package it for use with production systems, mobile phones or other programming laj -- language. This functionality is experimental but today and going into 2.0 you can easily export your models for use with tf. serving, tf.lite and more. You can reload the models back into that you be Python to continue training and using in your normal workflow. That is where we are today in the alpha. We are obviously not done yet. Let me tell you a little bit about what is coming up in the next few months. We talked about distributing over multiple GPU but that is just the start. We can swap out strategy and scale up to multiple nodes. Here we cake the same Keras model and distribute it across multiple machines working in sync using collective ops. The multi-worker mirror strategy supports Ring and allows great performance. You can try it out today. If you are interested in a TensorFlow native solution to synchronous multi working training. We are excited to say using Keras and distribute strategies you will be able to use the same code to distribute models on TPUs. When released, this strategy will work on Google Cloud TPUs and Colab which can access TPUs directly. As we approach the final 2.0 release we will continue to bring scalability into Keras and allow parameter service strategy, the multi node asynchronous training that estimator does today. For those that want a higher level API, we will be adding support for partition variables across many machines for some of the largest scale models like those we run at Google. And that's just a sneak peek so please check out the 2. 0 preview and check out Keras if you haven't already. You will hear more about Keras and high-level APIs throughout the day and we have a workshop tomorrow that Martin mentioned. With that, let me introduce Raziel who will tell you more about TFLite. Thanks, Karmel. Hi, everyone. My name is Raziel and I am an engineering lead. Today I will be covering at -- a lot of the work the team has done over the past year that you can take advantage of right away. I am cover a lot of work we have for this year. Before we jump into that, let me give you a brief introduction of what TensorFlow Lite is. TensorFlow Lite is our infrastructure framework to run machine learning on device. The reason why we decided to built TensorFlow Lite was because machine learning you think of it as running on the server, but more and more it is moving to edge devices like mobile devices, cars, Wearables, you name it. There is more machine learning moving to different kinds of devices. There is good reason for that. One, you have access to a lot more data because you have access to the algorithm, camera, and don't have to stream it all the way to the server so you can do a lot of stuff. It means you can build faster and close-knit interactions. Another advantage of running machine learning on devices is it has a strong privacy preserving component because the data stays on the device and doesn't have to go to the server. However, that means basically by building machine learning on devices we can build products that otherwise wouldn't be able to really work if you really rely only on server-side execution. However, you know, it is hard. Perhaps one obvious thing is you don't have the same compute power you have on the server. If you have a new shiny machine learning model that consumes too much compute power, it is not like in the server where you can throw perhaps more servers at it and be done with it. You have to be mindful of the resources that you have on the device. Another of those resources is the memory. You have restricted memory so you have to be careful about how you craft your model, perhaps try some compression techniques. Last but not least, many of these devices are battery powered and even if you have a model that fits in memory and can execute, well, you don't want to consume all the battery in a battery powered device. For all these reasons, we built TensorFlow Lite and we want to make deploying machine learning on edge devices as easily as possible. What can you do with it right now? You can do a lot of things. These are different tasks that are supported by TensorFlow Lite already. If you look at it when you hear machine learning you typically fall into one of these tasks. You know, text-to-speech, image processing, audio processing, a lot of content generation so really, like, everything is already there for you to take advantage of and because we are already able to support these tasks, it means it has a lot of usage already. TensorFlow Lite is currently deployed in over 2 billion devices in production. This not only means Google properties, some of which are core to Google like the assistant and Google photos but come from other large companies and those are frameworks. A lot of our infrastructure is power and frameworks. I can tell you many reasons why it is a good idea to use our infrastructure but we thought it would be best if we bring two of our component clients to tell you how they use TensorFlow Lite. Our first presenter is Alex coming from the Google Assistant which has a wide variety of devices it is running. Let's hear why they are using TensorFlow Lite and how they are doing. [Applause] Thanks, Raziel. I am Alex. I am an engineering lead on our embedded speech recognition team at Google. One of the key applications for speech recognition at Google is the Google Assistant. Google recently announced the Google Assistant is on about a billion devices ranging from phones, speakers, smart displays, cars, TVs, laptops, Wearables, you name it we are trying to put the assistant into it. That means high end devices, low end devices, battery powered, plugged in, and a large range of operating systems. The neural networks I build have to be able to run anywhere. For assistant, we have a couple key speech on-device capabilities. The first is the hot word. You need to be able to say hey, Google to activate the device and we need to make sure it is your voice. We have to have a tiny memory and computational input and we are extremely latency sensitive. Our other major application is on device speech recognition. We use this when you are offline or have a bad network connection. Instead of using the server, we run speech recognition on the device. Here we are trying to run large models as fast as possible to get the best probably accuracy. It is an a high load returning into short burst typically. Over the years, we have been working on this for a long time, before there was TensorFlow Lite, before there was TensorFlow, so we have been investing a lot for a long time in building our own on-device neural net libraries and shipping them to production in Google product for a long time. As a team, we have invested in keeping the code size tiny, memory small, and optimizing for all different platform and all of our code after many years production use is hardened and as bug-free as you can get it, hopefully. Over the last year, we decided to migrate to TFLite. This wasn't an easy decision for us and as we switched over we checked carefully to make sure TFLite would beat our existing models and we have moved all over in the last year. We are excited because this lays the ground work for us moving to a new standard that is being widely adopted and we think TFLite will help us accelerate models on GPUs and EdgeTPUs and new accelerators coming. Thanks. I will turn it back over to Raziel. [Applause] Thanks, Alex. So, Alex analyzed the key goals for TensorFlow Lite. We want to be able to support your current-use-cases, but perhaps equally important or more important, is we want to cater to your needs that you will have 1-2 years in the future. Our next presenter is Huiije from a China-based company that has several applications and hundreds of millions of users. Let's welcome him. Thanks, Raziel. I am Huiije from NetEase in Youdao. We have built a lot of dictionary translation and note taking applications in China. We have over 800 million users in total. We also have 22 million daily active users. In the past years, we have built several dictionary translation and note taking applications. We have the most popular dictionary application in China, for example, which is Youdao dictionaries and we have the most popular translator in China and the most popular dictionary and language learning application in India which is Youdao dictionary. We provide features for users in these applications so they can conveniently look up words in the scenario such as, for example, you can use your camera to recognize the words from the images and do the OCR translation on your devices. We use the TensorFlow Lite to use theOCR and translation here. We have provided the photo translation in your applications. For example, you can use camera to take a photo from the many scenario and it will do the whole image of OCR and translate the text into another language. Then it will erase the text on the image and replace the original text to the translated text to present users with the translation. In this scenario, we also use the TensorFlow Lite to accelerate our OCR and translation services here. The OCR and translation is very sensitive to the binary size and also computational resources and responding time. So we chose the TensorFlow Lite to accelerate our abilities on device to provide more efficient on device inference here. That is why we chose the TensorFlow Lite to do this. Thanks. [Applause] Really exciting to see how our >> INSTRUCTOR: -- infrastructure is used by hundreds of users around the world. TensorFlow Lite is geared not only towards Google but everybody should be able to take advantage of it. For the rest of the presentation, I will cover what I said at the start. A lot of features that are available over the past year and roadmap for this year. We organized our engineering in four things. First is usability, overall it means we want to be able to get you up and running as easily and fast as possible. Now once you have your model basically we want to get you executing that model as fast and possible in the hardware that you care about. Then with optimization we look at certain things we can do to make it faster and smaller. Finally, with our documentation efforts, we want to give you all the resources you need. Let's start with usability. There is different ways of getting up and running in TensorFlow Lite but at the end of the day you will most likely train the model in TensorFlow and convert it to TensorFlow Lite. The flow is very simple. You train in TensorFlow, you get a model and use the TensorFlow Lite convertert and get a model you can execute on different kinds of devices. However, there is points of failure. TensorFlow Lite made the decision to take a subset of TensorFlow and optimize those. That means basically we don't have all the same hundreds of operations that TensorFlow has. The other reason is some semantics. We don't support jet on TensorFlow like control flow which is typically used for reoccurring neural networks. We have made already like strides trying to basically make this easier and better for you. Last year we launched TensorFlow select and this basically means you can execute TensorFlow ops in TensorFlow Lite. In the pipeline, we are making improvements to TensorFlow Select by enabling selective registration and I will explain what all this means in the following slides. We are working on contraflow support as well. Select, like I said, means you can execute TensorFlow and TensorFlow Lite. It comes at the cost of some binary size increase because right now it is basically all or nothing, but that is why we are building selective registration. Selective registration is something you can take advantage of already on TensorFlow Lite for our building ops. You include in the binary only the ops you are using. We are bringing that to TensorFlow Select. We are trying to blur the line of what the TensorFlow and TensorFlow Lite ops are and achieve one of the key points is blur the performance between one and the other. Now, contraflow is something we are working on. This is one of fully lighter pain points we are trying to convert occurring neural networks to TensorFlow Lite. We are adding supports and loops and conditionals. All of this is part of a bigger effort to revamp our converter and we took your feedback very seriously and we wanted this converter to answer these three questions. Some Something fails and what went wrong and what can you do to fix it. OK. Jumping to the next theme is performance. We want your models to execute as fast as possible in the hardware you have available. If you saw in Megan's slides, we have made huge slides in these areas. This is an example of a model. The execution of the CPU, floating point compared to the fast version in CPU. With the recently developing preview we launched, we get huge gains. Like over 7X improvements. By using the HTP we get crazy speed ups. 62X. Again, this is an area we are working really hard. Again, some things that are already available. We have a very nice benchmark and profiling tool and a few delegates. For those not familiar with the concept of TensorFlow delegates, that is our abstraction layer that allows executing models on the CPU. In the pipeline we are working on CPU improvements because we know CPU is important. I am not listing here in the pipeline anything around newer delegates because it tends to be a partnership with other companies and I don't have anything to announce yet, but we are working on that. Our benchmarking tool is basically a profiler. We added some features. Personal threading. You can see statistics for opposite execution and it supports the neural networks API. If you execute the profiler you get a lot of information. This is just a snippet of all the stuff that you get. In this case, we are showing you here information for each op in your graph and how long it took to execute. Then you also get a nice summary of the archives in your graph and many useful statistics so you can decide maybe there is a way that you can, you know, change one for another type that is more efficient or tweak your graph in some way. Now let's talk about the delegates which is an abtraction layer for extracting graphs. The way it works is very nice. In the TensorFlow Lite you prepare a model, right? You are not necessarily saying this model is for this hardware. You just prepare your model. Then inpreter will take it and for those delegates you raised it will go one by one and it will say how much of these graph can you execute. What the delegate cannot execute, it will be executed in CPU. So, by doing this basically you will always be able to execute, right? And if anything, you will get better performance if the delegate can consume more of your graph. We have already an edge-TPU delegate. It has a small power footprint and later on we will have more information about the edge-TPU and a very cool demo. The next delegate is the GPU delegate. These are available right now. We see very nice performance improvements at the cost of relatively small binary size increase. How do you enable the GPU delegate? Just like any other delegate, this is an example of the interpreter and executing one inference. The only thing you need to do is share and pass to interpreter and that is it. So, stuff that we are working on for the GPU deldelegate, we want to add more ops. We want to improve performance even further. We want to finalize APIs so this is generally available. The final delegate I want to mention is the neural network API. This will allow you to execute any other hardware that the API supports. It should be just transparent for you. Finally, CPU is still very important for many use-cases. Most devices have some form of a CPU. We are targeting further improvements on the arm and X86 architectures. The next theme is optimization. Maybe we can do 2X model to make it faster and smaller. We already did some work here. Last year we released this quantization tool that gives nice performance on CPU. This is the first launch that we have while we are calling model optimization toolkit. It will be available for use for everybody. In the pipeline some of these new tech techniques we will put in the toolkit is more quantization work. It gives further improvement on CPU and allows you to execute more hardware. A lot of the hardware like neural processing units are fixed point base with no floating base support. We are working on a technique called production pruning. So quantization means changing the precision of your model and how it is executed on the device. We launch post training quantization tool where you take your save pod model and when you convert to TensorFlow Lite you turn on the flag and it updates. Trying to get the most performance and small Keras loss. We see nice improvements this way. If the model is is fully quantized and you get 4X. For convolutional, we see 10-15% performance improvements but for fully connected RNNs we see up to 3X. This is simple to try. If you are using TensorFlow Lite or plan to use it, give this a try. You have to validate how the accuracy is affected in your particular model but so far we have seen very good numbers. Many of the two billion devices we mentioned before are running this way. In the pipeline, again, we are making further improvements in the quantization approaches that we have. We are working on this for quantization. We want to make it powerful by enabling quanterize straining and easy to use. We are bringing more post-training quantization support. This is an example of Keras model. It is very easy with two dense layers. Let's say you want to train with quantization those two dense layers. Well, soon enough, you will be able to just import your API and say quantatize those two layers. The post-training quantization for fixed point will work very much like the one we already support where you have the conversion path and the only thing you need to parse extra is some calibration data which is the typical model your sees as input. If you have an image processing model, you just need to pass some images. You don't really need to pass a ton of images. In preliminary numbers, model Nets, we see with 25 images we do well. These are preliminary numbers we see with this tool. We have the float baseline and then if you do quantization training you get almost the same accuracy with all the benefits. Then even further if you really, for some reason don't want to invest in this training or you have a model, you know, that you got somewhere, you can still quantize it with post training and you see the accuracy is almost the same. The other technique we are heavily investing in is connection pruning meaning we prune connections in the network and those connections become Keras and that creates a sparse tensor. The benefits of a parse tensor is you can compress them and create potentially faster models if you have less computations to do. Coming very soon, we are going to add a training pruning as a Keras-base API and in the pipeline we are working on first class support of this parse execution in TensorFlow Lite. Again, in our example from Keras you have two dense layers, want to prune connection from those layers, and the only thing you need to do is say prune. It is very, very simple. When you convert the model to TensorFlow Lite format, basically by just even seeding the file you will get nice compressions. These are some, again, very preliminary numbers. A mobile net which is pretty hard to prune compared to other perhaps larger models, we see various models of 50% and even 75%. Finally, documentation, again, we want to give you all the resources you need to get the most out of TensorFlow Lite. We have already done pretty nice improvements. We have a site with more content. We currently have a model representation with five models and representations you can try. We are working on new models and looking to expand our repository. The goal is not only that it is pretty but that it answers your questions. Please give us more feedback if there is things that are still lacking there. Yeah, we revamp the documentation and very important, we also have the roadmap for the foreseeable future. A lot of the stuff I am talking about now you can see there in more detail perhaps. And then, again, we have our repository with all these applications and tutorials and we are expanding it. We want to make the repository into something that, you know, if you don't want to go, or before you start trying to train something, perhaps you have a model you can reuse. By the way, TensorFlow mobile is aggregated especially using TensorFlow select allowing you to use TensorFlow Lite and there is no need for TensorFlow mobile except if you are doing training on the device. That is the reason we are keeping that there. Yeah, training. So, if you see we are building a lot of the basic infrastructure that you will need when you are trying to prepare a training pipeline to execute on device. I don't have anything to announce now but I just wanted to let you know that we are working on it and thinking a lot about it. What about TensorFlow 2.0? Well, it should work. You should be able to convert it to TensorFlow Lite. Before we go two last things. One is there was a lot of content in this talk. Tomorrow we have breakout sessions and will cover TensorFlow Lite in general and optimization in particular. And I want to introduce Pete Warden who is going to talk about some very cool products. [Applause] Awesome. Thanks so much, Raziel. I am really excited to be here to talk about a new project that I think is pretty cool. So, TensorFlow Lite for microcontrollers. What is that all about? This all comes back to when I actually first joined Google back in 2014. There were a whole bunch of internal projects I didn't know about as a member of the public that sort of blew my mind but one in particular came about when I actually spoke to Raziel for the first time. He was on the speech team at the time working with Alex who you just saw. He explained that they use neural networks models of only 13 kilobytes in size. At that time, I only really had experience with image networks and the very smallest of them was still like multiple megabytes so this idea of having a 13 KB model was amazing for me. What amazed me even more was when when he told me why they had to be so small. They needed to run them on dsp and other embedded chips in smartphones so Android could listen for wake words like hey Google while the main CPU was powered off to save the battery. These microcontrollers often only had 10 kilobytes of RAM and flash storage. They couldn't rely on Cloud connectivity because the amount of power that would be drained to send data over would have been prohibitive. So, that really struck me. That conversation and the continued work that we did with the speech team because they had so much experience doing all sorts of different approaches with speech. They had spent a lot of time and a lot of energy experimenting and even within the tough constraints of embedded devices, neural networks were better than any of the traditional method they used. I was left wondering if they would be really useful for other embedded sensor applications as well and it left me really wanting to see if we could actually build support for these kind of devices into TensorFlow itself, so that more people could actually get access to it. At the time, only people in the speech community really knew about the groundbreaking work that was being done and I wanted to share it a lot more widely. So, today I am pleased to announce we are releasing the first experimental support for embedded platforms in TensorFlow Lite. To show you what I mean, here is a demonstration board that I actually have in my pocket. This is a prototype of a development board built by SparkFun and it has a cortex M4 processor with 384 kilobytes of RAM and a whole MB of flash storage and it was built by to be extremely low power -- Ambiq. It is able to run on a single core battery like this for many days potentially. I am going to take my life in my hands now by trying a live demo. Let us see if this is actually -- it's going to be extremely hard to see unless we dim the lights. There we go. What I am going to be doing here is by saying a particular word and see if it actually lights up the little yellow light. You can see the blue LED is flashing and that is telling me it is running inference. If I try saying "yessaying "yes", "yes". I knew I was taking my own life into my own hands. There we go. As you can see, it still is far from perfect but it is managing to do a job of recognizing when I say the word and not lighting up when there is unrelated conversations. So why is this useful? Well, first this is running entirely locally on the embedded chip, so we don't need to have any internet connection. It is a good useful first component of a voice interface system. The model isn't quite 13 kilobytes but it is down to 20 kilobytes so it only takes up 20 kilobytes of flash storage on this device. The footprint of the TensorFlow Lite for microcontrollers is only another 25 kilobytes and it only needs about 30 kilobytes of RAM to operate. It is within the capabilities of a lot of different embedded devices. Secondly, this is all open source, so you can actually grab the code yourself and build it yourself, and you can modify it. I am showing you here on this particular platform but it actually works on a whole bunch of different embeded chips. We really want to see lots more supports so we are keen to work with the community on collaboratoring to get more devices supported. -- collaborating. You can also train our own model. Just something that recognizes yes isn't all that useful, but the key thing is that this comes with a toil you can use to actually train your own models. It always comes with a dataset of 100,000 utterances of about 20,000 common words. The first link there, if you could go to that link, and contribute your voice to the open dataset, it should actually increase the size and the quality of the dataset that we can actually make available. That would be awesome. You can actually use the same approach to do a lot of different audio recognition to recognize different kinds of sounds and even start to use it for similar signal processing problems like, you know, things like predictive maintenance. So, how can you try this out for yourself? If you are in the audience here, at the end of today, you will find that you get a gift box, and you actually have one of these in there. [Applause] All you should need to do is remove the little tab between the battery and it should automatically boot up pre-flash with this yes example. You can try it out for yourself and let me know how it goes. Say yes to TensorFlow Lite. We also include all the cables so you should be able to program it yourself through there serial port. These are the first 700 boards ever built so there is a wiring issue. It will drain the battery. It will last more like hours than days but that will actually, knock on wood, be fixed in the final product that is shipping. And you should be able to develop with these in the exact same way you will with the final shipping product. If you are watching at home, you can pre-order one of these from SparkFun right now for, I think, it is $15. And you will also find lots of other instructions for other platforms in the documentation. We are trying to support all of the -- or as many of the modern microcontrollers that are out there that people are using as possible. And we welcome collaboration with everybody across the community to help unlock all of the creativity that I know is out there. I am really hoping I am going to be spending a lot of time over the next few months reviewing pull requests. And finally, this was my first hardware project so I needed a lot of help from a lot of people including to TensorFlow Lite team especially Raziel, Tim, and Andy, Allisster, Nathan, owen and Jim at SparkFun were life savers. We got these in our hands in the middle of the day yesterday. And also those at Ambiq who actually designed this process and helped us get the software going. A lot of people are arms as well including a big shout out to Neal and Zach. This is a very early experiment but I can't wait to see what people build with this. One final note, I will be around to talk about M CUs with anyone interested on day two. I am really looking forward to chatting with everyone. Thank you. [Applause] Thanks, Pete. We really hope that you try this. It is the early stages but you see a huge help to make this happen. We think it will be really impactful for everybody. Now, before we go again, and I promise this is the last thing you hear from me, I want to welcome June, who is going to talk about how by use TensorFlow Lite with the edge-TPU delegate are able to train these usable machines. [Applause] Thanks, Raziel. Hi, my name is June Tate-Gans and I am one of the lead software engineers inside Google's new Coral group and I have been asked to give a talk about the edge-TPU based teachable demo. Coral is a platformplatformplatform for products with on demand learning. Our first product is a single board computer and a USB stick. What is the edge-TPU? It is a Google designed asic. It doesn't require a network connection to run and allows for a whole new range of applications and machine learning. The first product we built is the Coral Dev Board. This is a single board computer with a removable som. It runs Linux and Android and the som has a gigabyte of ram, Wi-Fi and Bluetooth and the EdgeTPU. The second is the corilatored board connected to whatever you need be it Raspberry Pi or a Linux work station. This teachable machine shows off a form of edge training. Traditionally, there is three ways to do this. There is K-nearest neighbors, weight imprinting and last layer retraining. For this demo, we are using the K-nearest neighbors approach. In this animated GIF, you can see the TPU enables very high classification rates. The frame rate is actually the rate at which the TPU is classifying the images that I am showing it. In this case, you can see we are getting about 30 frames per second. It is essentially real-time classification. With that, I actually have one of our teachable machine demos here. If we can turn this on. There we go. OK. So, on this board we have our EdgeTPU development board assembled with a camera and a series of buttons. Each button corresponds with a class and lights up when the model identifies an object from the camera. First we have to plug this in. Every time I take a picture, by pressing one of these buttons it associates that picture with that particular class and because it is running inference on the EdgeTPU it lights up immediately. The first thing I have to do is train it on the background. I will press this blue button and where you can see it immediately turns on and this is because it is doing inference in realtime. Now, if I train one of the other buzz buttons using something like a tangerine, you can see it can classify between this tangerine and the background. I can grab other objects, such as this TensorFlow Lite sticker. It looks similar. It is the same color. Let's see what was the class I used? Yellow. OK. Sorry. Now, even though it is a similar color, it can still discern the TensorFlow Lite logo from the tangerine. [Applause] you can imagine in a manufacturing content, your operators are no knowledge of machine learning or training, can adapt your system easily and quickly using this exact technique. That is about it for the demo but before I go, I should grab the clicker and also I should say, we are also giving away some EdgeTPU accelerators. For those here today, we will have one available and for those on the livestream you can purchase one at Coral.Google.com. We also have links for you up here. Please take a look at the TensorFlow Lite and our new hardware and thank you very much. [Applause] Thank you, June. That was awesome. So, that is the end of our first block of talks. We are now going to have a break that's about 45 minutes. We want to be back here by noon. Sorry we ran a little long. Thank you for coming up. After the break, we will be talking about building models, Swift, and the community. It is going to be a lot of great talks. Go grab something to eat and come on back here by noon. Thank you very much. Tf. tf.tf.tf.tf. TF an TensorFlow 2. flight 370 TensorFlow Lite TensorFlow Lite Swift Keras tf. touchdown megabytes CPU GPU TPU SIGs tf. function function function TUNGZ nction function function tf.function tf. function autograph Alexandre Passos SIGs SIGs Edd Swift TensorFlow 2. FASZ test test test test Hello, everyone. Typhl TIFL TIFL te TFLite TFLite K-nearest Youdao Youdao Youdao Youdao Youdao Youdao test test test test test Hi, everyone. My name is Gal Oshri and I am here to talk about TensorBoard. How many have used TensorBoard before? It helps you visualize a lot of data coming out of TensorFlow. It has a wide variety of features but today we will talk about a couple new additions. We will actually switch over to rademo. This is Colab . I made sure we installed TensorFlow 2. 0 alpha and a couple other sets so we don't have to install it now. We are going to use the dataset and train a really simple keras sequintial model. We will train it with the fit API and give it the TensorFlow callback. If you have used Colab before, you will know we will need to download the logs to our local machine, make sure TensorBoard and set-up and point at it and go back to Colab. We are enabled showing TensorBoard directly within Colab. You will notice the way we start TensorBoard here is exactly the same as in the command line. It is the same command just as the magic cell and magic function in front of it. The same thing will also work in Jupyter notebooks. Let's see what is different. When using the keras callback we have trained validation showing up on the same chart to make it easier to compare them in accuracy, loss and other metrics. This makes it easier to detect things like overfitting. In the graph's dashboard, while seeing the ops and exillerary information is useful for many scenario, sometimes you want to see the model you created in keras. So you can switch to the keras tag and just few that model. We can expand this to see the actual layers we added. There is several other APIs for using TensorBoard within notebooks that let you change the height of the cell as well as list the active instances within your Colab notebook but we are not going to look at that because I want to switch gears and talk about hyperparameter training. We picked the dropout rate, number of units in the dense layer and the optimizer. If we really care about the modeled's performance we will want to try out different values and experiment. The way this looks in TensorBoard is you might include all that information about what the values are in the run names. As you can see here in the bottlebottle -- bottom left. It is not the best experience so let's see if we can do something better. What I'm going to show next is something that is going to change in terms of the API and the UI but it is available in the TF 2.alpha today. We will do several different imports and define which values of the hyperparameters we want to try. We will start out with just a few so we don't take up too much time during the demo. We are going to log a summary that tells TensorBoard what were the hyperparameters we care about and what were the metrics. We then wrap the existing training code that we had, just to make sure we logged the accuracy at the end on the validation set, and also tell TensorBoard that the experiment has started and finished. This time we are going to start TensorBoard before doing our training because in most cases your training will take longer than one minute and you want to view the TensorBoard while your model is training to understand its progress. We started it. It has no data but once a couple epox of training have finished we can refresh it and start to see something. You will notice in the top we have the h-parameters dashboard which show a table where each run is represented by a row and we have columns. As the runs finish, the table becomes populated with them. On the left, we have ability to filter and sort. We can say we don't actually care about the number of units or only want to see experiments where the accuracy is at least 85. Before we proceed further, I want to actually cheat a little bit and log and access some completed experiments where we have run a wider range of combinations of values for the hyperparameters. Now, while this is loading, I want to point out that I am pointing TensorBoard directly at a folder in my Google Drive. I had all my TensorFlow logs on another machine, uploaded them to my Google Drive and I can access them directly within my Colab notebook. This takes a moment to load but, hopefully, when I refresh we can see it. I can switch over to the H-parameters dashboard and see the experience. I can switch over to this view that shows the visualization of access for each hyperparameter and each metric. Each run is represented by a line that passes through all these axis at the points corresponding to hyperparameters and values. I can click and grab to select a range. I have selected the experiments with the relatively high accuracy and they become highlighted in this visualization. I can immediately see that all these experiments used the add-in optimizer as asupposed to SGD and had a relatively high number of units in the dense layer. This gives great ideas about what we can experiment with next. I can skew the scatter plot view which shows the correlation between the different hyperparameters and metrics. I can, again, select a region here to view those points across the other charts. Just to summarize, we have looked at TensorBoard in Colab. An easier way to compare the trend validation runs, visualizing the keras conceptual graph, and better hype parameter tuning with the H-program dashboard. All this information is available as documentation at tensorflow.org/TensorBoard. We also have a demo station upstairs and would love to hear from you. Next up is Alex who will talk about tf.function and autograph. Thank you very much. Hello. I am Alex. I am here to tell you about how you are going to build graphs in TensorFlow 2.0. This might make you a little uncomfortable because we already spent time early telling you in TensorFlow 2.0 we use eager execution by default. So why am I taking that away from you? And I am not. Graphs are useful for quite a few things. The two ones I care the most about personally are that some hardware like our TPUs really benefit from the full program optimization if we have graphs and if you have graphs you can take your model and deploy it on mobile and server devices and make it available to as many people as you can think of. At this point you are probably -- I remember TensorFlow 1.0 and the kind of code I had to write to use graphs. Where -- is he going to tell me I have to keep doing that? No, with TensorFlow 2.0 we are changing the programming model which builds the graphs in TensorFlow. We are removing that model where you add a bunch of nodes and rely on session.run to prune things out and figure out the precise things you want to run and replacing it with a simpler model based on the idea of the function. We are calling it tf. function because that is the main API entry point. With tf. function, many things you are used to are going to go away and I dearly hope you won't miss them. The first thing going away is you will never have to use session.run anymore. [Applause] If you have used TensorFlow execution, you know how it work. You have tensors and operations. Tf.function is just like an operation except one that you get to define using a composition of the other operations in TensorFlow however you wish. Once you have your tf.function, you can call it inside another function, you can take its gradient, you can run it on the GPU, on the TPU, on the CPU, on distributed things just like you would do with any other TensorFlow operation. We are letting you define oprations in Python and making it as easy as possible and trying to preserve as many of the semantics of the Python programming language you know and love and when you execute these functions in a graph. So, obviously the first thing you would think is that is it actually faster? If you look at models that are large conconvolutions it is not any faster because they execute plenty fast. Res models get small and you can measure the difference in performance. I show for this tiny LSTM cell there is a 10-fold speed up if you use tf.function verse if you don't use tf.function to execute it. As I was saying, we really try to preserve as much as the Python semantics as we can to make this code easy at a use. If you have seen TensorFlow graphs you know they are not polymorphic polymorphic. Python code tends to be free in the things it accepts. With tf. function we do the same. Under the hood, when you call a tf. function, we look at the tensors you are passing and see if you made a graph that is compatible, if not we make a new one. We hide them so you can use your tf.function as you would use a normal TensorFlow operation and eventually you will get all the graphs you need built up and your code will run BLAGZ blazingly fast. If you want accease the graphs you are generating you can get them. We expose them to you so you have need to manipulate the grap or do weird things I don't approve you can still do it. The main reason we changed this model, really, is not to replace session.run with tf.function but it is by changing the promise for what we do to your code, we can do so much more for you than we could do before. With the model where you add nodes to a graph and prune them it is hard for it TensorFlow Runtime to know what order you want the operations to be executed in. Almost every TensorFlow operation is stateless so it doesn't matter but for where it does you probably had to use control dependencies other things to make it work. I am here to tell you you will never have to use control dependencies again if you are using tf.function. The promise behind tf.function is you write code you would like to run eagerly, we take it and make it run fast. As we trace your Python code to generate a graph, we look at the operations you run and every time we see a stateful operation, we add the minimal necessary set of control dependencies to ensure the resources accessed are accessed in the order you want them to be. If you have two variables you are updating we will do that in parallel. If you have one variable updating many times we will order the updates so you are not surprised by them happening out of order or something like that. There is really no crazy surprises and weird identifying behavior. You should never need to explicitly add control dependencies to your code but you will still get the ability of knowing what order things execute. If you want to check something before something else, just put that line of code before that line of code like how you do normal programming. We can simplify how you use variables also. You know variables are useful and let you persist and checkpoint and all those things but they can be a little finically.Thifinicy. Another thing we are removing from TensorFlow is the need to manual initialize variables yourself. The story for variables is a little complicated because as we try to make code compatible with eager execution and graph semantics you will find examples where it is unclear what we should do. My favorite one is this one. If you run this in TensorFlow 1. X and session run repeatedly you will get a series of numbers that goes up. But if you run the code eagerly, every time you run it you will get the same number back. If I want to wrap this code of tf.function, which one should it do? Follow the 1.X behavior or the eager behavior? I think if we took a poll I don't agree with myself. This is an area. Non ambiguously created variables is perfectly OK. You can create a variable and capture it by closure in a function. That is a way of a lot of TensorFlow is written. You can write the function so it only creates variables the first time it is called. This is what most libraries in TensorFlow do under the hood. This is how keras is implemented, sonnet, and all sorts of other libraries that use TensorFlow variables. They try to take care to not create variables every time they are called otherwise you are creating too many variables and not training anything. Code that behaves well gets thrown into function and it is fine. If you have seen this, I didn't need to call the initializer for this variable I am creating. It is even better. I can make the initializer depend on the value of there argument of the FUNGSZFUNGSZ -- function in arbitarily ways and we add the dependencies to make sure the state updates happen in the way we want, so there is no need to worry about this. You can create variables like how you would use in a normal programming language and things behave the way you want. I am really happy about the autograph integration as well. If anyone has used TensorFlow with autograph it can be awkward but we are finally breaking up with the prior forms and you can write code that looks like this. I have a wild loop here and this is probably the worst way to make a tensor. If you put this in a tf. function it will put it in a graph and execute it. This is nice and great. How does it work? Under the hood things like tf. file are still there but we have tf. autograph that rewrites control expressions into something that looks like this which is not how you would want to write code. This then can be taken by TensorFlow and turned into fast dynamic graph code. How does this work? To explain that I would like to take a step back and think about how does anything in TensorFlow work. You can have a tensor and do tensor plus tensor times other tensors. Just use a tensor like you would a floating point number. Python has a thing called operator overloading that let's us change the behavior of standard Python operators when applied on our custom data types like tensors. We can add these underscores and change how TensorFlow does addition and subtraction. Python doesn't let us write underscore underscore. It makes me very sad. If you this can -- if you think about this for a few second, you can come up with rewrite rules that will let us lower to byte code with underscore-underscore if that is under writeable. If code looks like this, you can write this as condition.if a and B. You need to do fiddling of the scopes because I am sure you know Python's lexical scoping is not as lexical as you think. Names can leak out of scopes and it is a little messy but that is also a mechanical transformation. If this is potentially a mechanical transformation let's do this. We wrote this compiler called autograph that takes your Python code and rewrites it in a form that let's us call underscore underscore if and etc on tensors. This is all does but this unlocks a lot of the power of native Python control into your tensor graphs. On this function, I have two loops. One it is a static Python loop because where write four I in range. I is an integer. Autograph sees this and leaves it untouched. You still get to use Python control flow to chose how many layers your network will have or iterate over a sequential but when your control flow does depend on the properties of tensors like in the second loop for intf.range then autograph sees it and turns it into a dynamic tf.wildloop. This means you can implement this in 10 lines of code just like how you would use in a normal language which is pretty nice. And anything that you can do in a TensorFlow graph you can make happen dynamically. You can make prints and assertions happen. Notice here that I don't need to add control dependencies to make sure they happen in the right order because of the thing we were talking about earlier. We already do. We control the dependencies automatically for you to try to really make your code look and behave the same as Python code. The TensorFlow Runtime is not powerful enough to support everything that Python can do. So, for example, if you are manipulating list of tenors at run-time you should still use a tensor array. It works very well and compiles to eefficient TensorFlow code. You no longer need to write the boilerplate code associated with it. This is how you stack a bunch together in a loop. Wrapping up, I think we have changed a lot in TF 2. I hope you agree the changes are worth it. I will quickly walk through you a dif of what your code will look look before and after. Session run goes away. Control dependencies go away. Variable initialization goes away. Cond and wildloop go away and you just use functions like a normal language. Thank you and welcome to TF 2. All the examples are on tensorflow. org/alpha and dig you will find a Colab notebook with these and more and play around with the tf.function for auto. Now I would like to introduce Ryan who is going to tell you about all the ways we have of feeding data into TensorFlow. I am here to tell you about TensorFlow datasets. It is on GitHub and pipi. First a little set-up. This is the partnership care about. TensorFlow 2.0 makes it look nice. You get predictions, your loss, gradient, update, it all looks fantastic. And they are good together. Data and models. Really, I just want to focus in on this one piece. This one line. Iterating through your training dataset. In TF 2, with the training data API, you have this expressive simple way of expressing high performance but flexible in put pipelines. We might start with a TF dataset and we can shuffle and repeat and build complex pipelines and we know this will be performance. This map call might be doing image preprocessing on each record in the dataset. It is happening in parallel you can see here. TensorFlow will be spinning up a bunch of threads so the paping is high performance. At the end, you have the ability to pre-fetch. As your tight model training loop is going, you can be preparing batches ready to feed into the iteration. This is fantastic bit but unfortunately the source data is a little out of step. We want this tight iteration with four iputs and targets. What is hidden is how did you get the data into a format that you can feed into the model in the first place? Every source dataset when is all a little bit different. This makes sense when you distribute data at rest there is some natural format you would like to distribute it in. We are not interest in data at rest. We are interested in having data in a format that is ready to move into a TF data pipeline. What we really want is something that is good together. We want these two pups, data and model, to really be simpatico. It is about getting data into a format that is ready to move into a tf. data pipeline. Here is TensorFlow datasets. You import TensorFlow datasets and this load API for mnist, and we are grabbing the training split of the dataset. We are asking for supervised tuples. We will get input and target tuples. We get a tf. data dataset and we can build a complex pipeline. We are adding shuffling and batching in this case. We are off and we can iterate through our inputs and do all the modeling things we need. Load hides a lot of the complexity. What is happening here with load is that mnist is being fetched from the source, it is being pre-processed, put into a standard format, and it is being documented with statistics and everything which I will show you in a second, and then you are building the sort of front end where you have preprocessed data on disks ready to move. Typically you would only do this one and tf.ds will tash the results for you and you are getting a tf.data pipeline out of it. We don't just for mnist. We have imagenet. The first step is running this batch script, muck with the files over here, make sure you filter out these records and things like that and of course this is all taken care of for you. ImageNet is gated with a password and username and so currently TensorFlow datasets ask you to download the data and put it in a certain place and TensorFlow datasets takes it from there. We are working with cagal to get ImageNet to be downloaded without any user interaction. We are imdb reviews in this case. Note the API doesn't change. This is a tfts data call. Of course, text is a notoriously difficult format to work with when you are interested in doing machine learning on it you have to do encoding. We are shipping with three really powerful encodings one is byte level, character loading encoding and another is subwords. We have a great subword text encoder. We will get imdb reviews already encoded using a vocabulary with about 8,000 tokens. Each of these datasets is packaged together as a dataset builder. If you want to add a dataset to TensorFlow datasets, which I hope every single one of you do, there should be about a thousand new datasets in a week, dataset builder is simple. It does those exact three things lat load hiding. It has a method called download and prepare taking source data on the local web or local directory and produces preprocessed files. It takes data at rest and put it into a format of data ready to move. The second is as dataset and that takes the preprocessed files on disk and produces a tf.data.dataset. It third is metadata about the dataset. Dataset.info has the features in the dataset and each feature documents the name, shape and type. So in this case, the image for mnist is 28, 28 by 1, type int 8. The number of classes is documented in the class label feature and you get statistics on the dataset and what are the splits exposed, train and set, and how many records in each one. Mnist has 60,000 in the training and 10,000 for it testing. If you want to grab a dataset from TensorFlow datasets and filter for all the ones that are trivially supervised datasets, you can look for supervised keys and pick up the features you need. For text, again, one of these things that is a bit annoying to work with often, so this is imdB reviews. The text feature contains the encoder. We see it is a texting encoder with a vocabulary size of 8, 185. We support numpy usage as well. We want to make TensorFlow datasets really portable. You can call this tfts as umpy arrays. Coming back to our initial example, you know, we have this beautiful, you know, model and data, but of course trained dataset -- where did this come from and how did you get the data? With TensorFlow datasets a simple change and you have something that works end-to-end, out of the box, which is something we are really happy about and hope you are too. You can come find us on pypi and GitHub. The community on GitHub has gotten surprisingly active surprisingly quickly. Come join us. We could use your data if you are distributing a dataset and would like to make your dataset famous, please add it. Come help us on GitHub. There are dataset requests and you can help implement a dataset that has been requested or help us develop. We develop out in the open and develop straight out on GitHub. Thanks very much. Hopefully you guys find this useful and come help us out. Thank you. Next is something that I think is really great. Chris Lattner and Brendon telling you about Swift for TensorFlowTensorFlow. Thanks, Brian. Fantastic. Hi, I am Chris and this is Brennan and we are super excited to tell you about a new approach to machine learning. Here in the TensorFlow team, it is our jobs to push the state-of-the-art of machine learning forward. We have learned a lot over the last few years with deep learning. We have incorporated most of that all into TensorFlow 2. 0 and we are really excited about it. Here we are looking further beyond than TensorFlow 2.0. Eager mode makes it easy to train dynamic model, but deploying it still requires you take that and write a bunch of C++ code to help drive it. That could be better. Some researchers are interested in taking machine learning models and integrating them into larger applications. That often requires writing C++ code also. We want more expressive and differential mechanisms and we are excited to define reusable types that can be put into new places and used with automatic differentiate. We want to make your more productive by taking errors in your code and improving your iteration time. Now, what we are really trying to do here is lift TensorFlow to entirely new heights. We need to innovate at all levels of the stack and this includes the compiler and the language. That is what's Swift for TensorFlow is about all. We think that applying new solutions to old problems with help push machine learning even further than before. Let's jump into some code. First, what is Swift? Swift is a modern and class platform language designed to be easy to learn and use. Swift uses types which are great because they can help you catch errors earlier and encourage good API design. Swift uses type inference so it is easy to use and elegant but it is also open source and has an open evolution language processing allowing us to change the language and make it better. This is how you define a model in Swift for TensorFlow. We are laying out our layers here and we can define a forward function which com poses them together in a linear sequence. You probably noticed this looks like keras and that is no error. All we have to do to train is substantiate the model, pick an optimizer and random data and a training loop. We will write it by hand because that gives you the maximum flexibility to play with constructs and you can do whatever you want. Some of the major advantages of Swift for TensorFlow is the workflow. Instead of telling you about it, what do you think Brian? -- Brennan. What could be easier than opening up a browser tab? This is Google Colab hosted Jupyter notebooks and comes with Swift for TensorFlow built right in. Let's see it in action. Here is the layer model or the model that Chris just showed you a couple slides ago. We are going to run it using some random training data right here in the browser. We will instantiate the data and trained it using Swift for TensorFlow in our browser on training data right here. We can see the training losses decreasing over time so that is great, but if you are ever like me when I try to use machine learning in any application, I start with the simple model and I have to iterate. I have to tweak the model to make it fit better to the task at hand. So, since we are trying to show you the workflow, let's edit this model, let's make it more accurate. Here we are. Now, let's think a little for a moment. What changes do we want to make to our model? This is deep learning after all so the answer is always to go deeper. Not just sequential layer but skip connections are a good idea to make sure your model continues to train effectively. Let's go through and actually add an extra layer to our model, let's add skip connections, and we will do it all right now in under 90 seconds. Are you ready? Here we go. We need to define our additional layer first. We will fill in this dense layer. One thing you can see is that we are using tab auto complete to help fill in code as we are trying to develop and modify our model. Now, we are going to fix up the shapes right here really quick so that the residual connections will all work. If I can type properly that would go better. Great. We have now defined our model with the additional layers. All we need to do is modify the forward pass so that we add those skip connections. Here we go. The first thing we need to do is store in a temporary variable the output of the flattened layer. Then we will feed that output to our first dense layer. Dense. applied to tmp in context. For the cuda graphs here is our dense applied. Tmp plus tmp2 in context. Run that. Yes, it worksism -- works. Let's see how it does. We will reinstantiate our model and run the training loop. This loss is now substantially lower. This is an example of what it is like to use Swift for TensorFlow as you apply and develop models for applications and challenges. [Applause] Thank you. But Swift for TensorFlow was designed for researchers and researchers often need to do more than just change models and change the way architecture fits together. They need to define entirely new abstractions and layers. Let's see that live right now. Let's define a new custom layer. Let's say we had the brilliant idea we wanted to modify the standard dense layer that takes weights and biases and add an additional bias set of parameters. We will define this double bias dense layer right here. I am going to type this quickly. Stand by. 15 seconds. Here we go! [Laughter] That was great. Let's walk through the code so you can see what is going on. The first thing that we have is we define our parameters. These are w, like our weights for our neurons and B1 and B2. We define an initializer that takes an input and output size just like dense does and we use that to initialize our parameters. The forward pass is very simple to write. Here is just aplied to and we take the matrix multiplication of input by our weights and add in our bias terms. We have just defined a custom layer in Colab in just a few lines of code. Here is model two and we will use our double bias dense layer and we are going to instantiate and use our custom handwritten training loop. Because TensorFlow can statically analyze your code, it can be really helpful to you. I don't know about you but I regularly put typos in my code and Swift for TensorFlow is helping you out. It is saying look, you mistyped softcross-in. This is not the right idea but this is an example of how easy it is for researchers to experiment with new ideas really easily in Swift for TensorFlow but let's go deeper. Swift for TensorFlow is, again, designed for researchers and researchers need to be able to customize everything. That is the whole point of research. So, let's show an example of how to customize something other than just a model or layer. You may have heard that large GPU clusters or TPU super pods are, like, delivering massive breakthroughs in research and advancing the state-of-the-art in certain applications and domains. You may have always heard as you scale up you need to increase your batch size. Let's say you are a researcher and you want to try and figure out what are the best ways to train deep neural networks at larger batch sizesment if you are a researcher, you probably can't guy a whole GPU cluster or went a whole TPU super pod all the time for your experiments but you often have a GPU under your desk. Let's see how we can simulate that on a single machine. We will do it all in a few lines of code here. Here is our custom training loop and the standard part. This is our 1-10 training epics. What we will do is instead of just applying our model forward once, we have an additional interloop, right? We will run our forward pass, we will run our model four times, and we are going to take the gradients for each step and aggregate them in this grads variable. This simulates running on four independent accelerates in a data parallel fashion on a batch that is four times as large as what we actually run. We will then use our optimizer. That is all there is to it. We are really excited by this flexibility and capabilities that Swift for TensorFlow brings to researchers. Back over to you, Chris. Thanks, Brennan. [Applause] So I think the focus on catching errors early and also productivity enhancements like code completion can help you in a lot of ways. It's not just about automating typing of code, but it can be about discovery of APIs. So another thing that is really cool about Swift as a language is it has good interoperability with C code. You can import header files and call directly from C without wrappers or boilerplate or anything involved. It just works. We have taken this approach in the TensorFlow and brought it to the world of Python. One of the cool things about this is that allows you to combine the power of Swift for TensorFlow with all the advantages of the Python Eco system. How about we take a look? Thanks, Chris. The Python data science ecosystem is incredibly powerful and vibrant. We wanted to make sure you didn't miss your favorite libraries and utilitiesio you are -- utilities you were used to. Let's see how this works in context to numpy. We import py plot from the library and numpy and assign it to np. After that we can use np just as if we were in Python. We will call sine and cosine and pass them to py plot. When we run the cell, it just works exactly as you would expect. [Applause] Thank you. Now, this sort of kind of looks like the Python code you are used to writing but this is actually pure Swift. It just works seamlessly. This is maybe a bit of a toy example. Let's see this a little bit more more in context. OpenAI has done a lot of work in the area of reinforcement learning and developed a Python library called OpenAI gym which contains a collection of environments that are very useful when you are trying to train a reinforcement agent across a variety of different challenges. Let's use OpenAI gym to train a reinforcement agent in TensorFlow right in our browsers. The first thing we need to do is import gym and define a few hyperparameters and now define our neural networks. We will pick a simple two layer dense network in this case and it is just a sequential model. After that we have helper code to filter out bad or short episodes and whatnot. Here is the real meat of it. We will use gym to instantiate the cart pole environment and we will instantiate our network in the optimizer and here is our training loop. We are going to get a bunch of episodes, we are going to run our model, get the gradientgradients and apply those to the optimizer and record the mean rewards as we train. Very simple, straightforward Swift. And here you go see us training a Swift for TensorFlow in an OpenAI gym environment using the Python bridge. Totally seamless. Afterwards, you can keep track of the parameters of the rewards. In this case we will plot the mean rewards as the model trained using Python Numpy. Totally seamless. You can get started using Swift for TensorFlow using all the libraries you know and love and take advantage of what Swift for TensorFlow brings to the table. Back over to you, Chris. Thanks, Brennan. One thing I love about this is it is not just about leveraging big important libraries. We are working on the ability to integrate Swift for TensorFlow and Swift for Python together which we think will provide a nice path to help you incrementally move code from one worth to the other. I think it is fair to say calculus is an intergral part of machine learning. We think differentiable programming is so important we built it right into the language enabling more flexible and custom work with derivatives. We think this is really cool. I would like to take a look. We have been using Swift for TensorFlow's differential programming throughout the demo but let's really break dedown. Here we define my function that takes two doubles and returns a double based on some products and sums and quotients. If we want Swift for TensorFlow to automatically compute the derivative for us, I would annotate it at differential. Swift for TensorFlow will then derive the derivative for this function right when we run the cell. To use this auto generated derivative, use gradient which takes a closure to evaluate and a point you want to evaluate your closure at. Here we go. This is what it is to take the derivative of a function at a particular point. We can change this around. This one is my favorite. Tasty number. That works nicely. One thing to note is we have been taking the partial derivatives of my function with respect to A. Of course you can take the partial derivative and get a full gradient of my function like so. Often with neural networks, however, you want to get not just the gragradients, you often want what the network predicted? This is useful to compute accuracy or other debugging sort of information. For that, you can use value with gradient. That returns a tuple containing the value and gradient shockingly enough. One thing to note, in Swift tuples can have named parameters. You can see it prints out nicely and you can access values. We think this is another nice little thing that helps make writing and debugging code and more importantly reading it and understanding it later a little bit easier. The one thing I want to call out is throughout this we have been using normal types. These are not tensor of something. It is just plain old double. This is because automatic differentiateation is built right in and it makes it easy to express your thoughts very clearly. But even though it is built into the language, we have worked very hard to make sure automatic differiation is customizeable. Let's show you. Let's say you want to define an algebra in 2d space. You are certainly going to need a point data type. Here we define a point instruct with X and Y and mark it differentiable. We can define helper functions on it like dot or other helper functions. Swift for TensorFlow when you try and use your code it will often automatically infer when you need gradients to be automatically computed for you by the compiler. It is a good idea to document your intentions so you can annotate your helping functions as at differentiable. The other reason we recommend doing this is because it helps catch errors. Swift for TensorFlow is telling you that hey, you can only differentiate functions that return values that conform to differentiable but int doesn't conform to differentiable. This is telling you my helper function returns an int and int is about taking small steps and integers are very discreet. Swift for TensorFlow is helping to catch errors right when you write the code and tell you what is going on. The solution, of course, is to just not mark it as a differentiable and the cell runs just fine. Let's say we also wanted to go beyond just defining the dot product. Let's say we wanted to define the magnitude helper function. That is the magnitude of the vector defined by the origin to the point in question. To do that, we can use the distance formula and we can define the extension on point that does this. We will pretend for a moment that Swift doesn't include a square root function because I want a good excuse for you to see the interopera interoperability with C. We will use C's square root function. We can define the magnitude and it -- no, it doesn't quite work. Let's see what is going on. We wanted magnitude to be differentiable and it is saying you can't differentiate the square root function because this is an external function that hasn't been marked as differentiable. The square root is a C function compiled by the C compiler and that can't automatically compute derivatives for you. This is saying it will not work and this is excellent because it gives me a great excuse to show you how to write custom gradientss. We define a wrapper function here that calls to the C square root function. The backwards pass, we take our double, and we return to a tuple of two values. Rather the first element in the tuple is the normal value and the forward pass. The second a pullback closure and this is where you define the backwards pass capturing whatever values you need from the forward pass. We are going to run that. We are going to go back up to the definition of magnitude and change it from square root to my square root, rerun the cell and it works. We defined point, dot and magnitude. Here I defined the silly function and marked it as differentiable. We are going to take two points, we are also going to take a double, right? You can mix and match differentiable data types totally fluidly. We will do magnitude and dot products. It is a silly function. We can use it, compute the gradient of this function at arbitrary data points. You can get the value of there function and full gradients in partial derivatives with respect to individual values. That has been a quick run through of custom gradients J custom data types with the differentiate built in. Let's put all this together and show how you can write your own debuggers as an example of how this power is all in your hands. Often when you are developing models, you want to -- or debugging models, you want to be to see the gradients at different points within your model. We can define in regular Swift code a gradient debugger. It will take as input, a double, and it is going to return it just like normal for the forward pass, right? It is an identity function. We will get the gradient, print it and return it. We are just passing it through printing it out. We have defined this gradient debugger and we can use it in our silly function to see what is going on as we take derivatives. Gradientdebugger, here we go. We can rerun that. And when we take the gradients, we can see that for that point in the silly function, of a. b, the gradient is 3.80. That's been a brief through of how automatic differiation works and how you can extract the power to build whatever systems you need. Back over to you, Chris. The algorithms we are defining were built in the 1970s. There is a tremendous amount of depth and I am excited to see what you all can do it about. If you are interested in learning more, we are detail documents online. The language of Swift has low level performperformance and there is no gill to get in the way of concurrency. They have advanced compiler techniques to identify and extract graphs for you. The consequence of this together is we think Swift has the world's most advanced eager mode. You may wonder why do we care about this stuff? We are seeing trends in the industry where people are defining neural networks. This requires you to export graphs and write a bunch of C++ code to load and orchestrate them in various ways. Let's look at an example of this. Alpha go- zero is amazing work. You have deep learning and it drives through it Monte Carlo. It is the combination of all three things that make AlphaGo Zero possible. This is possible today. If you are an advanced team like DeepMind you can do this. We think breaking down barriers like this can lead to new breakthroughs in science and drive progress forward. Instead of talking about it, again, let's take a look. MiniGo is an Open Source Go player inspired by DeepMind's AlphaGo Zero. The miniGo project were writing everything in normal TensorFlow and it was working great until tay started trying to run at scale on large clusters of TPUs. They ran into performance problems and had to rewrite things into C++ to effectively utilize the modern accelerators. We have reimplemented Monte Carlo research and the rest of MiniGo player in Swift. We define a helper function, these are our white and black players, we are going to run basically play the game until we have a winner or looser. So, let's actually run this. We define a game configuration and we will play between a Monte Carlo tree search versus a random player to see how easy it is to flip back and forth or mix and match between deep learning and other arbitrary machine learning algorithms right here in Swift. Here you go. You can see them playing white, black, playing different moves back and forth and it just goes. We think that Swift for TensorFlow is going to unlock whole new classes of algorithms and research because of how easy it is to do everything in one language with no barriers and not rewriting everything into the C++. Back to you, Brennan. The cool thing about this is you can do something like this in a work workbook. We have seen families of techniques that can be bound together and fused and bringing this to more people we think will lead to new kinds of research. Our work on usability and design is not just about high-end researchers. We love them but Swift is widely used to teach new programmers to learn how to code. I am excited to announce a new journey we are embarking on with Jeremy Howard. At fast.ai we are always looking to push the boundaries of what is possible especially pushing to make recent advances more accessibility. We have been involved with this and building the world's best document classifier. Hundreds of thousands have become deep lun learners. We think with Swift for TensorFlow we can go even further. We are announcing a new program caught by someone that knows Swift well. Chris, I think he means you. I am super excited to help teach the next generation of learners and excited Jeremy is bringing his expertise in Swift and helping us shift the higher level APIs. The most important part is Swift for TensorFlow is really TensorFlow at its core and we think this is super important. We have worked hard to make sure it integrates with all things going on in the big TensorFlow family and we are very excited about that. You may be wondering where you can get this. Swift for TensorFlow is Open Source. You can find it on GitHub now and you can join our community. It also works great in Colab as you have seen today. We have tutorials and examples and the demos you saw today are available in Colab. We have released our 0.2 release including all the basic infrastructure and underlying technology to power these demos and examples and we are actively working on high-level APIs right now. This is not ready for production yet as you can guess but we are excited about shaping the future, building this out, exploring this new program model, and this is a great example for advanced researchers to get involved and help shape the platform. We would love for you to try it out and let us know what you think. Next up, I would like to invite Edd to talk about the TensorFlow community. Thanks, Chris. Hey, everybody. I work with TensorFlow helping build community and collaboration around the open source project. Usually I put the thank you slide at the end for listening to me but actually I want to thank you for contributing and being part of the TensorFlow project. Whether you are here in this room or on the livestream, there has been some amazing talk on YouTube from people in China, and India, and Japan, and all over the world joining us virtually today. Thank you for your contributions. The project is where it is today because of you. Of course, in core TensorFlow alone we have had this many commits. This figures out over 50,000 from over 18,000 contributors and much more than just code commits. 39,000 stackoverflow questions about TensorFlow, we have 66 machine learning Google developer experts, many of whom who are here with us today. Welcome and thank you guys. It is really great to have you with us and thank you for everything you do helping teach people about TensorFlow. We have had 14 guest posts to the TensorFlow blog and that keeps going up. There are so many ways people are contributing. Thank you. You are really helping build out the TensorFlow ecosystem. I want to discuss how we are growing the Eco system and report back on some of the changes we have made over the last year. I am going to cover how we are making it easier to get involved in TensorFlow and how we are trying to consult betwer the users in the community and be more transparent about our development. Going to cover how we are empowering everybody to get involved and to do more and increasing in the number of contact points. I will go a bit more into depth about the conference announced today. Let's talk about how we are making contributions to TensorFlow easier. One of there most important thing is increasing its modularity. We are trying to make it less of a mono lift. When you come and want to contribute to an open source project it helps to know where to go. We are creating more surface area. Our special interest groups play a big part in this and I will talk a bit more about them later. It is not just code. There is so many more places to contribute this year compared to last year. I will talk about the documentation groups, groups involved in testing, groups who are blogging and on YouTube and more. I was super excited to see we published TensorFlow tutorial in Korean. That isn't something we did on our team. That is part of the community. We were similarly able to publish it in Russian thank you to Andrew Stepin. I am also really excited about the TensorFlow 2.0 testing group led my Paige Bailey. This is a bunch of contributors and Google developer experts who are working to get TensorFlow 2.0 a test. You see an example of friction load and folks are going through ML loads and documenting what they find awesome and also things that could be a little better. This group meets weekly and often has guest talks from maintainers and SIGs leaders and it helping bring TensorFlow 2.0 from cutting edge to something thoroughly tested and ready to use. We mentioned we have over 14 posts from guests on the TensorFlow blog. This is from a great post about real-time segmentation with TensorFlow.js that comes from a grad student and researcher at itp. Whether it is testing, whether it is documentation, whether it is blogs or conference talks, thank you. Now, I want to talk a little bit about TensorFlow RFCs. As you probably know, RFC means request for comments. This time last year, we were not that organized about how we evolved TensorFlow's design in terms of communicating and I stood on this stage and told you about launching the RFC process. We have accepted 21 of them over the period of the last year. This is our way to communicate design. We post an RFC about a design and consult widely. This isn't just about code coming outwards, it can be created on and commented on by anyone. We have several RFCs brought by the broader community and I expect to see several more. One of the things I am most proud about is how the RFC process is underpinning the 2.0 transition. All the major changes have been proposed and consulted with in RFCs. This isn't just a great way of consulting and getting feedback, going forward you have a big repository of documentation about why design choices were made a certain way in TensorFlow and great educational results for people who are coming on and want to get involved in contributing to the project. I really want to give a big thanks to anyone who authored or reviewed an RFC. You played a vital role in making TensorFlow better. Let's talk a bit about the social structure of TensorFlow. You know, last year I talked about how coming to a large project can be a little bit daunting. You don't know where people are, whether people have your interest in common. We created the special interest groups or SIGs as a way of organizing our work. There are so many uses of TensorFlow, so many environments and architectures. Many of them are outside of the scope of the core team and we want to enable TensorFlow to grow and be more sustainable by creating a way for like-minded people to collaborate. This is why SIGs exist. They are groups of people working together for a define project focus. We started last year and we have six up and running now. I am going to give you a quick state of the SIGs and many, in fact most of the leaders, are here with us. I will give a shout out to them and hopefully you will be able to talk to them in the launch and tomorrow. SIG addons first. Martin mentioned at the beginning of the day that TF contrib is no longer a part and SIG addons are where that is going. They conform to these well defined APIs. Most there is already an RFC posted about where you can find things that you used to find in contrib. How can you get involved if you have a favorite and want to step up and be a maintainer for everybody and how you can join in the project. I encourage you to take a look at that. SIG build is where TensorFlow meets the outside world. It is not always the most glamorous piece of work but building it and packaging and distributing it is tough. Thank you to Jason and Austin who lead that SIG. It is home for third party builds for architectures we don't ship. IBM power and SIG helps us be a better neighbor in the Python ecosystem. Machine learning generates a lot of extreme situations and changes in the way we package and distribute software. SIG-IO helps connect other systems and your data NLT real world exists using systems in other formats. This group is led my yang tan and antoine. If you are using any of these in the Apache Eco system or the environments you can use this addaddon. This group has already dropped four releases and shipped our integration with their module too. SIG networking is where a lot are going to. If you are using gdr verbs this is where you can find this. If you are interested in this, or any of the other SIGs, please talk to the leaders. They want help and are up and running and in a great place to bring people on. If you are looking for a way to get involved in TensorFlow, this is an ideal one. Let me talk about SIG TensorBoard. This is a great time to be involved as the TensorBoard team is trying to figure out how we can best enable people using TensorBoard for creating plugins or at scale. If you go to the demo area above and go to the TensorBoard stand you will find Gal and Moni there who would be happy to talk to you. This is the URL for anything you want to do joining the TensorFlow community from docs to testing to SIGs and all the other ways to be involved. The developer mailing list. Please head there. If you are here with us, rather than on the livestream, we are doing a contributor luncheon tomorrow where many of the core team and SIGs will be there to talk to you. Finally, let's move on and talk about TensorFlow World. I am so excited about this. It is our vision to bring together the amazing people who are part of our community and give space for everyone to connect with each other. There is so much we can learn from how we are all working with TensorFlow. Working with O'Rielly media, we will have this event at the end of October here in Santa Clara. It will be four days that really celebrate the TensorFlow ecosystem. We will have content, talks, tutorials, there will be an expo and place for vendors to present. We understand TensorFlow gets out into the real world there is a large ecosystem beyond folks in this room. We are really excited and that means we can bring everyone together. The main point of doing something like this is to connect all the amazing users and everyone with experience to share. As you heard the call for proposals is now open. If you have anything to share with your work, your product, your company, head to TensorFlow World and put in a proposal. You have about 4-5 weeks while the call for proposals is open and we will be selecting talks a few weeks after that. I really hope you will lend your voice to this event and I am so excited to see you in October. Once again, thank you. We appreciate how much you are part of our community. In my job, the best thing every day is when I get to talk to folks developing or using TensorFlow. It is so exciting to work with everybody here. This is the way you can meet me. Please do. If you have any issue or devire to contribute, reach out. We are really happy to talk to you. Thank you very much for your attention. -- desire. [Applause] Hey, everybody. Thank you, Edd. That was wonderful. We are at another break. Come back here at 2:25. We will be talking about TensorFlow in production. Then the next block of talks is about probability and so forth. Don't forget to go upstairs. We have deem demos of all kinds of stuff and Googlers galore. Check it out. Thanks a lot. . >>PAIGE BAILEY: Eager execution and all of the things that we have loved to come to know and love through the last several iterations of TensorFlow development. We have a lot of tutorials and documentation that have been recently released, it is about making you productive as quick as possible, and giving you scalable, reproducible learn ing. >>LAURENCE MORONEY: Yes, and with it being open source, of course, is the community is just a huge part of what we do. >>PAIGE BAILEY: We could not do it without the community. >>LAURENCE MORONEY: And that is why we are doing the live stream. >>PAIGE BAILEY: If you have questions, #askTensorFlow is the hashtag. >>LAURENCE MORONEY: And one question that we are going to cover, you will get today, it is great to do my training for a fixed number of epochs, but when I reached the desired accuracy metric, how do I continue training? >>PAIGE BAILEY: It is not easy to compute if you have gotten to a point where your boss says, okay, cool, 99 percent accuracy, that is fine for us today. >>LAURENCE MORONEY: Should we take a look at how we are doing that? >>PAIGE BAILEY: Absolutely. >>LAURENCE MORONEY: I opened up a codelab where you can see where I'm causing call backs, and call backs are the ways that you would achieve this. And at the top, you can see the class, the call back. And then when an epoch ends in training, I can look at the logs. If the accuracy log is greater than 60 percent, he is really happy that I'm on the training, 60 percent accuracy, then I would cancel the training. And then to be able to set it up, I would create an object, call backs, an instance of the class, and I love the click-in, the little things that I love, and call backs equal call backs, and when I do the training, I will show off and make my run time tied to the GPU, so it is really fast. So let's do a little bit of training with this one, so I'm doing this live and I'm connecting to the VM, it is training, getting ready to -- >>PAIGE BAILEY: The RAM and disk utilization. >>LAURENCE MORONEY: We are on the first epoch, that is progressing away, 60,000 images being trained , and I hit accuracy of 83, which is really good. But it reached 60 percent accuracy, so I canceled the training. Call backs are helpful, before I learned about call backs, it is like, I would set something up to train for 100 epochs, go to sleep, wait up the next morning, and after three epochs, it didn't do its job and I wasted the time. >>PAIGE BAILEY: Yes, call backs are incredibly invaluable, it does not just apply to accuracy either. There's a bunch of additional metrics that can be useful for your particular workflow, and this code works in TensorFlow 2.0. >>LAURENCE MORONEY: Absolutely. >>PAIGE BAILEY: So it is Caras, I love Caras. >>LAURENCE MORONEY: It is a love affair, and one neat thing about Caras that you might not realize, is the same code for TensorFlow 1.x is the same for 2.0, so what is going on behind the scenes is it is equally in 2.0 instead of a graph mode. So this codelab, 1. 13, this code will run in 2.0 without you modifying the code. >>PAIGE BAILEY: Absolutely. >>LAURENCE MORONEY: Should we take the next question? >>PAIGE BAILEY: Cool. So the next question is from Twitter it looks like. And what about all the web developers who are new to AI ? Does the version 2.0 help them get started? >>LAURENCE MORONEY: Oh web developers. >>PAIGE BAILEY: I just finished talking to two folks from the TensorFlow JS team about the cool new things they have seen from the community, a vibrant ecosystem of artists and creators that are using browser-based and server-based tools to create these machine learning models, training and running them. >>LAURENCE MORONEY: Yep. And for web developers, there's a whole bunch of ways you can get started with this. So you mentioned TensorFlow JS, we will start with that, but it is a JavaScript library. And this will allow you to train models in the browser, as well as executing them. And that blew my mind. >>PAIGE BAILEY: And the node bindings, being able to use the GPU inside your laptop with, you know, Google Chrome or your favorite flavor of browser, to train a model . That is absurd. >>LAURENCE MORONEY: And node bindings as well, with node.JS. It is not only in JavaScript, but server-side JavaScript with node. And am I supposed to say Node.js? I will say node. And, of course, by the fact that it is a node, one of my personal favorites, are you familiar with Cloud Functions for Firebase? >>PAIGE BAILEY: I'm not, I'm intrigued. >>LAURENCE MORONEY: I used to work on the Firebase team, so shout out to my friends on Firebase. >>PAIGE BAILEY: I have heard so many good things about Firebase. >>LAURENCE MORONEY: It is for mobile and web developers, and one of the things that Firebase gives you are these things called Cloud Functions for Firebase, I called up the web page, and you can execute the functions on the back end without maintaining a server infrastructure and execute them in response to a trigger, such as an analytic event, or a sign-in event. >>PAIGE BAILEY: Or you get new data and you have to process it. >>LAURENCE MORONEY: There you go. And the fact that they run JavaScript code on the back end, and now you can train models in a cloud function. >>PAIGE BAILEY: That's amazing. >>LAURENCE MORONEY: So web developers, a lot of great options for you. However it is you want to do it, in the browser, the back end, you can get started with it. >>PAIGE BAILEY: And TensorFlow 2.0, if it gives additional tools for application developers, I think it would mostly be in terms of those codes and tutorial that we are mentioning before. And we also released some courses. >>LAURENCE MORONEY: Yep. >>PAIGE BAILEY: So it is easier than ever to get started. And, um, the models that you create can be deployed to TS lite. >>LAURENCE MORONEY: And we have been talking about Caras, and the layers are supported in TensorFlow.JS. It is not just for Python developers, if you are a JS developer, you can define your layers. >>PAIGE BAILEY: And an R developer, they have Caras for R, which was created by JJ Layer and Francois Chalet. >>LAURENCE MORONEY: A lot of options for developers. And next question, from Twitter, are there any JS learning examples from object detection? >>PAIGE BAILEY: Node.JS is popular, we have learned. >>LAURENCE MORONEY: So object detection. How do we answer this? So I'm -- it depends on what you mean by object detection because, in Google, we talk about object detection and use that term for, in an image, you have a lot of objects and bounding boxes for them, there are samples for that. >>PAIGE BAILEY: And a lovely thing is the community is adept at creating TensorFlow. JS examples, for example, code pens, and Victor Dibia, a machine learning expert, for tracking hand movements. So if you want to create -- >>LAURENCE MORONEY: You have an incentive. >>PAIGE BAILEY: I have a limited edition collection of TensorFlow socks, and I would love to send you a bright, shiny pair of limited edition socks. >>LAURENCE MORONEY: They are classics, they have the old logo. >>PAIGE BAILEY: Not the new logo, but the old box style ones. >>LAURENCE MORONEY: I love them both. And, like the question was about transfer learning, and we did not have a demo of object detection, I would love to show Transfer Learning and being able to detect a single item in the frame. To do that, we call image classification. So can I roll the demo? >>PAIGE BAILEY: Please do. Are you going to do your favorite -- >>LAURENCE MORONEY: I am a child of the '80s, I love Pac-Man. So you will notice that it says loading mobile net now, so it downloaded the mobile net model. So I'm going to add some new classes to the mobile net model and then use transfer -- >>PAIGE BAILEY: There it goes. No, no, go for it. >>LAURENCE MORONEY: So Pac-Man, old arcade game, you try to move away from the ghosts. I will train it to move up when I'm pointing up, I will gather samples, about 50, when I go right like this, and turning left is going to be hard. I didn't think it through. Bear with me, so maybe I will do left like this, get my head out of the way, a few samples like that, and then down will look like this, and hopefully these aren't weird gestures in some country. And something like that, okay. So I have now picked 50 samples of these, I will retrain these in the browser with transfer learning. So if I look at the rate, I will start training and I will see -- it starts going quickly, you can see the loss started at 4, it went town down down down, all to zero, and probably a digit beyond the 6 digits, never at zero, it is a very low loss, now we will give it a try. I will stop playing the game, move left, you can see the bounding box around it, kind of shows that I'm up, right, watching the screen. And now you can see that I have trained it, going right this time. Right, right, and up. And it thinks it is down, up, and right. And I'm not good at the game. >>PAIGE BAILEY: You are using it as an excuse to play Pac-Man all day. >>LAURENCE MORONEY: It is a great example of transfer learning, and you can see in JavaScript how easy it is to extract the features from mobile net and then retrain it, and it is moving as I'm talking. And so enough fun with Pac-Man -- >>PAIGE BAILEY: Transfer learning can be used for a variety of cases. So be sure to look at the examples listed on the website. And this is on Twitter, which is very popular. Hi, dear ask TensorFlow. >>LAURENCE MORONEY: We have to respond to that, they said dear. >>PAIGE BAILEY: Yes, are you going to publish the updated version of TensorFlow, the poet's tutorial from Pete Warden on TS2. 0, TS lite, and a lot of other shenanigans. >>LAURENCE MORONEY: And the network API. >>PAIGE BAILEY: I love Pete Warden's codelab on TensorFlow for poets, and he had a great talk today. >>LAURENCE MORONEY: I didn't get to see it, I was preparing. >>PAIGE BAILEY: It is about creating tiny models and putting them on computers on mobile and embedded devices. >>LAURENCE MORONEY: So at some point, we will update it. I don't think there is an updated version available, but one of the things that I liked about the TensorFlow for poets codelab is it got me building a model quickly for the mobile device. And the draw back of that is there are a bunch of scripts that I ran, I didn't know what I was going on with them. And I decided a bunch of TensorFlow lite examples, image classification, object detection, image recognition, and speech detection. They are all open source, Android and iOS, and they have full instructions on how to do it yourself. The image classification is fun, and I will try and run that in my, um, Android Emulator, so we can see it running in an emulated environment. So let me get that started. You can see it being cached. For example, here, it is doing a classification of what is in the background, if I hold up a water bottle, this way, it detects it is a water bottle. This is running in the Android Emulator, okay? And this is using TensorFlow lite, and this is the sample on there that does the same thing that you would seen in TensorFlow for poets, it is using mobile net and building an application around mobile net. If you look, running in the emulator, I'm getting inference times in the 100 and 170 milliseconds. >>PAIGE BAILEY: The ability to take large scale models and sort of pull them done to a manageable side on a mobile, or an embedded device, is huge, and I'm excited to see what TensorFlow lite does this year. >>LAURENCE MORONEY: So we are working on a bunch of tutorials, both samples are out there, if you look at the GitHub page, you will see that there are -- there are details on how it is built. Let me go back on here. And there are details how it is built, how you can put it all together and compile it. You don't need to use build TensorFlow to use TensorFlow lite, so you can go and start kicking the tires on these applications for yourself. >>PAIGE BAILEY: Excellent. >>LAURENCE MORONEY: All right. We will have more codelabs, I would love to get Pete's TensorFlow for poets updated, hopefully for I/O. >>PAIGE BAILEY: And watch Pete's talk. It was so good, people were talking about putting models on comadors. I was nerding out a little bit. >>LAURENCE MORONEY: And that's why we're here. Cool! So -- do we have time for one more question before break? So we are heading to break now, so Paige and I will be back after the break with the rest of your questions. Please keep asking them on the live stream and on social media. Like I said, anything that we cannot get to, we will try to get to offline and we will reach out back to you. >>PAIGE BAILEY: Absolutely. So super excited to have you here, super excited to be at the TensorFlow Developer Summit. >>LAURENCE MORONEY: And we will see you back here in a few minutes. >>PAIGE BAILEY: Yep. [Music]. PAIGE BAILEY: Welcome back. I'm Paige Bailey. LAWRENCE MORONEY: I'm Laurence Moroney. PAIGE BAILEY: We're here to answer your TensorFlow questions with #askTensorFlow. LAWRENCE MORONEY: And we will try to get back to you today, if not, we will reach out to you later to answer them. PAIGE BAILEY: We will go to the first question. LAWRENCE MORONEY: This is on Twitter, once I installed TensorFlow GPU, it did not work at first, failed to run native TensorFlow run time. I remember seeing this one, it was on YouTube in response to a pip install TensorFlow GPU in codelab, because once upon a time, you had to pip install TensorFlow GPU to use it, and now you are running into issues, and the reason is good, it is good news, you don't have to run it anymore. So if we switched to colab, this is a notebook that I was showing you earlier on. If you wanted to use GPU in colab, pick the run time type, pick GPU as the hardware accelerator and now you don't need to pip install TensorFlow GPU, it does it under the hood behind the scenes, it is really cool and earlier why I trained this so quickly, I used the GPU and there is no pip install GPU on here. PAIGE BAILEY: When we were installing TensorFlow 2.0, we had an issue with the GPU install, and you needed specific Kuta drivers. Kolob is a great tip if you -- LAWRENCE MORONEY: >>AND: GPU stuff, this is what I ran into a number of times when using the GPU, you have to carefully take a look at the version of Kuda and see the DNN that you are using, because I made the mistake that I just went to the vendor's website, I downloaded the latest version, I installed them, and then I saw TensorFlow was actually supporting a slightly earlier version. So if you do get an error when you are using GPU, take a look at the version of the driver it is looking to support and then from the VEPD er vendor's website, download that version. PAIGE BAILEY: Driver issues, they are always a treat. LAWRENCE MORONEY: Exactly. And it is like -- it is one of the things that makes our job interesting. PAIGE BAILEY: Absolutely. LAWRENCE MORONEY: Haha, all right. Should we look at the next one? PAIGE BAILEY: Oh, let's go. Adi Kumar had at least eight excellent questions on Twitter. LAWRENCE MORONEY: He wins the volume award. PAIGE BAILEY: He absolutely does, we will get to all of them, not in this Ask TensorFlow segment, but we will focus on one for today and then answer the rest offline. LAWRENCE MORONEY: And I think a lot of them were about file formats and how do I use different file formats. Should I drill into that? PAIGE BAILEY: Absolutely. The question is, which is the preferred format for saving the model going forward, saved model, or something else? If we look at the laptop, we can take a gander at one of the slides from the keynote this morning, that Keras is a driver of 2.0 and this is at the heart of the deployment. You can see it for TensorFlow serving, lite, and JS, and a lot of other language bindings. We are pushing for the saved model. LAWRENCE MORONEY: And you cannot go wrong. PAIGE BAILEY: It is easier to use than the other deployment options we have seen before. LAWRENCE MORONEY: So I think the guidance is in the recommendation, not just for AD, but for everybody else. When you are thinking about saving your models, take a look at save model, consider using it, and not only is the advantage of the file format, but just how it is supported across all of these. PAIGE BAILEY: And an excellent point of TensorFlow 2. 0, I will keep selling it, we have a number of code samples and tutorials today on saved model. LAWRENCE MORONEY: I was playing with the lite stuff, saving as a saved model, and the TensorFlow lite process was a lot easier for me. So it is really refined, we are iterating on that, and I think it looks really cool. PAIGE BAILEY: Yep! Excellent. LAWRENCE MORONEY: So thank you for all of those questions, Adi, we will try to answer the rest of them, most of them were focused around the file format and hopefully saved model will help you. PAIGE BAILEY: Let's go to the next one. LAWRENCE MORONEY: This is from Eli Gaguba, is it possible to run TensorBoard on codelabs? PAIGE BAILEY: You are going to be so delighted. LAWRENCE MORONEY: Because before if was running on codelabs, we were talking about, we really want to run codelabs. PAIGE BAILEY: It is so painful! If you wanted to get it working in a collab notebook, it was not really approved by our bosses or in general. But, yes, so the good news is that you can run TensorBoard in codelab. LAWRENCE MORONEY: And before it was publicly announced, when it was announced internally, we got an email from Paige with a lot of smily faces. So thank you for your question, it made Paige's day. PAIGE BAILEY: And I'm downloading files in hopes to play with it a little bit, but here you can see it working. You should be able to do different operations like smoothing , and changing the values and using the embedding visualizer from the Collab notebook to improve your accuracy and model debugging. Another thing that we are working very, very hard on is you don't have to specify ports. You don't need to remember if you have multiple TensorBoard instances running, that you are using 6006, or whatever, for another. It automatically selects one that's a good candidate, and it would create a good fit for you. So the team is phenomenal, if you have any interest whatsoever in TensorBoard at all, I recommend stalking the POs, like I do. That's how I found out that TensorBoard was added to Jupiter notebooks and also to codelab. So it is so exciting, we will have this link in the documentation as well, the little notes underneath for you to play it. LAWRENCE MORONEY: It is so great to have a PR stalker [ laughter ]. PAIGE BAILEY: I get push notifications in my phone, it is a problem. But they are doing such great work. LAWRENCE MORONEY: So yes, TensorBoard is working in Collab. PAIGE BAILEY: And also project Jupyter notebooks! TensorBoard everywhere. LAWRENCE MORONEY: And we all love it. I have not played with it that much, do the plugins work? PAIGE BAILEY: So TensorBoard is a collection of different visualization, you can see scalers, accuracy, histograms, you can see embedding visualizers which allows you to do a great Mnist example from TensorFlow a couple years ago, it is beautiful, and beholder, created by a community member. And the plugins, a couple of the Google Summer of Code projects this summer are focused on getting additional visualization plugins added to TensorBoard. And so in addition to the what if tool, and in addition to beholder, you can make your own TensorFlow plugin. LAWRENCE MORONEY: And what while I'm geeking out about it all day, I'm sure we need to move to another question. So the next question that came in: How would you use future columns with Keras? PAIGE BAILEY: I know you know the answer to this question. LAWRENCE MORONEY: And it is a passion of yours. Feature columns are a part of estimators and are a way of getting your data efficiently into estimators, and with people saying, like, hey, it is all there in estimates, what about Keras? You have been working on great stuff around that. PAIGE BAILEY: And I know that the YouTube channel has a series from Karmel. LAWRENCE MORONEY: Karmel is the engineering director for high-level APIs, she has a great series around high level APIs for TensorFlow, and is teaching about that and her and her team are working on parity for columns on TensorFlow 2. I have not checked the alpha yet, but it is on the way, if it is not there already. So you should be able to use them in Keras. PAIGE BAILEY: Yes, we are in the process of building out, if you wanted to migrate your model, using estimators to be more of a TensorFlow 2. 0 with Keras, I'm in the process of building a migration guide. If you have any interest around that, please feel free to reach out and we're excited to get that released pretty soon. LAWRENCE MORONEY: I'm excited to see that, personally, because the very first stuff I did in TensorFlow was with estimateers before I learned Keras, and I wanted to change them to use Keras without re-writing them. PAIGE BAILEY: Absolutely, Keras is just friendlier to work with, I feel. LAWRENCE MORONEY: They are both greatest estimators, and Keras is great for beginners, so hopefully we will get the best of both worlds. So next question, Jeff Thomas, looking for data sets for testing and comparing different training methods. Looking for new data sets, like MNIST is great, but after a while, people want something new. So what could you tell Jeff? PAIGE BAILEY: We can tell him about TensorFlow data sets! LAWRENCE MORONEY: And there's a blog post about it. PAIGE BAILEY: It is about creating the data ingestion pipelines for you to be able to easily use a variety of data sets with all of your deep learning and machine learning models with just a few lines of code. So if you are familiar with skikit learn, and all of this, it is nifty data ingestion practices, this feels very similar. It is easy to do, training and testing, and verifications. And we have an LOT of data sets readily available for you right now to go ahead and explore. And another thing that I would especially like to call out is, um, a member of our community -- so anybody can make your data famous, right? If you have a data set that you are using on your research lab, if you have, like, a bright and shiny CSV, that you think would be a cool add to this section. LAWRENCE MORONEY: I have never used a CSV called bright and shiny. PAIGE BAILEY: Everybody uses them, a community manager, Andrew, an undergraduate researcher at Stanford, added 200,000 of these to the Stanford ML group, in less than a day. It is as simple as take the template format for images or for audio or whatever you are using, add some additional metadata for the data set, change the types for the features you are using with that, and voila. LAWRENCE MORONEY: You are off to the races. PAIGE BAILEY: And you can make your data famous. LAWRENCE MORONEY: A wonderful thing about it, if you are getting started or if you are learning, if you look at the samples, pre-TensorFlow data sets, you get a lot of code about downloading the data, unzip it here, relabel it like this, put the files in these folders, take a CSV and make these features, but it is one or two lines of code and the training is set into data and test sets for you, that type of thing. So for learners, I found it exciting because I didn't have to go through 100 lines of code before I get to the neural network. PAIGE BAILEY: And also data sciency people, creating and training a model is understanding the shape, format, and the statistical distributions, and this is really helpful. LAWRENCE MORONEY: Yep. So that's all we have time for. We're going to take a break, we will be taking some more of your questions during the break. And then we will be back in just a few minutes to start answering them. Thanks very much. [Music]. . LAWRENCE MORONEY: Welcome back everybody to the live stream where Paige and I are answering questions that you posted on social media, on the live chat, Stack Overflow, and anywhere else you can find them. If you have questions, please hashtag #AskTensorFlow. PAIGE BAILEY: And if we do not answer your questions today, we will in further sessions, so keep asking, even if we don't necessarily get to them. LAWRENCE MORONEY: There are so many that we cannot get to them there today, a great problem to have. We have a show, Ask TensorFlow. So we will put these questions into the show, so please asking and this is what keeps the community engaged. So the first question, the pre-built binary for GPU on 16? PAIGE BAILEY: That is very specific. LAWRENCE MORONEY: I like specific questions, even if I can't answer them. So the pre-built binaries for TensorFlow tend to be associated with a specific driver for NVIDIA. So if you are looking at any of the pre-built binaries, look at what version of the driver you have supported. I'm not an expert on NVIDIA cards, and if you go over here, like on my laptop, I have -- I have called up, like, some of the what NVIDIAs say about the TensorFlow system requirements, and the specific versions of the drivers they support. And one gotcha that I found in working with GPUs is it is easier to go to the driver vendor and download the latest version, but that is not one that TensorFlow is built for that it supports. So make sure that they actually match each other, you should be good to go even with that particular card. PAIGE BAILEY: If you have warm feelings and excitement for builds for TensorFlow, we have an interest group, Sync build. So please go to the community section of our GitHub and check up the listserv and joining it in and joining our weekly stand-up. LAWRENCE MORONEY: Right. So thanks, Arthur, for that question. And then the next question is a really funny one. How many times have you been asked this today? PAIGE BAILEY: At least 12. At least, and then the other flavor of it is, is this symbol that I use all the time, is this supported in TensorFlow 2.0, and if not, what has changed? LAWRENCE MORONEY: Yep. And I -- I mean, people have invested so much time in 1. x, they don't want it to be deprecated or go away. So do scripts for 1 work with 2? PAIGE BAILEY: The sad fact is probably not. They would not work with TensorFlow 2.0 out of the box. But, we have created an upgrade utility to use, it is automatically downloaded with TensorFlow 2.0 when you download it, and you can check out what it is doing in this Medium blog post that I and my colleague, Ana, created, as well as this TensorFlow 2.0 video, and the gifs, the best communication medium possible, shows you how you can use the upgrade script on an N file. So any, sort of, arbitrary Python file or Jupyter notebooks, a machine learning IDE extension that allows you to do that as well. And it will give you an export. text file that shows you the symbol renames and the manual changes if you have to add them. And usually you do not. So to see it in action, you can take a look at this particular text generation example that we have running a Shakespeare, well, take all of the corpus of Shakespeare text, trains it against the Shakespeare text, and generates something that he could have potentially written, should he have had access to deep learning resources. LAWRENCE MORONEY: I know you all, and I will up hold the wildest humor of your idle dream. PAIGE BAILEY: I didn't know you knew Shakespeare. LAWRENCE MORONEY: I played Henry the IV. PAIGE BAILEY: I was in much ado about nothing. LAWRENCE MORONEY: So while we are on much ado about nothing, the Jupyter notebook. PAIGE BAILEY: Export the Python file, and to reupgrade it. LAWRENCE MORONEY: You have to reconnect the run. PAIGE BAILEY: So starting it. And the requirements, we can check that we are using TensorFlow alpha, and like I mentioned before, all you have to do is preface this was a bang, the name of the Python file is text generation, and I wanted to create an upgrade, shift enter, and it does its upgrading magic and, very quickly, it tells me all of the things that we need to change in order to make it 2. 0 compatible and create that file for me off to the side. So if I wanted to run this model, it should be able to -- it should be able to train as it would. So let's just check to make sure that that would be the case. LAWRENCE MORONEY: I think a lot of the errors you are seeing here, it is renamed APIs rather than breaking changes within the APIs as well. PAIGE BAILEY: This is true. You have re-names and additional key words. So for more information about that, and to see it running in a video, you can go and take a look there. LAWRENCE MORONEY: Sounds good, and you have some gifs in there? PAIGE BAILEY: Absolutely. LAWRENCE MORONEY: And any jifes for those that don't say gifs? PAIGE BAILEY: It is in the works. LAWRENCE MORONEY: [ Laughter ], so when it comes to upgrade, there's a few little gotchas, so hopefully the blog post, the videos, will help you get around them. PAIGE BAILEY: And the community you are mentioning before, we had such an interest in testing TensorFlow 2.0 and trying it out against historic models that we've formed a weekly testing stand-up, and also we have a migration support hour that is being implemented with the internal support hour. So if you have an external group that is interested in upgrading your model, please join the testing group and we can get you situated. LAWRENCE MORONEY: And a lot of the stuff that we have seen in Keras, you have the great slide where she is training -- PAIGE BAILEY: The code is exactly the same. LAWRENCE MORONEY: Exactly the same. So while there is stuff changing under the hood, a lot of the surface-level code that you are writing in Keras is not changing. PAIGE BAILEY: If you are used Keras, you are probably not going to have any problems. LAWRENCE MORONEY: So good stuff. Should we move on to the next question? So I know we can talk about 2.0 all day, but we just mentioned Keras, and it appears. PAIGE BAILEY: So I guess I can ask you this question. What is the purpose of keeping estimators in Keras as separate APIs, is there something native to Keras models that allows for distributed training, or train and evaluate? LAWRENCE MORONEY: The purpose of keeping them, there are many purposes. So for me, the main purpose that I would like to think of is, because a lot of people are using them. PAIGE BAILEY: Including internal Google teams that would tar and feather us if we removed them [ laughter ]. LAWRENCE MORONEY: And so, when it comes to estimators, estimators are really great for large-scale training. PAIGE BAILEY: Yes. LAWRENCE MORONEY: And it is like a lot of times, you are doing the large scale training, keep going with estimators, they are great. When I first started with TensorFlow, I started with estimators, I could not figure out what a node was in a neural network and all of these concepts that I had to learn, where I had a simple estimator that I can use to do a DNN or something like that. And so, you know, they are there for a reason and they are the same for the reason. Keras is one of the things that, from the point of view of making life easier for developers that we have been doubling down on TensorFlow 2, and the code is the same between 1 and 2, and if -- the layers API makes it super simple for you to design a neural network, and the fact that you can go low level beyond that, you know, to define your own layers allows you to drive stick rather than automatic. PAIGE BAILEY: And what is beautiful about Keras 2.0 is you have Keras the way you are used to using it, and if you want customizations, there's a sub-component, if you wanted to go lower, we have something called TS module and we have even exposed some of the basic, most core ops in TensorFlow as well. So, at any sort of level you want to interact with the API, you can. LAWRENCE MORONEY: Yep, and I think there is another part of the question that is around distributed training. So you can scroll down there, and there is something called distributed strategy. PAIGE BAILEY: Yep. LAWRENCE MORONEY: With Keras and with TensorFlow 2, and the whole idea behind that is you can distribute your training, maybe across multiple GPUs on the same machine, maybe across multiple GPUs on different machines, and maybe across CPUs all over the place. So distribution strategy is a part of that, to help you with that. So estimators in Keras, we love them both, they are still there, and hopefully this is something that will help you with a question. I think we have time for just one more. PAIGE BAILEY: Absolutely. LAWRENCE MORONEY: Oh, this is a Paige question. PAIGE BAILEY: This is totally a me question. I'm the Python person. So ask TensorFlow, when will TensorFlow be supported in Python 3. 7 and be accessed in Anaconda3? I can answer Python 3.7, and I would love to speak a little bit more about support for Python going forward. LAWRENCE MORONEY: Please, please. PAIGE BAILEY: So, to answer the 3.7 question, I'm going to bounce over to the TensorFlow 2. 0 project tracker, and these are all of the standing issues that we have when doing development for TensorFlow 2.0, it is transparent -- LAWRENCE MORONEY: You can see your avatar. PAIGE BAILEY: I have filed many issues and they are transparent to the public. If you want a context on where we stand currently and what we have yet to do, this project tracker is a great way to understand that. And please take a look at 3.7 and, there we go. In process of releasing binaries for Python 3.5 and 3. 7, we issued 25, 420, it is going off the screen, and 4.29. You can take a look at that issue and it is currently in progress, there is not really an eta, but something that we want complete by the time that the Alpha RC is released. So that is wonderful to see. And there is also, um, a website called Python three statements, I think it is Python3statements. com, maybe it is .org. There we go! LAWRENCE MORONEY: Yay. PAIGE BAILEY: Cool. TensorFlow made the commitment as of January 1, 2020, we no longer support Python 2 and we have done that with a plethora of our Python community. So TensorFlow, Scikit-learn, etc. , we are committed to Python 3 and Python 3 support. So you will -- you will be getting your Python 3 support and we are firmly committed to having that. LAWRENCE MORONEY: And a nice thing about the issue tracker is it is not a big, hey, we have it, coming to a random point in the future, it is transparent and you can see what we are doing. PAIGE BAILEY: And you can see our people commenting and the engineers commenting back, yeah, man, I was running it last night, one more test and it is it. LAWRENCE MORONEY: That is all we have time for, and remember, #AskTensorFlow on Twitter, Stack Overflow, YouTube. PAIGE BAILEY: Send a carrier pigeon if that's your preferred method of communication. LAWRENCE MORONEY: I prefer owls. PAIGE BAILEY: Owls are awesome. LAWRENCE MORONEY: If you have questions, we will do the best to answer them, we will put a lot on YouTube.com/TensorFlow. So don't forget to subscribe, thank you so much for being engaged. PAIGE BAILEY: Thank you. [Music]. Often I hear sometimes from researchers, I really only do research and care about training the machine learning model and I don't mean these instruments. What I would argue is research often leads to production and where we want to avoid is researchers having to reimplement their hard work in the model they have work when they want to put the model into the production. We really want the research community to build the models and the framework in the community. A second comment that I hear often is well, I only -- and all are built to scale up. I don't really need all of these heavy tools. What we have seen time and time again at Google, is small data today becomes large data tomorrow. There is really no reason why you would have to reimplement your entire stack because just your dataset grew. We want to make sure you can use the same tools early on in your journey so the tools can grow with you and your product with the data so that you can scale the exact same code to hundreds of machines. We are built TensorFlow Extended and it has had a profound impact in how we do production and becoming an AI first company. TFX powers some of the most important alphabet companies. It is used at six different alphabet companies and within Google it is used with all of the major products and all of the products that don't have billions of users. I said before we want to make TFX available to all of you because we have seen the profound impact it has had on our business and we are really excited to see what you can do with the same tools in your companies. A year ago, we talked about the libraries that we had open sourced at that point in time. We talked about TensorFlow transform, the training libraries, estimators, and keras, and TensorFlow Serving. I made the point that back then, as today, all of these are just libraries. They are low-level libraries you have to use independently and stitch together to make use and trend for your own use-casesuse-cases. We added TensorFlow validation later that year. It was valuable to release these libraries because some of our most important partners externally have had a huge impack. Airbnb uses TensorFlow server and our friends at Twitter published a fascinating blog post of how they use TensorFlow to rank tweets on their home timeline and used it to analyze data and used TensorFlow Hub to share the word embeddings they used for these models. Coming back to this picture, for those who saw my talk last year, I promised everyone there would be more. This is just a set of libraries right now. Today for the very first time, we are sharing the horizontal layers that integrate all of these libraries into one end platform into one end program called TensorFlow Extended. First, we have to build components. At the top in orange are libraries from the past and in blue is current libraries. Libraries are low level and flexible. We can build many components part of the machine learning pipeline. I will go into detail in each one of these components later. A component is no longer just a library. It is a package binary or container that can be run as part of a pipeline. It is well defined input and outputs. In the case of model validation, it is the last validated model. A new candidate model and validation. It has a well defined configuration and most importantly it is one configuration model for the entire pipeline. You can figure a TFX pipeline end-to-end. Some of you may have noticed because model validation needs to last, it needs context and needs to know what was the last model validated. We need to add a metadata store that keeps a record of previous runs so SH of the more advanced capabilities can be enabled. How does this context get created? The trainer produces new model, the model validator knows about the last validated model and candidate model and downstream we take the new candidate model and validation outcome and if the out' come is positive we push to the server and if it isn't we don't because we tonight -- don't want to push a model that is not neter. The metadata store is new. When most people talk about machine learning learn floes and pipelines they think about task dependency and think about one component and when that is finished there is another component that runs. However, all of you who do machine learning in production know we need data dependencies because all components consume and create artifacts and as the example of model validation has showed, it is incredibly important to know these dependency. We need a system that is task and data aware so each component has a history of all the previous runs and knows about all of the artifacts. What is in this metadata store? Most importantly, type definitions of artifact and their properties. For TFX, it contains the definition of all artifacts that have been consumeed and produced by components and all of their properties. It is an extensible type system and you can add new properties to the artifacts if you need to track more properties of those. Secondly, we keep a record of all of the execution of the components. With that execution, we store all of the imported artifact and the output artifacts that were produced and the Runtime components. If you want to track the code snapshot, you can store it in the metadata store as well. Putting these things together allows us to do something called linear tracking across all executions. If you know every execution, all inputs and outputs, you can piece together a story of how an artifact was created. We can say what were the upstreams that went into producing the artifact and what were the downstream runs artifacts that were produced using the artifact as an input. That is a powerful capability and let me talk you through some of the examples of what this enables. The first is pretty straightforward. Let's say I want to list all of the training runs I are done in the past. I am interested in the trainer and I want to see all the training runs that were recorded. In this case I had two training run and see all the properties. This is pretty straightforward and nothing new to see here. We can visualize the lineage and all the information we have. The first comment on this slide is we are working on a better UI. This is just for demonstration purposes. If you look at the end of the graph, you see the model expert path. This is the specific instance of the model that was created and we see the model was created by the trainer and the trainer created this model by consuming a schema, transform and examples. These are specific instances so the IDs are not just number. They are schema of ID number 4 and for each of those artifact WEEZsee they were created upstream. This allows forward and backward in the artifact and the narrative I used was walking back from the model but you could look at training data and say were what the artifacts produced using that training data. This slide shows a visualization of the data distribution that went into the model. At first glance, this may not be something like that that is earth shattering because we have done this before. We can compute and visualize statistics but if you look at the code snippet, we are fought referring data or statistics. We are referring to a model. We say show me the distribution of data the model was trained on and we can do this because we have a track record of all the data and statistics that went into this model. We can do a similar thing in the other direction saying for a specific model show me the sliced metrics that were produced downstream by TensorFlow model analysis and we can get this visualization again just by looking at a model and not specifically pointing to the output of TensorFlow analysis. We know all the models that were trained and where the checkpoints lie so we can start support to the historic runs and look at the tensor port for the models you have trained in the past. Because we have a track record, you can launch tensor point and compare two models in the same instance. This is really model tracking and experiment comparison after the fact. We enabled this by keeping a track record of all of this. If you have multiple models, you can look at the data distribution for multiple models and this usually helps with debugging a model. If you train the same model twice, or on data and it behaves differently, sometimes it can pay off. We are overlaying two distributions of the statistics for one model and the other and you would see if there is a considerable drift between those two. All of these are enabled by this lineage tracking. Another set of use-cases is visualizing previous runs over time. So if you train the same model over time over new data we can give you a time series graph of all metrics over time and you can see if your model improves or gets worse over time as you re-train it. Another powerful use case is carrying over state from previous models because we know you have trained the model in the past we can do something called warm starting. We can reinitialize the model with weights from a previous run. Sometimes you want to reinitialize the entire model or just an embedding and in this way we can continue training from where we left off with a new dataset. And another very powerful application of this is being able to reuse previously outputed outputs. If you have a pipeline that ingest data, applies transformation to data and then you train the model, every time you make a small change to the model you don't want to recompute every upstream. There is no reason just because you changed something in the model. Because we have a track record of all the previous steps, we can make a decision of saying your data hasn't changed, your transfer code hasn't changed and we will reuse the artifacts that produced the upstream and you can iterate much, much faster. This improves iteration speeds and saves compute because you are not reexot -- recomputing things you already computed in the past. We talked about components and how do we orchestrate TFX pipelines? Every component has something we call a driver and publisher. The driver's responsibility is to basically retrieve state from the metadata store to inform what work needs to be done. In the example of model validation, the driver looks into the metadata store to find the last validated model. The publish keeps the record of everything going in the component, produced and Runtime configuration so we can do the lineage tracking I mentioned earlier. In between is the executer who is unaware of the metadata because it is important to make that piece relatively simple because if you want to change the training code in a component you should not have to worry about drivers and publishes, just the executer. It makes it easier to write new components for the system, also. We have one shared configuration model on top that configures end-to-end pipeline. As you can see this is a Python dsl, you see it has an object for each component from top to bottom. The training component you can see basically receives configuration and says your input come from the transer outoutput and the schema that was inferred. Let's see what is inside the of the trainer and that is really just TensorFlow code. We use an estimator and an estimator train and evaluate method to train this model. It takes an estimator and we just use one of our pre-made estimators. This is a wide and deep model you can instantiate and return. We don't have an opinion on what this code looks like. It is just TensorFlow. Anything that produces a safe model as output is fair game. You can use a keras model that produces an inference graph or if you chose to you can use lower level APIs in TensorFlow as long as it produces a safe model in the right format that it can be useded in TensorFlow or for model analysis, you can write any type of TensorFlow code you want. So, if you have noticed, we still haven't talked about orchestration. We are a configuration, components and metadata store. I know some of you might be thinking is he going to announce a new orchestration system and the good news is no. At least not today. Instead we talked to a lot of users and unsurprisingly found out that there is a significant install base of orchestration systems in your companies. We just heard from Airbnb, of course they developed airflow and there is a lot of companies that use Kubeflow and there is a number of other orchestration systems. We made the choice to select any number of orchestration systems because we don't want to make you adopt a new orchestration system to use the TFX pipelines. Reason number two is we want you to extend TFX pipelines. We published our version of what a TFX pipeline looks like and the components we use at Google, but we want to make it easy for you to add components before, after and in parallel to customize the pipeline to your own use-cases. All these orchestrations are made to be able to express arbitrary workflows and if you are familiar withing of those orchestration systems you should be able to use them for a use case. Here are two examples of what it looks like with airflow and Kubeflow pipelines. On the left you see the same TFX pipeline configured that is executed on AirFlow. In the Chicago taxi cab example we used 10,000 records. On the right side you see the same pipeline executed on Kubeflow on data Cloud so you can scale it up to the 100 million rows in that dataset. It is the same configuration and components running in both environments and you can chose how you want to orchestrate them in your own favorite orchestration system. This is what this looks like if it is put together. TFX goes all the way from raw data to your deployment environment. We discussed a shared configuration model at the top. The metadata system that keeps track of the runs no matter how you orchestrator the component and two ways to publish them with airflow and Kubeflow but you can chose to orchestrate the TFX pipeline in any way you want. All of this is available now so you can go on GitHub/TensorFlow/TFX. Tomorrow we have a workshop where you can get hands on experience with TensorFlow Extended and there is no prerecks and you don't to bring your own laptop. We will jump into an example with the Chicago taxicab dataset. This is a record of cab rides in Chicago for some period of time. It contains everything you would expect. It contains when the trip started, when it ended, where they started and ended, how much was paid for it and how it was paid. Some of the features need transformation so longitude and latitude features need to be packaged. We treat them as categorical features. Strings need to be integerized and some of the flows need to be normalized. We feed the dense features into the deep part and all the others use the wide part. The label we are trying to predict is a bullion which is if the tip is larger than 20% of the fare. Really, what we are doing is building a high tip predictor just in case if there is any cab drivers in the audience or online, come find me later. I think this will be beneficial to you if you can predict if a cab ride give as high tip or not. Let's jump in and start with data validation transformation. The first part of the TFX pipeline is ingesting data, validating that data if it is OK, and then transforming so it can be fed into a TensorFlow graph. We start with the examplexamplegenwhich takeexamplegen. This is extensible. You can ingest any type of data in the pipelines. What is important is after the step the data is in a well defined place where we can find it, in a well defined format because all downstream components standardize and it is split between training and develop. You have seen the configuration of these components before. It is very minimal configuration in most of the cases. Next we move on to data analysis and validation. I think a lot of you have a good intuition of why it is important. Machine learning is the process of taking data and learning models that predict some field in your data and you are always aware if you feed garbage in you get garbage out. There is no hope in learning a good machine learning if the data is wrong or has errors in them. This is reinforced if you have continuous pipelines produced on data from a bad model and you are reinforcing the same problem. Data understanding is critical for model understanding. There is no hope in understanding why a model isn't predicting something if the data isn't right. The question you may ask as a cab driver is why are trip predictions bad in the morning hours and I will try to answers these with the TFX toolsets we V. There is a lot of care taking with code. It is previewed, checked into repository, version controlled and data needs to be first-class citizens in these systems. With the question what are expected values from payment types? That is a question about the schema of your data and what we would argue is that the schema needs to be treated with the same care as you treat your code. And catching errors early is absolutely critical because I am sure as all of you know errors propagate thew the system. If the data is not OK, every downstream goes wrong. These errors are hard to fix if you catch them late in the process. Really catching those problems as early as possible is absolutely critical. In the taxicab example, you would ask a question is this new company I have in my dataset a typo or a real company which is a natchal evolution of my dataset. The statistics training program takes in the data and it can be training or serving logs in which case you can look at the skew between serving and training data. Statistics really capture the shape of your data and the visualization components draw your attention to things that need your attention such as if a feature is missing most of the time it is highlighted in red. The configuration for this component is minimal as well. Let me zoom into some of these visualizations. One question I posted earlier was why are my predictions bad in the morning hours. You could look at the dataset and see for trip start hour in the morning hours before 2:00-6:00 A.M. you don't have much data because there is not that much taxi trips at that time and not having a lot of data in a specific area can mean your model is not robust or has higher variance and this could lead to worse predictions. Next schemagen. It takes the output and infers a schema for you. There is very few features in this case so you could handwrite the schema. If you have thousands of features, it is hard to handwrite that expectation so we infer the schema for you and it represents what you expect from your data and what good data looks like and what values your string features can take on and so on. Again, very minimal configuration. The question we can answer is what are expected values for payment types. If you look at the bottom, you see the field payment type can cake ton cash, credit card, dispute, no charge, p card and unknown. Next time I run this, and this field takes on a different value, I will get an anomaly which comes from the example validator. The example validator takes the statistics and schema as input and produces on anomaly report. That anomaly report tell you if the data is missing feature, has the wrong relevancey, if the distribution has shifted and deis important to highlight the anomaly report is human readable so you can look at it and understand what is going on but it is always machine readable so you can automatically make decisions on the anomalanomaly. Here you can see the field company has taken on unexpected string values. That just means these string values were not in your schema before and that can be a natural evolution of data. The first time you run it maybe you didn't see trips from those taxicab companies but when you look you see they are normal so you can update or if you saw a lot of scrambled text you would know there is a problem in your data that you would have to go and fix. Moving on, we actually get to transform. Let me just recap the types of transformations we want to do. I highlighted them in blue. We want to convert the strings to ints and for the tense features we want to normalize them. All of these transformations require you to do a full pass. To pact packatize you need to figure out the boundaries. For a string to int you need to see all the string values. This is exactly what we build TensorFlow transform for. It allows you to express the preprocessing function of your data that contains some of these transformations that require a full pass of your data and enable automatically to run the data processing graph to compute those statistics. In this case, you can see the orange boxes are statistics that we require, for minimalization we require a mean and standard deviation and TensorFlow transform says scale to C store and it will create a data processing graph for you that will compute the standard mean and data and return the results. That graph is a hematic graph that contains all the information you need to apply your transformations. That graph can then be used in training and surveying guaranteeing there is no drift between them. This eliminates the chances of skew and at serving time you need to feed in the raw data and all the transformations are done as part of a TensorFlow graph. The transfer component takes in data, the schema allows you to parse the data, and code in this case the system provided preprocessing function, and it produces a hematic graph that gets attached to your training. It can materialize the transform data and that is a performance optimization you need. It can pay off to serialize some of the training before the next step. The component takes in a module file, a file where you configure preprocessing function. In this code snippet, the actual code isn't important but I want to highlight the last line in this code snippet and that is how we transform our label because there label is a logic expression of saying it is greater than 20% of my fare. I want to highlight this because you don't need analyze phase for all of your transformations. In cases where you don't need analysis phases, the transformation is just a regular TensorFlow graph. To square something to C square and integerize schemes you need these phases. This is the user code you would write and transfer creates the graph and returns the transfer graph you need to apply these transformations. Now we are done with this we haven't trained a machine learning model right but we have made sure we know our data is in a place where we can find it, it is in a format we can understand and split between training and deval and we know our data is good because we validated them and we are applying transforms consistently between training and serving. This bringstuse the training step. This is where the magic happens. The training steps is the TensorFlow graph and step. The training component takes in the output of transform as mentioned which is the transform graph and optionally the materialized data, schema and training codes that you provide. It creates output to TensorFlow models. Those models are in the safe model format the standard serialized model in TensorFlow which you heard quite a bit about this morning. We produced two of them. One is the inference graph and another is the evaluate graph which contains the metrics and necessary annotations to perform TensorFlow model analysis. This is the configuration you have seen earlier and again the trainer takes in a module file and the code that is actually in that module file, again, I am going to show you the same slides to reiterate the point is just TensorFlow. In this case it is the train and evaluate method from estimators and the estimator being returned here. Again, just to make sure you are aware of this, any TensorFlow code here that produces the safe model in the right format is fair game. This works really well. With this, we have now trained the TensorFlow model and now I am going to hand it off to my colleague Christina who will talk about model evaluation analysis. -- model validation. Thank you, Clemens. Hi, my name is Christina and I am a software engineer on the Google brain team. I am here today to tell you about some tools my team and I have built to help make the end-to-end lifecycle of the machine learning pipeline easier. I am going to start by talking about model analysis and salad validation. These are two different components in TFX but they are similar in how they are executed. The main difference is how you, as an end user, will use them. I will start by talking about the evaluator. Why is this important? We have gathered data, cleaned that data, trained a model, but we really want to make sure that model works and some model evaluation can help you assess the overall quality of your model. You also may want to analyze how your model is performing on specific Slices of the data. So, in this case with the Chicago taxi example that Clemens started us off with, why are my tip predictions sometimes wrong? Slicing the data and looking at where you are doing poorly can be a real benefit because it identifies low-hanging fruit where you can get gains in accuracy by adding more data, or making some other changes to make some of these segments improve. You also want to track your performance over time. You are going to be continuously training models and updating them with fresh data so your models don't get stale and you want to make sure that your metrics are improving over time and not regressing and model evaluation with help with this. The component of TFX that supports this is called evaluator. You need to specify the splits in data you find more interesting so the evaluateator evaluator. A process is run to generate metric for the overall slice and the Slices you have specified. The output of the evaluator is evaluation metrics. This is a structured data format that has your data, the splits you specified, and the metrics that correspond to each split. The TensorFlow model analysis library also has a visualization tool that allows you to load up these metrics and dig around in your data in a user-friendly way. So going back to our Chicago taxi example, you can see how the model evaluator can help you look at your top-line objective and how well can you predict trips that result in large tips. The TFMA visualization shows the overall data. Maybe you want to stay 95% accuracy. That is a lot better than 94.7. Maybe you want to bump that up. Then you can dig into why your tip predictions are sometimes wrong. We sliced the data by the hour of the day the trip starts on and sorted by poor performance. When I look at this data, I see trips that start at 2-3:00 a.m. were performing quite poorly in these times. Because of the statistics generation tool Clemens talked about I know the data is sparse here but if where didn't I would think maybe there is something that people that get taxies during that time that causes erratic tipping behavior. You also want to know if you can get better at predicting trips over time. You are continuously training the model for new data and hoping you get better. The TensorFlow model analysis tool that powers the evaluator in TFX can show you the trends of your metrics over time. Here you see three different models and the performance over each with accuracy and AU C. Now I will move on to talking about the model validator component. With the evaluator, you were an active user, you generated the metrics, you loaded them up into the UI, you dug around in your data, you looked for issues you could fix to improve your model, but eventually you are going to iterate, your data is going to get better and model improve and you will be ready to launch. You will have a pipeline continuously feeding new data into this model. Every time you generate a new model with new data you don't want to have to do a manual process of pushing this to a server somewhere. The model validator component of TFX acts as a gate that keeps you from pushing bad versions of your model while allowing you to auto automate pushing. We want to avoid pushing models with degrated quality specifically in an automated fashion. If you train a model with new data and the performance drops but say it increases in certain segments of the data you care about, maybe you make the judgment call this is an improvement call overall so we will launch it. But you don't want to do this automatically you want to have say before you do this. This acts as your gatekeeper. You want to avoid breaking downstream components if your model started outputing something your server binary couldn't handle you would want to know that before you push. The TFX component that supports this is called the model validator taking similar input and output to the model evaluator and the libraries that compute the metrics are pretty much the same under the hood, however, you provide two models. It then runs on your eval split data and compares the metrics on the same data between the two models. If your metrics have stayed the same or improved, you go ahead and bless the model. If the metrics that you care about have degraded, you will not bless the model, get some information about which metrics failed so you can do further analysis. The outcome of this is a valididation outcome. It just says blessed if everything went right. Another thing to mote about the model validator is it allows next day eval of your next day push. Maybe the last model you blessed, it was trained with old data. With the model evaluator, it evaluates it on the new data. And finally, I am going to talk about the pusher. It pusher is probably the simplest component in the entire TFX pipeline but it does serve quite a useful purpose. It has one input which is that blessing that you got from the model validator. Then the output is if you passed your validation then the pusher will copy your saved model into a file system destination you have specified. Now you are ready to serve your model and make it useful to the world at-large. I am going to talk about model deployment next. So this is where we are. We have a trained saved model. A saved model is a universal serialization format for TensorFlow models. It contains your graph, your learned variable weights, your assets like embeddings and vocabulary, but to you this is just an implementation detail. Where you really want to be is you have an API, you have a server that you can query to get answers in real-time or provide those answers to your users. We provide several deployment options and many of them are going to be discussed at other talks in this session. TensorFlow. js is optimized for serving in the browser. TensorFlow Lite is optimized for mobile devices. We heard a talk about how Google Assistant is using TensorFlow Lite to support model inference on their Google Home devices. TensorFlow Hub is something new. André is going to come on in five minutes and tell you about that so I am not going to step on his toes. I am going at a talk about TensorFlow Serving. If you want to put up a resting API you would want to use TensorFlow Serving. Why would you want to use this? For one thing, TensorFlow Serving has a lot of flexibility. It supports multi tenancy. You can run multiple models on a single server instance and run multiple versions of the same model. This can be useful when you are trying to canary a new model. Say you have a tried and tested version of your model, you have created a new one, it past your evaluator and validation but you want to do testing with real users before you completely switch over. TensorFlow Serving supports this. We also support optimization with GPU and TensorFlow RT. You can expose a gRPC or REST API. TensorFlow Serving is optimized for high performance. It provides low let latency and request batching and traffic isolation so if you are serving multiple models on a single server, a traffic spike in one of those models won't affect the serving of the other. And TensorFlow Serving is production ready, finally. This is what we use to serve many of our models inside of Google. We have served millions of QPS with it. You can scale in minutes particularly if you use the Docker image and scale up on Kubernetes. We support dynamic version refresh so you can specify a version refresh policy to take the latest version of your model or you can pin to a specific version. This can be useful for rollbacks if you find a problem with the latest version after you have already pushed. I am going to go into a little bit more detail about how you might deploy a REST API for your model. We have two different options for doing this. The first, the top command, is using Docker which we really recommend. It requires a little bit of ramp up at the beginning, but you will really save time in the long run by not having to manage your environment and not having to manage your own dependencies. You can run locally on your own host but you have to do all that stuff long term. I will go into more detail on the Docker run command. You start with Docker run, chose a port you want to bind your API to, provide the path to the saved model generated by your trainer and hopefully it was pushed by the pusher, you provide the model name, and you tell Docker to run the TensorFlow Serving binary. Another advantage of using Docker is that you can easily enable hardware acceleration if you are running on a host with a GPU and Nvidia host installed you can run on an accelerated hardware. If you need further optimization we support tensor RT which is optimized for deep learning inference. Your Chicago taxi example that we have been using here probably wouldn't benefit from this but if you had an image recognition model, ResNet, you could get performance boosting and cost savings by using TensorRT. We provide a command line allowing you to convert this model and a simple change to the command line and you are running on accelerated GPU hardware TensorRT optimization. To put it all together again, we introduced TensorFlow Extended or TFX and showed you how the different components that TFX consist of can work together to help you manage the end-to-end lifecycle of your machine learning pipeline. First, you have your data and we have tools to help you make sense of that, process it, and prepare it for training. We then support training your model and after you train your model we provide tools that allow you to make sense of what you are seeing, of what your model is doing and to make improvements, also to make sure you don't regress. Then we have the pusher that allows you to push to various deployment options and make sure model available to serve users in the real world. To get started with TensorFlow Extended, please visit us on GitHub. There is more documentation at tensorflow.org/TFX and some of my teammates are running a workshop tomorrow and they would love to see you there. You don't need to bring a laptop. We have machines setup and ready to go and you can get hands-on experience using TensorFlow Extended. Thank you. Please welcome André who will talk about TensorFlow Hub. Thank you, Christina. Could I get the notes on the presentation? Thank you. Hello. My name is André and I am a software engineer in Zurich and I am the technical lead for the TensorFlow Hub project. It is a library and project for using machine learning. We Open Sourced it last year. Today I will give you an update on the project. Let's start by talking about when you want to use it. If you have problems collecting enough data to train models from scratch than this is for you. Additionally, since it is so easy to reuse computations and weights, it becomes possible to leverage features without having to learn how to fit them into neural networks. Images, text and videos are features you can use with a single line of code. You might also have encountered problems where codebases become really coupled and experimentation becomes slower over time. By defining an artifact that does not depend on code, hub allows for more maintainable systems similar to how libraries help software engineers. The main concept is this pre-trained building blocks that we call modules. Typically we start by training a large model with the right algorithm and data. From this large model, we are interested in just a part which is reusable. This is typically a bottleneck layer or disrooted representation. Then we can package them into a saved model that defines the commutation and no longer depends on the original code. You can share with web servers, cloud and artifact system. You can bring the module back as a piece of a new model to solve a new task. Since the model is defined by TensorFlow primitives you can fine tune the waits weights and adjust them. For the last few months, we have been making it easier to use. This concept of saving a part of the model and loading it back is getting integrated right in the core of TensorFlow. We have added saveModel features that make it possible to share more than just signatures and with eager execution it becomes even easier to select the part the network that gets exports. Let's look at a few examples. In TensorFlow 2. 0, we can load a module with hub.load. We can use tf.saved model. load if it is on the system. Once loaded, we can call it right away. Due to the new capabilites you can share any object composed of TensorFlow primitives. Text features has two members. Call which is a TFX function and embedding which is a TF variable. We are really excited that we added support for polymorphic functions on saved model. This will provide a more natural interface than we had before with signatures. Here we see an image representation module being loaded and being used in inference mode or used during training mode where batch mode is on. When it is on, we can control some of its parameters. This is just backed by TF graphs so we no longer need to be selecting things here and there. We just get the API that looks very like Python. Additionally, we have added the new symbol. Hub.keraslayer. This makes integrating hub models into a keras model easier. In this example, we see how easy it is to build a text sentence -- a sentence classification model. We have three layers. The top model is this keras layer, is a layer that receives inputs and outputs representation. Then we have a dense layer and a classification layer. Since the keras layer with this MLM layer, we can just feed sentences straight into the model. We never have to define the logic. If we wanted to try other text models we could just change that string. That is the easiest way to try the new research. We have released 0.3 version of TensorFlow Hub. We have hub.load and hub.keraslayer and they are usable in TensorFlow 2.0 both in eager and in graph mode. To let you preview this functionality, we have published modules in this format and the next step for us is to backport existing models. A bit more practical now. Google AI and DeepMind teams have been sharing information on tfhub.dev. This was launched last year. One of the most popular was the universal sentence encoder. This encoded short sentences into a vector that could be used for many natural language tasks. The team added a cross-lingual version of these. Sentences with similar meaning, independent of the language, end up in points close together. You can learn the classifier using English data and you could run it on other languages. We have also added image augmentation models. They have been shown to transfer to new tasks. An interesting thing is you could grab this module and one of the image representation modules and you could string them together. The image augmentation model would reduce the amount of data by data augmentation and the image feature vector by transfer learning. There are many more. We have BERT model for tasks, we have biggan models and more. Some were designed especially for low resources. Additionally, we have been working to make the modules more integrated. We have a common line utility to convert the hub model into TF model. It can be used with a library for ML and they can also be used inside TF transform. If you want to try it you can go to tfhub.dev. Most of them include the link where you can see them in action. Thank you. Thank you very much. OK. Now you have all learned hopefully training neural networks that if I show up there is food. So, we are going to come back here at 4:45 to do a little wrap up. The wrap up session is going to be amazing not that this wasn't also amazing. We will be talking about probablyability, reinforcement learning, we will be talking about -- Elena is coming back to tell us about her stuff. It is going to be great. After that there is a party. Please stay with us. Come back. Don't forget to go upstairs and check out the demos. Thank you so much. Aida net et AdaNet da net auto M L test test test test Extreme weather is changing. There is more extreme rainfall, heavy flooding, forest fires. Being able to predict these extreme events more accurately is the biggest changes we are facing. There is 100 Tarabytes every day. Climate data is a big problem. We need things that are fast and can sift through the data accurately and rapidly. Deep learning is poised for problems in climate science. A lot of NERSC are using TensorFlow. We use TensorFlow to layer and to create the deep learning model, we started from segmentation models which have proven to be successful and we use TensorFlow to enhance the models until we found a set of models that perform well enough for the specific task. The network required 14 teraflops. If you want to do this on your work station, it would take months to train. It takes the largest computational resources on the planet. Two tennis courts in size, this computer is state-of-the-art and a million times faster than your common laptop. 3.3 exaflops. Imagine what you do at your work station and imagine having 27,000 times that power. We can do that now. We were surprised how good it scales. 2,000 nodes, 5,000 nodes. This was the first time anyone has ran an AI application at this scale. The climate scientists could express things in Python in TensorFlow and get all the code people are used to. We are entering a state where AI can contribute to the predictions of these extreme weather events. We can tackle things we never thought. Understanding diseases like Alzheimer's and cancer. That is like incredible. We have shown with a PRAM work like TensorFlow you can get to massive scale and accomplish goals. Genetics, neuroscience, physics -- that is immensely exciting for me. -- framework -- [Applause] Hello, everyone. I am Josh Dillon and I am a lead on the TensorFlow Probability team and today I will talk about probability stuff and how it relates to TensorFlow stuff. Let's find out what that means. OK. These days machine learning often means specifying deep model architectures and fitting them under loss and keras makes specifying this architecture easily but what about the loss? Choosing the right loss is tough. Improving even a reasonable one it be tougher. Once you fit the model, how do you know it is good? Why not use entropy or mode? Wouldn't it be nice if there was a plug and play with keras and the rest of TF model? This would make comparing models easier by maximizing likelihood and having as it SKT -- rapidly available statistics we can reject the bad ones and accept the good ones. Wouldn't it be great to say I want to maximize the log likelihood and summarize what I learned easily and in a unified way. Let's play with that idea. Here we have a dataset. These blue dots and our task is to predict, pretend task, is to predict the Y coordinate from the X coordinate. You might do this by specifying a deep model and choosing the mean squared error as the loss function. But our wish here is to think probalistically and that means maximizing the log likelihood like here with this Lambda function. We want to get back a distribution, a thing that has attached to it statistics we can use to evaluate what we just learned. If only such a thing were probably. Of course, it is and you can do this now. Using TensorFlow Probability distribution layers you can specify the model as part of your deep net. The loss now is actually part of the model, sort of the way it used to be, the way it is meant to be. We have two dense layers. The second one outputs one float that is parameterizing a normal distrubution mean through this distribution Lambda layer. Being we are able to find this line, that looks great, and the best part is once we instantite this model with test points we have back a distribution for which you get not just the mean but intropy variance and standard deviation and all these things. You can compare between this and other distributions as we will see later -- entropy. If we look at the data, something is fishy here, right? Notice as the magnitude of X increases, the variance of my increases also. What can we do to fix this? Answer? Learn the variance too. If we are fitting a normal why on earth do we think the variance would be one? That is what you are doing when you use mean squared error. To achieve this, all I had to do was make my previous layer output two floats. Passing one to the mean as normal, one in as the standard deviation of the normal, and press no change and now I have learned the standard deviation from the data itself. That is what the green lines are. This is cool because if you are a statistician, it means you are learning known unknowns. The data had variance and you learned it and it cost you basically nothing to do but a few key strokes. The way you saw to do this was self evident from the fact you were using a normal distribution for a curious constant sitting there. This is good but I don't know is there enough data for which we can claim this red line is the mean and the green lines are the standard deviation? How would we know? Is there anything else we can do? Of course there is. A keras dense layer has two components. A kernel matrix and a bias vector. What makes you think those point estimates are the best especially given your dataset itself is random and possibly inadequate to meaningfully and reliable learn those estimates. You can learn a distribution over weight, this is the same as learning an ensemble that is infinitely large. As you can see, all I had to do was replace keras. dense with with dense variation. The keyword here -- I like to think of the unknowns unknowns. As a consequence, of course,this means any model or instantiation is a random variable because the weight is a random available. That is why you see all the lines, there are many lines, we have an ensemble of them but if you averaged them and took the standard deviation that would give you an estimate of credible intervals over your prediction and you can go to your customer and say here is what I think will happen and this is how much you can trust me. We lost the head -- I just output two floats and feed that into my output layer and press no change. I am learning both known and unknown knowns and all it cost me was a few key strokes. So what you see here is an ensemble of standard deviations associated with the known-unknown parts, the variances observable in the Y axis, as well as a number or ensemble of these mean regressions. OK. That's cool. I like where this is going but I have to ask what makes you think a line is even the right thing to fit here? Is there another distribution we could chose? A richer distribution that would actually find the right form of the data? Of course the answer is yes. It is a Galsian process. The Gaussian process can see this. How can you do that if you are specifying mean squared error as your loss? You can't. It has to be part of the model. When you bake these ideas into one model you get to move it around fluidly between weight uncertainty and variance in the data and even uncertainty in the loss function you are fitting itself. So the question is, how can this all be so easy? How did it fit together? With TensorFlow Probability. TensorFlow Probability is a collection of tools designed to make probalistic reasoning in TensorFlow easier. It is not going to make your job easy. It is just going to give you the tools you need to express the ideas you have. You still have to have domain knowledge and expertise but you can encode that in a probalistic formulism and TensorFlow Hub has the way to do that. Gone are the days of hacking a model in R and importing it to a faster language. If you predict the light is green, you better be confident you should go. You can do that with probalistic modeling and TensorFlow Probability. We saw one small part of TFP. The tools are broken into two components. Those tools use FWL -- for building models and those useful for do analyzing. It is summary statistics like it is in other libraries. Our distributions support batch shape but for the most part they should be pretty natural and easy to use. We also have something we call bijectors which is a library for transforming random variables. This can be like taking the X of a normal and now you have a log normal. In comply -- complicated cases it is like mascots, regressive flows, MVPs and other sophisticated probalistic models. Edward2 helps you combine different variables as one. On the inference side, no library would be compete without Monte Carlo tools. We have transitions components one is called Markov chain Monte Carlo and variational inference tools. And of course we have our own optimizers that come up. The point is this toolbox has maybe not everything but certainly it has most of what you might need to do fancier modeling to get more. It doesn't have to be hard. You saw the keras examples were sequences of one-line changes. Of course, TensorFlow Probability is used widely around alphabet. DeepMind and Google brain and Google accelerated sciences, product areas, infrastructure areas use it for planning purposes, but it is also used outside of Google. Baker Hughes uses this to predict anomalies. For example, anyone who flew out here would be happy to know that bhge uses their prediction to predict the lifespan of jet engines. If we had a dataset using a failing engine that would be a tragedy. Using math they get around this by modeling models and then trying to in the abstract figure out if they are going to be good models. The orange boxes use TensorFlow Probability extensively. The orange border box is where they use TensorFlow. The basic flow is to try to treat the model itself as a random available and determine if it will be a good model on an otherwise incomplete dataset. From this they get remarkable results and dramatic increases in false positives and false negatives over large datasets in complicated systems. So the question is who will be the next success story? Try it out. It is an open source Python package built using TensorFlow that makes it easy to combine deep learning with probalistic models. You can pip install it. If you are interested in learning more about the bayessian methods, check out thus this book. If you are not a bayesian, that is OK too. If you are doing linear aggression, it could solve the problem on the order of 30 iterations which can't be said of standard gradient dissent. If you want to find out more, you can check out our GitHub repository where you can find several Jupyter notebooks. Thanks. [Applause] And with that, I would like to hand it off to Eugene and Sergio who will be talking about reinforcement learning. Thank you. Hello, everyone. How is it going? Very packed day? Lots of good stuff? Are you ready for more? Or not? OK. Good. So my name is Sergio Guadarrama and I am a software engineer at Google brain and I work in the TF agent steam. And I am Eugene Brevdo working on reinforcement learning. How many of you remember how you learned how to walk? You stumble a little bit, you try one step, doesn't work, you lose your balance, you try again. When you try to learn something hard practice you need to try many times. This is moving the legs and trying to learn how to walk. After learning to try multiple times, in this case is 1,000 times, it falls. It will train a little longer and then it is able to walk around and go from one place to another and find their way around the room. You have probably heard about all the applications including recommended assistance, data -- real robots like that, chemistry and math and always alpha go that play Go better than any human. Now, I have a question for you. How many of you have tried to auto implement an algorithm? I see quite a bit of hands. Very good. It is hard. Yeah. We went through that pain too. You know, many people who try get the first prototype right away. It seems to be working but then you miss a lot of different pieces. All the details have to be right, everything. There is a lot of pieces and a lot of things you need to do. We suffer through the same problem at Google and decided to implement a library that many people can use and we are happy to announce it is available online. You can go to GitHub and pip install it and start using it right away. Hopefully, you will provide feedback contributions so we can make it better over time. Now, where is TF age NLT -- agents? For people new, we have Colabs and things with documentation and samples so you can learn about it. For people that want to solve the complex problems, we have already ways to take their algorithm and apply it. For people who are researching and want to develop new algorithm they will not need to build all the pieces. They can build on top of it. You can start doing experiments right away. We build on top of the goodies in TensorFlow 2.0 you saw today. Tf.keras to build the networks and tf.functions when you want thinks to go faster and we make it mobile and extensible so you can cherrypick the pieces you need. For those who are not ready for the change yet, we make it compatible with TensorFlow 1.14. We go back to the sample of the little robot trying to work. This is how the call looks like. You have it define networks. An actor distribution network and a criteriaic and actor. Assuming we have data collected we can just train through it. TF-agents provide a lot of environments already. We also provide a state-of-the-art algorithms including dqn, and many others. There are more coming soon. They are fully tested with quality version tests and a speed test to make things keep working. As an overview of the system, it looks like that. On the left side you have all the collection aspects and we will have policy that will interact with environments and collects on experience. We probably put in some replay buffer to work later and on the right side we have all the training pipeline. We are going to write from this experience and our agent is going to learn to improve the policy by training a neural network. Let's focus for a little bit on this environment. How we define a problem and new task. Let's take another example. In this case it is breakout. The idea is you have to play this game move the power left and right and try to break the bricks on the top. You break the bricks you get rewards and the points go up. You let the ball drop and the points go down. The agent refers multiple frames and it will decide which action to take and based on that it will get reward and just loop again. How this look into the code is something like this. You define the specific specification. What kind of observation this environment provides. Any other information. Multiple things. Then the action speckspecs. In this case left and right but in many environments we have multiple actions. And a reset method because we will play the game a lot and have to reset the environment and a step method that taking an action will provide a new observation and give us a reward. Given that we define the policy, create the environment, reset the environment and start looking over it and start playing the game. Your policy is very good you will get a good score. To make the learning even faster, we make these parallel environments and you can run these games in parallel multiple times and wrap in TensorFlow so it will go even faster and then do the same loop again. What happened in general is like we don't want to define this policy by hand. Let me hand it over to Eugene who is going to explain how to learn those policies. Thank you, Sergio. Yeah, as Sergio said, I am giving an example of how you would interact with the environment by a policy. We are going to go into the a little bit more detail and talk about how to train policies to maximize the rewards. So kind of going over it again, policies take observations and emit parameters of a -- a distribution over the actions. In this case, the observations are an image, there is an underlying neural network that converts those images to the parameter of the distribution and the policy emits that distribution or you might sample from it to actually take actions. So, let's talk about networks. I think you have seen some variation of this slide over and over again today. A network in this case network used for deep Q learning is essentially a container for a bunch of keras layers. In this case, your inputs go through a convolutional layer and so on and so forth and the final layer emits logits over the actions you might take. The core method of the network is the call. It takes observations in a state, possibly an RNN state, and emits the logits and you update the state. OK. So let's talk about policies. First of all, we provide a large number of policies. Some of them specifically tailored to particular algorithms and particular agents but you can also implement your own. It is useful to go through that. A policy takes one or more networks. The fundamental method on a policy is the distribution method. This takes the time that is essentially containing the observations, passes that through one or more networks and emits the parameters of the output distribution in this case logits. It then returns a tuple of three things. The first is an actual distribution object. Josh spoke about the probability and here is this low probability category called distribution built from the logits, it emits the next state possibly containing RNN state information, and it also emits site information. Site information is useful -- side information is useful perhaps you want to emit information you want to login your metrics and that is not the action, or maybe you want to log some information that is necessarily for training later on so the agent is going to use that information to actually train. OK. So now let's actually talk about training. The agent class encompasses the main ML algorithm and that includes the training and reading batches of data and trajectries to update the neural network. Here is a simple example. First, you create a deep queue learning agent, you give it a network, you can access a policy specifically a collection policy from that agent, that policy uses the underlying network that you passed in, and maybe performs some additional work like maybe performs exploration. It also logs site information that is going to be necessary to be able to train the agent. The main method on the agent is called train. It takes experience in the form of batch trajectories from a replay buffer. Assuming you have trained the data and performed well you might want to take a policy that provides more greedy actions and just exploits and takes the best actions it thinks are the best and doesn't log any side information. That is the plan and policy. You can save this to a saved model and put it into the plain. For a complete example, here we have a deep Q learning network. It has other arguments describing what kind of keras layers to combine. You build the agent with that network. Then you get a tf.data dataset. In this case, you get it from a replay buffer object, but you can get it from any other dataset that emits the correct batch trajectory information and you iterate over that dataset calling agent.train to update the underlying neural networks that are updated in the reflected policies. Let's talk a little bit about collection. Given a collection policy, it doesn't have to be trained, it can have random parameters, you want to be able to collect data and we provide a number of tools for that, again, if your environment is something that is in Python you can wrap it. The core tool for this is the driver. Going through that first you create your batched environments at the top. Then you create a replay buffer. In this case we have a tf.uniform replay buffer. This is backed by TensorFlow variables. Then you create the driver. The driver accepts the environment and the collect policy from the agent and a number of callbacks. When you call driver.run what it will do is it will iterate, in this case it will take 100 steps between a policy and environment, it will create trajecto trajectories. After the buffer finished, you have 100 more frames of data. Here is kind of the complete picture. Again, you create your environment, you interact with that environment to the driver given a policy, those interactions get stored in the replay buffer, the replay buffer you read from with a tf. data dataset and then the agent trains with batches from that dataset and updates the network underlying the policy. Here is kind of the set of commands to do that. If you look at the bottom, here is that loop, you call driver run to collect data, it stores data in the replay buffer, and then you read from the dataset generated from that replay buffer and train the agent and you can iterate this over and over again. We have a lot of exciting things coming up. We have a number of new agents we are going to release. C51 and so on. We are adding support for contextual bandits, we are going to release a number of baselines and as well as a number of new replay buffers. In particular, we are going to be releasing some distributed replay buffers, in the next couple of quarters, and those will be used for distributed collection. Distributed collection allows you to parallelize your data collection across many machines and be able to maximize the throughput of your training URL algorithm that way. We are working on distributed training also using TensorFlow new distribution strategy API allowing you to train at a massive scale on many GPUs and TPUs and we are adding support for more environments. Please check out TF-agents on GitHub. We have a number of Colabs, I think 8-9, exploring different parts of the system. As Sergio said, TF-agents is built to solve many real-world problems and in particular we are interested in seeing your problems. For example, we welcome contributions for new environments, new RL algorithms for those of you out there who are RL experts. Please come chat with me or Sergio after the talks or file an issue on the GitHub issue tracker. And let us know what you think. Thank you very much. Thank you. [Applause] And now I would like to pass it off to Nick and Yannick who will talk about TensorFlow.js. Thanks, guys. Hey, everybody. My name is Nick and this is my colleague Yannick. We are going to talk about JavaScript and ML. TensorFlow.js is a library we launched last year. It is a library for training and deploying ML models in the browser and on node.js. We want to showcase what you can do today with a platform and where we are going. One of the great parts about the library is there is really no drivers to install. If you run it in the browser you can get out of the box GPU acceleration. The browser itself tends to be interactive and builds great applications and demos for using ML. Privacy is a very important part to the library. You can run inference and training locally on the client which works around all sorts of privacy issues you might have with doing server side inference or training. And what can you do today with the library? We have a collection of, like, pre-trained off the shelf models you can use without any knowledge of ML. We also have the ability to take existing Python models and convert them and run them in TensorFlow.js. We also have a full-stack for training, inference and low level linear algebra and that runs in a browser in node.js. We also have a bunch of platforms that JavaScript can run on outside of just the browser. The first thing I want to showcase is some of our new off the shelf models we have launched. The first one is a bunch of pre-trained -- or off the shelf models are a bunch of pre-trained models. They are image, audio and text classification models. The APIs are all user-friendly. You don't have to worry about converting images to tensors. They are available on MPM and we have precompiled hosted scripts a well. We are working on this a lot in the upcoming year and we will have pore and more models as we go forward. The first model was body pics. I want to show you how easy it is to use this model. First include the library and body pics. It is two simple imports. The next step is to create an image DOM. This is Frank. He is trying to do something on the couch but I want to actually load the model and find body parts on Frank. The first thing to do is to load the body pics model. Just a simple one-line call. The next step is to call one of the methods we expose which is estimate person segmentation and I can pass in a DOM element. There is also a bunch of really easy to use methods for doing filtering on the image. So I can take the results of that and render it directly on there DOM. It shows head, body, arm and so on. Another model we launched a couple weeks ago is the toxicity model. To use this model, we will use the prehosted scripts. Two lines of code. I will load the toxicity model and ask the model to classify just some really lovely text, pretty pg, you suck, and I will get a result back as a JavaScript object that has seven labels we identify different types of toxic-type text and the probabilities and if it matches. We also have the ability to take pre-trained Python models for run them directly in browsers. If you have a pre-trained model trained in Python world we have a command line tool that makes it easy to serialize the model as a JSON object and the weights and distribute them in a web format. We support saveModel, TensorFlow Hub and keras models. It supports over 170 and counting ops and we will be TensorFlow 2.0 compatible. I want to walkthrough how simple it is to use this. I is a Python model and run it through the command line tool and I can easily load that in my JavaScript application. Very simple. I want to hand it off to Yannick to walk through the pre-trained model. TensorFlow.js allows you to train in the browser and in node. The primary tool for this is the layers API which is a keras-compatible API for authoring models. We are going to take a quick look at what TF.js code looks like. The first step to is import the library and when working in node.js you can also use the node.js bindings which execute the TensorFlow operations using native compile C++ code. If you are an a system that supports kuda, you can import -- cuda -- you can -- this is what creating a convolutional model for a classification task looks like. It is similar to keras code in Python. We start by instantiating a model and add a flat operation and a dense layer with output classes. Similar to Python we use model. compile and specify loss function and optimizer and model.fit is the function that drives the train loop. In JavaScript it is an a sync function so we want to wait for the result or to be done. Once it is done training we can save the model. We support saving to a number of targets both on the client and server. And you can use model. predict to get a result from the model, finally. Over the past year, we have also heard feedback from the community on ways we can improve the experience of training with TensorFlow.js. We would like to show you progress we have made. First up it is tf.data. It is an an API for managing data training. It provides a whole set of utility functions for dataset transformation. Finally, it works with streams and the lazy evaluation allows you to work with data that doesn't fit in memory which can be quite important. Let's look at a simple example. We load up a CSV file and use the is label attribute. For example, in this map transformation, the price data has been separated out into that Y object and the rest of the features are in the X object. Once we flattened the data we can apply typical ML transformations including things like shuffling, an ML best practice, and batting which will do the work of creating properly sized mini batches for training and know what to pull into memory when the training loop is running. Other things include things like normalization. Finally we run the train loop. Model.fitdataset supports models and pulls the right stuff into memory as needed. And that's tf.data. So the other area we have been responding to community feedback is visualization. I want to talk about tfjs-vis which is model of internals as well as evaluation metricsism you can view internals and evaluation metrics. First we import the library and you should node we provide tfjs-vis as a separate package. To visualize training behavior we can use this showshow.fitscallbacks function. In one line, fit callbacks will plot the selected metrics. In this case loss and accuracy metrics on the batch end and at the end of each epoch. This allows us to adjust hyperparameters as usual. You can look at model internals with show. model summary and show.layer summary. Here we see the architecture of the model including output shapes of SARSAR -- various layers. We see the distribution of layers in the first convolutional layers including statistics like nans and zero counts which are useful for debugging models. We want to finally announce TensorBoard support in node.js. You can monitor when using the TensorBoard layer in node.js. A single line will generate the necessary callbacks to write the methods to a TensorBoard log file using this command. Then you can open in TensorBoard and look at how your training is going just like you may be used to. With that, I am going to hand it back to Nick to talk about some of the platforms we execute on. JavaScript is an interesting language because it runs in a lot more places than you think. There is the traditional browser front for running JavaScript in the browser. We all know about that. Node. js is a big server-side solution. Very popular. But there is also a growing trend with JavaScript in more places. One of them is desktop applications and electron is a very popular platform for developing applications. Those who have used Spotify or visual code those are good examples of electron. JavaScript is moving into the mobile space also. I want to highlight a couple examples we have seen in the industry on all four platforms. First is the browser. Our friends at Google creative labs have built experiments on how creative tools could be more accessible to everyone. There is going to be a great lightning talk on this and they will talk about everything they have built with this project. Uber has built an in-browser tool for model agnostic visualization of ML performance. They use TensorFlow. js for acceleration of their linear algebra. K means clustering and so on. They are also giving a great lightning talk about how they use TensorFlow. js to solve this problem for their platform. This is all in browser, again. Another cool industry example is Airbnb. Airbnb built an identity document detection model that they use as a full TensorFlow ecosystem solution. On our Airbnb profile, if you were to upload a government-issued ID it is a very big trust issue. They built a TensorFlow model to detect if a profile picture that you are trying to upload directly in the client contains government-issued IDs. They use this in the browser using TensorFlow. js as well as on their mobile devices for TFLite. On node.js, a good example of this being used is clinic doctor. This is a node.js performance analysis tool. They use our node. js bindings to filter out gc spikes on node processes that are running to give a true accurate CPU performance benchmark. On the desktop, our team here at Google with magenta and their music generation models, have built a series of desktop plugins for the Ableton studio. These are full desktop applications that use the magenta models and accelerate them all into the GPU and desktop applications. We have a demo at your booth for how this works. The cool part is all JavaScript and GPU acceleration on the desktop with no cuda drivers all through the findings. Mobile is another interesting space. WeChat, for example, is one of the most popular apps in the world with over 1 billion total users and a subapplication platform called the mini programs. They are great because it is no-need install app in advance used on demand and has over 1 million apps and 1.5 million developers. The mini program is built using JavaScript and makes it easy for developers to create, deploy and share on the WeChat platform. I want to show a demo of a tfjs models running on WeChat. I have just a regular iOS device and I will open WeChat. Someone shared this tfjs example and I can load up the application. It is running our post net model and if I aim it at Yannick here, there we go, this is just purely done in JavaScript and it is running our off the shelf mobile model doing about 30 frames a second. This is all done with the WeChat JavaScript platform. platform. Thank you. So, all this work over the past year and the fantastic projects created by the community, makes us very excited to announce TensorFlow.js 1.0 today. It is available now and we are super excited to see what the community builds with it and hopes the API stability makes it easier going forward. We are focused on providing a stable API you can build applications on and manage upgrade easier and bringing market improvements in performance particularly on mobile devices. We will look at that in a bit more detail. To look at the update closer, since announcng TensorFlow.js last year at the Dev Summit we have been working on improvements across a number of platforms. We see increases of about 1. 4X to 9X in some extreme cases. This chart shows inference performance with a batch size of 1 on mobile net in Chrome. It is a mobile image classification and we see inference times going from about 15 milliseconds to about 150 milliseconds on the pixel 2. We have done performance on the iOS devices as well so really excited for you to try this. What is next for us? Well, we want to enable you to execute saved models on our node.js back end without going through the conversion process and this will open up many more models to serve using the js. node stack. We are always keeping an eye on browser acceleration proposals like simd and WASM. The browser is only going to get faster and so will we. We want to expand the platforms on which TensorFlow.js can run. We are working on platforms like the Raspberry Pi and other hybrid mobile platforms that run JavaScript. So thanks and for more information on the things we talked about, you can visit any one of these links. Thank you. And last up is Elena to talk about her work using machine translation in the Vatican secret archive. Thank you, Nick and Yanni. It is great to be here. Just one second. I am excited to be here and to talk to you about the In Codice Ratio project that is going on at Roma university. To talk to you about TensorFlow help us build a model that is able to transcribe ancient manuscripts. You can see the polyographers and archivist and on the left is the data science team. That is why I think the name, In Codice Ratio, reflects us well. In Latin, it means a known edge for transcript but in Italian it means software code so it is knowledge through software. You might ask yourselvesyourselves, what brings polyographer and archivist and data scientists together. They both want to discover knowledge from big data. Historical archives are an endless source of important cultural and historical information. Just to give you a scale of how large this information can be, let's compare for a second the sides of the Vatican secret archive to the height of Mt. Everest. If you were to take each shelve of the Vatican secret archive, you would get to 85 kilometers tall. That is about 10 times the size of Mt. Everest and the content spans centuries and continents. There you have examples of letters coming from China, from Europe, from Africa, and of course from the Americas. What is our goal? To build tools and technology that enables in general to perform large scale analysis on historical archives because right now the process is entirely manually. You have to go there, consult the documents manually and be able to read the challenging handwriting and if you find information that may be linked to another collection, you have to do it all by yourself. But first, we have to face the very first challenge and that is when you are dealing with web content, for example, if you want to extract data from the internet, that is already text. We said we are dealing with historical documents and that is often scans. Traditionally OCR is fine for printed text but then you get to this. This is a medieval handwriting and a language nobody uses anymore. It is a handwriting nobody is able to write or read anymore for that matter. It is heavily aggregated and you want to get text out of it. You might want to train a machine learning model but then we come to the second challenge and that is scalability in the dataset collection process. The graph you see there is an algorithmic scale and it might show you something you know that it is known as the zip flow that tells you that there is very few words and most of the words do not occur that often. What does that mean for us? If we want to collect data at vocabulary or word level, we have to annotate thousands of lines of text which means thousands of -- hundreds of pages. Similar systems do exist. There are state-of-the-art systems but most of the polyographer even when they know of the tools get discouraged using them because they say it is not cost-effective because it can take up to months or even years on these documents to get a transcription service they will use once or twice whereas they would like to do it faster. We asked ourselves how can we scale on this task? We decided to go by easier steps, simpler steps. The very first things we did was to select data for single characters. This enabled us not to involve polyographer but people are less experience. We built a custom crowd sourcing platform that worked like capcha solving. You see an actual screen there. The workers were presented with an image and target and had to match the target and select the areas inside the image. In this way, we were able to involve more than 500 high school students and in about two week's work we made more than 40,000 annotations. Now we had the data and we wanted to build a model. When I started working at the project, I was pretty much a beginner in machine learning and so TensorFlow helped me put in practice what I was studying in thereytheory. It was a great help that I could rely on tutorials and the community and where everything else failed even the source code. So, we started experimenting and we decided to start small first. We didn't want to overkill. We wanted the model to fit exactly our data. We started small and proceeded incrementally and in a constant cycle of tuning hyperparameters and the best optimizer and initializers and the number of layer and the type of layers and then evaluaing and training again. Then we used keras which allowed us to keep the code spall and readable. This is what we settled for. It might look trivial but it allowed us to get up to 94% average accuracy. Where does this fit in the whole scheme of the transcription center? We have the input image. We are relying on over segmentation and that is a bit old school but it allows us to feed the single characters or combinations of characters inside of the classifier which then produces different transcription according to a latent language model which we also built from publically available sources. How good do we get? We get about 65% exact transcription and we can get up to 80% if we consider minor spelling errors or if the segmentation is perfect. If we had perfect segmentation we could get up to 80%. We can see it can be more challenging. What are plans for the future? We are very excited about the integration integration of TensorFlow and keras. What we found out was that sometimes some were lagging behind and sometimes we wanted to get to one part of the features or from keras or TensorFlow and we found ourselves doing lots of -- I don't know if that is your experience as well but we found ourselves doing back and forth between TensorFlow and keras and now we get the best of the two worlds and are very excited about that. How do we plan to expand our machine learning system? First thing first, we are trying units that achieved good results on medical imaging. We are planning to use them to get rid of the segmentation and achieve the result of classifying together. Those examples worked particularly well. Of course, since there could be ambiguitywe could do error correction and transcripttranscription. We want to evolve and move to word level and even sentence level annotated characters, but still our focus is calbility and we want to involve polyographers as little as possible. This is generated input from GAN but we plan to use an encoder to evolve the dataset with little human interaction, like the less we can. And in the end this brings us to use sequence model that will take full advantage of the sentence level context, for example, and could be able to solve things that we could not be able to solve with single, let's say, with single character classification, for example abbreviation. Many words occur aggregated in this text just like you would text saying me too and use 2 the number or for you and that is the same with this kind of manuscripts. That is one application you could have. We are planning to use sequence models to get to a neural model because so far we have only experimented with statistics. And one last thing, I mentioned the people in the team but there is so many people I would like to thank that were not in those slides. First of all Simone who should have been here but he couldn't make it. And then py school of AI and Luke for their amazing mentoring and a high school teacher that actually allowed us to involve those students that work for the platform. And of course, all the graduate and undergraduate students that worked with us and helped us achieve what we have achieved and what we plan to achieve in the future. And of course, that you can -- thank you for your attention. [Applause] Thank you, Elena. All right. This is the end of day 1. We hope you had a great time. A big thank you to our speakers, our staff, our crew, and to the audience here in Sunnyvale and all around the world. With that, let's take the livestream out with a look at some highlights. Hello, everybody. Right now, right now, before you leave, there is a survey. Please, it has been emailed to you right now. Please do the survey. This is machine learning. We need data. No, seriously, like we can't bring these events if you can't find out what you think of them. Please, please fill out the survey. Next we have a social night. Tomorrow we have talks, breakout sessions, we have coding, we have workshops. Doors open at 8:00 A.M. and lightning talks start at 9:50. We will have breakfast and start lightning talks. We have breakout on performance, mobile and 2.0 and we have an amazing series of research talks. Please, please, we will see you tomorrow. Go have a drink. [Applause]
B1 model paige bailey moroney paige bailey data TensorFlow Dev Summit 2019 Livestream 6 0 林宜悉 posted on 2020/03/31 More Share Save Report Video vocabulary