RICHARD WEI: Good morning, and welcome.

My name's Richard.

JAMES BRADBURY: I'm James.

RICHARD WEI: We're thrilled to talk to you

about a next-generation machine learning

platform, Swift for TensorFlow.

TensorFlow is a world-leading ecosystem for machine learning,

and we're always looking to make it better

for all kinds of users.

TensorFlow 2.0, available in alpha,

makes it easier than ever to get started with ML.

But we're also looking further beyond.

In this session, I'm going to tell you

about a project to rethink what a machine learning

framework could look like.

You might hear Swift for TensorFlow and think,

this is going to be about iOS apps.

But Swift is also a general purpose language

that we think will help enable the future of ML.

When figuring out what a next-generation machine

learning platform should look like,

we have been guided by three fundamental goals.

First, we would like to make sure

that every developer can be a machine learning developer.

Second, we'd like to make sure that every application can

benefit from the power of ML.

Third, we've seen that the frontiers of ML research are

pushing the boundaries of a software stack split between

Python and C++.

So we want a unified platform in the friendly and fast language.

This is why we decided to develop something

a little outside the box.

So we've taken the Swift programming language,

a general purpose and cross-platform language that

compiles to native code, and built general machine learning

functionality right into its core,

and combined its power with the TensorFlow ecosystem.

And this is Swift for TensorFlow.

Swift is a language designed to be easy to learn and use.

Let's take a look.

Swift's syntax is familiar and free of boilerplate.

Swift has types, so it catches errors early for you.

But it uses type inference, so it feels clean and simple.

Swift encourages fluent API design.

So your code is fluent and readable

at both the definitions side and the call side.

Swift is open source and has an open process

that allows us to propose new changes, new language features

for all Swift developers.

Swift supports prototyping platforms

such as a command line interpreter, Jupyter Notebooks,

and Playgrounds.

Just import your favorite libraries

and get started right away.

The TensorFlow library includes our carefully curated Tensor

APIs, layer APIs, as well as general abstractions

that help you build your own.

Now let's look at how to build a deep learning model in Swift.

This is an image classification model.

This is built out of several layers,

forming a simple convolutional neural network.

The call function applies every layer sequentially,

producing a final output.

Once we have defined the model, it's time to train it.

We start by initializing the model

in the stochastic gradient descent optimizer.

We'll be using random data here, x and y.

Now we write a training loop, apply the model,

compute the loss from labels using the softmaxCrossEntropy()

function, and compute the gradient of the loss with

respect to the model's parameters.

Finally, we can use the optimizer

to update all the differentiable variables and all parameters

in the model.

Our layer and optimizer APIs are inspired

by libraries like Keras, and are designed to be familiar.

A huge part of Swift for TensorFlow is workflow.

Lots and lots of research breakthroughs

in machine learning come from prototyping ideas.

Google Colab is a hosted Jupyter Notebook environment

that allows you to prototype your ideas right

in your browser.

A lot of researchers and developers

use Python with TensorFlow in Colab today.

And now we've made Colab work seamlessly with Swift.

Now let's show you Swift in action.

This is a Colab notebook.

As we start typing, we get an immediate code completion.

Here we use functional map to add 1

to each element in an array.

And it prints just like you expect.

Now let's look at a basic workflow in more detail.

Here's the dictionary of numbers.

And I want to find the largest even number in each array.

To do that, I'm using a for loop that loops through all elements

where the element is both greater than the largest--

than the last largest one found and is even.

Let's run that and see what happens.

Oops, what's wrong?

Well, it looks like Int doesn't have a property called isEven.

No problem, let's add one.

Even though the Int type is defined

in the standard library, we can still

add methods and properties to it.

In Swift, this is called an extension.

Here we define a Boolean computed property

called isEven, implement it in terms of the isMultiple method.

As you can see, Swift's syntax and naming conventions

look more like English than other programming languages.

And now everything works.

By importing TensorFlow, we get the power

of TensorFlow operators as functions and methods

on a Tensor type.

Many of these methods are the same

as they would be in NumPy and Python TensorFlow.

Other APIs make use of Swifty features,

like generics and argument labels.

TensorFlow also provides deep learning building blocks,

such as layers and optimizes.

Here's the same simple [INAUDIBLE]

that we just showed you.

We're on GPU, and it's running really fast.

Running the cells train the model.

It is that simple.

So that was Swift for TensofFlow in Colab.

Writing Swift for machine learning

is not all that different from writing dynamic languages.

With Colab, you can start Swift right away in your browser,

and it gives you code completion, free GPUs,

and you can even install a package, using Swift package

manager, right in your notebook.

You don't have to install anything.

It's really handy.

Next up, interoperability--

Swift interoperates transparently with C.

So you can literally just import a C header and use it.

But how about all the libraries--

everybody's favorite machine learning and data science

libraries in Python?

Do we have to read/write everything in Swift?

The answer is, we don't.

In Swift for TensorFlow, we have extended

Swift's seamless language interoperability to Python.

As you start using Swift for TensorFlow,

we want to make sure that you didn't miss all the utilities

and libraries that you're used to.

Now let's show you how interoperability

works in Swift.

Python interoperability is enabled by a lightweight Swift

library called Python.

Let's start by importing that.

Python.import lets you import any Python package installed

with pip.

Hundreds of popular packages are pre-installed in Colab.

Here we're going to use NumPy and Matplotlib.

We also enable their in-notebook visualization functionality.

Now we can call some common NumPy and plotting functions

directly from Swift exactly as if we're writing Python,

and it just works.

Now, OpenAI Gym is a Python library

of reinforcement learning environments, little games

for AIs.

Let's import that and solve one of the games.

One of these games is a cart-pole environment

where you train a neural net to balance a pendulum.

We'll use a simple two-layer neural network.

In each episode of the game is a loop.

We get observations from the Python environment

and call our Swift neural net to choose an action.

After running the game a few times,

we use the games that went well as training data

and update our neural net's parameters so that it will

pick better actions next time.

Repeating this process, we're getting better and better,

higher and higher rewards.

Finally, the problem's solved.

We can also plot the mean rewards.

As you see, you can mix and match

idiomatic Swift code, Python code, and a neural net defined

inline in Swift.

It is so seamless.

In fact, Python interoperability isn't hard-coded in the Swift

compiler.

It's actually written as a library in pure Swift.

And you can do the same--

do the same to make your libraries--

to make Swift interoperability work with Ruby or JavaScript

and other dynamic languages.

Now, in cutting-edge research and applications,

people often need to dive into low-level implementations

for better performance.

Swift's C interoperability makes that surprisingly easy.

Importing Glibc gives you access the C standard library from

qsort() all the way down to malloc() and free().

Here we allocate some memory using malloc(), store string,

and print it.

Now it's probably fair to say that Swift in Colab

gives you a better C prototyping experience than the C

language itself.

Now I hope you'll agree that the best parts of Swift

for TensorFlow is how easy it is to work

with code that is not in Swift.

And that is interoperability.

Having a few Python libraries in hand

is incredibly useful for day to day tasks in data science.

Swift for TensorFlow embraces interoperability

and lets you use C and Python without wrappers--

so seamless.

Now it's time to talk about the dark magic in Swift

for TensorFlow, differentiation.

It is fair to say that calculus is an integral part

of deep learning.

This is because machine learning models are really

just mathematical functions expressed in code.

But the special thing about them is that we don't just

want to evaluate those functions,

we also want to change the parameters,

or train them by changing their parameters based

on their derivatives.

A common technique adopted by deep learning frameworks

is called automatic differentiation.

Different from traditional programming,

doing calculus in code requires the language or the library

to understand the structure of your code.

But who to understand your code better than the compiler?

We think that computing derivatives from code

is not just a simple technique, but also a new programming

paradigm in the age of machine learning.

So we've integrated first-class automatic differentiation,

differentiable programming right into Swift.

First-class differential programming

means that native functions like this can be differentiated.

Here's a function of floats.

You can use a gradient operator to get its gradient function.

And just call it.

It's that simple.

And you can import TensorFlow, of course,

to compute gradients over tensors in the exact same way.

In fact, we've been using this feature in the model training

examples that you just saw.

Now instead of telling you how cool it is, let's show you.

Here's a simple function that works

with the native double type in Swift.

It squares x and adds y to it.

If we mark this function with differentiable,

the compiler checks to see if it can differentiate everything

that this function calls.

Now f is a differentiable function.

One of the things we can do with a differentiable function

is to get its gradient.

A gradient represents derivatives

of the output with respect to each of its inputs.