Hello?

Hi!

Welcome back!

We made it to 5:00.

Oh, my gosh!

Woohoo!!

Having a great day so far?

Yeah?

The weather is beautiful?

Awesome.

So this is going to be our last talk in this room, so after this talk is done, I'm going

to encourage everyone to head over to the SitePen Track to watch our last talk of the

day, and then there's an awesome party, of course, so our last talk is going to be by

Andy Ruestow and Bryan Donovan, and this is kinda cool, actually, they are going to show

us how to use TensorFlow to rate yoga poses.

So the fun fact for both of them is kind of funny because it's like the exact opposite.

So Andy actually is going to be the one doing the yoga and he doesn't really do yoga.

He's climbed a lot more mountains than he's done yoga, but Bryan has done a lot more yoga

than he's climbed mountains, so they complement each other very nicely.

So let's give it up for Bryan and for Andy.

[applause]

Hey, everybody, thanks for coming.

>> Hello.

My computer is locked here, so you can watch me type my password.

All right, so we're going to get started here with the yoga pose and if you've been in this

room earlier today, you've probably seen a lot of really great presentations.

David just gave a really great one about Imposter Syndrome which I think we're both feeling

a little bit right now.

>> Absolutely.

>> And he also mentioned some things about confirmation bias and a quick little story

about that.

I've got a wife and two kids, and they were lucky enough to come along with us on the

trip here today.

And any time I go on a work conference, my wife calls it a work vacation.

So I always explain, no, it's -- we're actually, we're learning, it's, you know, very dedicated,

we are spending a lot of learning new technologies, not learning people, so when we're driving

up to the resort yesterday and we saw the palm trees and the beautiful son, I think

the confirmation bias set in and this is in fact a work vacation.

>> She needs a vacation.

>> Well.

She's at the beach today.

>> So just a little bit about Bryan and myself.

Buyian is a software lead at Lockheed Martin.

He's been developing software for pretty much his whole life but over the last 20 years,

really as a lead developer, chief architect of a lot of cool systems.

He is really the driving force, like the conductor and really the engine to drive a lot of our

programs forward and that's where my train analogy runs out of steam.

Myself, I'm a DevOps tech lead which means that I'm not good as a developer and not good

at relations, but what I'm really good at is to be able to enable developers to be the

best at what they're at and for humans it is being creative and we're best at solving

problems and a problem we're not really good at solving is doing repetitive tasks over

and over again.

I take a lot of joy in automating all of those things, so that the creative people, especially

like Bryan, can do what they're best at.

>> Yeah, DevOps makes our life better, absolutely.

>> Andy: I little bit more on the personal side.

And to highlight how we're opposites.

I live in Upstate New York and that means that I enjoy winter and I have two small humans

as roommates and like I mentioned, they're at the beach today.

They say that you should try everything at least once once in life just to see what it's

like.

I've done the science on this next one, you guys -- don't get hit by a car.

It's not so much fun.

And am primarily a carnivore.

>> Bryan: For myself, I live in Los Angeles, where we experience zero months of winter

each year.

I live near a beach, Venice Beach, I have two roommates, wife and dog and I've been

hit by zero cars, a much better experience.

Herbivore, so no meat and what we can both agree on it beer and coffee.

>> Andy: I think that's what make us such great friends.

>> Bryan: That's right.

>> Andy: So why are you here?

Primarily to see us make fools of ourselves.

Bryan called me up a few months and said hey, why don't you come to Southern California

and go to a conference and a couple weeks later I said, hey, Bryan, why don't we present

at that conference.

So a few of the technologies we use for yoga possess.

Node.js, I don't think they'd let us in the door if we didn't use.

>> React, TensorFlow one of the fun toys that we don't get to play a whole lot with our

day jobs.

Some pose estimation that happens in real time in the browser and then some of the deployment

things that I find pretty interesting.

>> Bryan: So React, yes, the one with hooks, really great if new tech in 16: 8 I'm sure

you've seen a lot of those, and great, we're moving over to hooks, we're going to use hooks

for this whole thing.

Look how great use state is and then I'm off using use effect and then I want to share

some global states and now I'm using use context and then I have this mutable thing that I

want to keep track of and so now I'm into useRef and then you want to use the set interval.

So you're using a custom hook called useInterval.

And it's super interesting.

Dan wrote a blogpost on how it can be used instead of set interval.

>> Yeah, it's really interesting how the underlying hook technology allows you to take the ones

that React has published and use them.

But also extend them in ways that you can be creative about and find new uses about,

like the one that Dan published for handling intervals.

TensorFlow, how many people have heard of TensorFlow?

Several.

How many have used it before?

Fewer, great.

So TensorFlow.js is cool.

It allows you to train and deploy models in a browser or in a node environment.

I think it makes machine learning easy.

We work with a lot of data scientists, and they're really big nerds and they make machine

learning actually easy, like the things that they do is really impressive and it makes

our life as more front-end and application developers, easy to take what they've done

and really just implement it, so it was fun for us to jump into TensorFlow and actually

start playing with some of the models, creating our own and seeing how we can work with them.

Another great thing about TensorFlow is the docs.

There's a handful of tools that I think have really amazing docs.

Docker is one of them.

TensorFlow had really great docs.

Bootstrap is amazing that stuff.

So when a tool has really great docs, I think it makes it easier to understand it or at

least how to use it.

>> Bryan: Specifically out of TensorFlow there's a model called pose net and what it's used

for is to determining pose information in real time or you'll feed it a video or image

and what that model will give back to you are 17 different body parts, it will show

you where somebody's eyes are, where the nose is, shoulders and you can take that pose then

and basically run your algorithm against those 17 different body parts to do whatever you

want.

In our case, we're going to be scoring some yoga poses, but the pretty cool thing about

PoseNet where we get to see the browser and how far it's come is it runs completely in

the browser, so there isn't any data being sent out to some external server.

That pose estimation, that neural network is all running exclusively with your browser

and just use a convolutional neural net to basically return those pose and there's a

neural net under the hood that's using to determine oh, a human is in this image, these

are the parts of that human and then mapping out the XY coordinates that you can then use.

Andy: So out of the box, PoseNet ships with a handful of models.

It's a mobile mold.

It gives a lot less accuracy when coming up with the key point positions and confidence

in each of those.

There's a beefier one, called ResNet 50 which I really wanted to work and it really crushed

the processer, to the point where we were getting like 10 to 15 frames a second, which

sucks, so we couldn't go with that one.

So we used mobile net v1.

Then like Bryan mentioned, out of the box it can detect multiple people which is kind

of cool and could have some implications for different ways that you could use Posenet.

So let's have some fun with the first toy that we created.

Face detection and tracking so here's Will with some really cool glasses on.

Bryan, I don't know if you want to step out and show some code here.

>> Bryan: Absolutely so basically here ... >> Andy: Did you hit the go button?

You have to -- perfect.

There we go.

>> Bryan: Thank you.

Basically see we're grabbing that pose, the pose consists of those -- I'm 17 points that

we talked about before, and we're basically going to go into that pose, look at the key

points and say where is the nose, where is the eye, and just do some simple math here

to determine how wide the glasses are, we'll do a ratio of the image that we're going to

be using.

We'll do a calculation so just doing the inverse tangent of looking to calculate the angle

of the face so we can apply the glasses in the correct orientation and just doing a quick

translation based on the nose so we can place them in the correct spot, applying the rotation

based on the angle and we're going to draw that image onto the canvas.

So what does that look like?

Sorry, Andy.

>> Andy: Yeah, no worries.

So you can see we fully embraced the saved by the bell theme here.

Do you want to talk about how you added some key press magic here?

>> Yeah, so basically what we do, there's Andy.

Bryan: We wrapped the Pose net model in the React and basically we're hooking the keyboard

and we can do some simple logic to say we want to render this canvas on top of the video

we're seeing.

This harsh light is a little rough, but -- hahahaha [applause]

Andy: So I think we also learned something pretty important about users by coming here

and being on this stage.

It's super-easy as developers to build something in a way that you expect your users to interact

with it.

So for Bryan and I sitting in an office, sitting in our desks in our homes.

>> Andy Bryan: Nice ambient lighting Andy: Yeah, and then you come up here and

you have white background and harsh lighting so I think it's great to think that users

are going to abuse your systems, so here's a great example.

There we got Bob there.

Here's another one of the technologies that we used.

It's the Canvas API.

This is pretty need.

You're able to do basic drawing manipulations just with JavaScript.

It's native to what, all browsers?

And there's a bunch of libraries that wrap that, that make things easier.

I guess we did it the hard way just by manipulating Canvas directly but there's always a break

point where you realize, do I need this additional library?

Do I not need it?

And I think it's a good practice to try and do things yourself first, insofar as you can

still make progress.

As soon as you find yourself running into a brick wall and the progress is slowed, it's

a great opportunity to bring in some of those other tools, but because we had this simple

use case, we decided to use Canvas just directly.

A few other things we did here related to Canvas and also having access to user media

is to grab the camera that user might have available.

So there's a few different ways that that can be tuned to inspect which devices a user

has available to them whether it's a front-facing or rear-facing camera.

Bryan: So once you have those 17 key points to actually start being able to understand

or how to score how that those body parts are in relation to something else, we need

to turn those into vectors, so just basically taking each one of the 17 key points, iterating

through those, creating vectors through an additional key point and basically we're creating

a bunch of XY vectors in that space that we can start to say, OK, based on this vector,

how is it going to compare to some other vector that you're expecting?

And the way we did that is we created basically like a gold standard of a yoga pose and we

said, OK, if we were to look at that yoga pose, what would it look like in X-Y vector

space and create that model.

So that's a algorithm that's running that says let's compare that vector to another

vector and use something called cosine similarity.