Hello, welcome to another holiday coding challenge!

Again with the snowflakes!

This is my second snowflake challenge,

but in this time instead of creating an algorithm

to draw a snowflake, I'm going to ask a neural network

to draw a snowflake for me.

Now, there's a lot of pieces to all of the work

that's gone in to making the fact

that I'm going to be able to do this

in well what probably will be about four and a half hours,

but really could take just about 15 minutes possible.

The first thing I want to reference is the Magenta Project.

So the Magenta Project is a make music and art

using machine learning research project,

it's an open source research project from Google,

and it has a ton of examples,

projects with TensorFlow in both Python and JavaScript,

and you can look through at all these featured projects.

The project I want to mention is, I'll just click over here,

is this project called SketchRNN,

or draw together with a neural network.

Now, if you've watched any of my previous videos

about the Quick, Draw dataset,

the Quick, Draw dataset is this huge collection

of millions upon millions of doodles

that users from around the world made,

playing the Google Quick, Draw game.

So what the researchers from Google Brain did,

you can see them credited here David Ha,

Jonas Jongejan, and Ian Johnson

wrote this paper about how a neural network

could be trained on all of those drawings,

so it could learn okay when somebody's drawing a cat

they kind of go like this, and then they go like this,

and it could start to imagine new drawings of cats.

And it's called SketchRNN because the kind of neural network

that it's using is something called

a Recurrent Neural Network.

And you might have also seen this,

Nabil Hassein was here and did a guest video

about generating texting with a Recurrent Neural Network.

A Recurrent Neural Network is good

at learning about sequential information,

like hello world is sequential information.

H-E-L-L-O space W-O-R-L-D.

Music is sequential information (singing a scale)

that's a sequence of notes and rhythms.

A drawing is a sequence of vectors.

So you can read this paper all about how they used

the Google Quick, Draw dataset

to train a model to create new drawings.

You could also look at their demo,

and this is the code by the way for their demo.

There's a lot involved here in all of these pieces,

and here's the demo right here.

So, I'm going to pick cat,

and what I can do is I can start drawing,

oops, I can start drawing a cat,

I'm just going to draw a circle,

and then it will pick up for me

and finish the rest of the cat.

So at some point I will do a part two of this video

where I actually create an interactive version like that,

but one of the things that is in now the ml5 project,

ml5 is a friendly machine learning for the web library

that's built on top of TensorFlow.js.

It works with p5 which is the open source JavaScript library

that I use in a lot of my video tutorials,

is it has a SketchRNN module for you.

So on the one hand you might want to go

and look all the way through all this code

and read the research paper to learn how it works,

but if you want to quickly get up and running

with playing with a SketchRNN model,

what we've tried to put in ml5 will help you.

Now, (laughs) you might notice that if I go

click on reference, at the time of this recording

it doesn't say SketchRNN here.

So this is actually a new feature.

There's a couple issues in the GitHub repository

for ml5 about things that don't work perfectly,

or need to be added, or fixed, we welcome contributions.

I will come back and maybe do some more videos with it

once it's gotten further along and there is documentation

but for the holiday season let's see

if we can get it to draw a snowflake for us.

So, I'm not going to be able to use the website

because the documentation isn't there,

but I've got some documentation

over here on my invisible computer (laughs)

that I can look at if I need to.

So what do I have?

I have a blank p5 sketch with just setup and draw,

and also at index.html I should call this

Snowflake SketchRNN.

I have also in addition to the p5 libraries being imported,

I have the ml5 library and again

at the time of this recording it's version 0.1.3.

Hopefully there'll be future updates to improve SketchRNN

and there'll be a higher numbered version.

So the first thing that I want to do

is I want to make a variable and I'm going to call it model.

So model, I could call it SketchRNN,

maybe I should call it SketchRNN,

I'll just call it model, model is the thing

that's going to hold the SketchRNN model.

It is all of the information,

the brain of that neural network

that we can ask to generate new stuff.

So we can say, hey give me a new point

along the path that you're imagining to draw.

So I'm going to say model equals ml5

SketchRNN,

and then I'm going to just put a callback in here, model ready,

and this is the important thing, what goes here.

So there are a bunch of available models with SketchRNN

and we can see that list here, whoops, here.

For example there is cat, let's start with cat maybe

but we can see all the other ones like

the Mona Lisa apparently is in there, bird.

So let's, whatever is available we can put cat in here.

I'm asking ml5 to load the cat model.

It's actually doing this from the cloud,

the model, all of the data for this machinery model

is stored on a server somewhere,

and ml5 knows how to load that for you.

So, then what I'm going to do is add a callback,

I'm going to say model ready.

I'm going to write a function called model ready

and I'm just going to say console.log model ready.

So let's just run this code and see if it even just works

loading the model without an error.

So I'm going to close this demo down,

we have my own here.

Okay, great the model is loaded,

and just to be sure,

oh yeah initialized SketchRNN model ready,

just to be sure if I put in here like tiger,

which I don't think is something in there.

Oh, maybe it is (laughs)

guacamole, that must not be something in there.

I spelled guacamole wrong even.

Okay, right it couldn't even find guacamole

and I know that's spelled wrong,

apologies to guacamole lovers all over the world.

Okay, so we've got the cat now what do I need to do?

The first thing I need to do as soon as the model is ready

is ask for something.

So what SketchRNN will give you

every time you ask for it to generate something

is what is referred to as a stroke.

Now, I'm not going to use the variable name stroke

in my code because stroke is a global function in p5

that sets the outline color of a shape,

and I don't want to get confused with that,

but the stroke is a JavaScript object with a few properties.

It has dx, dy, and I think it's just called pen.

And dx is a floating point number, some number like 1.3,

dy is some number like negative four,

and pen is a string.

It can either be up, down,

or end.

And what this refers to is a particular

stroke of a pen basically.

So if you could imagine dx for the change in x, delta x,

dy for the change in y, delta y.

If I were to go over 1.3 units and up negative four units,

basically the stroke is this.

This is the path of the pen,

that's what SketchRNN should draw right now,

and it should either draw it as a line if the pen is down.

If the pen is up it should just move

from here to there, right.

Because if I'm drawing a cat, I'm going to draw this,

then I'm going to pick my pen up, move over here,

so that action is also a stroke

but it's a stroke with the pen up.

And then pen end is when the drawing is finished.

So this is what ml5 does, the actual native SketchRNN model,

just gives you all these things as array of numbers,

and so we've kind of made it as a JavaScript object

that makes it a little bit easier to read hopefully.

Okay, so now coming back over here

I think I just say model.generate

got sketch (laughs).

Oh, oh, oh, yes!

Now actually, an important thing that I should probably do,

I don't necessarily need to do this the first time,

but because this is a kind of machine learning model

that is giving us sequential information

the next stroke, like the one that comes after this one

is quite important, it's related to this one.

But if I want to start over and draw a new cat

I need to kind of go back to the beginning

and say start a new cat over.

And so a model.reset function is the thing that does that.

So model.reset I'm going to call that,

It should be reset the first time you load it,

but I'm going to call that anyway.

Then I can say model.generate got sketch

and then I'm going to write a function.

I'm going to call it got sketch.

It should get that stroke path.

Let's just call that S and let's have a variable.

I'm going to have a global variable called stroke path.

And I'm going to set that I just going to

set it to null when the program starts,

and then when I get a new sketch I'm going to say

stroke path equals S, and then I'm going to say

console.log stroke path.

So let's see if we get this first thing.

Model loaded!

Undefined oh.

(laughs)

That was close!

(sad trumpet sound)

I forgot, this is actually very hard for me,

I always make this mistake, ml5 is written in such a way

that uses something called error first callbacks,

this is different than how p5 is written.

So this got sketch function is receiving,

what was actually undefined there was the error,

there was no error.

All the callbacks need to have an error argument first.

So, if I wanted to be really thorough about error handling

I might do something like this.

So I could console out the error,

and then if there is no error

now just console log out the stroke path.

So this should work now.

Load that model!

Ah there it goes!

So look at this, I need to move

this many pixels in the x direction,

this many pixels in the y direction, and the pen is down.

So this should be easy enough for a sample then.

If I have an x and a y global variable,

I'm going to start them in the middle.

Then I am going to, in the draw loop I'm going to say

if stroke path is not equal to null, right.

I got to check do I have a new stroke path, right.

The new stroke path will come in the callback here.