Oh, my ukulele is very out of tune.

What luck this happens to be the make a ukulele tuner

coding challenge.

What a coincidence.

Well, I'm here in my new studio located--

it's not my studio, but it is a studio.

It's in Brooklyn at New York University.

And I'm doing my first coding challenge from here.

I don't really know what makes sense.

If this is a moment to do some sort of particular coding

challenge, I don't know.

I'm just going to do we're going to tune

the ukulele because the tuner I have, the battery is broken.

My ears are not so good.

So we'll see if this works.

I'm going to use the ml5.js library.

This is a JavaScript library that I'm lucky enough

to participate in its development

and I try to make tutorials with, so this works well.

It has a pretrained model for pitch detection, which

I'll talk about in a moment.

And then I'm also going to use the p5 Web Editor.

It'll be a little tricky to do this in the p5 Web Editor

because it's going to involve having

to upload a bunch of stuff, but the good news

is I already have the ml5 library right here.

All you need to do if you're in the p5 Web

Editor at editor.p5js.org, if you go to the ml5 website

under Getting Started, click on this link, which will open

a sketch in the p5 Web Editor with the ml5 library already

imported.

OK, so we're there.

I can make sure this is working by saying something like,

console.log ml5.version.

I'm going to hit Run.

And I see the version down here, so I've got ml5 going.

Now, what do I need to do next?

So if I go to the ml5 website, I'm going to Reference.

And in Reference, I can see that the various functionality

of ml5 is divided into different categories based on media.

And what I definitely want to do is look at sound

here because I want to find a sound model.

And lo and behold, there is a model called pitchDetection.

Now, I should mention that there are

ways of analyzing a sound for pitch that you don't

need machine learning for.

You could do FFT analysis and look at the different various

amplitude of different frequencies and pick the one

that's--

there's a variety of ways.

And people much smarter who know much more about sound

could tell you how to do that.

And I'm sure you could find other tutorials, but I'm here.

I want to try to use the ml5 library.

But this really begs the question,

like, well, how is this working?

You said something about pretrained model.

So ml5 comes with a certain model known as CREPE.

I don't know if that's how you pronounce it.

I like to say crepe, which is a Convolutional Representation

for Pitch Estimation or CREPE.

I like fruit-- a little banana--

and maybe a little Nutella is kind of good.

I don't know.

It's too much for me, the Nutella.

This is about pitch detection though.

That's what this video is about.

And so you can read a lot more about this particular model

and what data was used to train it, which is always

a question you should ask when you're using someone's

pretrained model because there's a lot of things that

can go wrong, or right, or be problematic potentially

about a model based on the data it was trained.

And this paper describes that in more detail.

You could click on this link over here to see a demo of it

in the browser, but we're going to do this in ml5.

And a big thank you to Hannah Davis,

who actually did the porting of this model into ml5.

And I'll include some links to her work

in this video's description.

OK, so here I am on the ml5 documentation page.

And it looks a little bit like, what's going on here?

So I need to create a pitch object.

And there's some sort of, like, string ./model/.

What's that?

First thing I want to tackle here is what this ./model/ is.

So a lot of times when using ml5,

it's going to load the model files from a URL.

And you might actually put the URL into your code

or ml5 might just know the URL automatically-- it's

saved on a Google server or some other server,

it's saved on GitHub and pointed to by ml5.

In this case, this is a case where I actually

need to have the model files with my code.

This was probably going to change.

Just by making this video, I've realized we probably

should host a version of the CREPE model

that you could access through ml5 more easily.

But luckily, if I go to the ml5-data-and-models GitHub

repo, where I am right here-- github.com/ml5js.

I can navigate to models/pitch-detection/crepe.

And these are all the model files.

So this is very typical of any pre-trained machine

learning model.

There will be a JSON file.

This is essentially a file that describes the model.

And then there's all these other files,

which are the actual weights, the numbers,

the secret sauce of the model after it's been trained.

All of those little parameters of the neural network

are all stored in all of these files.

Now, I've actually downloaded all of these already.

And you can see them.

They're right here in a folder on the desktop here.

So what I want to do right now is add all of them

to the P5 web editor.

Let's see how that goes.

I've really only worked with uploading little media files

and sound files, but I think this is going to work.

So I'm going to go here.

I'm going to create a new-- it's hard for you to see this,

but I'm going to create a new folder.

And I'm going to call this crepe.

And then I'm going to do Add File.

And I'm hoping that I could just select all of these

and drag them here.

So I forgot that the web editor currently only supports

certain file types, like JSON, or CSV, or JPEG image

files, sound files.

So these model files that include all the weights,

they can't be uploaded to the web editor.

And that's something that might change in the future.

But luckily, I can actually just point to the model files

that are on GitHub itself.

So this particular URL right here

where all these model files are stored,

there's actually a way to turn any file that's

sitting on GitHub into a URL that you

can load from a content delivery network.

And so a way of doing that, this is a nice blog

post that I found on gomakethings.com

that just shows this base URL.

So I can always access files through this URL,

cdn.jsdelivr.net/gh for GitHub, and then the path to the user

name, the repo, and the path of the files.

So I actually have done that right here.

I'm going to hit Refresh.

And you can see look, this is that model.json.

file.

And now, I can actually look and see,

oh, it's all of the configuration information

about this particular model.

And I can grab this, and I can put this into my code.

So let me close this.

I'm going to go to the top.

I'm going to say const modelurl equals,

and I'm going to paste that in there.

So now, I actually want just the path,

because I want the model to load all the files.

So I'm going to then remove model.json.

And this is the path to the crepe model.

And I can go ahead and just delete this

from here, this folder.

It's gone.

I guess I need to also delete the files one at a time.

Let me do that.

And now, I am ready to start putting in some code.

So I'm going to make a variable called pitch detector.

Let's just call it pitch.

I'm going to say pitch equals ML5 pitch detection.

Let me give myself some more space.

Pitch detection.

I'm going to go back to the reference page.

And these are the things that I need to load.

So let me copy-paste.

These are the parameters that I need to pass in.

I'm sitting here wondering why I have an error.

And of course, if I declare a variable as a const,

I can't just have it not equal anything and then assign it

later.

So this is going to have to be let.

And I'm going to make a lot of people angry right

now by making everything let just to simplify things.

And now, I have the--

I've now created a pitch detection object.

Now, what I want is to give it a bunch of arguments

to create itself, one of which is the model itself.

So this is no longer a local directory of files.

I'm going to say modelurl.

I need to get this audio context and mic stream.

Let me come back to that.

But I also need a model loaded function,

so that I know that the model has been loaded.

All right.

So audio context and mic stream, what are those things?

Well, let's hope the documentation tells us.

Audio context, the browser audio context to use.

Stream, media stream, the media stream to use.

I'm a little bit lost, to be honest.

Can I get some more information?

You know what I think we might do here is

let's just look at the example.

So the example here, if I look at the example code,

is showing, ah, perfect.

So an audio context I can just get by saying

get audio context.

Perfect.

So this happens to be something that's built into JavaScript.

It's part of the Web Audio API, I would assume.

And I'm sure somebody in the chat or someone

will leave a comment to explain what this is a little bit more.

But I'm just going to go ahead and put it in here.

I'm going to say audio context equals get audio context.

And then for the mic stream, this

is actually me connecting to the built-in microphone.

Or potentially, I could specify a different microphone.

And that I'm going to connect by using the P5 Sound Library.

So I can make a variable called mic,

and then I can say mic equals P5 Audio In, I

believe is the function.

So this is the function.

This is part of p5sound.js, which

incidentally is a library that I am accessing here

in indexed HTML.

And look at this.

I am on such an old version of P5.

Let's update this stuff.

And I think the current version is 9.0.

So while I'm here, I'm going to update that, go back to here.

And then I think I can get this mic stream from the P5 Mic

Object.

I'll just look it up here, mic.stream.

Perfect.

So this P5 audio-- oh, and I need to say mic.start,

start pitch.

I don't see any function.

Oh, interesting.

Look at this.

So I wasn't paying close attention.

I didn't really think what I'm doing, because I've

got to do two things here.

I need to load them--

I need to access the microphone, and I need to load the model.

When I load the model, I want to connect it to the microphone.