And welcome to a video about neural networks that speak.

Every voice in this video is generated from a neural network.

I know people will claim they could told me all I could hear the difference.

There's absolutely no way I would be fooled by this.

I have some of the best years, not too big, not useful.

Just right.

Give it long enough and you might realize something is wrong.

But I think this voice is pretty convincing.

It's not a question of if there will come a time where we can trust audio or video.

It is.

Right now.

We live in a time we're almost full of our information.

Comes from Texas Rodeo and value only know all of this can be fixed.

It's entirely possible that made things we've already seen or heard were faked like this.

I wonder how people will use this technology.

It seems like it would mostly be used for bad things.

What can we even do about this?

The reaction of many will be to just ban things like this, and this is not necessarily the answer.

The truth is, Impersonating others for nefarious purposes is already legal van actress from other countries do not care about our laws.

They will add this technology to their arsenal of cyber attack weaponry.

Citizens must remain vigilant where they get their information and must remain aware that the friend of fake all you and that he was here.

Now we live in an hour where most people get their information online from many sources, including social media.

While this has allowed for more freedom of the press which underpins our democracy, it has also along for foreign legions to pin afraid your flow of information.

That is why tonight I am proposing that we built an Internet wallet is a horrible If we look at Young or Russia, they do not have this crowd.

I don't think you understand how this wall would stop the flow of then Information might be the dumbest idea I have ever this wall you've ever seen but know this would be a beautiful Absolutely not.

People would come to visit just to see my Internet wall.

No, I estimate that the money made from cheers to see the Internet wall would pay for the Internet Will continues.

We might have this, Kyle, the issue here is just like any fig news.

Often the truth takes a long time to get out.

And some people will just never see the correction.

With the last year's presidential election, we saw just how much fake information could impact is by clocking up information.

And that was just with fake text.

I wonder how much more destructive fake audio and video will be all right and back to reality.

Or is it just getting or am I?

Anyway, I, uh, would like to do now is show you guys how you could get started with this, what's involved and hopefully simplify it a decent amount.

I had no idea how complicated sound could actually be with Spectra Grams and things like 48 transforms all this stuff, eh?

So I'm not gonna be getting too deep into the weeds, just kind of show you how you could do what I've done and kind of what I found on by no means done with doing fake audio.

And the next thing I'd like to do is fake video on.

Then combine the two.

So, um, there's still a lot I'm trying to do, and I just I don't really feel like doing the line by line thing.

But I thought I would share with you guys what I found so far.

So with that, the model that I ended up going with is this D C T.

T s model, which is, um I used this code here, but it's based on the following paper which take note was written in late 2017.

So almost two years ago, actually, But you know, the history of doing Texas speech on computers is and has always been, pretty much Someone goes into a studio and records hundreds of hours of audio, and then we use things like digital signal processing, or DSP is I'll probably refer to it from now on.

Uh, to do things like working on the pauses between words and sentences and then do do things to do, like the speech pattern and stuff like that.

It's all these like tools and hard coded, rule based things for Texas speech.

In most cases, even when doing Texas speech with neural networks, historically, people have applied Maur rule based things to the output.

Now, obviously, the dream would be to do text to speech without any of that just simply throw in some examples, get output.

That is just awesome.

So that was the goal.

And also side goal is to figure out How can we do this?

And how quickly can we do this?

And could we just do this with, like, anything on like YouTube?

Like, could we just take just about any YouTube video and mimic a voice on that video?

So with that in mind, this is the model I ended up with.

But I did try other models, including Taco Tron and Taco Tron V two, and I found this one to be the best, which is kind of curious and one of the things that you might notice if you've been reading and not listening or both, is this as deep convolution.

All that's right.

It's a convolution, l nor network.

There is not a recurrent cell to be seen, which is also interesting because traditionally you would think of a task like this to go from text to speech.

Text is a sequence.

Speech is also it's not like a single scaler value, right?

It's a sequence, so most of the time you would think, um, I would use the sequence to sequence model, which is a recurrent neural network.

Nope.

This is using deep convolution neural networks with attention.

Now this is guided attention, which will explain in a moment.

But interestingly enough, it's turning out over the last couple years to be the case that deep convolution, all neural networks are actually outperforming recurrent neural networks at these sequence to sequence types of tasks.

They're doing it in their training faster, and they're getting better results.

And that's awesome.

I'm not sure that I would be ready to say recurrent neural networks are dead.

We just need a better way to do recurrent neural networks because the biggest thing is the amount of data that's required and how slow they are to train.

So if we could speed them up, we might see good things.

So, um, for example, I recall using a recurrent neural network on like M n'est data.

So it's not like recurrent nor networks aren't like.

They're just as, um, trying to think a topic wide as a convolution Eleanor network can be, So I wouldn't necessarily write them off completely.

Um, don't forget, Neural networks themselves got written off completely not long ago, So anyway, um, yeah, Cool.

So what's guided?

Attention guided attention is this.

It's basically the assumption that everything is gonna be a linear.

So the input to the output will always be linear, as opposed to consider a machine translation task.

Where let's say you're going from English to Spanish and you've got big house needs to translate to Casa Grande A.

Okay, well, the adjective to describe that noun in English comes before the noun, but in Spanish comes after the noun.

So it's not a linear relationship.

In all cases, other words might be, but some are gonna be flipped around.

And so attention is there to actually help with this sequence of sequence, which was previously mostly used for recurrent neural networks.

But now we're using them in a confrontational neural networks as well.

And, it turns out, with convolution, all neural networks plus attention, we can do the exact same thing with the added benefit of doing it way faster.

The other nice benefit that I found is the end result is just better, so really cool.

So, for example, I just want to play a quick example of, um, let me see where it is Yes, o of a Texas speech that is pretty close to the typical ones I would find by training a mediocre sample size of, say, 600 samples.

So I'm just gonna play one real quick.

Hi.

Is lying programming like that?

Okay, um, you probably might not even understand what is being said there, but I'll play it one more time.

It's saying Python is the one and only true programming language is one.

So, yeah, that doesn't work well, So that's Taco Tron on 500 samples.

One thing to note, Taco Tron requires 100 more like 100 plus hours of sample audio and weeks to months to train.

So I did not do Taco Tron justice.

Also, there's now Taco John V, too.

So, um, just understand that that's just a example.

But I wanted to show you because every time I've ever played with Texas speech, that's about what it sounded like.

Like in terms of Texas Beach with neural networks.

So there's something there you can tell.

The neural network is kind of grasping onto things, but it's not.

It's just not there.

It's just simply not there.

So then I trained this deep convolution alone.

Neural network with this, this guy's code, Um, and I'll try to put a link in the description.

If I forget someone remind me.

You can also just google like d c T T s get hub.

And I'm confident you'll find this page.

Um, Anyways, I trained this model, and this one is on the l J speech data set, which is here, Um, which is a female voice.

I think it I don't even know.

I don't I don't know if it's Kate Winslet's voice, because there's he's got these other sample audio's You've got Kate Winslet, Nick Offerman, and then I'm not sure whose voice it is, but it's a singular voice.

You need to train with just one voice.

Otherwise you'll get a mashing of voices, which maybe that's what you want, but probably not.

Anyway, I used the L J speech data set and trained for I want to say about, like, three days like the 1st 1 The so this this model has two networks will explain that Maurin a moment the 1st 1 took about one day.

The 2nd 1 took about two days and this is the result that I got.

Now, this voice is quite good, in my opinion.

And it was trained with many hours of sample audio from an audiobook.

Okay, so that is still a little robotic.

You can, if you really pay close attention, you can kind of hear these weird little fluctuations with somebody who knows more than I do.

I really don't know much about digital signal processing and all that stuff.

Someone who's really good at it could fix that voice and use like something really specific, probably for that voice and fix it.

But this is straight from the from the D C T t s model output from here on, get up.

And that's pretty good.

I mean, that's clearly a person we would accept listening to that audio.

Um, I think I was just I was pretty staggered by those results because I've never seen a Texas speech model produced such results besides Taco Tron, which is a big pain to train now.

That data set consists of 13,000 samples, and it's about 24 hours total of audio.

It's a lot of audio.

So my initial thought was, What can I do this with YouTube and how little how few samples can I use?

So then I started.

I tried to use my own videos because I felt like if I'm gonna fake someone's voice a price should do my my own first and then do other people's.

It's kind of like if you're a police officer and you wanna have a Taser, they make you taste yourself first.

It's got the same thing.

Um, so So So So my hope was that I could take my YouTube videos and then also take the caption file for the YouTube videos.

So I tried that.

But the problem is, the caption file doesn't quite like the time stamps on the caption file aren't quite right.

And then the caption file also eliminates often things like, um and, uh, other like stutters and seven.

So it's just because of the guided attention.

Um, this is such an integral part here because Theus assumption is everything will be linear.

Everything will line up.

We're saying with 100% certainty, So we need it to actually lineup.

So has to be perfect.

Unfortunately, caption file no go so that I'm thinking, Well, maybe captions or that way because, um because it's better for the person to read it so you could read it faster if we don't do include this stutters and stuff.

So then it's like, Okay, let's try using, like Google's speech to text.

Try that similar problems.

It would chop things.

It would leave things out.

It would fix stutters.

It would get rid of bums and us.

So then I'm like, Well, I guess I'll manually transcribed this stuff And I don't think probably anyone watching this video has manually transcribed audio.

But let me just say it is the worst thing.

It is the most painful thing.

I can't think of a thing more painful than transcribing audio Attn Least, unless it's like, really clean audio.

So in the in my case, there's a lot of stutters.

There's this and, uh, lots of us, and lots of times, I, like, repeat a word twice or three times, and sometimes it's really hard to hear, and you have to go back and play and like it is, What I found personally is my like, I just naturally will nix when people say, uh um, like a lot of times, I just like, don't even hear it.

So I'm sitting here typing it out, and then usually I'd play it back one more time or two more times after it was fully typed out just to make sure.

And it was just staggering How many times I would miss things, even like to twice I would miss things when I'm, like, really trying to pay attention to it.

So, um, really interesting.

And also just painful just painful to, uh, to, ah transcribed things.

So then I'm like, Oh, my gosh, because they're, you know, the L J speech data set is 13,000 samples.

I was finding I could do about 100 samples and our transcribing of so and then I could do it, Max about like 200 a day.

Beyond that, it was like just killer just too, too brutal.

And so I do the math.

That's two months to get a new voice and truck outside my house.

A love a love on your recording.

Like every time I record big trucks have to come by, but okay, so after manually transcribing about 600 samples, the result I got Waas Python is one only short of programming language.

So that one's not too bad.

There's a couple of small issues.

One is it sounds like I'm underwater.