Everybody.

And welcome back to the unconventional neural network Siris In this video, only doing is running over some of the results from the neural network that was trained just to do addition, we're gonna talk about some statistics.

We're gonna see how it did and then what we're going to be looking looking to do, moving forward.

So with that, let's hop in.

First of all, how did we do?

So one thing I did was I wrote a few different kinds scripts just for comparing how we're doing as time went on.

So so one thing week we can do is inside model.

We should have, um, the train log directory.

So one option we have is to just do si m d Here.

In addition, Python.

Uh, actually, sorry.

Tensor tense, sir.

Board logger equals train.

Underscore.

Log.

Run that.

And then we want to open this.

So let me just pull that up over here.

So these are the results.

So we can We can obviously see how loss was doing.

So actually, this model could have continued to train.

Well, actually, I'm sorry this problem isn't fully loaded yet.

We've prided to wait a minute.

Yes, you can see it's slowly certain level out.

We'll just we'll count.

Need to wait for that to to finish processing.

So when one way that we can see how models doing is to look at train log butt, part of the issue is with with loss.

The loss is really all about the weights.

And when when we're looking at something like a mathematic problem, or in our case, like when we're doing the Mt.

Chatbots, even there's a reason why we used other metrics like Blue and PPL because, uh, loss could mean, Sure, we're learning something.

But are we learning the thing we were hoping to learn?

That might be a different question, so you often want to come up with separate metric for that.

So what I did was just manually as this model trained, I just took the output Dev as they came through, and I just rename them quickly.

So as they were coming, um, I would just say at what step point was that depth.

So at five k steps, 10 case steps and and so on all the way to 57 k when it finished training um, after about 100 K, I just went to bed.

So that's why I didn't I didn't get anything in between there.

Hopefully, our tensor board is is done now, so we can kind of pull that up, and I'm just gonna refresh it.

So let's just open up loss.

Okay, so So we can actually see after about 1 20 k steps.

It looks like I'm gonna guess that the learning rate was taken down there as well, Let's see, at 120 0 actually, yes.

So right before I went to bed, Um, I recognize that the learning rate was one negative five or should have been wanting negative five.

Yeah.

I wonder why That says one you negative for, um weird.

Oh, this is negative.

One negative for So the middle ground here is 20 negative five.

Anyway, um, before I went to bed, that was gonna be the last epoch, and I thought it was still only about 50% accurate.

So I decided, You know what?

Let's add somewhere a pox.

And when I did that, it jumped back up.

So interestingly enough, we were actually kind of done learning.

Um then I allowed it to jump back up and look how we just boom just jumped away down in los Um so if anything looking back, this is actually the first time I'm looking at this, uh, that jump up in learning rate.

Basically, what that tells me is we jumped down way too early.

We shouldn't have jumped down that early because putting it back up like we were were fixing a plateau right there basically, but allowing it to jump back up again comes back down here, and that's probably what sealed the deal.

So cool.

I'll keep that in mind because we're gonna make things a little more complex here, So, uh, cool.

So that's that's the tensor board.

But that doesn't tell us, You know, learning rate doesn't tell us.

Did it actually learn math?

So instead, what I used was the output deva file.

So just open one of those up.

Ah, do sublime text and we can see Okay, that's basically the output from the, um, the model.

And it's because it's a character level model.

The spaces aren't actually spaces spaces between three outputs.

Now, the input to these, um, was basically would be like new data on then from.

So from this, we should get this.

Is that correct?

That's right.

Right.

Anyway, um six.

Yeah.

All right.

So yeah, from here, we should get this obviously minus the spaces.

And the question is, is this corrector is incorrect now, even at 5000 steps?

Um, I just I know that this is probably about Kurt lips.

I know it's about correct because they're pretty.

They're at least close there within the same range.

The same ballpark, right?

But not all these air.

Correct.

So what I did was I just kind of manually saved those output Debs as they go.

So those were just that's, like, our way of validating things.

So every 5000 steps would about 5000 steps.

It would go to the thes testing one's, which contains samples out of So they're out of sample testing from, like, the training data itself.

So this should be samples that the model has never seen, and this is how it does on those.

So what I did was at least started with this file.

So let me open that up and we'll talk about that.

So?

So this one opens Dev.

10 k.

But we can start with five k even.

And what it's gonna do is gonna open up the test file.

So test from it's gonna open up the output file, and that's gonna open up test dot too.

So So what was it supposed to be?

And then what we're gonna do is just create a couple of counters here.

We're gonna follow pep eight.

We should put our spaces here.

Okay?

And you're Anyway, um, we're gonna do is we're gonna count, basically, how often were we correct versus how many total samples did we have in the set?

And, uh, and then we can get an actual accuracy rating, so that was like, the first thing I started doing.

Um, man, I got a lot of pep.

Eight paid issues here.

Every now in a while, someone says my coat is ugly, and I say you're banned from the channel, so let's go ahead and run this.

So as Reese can see here, basically, um what the output of this is what the input waas.

What are desired output was and what the model said, Hey, this is the output of like this, like the output of 43.8000 plus 98.8.

Basically, it should have been 1 42,000 and some change, and we said 1 44 So actually, almost immediately, we can see Hey, this this model is at least on the right track.

But at this stage in the game, um, to me, I'm starting to think, you know, is it only getting close because it's basically brute?

Forcing this and it's it's finding another example where it was close to 43 8 away, plus something close to 98,000 got millions of samples.

Um, is it just getting close because it's it's just like putting the output of what that was back in back in that example?

And so at least my line of thinking here is one that could be the case so long is accuracy, you know, actual.

Where are you?

Actual accuracy doesn't improve.

That's one.

So that's one symptom of maybe it's just brute force.

The other symptom of brute force is doing something like we're going to do later, which would be tracking the total absolute difference of the model out versus the desired out.

So so So not only should accuracy be rising over time, but the total difference between accuracy and what we intended kind of like loss.

Only this is much more exact to our task.

Um, we'd like to see that going down as well over time.

And if that starts going down over time, we can be relatively comfortable that no, it's actually seeming to learn some sort of math here, Um and then, of course, at the final stage of the game, if it is able to get any accuracy, so the 0.0 means it got none of them right, if it's able to learn any accuracy and that starts going up them were to assume that it's learned math.

And my argument further is that we had 100,000 so it 0 to 100,000 plus 0 to 100,000.

So that's 100,000 combos here, 100,000 combos here to get the total number of combinations on both sides of that plus sign that be 100,000 times 100,000 which should come out to 10 1,000,000,000 possible combinations.

Now the output isn't 10 billion possible combinations of output, but as far as inputs concerned, that's 10 billion.

And we only fed the right Think three million So someone can feel for you.

Correct me.

Sometimes I get those these kinds of scales wrong and stuff like that.

But to my understanding, like, you know, you could have 0 to 100,000.

Here's a 100,000 and then even here, you've got five and they've got 100,000 examples of five.

Plus what?

So 100,000 times 100,000, 10 billion?

So it can't in my mind be brute force either there.

But it is simple math.

It's just edition.

Okay, let's see how we did later on.

So that was five K.

How about at I don't remember.

I think I had a 15 K.

Let's just test it real quick.

So even at 15 k, we got some.

Yea is in there and it's 1.2%.

You might be asking house at 1.2.

We have 500 samples, so that's why so anyways, we have some.

Suddenly we're making some accuracy now, Like I was saying, the other thing that you might want to start doing is tracking the total difference.

Let's see if we're Yeah, I'm doing that in this one.

So in this one Ah, we start tracking not only the total accuracy, but also the total difference.

So So, starting from let's just do five K and then we'll jump to 25 cakes at five K.

The total difference.

So the, you know, think of it again.

It's just like the loss.

The total is almost 400,000.

Ah, but by the time we jump to model after 25 k steps, the total loss or the total difference, I'm calling it lost because that's kind of what it is in my mind for this exact problem is not lost.

Anyway, the total difference that it was off was only 21,000.

That's a massive improvement after just a 20,000 steps.

Now I know everybody's waiting for the final, um result.

So let me open up comparing number three.

And basically what I did here is, rather than printing and, um, you know, the actual desired out in the model out.

I'm just running through.

Well, where did it go?

Oh, I opened it in idle.

I'm so used to using Idol it's open it in Sioux blame.

So in this one, I just iterating over house really mad.

But I reiterate over all of the checkpoints that we have, so we can kind of see how it does over time.

So my apologies.

I just have to fix it.

And then pretty soon it's gonna get angry at me over violating 78.

Anyway, go and run it.

And as we can see, the correct rate was zero.

Here, the sum of all the differences.

400 k very quickly.

It drunk jumps down all the way down to actually after the full 2 57 case steps, although there's quite the gap between one and three and 2 57 So I went to sleep.

Um, we got 100% accuracy.

So this model again, there's 10 billion plausible combinations.

The model got 100% accuracy on the ad.

A sample testing, no difference, obviously because of 100% accuracy.

Um, so we could definitely train a model to do math, So this was actually really surprising to me to get a perfect score through that, And the other thing we can do is we can actually run the difference.

I haven't actually done this.

So So if we find that it's actually not accurate, I'm gonna be pretty pretty angry on video.

But let's go ahead.

Run it.

So what was the words that could go wrong?

Um, so this is just running inference, not pie.

Now, let's just do like a 567 Plus that, um I don't even know what the answer should be.

It was bought the old trusty calculator.

Where is the old calculator?

Ah, so what do we even put in?

567 567 plus 40 to 14.

47 81.

So the model's top results was 47 81.

Now, the way this works is, um it has a thing called the beam beam with and that's how many predictions it's gonna plausibly make.

And then from there you score them.

Um, basically, this is the model's top choice, But you can see the model was thinking about maybe it's these other things too.

But it chose the top result as 47.

Anyone.

That's when we go with Ah, we could do something Super simple.

Oh, so this one that actually gets wrong five plus two.

It believed it was two sevens, actually.

All the way down here, I'm gonna go ahead and guess it's probably not that great at simple math, Um, or like, small numbers, because that I bet small numbers were just, like, super rare for that should work.