Everybody.

And welcome to part five of these self driving cars with Carla Tensorflow, Kare Aas and other things.

Basically, we're trying to reinforcement learning here.

Okay, that's a tutorial.

Siri's where we left off.

We created the Caryn and we created the agent, and now we're just going to tie everything together and actually hopefully start the training process.

So with that, we're gonna make a couple of imports here real quick.

Import tensorflow as t f import Kare aas Dante back end dot tensor flow underscore Back end as back end and then from threading will import capital t threading with those imports, we're gonna go to the very bottom, and we're gonna start our main dare.

I call it a loop, but really, if statement if, uh, name equals Dunder.

So if this is the script that's running, we're gonna just set, we'll say F p s is 60.

Um, you'll see why in a moment we'll wind up.

Probably.

There's got to be tweet.

This is Ah, we we wish kind of number.

We're not gonna get 60 but anyway, uh, and then yep, rewards will just prime that with a negative 200 to begin.

Uh, random dot seed one.

We're gonna just set some of random seeds.

Basically, this is just for repeatability mp dot random dot seed one and then the tense airflow dot set underscored random underscore.

Seed said that the one as well, this'd so we get as repeatable is possible things so we can kind of compare into bug.

So now what we want to dio is most people probably don't have to do this.

But you're running multiple things on your GP you or you're like me.

And for some reason, you're GPO really wants to allocate more memory than it possibly has.

We're gonna do GPO underscored options.

Also, if you don't want to run like multiple agents, you have to do this.

We're in a T f G Pew options.

Uh, and then we're gonna say, per underscore, processors were GPU our score memory underscore for action.

And then we're gonna say that whatever memory for action is, so scroll up to the bottom or the top here, minus set toe eight.

I'm actually gonna downgrade that to six, even, uh, for you, you can probably say one point.

Oh, uh, or 00.9 or you know, whatever.

Especially if you want to continue using your computer while on agent trains.

You consent it even lower than that.

And then you might not even notice that you've got training going on in the background.

And let's try to play some video game or something.

Um, so anyway, that's how you can tell, Basically, tensorflow, You know, it's pretty obvious, but this is per process.

So basically per model, each model will, in my case, use 60%.

So if you wanted to have, like, five models, each model would have to use less than point to write for your memory for action.

So Okay, cool.

Coming down here.

That's the GPU options.

Now we need to set that, so we'll stay back end dot set underscore session TF dot capital s session config equals t f dot fig Perrotto.

That's title cased.

And then we're just gonna say GPU options equals GPU options.

Cool.

So tensorflow now will hopefully listen to me.

Okay, Now that we've done that, we're almost ready to start training.

But we need somewhere for our models to go so and say if not os dot path Thought is, is your Come on.

Oh dear.

But models, so that path doesn't exist us dot Make ders models done.

Then we're gonna define our agent equals D Q and agents, And then we need and that's gonna be car end.

Cool.

So once we have that, we're ready to initialize the trainers or as a trainer, underscore thread Eagles Capital T thread Target will be equal to agent dot train in Loop.

Lehman equals true, uh, and then trainer thread dot start And then, while not asian dot training initialize.

So until that flag ish is true, we're just going to timed out sleep for a minimal amount of time.

Basically, just wait here until we're ready to go.

And then, uh, just like if you recall training loop here we do a model dot fit, um, to kind of prime the fit mints because the first one's gonna take so long.

And then we engage in our infinite loop.

We kind of want to do the same thing with the predictions because your first prediction will also take an exceptionally long amount of time.

So we're just going to say agent dot get underscore cues, n p dot ones and we're going to say end dot ym.

Mmm.

Mmm mmm.

Actually, now, hold on.

This might be different in my notes.

Let me go to tippy top here.

No, we do.

If we have, uh, well, we've got court end, so m dot Yeah, in height and then comma m don eme with I still see this in Versace image.

Um, that has me a little cautious at the moment.

I think we've got an air, their imjin image height.

So we got a couple of those image height, and I don't Yeah, and we don't have any definition for those, so that's that's gonna That would have been a problem.

So what we want to do is I'm going to control H.

I'm gonna search for image height, and I'm gonna replace that with him.

Height, replace all, and then I'm gonna do the same thing for width.

So wits.

And that needs to be INGE with him with cool.

So that would have bitten me.

Don't worry.

We'll have other errors that I'll hit.

I'm positive.

So, uh, in height and width, uh, and then three.

So again, it's just gonna get some Q values for something that is totally irrelevant and not important.

Anyway, I totally just dribbled.

You saw nothing.

Um okay.

Uh, now we want to Dio is basically start iterating over however many episodes we have, so we're gonna say four Episode in T Q d.

M did I importante que dio?

Uh no.

From T to d m.

We're going to import tedium back to the bottom.

Okay.

Four episode in teak UDM.

Uh, range 12 episodes plus one, and then we're gonna say asking equals true.

This is for our Windows brothers.

Um, so it looks pretty still, and then So if you're on Lennox or something, you don't have to say, ask equals true.

And then we're gonna say unit is episodes.

I don't forget what the default unit is, but at least it'll make sense at this, So m iss Okay, then.

Ah, what we want to do is say end dots collision underscore.

Hist is going to be an empty list and trying to side if I want to put this, Uh, no.

Uh So this is where our collision censors gonna start slapping in information.

Uh, the next thing we're gonna say is agent dot tensor board dot step Steppe Eagle two episode.

So you've got you've got steps.

And normally what would happen is every frame Wouldn't theory be a step to tensorflow?

Because every every frame will It would even be wolf the bachelor.

Anyway, it would at least every frame be a step.

Let me put it that way.

And we don't want that.

That's way too absurd.

So instead, we're actually saying Okay, every episode is a step.

Um, So now tensor board will show us, uh, basically a plot per episode instead of intra episode.

D'oh!

Uh, we're gonna say episode underscore.

Reward is equal to zero, and then step equals one.

Let me fix that.

And then we're gonna say current underscores state equals m dot resets gives us our first observation ever done.

Will be false to start, and then, eh, piss episodes start equals time.

That time I'm getting over a cold, and I can the brain is still a bit foggy, but, uh okay, So for every single episode, we, um we at least set these values we do in every set.

And now we start the actual while True, which basically is wow, not done Pretty much Um, but just because the done flag gets triggered, we're gonna do a couple of things in the exit immediately.

So we're actually just going to, like, we want to dictate exactly where we exit s.

So anyway, we're just gonna use a while true.

And then, if done at a specific point, we break.

So while true, if n p dot random dot random is greater than whatever the Epsilon value currently is, we're gonna say action equals NP Don't Argh, Max Agent, get underscore queues for the current state.

But if that's not the case, we're going to say action equals np dot random dot rand int between zero up to three and then we're gonna say time dot salute, sleep one divided by MPs.

So So what's the deal there?

So again, we're not gonna actually get 60 frames per second.

We're probably gonna get like, I don't know, 20 to 30 on a really nice GPU and then four on a badge.

Eep You were really probably none.

You can't run Carl on a bad GPO, unfortunately, uh, but the so when epsilon is really high, you're gonna take a lot of random actions.

Well, taking a random action.

Um, we get to avoid the agent.

I get cues, so we get to avoid doing a dot predict.

So we get to avoid using the neural network at all, which is what's gonna take time.

Whereas a random dot rand int this is like a nearly instant operation is gonna happen really quick.

So this is problematic because we actually want the agent to whether it takes a random action or takes ah, model action.

We want that time that it takes to be about the same right, because otherwise because frames per second really helps the model.

So the higher the frames per second, the better the models probably going to behave.

And if you take a random action but like a split second later, that is way quicker than normal.

You take a model action just not good.

We want we definitely want the friends per second of either one of these to be as close as possible.

So the way that you can eventually figure out what friends per second you're getting is at least wait until the, uh, your memory replays filled up and you're actually making predictions and then hat like set your Absalon toe like zero and then read your friends for a second.

That's how many friends were saying that you'll get for if you predicted all the time and then you can set this value, you can also guess.

I mean, it's probably unfair of me to say 60 at all to even start with.

No one's gonna get 60.

I don't think I don't think I would get 60.

So maybe put 20 or something like that.

Um, I'm gonna put 60.

I put.

I'll just stick with 20.

That's why we're not gonna get I actually might get 20 on this machine, but it's it's unlikely, But anyway, I'll go with that.

That's fine.

So, uh, all right, once we've done that.

So while true, we take our action.

Uh, the next thing we want to do is actually dio new state comma reward.

Done underscore because we don't have a value for that where they end dot step based on whatever action we decided to take Episode underscore reward plus equals, Whatever the reward Waas Then we want to update.

Yes, I'm happy with that.

Thank you.

Look, we want to update, so we'll say agent dot updates underscore a replay underscore memory.

And we're gonna update that with our transition.

Basically, And that transition is current state comma action reward new states and then done.

Okay, so then once we've done that step plus equals one, and then finally, if we are done, break cool.

Um, and then if done, break, um, and then we would come back here true.

And then four Actor in end dot actor underscore list, lest list actor don't D Uh, Okay, so So let me break this part.

So this is what we'll do during the episode.

So we're just updating a replay memory and then also take note.

So So so here we're making our actual predictions.

And then we're updating the replay memory for, like, the target model on all that.

And then you might be wondering what's going on.

The target model.

Well, don't forget, we are training in loop here for that actual target model.

So that's off doing its own thing.

It's taking an information from where well, from our games, where we're actually updating the replay memory.

So So basically, this is doing everything at this point that we need to do both to play and train so predict and trained at the same time.

Okay, Now, uh, we want to track some stats, but I think I'm gonna not type that out.

There's one.

I think we've typed it out before, into it will waste time.

Um, so we do have some stats that we want to collect and trying to decide if there's anything else here in the texts of pulled or text based version.

Um, did I not post?

I must not have posted the owner that I know what that is that So here I am.

I suppose we could take because we need to take this.

And we also need to take our tensor board script, so I'm gonna grab both of those things.

So I put a link in the description for both of those things.

And then basically, when you get there to part five, um, so just in the description, there would be a link that says, like, text based version, it'll be sample coded, text based tutorial or something along those lines.

The first little block is the code that we started with.

But if you continue scrolling down, usually it's at the very bottom, but actually at the very bottom of gotta play script for us.