Hopefully you can hear me well.

Yes?

Yes.

Great!

So, welcome to Course 6.S094.

Deep Learning for Self-Driving Cars.

We will introduce to you the methods of deep learning,

of deep neural networks using the guiding case study of building self-driving cars.

My name is Lex Fridman.

You get to listen to me for a majority of these lectures

and I am part of an amazing team with some brilliant TAs.

Would you say brilliant?

(CHUCKLES)

Dan Brown.

You guys want to stand up?

They're in the front row.

Spencer, William Angell.

Spencer Dodd and all the way in the back.

The smartest and the tallest person I know, Benedict Jenik.

Well you see there on the left of the slide is a visualization of one of the two projects that one of the two simulations, games that we'll get to go through.

We use it as a way to teach you about deep reinforcement learning but also as a way to excite you.

By challenging you to compete against others

if you wish to in a special prize yet to be announced.

Super secret prize.

So you can reach me and the TA's at deepcars@MIT.edu if you have any questions about the tutorials, about the lecture, about anything at all.

The website cars.mit.edu has the lecture content.

Code tutorials, again like today, the lectures slides for today are already up in PDF form.

The slides themselves, if you want to see them just e-mail me but there are over a gigabyte in size because they're very heavy in videos so I'm just posting the PDS.

And there will be lecture videos available a few days after the lectures were given.

So speaking of which there is a camera in the back.

This is being videotaped and recorded but for the most part the camera is just on the speaker.

So you shouldn't have to worry.

If that kind of thing worries you then you could sit on the periphery of the classroom

or maybe I suggest sunglasses and a moustache, fake mustache, would be a good idea.

There is a competition for the game that you see on the left.

I'll describe exactly what's involved

in order to get credit for the course you have to

design a neural network that drives the car just above the speed limit sixty five miles an hour.

But if you want to win, we need to go a little faster than that.

So who's this class is for?

You may be new to programming,

new to machine learning,

new to robotics,

or you're an expert in those fields but want to go back to the basics.

So what you will learn is an overview of deep reinforcement learning,

of convolutional neural networks,

recurring neural networks

and how these methods can help improve each of the components of autonomous driving -

perception, visual perception, localization, mapping, control planning and the detection of driver state.

Okay, two projects.

Code named "DeepTraffic" is the first one.

There is, in this particular formulation of it,

there is seven lanes.

It's a top view.

It looks like a game but I assure you it's very serious.

It is the agent in red,

the car in red is being controlled by a neural network and we'll explain

how you can control and design the various aspects, the various parameters of this neural network

and it learns in the browser.

So this, we're using ConvNet.JS

which is a library that is programmed by Andrej Karpathy in javascript.

So amazingly we live in a world where you can train in a matter of minutes

a neural network in your browser.

And we'll talk about how to do that.

The reason we did this

is so that there is very few requirements to get you up and started with neural networks.

So in order to complete this project for the course,

you don't need any requirements except to have a Chrome browser.

And to win the competition you don't need anything except the Chrome browser.

The second project code name "DeepTesla"

or "Tesla"

is using data from a Tesla vehicle

of the forward road way

and using end-to-end learning

taking the image and putting into convolutional neural networks

that directly maps

"or aggressor" that maps to a steering angle.

So all it takes is a single image

and it predicts a steering angle for the car.

We have data for the car itself

and you get to build a neural network

that tries to do better,

tries to steer better or at least as good as the car.

Okay.

Let's get started with the question,

with the thing that we understand so poorly at this time

because it's so shot in mystery

but it fascinates many of us.

And that is the question of: "What is intelligence?"

This is from a March 1996 Time magazine.

And the question: "Can machines think?"

is answered below with, "they already do."

So what if anything is special about the human mind?

It's a good question for 1996,

a good question for 2016,

2017 now,

and the future.

And there's two ways to ask that question.

One is the special purpose version.

Can an artificial intelligence system achieve a well defined,

specifically, formally defined finite set of goals?

And this little diagram

from a book that got me into artificial intelligence as a bright-eyed high school student

they are artificial intelligence to modern approach.

This is a beautifully simple diagram of a system.

It exists in an environment.

It has a set of sensors that do the perception.

It takes those sensors in.

It does something magical.

There's a question mark there.

And with a set of affectors acts in the world, manipulates objects in that world,

and so special purpose.

We can,

under this formulation,

as long as the environment is formally defined,

well defined;

as long as a set of goals are well defined.

As long as the set of actions,

sensors,

and the ways that the perception carries itself out as well defined.

We have good algorithms

which will talk about

that can optimize for those goals.

The question is,

if we inch along this path,

will we get closer to the general formulation,

to the general purpose version of what artificial intelligence is?

Can it achieve poorly defined,

unconstrained set of goals

with an unconstrained, poorly defined set of actions

and unconstrained, poorly defined utility functions rewards.

This is what human life is about.

This is what we do pretty well most days.

Exist in an undefined, full of uncertainty, world.

So, okay.

We can separate tasks into three different, categories,

formal tasks.

This is the easiest.

It doesn't seem so, it didn't seem so at the birth of artificial intelligence

but that's in fact true if you think about it.

The easiest is the formal tasks,

playing board games, theory improving.

All the kind of mathematical logic problems that can be formally defined.

Then there is the expert tasks.

So this is where a lot of the exciting breakthroughs have been happening

where machine learning methods,

data driven methods,

can help aid or improve on

the performance of our human experts.

This means medical diagnosis, hardware design,

scheduling,

and then there is the thing that we take for granted.

The trivial thing.

The thing that we do so easily every day when we wake up in the morning.

The mundane tasks of everyday speech,

of written language,

of visual perception,

of walking which we'll talk about in today's lecture

is a fascinatingly difficult task

on object manipulation.

So the question is that we're asking here,

before we talk about deep learning,

before we talk about the specific methods,

we really want to dig in and try to see what is it about driving,

how difficult is driving.

Is it more like chess which you see on the left there

where we can formally define a set of lanes,

a set of actions and formulate it as there's five set of actions - you can change your lane,

you can avoid obstacles.

You can formally define an obstacle.

You can the formally define the rules of the road.

Or is there something about natural language,

something similar to everyday conversation about driving

that requires a much higher degree of reasoning,

of communication,

of learning,

of existing in this under-actuated space.

Is it a lot more than just left lane,

right lane,

speed up,

slow down?

So let's look at it as a chess game.

Here's the chess pieces.

What are the sensors we get to work with on an autonomous vehicle?

And we get a lot more in-depth on this

especially with the guest speakers

who built many of these.

There's radar.

There's the Rays sensors.

Radar lidar.

They give you information about the obstacles in their environment.

They'll help localize the obstacles in the environment.

There's the visible light camera

and stereo vision that gives you texture information,

that helps you figure out not just where the obstacles are

but what they are,

helps to classify those,

has to understand their subtle movements.

Then there is the information about the vehicle itself,

about the trajectory and the movement of the vehicle that comes from the GPS

an IMU sensors.

And there is the rich state of the vehicle itself.

What is it doing?

What are all the individual systems doing

that comes from the canned network.

And there is one of the less studied

but fascinating to us on the research side is audio.

The sounds of the road

that provide the rich context

of a wet road.

The sound of a road that when it stop raining

but it's still wet,

the sound that it makes.

The screeching tire

and honking.

These are all fascinating signals as well.

And the focus of the research in our group,

the thing that's really much

under-investigated

is the internal facing sensors.

The driver,

sensing the state of the driver,

were they looking?

Are they sleepy?

The emotional state.

Are they in the seat at all?

And the same with audio.

That comes from the visual information and the audio information.

More than that.

Here are the tasks.

If you were to break into modules the tasks

of what it means to build a self-driving vehicle.

First, you want to know where you are.

Where am I.

Localization and mapping.

You want to map the external environment.

Figure out where all the different

obstacles are,

all the entities are,

and use that estimate of the environment

to then figure out where I am,

where the robot is.

Then there is scene understanding.

It's understanding not just the positional aspects