that the Toyota team uses, which I really love. They call it the mobility

[teammate] concept. In this particular model, the car is a virtual

copilot that arguably drives even better than you do.

It drives better than you do because it's paying attention all the time. It's

paying attention because it's a computer. It sees all around itself. And if we could

create the technology to make it possible to drive by itself, this virtual

copilot will keep you out of harm's way. I kind of imagine this idea where your car

essentially has this virtual bubble around it, virtual force field around it.

You never run into anything else around it. It knows exactly where to go. And it

keeps you out of harm's way. The second vision is a car that has no drivers at

all. In both of these examples, the computing

capability necessary to achieve autonomy,

self-driving capability, is going to be far greater than anything that's currently

available. And we'll talk about that in a second and make it clear why that's

so. In both of these visions, if we could realize it, the

contribution we can make to society is really, really wonderful and great. And so

our vision is to create the computing platform necessary to make this possible.

The self-driving program loop basically works like this. There's a sensing part.

You get all kinds of sensors. You want sensors that can see during the daytime,

during nighttime, during rain, during fog. All of the different types of conditions

that you could possibly imagine

surrounding the car, you would like to be able to sense. It could be radars.

It could be LIDARs. It could be cameras. It could be ultrasonics. You have inertial

measurement in units. You have GPS. All of these sensors contribute to sensing

where you are and what's around you.

There are several different [blocks] that I'll talk about. The top block is the map,

the precision map. That map was generated in advance. It was done through scanning,

probably LIDAR scanning. You see these cars that are running around mapping

the world in 3D, measuring the world and precisely mapping every part.

In the middle, that block is called localized. That has something to do with where you

are. Based on what you measure, based on what you perceive, we have to figure out

where you are within a few centimeters. The bottom block is perception. Perception --

seeing everything around you,

whether it's during daytime, driving directly into sunlight, in snow, in

fog, in rain. Whenever it is, we need to perceive where you are and

what's around you. And based on all of that information, we have to figure out a

way to plan your path. So starting from where you are, you perceive the world. We

figure out a way to calibrate that with a pre-known map. And then, based on

everything that's moving around you,

your movement where you are in the world all the movement of all the other

objects around you -- whether it's pedestrians crossing or bicyclists or

motorcyclists or cars that are moving all around you -- you need to find a way to

find your path. That's called path planning. You do this in basically

an infinite loop. You just keep doing this continuously. You're perceiving the

world continuously, you're comparing it against the map continuously, you're

localizing continuously, and you're planning the next step

continuously. And we do this as fast as we can so we can control the car and

make the car drive and path plan, so that it keeps you out of harm's way. It turns

out all of this is relatively hard to do. Each and every single one of these

blocks are hard to do.

Perception is hard, localization is hard, path planning is hard,

ffiguring out what sensors to use, using the right amount of sensors,

but modestly so we can make it cost-effective -- hard to do. All of these

components are hard to do, and there's innovation,

R&D, and discovery in every single component of it.

Self-driving cars is hard. It turns out that driving is hard.

When you think about highway driving, we've constrained the problem enough

that highway driving has become relatively easy, per se, to the point

where grandparents can drive. We all largely stay in our lane, we're traveling

about the same speed, and so relative to each other we're hardly moving. Highway

driving is relatively easy. And even then, there are many conditions by which

highway driving hasn't been solved. We have highway driving in all kinds of

conditions. If a truck carrying some junk, some parts of it fell off onto a

road -- that happens, as you know, pretty much all the time. And when that happens,

your car has to do the right thing, take evasive action, and keep you out

of harm's way. So, even highway driving is hard. But we can't realize

the full potential of our vision unless we can solve also city driving. Now, in

city driving, almost none of us are going in the same direction.

You've got cars coming sideways, you've got people walking all over the place, bikers

are of course on the same road you are, motorcyclists. And people are of

course sometimes following the rules and largely not. So, you can't read a manual

from the DMV and figure out exactly what program to write into

your car to cause it to drive properly. And so city driving is very chaotic. It's very,

very hard, and the perception problem, as you can imagine, explodes. You have to

perceive all kinds of cars, all different types of people, all different types of

bicyclists, all different types of environments.

Sometimes the road is being fixed; sometimes there's construction going on.

Sometimes the lane is painted over from a new lane, and all of the sudden the number of

lanes you look at with your car is rather confusing for the car. And so,

everything around that environment is chaotic.

It's complex, it's unpredictable, and oftentimes it's even hazardous. So, self-driving

cars are hard.