BRETT KUPREL: I'm Brett Kuprel.

I'd like to tell you about some of the work

we're doing at Stanford on skin cancer image classification.

This project has been a collaboration

between the Artificial Intelligence

Lab and the medical school.

So, let me warm up with some facts

to motivate the threat of skin cancer.

It's the most common cancer in the United States.

One in five Americans will develop skin cancer

at some point in their lifetime.

And in 2017, it's estimated that there

will be 87,000 new cases of melanoma, which

is the deadliest form of skin cancer,

and almost 10,000 deaths from it.

Fortunately, there is good news, there's hope.

The survival rate for melanoma is 98%

if you can detect it early on.

Also, by 2020, it's estimated that there

will be $6.1 billion smart phones in circulation globally.

So, this collaboration began a couple of years

ago when a dermatologist at Stanford

saw the recent breakthroughs in computer vision.

And he e-mailed the director of the Artificial Intelligence

Lab.

And he said, if your program can differentiate between hundreds

of dog breeds, I believe it can make a great contribution

to dermatology.

So, the first step to making a contribution

is acquiring a data set.

So, we have acquired a data set of almost 130,000 images spread

out over 2000 diseases.

We worked with the medical school

to clean it up and put it into this nice taxonomy, a subset

of which is shown here.

So, in this taxonomy, you can see green nodes and red nodes.

Green is safe, red is dangerous, black is deadly.

And there's also some orange nodes,

which could go either way.

It's not clearly a binary task.

So, on the next couple of slides I'll

show some benign and malignant lesions.

So here's some malignant lesions.

Here's some benign lesions.

So, you can kind of see flipping, back and forth,

a visual distinction.

But there's also some lesions that

are very similar between the two, which I highlighted green,

for instance.

OK, so now we have a data set, next up is to train it.

We find that training on finer classes

results in better performance.

So, if you consider the figures shown,

if we train on the green nodes, we actually train on more--

like 700 of them-- but just, imagine

we trained on the green ones.

At inference time, we would sum the probabilities

up the tree to the red nodes, the classes of interest.

And then further, if we're interested in a binary task

of distinguishing malignant melanoma from its binary

look alike--

from its benign look alike, melanistic benign lesions,

we would renormalize the probability sum to 1.

And this is consistent with Bayes' rule condition

on the information, that the disease

is one of two things, not one of all the nine red nodes.

So, we found that we got the best performance by fine-tuning

an [INAUDIBLE] pretrain model.

We tried a few different architectures.

We train with AlexNet, VGG Inception, versions one

and three.

We found Inception version three worked the best.

We also tried feeding with a spatial transformer network,

because many of our images have lesions taken from far away.

We didn't find improved performance,

and using Occam's Razor, we just didn't use it.

So, the next step is evaluation.

We used two metrics to compare with dermatologists--

sensitivity, which is known as the true positive rate,

and specificity, known as the true negative rate.

So, we show the doctors--

dermatologists-- a bunch of images of benign and malignant

lesions.

We can then calculate their sensitivity

as the percent of malignant lesions

they were shown that they ordered a biopsy for.

Similarly, the specificity is the percent of benign lesions

they were shown that they did not order a biopsy for.

And if they ordered a biopsy, it goes

to a pathologist, which results in a near-perfect diagnosis.

And for a network, we actually, for a classifier,

we output a malignant probability.

So, imagine we fed the same images through,

and we got the probabilities as shown.

Then the sensitivity would be the percent

of malignant lesions that falls to the right of some chosen

threshold.

Similarly, the specificity would be the percent benign lesions

that lie to the left of the threshold.

And in this particular threshold,

it would be a sensitive classifier,

because almost all malignant lesions lie

to the right of the threshold.

And we can vary this threshold to get

a whole range of sensitivity and specificity pairs.

So, doing that, we get these results

on three different tasks.

So, the first task is the most prevalent type of skin cancer.

Second task is the most dangerous type

of skin cancer versus its look alike.

And the third task is the same as the second task,

except it's using dermoscopy images, which

are taken with a dermoscope.

It's a special device that shines polarized light

at the lesion to expose underlying layers of tissue.

And you might think that dermoscopy is harder,

but, no, it's just a completely different set of lesions.

And we also see that the dermatologists don't

do as well in that data set.

So, let's see how the dermatologists do.

So, this is the most important slide.

We can see, for one, the line is more jaggedy than the previous,

and that's because this is a subset of our test set.

Dermatologists have important things to do.

They can't just classify thousands of images.

We can do it really fast.

Another thing you notice is dermatologists

are actually widely varied in their performance.

Anyway, from these plots, we conclude

that we have achieved dermatologist level performance

at skin lesion image classification of skin cancer.

So, here's some confusion matrices on a nine way

classification task.

It's interesting to look at these to see the similarity.

One thing you might notice is that we often

mispredict inflammatory.

It's such a broad category.

Another thing you can notice is that dermatologists

will err on the side of guessing a benign lesion is malignant

than the malignant lesion is benign.

Because that would be a deadly mistake.

And you can see that from this box.

OK, so I brought a demo of this classifier.

So this one-- I can't actually read the text, but--

Yeah, you can see it's malignant pigmented.

Let's do a couple of these.

This one's epidermal malignant.

There, that's probably good.

Don't want to embarrass myself.

So, I just want to say I'm honored at how well received

our work has been.

It's remarkable that Andre and I didn't know anything

about dermatology going into this,

and I think that's kind of the spirit of deep learning.

You can just get a large enough data set

and feed it through Google's latest image classifier.

Fine tune it, let them do all the hard work, take the credit.

And it's just a really amazing time to be an AI researcher,

as evidenced by three covers of "Nature" dedicated

to breakthroughs in artificial intelligence, two of them

here at Google.

I just wanted to comment a little bit on the future of AI

applied to dermatology.

And I think this could also apply to other skin diseases.

But there will be increased access, because a lot of people

might have access to a smartphone who don't

have access to a dermatologist.

Also, it will be more convenient to classify your lesion,

and because it's more convenient,

it will hopefully lead to earlier detection.

And the survival rate is much better

when you detect it early.

Thank you.