Human Computation - VoiceTube: Learn English through videos!

Subtitles section Play video

MALE SPEAKER: Today we're very pleased, very happy, to have
Luis Von Ahn here today, from Carnegie Mellon University.
His talk is on human computation.
Luis is a very new assistant professor in computer science
at the School of Computer Science at Carnegie Mellon
University.
He received his Ph.D. in 2005, and I'm told he was the
hottest new graduate on the market, with offers from just
about every university out there, including corporate
offers, too.
He received his B.S. from Duke University.
He received a Microsoft Research Fellowship Award.
His research interests include encouraging people to work for
free, as well as catching and thwarting cheaters in online
environments.
His work has appeared in over a hundred news publications
around the world.
New York Times, CNN, USA Today, BBC, and
the Discovery Channel.
Luis holds four patent applications and has licensed
technology to major internet companies.
Please join me in welcoming Luis Von Ahn.
[APPLAUSE]
LUIS VON AHN: Can you hear me now?
OK.
So, I want to start by asking a question to the people in
the audience.
How many of you have had to fill out a registration form
for something?
Like Yahoo, Hotmail, or Gmail, or some sort of web form where
you've been asked to read a distorted sequence of
characters or a distorted word such as this one?
How many of you found it annoying?
Awesome.
OK, well, that was part of my thesis.
That thing is called a CAPTCHA, and the reason it's
there is to make sure that you, the entity filling out
the web form, are actually a human, and not some sort of
computer program that was written to submit the form
millions and millions of times.
The reason it works is because humans--
at least non-visually impaired humans--
have no trouble reading distorted characters, whereas
computer programs simply can't do it as well yet.
More generally, a CAPTCHA is just a program that can tell
whether its user is a human or a computer.
OK, let me say that another way.
A CAPTCHA is a program that can generate and grade tests
that most humans can pass, but current computer
programs can not.
Notice the paradox here.
A CAPTCHA is a program that can generate and grade tests
that it itself cannot pass.
So in that way, CAPTCHAs are a lot like some professors.
[LAUGHTER]
Just to make things crystal clear, let me give you an
example of one of these programs that can generate and
grade tests that most humans can pass, but current computer
programs cannot.
Here's how the program works.
First, the program picks a random string of letters.
O-A-M-G, in this case.
Then the program renders the string into a randomly
distorted image, and then the program generates a test,
which consists of the randomly distorted image and the
question, "What are the characters in this image?"
CAPTCHAs are used all over the place, for all kinds of
things, and I could spend the next hour talking about all
the different applications of CAPTCHAs.
But since I don't want to do that, I want to illustrate one
of the applications through a little story.
So a few years ago, Slashdot--
which is a very popular website--
put up this poll in their site, asking which is the best
computer science graduate school in the United States?
This is a very dangerous question to ask over the web.
As with most online polls, IP addresses of voters were
recorded to make sure that each person could only vote,
at most, once.
However, as soon as the poll went up, students at CMU wrote
a program that voted for CMU thousands and
thousands of times.
The next day, students at MIT wrote their own program.
And a few days later, the poll had to be taken down with CMU
and MIT having, like, a gazillion votes and every
other school having less than 1,000.
I guess the poll worked in this case.
[LAUGHTER]
I'm just kidding.
But in general, this is a huge problem.
You simply cannot trust the results of an online poll,
because anybody could just write a program to vote for
their favorite option thousands and
thousands of times.
One solution is to use a CAPTCHA to make sure that only
humans can vote.
CAPTCHAs have many, many other applications.
Another one is in free email services.
For instance, there are several companies that offer
free email services--
Yahoo, Microsoft, Google--
and up until a few years ago, all of them were suffering
from a very specific type of attack.
It was people who wrote programs to obtain millions of
email accounts every day, and the people who wrote these
programs were usually spammers.
So if you're a spammer and you want to send spam from, say,
Yahoo, you run into the problem that each Yahoo
account only allows you to sound, like,
100 messages a day.
So if you want to send millions of messages a day
from Yahoo accounts, you have to own
millions of Yahoo accounts.
And this is why spammers wrote programs to obtain millions of
Yahoo accounts.
And the solution--
or one solution-- and this is what we originally suggested
to Yahoo-- was to use a CAPTCHA to make sure that only
humans can obtain free email accounts.
Now, since CAPTCHAs are used all over the place to stop
spammers from doing bad things, spammers have started
coming up with all kinds of dirty hacks to get around the
CAPTCHAs that are being used in practice.
So let me explain a couple of them.
Here's one.
I'm sure a lot of you have heard of this.
CAPTCHA sweatshops.
Spam companies actually are hiring people to solve
CAPTCHAs all day long.
And they are usually being hired in other countries where
the minimum wage is a lot lower, and this
is currently happening.
But there's at least two consolations.
First, it's at least costing them some.
So whereas before, they could get the accounts for free, now
it costs them a fraction of a cent per account, so they
can't get that many.
Second, CAPTCHAs are actually generating jobs in
underdeveloped countries.
[LAUGHTER]
So this is one dirty hack.
There's an even dirtier hack, and I'm sure a lot of you have
heard of it, and this is what some porn companies are
allegedly doing.
And I'm going to emphasize the word "allegedly." So, porn
companies also want to send spam.
They also want to break CAPTCHAs, and here's how they
are allegedly doing it.
They write a program the fills out the entire registration
form, say, at Yahoo.
And whenever the program gets to the CAPTCHA,
it can't solve it.
So what it does is it copies the CAPTCHA
back to the porn page.
Now, back at the porn page, there's a lot of people
looking at porn.
And suddenly, one of them gets this screen saying, "If you
want to see the next picture, you got to tell me what word
is in the box below." And you know what people do?
They type the word as fast as possible.
[LAUGHTER]
And by doing so, they are effectively solving the
CAPTCHA for the porn company bot.
That is, they're effectively obtaining a free
email account for them.
So pornographers, they're really, really smart.
So CAPTCHAs take advantage of human processing power in
order to differentiate humans from computers, and it turns
out that being able to do so has some very, very nice
applications in practice.
Now that I've told you about CAPTCHAs, now I can tell you
what this talk really is about.
This talk is not about CAPTCHAs.
This talk is about human computation.
Sort of the flipside of CAPTCHAs.
The idea is there's a lot of things that humans can easily
do that computers cannot yet do.
I want to show you how we can solve some of these problems
by just making good use of human processing power.
And I think the best way to introduce the rest of the talk
is with a little statistic, and the statistic is that over
9 billion human hours of Solitaire were played in 2003.
9 billion.
Now, some people talk about wasted computer cycles.
What about wasted human cycles?
Just to give you an idea of how large this number really
is, let me give you two other numbers.
First is the number of human hours that it took to build
the Empire State Building.
Turns out it took 7 million human hours to build the
entire Empire State Building.
That's equivalent to about 6.8 hours of people playing
Solitaire around the world.
Now, in case you don't think the Empire State Building is a
monumental enough task, let me give you another number.
The Panama Canal.
It turns out it took 20 million human hours to build
the entire Panama Canal, and that's equivalent to a little
less than a day of people play Solitaire around the world.
I want to show how we can make good use of these
wasted human cycles.
And that is what I mean by human computation.
In this talk, we're going to consider the human brain as an
extremely advanced processing unit that can solve problems
that computers cannot yet solve.
Even more, we're going to consider all of humanity as an
extremely advanced and large scale distributed processing
unit that can solve large scale problems that computers
cannot yet solve.
I claim that the current relationship between humans
and computers is extremely parasitic.
We're parasites of computers.
What I want to advocate for in this talk is more of a
symbiotic relationship, a symbiosis.
One in which humans solve some problems, computers solve some
other problems, and together we work to
create a better world.
[LAUGHTER]
OK, I'm getting freaky.
But more seriously, I want to talk about some problems that
computers cannot yet solve, and I want to show you how we
can easily solve a lot of these problems by just making
good use of human processing power.
The first problem that I'm going to talk about is that of
labeling images with words.
So the problem is as follows.
When inputting an arbitrary image, we want to output a set
of key words that properly and correctly describe this image.
[LAUGHTER]
As you should all probably know, this is still a
completely open problem in computer vision and artificial
intelligence, in the sense that computer programs simply
can't do this.
However, a method that could accurately label images with
words would have several applications, one of which
you've probably already seen, and that is image
search on the web.
So Google, for instance, has Google Images.
You can go there, type a word like "dog," and get back a lot
of images related to the word "dog." Now, it is the case
that there's no computer program out there that can
tell you whether an arbitrary image from the web contains a
dog or not, so the way Google Images works-- and image
search on the web works, roughly--
is by using file names in html text.
So if you search for "dog," you get back a lot of images
named dog.jpg or dog.gif, or that have the word
"dog" very near them.
Of course, the problem with this method is that it doesn't
always work very well.
For instance, this is not any more, but it used to be the
first page of results for the query "dog" on Google Images.
There is an image of a rabbit, there.
There's a guy in a blue suit.
What the hell?
But if we have methods such that for every image on the
web could give us accurate textual descriptions of those
images, we could potentially improve the accuracy of image
search on the web.
Such a method would have many other applications.
Another one is inaccessibility.
So it turns out that the majority of the web is not
fully accessible to visually impaired individuals, and one
of the biggest reasons is images.
So blind people actually surf the web.
The way they do it is they use screen readers, programs that
read the entire screen to them out loud.
But whenever a screen reader reaches an image, it can't do
anything other than read the caption of that image.
Of course, the majority of images on the web don't have
proper captions associated to them.
So again, if we had a method such that for every image on
the web could give us accurate, textual descriptions
of those images, we could improve the
accessibility of the web.
Such a method would have many other applications, and so
what we want-- and what I'm going to tell you right now--
is a method that can label all images on the web.
Not only that, it's a method that can label all images on
the web in a way that's fast and cheap.
How are we going to do it?
Well, we're going to use humans, but we're going to use
them cleverly.
So normally, if you ask people to label images for you, you'd
have to pay them to do so.
And if you wanted to label all images on the web by paying
people, you'd have to pay a lot of money.
And even if you had a lot of money, if you wanted to label
all images on the web by paying people fast, you'd have
to find a lot of people who were willing to label images
for living.
Good luck with that.
My approach is much better.
Rather than paying people to label images for me, I get
them to want to label the images for free.
And in fact, they want to label the images so much that
in some cases they're even willing to pay me to label the
images for me.
How do I do that?
Well, I have an extremely, extremely enjoyable,
multiplayer online game called the ESP game that people
really, really like to play, and as people play, sort of as
a side effect, they actually label images for me.
Now, the ESP game has two very nice properties.
First, as people play the game, the labels that they
generate for images are accurate even if the players
don't want them to be so.
Second, as people play the game, they actually label
images very, very fast. And in fact, using a conservative
estimate, I'm going to show you later in the talk that if
the ESP game is put on a popular gaming site, we could
actually label all images on Google Image Search
in just a few weeks.
So how does the game work?
Well, first and foremost, the ESP game is a two player
online game.
So there's a web site.
You can go there to try to play the website.
Whenever you go to the website, you get randomly
paired with somebody else wanting to play the game.
That's your partner.
Now, you're not allowed to communicate with them, and
you're not told who they are.
It's just a complete stranger from the web.
And the goal of the game is for both you and your partner
to type the exact same word, given that the only thing you
two have in common is an image.
So you can both see the same image.
You know you can both see the same image, and now you're
told to type whatever the other guy's typing.
Turns out that what people do, the best strategy, is just to
type a lot of words related to the common image.
So basically, both players are going to be typing a lot of
words related to the common image until one of player
one's words is equal to one of player two's words.
They agree, they get points, and then they get happy.
That's the basic idea of the game.
Now, this word that the two players agree on is usually a
very, very good label for the image, because it comes from
two independent sources.
Let me give you a better idea of the basic move of the game.
Imagine you have two players, player one and player two.
And they're both paired, so they can both
see the same image.
And now they're told, "Type whatever the other guy's
typing." Notice, the players are not told, "Label the
image," or even what labeling an image might mean.
They're just told, type whatever
the other guy's typing.
So say at first, player one types "car," player two types
"boy." It's not the same word, so the game still goes on.
Say then, player one types "hat" and then "kid." Still
none of player one's words is equal to one of player two's
words, so the game's still going on.
By the way, player one cannot see any of player two's
guesses, and vice versa.
So they're just typing words completely independently,
until, say, player two types a word that player one had
already entered.
They agree and then they get a lot of points.
This is the basic move of the game.
The actual game looks a little more like this.
Basically, both players have a certain amount of time to
agree on as many images as they can.
So in 2 and 1/2 minutes, they have to agree on as many
images as they can.
That's basically the game.
Each time they agree on an image, they get a certain
number of points.
There's also a thermometer at the bottom that measures how
many images the two players have agreed on, and if you
fill the thermometer, you get like a gazillion points.
There's also a pass button, so players can agree to pass on
difficult images.
And another really important component of the game is this
thing we call "taboo words." If you've ever played the game
Taboo, you should be able to guess what these are.
Taboo words are words that are related to the image the
players cannot use when trying to agree on that image.
So in this case, for instance, you can't use "hat" or
"sunglasses," or any plural or singular of these words.
Now, where do taboo words come from?
They come from the game itself.
The taboo words are words that two other players have already
agreed on for this particular image.
So the nice thing about taboos words is that they guarantee
that each time an image passes through the game, it gets a
brand new, different label.
The other nice thing about taboo words is they make the
game more difficult, and therefore more fun.
I'm not talking about fun.
Is this game fun?
Well, amazingly, it really is a lot of fun.
So far, we've gotten over 15 million agreements-- that's
over 15 million labels-- with about 75,000 players.
Let me say that another way.
75,000 players have given us over 15 million agreements.
That means that on average, each player is playing a lot.
We have many people that play over 20 hours a week.
That's like a full time job.
We've had playing streaks that are longer
than 15 hours straight.
[LAUGHTER]
I feel a little bad about this.
So by now, the game has a mechanism that if you've been
playing for longer than 15 hours, it will cut you off.
And as a promise to my department head, it's 10 hours
if you're from a .edu domain.
[LAUGHTER]
So, so far, over 15 million agreements.
What if you wanted to label the entire web?
Well, 5,000 people playing the game simultaneously could
label all images on Google Images in about two months.
The striking thing here is that 5,000 is not
a very large number.
In fact, individual games in popular gaming sites, such as
Yahoo, Polo.com, or MSN average over 5,000
players at a time.
So if you put the ESP game on a popular gaming site, you
could potentially label a lot of the images on the web in
just a few months.
A few more things about the game.
There's also a single player version of the game.
It's important to have a single player version of the
game for several reasons.
For one of them is the number of people playing the game is
not always even.
But also, whenever a player drops, it's important to just
basically have them keep on playing the single version
player of the game.
And how do you get a single player game?
Well, you simply can pair up a single person with a
prerecorded set of moves.
The idea is as follows.
Whenever you have two people playing, you record everything
that they do and when they do it.
So you record all the words they enter, along with timing
information.
And whenever we want to have a single player play, we simply
pair them up with a prerecorded set of moves.
So that single player is playing with somebody else,
just not at the same time.
One nice thing about this, notice this actually doesn't
stop the labeling process.
That single player is playing with somebody else, just not
the same time, so everything that I've said about labeling
remains true.
In fact, we can even go one step further.
We can do the zero player game.
We can also pair up prerecorded games with each
other to get more labels, and if you count all the extra
labels that the ESP game has collected so far, you get that
so far the ESP game has collected over 39 million
labels for images on the web, if you count all these.
Now, one thing that some of you may be wondering about is
what about cheating?
So for instance, could you try to cheat to
screw up the labels?
Something like, my office maid and I could try to log in to
the game at exactly the same time.
Maybe we'll get paired with each other, and if we get
paired with each other, we can agree on any word we
want for any image.
Or even worse, somebody could go to Slash.dot and type,
"Hey, everybody, let's all play the ESP game, and let's
all agree on the word 'A' for every image." Could happen.
Fortunately I've thought about this, and the ESP game has
several mechanisms that fully prevent cheating.
Let me tell you a few of the things that we
do to prevent cheating.
Here's one.
At random, we actually give players test images.
These are just images that are just there to test whether the
players are playing honestly or not.
And what they are is they are images for which we know all
the most common things that people enter for them.
And we only store a player's guesses and the words they
agree on if they successfully label the test images.
So if you think about it, in a way, this sort of gives a
probabilistic guarantee that a given label is not corrupt.
What's the probability that a given label is corrupt, given
that the players successfully label all
of their test images?
And this probability can be boosted by using the next
strategy, which is repetition.
So we only store a label after n pairs of players have agreed
on it, where it is a parameter that can be tweaked.
So every now and then, we actually delete all the taboo
lists for the images, and we put the image back into the
game afresh.
And we only store a label after n pairs of players have
agreed on it.
So if we let x be the probability of a label being
corrupt given that players successfully labeled all of
their test images, than after n repetitions, the probability
of corruption is x to the n.
This is assuming that the n repetitions are independent of
each other, but if x is very small, x to the n is really,
really small.
I'm going to say so far, we've collected lots and lots of
labels, and we have not seen cheating be able to screw up
our labels.
In fact, the quality of the labels that the ESP game has
collected so far is very high.
Let me now show you some search results.
Let me show you what happens when we search for the word
"dog," for instance.
Here's some dogs.
More dogs.
More dogs.
More dogs.
And I could go on forever.
Here's what happens when you sit for "Brittney Spears." You
got to show this whenever you show search results.
Here's what happens when you search for "Google." I
prepared this for this talk.
You get the founders.
And one really nice thing about this is that this slide
constitutes a proof that the word "Google" and the word
"search" really are synonyms. On input that,
people agreed on Google.
OK.
So let me now show you some sample labels.
So what I'm going to show you right now are some images,
along with the labels at the ESP game has collected for
them so far.
So here's an image, and here are the labels that the ESP
game has collected for it so far.
By the way, these could be ordered in terms of frequency.
They're not.
This is just the list of all the words that the ESP game
has collected for this image so far.
You should notice two things about this list of words.
First of all, it's extremely accurate, meaning all of these
words actually make sense with respect to the image.
Second, it's extremely complete, meaning almost
anything that you can imagine to describe something in this
image is in this list. Not everything, but
a lot of the things.
And in fact, this is true in general of the word lists
generated by the ESP game.
They are as accurate and as complete as those generated by
participants who are just paid to label images.
Let me show you more sample labels.
Here's another image.
Anybody know who this is?
Walter Matthau.
He's an actor.
And just to prep you for one of the labels, Walter Matthau
was in the movie Dennis The Menace, and he played the
character of Mr. Wilson.
Some of the labels that the ESP game has collected for
this image so far are--
[LAUGHTER]
So that first one seems a little wrong, but actually, if
you look carefully, you realize it's
really not that bad.
I like to tell people the ESP game has uncovered a major
conspiracy.
Now that we're on this topic, here's another image.
By the way, I have no political affiliations
whatsoever.
I'm not a US citizen, and what I'm about to show you are
simply the scientific results of what happens when you put
this image on the ESP game.
So some of the labels that the ESP game has collected for
this image so far are--
[LAUGHTER]
That last one, can you imagine how awesome the two players
must have felt when they agreed on that last one?
It must have felt great.
And in fact, this brings us to one of the reasons why people
really like the ESP game.
It's because they can feel a special connection with their
partner, especially when they agree on an off the wall word
like "yuck" for an image of President Bush.
In fact, it gets even better.
A lot of the emails we get actually suggest that players
feel a very, very, very special
connection with their partner.
Players like playing with partners of the
opposite sex better.
They want to know whether their partner's of the
opposite sex, and a lot of the emails say things like, "My
partner and I, we look at the world in exactly the same way.
Can you tell me their email address?" This is great
because, because I'm going to be rich soon.
More seriously, this brings us to the question of why do
people like the ESP game?
I mean, it's true that the game was designed to be
enjoyable, but what are the reasons that people like the
ESP game so much?
And to address that question, let me show you some of the
most common things that people have said of why
they like the game.
Here's what one person said.
By the way, I'm just going to let you read.
So this is the sense of connection with your partner.
Here are some of the other most common
things that people said.
[LAUGHTER]
That last one, if you think about it, it
makes perfect sense.
Although that was not expected, it
makes perfect sense.
The ESP game helps people learn English, because you've
got an image, you've got to say what it is in English.
And that brings up the question, could you have the
ESP game in multiple languages?
The answer is sure, but I don't want to talk about that.
So that's some of the most common things that people have
told us of why they like the game.
In addition, let me show you some of the things that people
have said about the game in blogs.
It was, at some point, in literally hundreds of blogs.
Here's a couple of them.
Here's what one guy said.
Sense of achievement.
But the best is the way this guy ends.
[LAUGHTER]
Here's another one.
So this guy actually likes the concept of the game, but
again, the best is the way this guy ends.
So not everybody likes their partner, and it completely
depends on whether you do well with them or not.
If you do well with them, you fall in love with them.
If you do badly, you think they're an idiot.
Of course, you're not the idiot.
They're an idiot.
Even though the game is symmetric.
But in addition to all those things that people have told
us, we continually do measurement to try to figure
out what are the things that make people play longer.
So let me explain one of these measurements to you.
At some point in the history of the game, I added this very
small message in the corner of the screen alerting you
whether your partner have already
entered a guess or not.
It's a very tiny message.
This is just a magnification of it.
It just tells you when your partner has already entered a
guess or not.
When this was added to the game, it wasn't added to all
the players.
It was just added to a small, random subset of the players.
And then we measured whether the players who had this
feature played longer than those who didn't.
And it turns out that those who had this feature played a
whopping 4% longer than those who didn't.
Now, you might not think that 4% is very large, but
actually, it's a statistically significant difference, and if
you think about it, just a very tiny message in the
corner of the screen makes people play 4% longer.
Now, in a way, the ESP game is kind of like an algorithm.
Much like an algorithm, it has an imput/output behavior.
Its input is an image.
It's output is a set of key words that properly
describe the image.
Much like an algorithm, you can analyze its efficiency.
You can prove that its output is not corrupt with high
probability, et cetera.
So what I want to do now is I want to refer to all games--
like the ESP game, that are kind of like algorithms--
I want to refer to them as games with a purpose.
And the idea that I want you to have in your mind is that
games with a purpose is like running a computation in
people's brains instead of silicon processors.
And what do I do now is I want to give you other examples of
games of a purpose.
So the next problem that I'm going to talk about is that of
locating objects in images.
On input an arbitrary image, the ESP game tells us what
objects are in the image, but it does not tell us where in
the image each object is located.
So what we would like to know is, we would like to know,
yes, there's a man in the image, but the
man is right there.
There's a plant in the image, but the plant is right there.
And not only that.
We would like to know precisely which pixels belong
to the man, which pixels belong to plant, et cetera.
And we would like to have this information for a large
percentage of the images on the web.
If we could have this information, we could do a lot
of really cool things.
For instance, we could have an image search engine were the
results are highlighted.
It tells you this is where the man is in
each one of your images.
That would be pretty cool.
But even better, if we have this information for a lot of
images, we could use this for training computer vision
algorithms. So computer vision has advanced significantly
over the last 20 or 40 years, but so far, it hasn't been
able to create a program that can, with high probability,
figure out where in the image each object is located.
And one of the major stumbling blocks is the
lack of training data.
But if we had this data for a lot of images on the web, we
could use it to train better computer vision algorithms.
So this is what the next game is going to do, and the next
game is called Peekaboom.
And here's how it works.
It's a two player game.
Much like the ESP game, both players don't know anything
about each other, and they can't
communicate with each other.
At the beginning of every round-- oh, by the way, the
player on the left, we're going to call him "Peek." The
player on the right, we're going to call him "Boom" So
Peek and Boom.
At the beginning of every around, Boom gets an image
along with a word.
In this case, it's the image of a butterfly and the word is
"butterfly." That image word pair comes directly
from the ESP game.
Peek, at the beginning of every round, gets nothing.
Just a completely blank screen.
And the goal of the game is for Boom to get Peek to guess
the word "butterfly." And the only thing that Boom can do to
help Peek guess the word "butterfly" is he can take his
mouse, put it somewhere in the image, and click.
And whenever Boom clicks, a circular area around that
click is revealed to Peek.
The actual circular area is a lot smaller
than the one I revealed.
I just didn't want to go through all the clicks.
But basically, when Boom clicks, a circular area around
the click is revealed to Peek.
And then Peek, given only the circular areas, has to guess
what word Boom is trying to make them guess.
Whenever Peek guesses the correct word, both players get
a lot of points, and then they switch roles.
Peek becomes Boom, and Boom become Peek.
Now notice, in this case the word was "butterfly," so Boom
clicked on the butterfly.
But had the word been "flower," Boom would have
clicked on the flower.
So by just watching where Boom clicks, we get information
about where each object is located in every image.
By the way, I'm brushing over a lot of details.
For instance, there's also hints.
So Boom can give hints to Peek about whether the word is a
noun, is it a verb, is it text in the image, et cetera.
Now, just to make things more clear, let's play a couple
rounds of Peekaboom.
So you guys have to guess what I'm trying to make you guess.
Here we go.
So now?
"Bush," awesome.
OK, you got it.
Bush.
Here's another one.
It's a verb.
"Pick." OK, very nice.
I love this image.
So, imagine we were back here, and I gave
you a different hint.
I told you it was a noun, and not only that, I started
pointing there.
What would you say this?
Hair, exactly.
So this is another mechanism of Peekaboom, and it's
something called pings.
So not only can Boom reveal part of the image.
After something has been revealed, he can also point to
somewhere, saying it's this, it's this.
This gives us extra information about where each
object is located in the image.
So this is the basic idea of Peekaboom.
This is what the Peekaboom screen looks like for one of
the players.
This is for the Boom player.
And now the first question is, is this game fun?
Well, it turns out it really is a lot of fun.
By the way, the statistics I'm going to show you right now
are a little outdated.
This is just for the first four months of gameplay.
So in the first four months of gameplay, 27,000 players gave
us 2.1 million pieces of data.
By a piece of data, I mean an image along with a word
correctly analyzed by a pair of players.
In the first 10 days after release, actually many people
played over 120 hours.
That's an average of over 12 hours a day.
So it's a lot of fun.
Here's the top scores list of Peekaboom, just to put things
in perspective.
This is for the first four months of gameplay.
Each time you play Peekaboom, you get on average 800 points.
So the top player there has 3.3 million points.
Even the lowest player in this list, in the first four
months, have played at least 270 hours of gameplay.
So people really love this game.
Now what about the data that it produces?
Is it any good, or how do we get
good data out of Peekaboom?
So let me explain how we get good data out of Peekaboom.
This is an image of Ronald.
The word is "Ronald." By the way, I love this image.
And the last three images were collected by searching for the
word "funny" using the ESP game.
So we get an image of Ronald, a word "Ronald," here's what
we do to get good data out of this.
We give the same image word pair to a
bunch of pairs of players.
And from each pair of players, we get a region of the image
that is related to the word.
Now we take all of these regions and intelligently
combine them, and get a really good idea of where the object
is located in the image.
And on top of that, we can add sort of where the pings are to
get more information about what the most
salient parts are.
And we can go even one step further.
We can take this information and combine it with image
segmentation algorithms to get pretty much the precise
outline of where the object is in the image.
Now I'll, say, this doesn't work for all
objects in the images.
It works like that perfectly for about 50% of the objects
in the images that we have data for.
For the rest, it works mostly, but it can miss
like a foot or something.
But even without using segmentation, we could just
use the Peekaboom data in a really, really boneheaded way
to come up with a search engine in which the results
are sort of highlighted.
And we've done this.
We have a search engine where you can search for "man,"
"dog." And for each image, it tells you here's the man,
here's the dog, here's the man, here's the dog.
And more man, more dog.
OK.
Forget about Peekaboom.
Brand new gain, Verbosity.
So this next game that I'm going to talk about, by the
way, has not yet been released, so I'm not going to
be able to show you any statistics.
But I'm just going to quickly explain what the idea is.
So what does Verbosity do?
The idea is it collects common-sense facts.
So what's a common-sense fact?
Here's an example of a common-sense fact.
Water quenches thirst. It's a true fact
that everybody knows.
Here's another common-sense fact.
Cars usually have four wheels.
Now, the thing about common-sense facts is that it
is estimated that each one of us has literally hundreds of
millions of them in our head.
And these are what allow us to act normal and navigate our
world successfully.
The other thing about common-sense facts is that
computers don't yet have them.
But if we could somehow put common-sense facts into
computers, we could potentially make them more
intelligent.
And I'm not even talking about making computers as
intelligent as humans.
Just a little more intelligent.
Like for instance, transforming our search query
into something better, that works better, or
something like that.
So if we could somehow collect a lot of common-sense facts
and put them into a computer, we could potentially use this
to make computers more intelligent.
And in fact, there's been a lot of projects that have
tried to do this, including one at MIT, and so far they
haven't been able to collect enough common-sense facts in
order to really make a difference, because the
process of entering common-sense facts into a
computer is extremely tedious.
So we're going to turn this into a game.
So for the next game that I'm going to talk about, the
input-output behavior of this game is as follows.
On input a word, this game is going to output a set of
common-sense facts about that word.
By the way, I'm oversimplifying here.
These common-sense facts are not just going to be
common-sense facts in English.
They're going to have some structure to them.
So there's going to be logical operators
inside them, et cetera.
So this is the input-output behavior.
On input a word, it's going to give common-sense
facts about that word.
And the way the game is going to work--
game called Verbosity--
and the way it's going to work is as follows.
It's a two-player word guessing game.
There's two players, a Narrator and a Guesser.
Same idea as the ESP game.
Basically, both players can't communicate with each other.
They don't know anything about each other.
At the beginning of every round, the Narrator gets a
word and has to get the Guesser to guess that word.
And what the Narrator can do to get to Guesser to guess
that word is he can pick one among many sentence templates
that they have. Which sentence templates are available to
them at the time vary depending on the word.
So he can pick one among many sentence templates, and fill
it with an appropriate word.
What's an appropriate word is a word that's not "milk," and
it's also a word that fits in grammatically with the
sentence template.
Whenever the sentence template is filled in,
it's sent to the Guesser.
Then the Narrator can pick another sentence template,
fill it with an appropriate keyword, and
send it to the Guesser.
And the Guesser, given enough hints about it, eventually has
to guess what word it is, and whenever the Guesser guesses
the correct word, both players get points.
The way we get common-sense facts out of this game is by
just watching what the Narrator says for each word.
By the way, I'm brushing over a lot of
details for this game.
This is just the basic idea, so high level idea is it's a
two-player game.
Player one and player two.
At the beginning of every round, player one gets a word,
and because of the rules of the game, has to give some
common-sense facts about the word.
Then those common-sense facts are sent to player two, and
player two, given only the common-sense facts, has to
guess what word player one got as input.
And if player two can guess the correct word, both players
get points.
This is the core mechanism of Verbosity.
Now, I want you to notice two things
about this core mechanism.
First, it's fun.
This is very similar to the core mechanism of a lot of
popular party games.
Basically, just word guessing games.
Second, this core mechanism actually gives output that is
already, in a way, verified.
Notice, we're getting all the common-sense
facts from player one.
But what's player two doing?
In a way, player two is verifying the output.
Because if player two can guess the word given only the
common-sense facts, then those common-sense facts must have
something to do with the word.
So in a way, it's giving output
that is already verified.
And the same core mechanism is exactly the same core
mechanism that was used in Peekaboom.
So in the case of Peekaboom, it's a two-player game.
Player one and player two.
At the beginning of every round, player one gets an
image along with a word, then has to give a region of the
image that is related to the word.
Then that region is sent to player two, and player two,
given only the region, has to guess what word
player one got as input.
The same mechanism as Verbosity.
And again, it's fun, and also gives output that is, in a
way, verified.
We're going to call all games that satisfy this mechanism,
we're going to call them asymmetric verification games.
So this is a general mechanism for building
games with a purpose.
So in general, for an arbitrary input-output
behavior, we could define a game as follows.
It's a two-player game.
We give the input to player one and have
them give an output.
Then we send the output to player two, and given only the
output, player two has to guess what
input player one got.
If player two can guess the correct input, both players
get points.
This mechanism has two very nice properties, that for a
lot of input-output behaviors, it's fun, and also, it gives
output that is, in a way, verified.
Of course, this doesn't work for all input-output
behaviors, but it works for a large class of them.
And these are asymmetric verification games, and it's
asymmetric because both players are doing something
slightly different than each other.
And it's also asymmetric, as supposed to symmetric
verification games, where you've already seen an example
of a symmetric verification game, and that's the ESP game.
So this is another general mechanism for creating games
with a purpose.
So for an arbitrary input-output behavior, you can
give both players the same input, and ask them to guess
what output the other player is going to give.
So if they both give the same output, they get points.
Again, this mechanism is fun for a lot of input-output
behaviors, and also has the property that the output it
gives is, in a way, verified because it comes from two
independent sources.
And now, we can start looking at the differences between
symmetric and asymmetric verification games.
So for instance, symmetric verification games, I claim,
put a constraint on the number of inputs per output.
The number of outputs per input, sorry.
If a given input has too many outputs, than a symmetric
verification game is never going to work, because both
players are never going to agree on the same output.
Asymmetric verification games put a constraint on the number
of inputs that yield the same output.
If there's too many inputs that yield the same output,
then given only the output, you'll never be able to guess
what input it came from.
I'm going to finish, now.
Hopefully, I've been able to convince you that there's a
lot of power into looking for clever ways of
utilizing human cycles.
In fact, if you think about it, this talk hints at a
paradigm for dealing with open problems in artificial
intelligence.
If you have something that you really can't solve in
artificial intelligence, then maybe you can turn the problem
into a test that distinguishes humans from computers.
Turns out that being able to do so has some very nice
applications in practice.
Or alternatively, maybe you can turn the problem into a
game, in which case you don't even need to solve your
problem anymore.
People will solve it for you.
One nice thing about this whole research agenda is that
it provides a much better motivation for
the movie The Matrix.
If you think about it, the motivation for the movie was
that in the future, computers become a lot more intelligent
than humans.
But rather than killing us, they actually have to keep us
around, because we generate power.
That makes no sense.
A much better motivation would be in the future, computers
become a lot more intelligent than humans.
But rather than killing us, they actually have to keep us
around, because there's a couple of problems that we can
solve that they cannot yet solve.
My ultimate research goal is to transform our human
existence to just eating, sleeping, drinking, playing--
never mind.
[LAUGHTER]
Www.captcha.net.
Www.espgame.org.
Peekaboom.org, and that's it.
Thank you.
[APPLAUSE]
Yes?
AUDIENCE: Does it concern you at all that the fact that
you're using a game will automatically give you a very
biased population of people that are giving us to problems
we want answers to?
And this population of people are the people that have way
more time on their hands, and are not motivated to maybe get
a job or do something [UNINTELLIGIBLE]?
[LAUGHTER]
LUIS VON AHN: Very good question.
It's true that the population is biased.
There's no question about that.
But for a lot of really simple things, I mean,
anybody can do it.
But it's true that the population is biased.
That's definitely true.
AUDIENCE: Have you seen any results?
LUIS VON AHN: I can tell you that the population is biased,
but I have not seen anything that really can tell me
because we're using gamers, it's like, this is happening
instead of the general population.
I have not seen that.
Yes?
AUDIENCE: I have a concern with asymmetric games where
the input is very similar to the [UNINTELLIGIBLE].
For example, when you said milk is close to cereal.
It's like a fraud question.
What if someone types in milk and I come up with pail--
P-A-I-L. I think it would be very obvious for his partner
to guess which question to ask.
LUIS VON AHN: Sure.
I didn't mention a lot of the mechanisms that we use to stop
that sort of cheating, but there's a lot of mechanisms.
For instance, we don't let them type anything that's not
a dictionary word.
Second, that word has to fit in with the template,
grammatically.
But still, I mean, there's a lot of mechanisms that try to
prevent that.
But you're right, that's a concern.
AUDIENCE: So I think the popularity of a lot of games
and these games in particular are [UNINTELLIGIBLE].
They're novel and different.
This is a new thing, let's try it out.
We might spend 100, 120 hours on this.
There was a site I remembered called Am I Hot or Not?
a few years ago.
Maybe I'm confessing something I shouldn't confess.
But you spend a few hours.
And 5 years from now the game won't exist. The question is,
if you view this as a strategic shift in how we use
human cycles, you're kind of hindered by the fact that this
will probably die out within a few months.
LUIS VON AHN: The answer to that is yes and no.
So, there are games whose popularity lasts for
thousands of years.
And there's a lot of these gaming sites that have games
that the popularity only last like six months or a year.
And what they do is very simple.
They have the same game concept and just redress it
with another name and something else, and all the
people come back.
This is also well known to nightclub designers.
Just change the name.
But it is true that popularity does die, but that is
completely game-dependent.
Some games, the popularity lasts longer than others.
So the ESP game has been running for well over two
years now, and the popularity has not died.
I mean, there was definitely an initial surge,
but it has not died.
So the amount of time that it works varies, and hopefully we
can find games that last for thousands of years.
Yes?
AUDIENCE: Towards the beginning of the talk, you
talked about accessibility and how vision impaired people
[UNINTELLIGIBLE] screen reader.
But I don't really see how these games
close the loop on that.
LUIS VON AHN: That's a very good question.
The way I explained, the ESP game only gives you keywords.
That's not quite enough for accessibility.
It's better than nothing, but it's not quite enough.
AUDIENCE: But you're not putting them
back into the websites.
LUIS VON AHN: Very good point.
I see what you're talking about.
But that's an engineering problem.
You could actually do that just with a server that they
connect to.
FEMALE SPEAKER: Plug an extension to the browser.
Something like that.
LUIS VON AHN: Sure.
FEMALE SPEAKER: Some people are better than others in any
game, so can you take people from the opposite end of the
spectrum, and there will be people who try
to disect your brain.
And the really bad people, you try to see how they improve.
LUIS VON AHN: I don't understand.
Say that again?
AUDIENCE: There's a spectrum of ability in any game.
So we can look at either end of a spectrum, and find the
really good people, and study what algorithm they use.
The really bad people, when they improve, see if
[UNINTELLIGIBLE] to your algorithm.
LUIS VON AHN: Right.
You can do that.
Yes, I agree.
Yes?
AUDIENCE: So you said you've running ruinning this game for
two years now, which means you must have an
obscene amount of data.
LUIS VON AHN: Yes and no.
I do have an obscene amount of data, but I recycle the
images, because I just don't want to have that many images.
So there are 39 million labels, but it's
not that many images.
AUDIENCE: You said the facts you get out of Verbosity are
not simply English sentences, but have some more logical
structure to them.
I wonder if you could say a few more words about that.
LUIS VON AHN: The reason is because of the templates.
We have templates.
We don't just let people free flow write English.
We have templates.
So out of those templates, we know things like, well, this
is for purpose.
So things like that.
AUDIENCE: Let's say I have a really boring job, like I'm
looking for defects in a manufacturing process.
How do I turn that into a fun game?
[LAUGHTER]
LUIS VON AHN: That's a very good question.
That would be really cool if we could figure out how to do
that for everything.
I don't know how to do that for everything.
I don't even know if it's possible to do it for
everything.
But that'd be really cool if we could figure out how to do
it for everything.
I don't know if it's possible to do it for everything.
AUDIENCE: From an ethical point of view, is there any
problem that people would probably be spending their
work hours playing the game, rather than
their free time hours?
And so you're not really gaining any productivity in
society as a whole.
LUIS VON AHN: Well, depending.
I mean, you're right about that.
But imagine we could turn everybody's work into
something fun.
That'd be really cool.
So, depending.
But one thing I should say about ethical is all these
games, they don't try to trick you into doing anything.
I mean, everybody knows what the purpose is.
AUDIENCE: Maybe "ethical" is the wrong word.
LUIS VON AHN: Right.
Yes?
AUDIENCE: A lot of these games are very good at getting basic
facts out of people.
Have you thought about how to get stuff that's a little bit
more nuanced?
Like if you leave milk in the fridge for three weeks, it's
going to go bad?
LUIS VON AHN: That's a very good question.
I mean, it depends a lot on the particular domain.
I don't know how to do it in general, but for instance, for
images, I can tell you.
The ESP game for images, most of the stuff that you get out
of the general ESP game is very general stuff.
I mean, the first word is going to be like "dog." Then
once "dog" becomes a taboo word, it's probably going to
be the breed of the dog, or something.
But very generally, usually things that everybody knows.
If you want to start getting things that only a few people
know, then you can do a few things.
So for instance, you can have people tell you what they want
to see images of.
So for instance, I like cars.
Can I see images of cars?
Then I'll be an expert on that sort of thing, and then you
can do that better.
Or you can use collaborative filtering
to try to give people--
you figure out what they're good at, and you give them
more images like that.
And so you can start getting better things like that.
But yeah, that's a very good question.
Yes?
AUDIENCE: It seems like you could also use these games to
solve problems that computers are already good at solving,
like you could have people add up numbers,
of things like that.
But those games likely would not be very fun.
LUIS VON AHN: They might be.
So like Sudoku.
AUDIENCE: I guess my question is-- right, but that's like a
constraint propagation problem that is a little
bit harder to solve.
LUIS VON AHN: Sure.
But given that Sudoku's human solved, computers are a lot
better at those.
AUDIENCE: Have you or anyone thought about the cognitive
aspects of games that are fun in this
model, versus that aren't?
And the computational models that are associated with it?
It seems like there's a lot of human
cognition interests there.
LUIS VON AHN: Yeah, definitely.
So part of the problem--
I should say two things.
There's a lot of research on trying to define and figure
out how to make things more fun.
Non-general computational things, but just how to make
games more fun.
But nobody really knows the answer to this.
I mean, this is an open problem.
AUDIENCE: In the future, if the market becomes competitive
do you think you'll have to start paying people money?
LUIS VON AHN: I don't know.
AUDIENCE: It depends on how much you're
making out of that.
LUIS VON AHN: Yeah, I don't know.
Yeah?
AUDIENCE: In the asymmetric verification games, how do you
eliminate when the first person makes a mistake in the
net, that mistaken output is sent to the second player, and
then after more output from the first guy that's then sent
to the second guy, they get the right answer.
How do you know what--
LUIS VON AHN: The way to do that is by using the
single-player game.
We take all these facts that we get and treat them all
separately, and sometimes you're just playing with a
computer and we're giving you certain facts that
we want you to verify.
And eventually, you just try to intersect which ones are
good and which ones are not, and you try to figure out.
Yeah, all the way in back.
AUDIENCE: How and when is the concept of the game presented
to the players?
LUIS VON AHN: How and when?
AUDIENCE: Yeah.
Like before they start playing or after they finish, you just
say, oh, by the way--
LUIS VON AHN: Oh, no.
Before.
Beforehand.
Beforehand.
Yeah.
Yes?
AUDIENCE: With the security issues now, with things like
Clicker or Picasa Web, where the images are controlled by a
specific entity?
LUIS VON AHN: What do you mean?
AUDIENCE: I could see implications coming from this
such that, you know, well, we want to provide all these
results, but we're going to basically try and have a
monopoly on the best images available.
Which seems kind of anti--
I don't know.
It just seems like it would be more--
people trying to get more control over content.
LUIS VON AHN: I don't understand what you mean.
There are problems with copyright.
I mean, Google knows about that.
So, yeah, there are problems with that.
But I don't know what else.
I mean--
Yes?
AUDIENCE: So, this talk was about generalizing past ESP
games to all kinds of things with human computation.
So you've looked at a bunch of these now.
From what I can tell so far, it's opportunistic.
Given a new task where you know people are better than
computers, is there some procedure for coming to figure
out what the right game is to get at that?
LUIS VON AHN: That would be great.
But I mean, in the same way that, for instance, if I give
you a new task and you have to come up with an efficient
algorithm for it, there's no procedure to coming up with an
efficient algorithm to solve something.
I don't think there will be a procedure, given a problem,
here's a game for it.
I think it's going to be an art, much like coming up with
efficient algorithms.
AUDIENCE: What does that mean for your research strategy and
research agenda?
Is it to just continue to find,
opportunistically, more of these?
LUIS VON AHN: Yes.
So basically, it's similar to what happens
in algorithm design.
I mean, people try to come up with general things.
So there's things like dynamic programming that works for a
lot of things.
So that's the best you can hope for, and that's sort of
what I'm trying to do.
But yeah, I don't think there'll ever be--
well, I don't know.
I'm not holding my breath that there'll ever be a method that
will just, given a problem, output a game.
MALE SPEAKER: OK, I'll ask if nobody else wants to.
So are you worried about the interface
between these two things?
Like, does the existence of these games and their
popularity reduce the value of the CAPTCHAs?
LUIS VON AHN: Oh, the CAPTCHA.
Yeah, yeah.
You can use these games to break the CAPTCHAs.
Yeah, definitely.
[LAUGHTER]
It's good to do research that breaks each other.
[LAUGHTER]
MALE SPEAKER: Thanks, Luis.
[APPLAUSE]