Lee Sedol is one of the world's greatest Go players,

這是李世石。

and he's having what my friends in Silicon Valley call

李世石是全世界 頂尖圍棋高手之一，

a "Holy Cow" moment --

此時，他正在經歷的是 我的矽谷朋友們稱之為

(Laughter)

「我的媽呀！」的時刻......

a moment where we realize

（笑聲）

that AI is actually progressing a lot faster than we expected.

在這一刻讓我們意識到，

So humans have lost on the Go board. What about the real world?

原來人工智慧發展的進程 比我們預期的要快得多。

Well, the real world is much bigger,

人類已在圍棋博弈中落敗， 那現實世界中情況又如何？

much more complicated than the Go board.

當然啦，現實世界要比棋盤 廣闊、複雜得多，

It's a lot less visible,

它也遠不如棋盤上那麽黑白分明，

but it's still a decision problem.

但仍然是個判定問題 （Decision Problem）。

And if we think about some of the technologies

如果我們思考一些 即將問世的新科技……

that are coming down the pike ...

新井紀子提到機器仍無法 「閱讀」，

Noriko [Arai] mentioned that reading is not yet happening in machines,

至少無法真正理解文本含義。

at least with understanding.

但這項能力最終會被機器掌握，

But that will happen,

而當這一切發生時，

and when that happens,

不久之後，

very soon afterwards,

機器就能讀遍所有人類寫下的東西。

machines will have read everything that the human race has ever written.

這會讓機器擁有比人類 更深刻的遠見和洞察力。

And that will enable machines,

就如我們在這場圍棋博弈中所見，

along with the ability to look further ahead than humans can,

如果機器能接觸到比人類更多的信息，

as we've already seen in Go,

那機器將能夠在現實世界中 做出比人類更好的決策。

if they also have access to more information,

那這會是一件好事嗎？

they'll be able to make better decisions in the real world than we can.

我當然希望如此。

So is that a good thing?

人類的全部文明， 我們所珍視的一切，

Well, I hope so.

都是基於我們的智慧。

Our entire civilization, everything that we value,

如果我們能獲得更強大的智慧，

is based on our intelligence.

那人類將無所不能了。

And if we had access to a lot more intelligence,

我在想，到時後就像 一些人所描述的那樣，

then there's really no limit to what the human race can do.

這會是人類歷史上最重要的事件。

And I think this could be, as some people have described it,

那為什麽有的人會說出 以下的言論呢？

the biggest event in human history.

說人工智慧將是人類的末日呢？

So why are people saying things like this,

這是新鮮事嗎？

that AI might spell the end of the human race?

這僅僅只是伊隆馬斯克、比爾蓋茲、 史蒂芬霍金的新發明嗎？

Is this a new thing?

實際上不是，這個概念 已經存在很長的時間了。

Is it just Elon Musk and Bill Gates and Stephen Hawking?

請看這段話：

Actually, no. This idea has been around for a while.

「即便我們能讓機器屈從於我們，

Here's a quotation:

比如說，在重要時刻關掉它。」

"Even if we could keep the machines in a subservient position,

我等會兒會再來討論 「關機」這一話題。

for instance, by turning off the power at strategic moments" --

「我們作為人類，仍應懷着謙卑......」

and I'll come back to that "turning off the power" idea later on --

這段話是誰說的呢？ 是艾倫 · 圖靈在 1951 年說的。

"we should, as a species, feel greatly humbled."

眾所皆知艾倫 · 圖靈是計算機科學之父，

So who said this? This is Alan Turing in 1951.

並且從很多方面來講， 他也是人工智慧之父。

Alan Turing, as you know, is the father of computer science

所以，當我們在思考「創造出 比自己更聰明的物種」這個問題時，

and in many ways, the father of AI as well.

我們不妨將它稱為「大猩猩問題」。

So if we think about this problem,

因為大猩猩的祖先們 在幾百萬年前就親歷此境，

the problem of creating something more intelligent than your own species,

我們可以去問大猩猩們：

we might call this "the gorilla problem,"

「這是不是一個好主意？」

because gorillas' ancestors did this a few million years ago,

圖片中，牠們正在開會討論 那麽做是不是一個好主意，

and now we can ask the gorillas:

過了一會兒，牠們總結出：「不。」

Was this a good idea?

這是個很爛的主意──

So here they are having a meeting to discuss whether it was a good idea,

作為靈長類的我們正岌岌可危。

and after a little while, they conclude, no,

你可以從牠們的眼神中 看到存亡攸關的憂傷。

this was a terrible idea.

（笑聲）

Our species is in dire straits.

「創造出比你自己更聰明的物種 並不是什麽妙計」

In fact, you can see the existential sadness in their eyes.

這種感覺很倒胃口。

(Laughter)

那我們能做些什麽呢？

So this queasy feeling that making something smarter than your own species

其實，除非停止人工智慧的研究， 否則束手無策。

is maybe not a good idea --

因為我所提到的人工智慧的各種裨益，

what can we do about that?

也因為我是人工智慧的研究人員，

Well, really nothing, except stop doing AI,

我可不同意就此止步。

and because of all the benefits that I mentioned

實際上，我想一直研究人工智慧。

and because I'm an AI researcher,

所以我們需要更加明確問題所在。

I'm not having that.

這個問題到底是什麽呢？

I actually want to be able to keep doing AI.

為什麽更強大的人工智慧 可能會是個災難呢？

So we actually need to nail down the problem a bit more.

還有一句名言：

What exactly is the problem?

「我們最好確保我們向機器發出的指令 與我們的真正目的相吻合。」

Why is better AI possibly a catastrophe?

這句話是諾伯特 · 維納在 1960 年說的，

So here's another quotation:

就在他看完一個早期的學習系統 （Learning System）之後。

"We had better be quite sure that the purpose put into the machine

這個系統在學習如何能把 西洋棋下得比發明它的人更好。

is the purpose which we really desire."

但如出一轍的一句話，

This was said by Norbert Wiener in 1960,

邁達斯國王也說過。

shortly after he watched one of the very early learning systems

他說：「我希望我觸碰的 所有東西都變成金子。」

learn to play checkers better than its creator.

結果他真的獲得了點石成金的能力。

But this could equally have been said

可以說，這就是他給機器下的指令。

by King Midas.

結果他的食物、飲料 和家人都變成了金子，

King Midas said, "I want everything I touch to turn to gold,"

最後他死於痛苦與饑餓當中。

and he got exactly what he asked for.

所以我們把這類問題叫做 「邁達斯國王問題」，

That was the purpose that he put into the machine,

這個比喻是要說明這種 不符合實際需求的 「目的」。

so to speak,

用現代的術語來說，我們把它稱為 「價值取向不一致問題」。

and then his food and his drink and his relatives turned to gold

「設錯了目標」不是唯一的問題，

and he died in misery and starvation.

還有其他的。

So we'll call this "the King Midas problem"

如果你給機器人設了個目標，

of stating an objective which is not, in fact,

即使簡單如「去把咖啡端來。」

truly aligned with what we want.

那機器人會對自己說：

In modern terms, we call this "the value alignment problem."

「什麼會讓我無法去拿咖啡？

Putting in the wrong objective is not the only part of the problem.

說不定有人會把我關機；

There's another part.

好，那我要想辦法阻止，

If you put an objective into a machine,

我得讓我的「關機」開關失效。

even something as simple as, "Fetch the coffee,"

我得盡一切可能防衛自己，

the machine says to itself,

免得別人干涉我去達成 所被賦予的任務。」

"Well, how might I fail to fetch the coffee?

這種專注的行事，以一種 極端自我保護的模式在執行，

Someone might switch me off.

實際上與我們人類 想要的目標並不一致。

OK, I have to take steps to prevent that.

這就是我們面臨的問題。

I will disable my 'off' switch.

而這就是這場演講的 核心想法，也是價值所在。

I will do anything to defend myself against interference

如果你想從這場演講中汲取什麽，

with this objective that I have been given."

那你只要記得： 如果死了，就不能端咖啡了。

So this single-minded pursuit

（笑聲）

in a very defensive mode of an objective that is, in fact,

這很簡單，記住就行了， 每天早晚覆誦三遍。

not aligned with the true objectives of the human race --

（笑聲）

that's the problem that we face.

實際上，這正是電影 《2001太空漫步》的劇情。

And in fact, that's the high-value takeaway from this talk.

HAL 有一個目標，一個任務，

If you want to remember one thing,

但這個目標與人類的目標不一致，

it's that you can't fetch the coffee if you're dead.

最後導致了衝突。

(Laughter)

幸運的是， HAL 並沒有超級智慧，

It's very simple. Just remember that. Repeat it to yourself three times a day.

它挺聰明的， 但還是比不過人類戴夫，

(Laughter)

戴夫可以把 HAL 關掉。

And in fact, this is exactly the plot

但我們可能就沒有這麽幸運了。

of "2001: [A Space Odyssey]"

那我們應該怎麽辦呢？

HAL has an objective, a mission,

我想要重新定義人工智慧，

which is not aligned with the objectives of the humans,

不再囿於傳統的概念：

and that leads to this conflict.

能明智地達成目標的機器。

Now fortunately, HAL is not superintelligent.

新的定義涉及三條原則。

He's pretty smart, but eventually Dave outwits him

第一個原則是利他主義原則，

and manages to switch him off.

也就是說，機器的唯一目標

But we might not be so lucky.

就是要最大化地實現 人類的目標、人類的價值。

So what are we going to do?

這種價值不是指多愁善感 或者假裝乖巧，

I'm trying to redefine AI

而是指人類所嚮往、追求的生活， 無論現狀如何。

to get away from this classical notion

事實上，這樣就違反了艾西莫夫定律，

of machines that intelligently pursue objectives.

定律裡的機器人必須維護自己的生存。

There are three principles involved.

而在這條原則裡 機器對自身生存與否毫不關心。

The first one is a principle of altruism, if you like,

第二個原則，不妨稱之為謙遜原則。

that the robot's only objective

這一條對製造出安全的機器人十分重要。

is to maximize the realization of human objectives,

它是指機器人不知道人類的價值是什麽，

of human values.

它只知道將該價值最大化， 但卻不知道該價值究竟是什麽。

And by values here I don't mean touchy-feely, goody-goody values.

這就避免了「追求單一目的 而不知變通」的現象。

I just mean whatever it is that the human would prefer

這種不確定性就變得很重要了。

their life to be like.

為了對我們有益，

And so this actually violates Asimov's law

機械就得大概明白我們想要什麽。

that the robot has to protect its own existence.

它要獲取這類信息，主要是 透過觀察人類的決策，

It has no interest in preserving its existence whatsoever.

所以我們的決策會揭露 我們生活的意願，

The second law is a law of humility, if you like.

所以，這三條原則，

And this turns out to be really important to make robots safe.

讓我們來看看要如何 應用到圖靈所說的問題：

It says that the robot does not know

「你能不能將機器關掉？」

what those human values are,

這是 PR2 機器人，

so it has to maximize them, but it doesn't know what they are.

這是我們實驗室裡的其中一台，

And that avoids this problem of single-minded pursuit

它的背面有一個大大的紅色開關。

of an objective.

那問題來了：它會讓你把它關掉嗎？

This uncertainty turns out to be crucial.

如果我們用傳統的定義製造它，

Now, in order to be useful to us,

我們給它一個「去拿咖啡」的目標， 它會想：「我必須去拿咖啡，

it has to have some idea of what we want.

但如果我死了，就不能拿咖啡了。」

It obtains that information primarily by observation of human choices,

看來， PR2 聽過我的演講了，

so our own choices reveal information

因此它說：「我必須讓自己的開關失靈，

about what it is that we prefer our lives to be like.

可能還要通過電擊把那些在 星巴克裡干擾我的人都擊暈。」

So those are the three principles.

（笑聲）

Let's see how that applies to this question of:

這無法避免，對吧？

"Can you switch the machine off?" as Turing suggested.

這種失敗看起來是必然的，

So here's a PR2 robot.

因為機器人會遵循一個 十分明確的目標。

This is one that we have in our lab,

那如果機器對目標 不那麽確定會發生什麽呢？

and it has a big red "off" switch right on the back.

那它的思路就不一樣了。

The question is: Is it going to let you switch it off?

它會說：「好的，人類可能會把我關掉，

If we do it the classical way,

但只有我做錯事了，才會把我關掉。

we give it the objective of, "Fetch the coffee, I must fetch the coffee,

沒錯，我真的不知道什麽才是錯，

I can't fetch the coffee if I'm dead,"

但我知道我不該做錯的事。」

so obviously the PR2 has been listening to my talk,

這就是第一和第二原則。

and so it says, therefore, "I must disable my 'off' switch,

「所以我應該讓人類把我關掉。」

and probably taser all the other people in Starbucks

事實上你可以推斷出機器人為了 允許讓人類關掉它所包含的動機，

who might interfere with me."

而且這與根本目標的 不確定性程度直接相關。

(Laughter)

當機器被關閉後，

So this seems to be inevitable, right?

第三條原則就起作用了。

This kind of failure mode seems to be inevitable,

機器開始學習它應追求的目標，

and it follows from having a concrete, definite objective.

因為它知道它剛才做的事是不對的。

So what happens if the machine is uncertain about the objective?

實際上，我們可以適當地 使用些希臘字母，

Well, it reasons in a different way.

就像數學家們經常做的那樣，

It says, "OK, the human might switch me off,

直接證明這一個理論：這樣的 機器人對人類是絕對有利的。

but only if I'm doing something wrong.

可以證明如此設計出來的機器人， 對我們的生活是是有益的。

Well, I don't really know what wrong is,

這個例子很簡單，

but I know that I don't want to do it."

但它是我們嘗試實現 能與人類和諧共處的 AI 的第一步。

So that's the first and second principles right there.

現在來看第三個原則，

"So I should let the human switch me off."

我知道各位可能還在為 這一個原則傷腦筋。

And in fact you can calculate the incentive that the robot has

你可能會想：「你懂的， 我行為舉止比較差勁。

to allow the human to switch it off,

我的機器人可不能被我帶壞。

and it's directly tied to the degree

我有時後會大半夜偷偷摸摸地 從冰箱裡找東西吃，

of uncertainty about the underlying objective.

東瞅瞅，西摸摸。」

And then when the machine is switched off,

有各種各樣的事 你是不希望機器人去做的。

that third principle comes into play.

但實際上不是那樣。

It learns something about the objectives it should be pursuing,

你行為不檢，

because it learns that what it did wasn't right.

不代表機器人就得有樣學樣。

In fact, we can, with suitable use of Greek symbols,

它會去嘗試理解你做事的動機，

as mathematicians usually do,

而且可能會在合適的情況下 幫助你、制止你。

we can actually prove a theorem

但這仍然十分困難。

that says that such a robot is provably beneficial to the human.

實際上，我們是要讓機器

You are provably better off with a machine that's designed in this way

為任何人、任何一種 可能的生活去預測：

than without it.

他們更想怎樣？更想要什麽？

So this is a very simple example, but this is the first step

這涉及到諸多困難，

in what we're trying to do with human-compatible AI.

我不認為這會很快地就被解決。

Now, this third principle,

實際上，真正的困難是我們自己。

I think is the one that you're probably scratching your head over.

就像我剛說的那樣， 我們做事不守規矩。

You're probably thinking, "Well, you know, I behave badly.

我們當中就有人是非常惡劣的。

I don't want my robot to behave like me.

如前所說，機器人 未必得要複製那些行為。

I sneak down in the middle of the night and take stuff from the fridge.

機器人沒有自己的目標，

I do this and that."

它是完全利他的。

There's all kinds of things you don't want the robot doing.

它的誕生不僅僅是為了去滿足 某一個人、某一個用戶的欲望，

But in fact, it doesn't quite work that way.

而是去尊重所有人的意願。

Just because you behave badly

所以它懂得抵制一些惡劣的行為，

doesn't mean the robot is going to copy your behavior.

它甚至能理解你為什麼惡劣，比如說，

It's going to understand your motivations and maybe help you resist them,

如果你是一個邊境護照官員， 你可能會收取賄賂，

if appropriate.

因為你得養家、供孩子們上學。

But it's still difficult.

機器人能理解這一點， 但不代表它也會學你偷錢，

What we're trying to do, in fact,

它反而會幫助你去供孩子們上學。

is to allow machines to predict for any person and for any possible life

我們的計算能力也是有限的。

that they could live,

李世石是一個傑出的圍棋大師，

and the lives of everybody else:

但他還是輸了。

Which would they prefer?

如果我們仔細觀察他的棋路， 他下錯了那幾步以致輸棋，

And there are many, many difficulties involved in doing this;

但這不意味著他想要輸。

I don't expect that this is going to get solved very quickly.

所以要理解他的行為，

The real difficulties, in fact, are us.

我們得從人類認知的模型回推過來，

As I have already mentioned, we behave badly.

它包含了我們計算能力上的局限，

In fact, some of us are downright nasty.

是一個很覆雜的模型。

Now the robot, as I said, doesn't have to copy the behavior.

但我們仍然可以嘗試去理解。

The robot does not have any objective of its own.

可能對於我這樣的 AI 研究人員來說，

It's purely altruistic.

最大的困難是，人有很多種，

And it's not designed just to satisfy the desires of one person, the user,

所以機器必須想辦法去協調、 權衡不同人之間的喜好、需求，

but in fact it has to respect the preferences of everybody.

而要做到這一點有多種不同的方法。

So it can deal with a certain amount of nastiness,

經濟學家、社會學家、 道德哲學家都理解這一點，

and it can even understand that your nastiness, for example,

我們正積極地尋求合作。

you may take bribes as a passport official

讓我們來看看，如果我們把這一步 走錯了會怎麽樣。

because you need to feed your family and send your kids to school.

比如說，你可能會與你的 人工智慧助理有這樣的對話，

It can understand that; it doesn't mean it's going to steal.

這樣的人工智慧可能幾年內就會出現。

In fact, it'll just help you send your kids to school.

可以把它想成是強化版的 Siri 。

We are also computationally limited.

Siri 對你說：「你老婆打電話 提醒你別忘了今天的晚宴。」

Lee Sedol is a brilliant Go player,

當然你早就忘了這回事：

but he still lost.

「什麽？什麽晚宴？你在說什麽？」

So if we look at his actions, he took an action that lost the game.

「呃.....今晚 7 點 慶祝結婚 20 周年。」

That doesn't mean he wanted to lose.

「我可去不了， 我晚上 7 點半要見秘書長。

So to understand his behavior,

怎麽會這樣呢？」

we actually have to invert through a model of human cognition

「呃，我可是提醒過你的， 但你沒有理會我的建議。」

that includes our computational limitations -- a very complicated model.

「我該怎麽辦呢？我可不能跟秘書長說 我有事，沒空見他。」

But it's still something that we can work on understanding.

「別擔心。我已經安排了， 讓他的航班延誤。」