I write fiction sci-fi thrillers,

我寫驚悚科幻小說

so if I say "killer robots,"

所以如果我談到「殺手機器人」

you'd probably think something like this.

你可能會覺得是這樣的東西

But I'm actually not here to talk about fiction.

但事實上我不是來這裡談論小說的

I'm here to talk about very real killer robots,

我是來這裡談論真正的殺手機器人

autonomous combat drones.

自主的無人戰機

Now, I'm not referring to Predator and Reaper drones,

我的意思不是掠奪者或是收割者無人戰機

which have a human making targeting decisions.

由人類來決定目標

I'm talking about fully autonomous robotic weapons

我說的是全自動的機械武器

that make lethal decisions about human beings

能有自主性的對人類做出致命性的決策

all on their own.

不倚靠人類

There's actually a technical term for this: lethal autonomy.

這有個專門術語：致命自主

Now, lethally autonomous killer robots

致命性自主殺手機器人

would take many forms -- flying, driving,

有多種模式－飛行、陸行

or just lying in wait.

或只是待命

And actually, they're very quickly becoming a reality.

事實上它們很快就要問世了

These are two automatic sniper stations

這是兩座自動狙擊台

currently deployed in the DMZ between North and South Korea.

目前部署在南北韓非軍事區

Both of these machines are capable of automatically

這兩台機器都是自動的

identifying a human target and firing on it,

能夠辨識人類的位置並開火射擊

the one on the left at a distance of over a kilometer.

左邊這台能瞄準超過一公里

Now, in both cases, there's still a human in the loop

這兩座狙擊台都還是有人力介入

to make that lethal firing decision,

來下達致命攻擊的決策

but it's not a technological requirement. It's a choice.

但並不是技術上的需要，而是選項

And it's that choice that I want to focus on,

這個選項就是我的重點

because as we migrate lethal decision-making

因為當我們將致命決策權

from humans to software,

由人類轉移到軟體設備

we risk not only taking the humanity out of war,

我們冒的風險不僅是將人性抽離戰爭

but also changing our social landscape entirely,

同時也完全改變了我們的社會景觀

far from the battlefield.

遠遠不是戰場可以形容的

That's because the way humans resolve conflict

因為人類解決衝突的方法

shapes our social landscape.

塑造了我們社會的樣貌

And this has always been the case, throughout history.

自古以來皆是如此

For example, these were state-of-the-art weapons systems

好比說這些是西元1400年時

in 1400 A.D.

最先進的武器系統

Now they were both very expensive to build and maintain,

儘管建造、保養代價高昂

but with these you could dominate the populace,

但是透過這些武器我們可以統治群眾

and the distribution of political power in feudal society reflected that.

封建社會下政治力量的分布顯示了這一點

Power was focused at the very top.

力量集中在最頂端

And what changed? Technological innovation.

什麼改變了？科技革新

Gunpowder, cannon.

火藥、大砲

And pretty soon, armor and castles were obsolete,

不久之後，盔甲和城堡就被淘汰了

and it mattered less who you brought to the battlefield

戰場上有什麼士兵不重要

versus how many people you brought to the battlefield.

數量有多少才是關鍵

And as armies grew in size, the nation-state arose

當軍隊壯大之後，出現了民族國家

as a political and logistical requirement of defense.

來滿足政治及後勤方面在防禦上的需求

And as leaders had to rely on more of their populace,

當領導人必須更加倚靠他們的人民

they began to share power.

就會開始分權

Representative government began to form.

代議政府開始成形

So again, the tools we use to resolve conflict

再一次，我們用來解決衝突的工具

shape our social landscape.

塑造了我們社會的樣貌

Autonomous robotic weapons are such a tool,

自主機械武器就是這種工具

except that, by requiring very few people to go to war,

但是因為發動戰爭所需的人數不多

they risk re-centralizing power into very few hands,

有極少數人再度集權統治的風險

possibly reversing a five-century trend toward democracy.

可能讓五百年來民主化的趨勢走回頭路

Now, I think, knowing this,

我想，了解了之後

we can take decisive steps to preserve our democratic institutions,

我們可以果斷的維持民主制度

to do what humans do best, which is adapt.

來做我們人類最擅長的事，那就是適應環境

But time is a factor.

但時間是一個因素

Seventy nations are developing remotely-piloted

有七十個國家正在研發

combat drones of their own,

他們自己的遠程無人戰鬥機

and as you'll see, remotely-piloted combat drones

如你所見，遠程無人戰鬥機

are the precursors to autonomous robotic weapons.

正是自主機械武器的前身

That's because once you've deployed remotely-piloted drones,

因為一旦部屬了遠程無人戰鬥機

there are three powerful factors pushing decision-making

就會有三個強而有力的因素

away from humans and on to the weapon platform itself.

迫使人們把決定權交給武器平台本身

The first of these is the deluge of video that drones produce.

第一個因素是無人戰機所拍攝的大量影像

For example, in 2004, the U.S. drone fleet produced

像是在2004年，美國的無人機艦隊

a grand total of 71 hours of video surveillance for analysis.

拍攝了長達71小時的影像監控以供分析之用

By 2011, this had gone up to 300,000 hours,

到了2011年，影像長度增為30萬小時

outstripping human ability to review it all,

超出人類檢視所有影像的能力

but even that number is about to go up drastically.

但即使數字之大，它依然大幅成長

The Pentagon's Gorgon Stare and Argus programs

五角大廈的「戈爾貢之眼(Gorgon Stare)」和 「百眼巨人(Argus)」計畫

will put up to 65 independently operated camera eyes

將會有65台獨立運作的攝影機

on each drone platform,

安裝在每架無人飛機上

and this would vastly outstrip human ability to review it.

而這將大幅超越人類所能檢視的能力

And that means visual intelligence software will need

這意味著視覺智慧軟體必須掃描影像

to scan it for items of interest.

找出我們感興趣的部分

And that means very soon

這表示不久後無人飛機

drones will tell humans what to look at,

將會告訴人類該看哪些部分

not the other way around.

而非由人類來決定

But there's a second powerful incentive pushing

還有第二個動機

decision-making away from humans and onto machines,

迫使我們把決定權交給機器

and that's electromagnetic jamming,

那就是電磁干擾

severing the connection between the drone

它會切斷無人飛機

and its operator.

和操作員的連線

Now we saw an example of this in 2011

我們現在看到的例子是在2011年

when an American RQ-170 Sentinel drone

美國的RQ-170哨兵式(Sentinel)無人飛機

got a bit confused over Iran due to a GPS spoofing attack,

在伊朗上空因為GPS受到欺騙攻擊而陷入混亂

but any remotely-piloted drone is susceptible to this type of attack,

這種攻擊對任何遠程無人飛機都有效

and that means drones

這表示無人飛機

will have to shoulder more decision-making.

必須去承擔更多決策

They'll know their mission objective,

它們會知道它們的任務目標

and they'll react to new circumstances without human guidance.

沒有人類指導也能對新的狀況作出反應

They'll ignore external radio signals

它們會忽視外來的無線電訊號

and send very few of their own.

只發出極少量的無線電訊號

Which brings us to, really, the third

這牽扯到第三個因素

and most powerful incentive pushing decision-making

也是最強烈的誘因

away from humans and onto weapons:

促使我們將決定權交給武器

plausible deniability.

那就是貌似合理的推託

Now we live in a global economy.

我們生活在一個全球經濟體下

High-tech manufacturing is occurring on most continents.

大部分的地方都有高科技工業

Cyber espionage is spiriting away advanced designs

網路間諜正在將先進的科技

to parts unknown,

帶到未知的地方

and in that environment, it is very likely

在這種環境下很有可能

that a successful drone design will be knocked off in contract factories,

成功的無人飛機設計，在灰市中量產

proliferate in the gray market.

擊敗取得授權的工廠

And in that situation, sifting through the wreckage

如此一來，即使仔細調查

of a suicide drone attack, it will be very difficult to say

無人機自殺攻擊的殘骸

who sent that weapon.

也難以斷定是誰發射這個武器

This raises the very real possibility

這就很有可能

of anonymous war.

會引起匿名戰爭

This could tilt the geopolitical balance on its head,

可能會讓地緣政治傾向一頭

make it very difficult for a nation to turn its firepower

讓一個國家難以將它的炮火對準攻擊者

against an attacker, and that could shift the balance

也會讓21世紀的維持的平衡

in the 21st century away from defense and toward offense.

由防禦轉為侵略

It could make military action a viable option

使得軍事行動變成可行的選項

not just for small nations,

不只是小國

but criminal organizations, private enterprise,

還有犯罪組織、私人企業

even powerful individuals.

甚至是有權力的人都能這麼做

It could create a landscape of rival warlords

這樣可能會引發軍閥割據的混戰

undermining rule of law and civil society.

破壞法律規範及文明社會

Now if responsibility and transparency

如果責任感以及透明度

are two of the cornerstones of representative government,

是代議政治的兩塊基石

autonomous robotic weapons could undermine both.

自主機械武器會破壞這兩者

Now you might be thinking that

你現在可能會覺得

citizens of high-tech nations

高科技國家的人民

would have the advantage in any robotic war,

在機械戰爭中佔有優勢

that citizens of those nations would be less vulnerable,

這些國家的人民比較不容易受傷害

particularly against developing nations.

尤其是對抗發展中國家的時候

But I think the truth is the exact opposite.

但是我認為事實剛好相反

I think citizens of high-tech societies

我認為高科技社會的人民

are more vulnerable to robotic weapons,

更容易受到機械武器傷害

and the reason can be summed up in one word: data.

其原因可以用兩個字來說明：資料

Data powers high-tech societies.

資料壯大了高科技社會

Cell phone geolocation, telecom metadata,

手機定位、電訊後設資料

social media, email, text, financial transaction data,

社交媒體、電子郵件、簡訊、金融交易資料

transportation data, it's a wealth of real-time data

和運輸資料等，都是大量的即時資料

on the movements and social interactions of people.

記載著人類的社交活動

In short, we are more visible to machines

簡單來說，比其任何其他時代

than any people in history,

機器對人類來說更顯而易見

and this perfectly suits the targeting needs of autonomous weapons.

而這正好符合了自動武器的目標需求

What you're looking at here

大家現在看到的

is a link analysis map of a social group.

是社會團體的連結分析圖

Lines indicate social connectedness between individuals.

線段代表著人與人之間的社會連結

And these types of maps can be automatically generated

這類的圖只要依據現代人所留下的資料

based on the data trail modern people leave behind.

就能夠自動產生

Now it's typically used to market goods and services

現在它應用於商品與服務的行銷

to targeted demographics, but it's a dual-use technology,

來統計目標群體，但這是雙面科技

because targeting is used in another context.

因為目標的鎖定可應用於另外一種情況

Notice that certain individuals are highlighted.

我們注意到某些人是特別顯著的

These are the hubs of social networks.

他們是社會網絡的樞紐

These are organizers, opinion-makers, leaders,

他們是組織者、輿論製造者、領導者

and these people also can be automatically identified

而這些人依據他們的溝通模式

from their communication patterns.

我們也能夠自動辨識

Now, if you're a marketer, you might then target them

如果現在你是一位營銷員，你可能將試用產品

with product samples, try to spread your brand

鎖定這些人，試著把你的品牌

through their social group.

推廣到他們的社會團體中

But if you're a repressive government

但如果你是專制的政府

searching for political enemies, you might instead remove them,

在搜尋政敵，你可能反而會除掉他們

eliminate them, disrupt their social group,

消滅他們、破壞他們的社會團體

and those who remain behind lose social cohesion

而剩下來的人就會失去他們的社會連結

and organization.

以及社群組織

Now in a world of cheap, proliferating robotic weapons,

在一個機械武器便宜、氾濫的世界

borders would offer very little protection

國界的保護微不足道

to critics of distant governments

無法防衛遠方政府的批評者

or trans-national criminal organizations.

或是跨國犯罪組織

Popular movements agitating for change

鼓吹改革的群眾運動

could be detected early and their leaders eliminated

在大眾充分了解其訴求之前

before their ideas achieve critical mass.

就會被察覺，並將其領導人消滅

And ideas achieving critical mass

而充分的把想法傳達給大眾

is what political activism in popular government is all about.

正是一般政治激進主義政府所追求的

Anonymous lethal weapons could make lethal action

匿名的致命武器能夠讓致命行動

an easy choice for all sorts of competing interests.

變成所有競爭者一個簡單的選項

And this would put a chill on free speech

這會讓民主的核心－言論自由及民眾在政治上行動

and popular political action, the very heart of democracy.

轉為冷漠

And this is why we need an international treaty

這也就是為什麼我們需要國際公約

on robotic weapons, and in particular a global ban

來規範機械武器，特別是

on the development and deployment of killer robots.

全球皆禁止研發及部屬殺手機器人

Now we already have international treaties

現在我們已經有國際公約

on nuclear and biological weapons, and, while imperfect,

來規範核武及生物武器，雖然並不完美

these have largely worked.

但卻很有效

But robotic weapons might be every bit as dangerous,

但是機械武器完全就是個危險的東西

because they will almost certainly be used,

因為它們幾乎一定會被使用

and they would also be corrosive to our democratic institutions.

也會侵蝕我們的民主制度

Now in November 2012 the U.S. Department of Defense

2012年11月，美國國防部

issued a directive requiring

直接要求

a human being be present in all lethal decisions.

所有致命決策都必須有人類參與其中

This temporarily effectively banned autonomous weapons in the U.S. military,

這暫時有效防止美軍的自主武器

but that directive needs to be made permanent.

但是這個要求必須永久有效

And it could set the stage for global action.

它也可以為全球立法行動鋪路

Because we need an international legal framework

因為我們需要國際性的法律架構

for robotic weapons.

來規範機械武器

And we need it now, before there's a devastating attack

我們現在就需要，以免毀滅性的恐怖攻擊事件

or a terrorist incident that causes nations of the world

讓世界上的國家

to rush to adopt these weapons

爭相採用這些武器

before thinking through the consequences.

完全不考慮後果

Autonomous robotic weapons concentrate too much power

自動機械武器把太強大的力量

in too few hands, and they would imperil democracy itself.

集中在太少人手中，而他們會威脅到民主本身

Now, don't get me wrong, I think there are tons

別誤解了我的意思，我認為

of great uses for unarmed civilian drones:

這些非武裝的民用無人飛機好用途不少

environmental monitoring, search and rescue, logistics.

環境監控、搜救、後勤

If we have an international treaty on robotic weapons,

如果我們有機械武器的國際公約

how do we gain the benefits of autonomous drones

我們要如何從自主無人飛機及載具中獲得好處

and vehicles while still protecting ourselves

同時又能夠保護我們自己

against illegal robotic weapons?

免於違法機械武器的侵害呢

I think the secret will be transparency.

我認為機密會變得透明化

No robot should have an expectation of privacy

任何的機器人在公共場所

in a public place.

都不得有隱私

(Applause)

(掌聲)

Each robot and drone should have

每一台機器人或無人飛機

a cryptographically signed I.D. burned in at the factory

在工廠裡都要烙上一個加密的身分

that can be used to track its movement through public spaces.

可以用來追蹤它在公共場所的動向

We have license plates on cars, tail numbers on aircraft.

我們的車子有車牌，飛機有機尾編號

This is no different.

這是同樣的道理

And every citizen should be able to download an app

每個市民都可以下載應用程式

that shows the population of drones and autonomous vehicles

來顯示無人飛機及自動載具

moving through public spaces around them,

在他們附近的公共空間移動的數量

both right now and historically.

不論是過去或是現在

And civic leaders should deploy sensors and civic drones

領導者應該要部屬感應器還有民用無人飛機

to detect rogue drones,

來偵測圖謀不軌的無人飛機

and instead of sending killer drones of their own up to shoot them down,

而且不是直接派自己的無人攻擊機將它們擊落

they should notify humans to their presence.

而是應該告知人們這些無人飛機的存在

And in certain very high-security areas,

在確定是高安全性的地區

perhaps civic drones would snare them

或許民用無人飛機

and drag them off to a bomb disposal facility.

可以將它們引導到炸彈處理設施

But notice, this is more an immune system

但要注意的是，這比較像是免疫系統

than a weapons system.

而不是武器系統

It would allow us to avail ourselves of the use

如此就可以讓我們

of autonomous vehicles and drones

獲得自主載具與無人飛機所帶來的好處

while still preserving our open, civil society.

同時保有我們開放、文明的社會

We must ban the deployment and development

我們必須禁止研發及部屬

of killer robots.

殺手機器人

Let's not succumb to the temptation to automate war.

我們不會屈服於自動化戰爭的誘惑

Autocratic governments and criminal organizations

專制政府及犯罪組織一定會屈服

undoubtedly will, but let's not join them.

但是我們不會加入他們

Autonomous robotic weapons

自主機械武器

would concentrate too much power

將過多權力

in too few unseen hands,

集中到極少數看不見的手中

and that would be corrosive to representative government.

而那會侵蝕代議政府

Let's make sure, for democracies at least,

我們要保證，至少為了民主的理由

killer robots remain fiction.

讓殺手機器人留在小說裡吧

Thank you.

謝謝