0:00:00.691,0:00:03.849
I write fiction sci-fi thrillers,

0:00:03.849,0:00:06.042
so if I say "killer robots,"

0:00:06.042,0:00:08.403
you'd probably think something like this.

0:00:08.403,0:00:10.958
But I'm actually not here to talk about fiction.

0:00:10.958,0:00:13.906
I'm here to talk about very real killer robots,

0:00:13.906,0:00:17.028
autonomous combat drones.

0:00:17.028,0:00:20.655
Now, I'm not referring to Predator and Reaper drones,

0:00:20.655,0:00:23.915
which have a human making targeting decisions.

0:00:23.915,0:00:27.044
I'm talking about fully autonomous robotic weapons

0:00:27.044,0:00:29.698
that make lethal decisions about human beings

0:00:29.698,0:00:32.115
all on their own.

0:00:32.115,0:00:36.115
There's actually a technical term for this: lethal autonomy.

0:00:36.115,0:00:38.971
Now, lethally autonomous killer robots

0:00:38.971,0:00:42.040
would take many forms -- flying, driving,

0:00:42.040,0:00:44.786
or just lying in wait.

0:00:44.786,0:00:47.895
And actually, they're very quickly becoming a reality.

0:00:47.895,0:00:50.379
These are two automatic sniper stations

0:00:50.379,0:00:54.516
currently deployed in the DMZ between North and South Korea.

0:00:54.516,0:00:56.687
Both of these machines are capable of automatically

0:00:56.687,0:01:00.211
identifying a human target and firing on it,

0:01:00.211,0:01:04.535
the one on the left at a distance of over a kilometer.

0:01:04.535,0:01:08.124
Now, in both cases, there's still a human in the loop

0:01:08.124,0:01:10.496
to make that lethal firing decision,

0:01:10.496,0:01:15.909
but it's not a technological requirement. It's a choice.

0:01:15.909,0:01:19.002
And it's that choice that I want to focus on,

0:01:19.002,0:01:21.643
because as we migrate lethal decision-making

0:01:21.643,0:01:24.752
from humans to software,

0:01:24.752,0:01:28.228
we risk not only taking the humanity out of war,

0:01:28.228,0:01:31.754
but also changing our social landscape entirely,

0:01:31.754,0:01:33.978
far from the battlefield.

0:01:33.978,0:01:38.487
That's because the way humans resolve conflict

0:01:38.487,0:01:40.220
shapes our social landscape.

0:01:40.220,0:01:42.853
And this has always been the case, throughout history.

0:01:42.853,0:01:45.514
For example, these were state-of-the-art weapons systems

0:01:45.514,0:01:47.593
in 1400 A.D.

0:01:47.593,0:01:50.737
Now they were both very expensive to build and maintain,

0:01:50.737,0:01:53.977
but with these you could dominate the populace,

0:01:53.977,0:01:57.866
and the distribution of political power in feudal society reflected that.

0:01:57.866,0:02:00.553
Power was focused at the very top.

0:02:00.553,0:02:04.081
And what changed? Technological innovation.

0:02:04.081,0:02:05.952
Gunpowder, cannon.

0:02:05.952,0:02:09.769
And pretty soon, armor and castles were obsolete,

0:02:09.769,0:02:12.302
and it mattered less who you brought to the battlefield

0:02:12.302,0:02:16.081
versus how many people you brought to the battlefield.

0:02:16.081,0:02:19.719
And as armies grew in size, the nation-state arose

0:02:19.719,0:02:23.399
as a political and logistical requirement of defense.

0:02:23.399,0:02:25.775
And as leaders had to rely on more of their populace,

0:02:25.775,0:02:27.608
they began to share power.

0:02:27.608,0:02:30.207
Representative government began to form.

0:02:30.207,0:02:33.495
So again, the tools we use to resolve conflict

0:02:33.495,0:02:36.799
shape our social landscape.

0:02:36.799,0:02:40.863
Autonomous robotic weapons are such a tool,

0:02:40.863,0:02:46.031
except that, by requiring very few people to go to war,

0:02:46.031,0:02:50.871
they risk re-centralizing power into very few hands,

0:02:50.871,0:02:57.386
possibly reversing a five-century trend toward democracy.

0:02:57.386,0:02:59.143
Now, I think, knowing this,

0:02:59.143,0:03:03.495
we can take decisive steps to preserve our democratic institutions,

0:03:03.495,0:03:07.474
to do what humans do best, which is adapt.

0:03:07.474,0:03:09.479
But time is a factor.

0:03:09.479,0:03:12.330
Seventy nations are developing remotely-piloted

0:03:12.330,0:03:14.487
combat drones of their own,

0:03:14.487,0:03:17.080
and as you'll see, remotely-piloted combat drones

0:03:17.080,0:03:21.552
are the precursors to autonomous robotic weapons.

0:03:21.552,0:03:24.319
That's because once you've deployed remotely-piloted drones,

0:03:24.319,0:03:27.703
there are three powerful factors pushing decision-making

0:03:27.703,0:03:32.303
away from humans and on to the weapon platform itself.

0:03:32.303,0:03:37.562
The first of these is the deluge of video that drones produce.

0:03:37.562,0:03:41.415
For example, in 2004, the U.S. drone fleet produced

0:03:41.415,0:03:46.727
a grand total of 71 hours of video surveillance for analysis.

0:03:46.727,0:03:51.226
By 2011, this had gone up to 300,000 hours,

0:03:51.226,0:03:54.375
outstripping human ability to review it all,

0:03:54.375,0:03:58.039
but even that number is about to go up drastically.

0:03:58.039,0:04:00.614
The Pentagon's Gorgon Stare and Argus programs

0:04:00.614,0:04:03.778
will put up to 65 independently operated camera eyes

0:04:03.778,0:04:05.816
on each drone platform,

0:04:05.816,0:04:09.119
and this would vastly outstrip human ability to review it.

0:04:09.119,0:04:11.279
And that means visual intelligence software will need

0:04:11.279,0:04:15.327
to scan it for items of interest.

0:04:15.327,0:04:16.675
And that means very soon

0:04:16.675,0:04:19.422
drones will tell humans what to look at,

0:04:19.422,0:04:21.919
not the other way around.

0:04:21.919,0:04:24.392
But there's a second powerful incentive pushing

0:04:24.392,0:04:27.775
decision-making away from humans and onto machines,

0:04:27.775,0:04:30.647
and that's electromagnetic jamming,

0:04:30.647,0:04:32.883
severing the connection between the drone

0:04:32.883,0:04:35.697
and its operator.

0:04:35.697,0:04:38.315
Now we saw an example of this in 2011

0:04:38.315,0:04:41.271
when an American RQ-170 Sentinel drone

0:04:41.271,0:04:45.578
got a bit confused over Iran due to a GPS spoofing attack,

0:04:45.578,0:04:50.692
but any remotely-piloted drone is susceptible to this type of attack,

0:04:50.692,0:04:52.744
and that means drones

0:04:52.744,0:04:56.364
will have to shoulder more decision-making.

0:04:56.364,0:04:59.407
They'll know their mission objective,

0:04:59.407,0:05:04.252
and they'll react to new circumstances without human guidance.

0:05:04.252,0:05:06.833
They'll ignore external radio signals

0:05:06.833,0:05:09.163
and send very few of their own.

0:05:09.163,0:05:11.169
Which brings us to, really, the third

0:05:11.169,0:05:15.031
and most powerful incentive pushing decision-making

0:05:15.031,0:05:18.373
away from humans and onto weapons:

0:05:18.373,0:05:21.666
plausible deniability.

0:05:21.666,0:05:24.553
Now we live in a global economy.

0:05:24.553,0:05:28.887
High-tech manufacturing is occurring on most continents.

0:05:28.887,0:05:31.801
Cyber espionage is spiriting away advanced designs

0:05:31.801,0:05:33.687
to parts unknown,

0:05:33.687,0:05:35.701
and in that environment, it is very likely

0:05:35.701,0:05:40.435
that a successful drone design will be knocked off in contract factories,

0:05:40.435,0:05:42.605
proliferate in the gray market.

0:05:42.605,0:05:45.065
And in that situation, sifting through the wreckage

0:05:45.065,0:05:48.025
of a suicide drone attack, it will be very difficult to say

0:05:48.025,0:05:52.425
who sent that weapon.

0:05:52.425,0:05:55.225
This raises the very real possibility

0:05:55.225,0:05:58.160
of anonymous war.

0:05:58.160,0:06:00.774
This could tilt the geopolitical balance on its head,

0:06:00.774,0:06:04.265
make it very difficult for a nation to turn its firepower

0:06:04.265,0:06:07.113
against an attacker, and that could shift the balance

0:06:07.113,0:06:10.877
in the 21st century away from defense and toward offense.

0:06:10.877,0:06:14.001
It could make military action a viable option

0:06:14.001,0:06:16.289
not just for small nations,

0:06:16.289,0:06:18.834
but criminal organizations, private enterprise,

0:06:18.834,0:06:21.313
even powerful individuals.

0:06:21.313,0:06:24.641
It could create a landscape of rival warlords

0:06:24.641,0:06:28.321
undermining rule of law and civil society.

0:06:28.321,0:06:31.937
Now if responsibility and transparency

0:06:31.937,0:06:34.321
are two of the cornerstones of representative government,

0:06:34.321,0:06:38.641
autonomous robotic weapons could undermine both.

0:06:38.641,0:06:40.187
Now you might be thinking that

0:06:40.187,0:06:42.433
citizens of high-tech nations

0:06:42.433,0:06:45.136
would have the advantage in any robotic war,

0:06:45.136,0:06:48.769
that citizens of those nations would be less vulnerable,

0:06:48.769,0:06:53.057
particularly against developing nations.

0:06:53.057,0:06:56.581
But I think the truth is the exact opposite.

0:06:56.581,0:06:58.832
I think citizens of high-tech societies

0:06:58.832,0:07:02.561
are more vulnerable to robotic weapons,

0:07:02.561,0:07:07.026
and the reason can be summed up in one word: data.

0:07:07.026,0:07:10.507
Data powers high-tech societies.

0:07:10.507,0:07:13.697
Cell phone geolocation, telecom metadata,

0:07:13.697,0:07:17.169
social media, email, text, financial transaction data,

0:07:17.169,0:07:20.701
transportation data, it's a wealth of real-time data

0:07:20.701,0:07:24.074
on the movements and social interactions of people.

0:07:24.074,0:07:27.849
In short, we are more visible to machines

0:07:27.849,0:07:30.091
than any people in history,

0:07:30.091,0:07:35.707
and this perfectly suits the targeting needs of autonomous weapons.

0:07:35.707,0:07:37.445
What you're looking at here

0:07:37.445,0:07:40.691
is a link analysis map of a social group.

0:07:40.691,0:07:44.325
Lines indicate social connectedness between individuals.

0:07:44.325,0:07:47.205
And these types of maps can be automatically generated

0:07:47.205,0:07:51.920
based on the data trail modern people leave behind.

0:07:51.920,0:07:54.397
Now it's typically used to market goods and services

0:07:54.397,0:07:58.813
to targeted demographics, but it's a dual-use technology,

0:07:58.813,0:08:02.173
because targeting is used in another context.

0:08:02.173,0:08:04.733
Notice that certain individuals are highlighted.

0:08:04.733,0:08:08.013
These are the hubs of social networks.

0:08:08.013,0:08:11.603
These are organizers, opinion-makers, leaders,

0:08:11.603,0:08:14.285
and these people also can be automatically identified

0:08:14.285,0:08:16.667
from their communication patterns.

0:08:16.667,0:08:18.813
Now, if you're a marketer, you might then target them

0:08:18.813,0:08:21.356
with product samples, try to spread your brand

0:08:21.356,0:08:24.185
through their social group.

0:08:24.185,0:08:26.138
But if you're a repressive government

0:08:26.138,0:08:30.948
searching for political enemies, you might instead remove them,

0:08:30.948,0:08:33.708
eliminate them, disrupt their social group,

0:08:33.708,0:08:36.877
and those who remain behind lose social cohesion

0:08:36.877,0:08:39.498
and organization.

0:08:39.498,0:08:42.822
Now in a world of cheap, proliferating robotic weapons,

0:08:42.822,0:08:45.457
borders would offer very little protection

0:08:45.457,0:08:47.403
to critics of distant governments

0:08:47.403,0:08:51.049
or trans-national criminal organizations.

0:08:51.049,0:08:54.542
Popular movements agitating for change

0:08:54.542,0:08:58.151
could be detected early and their leaders eliminated

0:08:58.151,0:09:01.062
before their ideas achieve critical mass.

0:09:01.062,0:09:03.653
And ideas achieving critical mass

0:09:03.653,0:09:07.589
is what political activism in popular government is all about.

0:09:07.589,0:09:11.586
Anonymous lethal weapons could make lethal action

0:09:11.586,0:09:15.368
an easy choice for all sorts of competing interests.

0:09:15.368,0:09:19.102
And this would put a chill on free speech

0:09:19.102,0:09:24.410
and popular political action, the very heart of democracy.

0:09:24.410,0:09:27.324
And this is why we need an international treaty

0:09:27.324,0:09:30.864
on robotic weapons, and in particular a global ban

0:09:30.864,0:09:34.772
on the development and deployment of killer robots.

0:09:34.772,0:09:38.026
Now we already have international treaties

0:09:38.026,0:09:41.412
on nuclear and biological weapons, and, while imperfect,

0:09:41.412,0:09:43.700
these have largely worked.

0:09:43.700,0:09:47.468
But robotic weapons might be every bit as dangerous,

0:09:47.468,0:09:50.756
because they will almost certainly be used,

0:09:50.756,0:09:55.783
and they would also be corrosive to our democratic institutions.

0:09:55.783,0:09:59.251
Now in November 2012 the U.S. Department of Defense

0:09:59.251,0:10:01.709
issued a directive requiring

0:10:01.709,0:10:06.228
a human being be present in all lethal decisions.

0:10:06.228,0:10:11.004
This temporarily effectively banned autonomous weapons in the U.S. military,

0:10:11.004,0:10:14.757
but that directive needs to be made permanent.

0:10:14.757,0:10:19.133
And it could set the stage for global action.

0:10:19.133,0:10:22.978
Because we need an international legal framework

0:10:22.978,0:10:25.116
for robotic weapons.

0:10:25.116,0:10:28.044
And we need it now, before there's a devastating attack

0:10:28.044,0:10:31.196
or a terrorist incident that causes nations of the world

0:10:31.196,0:10:33.120
to rush to adopt these weapons

0:10:33.120,0:10:36.891
before thinking through the consequences.

0:10:36.891,0:10:39.872
Autonomous robotic weapons concentrate too much power

0:10:39.872,0:10:46.155
in too few hands, and they would imperil democracy itself.

0:10:46.155,0:10:48.841
Now, don't get me wrong, I think there are tons

0:10:48.841,0:10:51.459
of great uses for unarmed civilian drones:

0:10:51.459,0:10:55.398
environmental monitoring, search and rescue, logistics.

0:10:55.398,0:10:58.224
If we have an international treaty on robotic weapons,

0:10:58.224,0:11:01.811
how do we gain the benefits of autonomous drones

0:11:01.811,0:11:04.459
and vehicles while still protecting ourselves

0:11:04.459,0:11:08.439
against illegal robotic weapons?

0:11:08.439,0:11:13.180
I think the secret will be transparency.

0:11:13.180,0:11:16.193
No robot should have an expectation of privacy

0:11:16.193,0:11:19.644
in a public place.

0:11:19.644,0:11:24.692
(Applause)

0:11:24.692,0:11:26.737
Each robot and drone should have

0:11:26.737,0:11:29.620
a cryptographically signed I.D. burned in at the factory

0:11:29.620,0:11:32.543
that can be used to track its movement through public spaces.

0:11:32.543,0:11:35.924
We have license plates on cars, tail numbers on aircraft.

0:11:35.924,0:11:37.765
This is no different.

0:11:37.765,0:11:39.777
And every citizen should be able to download an app

0:11:39.777,0:11:42.902
that shows the population of drones and autonomous vehicles

0:11:42.902,0:11:45.331
moving through public spaces around them,

0:11:45.331,0:11:48.064
both right now and historically.

0:11:48.064,0:11:51.612
And civic leaders should deploy sensors and civic drones

0:11:51.612,0:11:53.956
to detect rogue drones,

0:11:53.956,0:11:57.132
and instead of sending killer drones of their own up to shoot them down,

0:11:57.132,0:12:00.124
they should notify humans to their presence.

0:12:00.124,0:12:02.730
And in certain very high-security areas,

0:12:02.730,0:12:04.639
perhaps civic drones would snare them

0:12:04.639,0:12:07.480
and drag them off to a bomb disposal facility.

0:12:07.480,0:12:10.507
But notice, this is more an immune system

0:12:10.507,0:12:11.828
than a weapons system.

0:12:11.828,0:12:14.420
It would allow us to avail ourselves of the use

0:12:14.420,0:12:16.452
of autonomous vehicles and drones

0:12:16.452,0:12:20.747
while still preserving our open, civil society.

0:12:20.747,0:12:23.746
We must ban the deployment and development

0:12:23.746,0:12:25.608
of killer robots.

0:12:25.608,0:12:30.458
Let's not succumb to the temptation to automate war.

0:12:30.458,0:12:33.176
Autocratic governments and criminal organizations

0:12:33.176,0:12:36.132
undoubtedly will, but let's not join them.

0:12:36.132,0:12:38.023
Autonomous robotic weapons

0:12:38.023,0:12:40.074
would concentrate too much power

0:12:40.074,0:12:42.556
in too few unseen hands,

0:12:42.556,0:12:45.811
and that would be corrosive to representative government.

0:12:45.811,0:12:48.772
Let's make sure, for democracies at least,

0:12:48.772,0:12:51.376
killer robots remain fiction.

0:12:51.376,0:12:52.486
Thank you.

0:12:52.486,0:12:57.051
(Applause)

0:12:57.051,0:13:01.667
Thank you. (Applause)