The kill decision shouldn't belong to a robot

As a novelist, Daniel Suarez spins dystopian tales of the future. But on the TEDGlobal stage, he talks us through a real-life scenario we all need to know more about: the rise of autonomous robotic weapons of war. Advanced drones, automated weapons and AI-powered intelligence-gathering tools, he suggests, could take the decision to make war out of the hands of humans.

I write fiction sci-fi thrillers,

so if I say "killer robots,"

you'd probably think something like this.

But I'm actually not here to talk about fiction.

I'm here to talk about very real killer robots,

autonomous combat drones.

Now, I'm not referring to Predator and Reaper drones,

which have a human making targeting decisions.

I'm talking about fully autonomous robotic weapons

that make lethal decisions about human beings

all on their own.

There's actually a technical term for this: lethal autonomy.

Now, lethally autonomous killer robots

would take many forms -- flying, driving,

or just lying in wait.

And actually, they're very quickly becoming a reality.

These are two automatic sniper stations

currently deployed in the DMZ between North and South Korea.

Both of these machines are capable of automatically

identifying a human target and firing on it,

the one on the left at a distance of over a kilometer.

Now, in both cases, there's still a human in the loop

to make that lethal firing decision,

but it's not a technological requirement. It's a choice.

And it's that choice that I want to focus on,

because as we migrate lethal decision-making

from humans to software,

we risk not only taking the humanity out of war,

but also changing our social landscape entirely,

far from the battlefield.

That's because the way humans resolve conflict

shapes our social landscape.

And this has always been the case, throughout history.

For example, these were state-of-the-art weapons systems

in 1400 A.D.

Now they were both very expensive to build and maintain,

but with these you could dominate the populace,

and the distribution of political power in feudal society reflected that.

Power was focused at the very top.

And what changed? Technological innovation.

Gunpowder, cannon.

And pretty soon, armor and castles were obsolete,

and it mattered less who you brought to the battlefield

versus how many people you brought to the battlefield.

And as armies grew in size, the nation-state arose

as a political and logistical requirement of defense.

And as leaders had to rely on more of their populace,

they began to share power.

Representative government began to form.

So again, the tools we use to resolve conflict

shape our social landscape.

Autonomous robotic weapons are such a tool,

except that, by requiring very few people to go to war,

they risk re-centralizing power into very few hands,

possibly reversing a five-century trend toward democracy.

Now, I think, knowing this,

we can take decisive steps to preserve our democratic institutions,

to do what humans do best, which is adapt.

But time is a factor.

Seventy nations are developing remotely-piloted

combat drones of their own,

and as you'll see, remotely-piloted combat drones

are the precursors to autonomous robotic weapons.

That's because once you've deployed remotely-piloted drones,

there are three powerful factors pushing decision-making

away from humans and on to the weapon platform itself.

The first of these is the deluge of video that drones produce.

For example, in 2004, the U.S. drone fleet produced

a grand total of 71 hours of video surveillance for analysis.

By 2011, this had gone up to 300,000 hours,

outstripping human ability to review it all,

but even that number is about to go up drastically.

The Pentagon's Gorgon Stare and Argus programs

will put up to 65 independently operated camera eyes

on each drone platform,

and this would vastly outstrip human ability to review it.

And that means visual intelligence software will need

to scan it for items of interest.

And that means very soon

drones will tell humans what to look at,

not the other way around.

But there's a second powerful incentive pushing

decision-making away from humans and onto machines,

and that's electromagnetic jamming,

severing the connection between the drone

and its operator.

Now we saw an example of this in 2011

when an American RQ-170 Sentinel drone

got a bit confused over Iran due to a GPS spoofing attack,

but any remotely-piloted drone is susceptible to this type of attack,

and that means drones

will have to shoulder more decision-making.

They'll know their mission objective,

and they'll react to new circumstances without human guidance.

They'll ignore external radio signals

and send very few of their own.

Which brings us to, really, the third

and most powerful incentive pushing decision-making

away from humans and onto weapons:

plausible deniability.

Now we live in a global economy.

High-tech manufacturing is occurring on most continents.

Cyber espionage is spiriting away advanced designs

to parts unknown,

and in that environment, it is very likely

that a successful drone design will be knocked off in contract factories,

proliferate in the gray market.

And in that situation, sifting through the wreckage

of a suicide drone attack, it will be very difficult to say

who sent that weapon.

This raises the very real possibility

of anonymous war.

This could tilt the geopolitical balance on its head,

make it very difficult for a nation to turn its firepower

against an attacker, and that could shift the balance

in the 21st century away from defense and toward offense.

It could make military action a viable option

not just for small nations,

but criminal organizations, private enterprise,

even powerful individuals.

It could create a landscape of rival warlords

undermining rule of law and civil society.

Now if responsibility and transparency

are two of the cornerstones of representative government,

autonomous robotic weapons could undermine both.

Now you might be thinking that

citizens of high-tech nations

would have the advantage in any robotic war,

that citizens of those nations would be less vulnerable,

particularly against developing nations.

But I think the truth is the exact opposite.

I think citizens of high-tech societies

are more vulnerable to robotic weapons,

and the reason can be summed up in one word: data.

Data powers high-tech societies.

Cell phone geolocation, telecom metadata,

social media, email, text, financial transaction data,

transportation data, it's a wealth of real-time data

on the movements and social interactions of people.

In short, we are more visible to machines

than any people in history,

and this perfectly suits the targeting needs of autonomous weapons.

What you're looking at here

is a link analysis map of a social group.

Lines indicate social connectedness between individuals.

And these types of maps can be automatically generated

based on the data trail modern people leave behind.

Now it's typically used to market goods and services

to targeted demographics, but it's a dual-use technology,

because targeting is used in another context.

Notice that certain individuals are highlighted.

These are the hubs of social networks.

These are organizers, opinion-makers, leaders,

and these people also can be automatically identified

from their communication patterns.

Now, if you're a marketer, you might then target them

with product samples, try to spread your brand

through their social group.

But if you're a repressive government

searching for political enemies, you might instead remove them,

eliminate them, disrupt their social group,

and those who remain behind lose social cohesion

and organization.

Now in a world of cheap, proliferating robotic weapons,

borders would offer very little protection

to critics of distant governments

or trans-national criminal organizations.

Popular movements agitating for change

could be detected early and their leaders eliminated

before their ideas achieve critical mass.

And ideas achieving critical mass

is what political activism in popular government is all about.

Anonymous lethal weapons could make lethal action

an easy choice for all sorts of competing interests.

And this would put a chill on free speech

and popular political action, the very heart of democracy.

And this is why we need an international treaty

on robotic weapons, and in particular a global ban

on the development and deployment of killer robots.

Now we already have international treaties

on nuclear and biological weapons, and, while imperfect,

these have largely worked.

But robotic weapons might be every bit as dangerous,

because they will almost certainly be used,

and they would also be corrosive to our democratic institutions.

Now in November 2012 the U.S. Department of Defense

issued a directive requiring

a human being be present in all lethal decisions.

This temporarily effectively banned autonomous weapons in the U.S. military,

but that directive needs to be made permanent.

And it could set the stage for global action.

Because we need an international legal framework

for robotic weapons.

And we need it now, before there's a devastating attack

or a terrorist incident that causes nations of the world

to rush to adopt these weapons

before thinking through the consequences.

Autonomous robotic weapons concentrate too much power

in too few hands, and they would imperil democracy itself.

Now, don't get me wrong, I think there are tons

of great uses for unarmed civilian drones:

environmental monitoring, search and rescue, logistics.

If we have an international treaty on robotic weapons,

how do we gain the benefits of autonomous drones

and vehicles while still protecting ourselves

against illegal robotic weapons?

I think the secret will be transparency.

No robot should have an expectation of privacy

in a public place.

(Applause)

Each robot and drone should have

a cryptographically signed I.D. burned in at the factory

that can be used to track its movement through public spaces.

We have license plates on cars, tail numbers on aircraft.

This is no different.

And every citizen should be able to download an app

that shows the population of drones and autonomous vehicles

moving through public spaces around them,

both right now and historically.

And civic leaders should deploy sensors and civic drones

to detect rogue drones,

and instead of sending killer drones of their own up to shoot them down,

they should notify humans to their presence.

And in certain very high-security areas,

perhaps civic drones would snare them

and drag them off to a bomb disposal facility.

But notice, this is more an immune system

than a weapons system.

It would allow us to avail ourselves of the use

of autonomous vehicles and drones

while still preserving our open, civil society.

We must ban the deployment and development

of killer robots.

Let's not succumb to the temptation to automate war.

Autocratic governments and criminal organizations

undoubtedly will, but let's not join them.

Autonomous robotic weapons

would concentrate too much power

in too few unseen hands,

and that would be corrosive to representative government.

Let's make sure, for democracies at least,

killer robots remain fiction.

Thank you.

(Applause)

Thank you. (Applause)