[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.74,0:00:04.88,Default,,0000,0000,0000,,So I started my first job\Nas a computer programmer
Dialogue: 0,0:00:04.88,0:00:07.06,Default,,0000,0000,0000,,in my very first year of college,
Dialogue: 0,0:00:07.06,0:00:08.89,Default,,0000,0000,0000,,basically as a teenager.
Dialogue: 0,0:00:08.89,0:00:10.85,Default,,0000,0000,0000,,Soon after I started working,
Dialogue: 0,0:00:10.85,0:00:12.89,Default,,0000,0000,0000,,writing software in a company,
Dialogue: 0,0:00:12.89,0:00:16.63,Default,,0000,0000,0000,,a manager who worked at the company\Ncame down to where I was,
Dialogue: 0,0:00:16.63,0:00:18.23,Default,,0000,0000,0000,,and he whispered to me,
Dialogue: 0,0:00:18.23,0:00:21.57,Default,,0000,0000,0000,,"Can he tell if I'm lying?"
Dialogue: 0,0:00:22.06,0:00:25.24,Default,,0000,0000,0000,,There was nobody else in the room.
Dialogue: 0,0:00:25.24,0:00:30.50,Default,,0000,0000,0000,,"Can who tell if you're lying,\Nand why are we whispering?"
Dialogue: 0,0:00:30.50,0:00:33.32,Default,,0000,0000,0000,,The manager pointed\Nat the computer in the room.
Dialogue: 0,0:00:33.32,0:00:37.27,Default,,0000,0000,0000,,"Can he tell if I'm lying?"
Dialogue: 0,0:00:37.83,0:00:43.22,Default,,0000,0000,0000,,Well, that manager was having\Nan affair with the receptionist,
Dialogue: 0,0:00:43.22,0:00:45.62,Default,,0000,0000,0000,,and I was still a teenager,
Dialogue: 0,0:00:45.62,0:00:47.66,Default,,0000,0000,0000,,so I whisper-shouted back to him,
Dialogue: 0,0:00:47.66,0:00:51.53,Default,,0000,0000,0000,,"Yes, the computer can tell\Nif you're lying."
Dialogue: 0,0:00:51.53,0:00:53.10,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:00:53.10,0:00:56.52,Default,,0000,0000,0000,,Well, I laughed, but actually\Nthe laugh's on me.
Dialogue: 0,0:00:56.52,0:01:01.60,Default,,0000,0000,0000,,Nowadays, there are computational systems\Nthat can suss out emotional states,
Dialogue: 0,0:01:01.60,0:01:02.94,Default,,0000,0000,0000,,and even lying,
Dialogue: 0,0:01:02.94,0:01:05.29,Default,,0000,0000,0000,,from processing human faces.
Dialogue: 0,0:01:05.29,0:01:10.46,Default,,0000,0000,0000,,Advertisers and even governments\Nare very interested.
Dialogue: 0,0:01:10.46,0:01:13.31,Default,,0000,0000,0000,,I had become a computer programmer\Nbecause I was one of those kids
Dialogue: 0,0:01:13.31,0:01:15.61,Default,,0000,0000,0000,,crazy about math and science,
Dialogue: 0,0:01:15.61,0:01:19.07,Default,,0000,0000,0000,,but somewhere along the line\NI'd learned about nuclear weapons,
Dialogue: 0,0:01:19.07,0:01:22.05,Default,,0000,0000,0000,,and I'd gotten really concerned\Nwith the ethics of science.
Dialogue: 0,0:01:22.05,0:01:23.46,Default,,0000,0000,0000,,I was troubled.
Dialogue: 0,0:01:23.46,0:01:26.13,Default,,0000,0000,0000,,However, because of family circumstances,
Dialogue: 0,0:01:26.13,0:01:29.48,Default,,0000,0000,0000,,I also needed to start working\Nas soon as possible.
Dialogue: 0,0:01:29.48,0:01:32.80,Default,,0000,0000,0000,,So I thought to myself, hey,\Nlet me pick a technical field
Dialogue: 0,0:01:32.80,0:01:34.53,Default,,0000,0000,0000,,where I can get a job easily
Dialogue: 0,0:01:34.53,0:01:39.23,Default,,0000,0000,0000,,and where I don't have to deal\Nwith any troublesome questions of ethics.
Dialogue: 0,0:01:39.23,0:01:41.91,Default,,0000,0000,0000,,So I picked computers.
Dialogue: 0,0:01:41.91,0:01:45.22,Default,,0000,0000,0000,,Well ha ha ha, all the laughs are on me.
Dialogue: 0,0:01:45.22,0:01:49.25,Default,,0000,0000,0000,,Nowadays, computer scientists\Nare building platforms that control
Dialogue: 0,0:01:49.25,0:01:52.81,Default,,0000,0000,0000,,what a billion people see every day.
Dialogue: 0,0:01:52.81,0:01:57.62,Default,,0000,0000,0000,,They're developing cars that\Ncould decide who to run over.
Dialogue: 0,0:01:57.62,0:02:03.25,Default,,0000,0000,0000,,They're even building machines, weapons,\Nthat might kill human beings in war.
Dialogue: 0,0:02:03.25,0:02:06.77,Default,,0000,0000,0000,,It's ethics all the way down.
Dialogue: 0,0:02:06.77,0:02:09.90,Default,,0000,0000,0000,,Machine intelligence is here.
Dialogue: 0,0:02:09.90,0:02:13.45,Default,,0000,0000,0000,,We're now using computation\Nto make all sort of decisions,
Dialogue: 0,0:02:13.45,0:02:15.36,Default,,0000,0000,0000,,but also new kinds of decisions.
Dialogue: 0,0:02:15.36,0:02:20.55,Default,,0000,0000,0000,,We're asking questions to computation\Nthat have no single right answers,
Dialogue: 0,0:02:20.55,0:02:24.40,Default,,0000,0000,0000,,that are subjective and open-ended\Nand value-laden.
Dialogue: 0,0:02:24.40,0:02:25.93,Default,,0000,0000,0000,,We're asking questions like,
Dialogue: 0,0:02:25.93,0:02:28.33,Default,,0000,0000,0000,,"Who should the company hire?"
Dialogue: 0,0:02:28.33,0:02:31.09,Default,,0000,0000,0000,,"Which update from which friend\Nshould you be shown?"
Dialogue: 0,0:02:31.09,0:02:33.72,Default,,0000,0000,0000,,"Which convict is more\Nlikely to re-offend?"
Dialogue: 0,0:02:33.72,0:02:36.78,Default,,0000,0000,0000,,"Which news item or movie\Nshould be recommended to people?"
Dialogue: 0,0:02:36.78,0:02:39.79,Default,,0000,0000,0000,,Look, yes, we've been using\Ncomputers for a while,
Dialogue: 0,0:02:39.79,0:02:41.94,Default,,0000,0000,0000,,but this is different.
Dialogue: 0,0:02:41.94,0:02:43.68,Default,,0000,0000,0000,,This is a historical twist,
Dialogue: 0,0:02:43.68,0:02:49.04,Default,,0000,0000,0000,,because we cannot anchor\Ncomputation for such subjective decisions
Dialogue: 0,0:02:49.04,0:02:51.53,Default,,0000,0000,0000,,the way we can anchor computation
Dialogue: 0,0:02:51.53,0:02:56.35,Default,,0000,0000,0000,,for flying airplanes, building bridges,\Ngoing to the moon.
Dialogue: 0,0:02:56.35,0:02:59.93,Default,,0000,0000,0000,,Are airplanes safer?\NDid the bridge sway and fall?
Dialogue: 0,0:02:59.93,0:03:04.45,Default,,0000,0000,0000,,There we have agreed-upon,\Nfairly clear benchmarks,
Dialogue: 0,0:03:04.45,0:03:06.56,Default,,0000,0000,0000,,and we have laws of nature to guide us.
Dialogue: 0,0:03:06.56,0:03:09.71,Default,,0000,0000,0000,,We have no such anchors and benchmarks
Dialogue: 0,0:03:09.71,0:03:14.66,Default,,0000,0000,0000,,for decisions in messy human affairs.
Dialogue: 0,0:03:14.66,0:03:18.18,Default,,0000,0000,0000,,To make things more complicated,\Nour software is getting more powerful,
Dialogue: 0,0:03:18.18,0:03:22.54,Default,,0000,0000,0000,,but it's also getting less transparent\Nand more complex.
Dialogue: 0,0:03:22.54,0:03:24.74,Default,,0000,0000,0000,,Recently, in the past decade,
Dialogue: 0,0:03:24.74,0:03:27.52,Default,,0000,0000,0000,,complex algorithms have made\Ngreat strides.
Dialogue: 0,0:03:27.52,0:03:29.68,Default,,0000,0000,0000,,They can recognize human faces.
Dialogue: 0,0:03:29.68,0:03:32.67,Default,,0000,0000,0000,,They can decipher handwriting.
Dialogue: 0,0:03:32.67,0:03:35.68,Default,,0000,0000,0000,,They can detect credit card fraud\Nand block spam
Dialogue: 0,0:03:35.68,0:03:37.85,Default,,0000,0000,0000,,and they can translate between languages.
Dialogue: 0,0:03:37.85,0:03:40.44,Default,,0000,0000,0000,,They can detect tumors in medical imaging.
Dialogue: 0,0:03:40.44,0:03:43.50,Default,,0000,0000,0000,,They can beat humans in chess and go.
Dialogue: 0,0:03:43.50,0:03:48.32,Default,,0000,0000,0000,,Much of this progress comes from\Na method called machine learning.
Dialogue: 0,0:03:48.32,0:03:51.53,Default,,0000,0000,0000,,Machine learning is different\Nthan traditional programming,
Dialogue: 0,0:03:51.53,0:03:55.38,Default,,0000,0000,0000,,where you give the computer\Ndetailed, exact, painstaking instructions.
Dialogue: 0,0:03:55.38,0:03:59.58,Default,,0000,0000,0000,,It's more like you take the system\Nand you feed it lots of data,
Dialogue: 0,0:03:59.58,0:04:01.26,Default,,0000,0000,0000,,including unstructured data,
Dialogue: 0,0:04:01.26,0:04:03.57,Default,,0000,0000,0000,,like the kind we generate\Nin our digital lives.
Dialogue: 0,0:04:03.57,0:04:06.73,Default,,0000,0000,0000,,And the system learns\Nby churning through this data.
Dialogue: 0,0:04:06.73,0:04:12.89,Default,,0000,0000,0000,,And also crucially, these systems\Ndon't operate under a single answer logic.
Dialogue: 0,0:04:12.89,0:04:14.53,Default,,0000,0000,0000,,They don't produce a simple answer.
Dialogue: 0,0:04:14.53,0:04:15.70,Default,,0000,0000,0000,,It's more probabilistic.
Dialogue: 0,0:04:15.70,0:04:20.13,Default,,0000,0000,0000,,This one is probably more\Nlike what you're looking for.
Dialogue: 0,0:04:20.13,0:04:23.15,Default,,0000,0000,0000,,Now the upside is, this method\Nis really powerful.
Dialogue: 0,0:04:23.15,0:04:27.95,Default,,0000,0000,0000,,The head of Google's AI systems called it\Nthe unreasonable effectiveness of data.
Dialogue: 0,0:04:27.95,0:04:33.08,Default,,0000,0000,0000,,The downside is we don't really\Nunderstand what the system learned.
Dialogue: 0,0:04:33.08,0:04:35.12,Default,,0000,0000,0000,,In fact, that's its power.
Dialogue: 0,0:04:35.12,0:04:39.31,Default,,0000,0000,0000,,This is less like giving instructions\Nto a computer.
Dialogue: 0,0:04:39.31,0:04:43.62,Default,,0000,0000,0000,,It's more like training a puppy\Nmachine creature
Dialogue: 0,0:04:43.62,0:04:46.46,Default,,0000,0000,0000,,we don't really understand or control.
Dialogue: 0,0:04:46.46,0:04:48.60,Default,,0000,0000,0000,,So this is our problem.
Dialogue: 0,0:04:48.60,0:04:52.71,Default,,0000,0000,0000,,It's a problem when this artificial\Nintelligence system gets things wrong.
Dialogue: 0,0:04:52.71,0:04:54.24,Default,,0000,0000,0000,,It's also a problem
Dialogue: 0,0:04:54.24,0:04:56.48,Default,,0000,0000,0000,,when it gets things right,
Dialogue: 0,0:04:56.48,0:05:00.13,Default,,0000,0000,0000,,because we don't even know which is which\Nwhen it's a subjective problem.
Dialogue: 0,0:05:00.13,0:05:03.58,Default,,0000,0000,0000,,We don't know what this thing is thinking.
Dialogue: 0,0:05:03.58,0:05:08.19,Default,,0000,0000,0000,,So consider a hiring algorithm,
Dialogue: 0,0:05:08.19,0:05:13.10,Default,,0000,0000,0000,,a system used to hire people\Nusing machine learning systems.
Dialogue: 0,0:05:13.10,0:05:16.69,Default,,0000,0000,0000,,Such a system would have been trained\Non previous employees' data
Dialogue: 0,0:05:16.69,0:05:20.68,Default,,0000,0000,0000,,an instructed to find and hire\Npeople like the existing
Dialogue: 0,0:05:20.68,0:05:23.06,Default,,0000,0000,0000,,high performers in the company.
Dialogue: 0,0:05:23.06,0:05:24.18,Default,,0000,0000,0000,,Sounds good.
Dialogue: 0,0:05:24.18,0:05:27.01,Default,,0000,0000,0000,,I once attended a conference\Nthat brought together
Dialogue: 0,0:05:27.01,0:05:30.33,Default,,0000,0000,0000,,human resources, managers,\Nand executives, high-level people,
Dialogue: 0,0:05:30.33,0:05:32.03,Default,,0000,0000,0000,,using such systems in hiring.
Dialogue: 0,0:05:32.03,0:05:33.89,Default,,0000,0000,0000,,They were super-excited.
Dialogue: 0,0:05:33.89,0:05:38.56,Default,,0000,0000,0000,,They thought that this would make hiring\Nmore objective, less biased,
Dialogue: 0,0:05:38.56,0:05:41.51,Default,,0000,0000,0000,,and give women\Nand minorities a better shot
Dialogue: 0,0:05:41.51,0:05:43.87,Default,,0000,0000,0000,,against biased human managers.
Dialogue: 0,0:05:43.87,0:05:47.35,Default,,0000,0000,0000,,And look, human hiring is biased.
Dialogue: 0,0:05:47.35,0:05:48.44,Default,,0000,0000,0000,,I know.
Dialogue: 0,0:05:48.44,0:05:51.46,Default,,0000,0000,0000,,In one of my early jobs as a programmer,
Dialogue: 0,0:05:51.46,0:05:55.70,Default,,0000,0000,0000,,my immediate manager would sometimes\Ncome down to where I was
Dialogue: 0,0:05:55.70,0:05:57.17,Default,,0000,0000,0000,,really early in the morning
Dialogue: 0,0:05:57.17,0:05:59.11,Default,,0000,0000,0000,,or really late in the afternoon,
Dialogue: 0,0:05:59.11,0:06:02.92,Default,,0000,0000,0000,,and she'd say, "Zeynep,\Nlet's go to lunch!"
Dialogue: 0,0:06:02.92,0:06:05.11,Default,,0000,0000,0000,,I'd be puzzled by the weird timing.
Dialogue: 0,0:06:05.11,0:06:07.35,Default,,0000,0000,0000,,It's 4 pm? Lunch?
Dialogue: 0,0:06:07.35,0:06:10.74,Default,,0000,0000,0000,,I was broke, so free lunch. I always went.
Dialogue: 0,0:06:10.74,0:06:12.71,Default,,0000,0000,0000,,I later realized what was happening.
Dialogue: 0,0:06:12.71,0:06:17.28,Default,,0000,0000,0000,,My immediate managers\Nhad not confessed to their higher-ups
Dialogue: 0,0:06:17.28,0:06:20.32,Default,,0000,0000,0000,,that the programmer they hired\Nfor a serious job was a teen girl
Dialogue: 0,0:06:20.32,0:06:25.37,Default,,0000,0000,0000,,who wore jeans and sneakers to work.
Dialogue: 0,0:06:25.37,0:06:28.46,Default,,0000,0000,0000,,I was doing a good job.
Dialogue: 0,0:06:28.46,0:06:29.34,Default,,0000,0000,0000,,I just looked wrong and was\Nthe wrong age and gender.
Dialogue: 0,0:06:29.34,0:06:32.42,Default,,0000,0000,0000,,So hiring in a gender- and race-blind way
Dialogue: 0,0:06:32.42,0:06:35.12,Default,,0000,0000,0000,,certainly sounds good to me.
Dialogue: 0,0:06:35.12,0:06:36.63,Default,,0000,0000,0000,,But with these systems,\Nit is more complicated, and here's why.
Dialogue: 0,0:06:36.63,0:06:44.78,Default,,0000,0000,0000,,Currently, computational systems\Ncan infer all sorts of things about you
Dialogue: 0,0:06:44.78,0:06:46.86,Default,,0000,0000,0000,,from your digital crumbs,
Dialogue: 0,0:06:46.86,0:06:49.69,Default,,0000,0000,0000,,even if you have not\Ndisclosed those things.
Dialogue: 0,0:06:49.69,0:06:53.18,Default,,0000,0000,0000,,They can infer your sexual orientation,
Dialogue: 0,0:06:53.18,0:06:55.17,Default,,0000,0000,0000,,your personality traits,
Dialogue: 0,0:06:55.17,0:06:57.10,Default,,0000,0000,0000,,your political leanings.
Dialogue: 0,0:06:57.10,0:07:01.64,Default,,0000,0000,0000,,They have predictive power\Nwith high levels of accuracy,
Dialogue: 0,0:07:01.64,0:07:04.24,Default,,0000,0000,0000,,remember for things\Nyou haven't even disclosed.
Dialogue: 0,0:07:04.24,0:07:05.79,Default,,0000,0000,0000,,This is inference.
Dialogue: 0,0:07:05.79,0:07:08.91,Default,,0000,0000,0000,,I have a friend who developed\Nsuch computational systems
Dialogue: 0,0:07:08.91,0:07:12.63,Default,,0000,0000,0000,,to predict the likelihood of clinical\Nor post-partum depression
Dialogue: 0,0:07:12.63,0:07:14.81,Default,,0000,0000,0000,,from social media data.
Dialogue: 0,0:07:14.81,0:07:16.71,Default,,0000,0000,0000,,The results were impressive.
Dialogue: 0,0:07:16.71,0:07:19.84,Default,,0000,0000,0000,,Her system can predict\Nthe likelihood of depression
Dialogue: 0,0:07:19.84,0:07:24.10,Default,,0000,0000,0000,,months before the onset of any symptoms,
Dialogue: 0,0:07:24.10,0:07:25.50,Default,,0000,0000,0000,,months before.
Dialogue: 0,0:07:25.50,0:07:27.77,Default,,0000,0000,0000,,No symptoms, there's prediction.
Dialogue: 0,0:07:27.77,0:07:32.11,Default,,0000,0000,0000,,She hopes it will be used\Nfor early intervention.
Dialogue: 0,0:07:32.11,0:07:34.15,Default,,0000,0000,0000,,Great.
Dialogue: 0,0:07:34.15,0:07:36.21,Default,,0000,0000,0000,,But now put this in the context of hiring.
Dialogue: 0,0:07:36.21,0:07:39.28,Default,,0000,0000,0000,,So at this human resources\Nmanager's conference,
Dialogue: 0,0:07:39.28,0:07:42.17,Default,,0000,0000,0000,,I approached a high-level manager
Dialogue: 0,0:07:42.17,0:07:44.11,Default,,0000,0000,0000,,in a very large company,
Dialogue: 0,0:07:44.11,0:07:46.07,Default,,0000,0000,0000,,and I said to her, "Look,\Nwhat if, unbeknownst to you,
Dialogue: 0,0:07:46.07,0:07:55.90,Default,,0000,0000,0000,,your system is weeding out people with\Nhigh future likelihood of depression?
Dialogue: 0,0:07:55.90,0:07:58.75,Default,,0000,0000,0000,,They're not depressed now,\Njust maybe in the future, more likely.
Dialogue: 0,0:07:58.75,0:08:03.48,Default,,0000,0000,0000,,What if it's weeding out women\Nmore likely to be pregnant
Dialogue: 0,0:08:03.48,0:08:07.04,Default,,0000,0000,0000,,in the next year or two\Nbut aren't pregnant now?
Dialogue: 0,0:08:07.04,0:08:13.35,Default,,0000,0000,0000,,What if it's hiring aggressive people\Nbecause that's your workplace culture?"
Dialogue: 0,0:08:13.35,0:08:16.06,Default,,0000,0000,0000,,You can't tell this by looking\Nat gender breakdowns.
Dialogue: 0,0:08:16.06,0:08:17.54,Default,,0000,0000,0000,,Those may be balanced.
Dialogue: 0,0:08:17.54,0:08:19.84,Default,,0000,0000,0000,,And since this is machine learning,\Nnot traditional coding,
Dialogue: 0,0:08:19.84,0:08:23.13,Default,,0000,0000,0000,,there is no variable there
Dialogue: 0,0:08:23.13,0:08:26.06,Default,,0000,0000,0000,,labeled "higher risk of depression,"
Dialogue: 0,0:08:26.06,0:08:27.98,Default,,0000,0000,0000,,"higher risk of pregnancy,"
Dialogue: 0,0:08:27.98,0:08:30.20,Default,,0000,0000,0000,,"aggressive guy scale."
Dialogue: 0,0:08:30.20,0:08:33.79,Default,,0000,0000,0000,,Not only do you not know\Nwhat your system is selecting on,
Dialogue: 0,0:08:33.79,0:08:36.39,Default,,0000,0000,0000,,you don't even know\Nwhere to begin to look.
Dialogue: 0,0:08:36.39,0:08:37.60,Default,,0000,0000,0000,,It's a black box.
Dialogue: 0,0:08:37.60,0:08:40.63,Default,,0000,0000,0000,,It has predictive power,\Nbut you don't understand it.
Dialogue: 0,0:08:40.63,0:08:45.10,Default,,0000,0000,0000,,"What safeguards," I asked, "do you have\Nto make sure that your black box
Dialogue: 0,0:08:45.10,0:08:47.19,Default,,0000,0000,0000,,isn't doing something shady?"
Dialogue: 0,0:08:47.19,0:08:50.25,Default,,0000,0000,0000,,So she looked at me
Dialogue: 0,0:08:50.25,0:08:54.01,Default,,0000,0000,0000,,as if I had just stepped\Non 10 puppy tails.
Dialogue: 0,0:08:54.01,0:08:56.68,Default,,0000,0000,0000,,She stared at me and she said,
Dialogue: 0,0:08:56.68,0:09:01.73,Default,,0000,0000,0000,,"I don't want to hear\Nanother word about this."
Dialogue: 0,0:09:01.73,0:09:04.29,Default,,0000,0000,0000,,And she turned around and walked away.
Dialogue: 0,0:09:04.29,0:09:07.11,Default,,0000,0000,0000,,Mind you, she wasn't rude.
Dialogue: 0,0:09:07.11,0:09:13.86,Default,,0000,0000,0000,,It was clearly what I don't know\Nisn't my problem, go away, death stare.
Dialogue: 0,0:09:13.86,0:09:20.05,Default,,0000,0000,0000,,Look, such a system may even be less\Nbiased than human managers in some ways,
Dialogue: 0,0:09:20.05,0:09:22.68,Default,,0000,0000,0000,,and it could make monetary sense,
Dialogue: 0,0:09:22.68,0:09:29.14,Default,,0000,0000,0000,,but it could also lead to a steady but\Nstealthy shutting out of the job market
Dialogue: 0,0:09:29.14,0:09:31.95,Default,,0000,0000,0000,,of people with higher risk of depression.
Dialogue: 0,0:09:31.95,0:09:34.53,Default,,0000,0000,0000,,Is this the kind of society\Nwe want to build
Dialogue: 0,0:09:34.53,0:09:36.84,Default,,0000,0000,0000,,without even knowing we've done this
Dialogue: 0,0:09:36.84,0:09:41.54,Default,,0000,0000,0000,,because we turned decision-making\Nto machines we don't totally understand?
Dialogue: 0,0:09:41.54,0:09:43.41,Default,,0000,0000,0000,,Another problem is this:
Dialogue: 0,0:09:43.41,0:09:47.98,Default,,0000,0000,0000,,these systems are often trained\Non data generated by our actions,
Dialogue: 0,0:09:47.98,0:09:50.30,Default,,0000,0000,0000,,human imprints.
Dialogue: 0,0:09:50.30,0:09:54.26,Default,,0000,0000,0000,,Well, they could just be\Nreflecting our biases,
Dialogue: 0,0:09:54.26,0:09:57.88,Default,,0000,0000,0000,,and these systems could be\Npicking up on our biases
Dialogue: 0,0:09:57.88,0:10:00.51,Default,,0000,0000,0000,,and amplifying them
Dialogue: 0,0:10:00.51,0:10:01.53,Default,,0000,0000,0000,,and showing them back to us,
Dialogue: 0,0:10:01.53,0:10:06.46,Default,,0000,0000,0000,,while we're telling ourselves, "We're just\Ndoing objective neutral computation."
Dialogue: 0,0:10:06.46,0:10:09.92,Default,,0000,0000,0000,,Researchers found that on Google,
Dialogue: 0,0:10:09.92,0:10:13.01,Default,,0000,0000,0000,,women are less likely than men
Dialogue: 0,0:10:13.01,0:10:16.36,Default,,0000,0000,0000,,to be shown job ads for high-paying jobs,
Dialogue: 0,0:10:16.36,0:10:19.02,Default,,0000,0000,0000,,and searching for African-American names
Dialogue: 0,0:10:19.02,0:10:24.29,Default,,0000,0000,0000,,is more likely to bring up ads\Nsuggesting criminal history,
Dialogue: 0,0:10:24.29,0:10:26.89,Default,,0000,0000,0000,,even when there is none.
Dialogue: 0,0:10:26.89,0:10:30.46,Default,,0000,0000,0000,,Such hidden biases\Nand black box algorithms
Dialogue: 0,0:10:30.46,0:10:32.74,Default,,0000,0000,0000,,that researchers uncover sometimes
Dialogue: 0,0:10:32.74,0:10:34.41,Default,,0000,0000,0000,,but sometimes we don't know
Dialogue: 0,0:10:34.41,0:10:37.96,Default,,0000,0000,0000,,can have life-altering consequences.
Dialogue: 0,0:10:37.96,0:10:40.89,Default,,0000,0000,0000,,In Wisconsin, a defendant was sentenced
Dialogue: 0,0:10:40.89,0:10:44.17,Default,,0000,0000,0000,,to six years in prison\Nfor evading the police.
Dialogue: 0,0:10:44.17,0:10:47.72,Default,,0000,0000,0000,,You may not know this, but algorithms\Nare increasingly used
Dialogue: 0,0:10:47.72,0:10:50.82,Default,,0000,0000,0000,,in parole and sentencing decisions.
Dialogue: 0,0:10:50.82,0:10:53.80,Default,,0000,0000,0000,,He wanted to know,\Nhow is this score calculated?
Dialogue: 0,0:10:53.80,0:10:55.76,Default,,0000,0000,0000,,It's a commercial black box.
Dialogue: 0,0:10:55.76,0:11:00.68,Default,,0000,0000,0000,,The company refused to have its algorithm\Nbe challenged in open court,
Dialogue: 0,0:11:00.68,0:11:04.20,Default,,0000,0000,0000,,but ProPublica, an investigative nonprofit
Dialogue: 0,0:11:04.20,0:11:08.24,Default,,0000,0000,0000,,audited that very algorithm\Nwith what public data they could find,
Dialogue: 0,0:11:08.24,0:11:10.53,Default,,0000,0000,0000,,and found that its outcomes were biased
Dialogue: 0,0:11:10.53,0:11:13.98,Default,,0000,0000,0000,,and its predictive power was dismal,\Nbarely better than chance,
Dialogue: 0,0:11:13.98,0:11:16.45,Default,,0000,0000,0000,,and it was wrongly labeling\Nblack defendants
Dialogue: 0,0:11:16.45,0:11:18.88,Default,,0000,0000,0000,,as future criminals
Dialogue: 0,0:11:18.88,0:11:21.76,Default,,0000,0000,0000,,at twice the rate of white defendants.
Dialogue: 0,0:11:21.76,0:11:26.10,Default,,0000,0000,0000,,So consider this case.
Dialogue: 0,0:11:26.10,0:11:30.76,Default,,0000,0000,0000,,This woman was late to picking up\Nher godsister from a school
Dialogue: 0,0:11:30.76,0:11:32.53,Default,,0000,0000,0000,,in Broward County, Florida,
Dialogue: 0,0:11:32.53,0:11:35.14,Default,,0000,0000,0000,,running down a street\Nwith a friend of hers,
Dialogue: 0,0:11:35.14,0:11:39.47,Default,,0000,0000,0000,,and they spotted an unlocked\Nkid's bike and a scooter on a porch
Dialogue: 0,0:11:39.47,0:11:41.10,Default,,0000,0000,0000,,and foolishly jumped on it.
Dialogue: 0,0:11:41.10,0:11:43.24,Default,,0000,0000,0000,,As they were speeding off,\Na woman came out and said,
Dialogue: 0,0:11:43.24,0:11:45.83,Default,,0000,0000,0000,,"Hey, that's my kid's bike."
Dialogue: 0,0:11:45.83,0:11:47.92,Default,,0000,0000,0000,,They dropped it, they walked away,
Dialogue: 0,0:11:47.92,0:11:49.24,Default,,0000,0000,0000,,but they were arrested.
Dialogue: 0,0:11:49.24,0:11:53.00,Default,,0000,0000,0000,,She was wrong, she was foolish,\Nbut she was also just 18.
Dialogue: 0,0:11:53.00,0:11:55.91,Default,,0000,0000,0000,,She had a couple of juvenile misdemeanors.
Dialogue: 0,0:11:55.91,0:12:00.29,Default,,0000,0000,0000,,Meanwhile, that man had been arrested\Nfor shoplifting in Home Depot,
Dialogue: 0,0:12:00.29,0:12:04.40,Default,,0000,0000,0000,,85 dollars worth of stuff,\Na similar petty crime,
Dialogue: 0,0:12:04.40,0:12:09.96,Default,,0000,0000,0000,,but he had two prior\Narmed robbery convictions.
Dialogue: 0,0:12:09.96,0:12:14.43,Default,,0000,0000,0000,,But the algorithm scored her\Nas high risk, and not him.
Dialogue: 0,0:12:14.43,0:12:18.84,Default,,0000,0000,0000,,Two years later, ProPublica found\Nthat she had not reoffended.
Dialogue: 0,0:12:18.84,0:12:21.35,Default,,0000,0000,0000,,It was just hard to get a job\Nfor her with her record.
Dialogue: 0,0:12:21.35,0:12:23.44,Default,,0000,0000,0000,,He on the other hand did reoffend
Dialogue: 0,0:12:23.44,0:12:28.24,Default,,0000,0000,0000,,and is now serving an eight-year\Nprison term for a later crime.
Dialogue: 0,0:12:28.24,0:12:31.76,Default,,0000,0000,0000,,Clearly, we need to audit our black boxes
Dialogue: 0,0:12:31.76,0:12:34.44,Default,,0000,0000,0000,,and not have them have\Nthis kind of unchecked power.
Dialogue: 0,0:12:34.44,0:12:37.32,Default,,0000,0000,0000,,(Applause)
Dialogue: 0,0:12:38.02,0:12:40.43,Default,,0000,0000,0000,,Audits are great and important,\Nbut they don't solve all our problems.
Dialogue: 0,0:12:40.43,0:12:42.54,Default,,0000,0000,0000,,Take Facebook's powerful\Nnews feed algorithm,
Dialogue: 0,0:12:42.54,0:12:45.77,Default,,0000,0000,0000,,you know, the one that ranks everything
Dialogue: 0,0:12:45.77,0:12:48.81,Default,,0000,0000,0000,,and decides to show you
Dialogue: 0,0:12:48.81,0:12:52.90,Default,,0000,0000,0000,,what from all the friends\Nand pages that you follow.
Dialogue: 0,0:12:52.90,0:12:56.63,Default,,0000,0000,0000,,Should you be shown\Nanother baby picture?
Dialogue: 0,0:12:56.63,0:12:59.66,Default,,0000,0000,0000,,A sullen note from an acquaintance?
Dialogue: 0,0:12:59.66,0:13:01.86,Default,,0000,0000,0000,,An important but difficult\Nnews item? There's no right answer.
Dialogue: 0,0:13:01.86,0:13:05.87,Default,,0000,0000,0000,,Facebook optimizes\Nfor engagement on the site:
Dialogue: 0,0:13:05.87,0:13:07.43,Default,,0000,0000,0000,,likes, shares, comments.
Dialogue: 0,0:13:07.43,0:13:11.20,Default,,0000,0000,0000,,So in August of 2014,
Dialogue: 0,0:13:11.20,0:13:13.74,Default,,0000,0000,0000,,protests broke out in Ferguson, Missouri,
Dialogue: 0,0:13:13.74,0:13:18.18,Default,,0000,0000,0000,,after the killing of an African-American\Nteenager by a white police officer
Dialogue: 0,0:13:18.18,0:13:20.19,Default,,0000,0000,0000,,under murky circumstances.
Dialogue: 0,0:13:20.19,0:13:22.98,Default,,0000,0000,0000,,The news of the protests\Nwere all over my algorithmically
Dialogue: 0,0:13:22.98,0:13:24.64,Default,,0000,0000,0000,,unfiltered Twitter feed
Dialogue: 0,0:13:24.64,0:13:27.39,Default,,0000,0000,0000,,but nowhere on my Facebook.
Dialogue: 0,0:13:27.39,0:13:29.15,Default,,0000,0000,0000,,Was it my Facebook friends?
Dialogue: 0,0:13:29.15,0:13:31.20,Default,,0000,0000,0000,,I disabled Facebook's algorithm,
Dialogue: 0,0:13:31.20,0:13:34.34,Default,,0000,0000,0000,,which is hard because Facebook\Nkeeps wanting to make you
Dialogue: 0,0:13:34.34,0:13:36.61,Default,,0000,0000,0000,,come under the algorithm's control,
Dialogue: 0,0:13:36.61,0:13:38.42,Default,,0000,0000,0000,,and saw that my friends\Nwere talking about it.
Dialogue: 0,0:13:38.42,0:13:40.51,Default,,0000,0000,0000,,It's just that the algorithm\Nwasn't showing it to me.
Dialogue: 0,0:13:40.51,0:13:43.80,Default,,0000,0000,0000,,I researched this and found\Nthis was a widespread problem.
Dialogue: 0,0:13:43.80,0:13:45.51,Default,,0000,0000,0000,,The story of Ferguson\Nwasn't algorithm-friendly.
Dialogue: 0,0:13:45.51,0:13:49.44,Default,,0000,0000,0000,,It's not likable.
Dialogue: 0,0:13:49.44,0:13:51.18,Default,,0000,0000,0000,,Who's going to click on like?
Dialogue: 0,0:13:51.18,0:13:53.100,Default,,0000,0000,0000,,It's not even easy to comment on.
Dialogue: 0,0:13:53.100,0:13:56.90,Default,,0000,0000,0000,,Without likes and comments,\Nthe algorithm was likely showing it
Dialogue: 0,0:13:56.90,0:13:58.26,Default,,0000,0000,0000,,to even fewer people,
Dialogue: 0,0:13:58.26,0:14:01.01,Default,,0000,0000,0000,,so we didn't get to see this.
Dialogue: 0,0:14:01.01,0:14:03.69,Default,,0000,0000,0000,,Instead, that week, Facebook's\Nalgorithm highlighted this,
Dialogue: 0,0:14:03.69,0:14:06.98,Default,,0000,0000,0000,,which is the ALS ice bucket challenge.
Dialogue: 0,0:14:06.98,0:14:10.74,Default,,0000,0000,0000,,Worthy cause. Dump ice water,\Ndonate to charity, fine.
Dialogue: 0,0:14:10.74,0:14:13.35,Default,,0000,0000,0000,,But it was super-algorithm-friendly.
Dialogue: 0,0:14:13.35,0:14:15.92,Default,,0000,0000,0000,,The machine made this decision for us.
Dialogue: 0,0:14:15.92,0:14:19.60,Default,,0000,0000,0000,,A very important\Nbut difficult conversation
Dialogue: 0,0:14:19.60,0:14:21.41,Default,,0000,0000,0000,,might have been smothered
Dialogue: 0,0:14:21.41,0:14:24.44,Default,,0000,0000,0000,,had Facebook been the only channel.
Dialogue: 0,0:14:24.44,0:14:26.74,Default,,0000,0000,0000,,Now finally, these systems
Dialogue: 0,0:14:26.74,0:14:30.85,Default,,0000,0000,0000,,can also be wrong in ways\Nthat don't resemble human systems.
Dialogue: 0,0:14:30.85,0:14:33.78,Default,,0000,0000,0000,,Do you guys remember Watson,\NIBM's machine intelligence system
Dialogue: 0,0:14:33.78,0:14:37.13,Default,,0000,0000,0000,,that wiped the floor with\Nhuman contestants on Jeopardy?
Dialogue: 0,0:14:37.13,0:14:39.47,Default,,0000,0000,0000,,It was a great player.
Dialogue: 0,0:14:39.47,0:14:42.66,Default,,0000,0000,0000,,But then, for Final Jeopardy,\NWatson was asked this question:
Dialogue: 0,0:14:42.66,0:14:45.62,Default,,0000,0000,0000,,"It's largest airport is named\Nfor a World War II hero,
Dialogue: 0,0:14:45.62,0:14:48.12,Default,,0000,0000,0000,,it's second largest\Nfor a World War II battle."
Dialogue: 0,0:14:48.12,0:14:50.89,Default,,0000,0000,0000,,Chicago.
Dialogue: 0,0:14:50.89,0:14:52.64,Default,,0000,0000,0000,,The two humans got it right.
Dialogue: 0,0:14:52.64,0:14:57.36,Default,,0000,0000,0000,,Watson, on the other hand,\Nanswered "Toronto"
Dialogue: 0,0:14:57.36,0:14:59.74,Default,,0000,0000,0000,,for a US city category.
Dialogue: 0,0:14:59.74,0:15:02.60,Default,,0000,0000,0000,,The impressive system also made an error
Dialogue: 0,0:15:02.60,0:15:06.52,Default,,0000,0000,0000,,that a human would never make,\Na second grader wouldn't make.
Dialogue: 0,0:15:06.52,0:15:11.68,Default,,0000,0000,0000,,Our machine intelligence can fail\Nin ways that don't fit
Dialogue: 0,0:15:11.68,0:15:13.22,Default,,0000,0000,0000,,error patterns of humans,
Dialogue: 0,0:15:13.22,0:15:16.10,Default,,0000,0000,0000,,in ways we won't expect\Nand be prepared for.
Dialogue: 0,0:15:16.10,0:15:19.72,Default,,0000,0000,0000,,It'd be lousy not to get a job\None is qualified for,
Dialogue: 0,0:15:19.72,0:15:23.47,Default,,0000,0000,0000,,but it would triple suck if it was\Nbecause of stack overflow
Dialogue: 0,0:15:23.47,0:15:25.09,Default,,0000,0000,0000,,in some subroutine.
Dialogue: 0,0:15:25.09,0:15:27.02,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:15:27.02,0:15:31.63,Default,,0000,0000,0000,,In May of 2010, a flash crash\Non Wall Street
Dialogue: 0,0:15:31.63,0:15:36.46,Default,,0000,0000,0000,,fueled by a feedback loop\Nin Wall Street's sell algorithm
Dialogue: 0,0:15:36.46,0:15:41.82,Default,,0000,0000,0000,,wiped a trillion dollars of value\Nin 36 minutes.
Dialogue: 0,0:15:41.82,0:15:44.56,Default,,0000,0000,0000,,I don't even want to think\Nwhat error means in the context
Dialogue: 0,0:15:44.56,0:15:47.98,Default,,0000,0000,0000,,of lethal autonomous weapons.
Dialogue: 0,0:15:50.04,0:15:53.79,Default,,0000,0000,0000,,So yes, humans have always made biases.
Dialogue: 0,0:15:53.79,0:15:55.61,Default,,0000,0000,0000,,Decision makers and gatekeepers,
Dialogue: 0,0:15:55.61,0:15:59.76,Default,,0000,0000,0000,,in courts, in news, in war,
Dialogue: 0,0:15:59.76,0:16:02.55,Default,,0000,0000,0000,,they make mistakes,\Nbut that's exactly my point.
Dialogue: 0,0:16:02.55,0:16:06.83,Default,,0000,0000,0000,,We cannot escape\Nthese difficult questions.
Dialogue: 0,0:16:06.83,0:16:10.28,Default,,0000,0000,0000,,We cannot outsource\Nour responsibilities to machines.
Dialogue: 0,0:16:10.28,0:16:14.88,Default,,0000,0000,0000,,(Applause)
Dialogue: 0,0:16:17.09,0:16:22.74,Default,,0000,0000,0000,,Artificial intelligence does not\Nget us a get out of ethics free card.
Dialogue: 0,0:16:22.74,0:16:26.21,Default,,0000,0000,0000,,Data scientist Fred Benenson\Ncalls this math-washing.
Dialogue: 0,0:16:26.21,0:16:27.69,Default,,0000,0000,0000,,We need the opposite.
Dialogue: 0,0:16:27.69,0:16:33.38,Default,,0000,0000,0000,,We need to cultivate algorithm suspicion,\Nscrutiny, and investigation.
Dialogue: 0,0:16:33.38,0:16:36.87,Default,,0000,0000,0000,,We need to make sure we have\Nalgorithmic accountability,
Dialogue: 0,0:16:36.87,0:16:39.43,Default,,0000,0000,0000,,auditing, and meaningful transparency.
Dialogue: 0,0:16:39.43,0:16:42.64,Default,,0000,0000,0000,,We need to accept\Nthat bringing math and computation
Dialogue: 0,0:16:42.64,0:16:45.76,Default,,0000,0000,0000,,to messy, value-laden human affairs
Dialogue: 0,0:16:45.76,0:16:48.26,Default,,0000,0000,0000,,does not bring objectivity.
Dialogue: 0,0:16:48.26,0:16:50.40,Default,,0000,0000,0000,,Rather, the complexity of human affairs\Ninvades the algorithms.
Dialogue: 0,0:16:50.40,0:16:57.80,Default,,0000,0000,0000,,Yes, we can and we should use computation\Nto help us make better decisions,
Dialogue: 0,0:16:57.80,0:17:03.12,Default,,0000,0000,0000,,but we have to own up\Nto our moral responsibility to judgment
Dialogue: 0,0:17:03.12,0:17:06.04,Default,,0000,0000,0000,,and use algorithms within that framework,
Dialogue: 0,0:17:06.04,0:17:10.85,Default,,0000,0000,0000,,not as a means to abdicate\Nand outsource our responsibilities
Dialogue: 0,0:17:10.85,0:17:14.03,Default,,0000,0000,0000,,to one another as human to human.
Dialogue: 0,0:17:14.03,0:17:16.59,Default,,0000,0000,0000,,Machine intelligence is here.
Dialogue: 0,0:17:16.59,0:17:20.08,Default,,0000,0000,0000,,That means we must hold on\Never tighter to human values
Dialogue: 0,0:17:20.08,0:17:22.06,Default,,0000,0000,0000,,and human ethics.
Dialogue: 0,0:17:22.06,0:17:23.89,Default,,0000,0000,0000,,Thank you.
Dialogue: 0,0:17:23.89,0:17:28.91,Default,,0000,0000,0000,,(Applause)