[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.74,0:00:04.86,Default,,0000,0000,0000,,So, I started my first job\Nas a computer programmer
Dialogue: 0,0:00:04.88,0:00:06.84,Default,,0000,0000,0000,,in my very first year of college --
Dialogue: 0,0:00:06.86,0:00:08.37,Default,,0000,0000,0000,,basically, as a teenager.
Dialogue: 0,0:00:08.89,0:00:10.62,Default,,0000,0000,0000,,Soon after I started working,
Dialogue: 0,0:00:10.64,0:00:12.26,Default,,0000,0000,0000,,writing software in a company,
Dialogue: 0,0:00:12.80,0:00:16.43,Default,,0000,0000,0000,,a manager who worked at the company\Ncame down to where I was,
Dialogue: 0,0:00:16.46,0:00:17.73,Default,,0000,0000,0000,,and he whispered to me,
Dialogue: 0,0:00:18.23,0:00:21.09,Default,,0000,0000,0000,,"Can he tell if I'm lying?"
Dialogue: 0,0:00:21.81,0:00:23.88,Default,,0000,0000,0000,,There was nobody else in the room.
Dialogue: 0,0:00:25.03,0:00:29.42,Default,,0000,0000,0000,,"Can who tell if you're lying?\NAnd why are we whispering?"
Dialogue: 0,0:00:30.27,0:00:33.37,Default,,0000,0000,0000,,The manager pointed\Nat the computer in the room.
Dialogue: 0,0:00:33.40,0:00:36.49,Default,,0000,0000,0000,,"Can he tell if I'm lying?"
Dialogue: 0,0:00:37.61,0:00:41.98,Default,,0000,0000,0000,,Well, that manager was having\Nan affair with the receptionist.
Dialogue: 0,0:00:41.100,0:00:43.11,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:00:43.14,0:00:44.90,Default,,0000,0000,0000,,And I was still a teenager.
Dialogue: 0,0:00:45.45,0:00:47.47,Default,,0000,0000,0000,,So I whisper-shouted back to him,
Dialogue: 0,0:00:47.49,0:00:51.11,Default,,0000,0000,0000,,"Yes, the computer can tell\Nif you're lying."
Dialogue: 0,0:00:51.14,0:00:52.94,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:00:52.97,0:00:55.89,Default,,0000,0000,0000,,Well, I laughed, but actually,\Nthe laugh's on me.
Dialogue: 0,0:00:55.92,0:00:59.18,Default,,0000,0000,0000,,Nowadays, there are computational systems
Dialogue: 0,0:00:59.21,0:01:02.76,Default,,0000,0000,0000,,that can suss out\Nemotional states and even lying
Dialogue: 0,0:01:02.78,0:01:04.82,Default,,0000,0000,0000,,from processing human faces.
Dialogue: 0,0:01:05.25,0:01:09.40,Default,,0000,0000,0000,,Advertisers and even governments\Nare very interested.
Dialogue: 0,0:01:10.32,0:01:12.18,Default,,0000,0000,0000,,I had become a computer programmer
Dialogue: 0,0:01:12.20,0:01:15.32,Default,,0000,0000,0000,,because I was one of those kids\Ncrazy about math and science.
Dialogue: 0,0:01:15.94,0:01:19.05,Default,,0000,0000,0000,,But somewhere along the line\NI'd learned about nuclear weapons,
Dialogue: 0,0:01:19.07,0:01:22.03,Default,,0000,0000,0000,,and I'd gotten really concerned\Nwith the ethics of science.
Dialogue: 0,0:01:22.05,0:01:23.25,Default,,0000,0000,0000,,I was troubled.
Dialogue: 0,0:01:23.28,0:01:25.92,Default,,0000,0000,0000,,However, because of family circumstances,
Dialogue: 0,0:01:25.94,0:01:29.24,Default,,0000,0000,0000,,I also needed to start working\Nas soon as possible.
Dialogue: 0,0:01:29.26,0:01:32.56,Default,,0000,0000,0000,,So I thought to myself, hey,\Nlet me pick a technical field
Dialogue: 0,0:01:32.59,0:01:34.38,Default,,0000,0000,0000,,where I can get a job easily
Dialogue: 0,0:01:34.41,0:01:38.43,Default,,0000,0000,0000,,and where I don't have to deal\Nwith any troublesome questions of ethics.
Dialogue: 0,0:01:39.02,0:01:40.55,Default,,0000,0000,0000,,So I picked computers.
Dialogue: 0,0:01:40.58,0:01:41.68,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:01:41.70,0:01:45.11,Default,,0000,0000,0000,,Well, ha, ha, ha!\NAll the laughs are on me.
Dialogue: 0,0:01:45.14,0:01:47.89,Default,,0000,0000,0000,,Nowadays, computer scientists\Nare building platforms
Dialogue: 0,0:01:47.92,0:01:52.12,Default,,0000,0000,0000,,that control what a billion\Npeople see every day.
Dialogue: 0,0:01:53.05,0:01:56.87,Default,,0000,0000,0000,,They're developing cars\Nthat could decide who to run over.
Dialogue: 0,0:01:57.71,0:02:00.92,Default,,0000,0000,0000,,They're even building machines, weapons,
Dialogue: 0,0:02:00.94,0:02:03.23,Default,,0000,0000,0000,,that might kill human beings in war.
Dialogue: 0,0:02:03.25,0:02:06.02,Default,,0000,0000,0000,,It's ethics all the way down.
Dialogue: 0,0:02:07.18,0:02:09.24,Default,,0000,0000,0000,,Machine intelligence is here.
Dialogue: 0,0:02:09.82,0:02:13.30,Default,,0000,0000,0000,,We're now using computation\Nto make all sort of decisions,
Dialogue: 0,0:02:13.32,0:02:15.21,Default,,0000,0000,0000,,but also new kinds of decisions.
Dialogue: 0,0:02:15.23,0:02:20.40,Default,,0000,0000,0000,,We're asking questions to computation\Nthat have no single right answers,
Dialogue: 0,0:02:20.43,0:02:21.63,Default,,0000,0000,0000,,that are subjective
Dialogue: 0,0:02:21.65,0:02:23.98,Default,,0000,0000,0000,,and open-ended and value-laden.
Dialogue: 0,0:02:24.00,0:02:25.76,Default,,0000,0000,0000,,We're asking questions like,
Dialogue: 0,0:02:25.78,0:02:27.43,Default,,0000,0000,0000,,"Who should the company hire?"
Dialogue: 0,0:02:28.10,0:02:30.86,Default,,0000,0000,0000,,"Which update from which friend\Nshould you be shown?"
Dialogue: 0,0:02:30.88,0:02:33.14,Default,,0000,0000,0000,,"Which convict is more\Nlikely to reoffend?"
Dialogue: 0,0:02:33.51,0:02:36.57,Default,,0000,0000,0000,,"Which news item or movie\Nshould be recommended to people?"
Dialogue: 0,0:02:36.59,0:02:39.96,Default,,0000,0000,0000,,Look, yes, we've been using\Ncomputers for a while,
Dialogue: 0,0:02:39.99,0:02:41.50,Default,,0000,0000,0000,,but this is different.
Dialogue: 0,0:02:41.53,0:02:43.60,Default,,0000,0000,0000,,This is a historical twist,
Dialogue: 0,0:02:43.62,0:02:48.96,Default,,0000,0000,0000,,because we cannot anchor computation\Nfor such subjective decisions
Dialogue: 0,0:02:48.98,0:02:54.40,Default,,0000,0000,0000,,the way we can anchor computation\Nfor flying airplanes, building bridges,
Dialogue: 0,0:02:54.42,0:02:55.68,Default,,0000,0000,0000,,going to the moon.
Dialogue: 0,0:02:56.45,0:02:59.71,Default,,0000,0000,0000,,Are airplanes safer?\NDid the bridge sway and fall?
Dialogue: 0,0:02:59.73,0:03:04.23,Default,,0000,0000,0000,,There, we have agreed-upon,\Nfairly clear benchmarks,
Dialogue: 0,0:03:04.25,0:03:06.49,Default,,0000,0000,0000,,and we have laws of nature to guide us.
Dialogue: 0,0:03:06.52,0:03:09.91,Default,,0000,0000,0000,,We have no such anchors and benchmarks
Dialogue: 0,0:03:09.94,0:03:13.90,Default,,0000,0000,0000,,for decisions in messy human affairs.
Dialogue: 0,0:03:13.92,0:03:18.16,Default,,0000,0000,0000,,To make things more complicated,\Nour software is getting more powerful,
Dialogue: 0,0:03:18.18,0:03:21.96,Default,,0000,0000,0000,,but it's also getting less\Ntransparent and more complex.
Dialogue: 0,0:03:22.54,0:03:24.58,Default,,0000,0000,0000,,Recently, in the past decade,
Dialogue: 0,0:03:24.61,0:03:27.34,Default,,0000,0000,0000,,complex algorithms\Nhave made great strides.
Dialogue: 0,0:03:27.36,0:03:29.35,Default,,0000,0000,0000,,They can recognize human faces.
Dialogue: 0,0:03:29.98,0:03:32.04,Default,,0000,0000,0000,,They can decipher handwriting.
Dialogue: 0,0:03:32.44,0:03:34.50,Default,,0000,0000,0000,,They can detect credit card fraud
Dialogue: 0,0:03:34.53,0:03:35.72,Default,,0000,0000,0000,,and block spam
Dialogue: 0,0:03:35.74,0:03:37.78,Default,,0000,0000,0000,,and they can translate between languages.
Dialogue: 0,0:03:37.80,0:03:40.37,Default,,0000,0000,0000,,They can detect tumors in medical imaging.
Dialogue: 0,0:03:40.40,0:03:42.60,Default,,0000,0000,0000,,They can beat humans in chess and Go.
Dialogue: 0,0:03:43.26,0:03:47.77,Default,,0000,0000,0000,,Much of this progress comes\Nfrom a method called "machine learning."
Dialogue: 0,0:03:48.18,0:03:51.36,Default,,0000,0000,0000,,Machine learning is different\Nthan traditional programming,
Dialogue: 0,0:03:51.39,0:03:54.97,Default,,0000,0000,0000,,where you give the computer\Ndetailed, exact, painstaking instructions.
Dialogue: 0,0:03:55.38,0:03:59.56,Default,,0000,0000,0000,,It's more like you take the system\Nand you feed it lots of data,
Dialogue: 0,0:03:59.58,0:04:01.24,Default,,0000,0000,0000,,including unstructured data,
Dialogue: 0,0:04:01.26,0:04:03.54,Default,,0000,0000,0000,,like the kind we generate\Nin our digital lives.
Dialogue: 0,0:04:03.57,0:04:06.30,Default,,0000,0000,0000,,And the system learns\Nby churning through this data.
Dialogue: 0,0:04:06.67,0:04:08.20,Default,,0000,0000,0000,,And also, crucially,
Dialogue: 0,0:04:08.22,0:04:12.60,Default,,0000,0000,0000,,these systems don't operate\Nunder a single-answer logic.
Dialogue: 0,0:04:12.62,0:04:15.58,Default,,0000,0000,0000,,They don't produce a simple answer;\Nit's more probabilistic:
Dialogue: 0,0:04:15.61,0:04:19.09,Default,,0000,0000,0000,,"This one is probably more like\Nwhat you're looking for."
Dialogue: 0,0:04:20.02,0:04:23.09,Default,,0000,0000,0000,,Now, the upside is:\Nthis method is really powerful.
Dialogue: 0,0:04:23.12,0:04:25.19,Default,,0000,0000,0000,,The head of Google's AI systems called it,
Dialogue: 0,0:04:25.22,0:04:27.41,Default,,0000,0000,0000,,"the unreasonable effectiveness of data."
Dialogue: 0,0:04:27.79,0:04:29.14,Default,,0000,0000,0000,,The downside is,
Dialogue: 0,0:04:29.74,0:04:32.81,Default,,0000,0000,0000,,we don't really understand\Nwhat the system learned.
Dialogue: 0,0:04:32.83,0:04:34.42,Default,,0000,0000,0000,,In fact, that's its power.
Dialogue: 0,0:04:34.95,0:04:38.74,Default,,0000,0000,0000,,This is less like giving\Ninstructions to a computer;
Dialogue: 0,0:04:39.20,0:04:43.26,Default,,0000,0000,0000,,it's more like training\Na puppy-machine-creature
Dialogue: 0,0:04:43.29,0:04:45.66,Default,,0000,0000,0000,,we don't really understand or control.
Dialogue: 0,0:04:46.36,0:04:47.91,Default,,0000,0000,0000,,So this is our problem.
Dialogue: 0,0:04:48.43,0:04:52.69,Default,,0000,0000,0000,,It's a problem when this artificial\Nintelligence system gets things wrong.
Dialogue: 0,0:04:52.71,0:04:56.25,Default,,0000,0000,0000,,It's also a problem\Nwhen it gets things right,
Dialogue: 0,0:04:56.28,0:04:59.90,Default,,0000,0000,0000,,because we don't even know which is which\Nwhen it's a subjective problem.
Dialogue: 0,0:04:59.93,0:05:02.27,Default,,0000,0000,0000,,We don't know what this thing is thinking.
Dialogue: 0,0:05:03.49,0:05:07.18,Default,,0000,0000,0000,,So, consider a hiring algorithm --
Dialogue: 0,0:05:08.12,0:05:12.43,Default,,0000,0000,0000,,a system used to hire people,\Nusing machine-learning systems.
Dialogue: 0,0:05:13.05,0:05:16.63,Default,,0000,0000,0000,,Such a system would have been trained\Non previous employees' data
Dialogue: 0,0:05:16.66,0:05:19.25,Default,,0000,0000,0000,,and instructed to find and hire
Dialogue: 0,0:05:19.27,0:05:22.31,Default,,0000,0000,0000,,people like the existing\Nhigh performers in the company.
Dialogue: 0,0:05:22.81,0:05:23.97,Default,,0000,0000,0000,,Sounds good.
Dialogue: 0,0:05:23.99,0:05:25.99,Default,,0000,0000,0000,,I once attended a conference
Dialogue: 0,0:05:26.01,0:05:29.14,Default,,0000,0000,0000,,that brought together\Nhuman resources managers and executives,
Dialogue: 0,0:05:29.16,0:05:30.37,Default,,0000,0000,0000,,high-level people,
Dialogue: 0,0:05:30.39,0:05:31.95,Default,,0000,0000,0000,,using such systems in hiring.
Dialogue: 0,0:05:31.98,0:05:33.62,Default,,0000,0000,0000,,They were super excited.
Dialogue: 0,0:05:33.65,0:05:38.30,Default,,0000,0000,0000,,They thought that this would make hiring\Nmore objective, less biased,
Dialogue: 0,0:05:38.32,0:05:41.32,Default,,0000,0000,0000,,and give women\Nand minorities a better shot
Dialogue: 0,0:05:41.35,0:05:43.54,Default,,0000,0000,0000,,against biased human managers.
Dialogue: 0,0:05:43.56,0:05:46.40,Default,,0000,0000,0000,,And look -- human hiring is biased.
Dialogue: 0,0:05:47.10,0:05:48.28,Default,,0000,0000,0000,,I know.
Dialogue: 0,0:05:48.31,0:05:51.31,Default,,0000,0000,0000,,I mean, in one of my early jobs\Nas a programmer,
Dialogue: 0,0:05:51.34,0:05:55.20,Default,,0000,0000,0000,,my immediate manager would sometimes\Ncome down to where I was
Dialogue: 0,0:05:55.23,0:05:58.98,Default,,0000,0000,0000,,really early in the morning\Nor really late in the afternoon,
Dialogue: 0,0:05:59.01,0:06:02.07,Default,,0000,0000,0000,,and she'd say, "Zeynep,\Nlet's go to lunch!"
Dialogue: 0,0:06:02.72,0:06:04.89,Default,,0000,0000,0000,,I'd be puzzled by the weird timing.
Dialogue: 0,0:06:04.92,0:06:07.04,Default,,0000,0000,0000,,It's 4pm. Lunch?
Dialogue: 0,0:06:07.07,0:06:10.16,Default,,0000,0000,0000,,I was broke, so free lunch. I always went.
Dialogue: 0,0:06:10.62,0:06:12.68,Default,,0000,0000,0000,,I later realized what was happening.
Dialogue: 0,0:06:12.71,0:06:17.26,Default,,0000,0000,0000,,My immediate managers\Nhad not confessed to their higher-ups
Dialogue: 0,0:06:17.28,0:06:20.39,Default,,0000,0000,0000,,that the programmer they hired\Nfor a serious job was a teen girl
Dialogue: 0,0:06:20.42,0:06:24.35,Default,,0000,0000,0000,,who wore jeans and sneakers to work.
Dialogue: 0,0:06:25.17,0:06:27.38,Default,,0000,0000,0000,,I was doing a good job,\NI just looked wrong
Dialogue: 0,0:06:27.40,0:06:29.10,Default,,0000,0000,0000,,and was the wrong age and gender.
Dialogue: 0,0:06:29.12,0:06:32.47,Default,,0000,0000,0000,,So hiring in a gender- and race-blind way
Dialogue: 0,0:06:32.49,0:06:34.36,Default,,0000,0000,0000,,certainly sounds good to me.
Dialogue: 0,0:06:35.03,0:06:38.37,Default,,0000,0000,0000,,But with these systems,\Nit is more complicated, and here's why:
Dialogue: 0,0:06:38.97,0:06:44.76,Default,,0000,0000,0000,,Currently, computational systems\Ncan infer all sorts of things about you
Dialogue: 0,0:06:44.78,0:06:46.66,Default,,0000,0000,0000,,from your digital crumbs,
Dialogue: 0,0:06:46.68,0:06:49.01,Default,,0000,0000,0000,,even if you have not\Ndisclosed those things.
Dialogue: 0,0:06:49.51,0:06:52.43,Default,,0000,0000,0000,,They can infer your sexual orientation,
Dialogue: 0,0:06:52.99,0:06:54.30,Default,,0000,0000,0000,,your personality traits,
Dialogue: 0,0:06:54.86,0:06:56.23,Default,,0000,0000,0000,,your political leanings.
Dialogue: 0,0:06:56.83,0:07:00.52,Default,,0000,0000,0000,,They have predictive power\Nwith high levels of accuracy.
Dialogue: 0,0:07:01.36,0:07:03.94,Default,,0000,0000,0000,,Remember -- for things\Nyou haven't even disclosed.
Dialogue: 0,0:07:03.96,0:07:05.56,Default,,0000,0000,0000,,This is inference.
Dialogue: 0,0:07:05.58,0:07:08.84,Default,,0000,0000,0000,,I have a friend who developed\Nsuch computational systems
Dialogue: 0,0:07:08.86,0:07:12.50,Default,,0000,0000,0000,,to predict the likelihood\Nof clinical or postpartum depression
Dialogue: 0,0:07:12.53,0:07:13.94,Default,,0000,0000,0000,,from social media data.
Dialogue: 0,0:07:14.68,0:07:16.10,Default,,0000,0000,0000,,The results are impressive.
Dialogue: 0,0:07:16.49,0:07:19.85,Default,,0000,0000,0000,,Her system can predict\Nthe likelihood of depression
Dialogue: 0,0:07:19.87,0:07:23.78,Default,,0000,0000,0000,,months before the onset of any symptoms --
Dialogue: 0,0:07:23.80,0:07:25.17,Default,,0000,0000,0000,,months before.
Dialogue: 0,0:07:25.20,0:07:27.44,Default,,0000,0000,0000,,No symptoms, there's prediction.
Dialogue: 0,0:07:27.47,0:07:32.28,Default,,0000,0000,0000,,She hopes it will be used\Nfor early intervention. Great!
Dialogue: 0,0:07:32.91,0:07:34.95,Default,,0000,0000,0000,,But now put this in the context of hiring.
Dialogue: 0,0:07:36.03,0:07:39.07,Default,,0000,0000,0000,,So at this human resources\Nmanagers conference,
Dialogue: 0,0:07:39.10,0:07:43.81,Default,,0000,0000,0000,,I approached a high-level manager\Nin a very large company,
Dialogue: 0,0:07:43.83,0:07:48.41,Default,,0000,0000,0000,,and I said to her, "Look,\Nwhat if, unbeknownst to you,
Dialogue: 0,0:07:48.43,0:07:54.98,Default,,0000,0000,0000,,your system is weeding out people\Nwith high future likelihood of depression?
Dialogue: 0,0:07:55.76,0:07:59.14,Default,,0000,0000,0000,,They're not depressed now,\Njust maybe in the future, more likely.
Dialogue: 0,0:07:59.92,0:08:03.33,Default,,0000,0000,0000,,What if it's weeding out women\Nmore likely to be pregnant
Dialogue: 0,0:08:03.35,0:08:05.94,Default,,0000,0000,0000,,in the next year or two\Nbut aren't pregnant now?
Dialogue: 0,0:08:06.84,0:08:12.48,Default,,0000,0000,0000,,What if it's hiring aggressive people\Nbecause that's your workplace culture?"
Dialogue: 0,0:08:13.17,0:08:15.86,Default,,0000,0000,0000,,You can't tell this by looking\Nat gender breakdowns.
Dialogue: 0,0:08:15.89,0:08:17.39,Default,,0000,0000,0000,,Those may be balanced.
Dialogue: 0,0:08:17.41,0:08:20.97,Default,,0000,0000,0000,,And since this is machine learning,\Nnot traditional coding,
Dialogue: 0,0:08:20.100,0:08:25.90,Default,,0000,0000,0000,,there is no variable there\Nlabeled "higher risk of depression,"
Dialogue: 0,0:08:25.93,0:08:27.76,Default,,0000,0000,0000,,"higher risk of pregnancy,"
Dialogue: 0,0:08:27.78,0:08:29.52,Default,,0000,0000,0000,,"aggressive guy scale."
Dialogue: 0,0:08:29.100,0:08:33.67,Default,,0000,0000,0000,,Not only do you not know\Nwhat your system is selecting on,
Dialogue: 0,0:08:33.70,0:08:36.02,Default,,0000,0000,0000,,you don't even know\Nwhere to begin to look.
Dialogue: 0,0:08:36.04,0:08:37.29,Default,,0000,0000,0000,,It's a black box.
Dialogue: 0,0:08:37.32,0:08:40.12,Default,,0000,0000,0000,,It has predictive power,\Nbut you don't understand it.
Dialogue: 0,0:08:40.49,0:08:42.86,Default,,0000,0000,0000,,"What safeguards," I asked, "do you have
Dialogue: 0,0:08:42.88,0:08:46.55,Default,,0000,0000,0000,,to make sure that your black box\Nisn't doing something shady?"
Dialogue: 0,0:08:48.86,0:08:52.74,Default,,0000,0000,0000,,She looked at me as if I had\Njust stepped on 10 puppy tails.
Dialogue: 0,0:08:52.76,0:08:54.01,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:08:54.04,0:08:56.08,Default,,0000,0000,0000,,She stared at me and she said,
Dialogue: 0,0:08:56.56,0:09:00.89,Default,,0000,0000,0000,,"I don't want to hear\Nanother word about this."
Dialogue: 0,0:09:01.46,0:09:03.49,Default,,0000,0000,0000,,And she turned around and walked away.
Dialogue: 0,0:09:04.06,0:09:05.55,Default,,0000,0000,0000,,Mind you -- she wasn't rude.
Dialogue: 0,0:09:05.57,0:09:11.88,Default,,0000,0000,0000,,It was clearly: what I don't know\Nisn't my problem, go away, death stare.
Dialogue: 0,0:09:11.91,0:09:13.15,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:09:13.86,0:09:17.70,Default,,0000,0000,0000,,Look, such a system\Nmay even be less biased
Dialogue: 0,0:09:17.72,0:09:19.83,Default,,0000,0000,0000,,than human managers in some ways.
Dialogue: 0,0:09:19.85,0:09:21.100,Default,,0000,0000,0000,,And it could make monetary sense.
Dialogue: 0,0:09:22.57,0:09:24.22,Default,,0000,0000,0000,,But it could also lead
Dialogue: 0,0:09:24.25,0:09:28.100,Default,,0000,0000,0000,,to a steady but stealthy\Nshutting out of the job market
Dialogue: 0,0:09:29.02,0:09:31.31,Default,,0000,0000,0000,,of people with higher risk of depression.
Dialogue: 0,0:09:31.75,0:09:34.35,Default,,0000,0000,0000,,Is this the kind of society\Nwe want to build,
Dialogue: 0,0:09:34.37,0:09:36.66,Default,,0000,0000,0000,,without even knowing we've done this,
Dialogue: 0,0:09:36.68,0:09:40.65,Default,,0000,0000,0000,,because we turned decision-making\Nto machines we don't totally understand?
Dialogue: 0,0:09:41.26,0:09:42.72,Default,,0000,0000,0000,,Another problem is this:
Dialogue: 0,0:09:43.31,0:09:47.77,Default,,0000,0000,0000,,these systems are often trained\Non data generated by our actions,
Dialogue: 0,0:09:47.79,0:09:49.61,Default,,0000,0000,0000,,human imprints.
Dialogue: 0,0:09:50.19,0:09:53.100,Default,,0000,0000,0000,,Well, they could just be\Nreflecting our biases,
Dialogue: 0,0:09:54.02,0:09:57.61,Default,,0000,0000,0000,,and these systems\Ncould be picking up on our biases
Dialogue: 0,0:09:57.64,0:09:58.95,Default,,0000,0000,0000,,and amplifying them
Dialogue: 0,0:09:58.97,0:10:00.39,Default,,0000,0000,0000,,and showing them back to us,
Dialogue: 0,0:10:00.42,0:10:01.88,Default,,0000,0000,0000,,while we're telling ourselves,
Dialogue: 0,0:10:01.90,0:10:05.02,Default,,0000,0000,0000,,"We're just doing objective,\Nneutral computation."
Dialogue: 0,0:10:06.31,0:10:08.99,Default,,0000,0000,0000,,Researchers found that on Google,
Dialogue: 0,0:10:10.13,0:10:15.45,Default,,0000,0000,0000,,women are less likely than men\Nto be shown job ads for high-paying jobs.
Dialogue: 0,0:10:16.46,0:10:18.99,Default,,0000,0000,0000,,And searching for African-American names
Dialogue: 0,0:10:19.02,0:10:23.72,Default,,0000,0000,0000,,is more likely to bring up ads\Nsuggesting criminal history,
Dialogue: 0,0:10:23.75,0:10:25.31,Default,,0000,0000,0000,,even when there is none.
Dialogue: 0,0:10:26.69,0:10:30.24,Default,,0000,0000,0000,,Such hidden biases\Nand black-box algorithms
Dialogue: 0,0:10:30.27,0:10:34.24,Default,,0000,0000,0000,,that researchers uncover sometimes\Nbut sometimes we don't know,
Dialogue: 0,0:10:34.26,0:10:36.92,Default,,0000,0000,0000,,can have life-altering consequences.
Dialogue: 0,0:10:37.96,0:10:42.12,Default,,0000,0000,0000,,In Wisconsin, a defendant\Nwas sentenced to six years in prison
Dialogue: 0,0:10:42.14,0:10:43.50,Default,,0000,0000,0000,,for evading the police.
Dialogue: 0,0:10:44.82,0:10:46.01,Default,,0000,0000,0000,,You may not know this,
Dialogue: 0,0:10:46.03,0:10:50.03,Default,,0000,0000,0000,,but algorithms are increasingly used\Nin parole and sentencing decisions.
Dialogue: 0,0:10:50.06,0:10:53.01,Default,,0000,0000,0000,,He wanted to know:\NHow is this score calculated?
Dialogue: 0,0:10:53.80,0:10:55.46,Default,,0000,0000,0000,,It's a commercial black box.
Dialogue: 0,0:10:55.48,0:10:59.69,Default,,0000,0000,0000,,The company refused to have its algorithm\Nbe challenged in open court.
Dialogue: 0,0:11:00.40,0:11:05.93,Default,,0000,0000,0000,,But ProPublica, an investigative\Nnonprofit, audited that very algorithm
Dialogue: 0,0:11:05.95,0:11:07.97,Default,,0000,0000,0000,,with what public data they could find,
Dialogue: 0,0:11:07.99,0:11:10.31,Default,,0000,0000,0000,,and found that its outcomes were biased
Dialogue: 0,0:11:10.33,0:11:13.96,Default,,0000,0000,0000,,and its predictive power\Nwas dismal, barely better than chance,
Dialogue: 0,0:11:13.98,0:11:18.40,Default,,0000,0000,0000,,and it was wrongly labeling\Nblack defendants as future criminals
Dialogue: 0,0:11:18.42,0:11:22.32,Default,,0000,0000,0000,,at twice the rate of white defendants.
Dialogue: 0,0:11:23.89,0:11:25.46,Default,,0000,0000,0000,,So, consider this case:
Dialogue: 0,0:11:26.10,0:11:29.96,Default,,0000,0000,0000,,This woman was late\Npicking up her godsister
Dialogue: 0,0:11:29.98,0:11:32.05,Default,,0000,0000,0000,,from a school in Broward County, Florida,
Dialogue: 0,0:11:32.76,0:11:35.11,Default,,0000,0000,0000,,running down the street\Nwith a friend of hers.
Dialogue: 0,0:11:35.14,0:11:39.24,Default,,0000,0000,0000,,They spotted an unlocked kid's bike\Nand a scooter on a porch
Dialogue: 0,0:11:39.26,0:11:40.89,Default,,0000,0000,0000,,and foolishly jumped on it.
Dialogue: 0,0:11:40.92,0:11:43.52,Default,,0000,0000,0000,,As they were speeding off,\Na woman came out and said,
Dialogue: 0,0:11:43.54,0:11:45.74,Default,,0000,0000,0000,,"Hey! That's my kid's bike!"
Dialogue: 0,0:11:45.77,0:11:49.06,Default,,0000,0000,0000,,They dropped it, they walked away,\Nbut they were arrested.
Dialogue: 0,0:11:49.09,0:11:52.72,Default,,0000,0000,0000,,She was wrong, she was foolish,\Nbut she was also just 18.
Dialogue: 0,0:11:52.75,0:11:55.29,Default,,0000,0000,0000,,She had a couple of juvenile misdemeanors.
Dialogue: 0,0:11:55.81,0:12:00.99,Default,,0000,0000,0000,,Meanwhile, that man had been arrested\Nfor shoplifting in Home Depot --
Dialogue: 0,0:12:01.02,0:12:03.94,Default,,0000,0000,0000,,85 dollars' worth of stuff,\Na similar petty crime.
Dialogue: 0,0:12:04.77,0:12:09.32,Default,,0000,0000,0000,,But he had two prior\Narmed robbery convictions.
Dialogue: 0,0:12:09.96,0:12:13.44,Default,,0000,0000,0000,,But the algorithm scored her\Nas high risk, and not him.
Dialogue: 0,0:12:14.75,0:12:18.62,Default,,0000,0000,0000,,Two years later, ProPublica found\Nthat she had not reoffended.
Dialogue: 0,0:12:18.64,0:12:21.19,Default,,0000,0000,0000,,It was just hard to get a job\Nfor her with her record.
Dialogue: 0,0:12:21.22,0:12:23.29,Default,,0000,0000,0000,,He, on the other hand, did reoffend
Dialogue: 0,0:12:23.32,0:12:27.15,Default,,0000,0000,0000,,and is now serving an eight-year\Nprison term for a later crime.
Dialogue: 0,0:12:28.09,0:12:31.46,Default,,0000,0000,0000,,Clearly, we need to audit our black boxes
Dialogue: 0,0:12:31.48,0:12:34.10,Default,,0000,0000,0000,,and not have them have\Nthis kind of unchecked power.
Dialogue: 0,0:12:34.12,0:12:36.100,Default,,0000,0000,0000,,(Applause)
Dialogue: 0,0:12:38.09,0:12:42.33,Default,,0000,0000,0000,,Audits are great and important,\Nbut they don't solve all our problems.
Dialogue: 0,0:12:42.35,0:12:45.10,Default,,0000,0000,0000,,Take Facebook's powerful\Nnews feed algorithm --
Dialogue: 0,0:12:45.12,0:12:49.97,Default,,0000,0000,0000,,you know, the one that ranks everything\Nand decides what to show you
Dialogue: 0,0:12:49.99,0:12:52.28,Default,,0000,0000,0000,,from all the friends and pages you follow.
Dialogue: 0,0:12:52.90,0:12:55.17,Default,,0000,0000,0000,,Should you be shown another baby picture?
Dialogue: 0,0:12:55.20,0:12:56.39,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:12:56.42,0:12:59.01,Default,,0000,0000,0000,,A sullen note from an acquaintance?
Dialogue: 0,0:12:59.45,0:13:01.30,Default,,0000,0000,0000,,An important but difficult news item?
Dialogue: 0,0:13:01.33,0:13:02.81,Default,,0000,0000,0000,,There's no right answer.
Dialogue: 0,0:13:02.84,0:13:05.49,Default,,0000,0000,0000,,Facebook optimizes\Nfor engagement on the site:
Dialogue: 0,0:13:05.52,0:13:06.93,Default,,0000,0000,0000,,likes, shares, comments.
Dialogue: 0,0:13:08.17,0:13:10.86,Default,,0000,0000,0000,,In August of 2014,
Dialogue: 0,0:13:10.89,0:13:13.55,Default,,0000,0000,0000,,protests broke out in Ferguson, Missouri,
Dialogue: 0,0:13:13.57,0:13:17.99,Default,,0000,0000,0000,,after the killing of an African-American\Nteenager by a white police officer,
Dialogue: 0,0:13:18.02,0:13:19.58,Default,,0000,0000,0000,,under murky circumstances.
Dialogue: 0,0:13:19.97,0:13:21.98,Default,,0000,0000,0000,,The news of the protests was all over
Dialogue: 0,0:13:22.00,0:13:24.69,Default,,0000,0000,0000,,my algorithmically\Nunfiltered Twitter feed,
Dialogue: 0,0:13:24.71,0:13:26.66,Default,,0000,0000,0000,,but nowhere on my Facebook.
Dialogue: 0,0:13:27.18,0:13:28.92,Default,,0000,0000,0000,,Was it my Facebook friends?
Dialogue: 0,0:13:28.94,0:13:30.97,Default,,0000,0000,0000,,I disabled Facebook's algorithm,
Dialogue: 0,0:13:31.47,0:13:34.32,Default,,0000,0000,0000,,which is hard because Facebook\Nkeeps wanting to make you
Dialogue: 0,0:13:34.34,0:13:36.38,Default,,0000,0000,0000,,come under the algorithm's control,
Dialogue: 0,0:13:36.40,0:13:38.64,Default,,0000,0000,0000,,and saw that my friends\Nwere talking about it.
Dialogue: 0,0:13:38.67,0:13:41.18,Default,,0000,0000,0000,,It's just that the algorithm\Nwasn't showing it to me.
Dialogue: 0,0:13:41.20,0:13:44.24,Default,,0000,0000,0000,,I researched this and found\Nthis was a widespread problem.
Dialogue: 0,0:13:44.26,0:13:48.08,Default,,0000,0000,0000,,The story of Ferguson\Nwasn't algorithm-friendly.
Dialogue: 0,0:13:48.10,0:13:49.27,Default,,0000,0000,0000,,It's not "likable."
Dialogue: 0,0:13:49.30,0:13:50.85,Default,,0000,0000,0000,,Who's going to click on "like?"
Dialogue: 0,0:13:51.50,0:13:53.71,Default,,0000,0000,0000,,It's not even easy to comment on.
Dialogue: 0,0:13:53.73,0:13:55.10,Default,,0000,0000,0000,,Without likes and comments,
Dialogue: 0,0:13:55.12,0:13:58.42,Default,,0000,0000,0000,,the algorithm was likely showing it\Nto even fewer people,
Dialogue: 0,0:13:58.44,0:13:59.98,Default,,0000,0000,0000,,so we didn't get to see this.
Dialogue: 0,0:14:00.95,0:14:02.17,Default,,0000,0000,0000,,Instead, that week,
Dialogue: 0,0:14:02.20,0:14:04.50,Default,,0000,0000,0000,,Facebook's algorithm highlighted this,
Dialogue: 0,0:14:04.52,0:14:06.75,Default,,0000,0000,0000,,which is the ALS Ice Bucket Challenge.
Dialogue: 0,0:14:06.77,0:14:10.51,Default,,0000,0000,0000,,Worthy cause; dump ice water,\Ndonate to charity, fine.
Dialogue: 0,0:14:10.54,0:14:12.44,Default,,0000,0000,0000,,But it was super algorithm-friendly.
Dialogue: 0,0:14:13.22,0:14:15.83,Default,,0000,0000,0000,,The machine made this decision for us.
Dialogue: 0,0:14:15.86,0:14:19.35,Default,,0000,0000,0000,,A very important\Nbut difficult conversation
Dialogue: 0,0:14:19.38,0:14:20.93,Default,,0000,0000,0000,,might have been smothered,
Dialogue: 0,0:14:20.96,0:14:23.65,Default,,0000,0000,0000,,had Facebook been the only channel.
Dialogue: 0,0:14:24.12,0:14:27.91,Default,,0000,0000,0000,,Now, finally, these systems\Ncan also be wrong
Dialogue: 0,0:14:27.94,0:14:30.67,Default,,0000,0000,0000,,in ways that don't resemble human systems.
Dialogue: 0,0:14:30.70,0:14:33.62,Default,,0000,0000,0000,,Do you guys remember Watson,\NIBM's machine-intelligence system
Dialogue: 0,0:14:33.64,0:14:36.77,Default,,0000,0000,0000,,that wiped the floor\Nwith human contestants on Jeopardy?
Dialogue: 0,0:14:37.13,0:14:38.56,Default,,0000,0000,0000,,It was a great player.
Dialogue: 0,0:14:38.58,0:14:42.15,Default,,0000,0000,0000,,But then, for Final Jeopardy,\NWatson was asked this question:
Dialogue: 0,0:14:42.66,0:14:45.59,Default,,0000,0000,0000,,"Its largest airport is named\Nfor a World War II hero,
Dialogue: 0,0:14:45.62,0:14:47.87,Default,,0000,0000,0000,,its second-largest\Nfor a World War II battle."
Dialogue: 0,0:14:47.89,0:14:49.27,Default,,0000,0000,0000,,(Hums Final Jeopardy music)
Dialogue: 0,0:14:49.58,0:14:50.76,Default,,0000,0000,0000,,Chicago.
Dialogue: 0,0:14:50.79,0:14:52.16,Default,,0000,0000,0000,,The two humans got it right.
Dialogue: 0,0:14:52.70,0:14:57.04,Default,,0000,0000,0000,,Watson, on the other hand,\Nanswered "Toronto" --
Dialogue: 0,0:14:57.07,0:14:58.89,Default,,0000,0000,0000,,for a US city category!
Dialogue: 0,0:14:59.60,0:15:02.50,Default,,0000,0000,0000,,The impressive system also made an error
Dialogue: 0,0:15:02.52,0:15:06.17,Default,,0000,0000,0000,,that a human would never make,\Na second-grader wouldn't make.
Dialogue: 0,0:15:06.82,0:15:09.93,Default,,0000,0000,0000,,Our machine intelligence can fail
Dialogue: 0,0:15:09.96,0:15:13.06,Default,,0000,0000,0000,,in ways that don't fit\Nerror patterns of humans,
Dialogue: 0,0:15:13.08,0:15:16.03,Default,,0000,0000,0000,,in ways we won't expect\Nand be prepared for.
Dialogue: 0,0:15:16.05,0:15:19.69,Default,,0000,0000,0000,,It'd be lousy not to get a job\None is qualified for,
Dialogue: 0,0:15:19.72,0:15:23.44,Default,,0000,0000,0000,,but it would triple suck\Nif it was because of stack overflow
Dialogue: 0,0:15:23.47,0:15:24.90,Default,,0000,0000,0000,,in some subroutine.
Dialogue: 0,0:15:24.92,0:15:26.50,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:15:26.53,0:15:29.31,Default,,0000,0000,0000,,In May of 2010,
Dialogue: 0,0:15:29.34,0:15:33.38,Default,,0000,0000,0000,,a flash crash on Wall Street\Nfueled by a feedback loop
Dialogue: 0,0:15:33.40,0:15:36.43,Default,,0000,0000,0000,,in Wall Street's "sell" algorithm
Dialogue: 0,0:15:36.46,0:15:40.64,Default,,0000,0000,0000,,wiped a trillion dollars\Nof value in 36 minutes.
Dialogue: 0,0:15:41.72,0:15:43.91,Default,,0000,0000,0000,,I don't even want to think\Nwhat "error" means
Dialogue: 0,0:15:43.93,0:15:47.52,Default,,0000,0000,0000,,in the context of lethal\Nautonomous weapons.
Dialogue: 0,0:15:49.89,0:15:53.68,Default,,0000,0000,0000,,So yes, humans have always made biases.
Dialogue: 0,0:15:53.71,0:15:55.88,Default,,0000,0000,0000,,Decision makers and gatekeepers,
Dialogue: 0,0:15:55.91,0:15:59.40,Default,,0000,0000,0000,,in courts, in news, in war ...
Dialogue: 0,0:15:59.42,0:16:02.46,Default,,0000,0000,0000,,they make mistakes;\Nbut that's exactly my point.
Dialogue: 0,0:16:02.49,0:16:06.01,Default,,0000,0000,0000,,We cannot escape\Nthese difficult questions.
Dialogue: 0,0:16:06.60,0:16:10.11,Default,,0000,0000,0000,,We cannot outsource\Nour responsibilities to machines.
Dialogue: 0,0:16:10.68,0:16:14.88,Default,,0000,0000,0000,,(Applause)
Dialogue: 0,0:16:17.09,0:16:21.54,Default,,0000,0000,0000,,Artificial intelligence does not give us\Na "Get out of ethics free" card.
Dialogue: 0,0:16:22.74,0:16:26.12,Default,,0000,0000,0000,,Data scientist Fred Benenson\Ncalls this math-washing.
Dialogue: 0,0:16:26.15,0:16:27.54,Default,,0000,0000,0000,,We need the opposite.
Dialogue: 0,0:16:27.56,0:16:32.95,Default,,0000,0000,0000,,We need to cultivate algorithm suspicion,\Nscrutiny and investigation.
Dialogue: 0,0:16:33.38,0:16:36.58,Default,,0000,0000,0000,,We need to make sure we have\Nalgorithmic accountability,
Dialogue: 0,0:16:36.60,0:16:39.05,Default,,0000,0000,0000,,auditing and meaningful transparency.
Dialogue: 0,0:16:39.38,0:16:42.61,Default,,0000,0000,0000,,We need to accept\Nthat bringing math and computation
Dialogue: 0,0:16:42.64,0:16:45.61,Default,,0000,0000,0000,,to messy, value-laden human affairs
Dialogue: 0,0:16:45.63,0:16:48.02,Default,,0000,0000,0000,,does not bring objectivity;
Dialogue: 0,0:16:48.04,0:16:51.67,Default,,0000,0000,0000,,rather, the complexity of human affairs\Ninvades the algorithms.
Dialogue: 0,0:16:52.15,0:16:55.64,Default,,0000,0000,0000,,Yes, we can and we should use computation
Dialogue: 0,0:16:55.66,0:16:57.67,Default,,0000,0000,0000,,to help us make better decisions.
Dialogue: 0,0:16:57.70,0:17:03.03,Default,,0000,0000,0000,,But we have to own up\Nto our moral responsibility to judgment,
Dialogue: 0,0:17:03.05,0:17:05.87,Default,,0000,0000,0000,,and use algorithms within that framework,
Dialogue: 0,0:17:05.90,0:17:10.83,Default,,0000,0000,0000,,not as a means to abdicate\Nand outsource our responsibilities
Dialogue: 0,0:17:10.85,0:17:13.31,Default,,0000,0000,0000,,to one another as human to human.
Dialogue: 0,0:17:13.81,0:17:16.42,Default,,0000,0000,0000,,Machine intelligence is here.
Dialogue: 0,0:17:16.44,0:17:19.86,Default,,0000,0000,0000,,That means we must hold on ever tighter
Dialogue: 0,0:17:19.88,0:17:22.03,Default,,0000,0000,0000,,to human values and human ethics.
Dialogue: 0,0:17:22.06,0:17:23.21,Default,,0000,0000,0000,,Thank you.
Dialogue: 0,0:17:23.23,0:17:28.25,Default,,0000,0000,0000,,(Applause)