1
00:00:10,080 --> 00:00:17,985
applause
2
00:00:17,985 --> 00:00:22,900
Thank you very much, can you…
You can hear me? Yes!
3
00:00:22,900 --> 00:00:27,620
I’ve been at this now 23 years. We
worked, with… My colleagues and I,
4
00:00:27,620 --> 00:00:31,390
we worked in about 30 countries,
we’ve advised 9 Truth Commissions,
5
00:00:31,390 --> 00:00:36,410
official Truth Commissions, 4 UN missions,
6
00:00:36,410 --> 00:00:40,150
4 international criminal tribunals.
We have testified in 4 different cases
7
00:00:40,150 --> 00:00:44,240
– 2 internationally, 2 domestically – and
we’ve advised dozens and dozens
8
00:00:44,240 --> 00:00:49,120
of non-governmental Human Rights groups
around the world. The point of this stuff
9
00:00:49,120 --> 00:00:54,180
is to figure out how to bring the
knowledge of the people who’ve suffered
10
00:00:54,180 --> 00:00:58,770
human rights violations to bear,
on demanding accountability
11
00:00:58,770 --> 00:01:04,960
from the perpetrators. Our job is to
figure out how we can tell the truth.
12
00:01:04,960 --> 00:01:09,240
It is one of the moral foundations of the
international Human Rights movement
13
00:01:09,240 --> 00:01:14,220
that we speak Truth to Power. We
look in the face of the powerful
14
00:01:14,220 --> 00:01:19,299
and we tell them what we believe
they have done that is wrong.
15
00:01:19,299 --> 00:01:23,639
If that’s gonna work, we
have to speak the truth.
16
00:01:23,639 --> 00:01:29,470
We have to be right, we
have to get the analysis on.
17
00:01:29,470 --> 00:01:33,979
That’s not always easy and to get there,
18
00:01:33,979 --> 00:01:37,209
there are sort of 3 themes that
I wanna try to touch in this talk.
19
00:01:37,209 --> 00:01:40,379
Since the talk is pretty short I’m
really gonna touch on 2 of them, so
20
00:01:40,379 --> 00:01:43,619
at the very end of the talk I’ll invite
people who’d like to talk more about
21
00:01:43,619 --> 00:01:49,270
the specifically technical aspects of this
work, about classifiers, about clustering,
22
00:01:49,270 --> 00:01:53,620
about statistical estimation, about
database techniques. People who wanna talk
23
00:01:53,620 --> 00:01:56,990
about that I’d love to gather and we’ll
try to find a space. I’ve been fighting
24
00:01:56,990 --> 00:02:00,460
with the Wiki for 2 days; I think
I’m probably not the only one.
25
00:02:00,460 --> 00:02:04,959
We can gather, we can talk about
that stuff more in detail. So today,
26
00:02:04,959 --> 00:02:09,990
in the next 25 minutes I’m
going to focus specifically on
27
00:02:09,990 --> 00:02:14,520
the trial of General
José Efraín Ríos Montt
28
00:02:14,520 --> 00:02:20,200
who ruled Guatemala from
March 1982 until August 1983.
29
00:02:20,200 --> 00:02:25,180
That’s General Ríos, there in
the upper corner in the red tie.
30
00:02:25,180 --> 00:02:30,600
During the government
of General Ríos Montt
31
00:02:30,600 --> 00:02:35,610
tens of thousands of people were killed by
the army of Guatemala. And the question
32
00:02:35,610 --> 00:02:39,610
that has been facing Guatemalans
since that time is:
33
00:02:39,610 --> 00:02:44,080
“Did the pattern of killing
that the army committed
34
00:02:44,080 --> 00:02:49,690
constitute acts of genocide?”. Now
genocide is a very specific crime
35
00:02:49,690 --> 00:02:54,420
in International Law. It does not
mean you killed a lot of people.
36
00:02:54,420 --> 00:02:58,910
There are other war crimes for mass
killing. Genocide specifically means
37
00:02:58,910 --> 00:03:03,930
that you picked out a particular group;
and to the exclusion of other groups
38
00:03:03,930 --> 00:03:08,460
nearby them you focused
on eliminating that group.
39
00:03:08,460 --> 00:03:14,240
That’s key because for a statistician
that gives us a hypothesis we can test
40
00:03:14,240 --> 00:03:18,860
which is: “What is the relative risk,
what is the differential probability
41
00:03:18,860 --> 00:03:22,820
of people in the target group being
killed relative to their neighbours
42
00:03:22,820 --> 00:03:28,150
who are not in the target group?”
So without further ado,
43
00:03:28,150 --> 00:03:31,970
let’s look at the relative risk of
being killed for indigenous people
44
00:03:31,970 --> 00:03:36,880
in the 3 rural counties of
Chajul, Cotzal and Nebaj
45
00:03:36,880 --> 00:03:41,400
relative to their
non-indigenous neighbours.
46
00:03:41,400 --> 00:03:45,960
We have – and I’ll talk in a moment about
how we have this – we have information,
47
00:03:45,960 --> 00:03:51,490
and evidence, and estimations of the
deaths of about 2150 indigenous people.
48
00:03:51,490 --> 00:03:58,550
People killed by the army in the period
of the government of General Ríos.
49
00:03:58,550 --> 00:04:02,550
The population, the total number of
people alive who were indigenous
50
00:04:02,550 --> 00:04:07,370
in those counties in the census
of 1981 is about 39,000.
51
00:04:07,370 --> 00:04:14,500
So the approximate crude mortality
rate due to homicide by the army
52
00:04:14,500 --> 00:04:18,710
is 5.5% for indigenous people in
that period. Now that’s relative
53
00:04:18,710 --> 00:04:22,890
to the homicide rate for non-indigenous
people in the same place
54
00:04:22,890 --> 00:04:27,200
of approximately 0.7%. So what
we ask is: “What is the ratio
55
00:04:27,200 --> 00:04:30,530
between those 2 numbers?” And
the ratio between those 2 numbers
56
00:04:30,530 --> 00:04:35,600
is the relative risk. It’s approximately
8. We interpret that as: if you were
57
00:04:35,600 --> 00:04:41,339
an indigenous person alive in
one of those 3 counties in 1982,
58
00:04:41,339 --> 00:04:46,939
your probability of being killed
by the army was 8 times greater
59
00:04:46,939 --> 00:04:51,069
than a person also living
in those 3 counties
60
00:04:51,069 --> 00:04:56,179
who was not indigenous.
Eight times, 8 times!
61
00:04:56,179 --> 00:05:00,250
To put that in relative terms: the
probability… the relative risk of being
62
00:05:00,250 --> 00:05:04,720
a Bosniac relative to being Serb
in Bosnia during the war in Bosnia
63
00:05:04,720 --> 00:05:09,800
was a little less than 3. So your
relative risk of being indigenous
64
00:05:09,800 --> 00:05:13,310
was more than twice nearly 3 times
as much as your relative risk
65
00:05:13,310 --> 00:05:19,200
of being Bosniac in the Bosnian War.
It’s an astonishing level of focus.
66
00:05:19,200 --> 00:05:23,809
It shows a tremendous planning
and coherence, I believe.
67
00:05:23,809 --> 00:05:29,469
So, again coming back to the statistical
conclusion, how do we come to that?
68
00:05:29,469 --> 00:05:32,849
How do we find that information? How do we
make that conclusion? First, we’re only
69
00:05:32,849 --> 00:05:35,470
looking at homicides committed by the
army. We’re not looking at homicides
70
00:05:35,470 --> 00:05:39,409
committed by other parties, by
the guerrillas, by private actors.
71
00:05:39,409 --> 00:05:44,499
We’re not looking at excess mortality,
the mortality that we might find
72
00:05:44,499 --> 00:05:47,709
in conflict that is in excess of
normal peacetime mortality.
73
00:05:47,709 --> 00:05:51,470
We’re not looking at any of that,
only homicide. And the percentage
74
00:05:51,470 --> 00:05:55,330
relates the number of people killed by the
army with the population that was alive.
75
00:05:55,330 --> 00:05:58,650
That’s crucial here. We’re looking at
rates and we’re comparing the rate
76
00:05:58,650 --> 00:06:02,430
of the indigenous people shown in the
blue bar to non-indigenous people
77
00:06:02,430 --> 00:06:06,869
shown in the green bar. The width of
the bars show the relative populations
78
00:06:06,869 --> 00:06:11,829
in each of those 2 communities. So clearly
there are many more indigenous people,
79
00:06:11,829 --> 00:06:14,980
but a higher fraction of them are also
killed. The bars also show something else.
80
00:06:14,980 --> 00:06:18,049
And that’s what I’ll focus on for the
rest of the talk. There are 2 sections
81
00:06:18,049 --> 00:06:22,159
to each of the 2 bars, a dark section
on the bottom, a lighter section on top.
82
00:06:22,159 --> 00:06:27,779
And what that indicates is what we know
in terms of being able to name people
83
00:06:27,779 --> 00:06:31,249
with their first and last name, their
location and dates of death, and
84
00:06:31,249 --> 00:06:35,560
what we must infer statistically. Now I’m
beginning to touch on the second theme
85
00:06:35,560 --> 00:06:40,949
of my talk: Which is that when we are
studying mass violence and war crimes,
86
00:06:40,949 --> 00:06:48,749
we cannot do statistical or pattern
analysis with raw information.
87
00:06:48,749 --> 00:06:51,950
We must use the tools of mathematical
statistics to understand
88
00:06:51,950 --> 00:06:56,080
what we don’t know! The information
which cannot be observed directly.
89
00:06:56,080 --> 00:07:00,649
We have to estimate that in order to
control for the process of the production
90
00:07:00,649 --> 00:07:04,989
of information. Information doesn’t just
fall out of the sky, the way it does
91
00:07:04,989 --> 00:07:10,359
for industry. If I’m running an ISP I know
every packet that runs through my routers.
92
00:07:10,359 --> 00:07:14,959
That’s not how the social world works. In
order to find information about killings
93
00:07:14,959 --> 00:07:17,889
we have to hear about that killing from
someone, we have to investigate,
94
00:07:17,889 --> 00:07:22,119
we have to find the human remains.
And if we can’t observe the killing
95
00:07:22,119 --> 00:07:28,130
we won’t hear about it and many killings
are hidden. In my team we have a kind of
96
00:07:28,130 --> 00:07:33,760
catch phrase: that the world… if a lawyer
is killed in a big city at high noon
97
00:07:33,760 --> 00:07:38,259
the world knows about it before
dinner time. Every single time.
98
00:07:38,259 --> 00:07:41,850
But when a rural peasant is killed 3-days
walk from a road in the dead of night,
99
00:07:41,850 --> 00:07:45,489
we’re unlikely to ever hear. And
technology is not changing this.
100
00:07:45,489 --> 00:07:48,899
I’ll talk later about that technology is
actually making the problem worse.
101
00:07:48,899 --> 00:07:53,470
So, let’s get back to Guatemala
and just conclude
102
00:07:53,470 --> 00:07:57,950
that the little vertical bars, little
vertical lines at the top of each bar
103
00:07:57,950 --> 00:08:03,079
indicate the confidence interval. Which is
similar to what lay people sometimes call
104
00:08:03,079 --> 00:08:07,199
a margin of error. It is our level of
uncertainty about each of those estimates
105
00:08:07,199 --> 00:08:10,960
and you’ll notice that the uncertainty
is much, much smaller than
106
00:08:10,960 --> 00:08:14,509
the difference between the 2 bars. The
uncertainty does not affect our ability
107
00:08:14,509 --> 00:08:17,970
to draw the conclusion that there
was a spectacular difference
108
00:08:17,970 --> 00:08:21,900
in the mortality rates between the
people who were the hypothesized
109
00:08:21,900 --> 00:08:26,630
target of genocide and those who were not.
110
00:08:26,630 --> 00:08:30,520
Now the data: first we
had the census of 1981,
111
00:08:30,520 --> 00:08:35,339
this was a crucial piece. I think there’s
very interesting questions to ask
112
00:08:35,339 --> 00:08:39,609
about why the Government of Guatemala
conducted a census on the eve of
113
00:08:39,609 --> 00:08:44,540
committing a genocide. There is excellent
work done by historical demographers
114
00:08:44,540 --> 00:08:47,950
about the use of censuses in mass
violence. It has been common
115
00:08:47,950 --> 00:08:52,880
throughout history. Similarly,
or excuse me, in parallel
116
00:08:52,880 --> 00:08:57,420
there were 4 very large
projects. First, the CIIDH
117
00:08:57,420 --> 00:09:01,600
– a group of non-Governmental
Human Rights groups –
118
00:09:01,600 --> 00:09:06,610
collected 1240 records of deaths
in this three-county region.
119
00:09:06,610 --> 00:09:11,750
Next, the Catholic Church collected
a bit fewer than 800 deaths.
120
00:09:11,750 --> 00:09:16,539
The truth commission – the Comisión
para el Esclarecimiento Histórico (CEH) –
121
00:09:16,539 --> 00:09:22,000
conducted a really big research
project in the late 1990s and
122
00:09:22,000 --> 00:09:25,810
of that we got information about a little
bit more than a thousand deaths.
123
00:09:25,810 --> 00:09:30,450
And then the National Program for
Compensation is very, very large
124
00:09:30,450 --> 00:09:35,370
and gave us about 4700
records of deaths.
125
00:09:35,370 --> 00:09:40,659
Now, this is interesting
but this is not unique.
126
00:09:40,659 --> 00:09:45,769
Many of the deaths are reported in common
across those data sources and so…
127
00:09:45,769 --> 00:09:49,490
we think about this in terms of a Venn
diagram. We think of: how did these
128
00:09:49,490 --> 00:09:54,329
different data sets intersect with each
other or collide with each other. And
129
00:09:54,329 --> 00:09:59,130
we can diagram that as in the sense
of these 3 white circles intersecting.
130
00:09:59,130 --> 00:10:05,610
But as I mentioned earlier we’re also
interested in what we have not observed.
131
00:10:05,610 --> 00:10:09,490
And this is crucial for us because
when we’re thinking about
132
00:10:09,490 --> 00:10:13,420
how much information we have, we have to
distinguish between the world on the left,
133
00:10:13,420 --> 00:10:17,200
in which our intersecting circles
cover about a third of the reality,
134
00:10:17,200 --> 00:10:21,829
versus the world on the right where our
intersecting circles cover all of reality.
135
00:10:21,829 --> 00:10:26,390
These are very different worlds; and the
reason they’re so different is not simply
136
00:10:26,390 --> 00:10:29,710
because we want to know the magnitude,
not simply because we want to know
137
00:10:29,710 --> 00:10:34,490
the total number of killings. That’s
important – but even more important:
138
00:10:34,490 --> 00:10:40,160
we have to know that we’ve covered,
we’ve estimated in equal proportions
139
00:10:40,160 --> 00:10:44,430
the two parties. We have to estimate in
equal proportions the number of deaths
140
00:10:44,430 --> 00:10:48,340
of non-indigenous people and the
number of deaths of indigenous people.
141
00:10:48,340 --> 00:10:51,510
Because if we don’t get those
estimates correct our comparison
142
00:10:51,510 --> 00:10:56,080
of their mortality rates will be biased.
Our story will be wrong. We will fail
143
00:10:56,080 --> 00:11:01,840
to speak Truth to Power. We can’t have
that. So what do we do? Algebra!
144
00:11:01,840 --> 00:11:06,390
Algebra is our friend. So I’m gonna
give you just a tiny taste of how we
145
00:11:06,390 --> 00:11:09,650
solve this problem and I’m going to
introduce a series of assumptions.
146
00:11:09,650 --> 00:11:13,279
Those of you who would like to debate
those assumptions: I invite you to join me
147
00:11:13,279 --> 00:11:18,359
after the talk and we will talk endlessly
and tediously about capture heterogeneity.
148
00:11:18,359 --> 00:11:22,240
But in the short term,
149
00:11:22,240 --> 00:11:27,940
we have a universe N of total killings in
a specific time/space/ethnicity/location.
150
00:11:27,940 --> 00:11:30,690
And of that we have 2 projects A and B.
151
00:11:30,690 --> 00:11:34,619
A captures some number of
deaths from the universe N,
152
00:11:34,619 --> 00:11:40,169
and the probability with which a death is
captured by project A from the universe N
153
00:11:40,169 --> 00:11:44,600
is by elementary probability theory the
number of deaths documented by A
154
00:11:44,600 --> 00:11:48,740
divided by the unknown number
of deaths in the population N.
155
00:11:48,740 --> 00:11:52,969
Similarly, the probability with which a
death from N is documented by project B
156
00:11:52,969 --> 00:11:58,149
is B over N, and this is the cool part:
the probability with which a death
157
00:11:58,149 --> 00:12:01,949
is documented by both A and B is M.
158
00:12:01,949 --> 00:12:05,579
Now we can put the 2 databases together,
we can compare them. Let’s talk about
159
00:12:05,579 --> 00:12:09,370
the use of random force classifiers
and clustering to do that later.
160
00:12:09,370 --> 00:12:12,489
But we can put the 2 databases together,
compare them, determine the deaths
161
00:12:12,489 --> 00:12:17,429
that are in M – that is in N both
A and B – and divide M by N.
162
00:12:17,429 --> 00:12:23,060
But, also by probability theory, the
probability that a death occurs in M
163
00:12:23,060 --> 00:12:27,740
is equal to the product of
the individual probabilities.
164
00:12:27,740 --> 00:12:31,619
The probability of any compound event, an
event made up of two independent events is
165
00:12:31,619 --> 00:12:36,410
equal to the product of those two
events, so M over N is equal to
166
00:12:36,410 --> 00:12:41,420
A over N times B over N. Solve for N.
167
00:12:41,420 --> 00:12:45,140
Multiply it through by N squared, divide
by M, and we have an estimate of N
168
00:12:45,140 --> 00:12:49,360
which is equal to AB over M. Now, the
lights in my eyes, I can’t see, but I saw
169
00:12:49,360 --> 00:12:52,740
a few light bulbs go off over people’s
heads. And when I showed this proof
170
00:12:52,740 --> 00:12:57,180
to the judge in the trial of General Ríos
171
00:12:57,180 --> 00:13:01,529
I saw a light bulb go on over her head.
172
00:13:01,529 --> 00:13:04,379
It’s a beautiful thing,
it’s a beautiful thing.
173
00:13:04,379 --> 00:13:09,509
applause
174
00:13:09,509 --> 00:13:12,660
So we don’t do it in 2 systems because
that takes a lot of assumptions.
175
00:13:12,660 --> 00:13:16,069
We do it in 4. You will recall that we
have 4 data sources. We organize
176
00:13:16,069 --> 00:13:21,530
the data sources in this format
such that we have an inclusion
177
00:13:21,530 --> 00:13:26,249
and an exclusion pattern in the table on
the left, which… for which we can define
178
00:13:26,249 --> 00:13:29,810
the number of deaths which fall into
each of these intersecting patterns.
179
00:13:29,810 --> 00:13:33,729
And I’ll give you a very quick
metaphor here. The metaphor is:
180
00:13:33,729 --> 00:13:38,239
imagine that you have 2 dark rooms and you
want to assess the size of those 2 rooms
181
00:13:38,239 --> 00:13:42,049
– which room is larger? And the only
tool that you have to assess the size
182
00:13:42,049 --> 00:13:46,359
of those rooms is a handful of little
rubber balls. The little rubber balls
183
00:13:46,359 --> 00:13:50,400
have a property that when they hit each
other they make a sound. makes CLICK sound
184
00:13:50,400 --> 00:13:53,390
So we throw the balls into the first
room and we listen, and we hear
185
00:13:53,390 --> 00:13:57,190
makes several CLICK sounds. We
collect the balls, go to the second room,
186
00:13:57,190 --> 00:14:00,490
throw them with equal force – imagining
a spherical cow of uniform density!
187
00:14:00,490 --> 00:14:03,950
We throw the balls into the second
room with equal force and we hear
188
00:14:03,950 --> 00:14:07,799
makes one CLICK sound
So which room is larger?
189
00:14:07,799 --> 00:14:12,070
The second room, because we hear fewer
collisions, right? Well, the estimation,
190
00:14:12,070 --> 00:14:15,620
the toy example I gave in the previous
slide is the mathematical formalization
191
00:14:15,620 --> 00:14:20,070
of the intuition that fewer
collisions mean a larger space.
192
00:14:20,070 --> 00:14:23,329
And so what we’re doing here is
laying out the pattern of collisions.
193
00:14:23,329 --> 00:14:26,679
Not just the collisions, the pairwise
collisions, but the three-way and
194
00:14:26,679 --> 00:14:31,409
four-way collisions. And that
allows us to make the estimate
195
00:14:31,409 --> 00:14:37,439
that was shown in the bar graph of
the light part of each of the bars. So
196
00:14:37,439 --> 00:14:41,460
we can come back to our conclusion and put
a confidence interval on the estimates.
197
00:14:41,460 --> 00:14:45,910
And the confidence intervals are shown
there. Now I’m gonna move through this
198
00:14:45,910 --> 00:14:50,850
somewhat more quickly to get to the end of
the talk but I wanna put up one more slide
199
00:14:50,850 --> 00:14:56,240
that was used in the testimony
and that is that we divided time
200
00:14:56,240 --> 00:15:01,220
into 16-month periods and
compared the 16-month period of
201
00:15:01,220 --> 00:15:04,580
General Ríos’s governance – now it’s only
16 months ’cause we went April to July,
202
00:15:04,580 --> 00:15:07,679
because it’s only a few days in August, a
few days in March, so we shaved those off,
203
00:15:07,679 --> 00:15:12,310
okay… – 16-month period of General
Ríos’s Government and compared it
204
00:15:12,310 --> 00:15:17,110
to several periods before and after. And
I think that the key observation here
205
00:15:17,110 --> 00:15:21,809
is that the rate of killing
against indigenous people
206
00:15:21,809 --> 00:15:26,729
is substantially higher done under General
Ríos’s Government than under previous
207
00:15:26,729 --> 00:15:33,280
or succeeding governments. But more
importantly the ratio between the two,
208
00:15:33,280 --> 00:15:37,950
the relative risk of being killed as an
indigenous person, was at its peak
209
00:15:37,950 --> 00:15:42,639
during the government of General Ríos.
210
00:15:42,639 --> 00:15:46,709
Have we proven genocide? No.
211
00:15:46,709 --> 00:15:49,870
This is evidence consistent with the
hypothesis that acts of genocide
212
00:15:49,870 --> 00:15:53,539
were committed. The finding of genocide
is a legal finding, not so much
213
00:15:53,539 --> 00:15:58,580
a scientific one. So as scientists,
our job is to provide evidence that
214
00:15:58,580 --> 00:16:02,870
the finders of fact – the judges in this
case – can use in their determination.
215
00:16:02,870 --> 00:16:05,219
This is evidence consistent
with that hypothesis.
216
00:16:05,219 --> 00:16:08,189
Were this evidence otherwise, as
scientists we would say we would
217
00:16:08,189 --> 00:16:11,480
reject the hypothesis that genocide was
committed. However, with this evidence
218
00:16:11,480 --> 00:16:15,370
we find that the evidence,
the data is consistent with
219
00:16:15,370 --> 00:16:18,080
the prosecution’s hypothesis.
220
00:16:18,080 --> 00:16:25,320
So, it worked!
221
00:16:25,320 --> 00:16:29,049
Ríos Montt was convicted on
genocide charges. applause
222
00:16:29,049 --> 00:16:31,359
You can clap!
applause
223
00:16:31,359 --> 00:16:36,359
applause
224
00:16:36,359 --> 00:16:39,499
For a week!
mumbled, surprised laughter
225
00:16:39,499 --> 00:16:42,279
Then the Constitutional Court intervened,
226
00:16:42,279 --> 00:16:44,959
there I know a couple of experts on
Guatemala here in the audience
227
00:16:44,959 --> 00:16:47,839
who can tell you more about why that
happened and exactly what happened.
228
00:16:47,839 --> 00:16:52,669
However, the Constitutional
Court ordered a new trial,
229
00:16:52,669 --> 00:16:59,160
which is at this time scheduled
for the very beginning of 2015.
230
00:16:59,160 --> 00:17:02,970
And I look forward to testifying again,
231
00:17:02,970 --> 00:17:06,820
and again, and again, and again!
232
00:17:06,820 --> 00:17:12,680
applause
233
00:17:12,680 --> 00:17:16,989
Look, but I wanna come back to this point.
Because as a bunch of technologists…
234
00:17:16,989 --> 00:17:21,589
– there is a lot of folks who really like
technology here, I really like it too!
235
00:17:21,589 --> 00:17:25,559
Technology doesn’t get us to science
– you have to have science
236
00:17:25,559 --> 00:17:28,770
to get you to science. Technology helps
you organize the data. It helps you do
237
00:17:28,770 --> 00:17:32,050
all kinds of extremely great and cool
things without which we wouldn’t be able
238
00:17:32,050 --> 00:17:36,480
to even do the science. But you
can’t have just technology!
239
00:17:36,480 --> 00:17:40,970
You can’t just have a bunch of data
and make conclusions. That’s naive,
240
00:17:40,970 --> 00:17:44,529
and you will get the wrong conclusions.
‘The point of rigorous statistics is
241
00:17:44,529 --> 00:17:48,100
to be right’, and there is a little bit of
a caveat there – or to at least know
242
00:17:48,100 --> 00:17:51,620
how uncertain you are. Statistics is often
called the ‘Science of Uncertainty’.
243
00:17:51,620 --> 00:17:55,960
That is actually my favorite
definition of it. So,
244
00:17:55,960 --> 00:18:01,509
I’m going to assume that we
care about getting it right.
245
00:18:01,509 --> 00:18:05,489
No one laughed, that’s good.
246
00:18:05,489 --> 00:18:08,890
Not everyone does, to my distress.
247
00:18:08,890 --> 00:18:11,320
So if you only have some of the data
248
00:18:11,320 --> 00:18:15,490
– and I will argue that we always
only have some of the data –
249
00:18:15,490 --> 00:18:20,449
you need some kind of model that will tell
you the relationship between your data
250
00:18:20,449 --> 00:18:23,989
and the real world.
Statisticians call that an inference.
251
00:18:23,989 --> 00:18:26,200
In order to get from here to there
you’re gonna need some kind of
252
00:18:26,200 --> 00:18:30,469
probability model that tells you
why your data is like the world,
253
00:18:30,469 --> 00:18:33,960
or in what sense you have to tweet,
twiddle and do algebra with your data
254
00:18:33,960 --> 00:18:39,309
to get from what you can
observe to what is actually true.
255
00:18:39,309 --> 00:18:42,690
And statistics is about comparisons.
Yeah, we get a big number and
256
00:18:42,690 --> 00:18:46,169
journalists love the big number; but
it’s really about these relationships
257
00:18:46,169 --> 00:18:50,609
and patterns! So to get those
relationships and patterns,
258
00:18:50,609 --> 00:18:53,560
in order for them to be right, in order
for our answer to be correct,
259
00:18:53,560 --> 00:18:57,439
every one of the estimates we make
for every point in the pattern
260
00:18:57,439 --> 00:19:01,700
has to be right. It’s a hard
problem. It’s a hard problem.
261
00:19:01,700 --> 00:19:05,070
And what I worry about is that
we have come into this world
262
00:19:05,070 --> 00:19:09,400
in which people throw the notion of Big
Data around as though the data allows us
263
00:19:09,400 --> 00:19:14,230
to make an end-run around problems
of sampling and modeling. It doesn’t.
264
00:19:14,230 --> 00:19:19,120
So as technologist, the reason I’m,
you know, ranting at you guys about it
265
00:19:19,120 --> 00:19:24,540
is that it’s very tempting to have a lot
of data and think you have an answer!
266
00:19:24,540 --> 00:19:30,580
And it’s even more tempting because
in industry context you might be right.
267
00:19:30,580 --> 00:19:36,739
Not so much in Human Rights, not so
much. Violence is a hidden process.
268
00:19:36,739 --> 00:19:39,960
The people who commit violence have
an enormous commitment to hiding it,
269
00:19:39,960 --> 00:19:44,420
distorting it, explaining it in different
ways. All of those things dramatically
270
00:19:44,420 --> 00:19:48,350
affect the information that is produced
from the violence that we’re going to use
271
00:19:48,350 --> 00:19:53,730
to do our analysis. So we usually
don’t know what we don’t know
272
00:19:53,730 --> 00:19:58,220
in Human Rights data collection.
And that means that we don’t know
273
00:19:58,220 --> 00:20:03,829
if what we don’t know is systematically
different from what we do know.
274
00:20:03,829 --> 00:20:06,270
Maybe we know about all the lawyers
and we don’t know about the people
275
00:20:06,270 --> 00:20:10,070
in the countryside. Maybe we know
about all the indigenous people
276
00:20:10,070 --> 00:20:14,130
and not the non-indigenous people.
If that were true, the argument
277
00:20:14,130 --> 00:20:17,980
that I just made would be merely
an artifact of the reporting process
278
00:20:17,980 --> 00:20:21,740
rather than some true analysis. Now
we did the estimations why I believe
279
00:20:21,740 --> 00:20:25,009
we can reject that critique, but that’s
what we have to worry about.
280
00:20:25,009 --> 00:20:28,860
And let’s go back to the Venn diagram
and say: which of these is accurate?
281
00:20:28,860 --> 00:20:32,840
It’s not just for one of the
points in our pattern analysis.
282
00:20:32,840 --> 00:20:36,500
The problem is that we’re
going to compare things.
283
00:20:36,500 --> 00:20:40,890
As in Peru where we compared killings
committed by the Peruvian army against
284
00:20:40,890 --> 00:20:44,860
killings committed by the Maoist Guerillas
with the Sendero Luminoso. And we found
285
00:20:44,860 --> 00:20:51,460
there that in fact we knew very little
about what the Sendero Luminoso had done.
286
00:20:51,460 --> 00:20:55,779
Whereas we knew almost everything
what the Peruvian army had done.
287
00:20:55,779 --> 00:20:57,970
This is called the coverage rate.
The rate between what we know and
288
00:20:57,970 --> 00:21:02,750
what we don’t know. And
raw data, however big,
289
00:21:02,750 --> 00:21:07,510
does not get us to patterns.
And here is a bunch of…
290
00:21:07,510 --> 00:21:11,569
kinds of raw data that I’ve used
and that I really enjoy using.
291
00:21:11,569 --> 00:21:14,270
You know – truth commission testimonies,
UN investigations, press articles,
292
00:21:14,270 --> 00:21:18,309
SMS messages, crowdsourcing, NGO
documentation, social media feeds,
293
00:21:18,309 --> 00:21:21,180
perpetrator records, government archives,
state agency registries – I know those
294
00:21:21,180 --> 00:21:23,570
sound all the same but they actually
turn out to be slightly different.
295
00:21:23,570 --> 00:21:28,340
Happy to talk in tedious detail! Refugee
Camp records, any non-random sample.
296
00:21:28,340 --> 00:21:31,990
All of those are gonna take
some kind of probability model
297
00:21:31,990 --> 00:21:36,070
and we don’t have that many
probability models to use. So
298
00:21:36,070 --> 00:21:40,330
raw data is great for cases – but
it doesn’t get you to patterns.
299
00:21:40,330 --> 00:21:45,120
And patterns – again – patterns are
the thing that allow us to do analysis.
300
00:21:45,120 --> 00:21:49,289
They are the thing… the patterns are what
get us to something that we can use
301
00:21:49,289 --> 00:21:53,629
to help prosecutors, advocates and the…
302
00:21:53,629 --> 00:21:56,409
and the victims themselves.
303
00:21:56,409 --> 00:22:00,589
I gave a version of this talk, a
much earlier version of this talk
304
00:22:00,589 --> 00:22:04,630
several years ago in Medellín, Columbia.
I’ve worked a lot in Columbia,
305
00:22:04,630 --> 00:22:07,670
it’s really… it’s a great place to
work. There’s really terrific
306
00:22:07,670 --> 00:22:13,569
Victims Rights groups there.
And a woman from a township,
307
00:22:13,569 --> 00:22:17,310
smaller than a county, near to Medellín
came up to me after the talk and she said:
308
00:22:17,310 --> 00:22:21,150
“You know, a lot of people… you
know I’m a Human Rights activist,
309
00:22:21,150 --> 00:22:25,309
my job is to collect data, I tell stories
about people who have suffered.
310
00:22:25,309 --> 00:22:28,210
But there are people in my
village I know who have had
311
00:22:28,210 --> 00:22:32,910
people in their families disappeared and
they’re never gonna talk about, ever.
312
00:22:32,910 --> 00:22:38,090
We’re never going to be able to use
their names, because they are afraid.”
313
00:22:38,090 --> 00:22:45,349
We can’t name the victims. At
least we’d better count them.
314
00:22:45,349 --> 00:22:49,520
So about that counting: there’s
3 ways to do it right. You can have
315
00:22:49,520 --> 00:22:54,430
a perfect census – you can have all the
data. Yeah it’s nice, good luck with that.
316
00:22:54,430 --> 00:22:58,910
You can have a random sample
of the population - that’s hard!
317
00:22:58,910 --> 00:23:03,029
Sometimes doable but very hard.
In my experience we rarely interview
318
00:23:03,029 --> 00:23:07,140
victims of homicide, very rarely.
Laughing
319
00:23:07,140 --> 00:23:09,640
And that means there’s a complicated
probability relationship between
320
00:23:09,640 --> 00:23:13,670
the person you sampled, the interview
and the death that they talk to you about.
321
00:23:13,670 --> 00:23:17,300
Or you can do some kind of posterior
modeling of the sampling process which is…
322
00:23:17,300 --> 00:23:21,260
which is in essence what
I proposed in the earlier slide.
323
00:23:21,260 --> 00:23:25,020
So what can we do with raw data,
guys? We can collect a bunch of…
324
00:23:25,020 --> 00:23:28,930
We can say that a case exists. Ok
– that’s actually important! We can say:
325
00:23:28,930 --> 00:23:34,409
“Something happened” with raw data. We can
say: “We know something about that case".
326
00:23:34,409 --> 00:23:38,250
We can say: “There were 100 victims
in that case or at least 100 victims
327
00:23:38,250 --> 00:23:41,570
in that case”, if we can name 100 people.
328
00:23:41,570 --> 00:23:46,390
But we can’t do comparisons: “This
is the biggest massacre this year”.
329
00:23:46,390 --> 00:23:48,350
We don’t really know. Because we
don’t know about that massacres
330
00:23:48,350 --> 00:23:53,910
we don’t know about. No patterns. Don’t
talk about the hot spot of violence.
331
00:23:53,910 --> 00:23:59,420
No, we don’t know that. Happy to talk
more about that if we gather after,
332
00:23:59,420 --> 00:24:06,439
but I wanna come to a close here with
the importance of getting it right.
333
00:24:06,439 --> 00:24:11,380
I’ve talked about one case today. This
is another case, the case of this man:
334
00:24:11,380 --> 00:24:16,320
Edgar Fernando García. Mr. García was
a student Labor leader in Guatemala
335
00:24:16,320 --> 00:24:19,800
early in the 1980s. He left
his office in February 1984
336
00:24:19,800 --> 00:24:24,470
– did not come home. People reported
later that they saw someone
337
00:24:24,470 --> 00:24:28,810
shoving Mr. García into a
vehicle and driving away.
338
00:24:28,810 --> 00:24:33,900
His widow became a very important
Human Rights activist in Guatemala
339
00:24:33,900 --> 00:24:38,570
and now she’s a very important, and
in my opinion impressive politician.
340
00:24:38,570 --> 00:24:42,240
And there’s her infant daughter. She
continued to struggle to find out
341
00:24:42,240 --> 00:24:46,130
what had happened to
Mr. García for decades.
342
00:24:46,130 --> 00:24:50,400
And in 2006 documents came to light
in the National Archives of the…
343
00:24:50,400 --> 00:24:54,429
excuse me, the Historical Archives
of the national Police, showing that
344
00:24:54,429 --> 00:24:59,320
the Police had realized an operation
in the area of Mr. García’s office
345
00:24:59,320 --> 00:25:01,930
and it was very likely that
they had disappeared him.
346
00:25:01,930 --> 00:25:07,400
These 2 guys up here in the upper
right were Police officers in that area;
347
00:25:07,400 --> 00:25:11,359
they were arrested, charged with the
disappearance of Mister García and
348
00:25:11,359 --> 00:25:15,620
convicted. Part of the evidence used to
convict them was communications meta data
349
00:25:15,620 --> 00:25:19,510
showing that documents
flowed through the archive.
350
00:25:19,510 --> 00:25:23,699
I mean paper communications! We coded
it by hand. We went through and read
351
00:25:23,699 --> 00:25:28,459
the ‘From’ and ‘To’ lines
from every Memo. And
352
00:25:28,459 --> 00:25:34,229
they were convicted in 2010
and after that conviction
353
00:25:34,229 --> 00:25:38,699
Mr. García’s infant daughter – now
a grown woman – was clearly joyful.
354
00:25:38,699 --> 00:25:42,730
Justice brings closure to a family
that never knows when to start talking
355
00:25:42,730 --> 00:25:48,059
about someone in the past tense.
Perhaps even more powerfully:
356
00:25:48,059 --> 00:25:52,319
those guys’ grand boss, their boss's
boss, Colonel Héctor Bol de la Cruz,
357
00:25:52,319 --> 00:25:58,439
this man here, was convicted
of Mr. García’s disappearance
358
00:25:58,439 --> 00:26:02,069
in September this year [2013].
applause
359
00:26:02,069 --> 00:26:07,610
applause
360
00:26:07,610 --> 00:26:10,789
I don’t know if any of you have
ever been dissident students,
361
00:26:10,789 --> 00:26:15,330
but if you’ve been dissident students
demonstrating in the street think about
362
00:26:15,330 --> 00:26:19,300
how you would feel if your friends
and comrades were disappeared,
363
00:26:19,300 --> 00:26:23,419
and take a long look at Colonel Bol
de la Cruz. Here is the rest of the stuff
364
00:26:23,419 --> 00:26:25,626
that we will talk about if we gather
afterwards. Thank you very much
365
00:26:25,626 --> 00:26:29,086
for your attention. I really
have enjoyed CCC.
366
00:26:29,086 --> 00:26:36,086
applause
367
00:26:36,086 --> 00:26:47,203
Subtitles created by c3subtitles.de
in the year 2016. Join and help us!