0:00:00.725,0:00:03.836
Like many of you, I'm one of the lucky people.

0:00:03.836,0:00:07.236
I was born to a family where education was pervasive.

0:00:07.236,0:00:11.474
I'm a third-generation PhD, a daughter of two academics.

0:00:11.474,0:00:15.268
In my childhood, I played around in my father's university lab.

0:00:15.268,0:00:19.117
So it was taken for granted that I attend some of the best universities,

0:00:19.117,0:00:22.918
which in turn opened the door to a world of opportunity.

0:00:22.918,0:00:27.038
Unfortunately, most of the people in the world are not so lucky.

0:00:27.038,0:00:30.173
In some parts of the world, for example, South Africa,

0:00:30.173,0:00:32.878
education is just not readily accessible.

0:00:32.878,0:00:35.853
In South Africa, the educational system was constructed

0:00:35.853,0:00:38.726
in the days of apartheid for the white minority.

0:00:38.726,0:00:41.426
And as a consequence, today there is just not enough spots

0:00:41.426,0:00:45.278
for the many more people who want and deserve a high quality education.

0:00:45.278,0:00:49.158
That scarcity led to a crisis in January of this year

0:00:49.158,0:00:50.994
at the University of Johannesburg.

0:00:50.994,0:00:53.125
There were a handful of positions left open

0:00:53.125,0:00:56.094
from the standard admissions process, and the night before

0:00:56.094,0:00:58.654
they were supposed to open that for registration,

0:00:58.654,0:01:02.706
thousands of people lined up outside the gate in a line a mile long,

0:01:02.706,0:01:06.586
hoping to be first in line to get one of those positions.

0:01:06.586,0:01:08.894
When the gates opened, there was a stampede,

0:01:08.894,0:01:12.546
and 20 people were injured and one woman died.

0:01:12.546,0:01:14.486
She was a mother who gave her life

0:01:14.486,0:01:18.549
trying to get her son a chance at a better life.

0:01:18.549,0:01:21.706
But even in parts of the world like the United States

0:01:21.706,0:01:26.062
where education is available, it might not be within reach.

0:01:26.062,0:01:28.734
There has been much discussed in the last few years

0:01:28.734,0:01:30.723
about the rising cost of health care.

0:01:30.723,0:01:33.365
What might not be quite as obvious to people

0:01:33.365,0:01:37.387
is that during that same period the cost of higher education tuition

0:01:37.387,0:01:39.867
has been increasing at almost twice the rate,

0:01:39.867,0:01:44.147
for a total of 559 percent since 1985.

0:01:44.147,0:01:48.681
This makes education unaffordable for many people.

0:01:48.681,0:01:52.482
Finally, even for those who do manage to get the higher education,

0:01:52.482,0:01:55.107
the doors of opportunity might not open.

0:01:55.107,0:01:58.314
Only a little over half of recent college graduates

0:01:58.314,0:02:00.627
in the United States who get a higher education

0:02:00.627,0:02:04.090
actually are working in jobs that require that education.

0:02:04.090,0:02:05.930
This, of course, is not true for the students

0:02:05.930,0:02:07.882
who graduate from the top institutions,

0:02:07.882,0:02:10.514
but for many others, they do not get the value

0:02:10.514,0:02:14.050
for their time and their effort.

0:02:14.050,0:02:17.080
Tom Friedman, in his recent New York Times article,

0:02:17.080,0:02:21.448
captured, in the way that no one else could, the spirit behind our effort.

0:02:21.448,0:02:24.568
He said the big breakthroughs are what happen

0:02:24.568,0:02:28.467
when what is suddenly possible meets what is desperately necessary.

0:02:28.467,0:02:31.088
I've talked about what's desperately necessary.

0:02:31.088,0:02:33.600
Let's talk about what's suddenly possible.

0:02:33.600,0:02:36.719
What's suddenly possible was demonstrated by

0:02:36.719,0:02:38.287
three big Stanford classes,

0:02:38.287,0:02:42.167
each of which had an enrollment of 100,000 people or more.

0:02:42.167,0:02:45.551
So to understand this, let's look at one of those classes,

0:02:45.551,0:02:47.471
the Machine Learning class offered by my colleague

0:02:47.471,0:02:49.200
and cofounder Andrew Ng.

0:02:49.200,0:02:51.519
Andrew teaches one of the bigger Stanford classes.

0:02:51.519,0:02:52.728
It's a Machine Learning class,

0:02:52.728,0:02:56.246
and it has 400 people enrolled every time it's offered.

0:02:56.246,0:02:59.511
When Andrew taught the Machine Learning class to the general public,

0:02:59.511,0:03:02.127
it had 100,000 people registered.

0:03:02.127,0:03:04.136
So to put that number in perspective,

0:03:04.136,0:03:06.495
for Andrew to reach that same size audience

0:03:06.495,0:03:08.321
by teaching a Stanford class,

0:03:08.321,0:03:12.247
he would have to do that for 250 years.

0:03:12.247,0:03:15.733
Of course, he'd get really bored.

0:03:15.733,0:03:18.470
So, having seen the impact of this,

0:03:18.470,0:03:21.598
Andrew and I decided that we needed to really try and scale this up,

0:03:21.598,0:03:25.718
to bring the best quality education to as many people as we could.

0:03:25.718,0:03:27.213
So we formed Coursera,

0:03:27.213,0:03:30.350
whose goal is to take the best courses

0:03:30.350,0:03:33.667
from the best instructors at the best universities

0:03:33.667,0:03:37.695
and provide it to everyone around the world for free.

0:03:37.695,0:03:40.295
We currently have 43 courses on the platform

0:03:40.295,0:03:43.494
from four universities across a range of disciplines,

0:03:43.494,0:03:45.327
and let me show you a little bit of an overview

0:03:45.327,0:03:48.605
of what that looks like.

0:03:48.605,0:03:49.818
(Video) Robert Ghrist: Welcome to Calculus.

0:03:49.818,0:03:51.698
Ezekiel Emanuel: Fifty million people are uninsured.

0:03:51.698,0:03:54.969
Scott Page: Models help us design more effective institutions and policies.

0:03:54.969,0:03:57.377
We get unbelievable segregation.

0:03:57.377,0:03:59.169
Scott Klemmer: So Bush imagined that in the future,

0:03:59.169,0:04:01.547
you'd wear a camera right in the center of your head.

0:04:01.547,0:04:05.801
Mitchell Duneier: Mills wants the student of sociology to develop the quality of mind ...

0:04:05.801,0:04:09.466
RG: Hanging cable takes on the form of a hyperbolic cosine.

0:04:09.466,0:04:12.537
Nick Parlante: For each pixel in the image, set the red to zero.

0:04:12.537,0:04:15.514
Paul Offit: ... Vaccine allowed us to eliminate polio virus.

0:04:15.514,0:04:19.137
Dan Jurafsky: Does Lufthansa serve breakfast and San Jose? Well, that sounds funny.

0:04:19.137,0:04:22.753
Daphne Koller: So this is which coin you pick, and this is the two tosses.

0:04:22.753,0:04:26.440
Andrew Ng: So in large-scale machine learning, we'd like to come up with computational ...

0:04:26.440,0:04:32.049
(Applause)

0:04:32.049,0:04:34.323
DK: It turns out, maybe not surprisingly,

0:04:34.323,0:04:36.561
that students like getting the best content

0:04:36.561,0:04:39.448
from the best universities for free.

0:04:39.448,0:04:41.970
Since we opened the website in February,

0:04:41.970,0:04:46.328
we now have 640,000 students from 190 countries.

0:04:46.328,0:04:48.480
We have 1.5 million enrollments,

0:04:48.480,0:04:51.330
6 million quizzes in the 15 classes that have launched

0:04:51.330,0:04:56.246
so far have been submitted, and 14 million videos have been viewed.

0:04:56.246,0:04:58.764
But it's not just about the numbers,

0:04:58.764,0:05:00.405
it's also about the people.

0:05:00.405,0:05:03.381
Whether it's Akash, who comes from a small town in India

0:05:03.381,0:05:05.556
and would never have access in this case

0:05:05.556,0:05:07.045
to a Stanford-quality course

0:05:07.045,0:05:09.560
and would never be able to afford it.

0:05:09.560,0:05:11.598
Or Jenny, who is a single mother of two

0:05:11.598,0:05:13.565
and wants to hone her skills

0:05:13.565,0:05:16.700
so that she can go back and complete her master's degree.

0:05:16.700,0:05:19.836
Or Ryan, who can't go to school,

0:05:19.836,0:05:21.701
because his immune deficient daughter

0:05:21.701,0:05:25.084
can't be risked to have germs come into the house,

0:05:25.084,0:05:26.924
so he couldn't leave the house.

0:05:26.924,0:05:28.556
I'm really glad to say --

0:05:28.556,0:05:30.808
recently, we've been in correspondence with Ryan --

0:05:30.808,0:05:32.740
that this story had a happy ending.

0:05:32.740,0:05:34.643
Baby Shannon -- you can see her on the left --

0:05:34.643,0:05:35.994
is doing much better now,

0:05:35.994,0:05:40.192
and Ryan got a job by taking some of our courses.

0:05:40.192,0:05:42.436
So what made these courses so different?

0:05:42.436,0:05:46.156
After all, online course content has been available for a while.

0:05:46.156,0:05:49.868
What made it different was that this was real course experience.

0:05:49.868,0:05:51.594
It started on a given day,

0:05:51.594,0:05:55.228
and then the students would watch videos on a weekly basis

0:05:55.228,0:05:57.083
and do homework assignments.

0:05:57.083,0:05:58.874
And these would be real homework assignments

0:05:58.874,0:06:02.178
for a real grade, with a real deadline.

0:06:02.178,0:06:04.234
You can see the deadlines and the usage graph.

0:06:04.234,0:06:06.322
These are the spikes showing

0:06:06.322,0:06:10.111
that procrastination is global phenomenon.

0:06:10.111,0:06:12.687
(Laughter)

0:06:12.687,0:06:14.359
At the end of the course,

0:06:14.359,0:06:16.215
the students got a certificate.

0:06:16.215,0:06:18.375
They could present that certificate

0:06:18.375,0:06:20.528
to a prospective employer and get a better job,

0:06:20.528,0:06:22.588
and we know many students who did.

0:06:22.588,0:06:24.507
Some students took their certificate

0:06:24.507,0:06:27.629
and presented this to an educational institution at which they were enrolled

0:06:27.629,0:06:29.470
for actual college credit.

0:06:29.470,0:06:31.684
So these students were really getting something meaningful

0:06:31.684,0:06:34.518
for their investment of time and effort.

0:06:34.518,0:06:37.073
Let's talk a little bit about some of the components

0:06:37.073,0:06:38.965
that go into these courses.

0:06:38.965,0:06:41.593
The first component is that when you move away

0:06:41.593,0:06:43.890
from the constraints of a physical classroom

0:06:43.890,0:06:46.730
and design content explicitly for an online format,

0:06:46.730,0:06:49.258
you can break away from, for example,

0:06:49.258,0:06:51.673
the monolithic one-hour lecture.

0:06:51.673,0:06:53.458
You can break up the material, for example,

0:06:53.458,0:06:56.834
into these short, modular units of eight to 12 minutes,

0:06:56.834,0:06:59.808
each of which represents a coherent concept.

0:06:59.808,0:07:02.378
Students can traverse this material in different ways,

0:07:02.378,0:07:06.082
depending on their background, their skills or their interests.

0:07:06.082,0:07:08.602
So, for example, some students might benefit

0:07:08.602,0:07:11.362
from a little bit of preparatory material

0:07:11.362,0:07:13.433
that other students might already have.

0:07:13.433,0:07:15.873
Other students might be interested in a particular

0:07:15.873,0:07:18.959
enrichment topic that they want to pursue individually.

0:07:18.959,0:07:22.194
So this format allows us to break away

0:07:22.194,0:07:25.018
from the one-size-fits-all model of education,

0:07:25.018,0:07:29.010
and allows students to follow a much more personalized curriculum.

0:07:29.010,0:07:31.353
Of course, we all know as educators

0:07:31.353,0:07:34.713
that students don't learn by sitting and passively watching videos.

0:07:34.713,0:07:37.658
Perhaps one of the biggest components of this effort

0:07:37.658,0:07:40.250
is that we need to have students

0:07:40.250,0:07:42.659
who practice with the material

0:07:42.659,0:07:45.815
in order to really understand it.

0:07:45.815,0:07:49.083
There's been a range of studies that demonstrate the importance of this.

0:07:49.083,0:07:51.615
This one that appeared in Science last year, for example,

0:07:51.615,0:07:54.447
demonstrates that even simple retrieval practice,

0:07:54.447,0:07:57.239
where students are just supposed to repeat

0:07:57.239,0:07:58.639
what they already learned

0:07:58.639,0:08:00.559
gives considerably improved results

0:08:00.559,0:08:02.828
on various achievement tests down the line

0:08:02.828,0:08:07.132
than many other educational interventions.

0:08:07.132,0:08:10.094
We've tried to build in retrieval practice into the platform,

0:08:10.094,0:08:12.348
as well as other forms of practice in many ways.

0:08:12.348,0:08:16.492
For example, even our videos are not just videos.

0:08:16.492,0:08:18.535
Every few minutes, the video pauses

0:08:18.535,0:08:20.686
and the students get asked a question.

0:08:20.686,0:08:22.907
(Video) SP: ... These four things. Prospect theory, hyperbolic discounting,

0:08:22.907,0:08:25.999
status quo bias, base rate bias. They're all well documented.

0:08:25.999,0:08:28.766
So they're all well documented deviations from rational behavior.

0:08:28.766,0:08:30.390
DK: So here the video pauses,

0:08:30.390,0:08:32.646
and the student types in the answer into the box

0:08:32.646,0:08:35.869
and submits. Obviously they weren't paying attention.

0:08:35.884,0:08:36.753
(Laughter)

0:08:36.753,0:08:38.763
So they get to try again,

0:08:38.763,0:08:41.299
and this time they got it right.

0:08:41.299,0:08:43.492
There's an optional explanation if they want.

0:08:43.492,0:08:47.749
And now the video moves on to the next part of the lecture.

0:08:47.749,0:08:49.627
This is a kind of simple question

0:08:49.627,0:08:51.708
that I as an instructor might ask in class,

0:08:51.708,0:08:54.208
but when I ask that kind of a question in class,

0:08:54.208,0:08:55.508
80 percent of the students

0:08:55.508,0:08:57.374
are still scribbling the last thing I said,

0:08:57.374,0:09:00.695
15 percent are zoned out on Facebook,

0:09:00.695,0:09:03.151
and then there's the smarty pants in the front row

0:09:03.151,0:09:04.510
who blurts out the answer

0:09:04.510,0:09:06.717
before anyone else has had a chance to think about it,

0:09:06.717,0:09:09.589
and I as the instructor am terribly gratified

0:09:09.589,0:09:11.237
that somebody actually knew the answer.

0:09:11.237,0:09:14.029
And so the lecture moves on before, really,

0:09:14.029,0:09:17.558
most of the students have even noticed that a question had been asked.

0:09:17.558,0:09:20.165
Here, every single student

0:09:20.165,0:09:22.949
has to engage with the material.

0:09:22.949,0:09:24.885
And of course these simple retrieval questions

0:09:24.885,0:09:26.547
are not the end of the story.

0:09:26.547,0:09:29.517
One needs to build in much more meaningful practice questions,

0:09:29.517,0:09:31.870
and one also needs to provide the students with feedback

0:09:31.870,0:09:33.533
on those questions.

0:09:33.533,0:09:36.421
Now, how do you grade the work of 100,000 students

0:09:36.421,0:09:39.503
if you do not have 10,000 TAs?

0:09:39.503,0:09:41.857
The answer is, you need to use technology

0:09:41.857,0:09:43.352
to do it for you.

0:09:43.352,0:09:46.000
Now, fortunately, technology has come a long way,

0:09:46.000,0:09:49.268
and we can now grade a range of interesting types of homework.

0:09:49.268,0:09:50.795
In addition to multiple choice

0:09:50.795,0:09:53.948
and the kinds of short answer questions that you saw in the video,

0:09:53.948,0:09:57.208
we can also grade math, mathematical expressions

0:09:57.208,0:09:59.160
as well as mathematical derivations.

0:09:59.160,0:10:02.034
We can grade models, whether it's

0:10:02.034,0:10:04.210
financial models in a business class

0:10:04.210,0:10:07.194
or physical models in a science or engineering class

0:10:07.194,0:10:10.938
and we can grade some pretty sophisticated programming assignments.

0:10:10.938,0:10:12.857
Let me show you one that's actually pretty simple

0:10:12.857,0:10:14.337
but fairly visual.

0:10:14.337,0:10:16.814
This is from Stanford's Computer Science 101 class,

0:10:16.814,0:10:18.418
and the students are supposed to color-correct

0:10:18.418,0:10:20.010
that blurry red image.

0:10:20.010,0:10:22.028
They're typing their program into the browser,

0:10:22.028,0:10:26.086
and you can see they didn't get it quite right, Lady Liberty is still seasick.

0:10:26.086,0:10:29.842
And so, the student tries again, and now they got it right, and they're told that,

0:10:29.842,0:10:32.201
and they can move on to the next assignment.

0:10:32.201,0:10:35.349
This ability to interact actively with the material

0:10:35.349,0:10:37.033
and be told when you're right or wrong

0:10:37.033,0:10:40.159
is really essential to student learning.

0:10:40.159,0:10:42.434
Now, of course we cannot yet grade

0:10:42.434,0:10:45.268
the range of work that one needs for all courses.

0:10:45.268,0:10:48.569
Specifically, what's lacking is the kind of critical thinking work

0:10:48.569,0:10:50.491
that is so essential in such disciplines

0:10:50.491,0:10:54.088
as the humanities, the social sciences, business and others.

0:10:54.088,0:10:56.337
So we tried to convince, for example,

0:10:56.337,0:10:57.953
some of our humanities faculty

0:10:57.953,0:11:00.649
that multiple choice was not such a bad strategy.

0:11:00.649,0:11:02.840
That didn't go over really well.

0:11:02.840,0:11:05.273
So we had to come up with a different solution.

0:11:05.273,0:11:08.347
And the solution we ended up using is peer grading.

0:11:08.347,0:11:10.769
It turns out that previous studies show,

0:11:10.769,0:11:12.441
like this one by Saddler and Good,

0:11:12.441,0:11:14.929
that peer grading is a surprisingly effective strategy

0:11:14.929,0:11:18.143
for providing reproducible grades.

0:11:18.143,0:11:19.913
It was tried only in small classes,

0:11:19.913,0:11:21.400
but there it showed, for example,

0:11:21.400,0:11:23.882
that these student-assigned grades on the y-axis

0:11:23.882,0:11:25.193
are actually very well correlated

0:11:25.193,0:11:27.489
with the teacher-assigned grade on the x-axis.

0:11:27.489,0:11:30.649
What's even more surprising is that self-grades,

0:11:30.649,0:11:32.960
where the students grade their own work critically --

0:11:32.960,0:11:34.697
so long as you incentivize them properly

0:11:34.697,0:11:36.635
so they can't give themselves a perfect score --

0:11:36.635,0:11:39.826
are actually even better correlated with the teacher grades.

0:11:39.826,0:11:41.433
And so this is an effective strategy

0:11:41.433,0:11:43.537
that can be used for grading at scale,

0:11:43.537,0:11:46.273
and is also a useful learning strategy for the students,

0:11:46.273,0:11:48.528
because they actually learn from the experience.

0:11:48.528,0:11:53.177
So we now have the largest peer-grading pipeline ever devised,

0:11:53.177,0:11:55.681
where tens of thousands of students

0:11:55.681,0:11:56.879
are grading each other's work,

0:11:56.879,0:11:59.948
and quite successfully, I have to say.

0:11:59.948,0:12:02.208
But this is not just about students

0:12:02.208,0:12:05.249
sitting alone in their living room working through problems.

0:12:05.249,0:12:07.056
Around each one of our courses,

0:12:07.056,0:12:09.216
a community of students had formed,

0:12:09.216,0:12:11.096
a global community of people

0:12:11.096,0:12:13.628
around a shared intellectual endeavor.

0:12:13.628,0:12:16.280
What you see here is a self-generated map

0:12:16.280,0:12:19.241
from students in our Princeton Sociology 101 course,

0:12:19.241,0:12:22.000
where they have put themselves on a world map,

0:12:22.000,0:12:24.960
and you can really see the global reach of this kind of effort.

0:12:24.960,0:12:29.527
Students collaborated in these courses in a variety of different ways.

0:12:29.527,0:12:32.166
First of all, there was a question and answer forum,

0:12:32.166,0:12:34.310
where students would pose questions,

0:12:34.310,0:12:36.734
and other students would answer those questions.

0:12:36.734,0:12:38.447
And the really amazing thing is,

0:12:38.447,0:12:40.117
because there were so many students,

0:12:40.117,0:12:42.482
it means that even if a student posed a question

0:12:42.482,0:12:44.114
at 3 o'clock in the morning,

0:12:44.114,0:12:45.696
somewhere around the world,

0:12:45.696,0:12:47.770
there would be somebody who was awake

0:12:47.770,0:12:50.083
and working on the same problem.

0:12:50.083,0:12:52.041
And so, in many of our courses,

0:12:52.041,0:12:54.370
the median response time for a question

0:12:54.370,0:12:57.788
on the question and answer forum was 22 minutes.

0:12:57.788,0:13:02.365
Which is not a level of service I have ever offered to my Stanford students.

0:13:02.365,0:13:03.706
(Laughter)

0:13:03.706,0:13:05.648
And you can see from the student testimonials

0:13:05.648,0:13:07.335
that students actually find

0:13:07.335,0:13:09.856
that because of this large online community,

0:13:09.856,0:13:12.455
they got to interact with each other in many ways

0:13:12.455,0:13:16.648
that were deeper than they did in the context of the physical classroom.

0:13:16.648,0:13:18.992
Students also self-assembled,

0:13:18.992,0:13:20.855
without any kind of intervention from us,

0:13:20.855,0:13:22.758
into small study groups.

0:13:22.758,0:13:25.120
Some of these were physical study groups

0:13:25.120,0:13:26.946
along geographical constraints

0:13:26.946,0:13:29.668
and met on a weekly basis to work through problem sets.

0:13:29.668,0:13:31.568
This is the San Francisco study group,

0:13:31.568,0:13:33.887
but there were ones all over the world.

0:13:33.887,0:13:35.919
Others were virtual study groups,

0:13:35.919,0:13:38.908
sometimes along language lines or along cultural lines,

0:13:38.908,0:13:40.352
and on the bottom left there,

0:13:40.352,0:13:44.148
you see our multicultural universal study group

0:13:44.148,0:13:45.911
where people explicitly wanted to connect

0:13:45.911,0:13:48.917
with people from other cultures.

0:13:48.917,0:13:51.028
There are some tremendous opportunities

0:13:51.028,0:13:54.353
to be had from this kind of framework.

0:13:54.353,0:13:58.007
The first is that it has the potential of giving us

0:13:58.007,0:14:00.441
a completely unprecedented look

0:14:00.441,0:14:02.730
into understanding human learning.

0:14:02.730,0:14:06.193
Because the data that we can collect here is unique.

0:14:06.193,0:14:10.202
You can collect every click, every homework submission,

0:14:10.202,0:14:14.565
every forum post from tens of thousands of students.

0:14:14.565,0:14:16.908
So you can turn the study of human learning

0:14:16.908,0:14:18.841
from the hypothesis-driven mode

0:14:18.841,0:14:21.699
to the data-driven mode, a transformation that,

0:14:21.699,0:14:24.740
for example, has revolutionized biology.

0:14:24.740,0:14:28.164
You can use these data to understand fundamental questions

0:14:28.164,0:14:30.044
like, what are good learning strategies

0:14:30.044,0:14:32.740
that are effective versus ones that are not?

0:14:32.740,0:14:34.980
And in the context of particular courses,

0:14:34.980,0:14:36.517
you can ask questions

0:14:36.517,0:14:39.772
like, what are some of the misconceptions that are more common

0:14:39.772,0:14:41.949
and how do we help students fix them?

0:14:41.949,0:14:43.373
So here's an example of that,

0:14:43.373,0:14:45.389
also from Andrew's Machine Learning class.

0:14:45.389,0:14:47.597
This is a distribution of wrong answers

0:14:47.597,0:14:49.207
to one of Andrew's assignments.

0:14:49.207,0:14:51.100
The answers happen to be pairs of numbers,

0:14:51.100,0:14:53.371
so you can draw them on this two-dimensional plot.

0:14:53.371,0:14:57.149
Each of the little crosses that you see is a different wrong answer.

0:14:57.149,0:14:59.555
The big cross at the top left

0:14:59.555,0:15:01.703
is where 2,000 students

0:15:01.703,0:15:04.748
gave the exact same wrong answer.

0:15:04.748,0:15:07.075
Now, if two students in a class of 100

0:15:07.075,0:15:08.362
give the same wrong answer,

0:15:08.362,0:15:09.713
you would never notice.

0:15:09.713,0:15:12.273
But when 2,000 students give the same wrong answer,

0:15:12.273,0:15:13.970
it's kind of hard to miss.

0:15:13.970,0:15:16.162
So Andrew and his students went in,

0:15:16.162,0:15:17.682
looked at some of those assignments,

0:15:17.682,0:15:21.770
understood the root cause of the misconception,

0:15:21.770,0:15:24.290
and then they produced a targeted error message

0:15:24.290,0:15:26.539
that would be provided to every student

0:15:26.539,0:15:28.718
whose answer fell into that bucket,

0:15:28.718,0:15:30.802
which means that students who made that same mistake

0:15:30.802,0:15:32.828
would now get personalized feedback

0:15:32.828,0:15:37.227
telling them how to fix their misconception much more effectively.

0:15:37.227,0:15:41.038
So this personalization is something that one can then build

0:15:41.038,0:15:44.178
by having the virtue of large numbers.

0:15:44.178,0:15:46.490
Personalization is perhaps

0:15:46.490,0:15:48.913
one of the biggest opportunities here as well,

0:15:48.913,0:15:51.258
because it provides us with the potential

0:15:51.258,0:15:53.948
of solving a 30-year-old problem.

0:15:53.948,0:15:57.297
Educational researcher Benjamin Bloom, in 1984,

0:15:57.297,0:15:59.548
posed what's called the 2 sigma problem,

0:15:59.548,0:16:02.610
which he observed by studying three populations.

0:16:02.610,0:16:06.218
The first is the population that studied in a lecture-based classroom.

0:16:06.218,0:16:08.995
The second is a population of students that studied

0:16:08.995,0:16:10.714
using a standard lecture-based classroom,

0:16:10.714,0:16:12.794
but with a mastery-based approach,

0:16:12.794,0:16:14.714
so the students couldn't move on to the next topic

0:16:14.714,0:16:18.068
before demonstrating mastery of the previous one.

0:16:18.068,0:16:20.362
And finally, there was a population of students

0:16:20.362,0:16:24.890
that were taught in a one-on-one instruction using a tutor.

0:16:24.890,0:16:28.162
The mastery-based population was a full standard deviation,

0:16:28.162,0:16:30.450
or sigma, in achievement scores better

0:16:30.450,0:16:32.844
than the standard lecture-based class,

0:16:32.844,0:16:34.988
and the individual tutoring gives you 2 sigma

0:16:34.988,0:16:36.818
improvement in performance.

0:16:36.818,0:16:38.281
To understand what that means,

0:16:38.281,0:16:40.114
let's look at the lecture-based classroom,

0:16:40.114,0:16:43.033
and let's pick the median performance as a threshold.

0:16:43.033,0:16:44.371
So in a lecture-based class,

0:16:44.371,0:16:48.250
half the students are above that level and half are below.

0:16:48.250,0:16:50.348
In the individual tutoring instruction,

0:16:50.348,0:16:55.149
98 percent of the students are going to be above that threshold.

0:16:55.149,0:16:59.069
Imagine if we could teach so that 98 percent of our students

0:16:59.069,0:17:01.267
would be above average.

0:17:01.267,0:17:04.690
Hence, the 2 sigma problem.

0:17:04.690,0:17:07.089
Because we cannot afford, as a society,

0:17:07.089,0:17:10.161
to provide every student with an individual human tutor.

0:17:10.161,0:17:12.410
But maybe we can afford to provide each student

0:17:12.410,0:17:14.429
with a computer or a smartphone.

0:17:14.429,0:17:16.618
So the question is, how can we use technology

0:17:16.618,0:17:19.993
to push from the left side of the graph, from the blue curve,

0:17:19.993,0:17:22.731
to the right side with the green curve?

0:17:22.731,0:17:25.068
Mastery is easy to achieve using a computer,

0:17:25.068,0:17:26.473
because a computer doesn't get tired

0:17:26.473,0:17:29.546
of showing you the same video five times.

0:17:29.546,0:17:32.797
And it doesn't even get tired of grading the same work multiple times,

0:17:32.802,0:17:35.828
we've seen that in many of the examples that I've shown you.

0:17:35.828,0:17:37.682
And even personalization

0:17:37.682,0:17:39.818
is something that we're starting to see the beginnings of,

0:17:39.818,0:17:43.010
whether it's via the personalized trajectory through the curriculum

0:17:43.010,0:17:46.274
or some of the personalized feedback that we've shown you.

0:17:46.274,0:17:48.762
So the goal here is to try and push,

0:17:48.762,0:17:52.259
and see how far we can get towards the green curve.

0:17:52.259,0:17:57.618
So, if this is so great, are universities now obsolete?

0:17:57.618,0:18:00.610
Well, Mark Twain certainly thought so.

0:18:00.610,0:18:03.155
He said that, "College is a place where a professor's lecture notes

0:18:03.155,0:18:04.858
go straight to the students' lecture notes,

0:18:04.858,0:18:07.234
without passing through the brains of either."

0:18:07.234,0:18:11.281
(Laughter)

0:18:11.281,0:18:13.949
I beg to differ with Mark Twain, though.

0:18:13.949,0:18:16.614
I think what he was complaining about is not

0:18:16.614,0:18:19.364
universities but rather the lecture-based format

0:18:19.364,0:18:22.148
that so many universities spend so much time on.

0:18:22.148,0:18:25.307
So let's go back even further, to Plutarch,

0:18:25.307,0:18:27.534
who said that, "The mind is not a vessel that needs filling,

0:18:27.534,0:18:29.557
but wood that needs igniting."

0:18:29.557,0:18:31.747
And maybe we should spend less time at universities

0:18:31.747,0:18:34.318
filling our students' minds with content

0:18:34.318,0:18:38.118
by lecturing at them, and more time igniting their creativity,

0:18:38.118,0:18:41.373
their imagination and their problem-solving skills

0:18:41.373,0:18:43.871
by actually talking with them.

0:18:43.871,0:18:45.238
So how do we do that?

0:18:45.238,0:18:48.669
We do that by doing active learning in the classroom.

0:18:48.669,0:18:51.118
So there's been many studies, including this one,

0:18:51.118,0:18:53.198
that show that if you use active learning,

0:18:53.198,0:18:55.614
interacting with your students in the classroom,

0:18:55.614,0:18:58.310
performance improves on every single metric --

0:18:58.310,0:19:00.759
on attendance, on engagement and on learning

0:19:00.759,0:19:02.814
as measured by a standardized test.

0:19:02.814,0:19:04.678
You can see, for example, that the achievement score

0:19:04.678,0:19:07.548
almost doubles in this particular experiment.

0:19:07.548,0:19:11.949
So maybe this is how we should spend our time at universities.

0:19:11.949,0:19:16.526
So to summarize, if we could offer a top quality education

0:19:16.526,0:19:18.429
to everyone around the world for free,

0:19:18.429,0:19:21.250
what would that do? Three things.

0:19:21.250,0:19:24.671
First it would establish education as a fundamental human right,

0:19:24.671,0:19:26.037
where anyone around the world

0:19:26.037,0:19:27.958
with the ability and the motivation

0:19:27.958,0:19:29.909
could get the skills that they need

0:19:29.909,0:19:31.494
to make a better life for themselves,

0:19:31.494,0:19:33.511
their families and their communities.

0:19:33.511,0:19:36.142
Second, it would enable lifelong learning.

0:19:36.142,0:19:38.093
It's a shame that for so many people,

0:19:38.093,0:19:41.405
learning stops when we finish high school or when we finish college.

0:19:41.405,0:19:43.886
By having this amazing content be available,

0:19:43.886,0:19:46.629
we would be able to learn something new

0:19:46.629,0:19:47.765
every time we wanted,

0:19:47.765,0:19:49.094
whether it's just to expand our minds

0:19:49.094,0:19:51.053
or it's to change our lives.

0:19:51.053,0:19:54.198
And finally, this would enable a wave of innovation,

0:19:54.198,0:19:57.270
because amazing talent can be found anywhere.

0:19:57.270,0:20:00.278
Maybe the next Albert Einstein or the next Steve Jobs

0:20:00.278,0:20:02.893
is living somewhere in a remote village in Africa.

0:20:02.893,0:20:05.549
And if we could offer that person an education,

0:20:05.549,0:20:07.905
they would be able to come up with the next big idea

0:20:07.905,0:20:10.309
and make the world a better place for all of us.

0:20:10.309,0:20:11.469
Thank you very much.

0:20:11.469,0:20:19.052
(Applause)