This app knows how you feel — from the look on your face

Our emotions influence every aspect of our lives – how we learn, how we communicate, how we make decisions. Yet they’re absent from our digital lives; the devices and apps we interact with have no way of knowing how we feel. Scientist Rana el Kaliouby aims to change that. She demos a powerful new technology that reads your facial expressions and matches them to corresponding emotions. This “emotion engine” has big implications, she says, and could change not just how we interact with machines — but with each other.

Our emotions influence
every aspect of our lives,

from our health and how we learn,
to how we do business and make decisions,

big ones and small.

Our emotions also influence
how we connect with one another.

We've evolved to live
in a world like this,

but instead, we're living
more and more of our lives like this --

this is the text message
from my daughter last night --

in a world that's devoid of emotion.

So I'm on a mission to change that.

I want to bring emotions
back into our digital experiences.

I started on this path 15 years ago.

I was a computer scientist in Egypt,

and I had just gotten accepted to
a Ph.D. program at Cambridge University.

So I did something quite unusual

for a young newlywed Muslim Egyptian wife:

With the support of my husband,
who had to stay in Egypt,

I packed my bags and I moved to England.

At Cambridge, thousands of miles
away from home,

I realized I was spending
more hours with my laptop

than I did with any other human.

Yet despite this intimacy, my laptop
had absolutely no idea how I was feeling.

It had no idea if I was happy,

having a bad day, or stressed, confused,

and so that got frustrating.

Even worse, as I communicated
online with my family back home,

I felt that all my emotions
disappeared in cyberspace.

I was homesick, I was lonely,
and on some days I was actually crying,

but all I had to communicate
these emotions was this.

(Laughter)

Today's technology
has lots of I.Q., but no E.Q.;

lots of cognitive intelligence,
but no emotional intelligence.

So that got me thinking,

what if our technology
could sense our emotions?

What if our devices could sense
how we felt and reacted accordingly,

just the way an emotionally
intelligent friend would?

Those questions led me and my team

to create technologies that can read
and respond to our emotions,

and our starting point was the human face.

So our human face happens to be
one of the most powerful channels

that we all use to communicate
social and emotional states,

everything from enjoyment, surprise,

empathy and curiosity.

In emotion science, we call each
facial muscle movement an action unit.

So for example, action unit 12,

it's not a Hollywood blockbuster,

it is actually a lip corner pull,
which is the main component of a smile.

Try it everybody. Let's get
some smiles going on.

Another example is action unit 4.
It's the brow furrow.

It's when you draw your eyebrows together

and you create all
these textures and wrinkles.

We don't like them, but it's
a strong indicator of a negative emotion.

So we have about 45 of these action units,

and they combine to express
hundreds of emotions.

Teaching a computer to read
these facial emotions is hard,

because these action units,
they can be fast, they're subtle,

and they combine in many different ways.

So take, for example,
the smile and the smirk.

They look somewhat similar,
but they mean very different things.

(Laughter)

So the smile is positive,

a smirk is often negative.

Sometimes a smirk
can make you become famous.

But seriously, it's important
for a computer to be able

to tell the difference
between the two expressions.

So how do we do that?

We give our algorithms

tens of thousands of examples
of people we know to be smiling,

from different ethnicities, ages, genders,

and we do the same for smirks.

And then, using deep learning,

the algorithm looks for all these
textures and wrinkles

and shape changes on our face,

and basically learns that all smiles
have common characteristics,

all smirks have subtly
different characteristics.

And the next time it sees a new face,

it essentially learns that

this face has the same
characteristics of a smile,

and it says, "Aha, I recognize this.
This is a smile expression."

So the best way to demonstrate
how this technology works

is to try a live demo,

so I need a volunteer,
preferably somebody with a face.

(Laughter)

Cloe's going to be our volunteer today.

So over the past five years, we've moved
from being a research project at MIT

to a company,

where my team has worked really hard
to make this technology work,

as we like to say, in the wild.

And we've also shrunk it so that
the core emotion engine

works on any mobile device
with a camera, like this iPad.

So let's give this a try.

As you can see, the algorithm
has essentially found Cloe's face,

so it's this white bounding box,

and it's tracking the main
feature points on her face,

so her eyebrows, her eyes,
her mouth and her nose.

The question is,
can it recognize her expression?

So we're going to test the machine.

So first of all, give me your poker face.
Yep, awesome. (Laughter)

And then as she smiles,
this is a genuine smile, it's great.

So you can see the green bar
go up as she smiles.

Now that was a big smile.

Can you try a subtle smile
to see if the computer can recognize?

It does recognize subtle smiles as well.

We've worked really hard
to make that happen.

And then eyebrow raised,
indicator of surprise.

Brow furrow, which is
an indicator of confusion.

Frown. Yes, perfect.

So these are all the different
action units. There's many more of them.

This is just a slimmed-down demo.

But we call each reading
an emotion data point,

and then they can fire together
to portray different emotions.

So on the right side of the demo --
look like you're happy.

So that's joy. Joy fires up.

And then give me a disgust face.

Try to remember what it was like
when Zayn left One Direction.

(Laughter)

Yeah, wrinkle your nose. Awesome.

And the valence is actually quite
negative, so you must have been a big fan.

So valence is how positive
or negative an experience is,

and engagement is how
expressive she is as well.

So imagine if Cloe had access
to this real-time emotion stream,

and she could share it
with anybody she wanted to.

Thank you.

(Applause)

So, so far, we have amassed
12 billion of these emotion data points.

It's the largest emotion
database in the world.

We've collected it
from 2.9 million face videos,

people who have agreed
to share their emotions with us,

and from 75 countries around the world.

It's growing every day.

It blows my mind away

that we can now quantify something
as personal as our emotions,

and we can do it at this scale.

So what have we learned to date?

Gender.

Our data confirms something
that you might suspect.

Women are more expressive than men.

Not only do they smile more,
their smiles last longer,

and we can now really quantify
what it is that men and women

respond to differently.

Let's do culture: So in the United States,

women are 40 percent
more expressive than men,

but curiously, we don't see any difference
in the U.K. between men and women.

(Laughter)

Age: People who are 50 years and older

are 25 percent more emotive
than younger people.

Women in their 20s smile a lot more
than men the same age,

perhaps a necessity for dating.

But perhaps what surprised us
the most about this data

is that we happen
to be expressive all the time,

even when we are sitting
in front of our devices alone,

and it's not just when we're watching
cat videos on Facebook.

We are expressive when we're emailing,
texting, shopping online,

or even doing our taxes.

Where is this data used today?

In understanding how we engage with media,

so understanding virality
and voting behavior;

and also empowering
or emotion-enabling technology,

and I want to share some examples
that are especially close to my heart.

Emotion-enabled wearable glasses
can help individuals

who are visually impaired
read the faces of others,

and it can help individuals
on the autism spectrum interpret emotion,

something that they really struggle with.

In education, imagine
if your learning apps

sense that you're confused and slow down,

or that you're bored, so it's sped up,

just like a great teacher
would in a classroom.

What if your wristwatch tracked your mood,

or your car sensed that you're tired,

or perhaps your fridge
knows that you're stressed,

so it auto-locks to prevent you
from binge eating. (Laughter)

I would like that, yeah.

What if, when I was in Cambridge,

I had access to my real-time
emotion stream,

and I could share that with my family
back home in a very natural way,

just like I would've if we were all
in the same room together?

I think five years down the line,

all our devices are going
to have an emotion chip,

and we won't remember what it was like
when we couldn't just frown at our device

and our device would say, "Hmm,
you didn't like that, did you?"

Our biggest challenge is that there are
so many applications of this technology,

my team and I realize that we can't
build them all ourselves,

so we've made this technology available
so that other developers

can get building and get creative.

We recognize that
there are potential risks

and potential for abuse,

but personally, having spent
many years doing this,

I believe that the benefits to humanity

from having emotionally
intelligent technology

far outweigh the potential for misuse.

And I invite you all to be
part of the conversation.

The more people who know
about this technology,

the more we can all have a voice
in how it's being used.

So as more and more
of our lives become digital,

we are fighting a losing battle
trying to curb our usage of devices

in order to reclaim our emotions.

So what I'm trying to do instead
is to bring emotions into our technology

and make our technologies more responsive.

So I want those devices
that have separated us

to bring us back together.

And by humanizing technology,
we have this golden opportunity

to reimagine how we
connect with machines,

and therefore, how we, as human beings,

connect with one another.

Thank you.

(Applause)