Reach into the computer and grab a pixel

The border between our physical world and the digital information surrounding us has been getting thinner and thinner. Designer and engineer Jinha Lee wants to dissolve it altogether. As he demonstrates in this short, gasp-inducing talk, his ideas include a pen that penetrates into a screen to draw 3D models and a computer desktop prototype that lets you reach through the screen to manipulate digital objects.

Throughout the history of computers

we've been striving
to shorten the gap between us

and digital information,

the gap between our physical world

and the world in the screen

where our imagination can go wild.

And this gap has become shorter,

shorter, and even shorter,

and now this gap is shortened down

to less than a millimeter,

the thickness of a touch-screen glass,

and the power of computing

has become accessible to everyone.

But I wondered, what if there
could be no boundary at all?

I started to imagine
what this would look like.

First, I created this tool

which penetrates into the digital space,

so when you press it hard on the screen,

it transfers its physical
body into pixels.

Designers can materialize their ideas

directly in 3D,

and surgeons can practice
on virtual organs

underneath the screen.

So with this tool, this
boundary has been broken.

But our two hands still
remain outside the screen.

How can you reach inside and interact

with the digital information

using the full dexterity of our hands?

At Microsoft Applied Sciences,

along with my mentor Cati Boulanger,

I redesigned the computer

and turned a little space
above the keyboard

into a digital workspace.

By combining a transparent
display and depth cameras

for sensing your fingers and face,

now you can lift up your hands
from the keyboard

and reach inside this 3D space

and grab pixels with your bare hands.

(Applause)

Because windows and files have
a position in the real space,

selecting them is as easy
as grabbing a book off your shelf.

Then you can flip through this book

while highlighting the lines, words

on the virtual touch pad
below each floating window.

Architects can stretch
or rotate the models

with their two hands directly.

So in these examples,

we are reaching into the digital world.

But how about reversing its role

and having the digital
information reach us instead?

I'm sure many of us
have had the experience

of buying and returning items online.

But now you don't have to worry about it.

What I got here is an online
augmented fitting room.

This is a view that you get from

head-mounted or see-through display

when the system understands
the geometry of your body.

Taking this idea further,
I started to think,

instead of just seeing
these pixels in our space,

how can we make it physical

so that we can touch and feel it?

What would such a future look like?

At MIT Media Lab,
along with my advisor Hiroshi Ishii

and my collaborator Rehmi Post,

we created this one physical pixel.

Well, in this case, this spherical magnet

acts like a 3D pixel in our space,

which means that both computers and people

can move this object to anywhere

within this little 3D space.

What we did was essentially
canceling gravity

and controlling the movement by combining

magnetic levitation
and mechanical actuation

and sensing technologies.

And by digitally programming the object,

we are liberating
the object from constraints

of time and space, which means that now,

human motions can be
recorded and played back

and left permanently
in the physical world.

So choreography can be taught
physically over distance

and Michael Jordan's famous
shooting can be replicated

over and over as a physical reality.

Students can use this as a tool

to learn about the complex concepts

such as planetary motion, physics,

and unlike computer screens or textbooks,

this is a real, tangible experience

that you can touch and feel,
and it's very powerful.

And what's more exciting

than just turning what's currently
in the computer physical

is to start imagining
how programming the world

will alter even our daily
physical activities.

(Laughter)

As you can see, the digital information

will not just show us something

but it will start directly acting upon us

as a part of our physical surroundings

without disconnecting
ourselves from our world.

Today, we started by talking
about the boundary,

but if we remove this boundary,

the only boundary left is our imagination.

Thank you.

(Applause)