See invisible motion, hear silent sounds. Cool? Creepy? We can't decide
-
0:01 - 0:09So over the past few centuries,
microscopes have revolutionized our world. -
0:09 - 0:14They revealed to us a tiny world
of objects, life and structures -
0:14 - 0:17that are too small for us
to see with our naked eyes. -
0:17 - 0:20They are a tremendous contribution
to science and technology. -
0:20 - 0:23Today I'd like to introduce you
to a new type of microscope, -
0:23 - 0:26a microscope for changes.
-
0:26 - 0:29It doesn't use optics
like a regular microscope -
0:29 - 0:31to make small objects bigger,
-
0:31 - 0:35but instead it uses a video camera
and image processing -
0:35 - 0:41to reveal to us the tiniest motions
and color changes in objects and people, -
0:41 - 0:44changes that are impossible
for us to see with our naked eyes. -
0:44 - 0:48And it lets us look at our world
in a completely new way. -
0:48 - 0:50So what do I mean by color changes?
-
0:50 - 0:53Our skin, for example,
changes its color very slightly -
0:53 - 0:55when the blood flows under it.
-
0:55 - 0:58That change is incredibly subtle,
-
0:58 - 1:00which is why, when you
look at other people, -
1:00 - 1:02when you look at the person
sitting next to you, -
1:02 - 1:06you don't see their skin
or their face changing color. -
1:06 - 1:10When we look at this video of Steve here,
it appears to us like a static picture, -
1:10 - 1:14but once we look at this video
through our new, special microscope, -
1:14 - 1:16suddenly we see
a completely different image. -
1:16 - 1:20What you see here are small changes
in the color of Steve's skin, -
1:20 - 1:25magnified 100 times
so that they become visible. -
1:25 - 1:28We can actually see a human pulse.
-
1:28 - 1:31We can see how fast
Steve's heart is beating, -
1:31 - 1:37but we can also see the actual way
that the blood flows in his face. -
1:37 - 1:39And we can do that not just
to visualize the pulse, -
1:39 - 1:43but also to actually
recover our heart rates, -
1:43 - 1:44and measure our heart rates.
-
1:44 - 1:49And we can do it with regular cameras
and without touching the patients. -
1:49 - 1:55So here you see the pulse and heart rate
we extracted from a neonatal baby -
1:55 - 1:57from a video we took
with a regular DSLR camera, -
1:57 - 1:59and the heart rate measurement we get
-
1:59 - 2:04is as accurate as the one you'd get
with a standard monitor in a hospital. -
2:04 - 2:07And it doesn't even have to be
a video we recorded. -
2:07 - 2:10We can do it essentially
with other videos as well. -
2:10 - 2:14So I just took a short clip
from "Batman Begins" here -
2:14 - 2:15just to show Christian Bale's pulse.
-
2:15 - 2:17(Laughter)
-
2:17 - 2:19And you know, presumably
he's wearing makeup, -
2:19 - 2:21the lighting here is kind of challenging,
-
2:21 - 2:24but still, just from the video,
we're able to extract his pulse -
2:24 - 2:26and show it quite well.
-
2:26 - 2:28So how do we do all that?
-
2:28 - 2:33We basically analyze the changes
in the light that are recorded -
2:33 - 2:35at every pixel in the video over time,
-
2:35 - 2:37and then we crank up those changes.
-
2:37 - 2:39We make them bigger
so that we can see them. -
2:39 - 2:41The tricky part is that those signals,
-
2:41 - 2:44those changes that we're after,
are extremely subtle, -
2:44 - 2:47so we have to be very careful
when you try to separate them -
2:47 - 2:51from noise that always exists in videos.
-
2:51 - 2:54So we use some clever
image processing techniques -
2:54 - 2:58to get a very accurate measurement
of the color at each pixel in the video, -
2:58 - 3:00and then the way the color
changes over time, -
3:00 - 3:03and then we amplify those changes.
-
3:03 - 3:07We make them bigger to create those types
of enhanced videos, or magnified videos, -
3:07 - 3:09that actually show us those changes.
-
3:09 - 3:13But it turns out we can do that
not just to show tiny changes in color, -
3:13 - 3:16but also tiny motions,
-
3:16 - 3:19and that's because the light
that gets recorded in our cameras -
3:19 - 3:22will change not only if the color
of the object changes, -
3:22 - 3:24but also if the object moves.
-
3:24 - 3:28So this is my daughter
when she was about two months old. -
3:28 - 3:31It's a video I recorded
about three years ago. -
3:31 - 3:34And as new parents, we all want
to make sure our babies are healthy, -
3:34 - 3:37that they're breathing,
that they're alive, of course. -
3:37 - 3:39So I too got one of those baby monitors
-
3:39 - 3:41so that I could see my daughter
when she was asleep. -
3:41 - 3:45And this is pretty much what you'll see
with a standard baby monitor. -
3:45 - 3:48You can see the baby's sleeping, but
there's not too much information there. -
3:48 - 3:50There's not too much we can see.
-
3:50 - 3:53Wouldn't it be better,
or more informative, or more useful, -
3:53 - 3:56if instead we could look
at the view like this. -
3:56 - 4:02So here I took the motions
and I magnified them 30 times, -
4:02 - 4:06and then I could clearly see that my
daughter was indeed alive and breathing. -
4:06 - 4:08(Laughter)
-
4:08 - 4:10Here is a side-by-side comparison.
-
4:10 - 4:13So again, in the source video,
in the original video, -
4:13 - 4:14there's not too much we can see,
-
4:14 - 4:18but once we magnify the motions,
the breathing becomes much more visible. -
4:18 - 4:20And it turns out, there's
a lot of phenomena -
4:20 - 4:24we can reveal and magnify
with our new motion microscope. -
4:24 - 4:28We can see how our veins and arteries
are pulsing in our bodies. -
4:28 - 4:31We can see that our eyes
are constantly moving -
4:31 - 4:33in this wobbly motion.
-
4:33 - 4:34And that's actually my eye,
-
4:34 - 4:37and again this video was taken
right after my daughter was born, -
4:37 - 4:42so you can see I wasn't getting
too much sleep. (Laughter) -
4:42 - 4:44Even when a person is sitting still,
-
4:44 - 4:46there's a lot of information
we can extract -
4:46 - 4:50about their breathing patterns,
small facial expressions. -
4:50 - 4:52Maybe we could use those motions
-
4:52 - 4:55to tell us something about
our thoughts or our emotions. -
4:55 - 4:58We can also magnify small
mechanical movements, -
4:58 - 5:00like vibrations in engines,
-
5:00 - 5:03that can help engineers detect
and diagnose machinery problems, -
5:03 - 5:08or see how our buildings and structures
sway in the wind and react to forces. -
5:08 - 5:13Those are all things that our society
knows how to measure in various ways, -
5:13 - 5:15but measuring those motions is one thing,
-
5:15 - 5:17and actually seeing those
motions as they happen -
5:17 - 5:20is a whole different thing.
-
5:20 - 5:23And ever since we discovered
this new technology, -
5:23 - 5:27we made our code available online so that
others could use and experiment with it. -
5:27 - 5:29It's very simple to use.
-
5:29 - 5:31It can work on your own videos.
-
5:31 - 5:34Our collaborators at Quanta Research
even created this nice website -
5:34 - 5:37where you can upload your videos
and process them online, -
5:37 - 5:40so even if you don't have any experience
in computer science or programming, -
5:40 - 5:43you can still very easily experiment
with this new microscope. -
5:43 - 5:46And I'd like to show you
just a couple of examples -
5:46 - 5:48of what others have done with it.
-
5:48 - 5:54So this video was made by
a YouTube user called Tamez85. -
5:54 - 5:55I don't know who that user is,
-
5:55 - 5:58but he, or she, used our code
-
5:58 - 6:01to magnify small belly
movements during pregnancy. -
6:01 - 6:03It's kind of creepy.
-
6:03 - 6:05(Laughter)
-
6:05 - 6:09People have used it to magnify
pulsing veins in their hands. -
6:09 - 6:13And you know it's not real science
unless you use guinea pigs, -
6:13 - 6:17and apparently this guinea pig
is called Tiffany, -
6:17 - 6:20and this YouTube user claims
it is the first rodent on Earth -
6:20 - 6:22that was motion-magnified.
-
6:22 - 6:24You can also do some art with it.
-
6:24 - 6:28So this video was sent to me
by a design student at Yale. -
6:28 - 6:30She wanted to see
if there's any difference -
6:30 - 6:31in the way her classmates move.
-
6:31 - 6:35She made them all stand still,
and then magnified their motions. -
6:35 - 6:39It's like seeing
still pictures come to life. -
6:39 - 6:41And the nice thing with
all those examples -
6:41 - 6:43is that we had nothing to do with them.
-
6:43 - 6:47We just provided this new tool,
a new way to look at the world, -
6:47 - 6:52and then people find other interesting,
new and creative ways of using it. -
6:52 - 6:54But we didn't stop there.
-
6:54 - 6:57This tool not only allows us
to look at the world in a new way, -
6:57 - 7:00it also redefines what we can do
-
7:00 - 7:03and pushes the limits of what
we can do with our cameras. -
7:03 - 7:05So as scientists, we started wondering,
-
7:05 - 7:09what other types of physical phenomena
produce tiny motions -
7:09 - 7:12that we could now use
our cameras to measure? -
7:12 - 7:16And one such phenomenon
that we focused on recently is sound. -
7:16 - 7:18Sound, as we all know,
is basically changes -
7:18 - 7:20in air pressure that
travel through the air. -
7:20 - 7:24Those pressure waves hit objects
and they create small vibrations in them, -
7:24 - 7:26which is how we hear
and how we record sound. -
7:26 - 7:30But it turns out that sound
also produces visual motions. -
7:30 - 7:33Those are motions
that are not visible to us -
7:33 - 7:36but are visible to a camera
with the right processing. -
7:36 - 7:37So here are two examples.
-
7:37 - 7:40This is me demonstrating
my great singing skills. -
7:41 - 7:43(Singing)
-
7:43 - 7:44(Laughter)
-
7:44 - 7:47And I took a high-speed video
of my throat while I was humming. -
7:47 - 7:49Again, if you stare at that video,
-
7:49 - 7:51there's not too much
you'll be able to see, -
7:51 - 7:55but once we magnify the motions 100 times,
we can see all the motions and ripples -
7:55 - 7:59in the neck that are involved
in producing the sound. -
7:59 - 8:01That signal is there in that video.
-
8:01 - 8:04We also know that singers
can break a wine glass -
8:04 - 8:05if they hit the correct note.
-
8:05 - 8:07So here, we're going to play a note
-
8:07 - 8:10that's in the resonance
frequency of that glass -
8:10 - 8:12through a loudspeaker that's next to it.
-
8:12 - 8:16Once we play that note
and magnify the motions 250 times, -
8:16 - 8:19we can very clearly see
how the glass vibrates -
8:19 - 8:22and resonates in response to the sound.
-
8:22 - 8:25It's not something you're used
to seeing every day. -
8:25 - 8:28But this made us think.
It gave us this crazy idea. -
8:28 - 8:34Can we actually invert this process
and recover sound from video -
8:34 - 8:38by analyzing the tiny vibrations
that sound waves create in objects, -
8:38 - 8:42and essentially convert those
back into the sounds that produced them. -
8:42 - 8:47In this way, we can turn
everyday objects into microphones. -
8:47 - 8:49So that's exactly what we did.
-
8:49 - 8:52So here's an empty bag of chips
that was lying on a table, -
8:52 - 8:55and we're going to turn that
bag of chips into a microphone -
8:55 - 8:56by filming it with a video camera
-
8:56 - 9:00and analyzing the tiny motions
that sound waves create in it. -
9:00 - 9:02So here's the sound
that we played in the room. -
9:02 - 9:07(Music: "Mary Had a Little Lamb")
-
9:10 - 9:13And this is a high-speed video
we recorded of that bag of chips. -
9:13 - 9:14Again it's playing.
-
9:14 - 9:18There's no chance you'll be able
to see anything going on in that video -
9:18 - 9:19just by looking at it,
-
9:19 - 9:22but here's the sound we were able
to recover just by analyzing -
9:22 - 9:24the tiny motions in that video.
-
9:24 - 9:27(Music: "Mary Had a Little Lamb")
-
9:41 - 9:42I call it -- Thank you.
-
9:42 - 9:48(Applause)
-
9:50 - 9:52I call it the visual microphone.
-
9:52 - 9:56We actually extract audio signals
from video signals. -
9:56 - 9:59And just to give you a sense
of the scale of the motions here, -
9:59 - 10:04a pretty loud sound will cause that bag
of chips to move less than a micrometer. -
10:04 - 10:07That's one thousandth of a millimeter.
-
10:07 - 10:10That's how tiny the motions are
that we are now able to pull out -
10:10 - 10:14just by observing how light
bounces off objects -
10:14 - 10:16and gets recorded by our cameras.
-
10:16 - 10:19We can recover sounds
from other objects, like plants. -
10:19 - 10:25(Music: "Mary Had a Little Lamb")
-
10:27 - 10:29And we can recover speech as well.
-
10:29 - 10:32So here's a person speaking in a room.
-
10:32 - 10:36Voice: Mary had a little lamb
whose fleece was white as snow, -
10:36 - 10:40and everywhere that Mary went,
that lamb was sure to go. -
10:40 - 10:43Michael Rubinstein: And here's
that speech again recovered -
10:43 - 10:46just from this video
of that same bag of chips. -
10:46 - 10:51Voice: Mary had a little lamb
whose fleece was white as snow, -
10:51 - 10:56and everywhere that Mary went,
that lamb was sure to go. -
10:56 - 10:58MR: We used "Mary Had a Little Lamb"
-
10:58 - 11:00because those are said to be
the first words -
11:00 - 11:05that Thomas Edison spoke
into his phonograph in 1877. -
11:05 - 11:08It was one of the first sound
recording devices in history. -
11:08 - 11:11It basically directed the sounds
onto a diaphragm -
11:11 - 11:15that vibrated a needle that essentially
engraved the sound on tinfoil -
11:15 - 11:17that was wrapped around the cylinder.
-
11:17 - 11:23Here's a demonstration of recording and
replaying sound with Edison's phonograph. -
11:23 - 11:26(Video) Voice: Testing,
testing, one two three. -
11:26 - 11:30Mary had a little lamb
whose fleece was white as snow, -
11:30 - 11:34and everywhere that Mary went,
the lamb was sure to go. -
11:34 - 11:36Testing, testing, one two three.
-
11:36 - 11:40Mary had a little lamb
whose fleece was white as snow, -
11:40 - 11:46and everywhere that Mary went,
the lamb was sure to go. -
11:46 - 11:50MR: And now, 137 years later,
-
11:50 - 11:54we're able to get sound
in pretty much similar quality -
11:54 - 11:58but by just watching objects
vibrate to sound with cameras, -
11:58 - 12:00and we can even do that when the camera
-
12:00 - 12:04is 15 feet away from the object,
behind soundproof glass. -
12:04 - 12:07So this is the sound that we were
able to recover in that case. -
12:07 - 12:13Voice: Mary had a little lamb
whose fleece was white as snow, -
12:13 - 12:17and everywhere that Mary went,
the lamb was sure to go. -
12:17 - 12:21MR: And of course, surveillance is
the first application that comes to mind. -
12:21 - 12:24(Laughter)
-
12:24 - 12:28But it might actually be useful
for other things as well. -
12:28 - 12:31Maybe in the future, we'll be able
to use it, for example, -
12:31 - 12:33to recover sound across space,
-
12:33 - 12:37because sound can't travel
in space, but light can. -
12:37 - 12:39We've only just begun exploring
-
12:39 - 12:42other possible uses
for this new technology. -
12:42 - 12:45It lets us see physical processes
that we know are there -
12:45 - 12:49but that we've never been able
to see with our own eyes until now. -
12:49 - 12:50This is our team.
-
12:50 - 12:53Everything I showed you today
is a result of a collaboration -
12:53 - 12:55with this great group
of people you see here, -
12:55 - 12:58and I encourage you and welcome you
to check out our website, -
12:58 - 12:59try it out yourself,
-
12:59 - 13:02and join us in exploring
this world of tiny motions. -
13:02 - 13:04Thank you.
-
13:04 - 13:05(Applause)
- Title:
- See invisible motion, hear silent sounds. Cool? Creepy? We can't decide
- Speaker:
- Michael Rubinstein
- Description:
-
Meet the “motion microscope,” a video-processing tool that plays up tiny changes in motion and color impossible to see with the naked eye. Video researcher Michael Rubinstein plays us clip after jaw-dropping clip showing how this tech can track an individual’s pulse and heartbeat simply from a piece of footage. Watch him recreate a conversation by amplifying the movements from sound waves bouncing off a bag of chips. The wow-inspiring and sinister applications of this tech you have to see to believe.
- Video Language:
- English
- Team:
- closed TED
- Project:
- TEDTalks
- Duration:
- 13:18
Morton Bast edited English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Yasushi Aoki commented on English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Yasushi Aoki commented on English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Morton Bast edited English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Morton Bast edited English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Morton Bast edited English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Morton Bast edited English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide | ||
Morton Bast approved English subtitles for See invisible motion, hear silent sounds. Cool? Creepy? We can't decide |
Yasushi Aoki
108
00:05:30,708 --> 00:05:33,901
Our collaborators at Quantum Research
even created this nice website
->
Our collaborators at Quanta Research
even created this nice website
# See https://videoscope.qrilab.com/
114
00:05:48,470 --> 00:05:53,787
So this video was made by
a YouTube user called Tamez85.
->
So this video was made by
a YouTube user called Tomez85.
# See https://www.youtube.com/user/Tomez85
Yasushi Aoki
237
00:12:39,157 --> 00:12:42,176
other possible uses
for this new technology.
->
all the possible uses
for this new technology.