0:00:00.449,0:00:01.482
My passions

0:00:01.482,0:00:05.171
are music, technology and making things.

0:00:05.171,0:00:08.354
And it's the combination of these things

0:00:08.354,0:00:11.050
that has led me to the hobby of sound visualization,

0:00:11.050,0:00:15.128
and, on occasion, has led me to play with fire.

0:00:15.128,0:00:17.505
This is a Rubens' tube. It's one of many I've made over the years,

0:00:17.505,0:00:19.251
and I have one here tonight.

0:00:19.251,0:00:20.776
It's about an 8-foot-long tube of metal,

0:00:20.776,0:00:22.111
it's got a hundred or so holes on top,

0:00:22.111,0:00:23.864
on that side is the speaker, and here

0:00:23.864,0:00:26.040
is some lab tubing, and it's connected to this tank

0:00:26.040,0:00:27.641
of propane.

0:00:29.087,0:00:32.456
So, let's fire it up and see what it does.

0:00:37.887,0:00:39.778
So let's play a 550-herz frequency

0:00:39.778,0:00:41.293
and watch what happens.

0:00:41.342,0:00:49.825
(Frequency)

0:00:49.825,0:00:52.538
Thank you. (Applause)

0:00:52.538,0:00:54.526
It's okay to applaud the laws of physics,

0:00:54.526,0:00:55.982
but essentially what's happening here

0:00:55.982,0:00:57.742
-- (Laughter) --

0:00:57.742,0:01:01.750
is the energy from the sound via the air and gas molecules

0:01:01.750,0:01:04.254
is influencing the combustion properties of propane,

0:01:04.254,0:01:06.158
creating a visible waveform,

0:01:06.158,0:01:08.478
and we can see the alternating regions of compression

0:01:08.478,0:01:10.518
and rarefaction that we call frequency,

0:01:10.518,0:01:12.262
and the height is showing us amplitude.

0:01:12.262,0:01:14.590
So let's change the frequency of the sound,

0:01:14.590,0:01:16.015
and watch what happens to the fire.

0:01:16.015,0:01:26.145
(Higher frequency)

0:01:26.145,0:01:29.495
So every time we hit a resonant frequency we get a standing wave

0:01:29.495,0:01:31.195
and that emergent sine curve of fire.

0:01:31.195,0:01:32.773
So let's turn that off. We're indoors.

0:01:32.773,0:01:38.364
Thank you. (Applause)

0:01:38.364,0:01:40.711
I also have with me a flame table.

0:01:40.711,0:01:42.276
It's very similar to a Rubens' tube, and it's also used

0:01:42.276,0:01:44.397
for visualizing the physical properties of sound,

0:01:44.397,0:01:46.365
such as eigenmodes, so let's fire it up

0:01:46.365,0:01:48.615
and see what it does.

0:01:52.292,0:01:56.760
Ooh. (Laughter)

0:01:56.760,0:01:59.734
Okay. Now, while the table comes up to pressure,

0:01:59.734,0:02:01.406
let me note here that the sound is not traveling

0:02:01.406,0:02:04.152
in perfect lines. It's actually traveling in all directions,

0:02:04.152,0:02:07.261
and the Rubens' tube's a little like bisecting those waves

0:02:07.261,0:02:09.257
with a line, and the flame table's a little like

0:02:09.257,0:02:11.111
bisecting those waves with a plane,

0:02:11.111,0:02:15.111
and it can show a little more subtle complexity, which is why

0:02:15.111,0:02:17.464
I like to use it to watch Geoff Farina play guitar.

0:02:17.464,0:02:59.745
(Music)

0:02:59.745,0:03:01.510
All right, so it's a delicate dance.

0:03:01.510,0:03:04.055
If you watch closely — (Applause)

0:03:04.055,0:03:06.822
If you watch closely, you may have seen

0:03:06.822,0:03:09.294
some of the eigenmodes, but also you may have seen

0:03:09.294,0:03:13.911
that jazz music is better with fire.

0:03:13.911,0:03:15.921
Actually, a lot of things are better with fire in my world,

0:03:15.921,0:03:18.353
but the fire's just a foundation.

0:03:18.353,0:03:19.501
It shows very well that eyes can hear,

0:03:19.501,0:03:20.896
and this is interesting to me because

0:03:20.896,0:03:23.750
technology allows us to present sound to the eyes

0:03:23.750,0:03:26.613
in ways that accentuate the strength of the eyes

0:03:26.613,0:03:29.310
for seeing sound, such as the removal of time.

0:03:29.310,0:03:32.694
So here, I'm using a rendering algorithm to paint

0:03:32.694,0:03:35.157
the frequencies of the song "Smells Like Teen Spirit"

0:03:35.157,0:03:37.197
in a way that the eyes can take them in

0:03:37.197,0:03:39.441
as a single visual impression, and the technique

0:03:39.441,0:03:41.414
will also show the strengths of the visual cortex

0:03:41.414,0:03:43.030
for pattern recognition.

0:03:43.030,0:03:44.909
So if I show you another song off this album,

0:03:44.909,0:03:48.390
and another, your eyes will easily pick out

0:03:48.390,0:03:51.318
the use of repetition by the band Nirvana,

0:03:51.318,0:03:53.175
and in the frequency distribution, the colors,

0:03:53.175,0:03:56.223
you can see the clean-dirty-clean sound

0:03:56.223,0:03:57.430
that they are famous for,

0:03:57.430,0:04:01.430
and here is the entire album as a single visual impression,

0:04:01.430,0:04:03.310
and I think this impression is pretty powerful.

0:04:03.310,0:04:05.024
At least, it's powerful enough that

0:04:05.024,0:04:06.366
if I show you these four songs,

0:04:06.366,0:04:08.807
and I remind you that this is "Smells Like Teen Spirit,"

0:04:08.807,0:04:11.126
you can probably correctly guess, without listening

0:04:11.126,0:04:12.560
to any music at all, that the song

0:04:12.560,0:04:14.854
a die hard Nirvana fan would enjoy is this song,

0:04:14.854,0:04:17.110
"I'll Stick Around" by the Foo Fighters,

0:04:17.110,0:04:19.110
whose lead singer is Dave Grohl,

0:04:19.110,0:04:22.888
who was the drummer in Nirvana.

0:04:22.888,0:04:24.188
The songs are a little similar, but mostly

0:04:24.188,0:04:25.814
I'm just interested in the idea that someday maybe

0:04:25.814,0:04:30.226
we'll buy a song because we like the way it looks.

0:04:30.226,0:04:31.326
All right, now for some more sound data.

0:04:31.326,0:04:33.978
This is data from a skate park,

0:04:33.978,0:04:36.010
and this is Mabel Davis skate park

0:04:36.010,0:04:38.152
in Austin, Texas. (Skateboard sounds)

0:04:38.152,0:04:39.526
And the sounds you're hearing came from eight

0:04:39.526,0:04:41.742
microphones attached to obstacles around the park,

0:04:41.742,0:04:43.926
and it sounds like chaos, but actually

0:04:43.926,0:04:47.273
all the tricks start with a very distinct slap,

0:04:47.273,0:04:48.877
but successful tricks end with a pop,

0:04:48.877,0:04:50.670
whereas unsuccessful tricks

0:04:50.670,0:04:52.526
more of a scratch and a tumble,

0:04:52.526,0:04:56.536
and tricks on the rail will ring out like a gong, and

0:04:56.536,0:04:59.326
voices occupy very unique frequencies in the skate park.

0:04:59.326,0:05:01.264
So if we were to render these sounds visually,

0:05:01.264,0:05:02.671
we might end up with something like this.

0:05:02.671,0:05:05.127
This is all 40 minutes of the recording,

0:05:05.127,0:05:07.287
and right away the algorithm tells us

0:05:07.287,0:05:09.360
a lot more tricks are missed than are made,

0:05:09.360,0:05:11.695
and also a trick on the rails is a lot more likely

0:05:11.695,0:05:14.567
to produce a cheer, and if you look really closely,

0:05:14.567,0:05:16.300
we can tease out traffic patterns.

0:05:16.300,0:05:22.387
You see the skaters often trick in this direction. The obstacles are easier.

0:05:22.387,0:05:24.092
And in the middle of the recording, the mics pick this up,

0:05:24.092,0:05:26.896
but later in the recording, this kid shows up,

0:05:26.896,0:05:29.824
and he starts using a line at the top of the park

0:05:29.824,0:05:31.626
to do some very advanced tricks on something

0:05:31.626,0:05:32.729
called the tall rail.

0:05:32.729,0:05:34.601
And it's fascinating. At this moment in time,

0:05:34.601,0:05:38.104
all the rest of the skaters turn their lines 90 degrees

0:05:38.104,0:05:39.882
to stay out of his way.

0:05:39.882,0:05:42.425
You see, there's a subtle etiquette in the skate park,

0:05:42.425,0:05:44.025
and it's led by key influencers,

0:05:44.025,0:05:47.273
and they tend to be the kids who can do the best tricks,

0:05:47.273,0:05:49.703
or wear red pants, and on this day the mics picked that up.

0:05:49.703,0:05:53.604
All right, from skate physics to theoretical physics.

0:05:53.604,0:05:55.220
I'm a big fan of Stephen Hawking,

0:05:55.220,0:05:56.556
and I wanted to use all eight hours

0:05:56.556,0:05:59.143
of his Cambridge lecture series to create an homage.

0:05:59.143,0:06:02.207
Now, in this series he's speaking with the aid of a computer,

0:06:02.207,0:06:05.311
which actually makes identifying the ends of sentences

0:06:05.311,0:06:08.737
fairly easy. So I wrote a steering algorithm.

0:06:08.737,0:06:10.697
It listens to the lecture, and then it uses

0:06:10.697,0:06:13.384
the amplitude of each word to move a point on the x-axis,

0:06:13.384,0:06:16.017
and it uses the inflection of sentences

0:06:16.017,0:06:18.171
to move a same point up and down on the y-axis.

0:06:18.171,0:06:20.945
And these trend lines, you can see, there's more questions

0:06:20.945,0:06:22.815
than answers in the laws of physics,

0:06:22.815,0:06:24.848
and when we reach the end of a sentence,

0:06:24.848,0:06:27.183
we place a star at that location.

0:06:27.183,0:06:29.983
So there's a lot of sentences, so a lot of stars,

0:06:29.983,0:06:32.339
and after rendering all of the audio, this is what we get.

0:06:32.339,0:06:35.268
This is Stephen Hawking's universe.

0:06:35.268,0:06:42.205
(Applause)

0:06:42.205,0:06:44.692
It's all eight hours of the Cambridge lecture series

0:06:44.692,0:06:46.540
taken in as a single visual impression,

0:06:46.540,0:06:48.297
and I really like this image,

0:06:48.297,0:06:50.105
but a lot of people think it's fake.

0:06:50.105,0:06:52.449
So I made a more interactive version,

0:06:52.449,0:06:57.905
and the way I did that is I used their position in time

0:06:57.905,0:07:00.272
in the lecture to place these stars into 3D space,

0:07:00.272,0:07:02.673
and with some custom software and a Kinect,

0:07:02.673,0:07:05.336
I can walk right into the lecture.

0:07:05.336,0:07:07.136
I'm going to wave through the Kinect here

0:07:07.136,0:07:08.968
and take control, and now I'm going to reach out

0:07:08.968,0:07:12.207
and I'm going to touch a star, and when I do,

0:07:12.207,0:07:14.151
it will play the sentence

0:07:14.151,0:07:15.616
that generated that star.

0:07:15.616,0:07:19.432
Stephen Hawking: There is one, and only one, arrangement

0:07:19.432,0:07:22.231
in which the pieces make a complete picture.

0:07:22.231,0:07:26.379
Jared Ficklin: Thank you. (Applause)

0:07:26.379,0:07:29.552
There are 1,400 stars.

0:07:29.552,0:07:31.216
It's a really fun way to explore the lecture,

0:07:31.216,0:07:32.683
and, I hope, a fitting homage.

0:07:32.683,0:07:38.155
All right. Let me close with a work in progress.

0:07:38.155,0:07:41.138
I think, after 30 years, the opportunity exists

0:07:41.138,0:07:43.242
to create an enhanced version of closed captioning.

0:07:43.242,0:07:45.361
Now, we've all seen a lot of TEDTalks online,

0:07:45.361,0:07:48.288
so let's watch one now with the sound turned off

0:07:48.288,0:07:52.210
and the closed captioning turned on.

0:07:52.210,0:07:54.356
There's no closed captioning for the TED theme song,

0:07:54.356,0:07:56.469
and we're missing it, but if you've watched enough of these,

0:07:56.469,0:07:57.838
you hear it in your mind's ear,

0:07:57.838,0:08:00.821
and then applause starts.

0:08:00.821,0:08:03.011
It usually begins here, and it grows and then it falls.

0:08:03.011,0:08:04.988
Sometimes you get a little star applause,

0:08:04.988,0:08:07.474
and then I think even Bill Gates takes a nervous breath,

0:08:07.474,0:08:09.164
and the talk begins.

0:08:09.164,0:08:14.862
All right, so let's watch this clip again.

0:08:14.862,0:08:16.118
This time, I'm not going to talk at all.

0:08:16.118,0:08:17.485
There's still going to be no audio,

0:08:17.485,0:08:19.389
but what I am going to do is I'm going to render the sound

0:08:19.389,0:08:23.705
visually in real time at the bottom of the screen.

0:08:23.705,0:08:26.496
So watch closely and see what your eyes can hear.

0:08:47.880,0:08:49.760
This is fairly amazing to me.

0:08:49.760,0:08:53.093
Even on the first view, your eyes will successfully

0:08:53.093,0:08:56.181
pick out patterns, but on repeated views,

0:08:56.181,0:08:57.870
your brain actually gets better

0:08:57.870,0:08:59.526
at turning these patterns into information.

0:08:59.526,0:09:01.117
You can get the tone and the timbre

0:09:01.117,0:09:02.340
and the pace of the speech,

0:09:02.340,0:09:04.404
things that you can't get out of closed captioning.

0:09:04.404,0:09:06.588
That famous scene in horror movies

0:09:06.588,0:09:09.236
where someone is walking up from behind

0:09:09.236,0:09:11.196
is something you can see,

0:09:11.196,0:09:13.900
and I believe this information would be something

0:09:13.900,0:09:16.703
that is useful at times when the audio is turned off

0:09:16.703,0:09:19.644
or not heard at all, and I speculate that deaf audiences

0:09:19.644,0:09:20.821
might actually even be better

0:09:20.821,0:09:22.602
at seeing sound than hearing audiences.

0:09:22.602,0:09:24.100
I don't know. It's a theory right now.

0:09:24.100,0:09:25.621
Actually, it's all just an idea.

0:09:25.621,0:09:29.812
And let me end by saying that sound moves in all directions,

0:09:29.812,0:09:31.789
and so do ideas.

0:09:31.789,0:09:34.901
Thank you. (Applause)