Seeing the pattern for the pixels: Austin Troy at TEDxUVM

Austin Troy's work focuses on land use – particularly the causes and impacts of urban/suburban development and the effectiveness of policies in mediating those impacts. He is also interested in studying the role and importance of environmental assets in urban systems and in quantifying the spatial distribution of ecosystem services.

All right, thanks Chris
and thanks for having me here.

Can everybody hear me OK?

So, today I'm going to talk

about a type of data
we're all very familiar with

and I think most of us like, intrinsically.

And that is geographic data
and particularly imagery of the Earth.

And we've already seen
some examples of that today.

I'm going to start
with a little show and tell here.

I had to bring a prop, I couldn't resist.

My old Macintosh Power Book 145 from 1992.

This is the first computer
I had with a hard drive.

It came with 4, was it 4?

Actually with about 6 megabytes of memory.

That was a big deal
and I was just blown away.

I couldn't believe I had that much memory.

I'm just going to put this back here.

I couldn't believe
I had that much memory at my fingertips.

Today, we now all have computers.

I can go down to Staples
and buy a computer

that has a quarter million times
more memory than that

for about $400 or $600
or something like that.

Times have changed
and that's 20 years ago.

As a result,
with all this increased computing power,

we are drowning in data.

We're just absolutely drowning in data.

And one of the types of data
we are most drowning in

is remote sensing
or imagery data, satellite data,

aerial data, things like that.

And we've all played around
with this, I'm sure.

We all love Google Earth,
it's free and it's fun.

And it's just teaming with imagery.

So, what do we do with all this stuff?

How do we make use of this?

Here's an image of Baltimore.
This is urban Baltimore.

It's got all these great objects.

I can look in there and see,

it's hard with this projector,

but I can see trees and buildings,
things like that.

And let's just say

I wanted to actually do some kind
of a quantitative study with that.

Say I had to do something
that required knowing

where the trees really were.

I can see where the trees are,

but the computer doesn't know,
it has no clue what a tree is.

Let's just say I wanted to do
something like,

these are actually
the locations of crimes,

say I wanted to know
if the density of trees affects crime.

There's no way I can do that
with imagery the way we have it now,

in a computing environment.

Part of the reason for this

is that computers
aren't really good at recognizing things

the way that we can recognise things.

We are excellent are recognizing
things with very slight differences.

I can tell you within two seconds

that that's George Carlin
and that's Sigmund Freud.

That that is the Big Lebowski
and that is Eddie Vedder.

For me to train a computer
to recognise the difference

between the Big Lebowski, a.k.a. the Dude
and Eddie Vedder,

would take me unbelievable amounts
of time to do.

Yet I can do that instantly,
so that's an issue right here.

So, let's cut to the chase here.

On the left I have raw data,

color infrared remote sense imagery.

On the right I have
a classified GIS layer.

That is usable information.

The computer knows what's grass,
what's buildings and knows what's trees.

How do I get from one to the other?

This is a major major conundrum
in today's world of high resolution data.

Here's an image
of just a typical suburban area.

I look at it and I see
all sort of features

and I see that it is at a
very fine resolution.

If I were to be working
with remote sensing data 15 years ago,

I'd have coarse resolution imagery.

This is the same exact location
using 30 meter pixels.

Back then, classifying this stuff
was a qualitatively different thing,

because all I really needed to do
is get in the general ball park.

These pixels here
are sort of generally urbanized,

these pixels here are generally forested.

I didn't really have to know
about the specific identity of objects.

Now fast forward to today
and I've got imagery which I can zoom in

and I can see
a million different types of objects.

From cars in a parking lot
to shipping containers,

to the cranes
at the shipping containers facility

to the mechanicals on top of a building

to, I love this one,

this is the Sphinx
at the Luxor Hotel in Las Vegas.

Try telling a computer what that is.

Here's another Vegas one,
I love Google Earth in Vegas,

it's the best, it's so much fun.

This is a tropical fish-shaped pool.

Again, not so easy to tell a computer
what that is.

And this is the best of all.

I believe this is high-resolution
satellite imagery

of camels in the middle of Africa
and it's part of this Google,

National Geographic,
Africa mega fly-over project.

The number of possible things

I have to prepare computers
to be ready for,

the types of objects
out on the surface of the Earth

surface of the Earth,
is staggering.

And it's a project
that we'll never see finished,

that project of teaching computers
the artificial intelligence,

giving them the artificial intelligence
they need

to recognize all of this variation
on the Earth.

Here's another even better one.

This is actually a real thing,
this is the Colonel.

Someone actually did mega art
of the Colonel in the middle of Nevada.

It's interesting
how many of these come from Nevada.

(Laughter)

So, let's explain why it's difficult

to use the methods
we've always used in the past

for this generation
of high resolution imagery.

Here's a high resolution image
of Burlington.

If I try to classify each pixel,
pixel by pixel,

I get this awful pixelated
meaningless gobbledygook.

If I look at a particular object
like a house, that house is made up,

it's hard to see, but it's just dozens
of different pixel values

that don't really mean anything.

If I take a single tree, again,
is made up of a gobbledygook of pixels.

Now, if I zoom in on that tree,
for instance,

I will see that it's made up of pixels,
lots of different tones, different colors,

and if this is the direct representation
in classified pixels,

it's meaningless, right?

This is not finding objects.

So, I need to teach a computer
to see objects and to think like me.

So this means teaching a computer
to think like a human,

which means working
based on shape size, tone, pattern,

texture, site and association,
a lot of this is all spatial.

We have to stop thinking pixel by pixel
and start thinking of things spatially.

That means taking an image
and, what's called segmenting it,

turning it into objects.

And the process of segmenting it
is very difficult.

You have to train a computer
to segment imagery correctly.

And if I look, here is a house,
there's one side of the roof

and another side of the roof,
there is the driveway,

they're segmented as different objects

and I can then re-aggregate those objects

into something that is just a house
and another one that is just a driveway.

And at the end of the day,
what I'm going to end up with

is something like this.

I will be able to tell you the difference.

Even though the spectral signature
is the same of this roof and this road,

I know that their compactness
factor is different,

and because of that,
because of the shape metrics of them,

I can tell you which one's a roof
and which one's a road,

and I can start
classifying things in that way.

To do this requires huge rule sets.

The rule sets could be dozens and dozens
to over a hundred pages long

of all these classification rules
and I won't bore you with the details.

I also will make use of
a lot of ancillary data.

There's all sort of great GIS data
that helps me classify things now.

Most cities are collecting things
about building footprints,

we know where parcels are,
we know where sewer lines are

and roads are and things like that.

We can use this to help us,

but the most important
form of ancillary data

that's out there today is called LIDAR:
Light Detection and Ranging.

And LIDAR has been used
in engineering for a while

and it allows us to essentially create
models of the surface.

This is Columbus Circle
in Central Park in New York City

and this is a surface elevation
of the trees.

The LIDAR tells me
where the canopy of the trees is,

where the tops of the buildings are,
where the ground surface is too.

And I can create these incredibly detailed
models of the world

so now I'm not just working

with spatial spectral information,
reflectance information,

I'm also working with height information,
I know the heights of things

so I can see two objects
that are green and woody,

but I can tell that one of them
is a shrub and one of them is a tree.

And this is just zooming
into that stuff there.

Now the problem is,
this is incredibly data-intensive

and nobody's figured out until recently,
and I mean like maybe two years ago,

people were doing this
on a tile by tile basis

to work on one little tile of data
at a time

that might be, you know,

one, that just that red outline
that you see right there,

and that might be half a gigabyte
or something like that.

So we've worked on turning this
into an enterprise environment,

that's what we have to do,
make an enterprise environment out of this

so we can start looking
at thousands of tiles of data at a time,

and we've successfully done that.

My lab, which is the
Spatial Analysis Lab.

The Spatial Analysis lab is the lab I run,

and they've been doing this stuff
for a number of years,

and they've collected,
through 64 projects, 837 communities,

covering 28 million people,
almost 9000 square miles of data mapped,

250 billion pixels of land cover
products generated

and 110 terabytes of data.

So this is a major undertaking
but it's only the beginning.

Going back to the crime data
that I was telling you about those trees,

so here's Baltimore again.

Using this method,
we turn data into information.

We get trees,
we now know where trees are.

I overlay it with the crime.

I end up with information,
I can now do a study,

and we just submitted this
for publication.

We just found out in fact
there's a strong negative correlation

between trees and crime

even when adjust for about
fifty other things.

We couldn't have done that
without this sort of information.

So with that, I will say thanks
to the people

from the Spatial Analysis Lab,

and particularly Jarlath O'Neil-Dunne
who helped me put this together

and has been doing this research
for a long time and thanks to you.

Thank you. (Applause)