How we found the worst place to park in New York City — using big data

Six thousand miles of road,

600 miles of subway track,

400 miles of bike lanes,

and a half a mile of tram track,

if you've ever been to Roosevelt Island.

So these are the numbers that make up
the infrastructure of New York City.

These are the statistics
of our infrastructure.

They're the kind of numbers you can find
released in reports by city agencies.

For example, the Department
of Transportation will probably tell you

how many miles of road they maintain.

The MTA will boast how many miles
of subway track there are.

But most city agencies give us statistics.

This is from a report this year

from the Taxi and Limousine Commission,

where we learn that there's about,
13,500 taxis here in New York City.

Pretty interesting, right?

But did you ever think about
where these numbers came from?

Because for these numbers to exist,
someone at the city agency

had to stop and say, "Hmm, here's a number
that somebody might want want to know."

So they go back to their raw data,

they count, they add, they calculate,

and then they put out reports,

and those reports
will have numbers like this.

The problem is, how do they know
all of our questions?

We have lots of questions.

In fact, in some ways there's literally
an infinite number of questions

that we can ask about our city.

So the agencies can never keep up.

So the paradigm isn't exactly working,
and I think our policymakers realize that,

because in 2012, Mayor Bloomberg
signed into law what he called

the most ambitious and comprehensive
open data legislation in the country.

In a lot of ways, he's right.

In the last two years,
the city has released

a thousand data sets
on our open data portal,

and it's pretty awesome.

So you go and look at data like this,
and instead of just counting

the number of cabs,

we can start to ask different questions.

So I had a question.

When's rush hour in New York City?

I mean, it can be pretty bothersome.
When is rush hour exactly?

And I thought to myself,
well, these cabs aren't just numbers,

these are GPS recorders
driving around in our city streets

recording each and every ride they take.

There's data there,

and I looked at that data,
and I made a plot

of the average speed of taxis
in New York City throughout the day.

Well, you can see that from about midnight
to around 5:18 in the morning,

speed increases, and at that point,

things turn around,

and they get slower and slower and slower
until about 8:35 in the morning,

when they end up at around
11 and a half miles and hour.

The average taxi is going 11 and a half
miles per hour on our city streets,

and it turns out stays that way

for the entire day.

The entire day. (Laughter)

So I said to myself, I guess
there's no rush hour in New York City.

There's just a rush day.

Makes sense. But this is important
for a couple of reasons.

If you're a transportation planner,
this might be pretty interesting to know,

but if you want to get somewhere quickly,

you now know to set your alarm
for 4:45 in the morning, your'e all set.

New York, right?

But there's a story behind this data.

This data wasn't
just available, it turns out.

It actually came from something called
a Freedom of Information law request,

or a FOILrequest.

This is a form you can find on the
Taxi and Limousines Commission website.

In order to access this data,
you need to go get this form,

fill it out, and they will notify you,

and a guy named Chris Wong
did exactly that.

Chris went down, and they told him,
"Just a bring a hard drive down,

a brand new hard drive,
bring it to our office,

leave it here for five hours,
we'll copy data and you take it back."

And that's where this data came from.

Now, Chris is the kind of guy
who wants to make the data public,

and so it ended up online for all to use,
and that's where this graph came from.

And the fact that it exists is amazing.
These GPS recorders, really cool.

But the fact that we have citizens
walking around with hard drives

picking up data from city agencies
to make it public,

where it was already kind of public,
you could get to it, but it was "public,"

but it wasn't public.

And we can do better
than that as a city, right?

We don't need our citizens
walking around with hard drives.

Now, not every data set
is behind a FOIL request. Right?

So here is a map I made with the most
dangerous intersections in New York City,

based on cyclist accidents.

So the red areas are more dangerous,
and what it shows is first,

the East side of Manhattan,
especially in the lower area of Manhattan,

has more cyclist accidents.

That might make sense,
because there are more cyclists

coming off the bridges there.

But there are other hotspots
worth studying, right?

There's Williamsburg.
There's Roosevelt Avenue in Queens.

And this is exactly the kind of data
we need for Vision Zero.

This is exactly what we're looking for.

But there's a story
behind this data as well.

This data didn't just appear.

How many of you guys know this logo?

Yeah, I see some shakes.

Have you ever tried to copy
and paste data out of a PDF

and make sense with it?

I see more shakes.

More of you tried copying and pasting
than knew the logo. I like that.

Well, so what happened is, the data
that you just saw was actually on a PDF.

In fact, hundreds and hundreds
and hundreds of pages of PDF

put out by our very own NYPD,

and in order to access it,
you would either have to copy and paste

for hundreds and hundreds of hours,

or you could be John Krause.

John Krause was like,
I'm not going to copy and paste this data.

I'm going to write a program.

It's called the NYPD Crash Data Band-Aid,

and it goes to the NYPD's website
and it would download PDFs.

Every day it would search:
if it found a PDF, it would download it

and then it would run
some PDF-scraping program,

and out would come the text,

and it would go on the Internet,
and then people could make maps like that.

And the fact that the data's here,
once again, the fact that we have access to it

-- Every accident, by the way,
is a row in this table,

every single accident, you can imagine
how many PDFs that is --

the fact that we
have access to that is great,

but let's not release it in PDF form,

because then we're having our citizens
write PDF scrapers.

It's not the best use
of our citizens' time,

and we as a city can do better than that.

Now, the good news is that
the de Blasio Administration

actually recently released this data
a few months ago,

and so now we can
actually have access to it,

but there's a lot of data
still entombed in PDF.

For example, our crime data
is still only available in PDF.

And not just our crime data:

our own city budget.

Our city budget is own readable
right now in PDF form,

and it's not just us
that can't analyze it:

our own legislators
who vote for the budget

also only get it in PDF.

So our legislators cannot
analyze the budget

that they are voting for.

And I think as a city we can do
a little better than that as well.

Now, there's a lot of data
that's not hidden in PDFs.

This is an example of a map I made,

and this is the dirtiest waterways
in New York City.

Now, how do I measure dirty?

Well, it's kind of a little weird,

but I looked at the level
of fecal coliform