no. Okay. Yeah. Welcome, everyone.
Welcome to our kick off to open data Day,
which is tomorrow, 21st, where you hope the
weather holds off for us to have a good day
of discussions on open data as well as hackathon on
that, Um, so this is open day to
day and code across. And this is a collaboration
between the university libraries and code for an RV.
A brigade of code for America. Uh, here
in Blacksburg. Um, my name is Philip Young
. I am the Scarlet Communications librarian here at university
libraries. And so I work on a lot of
open projects Open scholarship, open access, open data
, which will be talking about tonight and tomorrow.
And we also work on open educational resources. Um
, in order to help students save a little money
on some of the they're learning materials. You may
have seen the posters outside the door about our events
next week. If you're interested in open textbooks and
other open educational resources, please go to our website
and look at some of the events that we have
on tap next week. And we also have our
We are librarian here, Anita, Laws and she'll
be glad to answer any questions that you might have
about open educational resources. Um, in the back
, we have a sign up sheet. Um,
where VT faculty can get NL. I credit for
attending tonight. So if you want to get an
l. I credit, please sign up there.
Also, we have a sign up box for the
open V t listserv, which is, uh,
listserv campus-wide listserv, where we talk about a variety
of open issues, including open access, open data
, and O er, um, in addition to
our speaker tonight, we have a distinguished guest that
I want to recognize A from the Virginia Tech Board
of Visitors. We have Nancy die in our presence
, so thank you very much, Nancy, for
coming tonight to our event. Um, as I
mentioned, this is a collaboration between the university libraries
here at Virginia Tech and code for Inter V.
Um, the co leaders of code for N RV
, R Neal, Neal, Farrah, Ben and
Ben Schoenfeld. And I want to introduce Ben to
come up and talk a little bit about code for
America. Everybody. My name is Ben. I
graduated from Virginia Tech in 2000 and nine with a
degree in computer engineering. Uh, and I'm a
software developer by day, but on nights and weekends
, I volunteer for code for America. And so
I'd like to take a few minutes just to tell
you about code for America since code across, Um
, an open day to day are a code for
America nationwide event. So how many of you are
old enough to remember a time when you had to
go into the bank and interact with someone, uh
, to actually get money or deposit a check?
Um, it seems, uh, the memory seems
quaint because we know how much personal finance has revolutionized
. Um has been revolutionized over the last 24 25
years. So now we can just take a picture
of a check, and minutes later, the money
shows up in our bank. Um and so this
this technology has, uh, has been or technology
has revolutionized the banking industry, and it's revolutionized almost
every other parts of our lives. Uh, but
there's one place that technology hasn't quite revolutionized yet,
and that's that's the government. So this is the
D. M. V. Uh, it's a
symbol of bureaucracy. Uh, so many people here
or everyone here has to interact with the NBA at
some point, and almost no one has a positive
experience. So if if 91% of Americans own a
cell phone, 67% use Facebook and 33% have a
tablet, why is this still how we engage with
government? There has to be a better way.
Um, good for America is all about trying to
revolutionize how we interact with government. The organization was
founded to change the culture inside government that supports bureaucracy
, breeds disengagement with citizens and makes it hard for
government to come up with innovative solutions to long standing
problems, all using modern network digital technology and user
centered design approaches. We take four approaches. We
work directly with government officials at the local level to
create the capacity inside government to build innovative solutions to
hard problems. We build communities of technologists and citizens
who want to lend their skills to help build their
governments. We build tools that make citizens citizen interactions
with government easier, simpler and more elegant so that
the experience of government is positive and breach trust.
We incubate and accelerate civic startups to create new economic
models for those tools and in this were influenced by
the idea that government should act like a platform.
Before the iPhone phones had 20 or 30 applications,
and now they have millions. And we know that
when governments open data, private companies can deliver innovative
solutions. So code for America has five parts.
Um, and tonight I'll just briefly talk about the
part we participated in, which is the brigade.
So we are code for New River Valley. Uh
, we are all volunteer chapter of code for America
. Right here in Blacksburg. We are the fourth
Brigade in Virginia. There's one in Hampton Roads,
one in Richmond, run in northern Virginia. And
we just started up in December. Um, so
like I said, there are good for America.
Brigades are springing up all over the country completely organically
. Uh, the brigade program started about three years
ago. Uh, this time last year, there
were 10 or 20 brigades, and now there are
about 100 brigades nationwide, Uh, and and internationally
, too. So our brigades at advocate for open
data and, uh, the best way we know
how to do this is by taking open data and
showing people what you can do with it and how
powerful it can be. In Hampton Roads, we
created a real time bus tracker. We got,
um, data from the bus company, and we
built that nice web app that you saw that you
can get to on your iPhone. But accessibility really
matters to us. So we also built a text
message interface because we want these technologies to get to
as many people as possible. This is what the
open data looks like that we got. So this
is just a big file. Um, and you
might recognize some numbers in there. Uh, there's
dates and lat long coordinates. Um, and so
we took that data, and we turned it into
what? You see what you saw on the iPhone
there? That map and help people find the schedule
and find how to catch the bus. And we
currently have over 200 users a day use that application
? Um, really, without any kind of promotion
behind it. So it's been really helpful to a
lot of people, and we're really proud of it
. Blacksburg has real time bus data to of course
, Blacksburg is a little more advanced in their transit
than a lot of places, thanks to all the
people who read the bus here. But, um
, we could do some really some really interesting and
innovative things with this this real time data, and
I hope to to work on some of those tomorrow
. Another application that brigades in Virginia developed is open
health inspection dot com. Uh, all restaurants are
visited, uh, quarterly, I believe, by
the health inspector to check for violations. And,
um, that information is public and it's up on
a website somewhere that's very difficult to access. Um
, it's hard to find, and it's very difficult
to get to on your mobile phone. So we
created an application. Uh, you can pull it
up on your smartphone, and, um, it'll
it'll figure out where you are, and I'll show
you the closest restaurants to you. I pulled this
up before I came in here, and, uh
, you can see that, uh, that these
are all the restaurants around. We also created this
scoring system because Virginia doesn't have a scoring system like
some states do. Uh, so we created this
one of our data. Scientists created this, uh
, to try to help people understand the violation system
. And finally, another project that we came up
with back in November was done with a collaboration with
some reporters at the run of times and they came
up. They came to our brigade and said,
Uh, we have difficulty searching on, uh,
searching court cases on the Virginia court website. They
make you search one locality at the time, and
we like to search all of them and 120 localities
takes a long time to search through. So in
just over a weekend, we built this site that
allows them to search through localities, and we get
about 10 users a day on this site. And
so it's been really helpful to, uh to our
journalists all over all over the state. Um,
so now we'll talk about some of the things we
have to accomplish tomorrow at open day to day.
Um, the main thing we'd like to we'd like
to do is have these round table discussions, and
the idea is just like the last example I provided
. If we get subject matter experts in the same
room with civic minded volunteers, we can create some
really interesting applications. And so we're gonna have these
roundtables where we sit down and talk about mapping and
G i s systems. We're gonna talk more about
journalism and see if we can come up with any
more ideas like the one we had there. We're
gonna talk about public policy and trying to use data
to make, um, data driven decisions in in
cities and and state government. And we're also going
to talk about health. Um, v b I
here at Virginia Tech had a great, um,
hackathon a few months ago about on on Ebola data
. And so we hope to carry that torch and
maybe come up with some more innovative things that we
can do at the same time in the background,
we're gonna be running a hackathon. Uh, so
the ideas that come out of these roundtables we hope
to start implementing, uh, immediately and some of
the themes like I discussed before transportation. We've got
that real time transit ap. We've also got some
some parking data, and we'd like to try to
make finding a parking spot in downtown Blacksburg a little
easier. Uh, we're like I said, we're
gonna work on the health data, the public policy
data. We've got a great new API for finding
out about information about Virginia businesses. Um, and
we've got the court data that I mentioned before that
we can look through and we've got some other great
data sets. Virginia, the governor. The governor's
office just launched an open data portal with with a
lot of great resources on it. We have to
leverage. And of course, there's a U.
S. Census api, that's that's really good.
And finally, if hacking is not your thing,
if you're not a programmer, uh, we're gonna
be we'll have a few activities. Um, for
anyone who knows how to Google, uh, we'll
have open street map editing where, uh, we'll
pull information into open street map. It's a really
easy editor. Just click and type. We've also
got to city censuses. So these are the open
data. Census is, uh, is a collection
of data sets. That code for America has found
that most cities have and should have, um,
openly accessible, and many cities have already contributed.
You can go out on the list and see other
cities and how accessible their data sets are and where
they are. And so we'd like to put Blacksburg
in there and get it on the map. We're
also going to do a new census called the local
Digital Services Census. And so that's a collection of
how accessible things like, um, how easy is
it to go find your town council members and and
figure out how to contact them? And how easy
is it to find? Um, you know,
bus schedules and things like that. So if you
can, If you can get on a computer and
find this information and then write down in the census
how easy it is to find, then you'd be
able to help us. Breakfast and lunch will be
provided tomorrow. We'll start here at nine o'clock in
the morning and go till five weather permitting. Uh
, and so we hope to see you all there
. Thanks. All right. Thanks, Ben.
So we're really honored to have tonight, uh,
follow Jake with who drove down here? Waldo Jake
with is the director of the U. S.
Open Data Institute, an organization building the capacity of
open data and supporting government in that mission in 2011
. In acknowledgement of his open data work. Jake
with was named a champion of change by the White
House and in 2012, and open Gove, champion
by the Sunlight Foundation. He went on to work
and open data with the White House Office of Science
and Technology Policy. Jake with is a 2005 Virginia
Tech graduate and lives near Charlottesville with his wife and
son. And just this afternoon, he was telling
me that he had an offer to speak about open
data in Europe, but he chose Blacksburg. Please
welcome. Although Jake with Yeah, I'm glad to
be here. I I have not been to Blacksburg
since 2007. Uh, and I graduated from Virginia
Tech in 2000 and five, although confess Italy as
a late life college student. So I'm going to
talk to you all for for a little while about
, uh first, I would explain generally about open
data, but it is how it works. Why
? It's useful, how it's different than closed data
or information generally, or things like that. Then
we're gonna look at just basically tell you a couple
of stories about the process of how I've opened a
couple of data sets. And then we'll do a
quick rundown of how things are looking in Virginia,
sort of a state of the commonwealth for data here
. So I want to begin with the question of
what is open data. So here's some context.
We have the whole universe of data, which is
the biggest circle there. So open data is just
a small subset of that. That's the portion of
all data is publicly accessible. And believe me,
this is not to scale in any way. This
circle for open it would be so tiny you would
not be able to spot. It's like when you
see, like, how many Earths can you fit
inside the sun like a million or whatever? It's
an absurd number. It's a little bit like that
in scale. There is some overlap with big data
open open data, but they're really basically unrelated.
People like to conflate them. The I think the
governor even lumps big data and open data into the
same bucket. Uh, this doesn't actually make any
sense in any way. A lot of the most
valuable open data is really, really small. Uh
, it is a little data it would fit on
a floppy disk. That's some really, really useful
data. So although my organizations work is an open
data, generally, most of my remarks here are
going to be confined to a subset of that which
is open government data. So here's the difference between
data and open data. Here's what allows data to
be transmuted into gold into open data, so the
first is being able to get it. You can't
get it. If it's not possible for you to
acquire it, then it is clearly not in fact
open. Ah, to be data it has to
be readable by software. Software has to be able
to do something with it. But there's another level
of of readable, and this is a little harder
to explain. And every time I explain it,
I hope I get a little better. And that
is the meaning of that data. The actual values
contained within it. Software has to be able to
understand it, so I'll make a short distinction here
, but I'll explain it better with some some graphics
in a minute here. If I have a spreadsheet
of a budget for a municipality and I take a
photo of it and I email it to somebody.
Well, that is data in the sense that it's
a photo that's on a computer. But the numbers
on those spreadsheets, I can do nothing with them
. I can't fiddle with the numbers and see how
it changes. It's just a picture, so that
is data in the abstract. But it's not data
in the sense that software can't read the information within
that spreadsheet. And so that's That's what I mean
by readable by software. Uh, if you're paying
for it, it's a lot less open if you
have to pay for that data in order to receive
it. It's not that it's not open. We're
talking about a sliding scale of openness here. It's
not an existential issue whether data is open, but
if you're paying for it, it's substantially less open
. And finally, you have to be able to
share it and do stuff with it. If it's
shared with you and only you and you can't share
it with anybody else, you can't change it.
You can't copyright prohibits that, or if license prohibits
it, it's also not very open. Now there's
a seven step measure and there's an 11 statement,
so there's a very spoiled down version of what it
means for data to be open. But we do
. We do speak of data being more or less
open. I'm trying to train myself to lose the
absolutist language. The data either is or is not
open. That is a false dichotomy. So let's
use weather data as an example, as it is
our most hackneyed example of open data. So I
will just keep going with it. Uh, so
going back in history, I think it's a good
argument you made that open data open. Government data
really started back in 18 70 and that is when
the National Weather Service was started. It wasn't called
that. Then, uh, it was created by
the White House. Was President. Grant had been
part of the initiative to to create this thing,
But the idea was this. We finally had the
ability to communicate faster than horseback. We had not
just an extensive train network, but we had telegraphy
, so weather observations were being used to make predictions
, and it was finally possible to say, Hey
, there's a storm here in the West year in
the East you might want to batten down the hatches
. And so this, uh, system was put
together by by the federal government for the general good
, but more specifically for the good of commerce to
be able to make these weather forecasts, particularly for
ships at sea and in the Great Lakes that there
was so much to be gained by saying, I'm
no sailor, but I know that sometimes putting on
the storm, you either want to get into Porter
out of port. And I don't really know which
weather circumstances warrant switch. But if you can know
that in advance, you can prevent huge amounts of
losses. And so electronic data just wasn't possible until
calligraphy. So that's part of why it came around
at that point. Uh, but they did name
it the Division of Telegrams and Reports for the benefit
of Commerce. That was the name of this department
, which was rather inelegant. And it was run
by the military not because it was seen as a
military function, but because the federal and government,
in 18 70 was such that there was no agency
or department have their act together adequately to monitor and
relay this information with the proper structure and authority in
such a regulated fashion that could be relied on except
for the Army, which is why they had run
it like that. Now, if we include analog
data, then the census would surely be our first
open data thing in the United States as prescribed in
the Constitution. But that, as with our photo
of a spreadsheet, the sense is sort of the
equivalent of a photo of numbers as well. Until
those numbers could be provided as data, they couldn't
be shared or moved or changed, and so on
, in the same way. So this is actually
the description that established this this early weather agency,
and you know that they were first specifically to the
magnetic telegraph that was the Internet of that. It
was the Victorian Internet. If there's a book by
that name, the Victorian Internet, which I recommend
highly, uh, everything, all the things we
wonder about now and the crazy things people do over
the Internet, we're all a thing. Them,
you know, online dating was a thing over topography
in the 18 seventies. Emoji, we done did
that like all of these, you know, the
shorthand and the goofy acronyms. This all happened in
our country almost 120 years ago. That was a
major cultural shift. It started then, uh so
, uh, let's step forward to the 19 seventies
. 100 years later, that is when the National
Weather Service, as it was then named, established
their first computer bulletin board system of BBS. Just
quick show of hands. Who here has ever heard
of a BBS? Not bad. Who here has
ever used a BBS? Fewer? Okay with some
. So it's okay for the rest of you all
before we had open access to the Internet before there
was a World Wide Web. So the same concept
, except instead of typing a website address instead,
you had to first discover it existed. So you
get a magazine like board watch, which would email
you like, Imagine getting a magazine every month with
like, here are the new websites that exists.
This is all of them. We're going to review
them for you, tell you what's good and what's
bad. So we got to make it a magazine
like that. It would be a list of phone
numbers for every website basically And so you get a
phone number and you plug the phone into your computer
into the modem and you dial out to that long
distance number and your computer would make a series of
squawking sounds. And then you would occupy a phone
line. One phone line for that website, such
as it is, and your phone line would be
in use and over your text interface. You could
download whatever information you wanted. But as like as
goofy as that sounds, it was a wonderful system
, and this is fundamentally the same concept under which
the Internet came to exist. The Internet. Instead
, you would dial up just once, and then
you could connect anything else on the Internet. Use
the Internet instead of phone lines. Basically well,
The National Weather Service created O. B. B
. S in the 19 seventies, where you could
dial up to their system in D. C.
No. Two or two area code and you could
download weather data. You can get real time observations
. You get historical data as these big text files
where you would be able to enter. I know
if they used an airport code or what at that
point, But you can get a big file to
be like you up here. The observations right now
. Or, you know, as recently as they
as they had them there. And that started just
a total change in how we use weather data in
this country. So there are no, like private
weather satellites. I remember a congressman. Oh,
it must have been 2010 or 2011 with all the
tea party members got into the house, a congressman
had a great idea. What are we using the
National Weather Service for? We got all these private
weather things. Shut it down. That went poorly
. Uh, that's where all of our weather data
comes from. It all comes from national. There
are no private weather satellites circling the earth. All
of our weather data comes from National Weather Service.
So all you know, the Weather Channel and so
on went to this guy is like, uh,
you might not know this you're going to put us
all out of business will be the end of all
weather data. You don't seem to understand how the
private sector works. Please stop. That was the
end of that. It was like a day and
a half long movement in the house that ended very
poorly. But this is how this this the system
they set up in 19 seventies. Now it's basically
the same thing. It's just on the Web and
the data that is, that is found there that
you get when you, if you were to go
and who goes to the National Weather Service website to
get their forecast like that's not really a thing.
You can like the data. Is there that same
data? That's what that's what made possible the Weather
Channel. The fact that they could dial up.
I assume they probably kept their modem dialed out all
the time. But you know, they could download
for, like, weather on the age to the
National Weather Service like there's no human in putting that
together at the National Weather Service. They're downloaded that
data from the National Weather Service update and get every
few minutes and automatically generating that message that you see
locally on your television. That's how they did it
on the Weather Channel 20 years ago, and that's
how they do it now. So now we have
dozens of websites every T V and radio station that
has weather constantly. Like they're not employing meteorologist.
Some are. But by and large, they're getting
all their data straight from the National Weather Service.
I have no fewer than seven weather apps on my
phone. I don't know what's wrong with me.
It's really more of a confession here. Uh,
these are made possible by National Weather Service data.
They all pull their data ultimately directly from the same
website. The you and I can go look at
one of them. My favorite is called Dark Sky
Dark says Great. It's only job is to keep
track of where you are, and it tells you
it's gonna start raining soon. That's it. It
was 20 minutes. Gonna start raining. Oh,
great. I can go inside or go outside like
, depending on, like my gardening like, what
am I doing now? It's a great system that
would not be possible if we didn't have this data
system. Uh, the entire electric power industry relies
on this incredibly heavily. You need to know how
much coal you need to order. And by the
way, that doesn't show up overnight, like you
need to have lots of advance warning. So if
it's going to be really hot in July, you
got to place that order and those trains gonna come
a long way. You don't have too much coal
because we're gonna put it. They don't have a
whole lot of space in the hopper like you need
to keep it moving from the trains and burn it
. So if you order too much or too little
, it's a problem. And that is true for
the entire electric power industry. Are trucking fleets are
airlines? All of our just in time delivery.
Uh, using a really conservative estimate, Uh,
our weather data is worth about$10 billion a year
to the economy, Probably worth a lot more.
But using a pretty strict definition, that's$10 billion
GPS network, by the way.$90 billion which
, of course, I'm just using to tell me
it's going to rain soon. It's not really the
highest use of a fleet of satellites circling the Earth
at all times to tell me a great restaurant nearby
. But that's amazing data that we've used to open
up so we can use yelp and foursquare and Facebook
. So just to make a quick comparison here.
This is what we're seeing. This isn't data.
This is information. The only purpose when you see
the weather printed in newspaper, all you can do
with that is read it and say Yep. Now
I know the weather. It doesn't flow. It's
only in the form that it's in. So imagine
if this is how the U. S government released
weather data. Just they printed it every day was
just like it came to your front door and you
read it like, yeah, that's the weather.
There would be no apps. There would be no
dark sky to tell me that it's about to rain
. Trucking fleets couldn't use this. This would not
be nearly as useful to power industry and so on
. A few of the economic benefits that do exist
from weather data would exist if we had it as
information instead of data and certainly if it was an
open data. So this is data. This is
meet our meteorological terminal aviation routine. Weather report.
Can anybody here read meat are similar? Question.
Is anybody here an amateur pilot because it's the same
question? No. Okay, so anybody who often
there's one person in the audience was like, Yes
, and they're very eager because nobody ever asked them
if they can read guitar. So this is their
moment to shine. I can barely read it.
Um, so this is a standard that's been used
since 1968 for weather data. Anybody who's a pilot
, they look at this right before they get they
got on their flight. So this is whether data
that I grabbed yesterday for KBC be the closest airport
, and this tells us the time the wind speed
and knots, the direction of the wind, the
cloud ceiling, the temperature, all of that.
So the trick is, uh, in theory,
software can read this, and humans can read it
in practice. I don't know what this says,
and I suspect none of us do. So we've
started moving towards some better systems represented data. So
this is a system known as XML. I understand
this is probably too small for most of you to
read on the screen here, but XML stands for
extensible markup language. This is a means of sharing
data that can be read by both humans and by
software. Some argue that XML is useless to either
, because it's really a pain to work with.
But this is how weather data is provided. So
I went right to the National Weather Service's website.
I chose view source in my browser, and this
exactly what it showed me. Um, so what
it lists out here and like pretty plain English is
things like weather. Mostly cloudy temperature negative. 17.6
in Celsius, zero in in Fahrenheit while I got
that right at the right moment. 2020.7 mile per
hour wind, gust, wind chill all that stuff
. So these are the current observations that I was
looking out here. That's all available in a way
that software knows what to do with it. It's
trivial for a programmer to write software that will look
at this source and display at the top of the
newspaper website. Here is the weather right now in
our town, Or to use that to put the
weather up on the National Weather Service to say,
Here's the weather in your town right now, but
it's also pretty easy for humans to you to read
it. Not that I recommend getting your weather like
this, but this is how far we've come.
We've moved from meet our two x m l.
And this is what powers the app on my phone
. It's grabbing this exact data. If I looked
at the weather right now on my iPhone, it
would detect where I am. It would get this
exact data feed from that very airport because it's the
one closest to us. So the point I'm trying
to make with this deep dive into weather is that
we have so much that is powered by open data
in this country very quietly in a way that most
people don't need to be aware of in the background
, Open data is substantially making the world go round
. So federal election results open data, campaign finance
, open data, crime incidents, transit data is
y'all y'all heard about earlier address coordinates area code locations
on and on and on. All these systems are
impossible. Without open data, zip codes would be
pretty useless to anybody but the USPS if they weren't
open data. But because there it is, open
information, then that can be used by UPS and
FedEx and DHL. Anybody else to be able to
use the same structure to be able to move packages
around in the world. That's now compare that,
by the way, to the U. K.
Where address databases are considered proprietary shall the Postal Service
and they're not shared. That really prevents a lot
of private innovation. As a result, it's not
a great system. Our shipping and transit system is
totally dependent on open data in so many ways,
the ability to get import and export data to be
able to ability to share the what's the word.
They use the bill of Laden, the bill of
lading. That's it. That's the term of art
in shipping. There's a whole standard for the bill
of lading. So instead of reading these physical things
outside of every box on every great on on a
ship that's transmit electronically weather data, the real time
location of ships and title data and all of that
is needed for how we move things across the seas
, there's similar analogue for how we move things across
land. This is crucial. We also use open
data in a bunch of places where you really wouldn't
think you'd find it. So my favorite example of
private open data is the stock market. There is
no higher example of open data being used outside of
government. In the stock market, stock market is
nothing but a huge pile of open data. Here's
what people are paying for this stock right now,
and here's what they paid yesterday and last week and
10 seconds ago or 10 milliseconds. It goes as
the case may be, that's all they do.
That's their job. Uh, it used to be
much more complicated. Now it's, here's the data
, and and there's a the ability to make trades
on there as well. But fundamentally, that's what
they trade in. Is that data, uh,
crucial? If we didn't have Imagine if NASDAQ's just
said, Yeah, we're shutting it down. We're
not providing stock date anymore. The time that NASDAQ
would continue to exist, we measured in hours because
why, what purpose do they serve anymore? Product
recalls. That's all. Open data. It's going
to be open data. OSHA fatality reports real time
earthquake data. There's an earthquake anywhere in this world
you're going to see it on. I think it's
an earthquake, Gov. Within about 10 to 30
seconds. So when we had a big earthquake just
a few miles from my house a few years ago
, and we had aftershocks pretty regularly. Routinely,
I think, was that an earthquake? Let me
check pulled off my iPhone. That's how I found
out there was an earthquake. By the time I
get to the USGS website, they know I know
how strong it is, Where it was centered,
how deep it is, that's magical, the real
time location of every airplane and cargo ship or even
large ship in the world. That's all available all
the time. It's amazing when I go to the
beach and go to the other banks and I see
ships sailing by in the horizon. I just pull
up one of the cargo tracking ships. So yeah
, that's what's going from that's where it's going to
. Here's a picture of it. Here's what it's
carrying. It's really cool now for me, it's
just really cool. If you're in that industry,
it's essential. Uh, one of my favorite weird
examples of open data is about a year ago,
Marvel, the comic company. They created what's known
as an API, an application programming interfaces like open
data. But instead of downloading a file, you
just ask for information about one thing like, What
is the weather in this place? Here it is
, uh, about Marvel Comics. So you know
, Spiderman and X men and so on, atomized
down to every tiny little element of every issue.
So you can say I would like a list of
every issue of Spider Man from the 19 seventies that
Stanley is on the credits for writing returned instantly.
Or who was the artist? Who's the artist who
drew the most covers between 1985 and 1987? Answer
right away. Now this sounds a little goofy,
but it's not really comic. Book collecting is huge
. I'd be very surprised if, UH Newman are
certainly library system at Virginia Tech didn't have a decent
collection of comics at this point. Uh, whatever
concept might have existed, decades about comics is like
, you know, Donald Duck is things for kids
. It's become serious literature collectors of these things.
They want to know what's in every issue so they
might use. There's a lot of software you can
get, like for desktop computer that can track issues
of, uh, the issues of comics that you
have if you can interface that as they do now
with Marvel's AP Well, it's not like you now
. Just used to be You just know the title
and maybe the date that it came out. Now
you're boring. Database of your collection of comics is
incredibly rich. The amount that you know. So
if you're wondering like, what was that one issue
where you don't need to wonder anymore answered, solved
. So they opened. It is in use in
all sorts of interesting places. Now, unfortunately,
not enough places Legislative data no often just doesn't exist
. Laws. The laws that govern us know usually
just not available as open data municipal G. I
s data registered corporations, restaurant inspections i. R
s nine nineties FCC political ad spending. There are
so many really great examples of incredibly valuable data that
either isn't available responsibly available, and you'll note that
it's all really crucial stuff. That's not a coincidence
. As a rule, the more valuable data is
, the more politically sensitive it is, and the
more likely it is, will be some reasons joined
up as to why that it shouldn't be released as
open data. Some of those concerns are legitimate.
A lot of it is often concern trolling. They're
not really concerned. It's just somebody doesn't want to
publish it. In fact, I just posed sitting
here reading, Reading Twitter is waiting for for us
to start here. About an hour ago, I
saw that two members of the General Assembly who also
holds state jobs. In fact, I saw Megan
Ryan in the back there. Quote in the article
, who also holds state jobs, refused to say
how much they're being paid at their state jobs.
These are members of the General Assembly who are also
on the state payroll, who say it's a secret
how much they're paid now. I don't know many
people who believe that it should be a secret how
much state employees are paid. If anything should be
public, it seems like to be that doubly so
if you're also a legislator, Tripoli. So if
you are in the case of one of these,
a legislator who has argued for ethics reform that we
need to have more of this data available that's not
available as open data, why? I think we
see why we have powerful members of the General Assembly
who don't want things known about them personally, and
this is not a great system. So it's one
thing to persuade people in a room like this why
government should open its data and why it's a good
idea. It's quite another to persuade government of the
same. So we hear a lot of failing arguments
. Some of these arguments can be successful in riling
up the crowd. Some of them can be successful
to persuade elected officials of the merits of it,
but not so successful at actually getting data published.
So one of them is we paid for it.
Uh, government has a fundamental obligation to be transparent
. There's another one. It has economic value.
We have a right to it. These are all
sound and ineffective arguments. Uh, and when I'm
speaking to audiences include a lot of people in government
. There are lot of nodding heads because they know
that somebody walking into their office and saying, I
demand that you published data because I'm an American and
the Constitution says you have to do this because it's
information it's public and FOIA and give it to me
like you're gonna find every reason you can not to
give them that data like that's just rude, and
people don't want to be treated like that. And
that's often how many of these arguments boil down.
What is really healthier is regarding open data is being
useful for governments. The government should produce data to
share between agencies between levels of government because that is
a better way for government to work and more on
this later. But the gist of it is that
if open data can make the jobs and, like
personal lives of people in government, easier, then
of course it's going to happen. Why wouldn't it
happen? But if it makes their jobs harder to
the extent to which it makes them likely to be
hauled before a Senate subcommittee to testify as to how
inaccurate data got published, then they're probably not going
to do it. Uh, but as a happy
byproduct of this idea, that government should share data
within government. The rest of us get this data
to so the first story here legislative video because I
love this story and because a reporter is actually working
on a story for about a Monday, I think
for The Virginian Pilot. So I had to tell
him this story about it a few days ago,
this high pride video out of the hands of the
Virginia General Assembly to make them open up their data
. So the Legislature streams video of its floor sessions
, not committee meetings. You're out of luck there
if you are not at their pre dawn meetings in
Richmond, and that's a long drive from here.
If you don't know. And also Virginia goes a
lot farther Southwest, so it's a multi day trip
for a lot of people. If you're not in
the literally predawn meetings in these rooms in the capital
, then you don't get to see them. But
it turns out that they're floor sessions. They stream
them live on the Web and they're saved to DVD
. They capture, they burn them live to DVDs
. So if it goes along, they have to
, like, switch out DVDs and you miss a
few seconds. Um, it's not a great system
, but that's how they do it. You can't
get archived video anywhere unless you go to the General
Assembly and you pay them for the DVDs. So
I learned this in oh, 2008 January 2000 and
eight. I thought this might be fun. So
I went to the General Assembly, said, Hey
, I'd like the DVD for yesterday's session to I
forgot it was the House or Senate clerk, So
I was like, No House clerk's office. Sure
, that'll be$10 okay for$10 and we waited
. I think few hours later they had the disk
ready for me and I took it home and I
ripped it, and I put it up on YouTube
and I did it again the next day, and
the third day I was told I was not allowed
to get the video anymore, and I wasn't quite
sure what the story was. But I talked to
a friend of mine, the Virginia A C L
U. They kicked me up to the national issue
. Who sent a threatening letter to I gather the
majority leader of the house, and then I could
get the video again. Uh, and I've been
doing it ever since. I get these stacks of
DVDs in the mail. It cost me$800 per
session. I buy all of the video one by
one, I load the DVDs into my eye mask
I ripped the video and I upload it and make
it available for free so that nobody else has to
buy the video. And the crazy thing is,
I started doing this in 2000 and eight. I'm
still doing it. I thought I'd do it for
a year and they be like, Oh, this
is Let's just start doing it. If this dude
can do it in his spare time, we can
probably manage it. As the oldest legislative body in
the hemisphere, I think we can probably swing this
. I was wrong. I vastly underestimated it.
So I upload all this video to the Internet archive
at archive dot org to be hosted permanently and for
free, which is great for me. There is
other than buying the freaking video, which is crazy
. Uh, there's no cost. I've tried by
the way to like, say, can you just
give it to me in like a memory sticks?
I don't need to rip. These are like upload
it so I could download it or like that would
be great. But no, they only have the
DVDs, and they will not use any other system
other than copying DVDs and I have these, my
, like cleaning out my attic a few weeks ago
. Like I just have, like, boxes of
Devi like, full of DVDs. I had to
throw them away. Like, I guess I told
him to the State Library. I don't know.
This is just This is a crazy system. So
here's a short clip of what that video looks like
. Seven of 473. Build a repeal and re
enact various sections of Virginia uniform. Foreign country money
judgments, recognition act. Exciting stuff with an amendment
. The John Sailing, Mr. Speaker, I
move the committee amendment questions on Adoption Committee of Atmosphere
that emotional say, I know the graphic in the
top right of the screen There, gentlemen. Thank
you, Mr Speaker. Members of the Senate And
now one of the three replaces the 1990 uniform Foreign
country, Money, Judgment Recognition Act and modernized process
for his mouth is out of sync with the audio
uniform. Law Council will be about the 20th ST
without the bill. You get the idea. So
, um, there are three bits of metadata on
the screen right now that I use. So the
first is each of the Chiron's. That's the industry
term that's used for the graphics. So one in
the corner identifies the bill that's being discussed right now
, and one identifies the person who's speaking. And
the third piece of metadata is delicate. Habib's face
There's 1/4 we can't see. That's the audio,
his voice pitch in his actual words. So I
use all five of those two index every second video
. So I once a second I look for these
. Kyra, Kyra is in the video, and
if I find them, I, uh, grab
that graphic I transform into black and white. I
inverse the colors and stretch the contrast. And then
I run it through OCR software to turn it into
text. And then I load it into the database
with a time stamp of the video, the day
, the session and the second in which that happened
. And I do some transformations. The database to
set to say Okay, delegate Gregory Hobby has this
idea. Is this legislator He's in this district.
Make sure that the name is accurate, that the
bill exists and so on. Um, and as
a result, I wind up with this is a
site I run about the General Assembly. Uh so
So here. Here's where the bill winds up.
The data winds up being used. So this bill
says that Quantico is allowed to set its own speed
limits. So imagine if you wanted to learn about
why delegate Dudenhoeffer introduced this bill. Well, because
we know the times in which the bill was discussed
, we can dynamically create an excerpt so you could
just watch the video and listen to the author of
the bill. Explain it. Uhh why the video
is not playing. For some reason, this can
be very dramatic. Anyway, uh, so,
uh, you would hear their delicate never explaining why
it's necessary for you to go to have that exemption
. I never call it being very persuasive. But
the value of this is that otherwise, when you
look at a bill, you have no idea why
it passed. But so to be able to look
at the video clips when it was discussed on the
floor of the General Assembly to see a discussion of
exactly why the will is being proposed in what accomplishes
it eliminates any mystery about legislative intent. You just
know and to be able to connect that with the
actual bills is is really important. Uh, we
did that. So in addition to using that data
on my site which, like, I don't really
much care about, um it's more that I also
give the data away. So this is the download
section of Richmond Sunlight, which is very glamorous looking
and that video index file at the bottom there,
if we open it up. This is a file
format called Jason JavaScript. Object notation is kind of
like XML. It's a way to represent data in
a really raw way. So every time somebody spoken
on the floor of the Legislature, we record here
the chamber, the name of the speaker of the
bill they were addressing when they started speaking when they
stopped speaking and then link to the video file.
Anybody can grab the data and do anything they want
with it, which is bound to be much more
interesting than anything that I've done with it that seems
ripe for some pretty great analysis. I still have
not persuaded the General Assembly to that they should do
this. They have all sorts of really lame excuses
as to why they haven't published this stuff. I
don't know what their deal is. Second story,
business records, Uh, the record of every corporation
registered in Virginia. There about 800,000 corporations in Virginia
that exists now or have existed in the past.
I think the records go back maybe 15 years.
These records are maintained by the State Corporation Commission,
and they're wildly valuable, as you can imagine for
all kinds of purposes. Um, just any business
in which you need leads. There is a strong
example. Uh, but no doubt we can all
think of many examples of why it might be useful
to know what businesses actually exist in your town.
In your county, in your state. Who owns
them? Who is on the on the board?
How many shares people own name changes. It's all
really useful stuff. So the only way to get
this data right now is from the State Corporation Commission's
website. One record at a time. There are
800,000, and you have to do it manually on
a website, and the available information they provide is
incomplete. They actually have more data than they bother
to display on the website. But like the Journal
assembly. They will sell it to you. It's
$150 a month subscription fee, which I've been paying
since April, I think. And I'm grateful to
my wife for her tolerance of this budget line item
. In our household, there are seven paying customers
. I foiled the list. I'm number seven.
Uh, I have a hard time believing that it
costs them less to administer this program than they make
an income. Uh, this can't be like a
very sensible use of their time billing me for$400
every three months. Uh, so, uh,
luckily, great organization called the Shuttleworth Foundation sent me
out of the blue$5000 this summer to defray my
cost of buying government data That should be free and
giving it away to actually make it free. Uh
, which was nice of them. So I had
to sign this five page contract. Remember the rules
of openness. I had to sign a contract to
receive public data. That's like a big red flag
, I think for the state Right there. So
So this, you know, fails to be open
data. Uh, but the contract I read over
carefully and I consulted with the Electronic Frontier Foundation in
the American Civil Liberties Union. Uh, can I
give this data away or they're gonna sue me?
And they said, I think you're okay. We
think you can actually get this data away. The
contract doesn't prohibit it. So I signed the contract
sending the check, and as soon as he gave
me access within 10 minutes, I'd written a script
. They would download the file, which they update
at 2 a.m. Every Wednesday morning and uploaded to a
website where anybody can download it for free so they
will never get any more customers. They are capped
at seven. Because you'd be crazy to pay for
something that I'm giving away for free. Uh,
and for me to download that data and give it
away cost me upwards of several cents each month for
the data transfer. Uh, so no big cost
for me either. This is what the data looks
like. It is really ugly. These are just
five random business records. Believe it or not,
there is a huge amount of data available in here
. Uh, good luck finding it, but it
is in there, and it was my unfortunate job
to get it. Imagine going spelunking without a flashlight
that what it was like for months trying to make
my way through this file and extract useful data out
of it. But I use that data to start
an open source project, which turns the data into
a spreadsheet and also into the aforementioned Jason, and
then loads it into a database named Elasticsearch that makes
it searchable. The heart of it all is this
little command line program. I named it Trump for
Beverly T. Crump, the very first SEC commissioner
, and note that despite being a woman's name because
it is somebody important in government, it's a dude
, Beverly, that's That's Virginia's government. It was
also 1900 the defense of Beverly. But what this
program does, it operates through the file, which
is, I think, millions of lines long and
breaks it up into tiny little pieces, fixes formatting
errors and text encoding errors, uh, and loads
it all into that database to be indexed, and
it pops up a website. Looks like this.
This is all free open source software. Anybody who
wanted to compete with me, I don't know why
I don't make any money off. This is no
advertising could download the software I've written and run it
and set up their own website. So this is
via businesses dot org. Anybody can download this bulk
data that I get, but I put it better
formats. There's an interface to review each record and
browse through it. But with all the data,
unlike the FCC's website, and I think that other
people will do much more clever things with us than
this ugly little website that I have slept together.
Also courtesy of Code for America, the Hampton Roads
chapter, a developer there without me having to do
any work whatsoever. Put an A p the application
programming interface on it for the site, which is
wonderful because of what it means is that somebody can
enter a U. R l like the one here
website address. So this is saying, Give me
any businesses named Gilley's in the city of Blacksburg,
and it returns exactly one result, and it informs
us that Gillies, a restaurant that I was fond
of when I lived around the corner from here in
Blacksburg, on college at Jan. Arguably, is
the president vice president, treasurer, secretary and Richard
agent. This is clearly a very capable woman.
She incorporated in 1983. There are 20,000 shares she
owns all of them. And we have this exact
same data available for 800,000 other businesses. A funny
outcome of this that I never anticipated in any way
. I got a phone call from a employee of
a town here in Virginia back in October. Who
said, Hey, I stumbled across this website of
yours. Could we get the records for all businesses
in our town? And I said, I suppose
I could geo code all the data and break it
up by town. Why do you need it?
She said. Well, you know, we have
a business license in a business license tax here,
but we only know if somebody starts a business here
. If they voluntarily fill out the form and tell
us they need to pay taxes, we have no
way of auditing it. We don't know what businesses
in or out of town. I feel like that
. Doesn't the State Corporation Commission tell you when there's
a new business in your talent? Said No,
no, no. We have no way of knowing
as but can't you ask them for the day?
They said, Well, no. And I explained
to her they could pay for the data, But
she said, We're not programmers. We wouldn't know
what to do with this big file that is so
ugly. So really, this seems like a terrible
system. So I taught myself how to over the
next few weeks, Geo code, all the data
, as we say, to assign latitude and longitudes
to every business and got the boundary data for every
town and county and city in the state and then
center the records for that town. I then heard
from the Commission of Revenue from my city who wanted
the same data and the round up being a couple
of stories in the press about a couple weeks ago
. And I've heard from a bunch of counties that
are saying, Oh my God, this would be
wonderful. We have no way of knowing who's not
paying. We did an audit of Charlottesville Records and
found that they're missing somewhere. Well, they're missing
1900 businesses, so I think about a third of
their businesses they don't know our in the city.
Now, not all businesses have to pay license fees
, and it's not clear exactly what percentage do.
But the average amount that businesses paying license tax to
Charlottesville every year is about 1500 change, so they're
missing out about$2 million in revenue. If all
those businesses are supposed to pay license taxes, if
we pretend that only 80% of them are exempted for
some reason, that strikes me as real unlikely.
There's still 600,000 year they're missing out on now,
just for scale. The city's budget is 100 50
year, and they get to go back and recover
five missing years. So even if it's a million
, they get to put a five. There's a
$4.5 million budget shortfall in Charlottesville right now. They
can fix that just by with this audit and which
they're doing right now. They're using these records to
audit. They need to hire a new auditor because
they've never tried to audit 1900 records, but they're
they're doing their best, uh, looking ahead.
I want the State Corporation Commission to do this.
It is crazy that I am doing this, That
we're going to have almost 200 localities when you include
towns failing to be able to audit their data unless
I show up and do something in my spare time
for fun. This is a really lousy system.
So there's, uh the State C. I.
O is becoming the chief administrative officer of the State
Corporation Commission. Uh, he knows about my work
. I'm optimistic that he'll show up and say,
Guys, what are you doing? Why are we
not helping our own localities? And by the way
, I have been told that this state is not
being used anywhere else either, including state Department of
Taxation. If they wanted to know what businesses are
not paying taxes, that they are not able to
use this data to audit their own records. This
is a terrible system. So when I say open
data by government for government, this is the example
. We're looking at$200 million across the state.
I mean, how conservative are optimistic? You want
to be in your in your estimates. Feeling more
concerned to say 100 million in missing taxes to localities
, because the state isn't sharing their data This is
crazy. So we wrap up with the state of
the Commonwealth, a quick rundown of where Virginia stands
in its provision of some key data sets. Now
there's nothing magic about these data sets that have listed
here. I happen to regard them as important for
states to provide. There is there are no particular
order, but let's let's go through them. Legislation
and legislators. The Legislature provides spreadsheets of bills and
votes. That's great. There's no budget data.
There's no legislative data. There's no committee data.
Uh, and getting the data is Unadvertised. It
requires registration. It's not updated off enough. I
give them a C Geo data. Virginia does two
amazing things with geographic data. The first is there
one of, I think two, maybe three states
in the country to publish as of December, it
was known as an address points database, every address
, every assigned address in the state. They have
a latitude and longitude for, and they'll give you
that list. It's available. You can download it
freely. That is huge. That is incredibly valuable
. The cost of geo coding data, which all
sorts of businesses need to convert addresses into actual locations
is prohibitive. When I needed to geo code my
Virginia business data it was going to cost me$1200
a week to use a commercial geo coded like,
Every time I run it, I have to pay
that that feet. Um, even if I was
really, uh, cautious in my my software development
, uh, and how I did it, it
would still be 1200 upfront and hundreds of dollars every
subsequent week. Now it's free. It doesn't cost
me a thing. Thanks, Virginia. Virginia does
another really great thing with its geo data. It
has an openly available geo coder. So they have
an AP, which I think we all know about
now and happy where you can give it an address
and it returns a cleaned up, proper version of
the address and its latitude and longitude. That's that's
magic. That's wonderful. That is incredibly rare thing
for states to do all the rest of the geo
data. And my God, there's a lot.
I mean, congressional districts and house districts and soil
types and watersheds. All these things it's complicated and
it's spotty. It's poorly documented, but by God
to give them so much credit for the address points
database and the openly available geo coding. I give
Virginia B plus on Geo data judicial records. There
is none f nothing. There's nothing. It's a
wasteland. Laws and regulations, best in the nation
. Nobody is even close. Virginia Has there not
been a second place? By the way, Virginia
has, uh, an A P I and bulk
data for all laws, all regulations, charters,
Constitution A and a bunch of pluses, like whatever
you're supposed to do with an eBay review like that
, a plus plus plus plus. Plus, I
could be proud of Virginia in that respect. Corporate
Data D For reasons that are clear to all of
you. Now the data exists. But come on
, it's awful. Campaign finance data. The State
Board of Elections provides bulk data for campaign finance reports
. They just started a couple years ago. They're
not going back there. Hard to gather, and
that's all that's provided. There are no election results
, but that's still a lot better than most states
. So on a curve, I'll give him a
B minus. They're doing some innovative stuff. Health
data. Now there's a lot of potential health data
out there So let's just set the bar real low
because that's where it needs to be for health data
, because it's in a bad way. Nationally,
let's just say restaurant inspections, nursing home inspections and
vaccination data. So there is no restaurant inspection data
, just the Canadian website that isn't even indexed by
Google. That's where. Health space dot com How
space dot c? A. I think it's the
site. Uh, that's where we keep our data
. And, like Internet Ghetto, Uh, so
that's not data that's barely information. The vaccine data
is messy, but it's their lousy data for nursing
home inspections. I give Virginia D on health data
and finally having a central data repository. So the
good news is there is a data. Virginia Gov
. The bad news is that it's bad. Uh
, it's not a proper data repository. It's a
website dressed up to look like a data repository.
It's just a pile of HTML, but at least
it exists. So see, uh, and I
think that's gonna be improving soon. So here's what
we need. Just example that I could go on
and on. But these are some core types of
data that Virginia should provide but does not in some
of this you'll notice is not very exciting. Some
of it is really boring a list of all schools
. Yeah, but how else are you gonna know
what all the schools are? And that's useful for
all kinds of things. And these things change schools
or move or you build new schools, their new
principles, Uh, the grades, they serve my
change. Um, you know, So there's a
lot of types of data like this that are all
elected officials that is not known. The state does
not know what all the elected officials are in the
state. Not only they not know. They don't
know how they're going to find out. This seems
like a problem. So for the new requirements to
file these ethics reports from elected officials across the state
state doesn't know who's supposed to file because they don't
know who is elected. They have no way of
finding out data. How does it work? Uh
, but I'll include with some positive trends. Geo
data is improving fast code for Hampton Roads have been
doing some great data opening up health data with our
open health inspections out by doing the really awkward work
of pulling that data, scraping as we call it
out of health space dot c. A business data
is improving with a new data and website. AP
. The current governor isn't interested in open data,
and they co opted me by getting me to write
the transition plan for the new governor for open data
and then helping them execute it. So I hope
that will add up to some sort of good.
So I think Virginia is finally on an upswing.
Uh, for years, it was embarrassing to the
conference is about open data, and people say,
Hey, Virginia, how are things there? Good
, Grim. It's real grim, like there was
never any sign to 2.2 to say, like things
are getting better. But in the last 12 months
, things have changed hugely, and I think it
just it's timing. I don't I don't think it
necessarily. Some of the credit goes to a new
governor in who's looking for new ideas, and that
happens with the change of administration, and some of
it comes to agency is just saying Now we have
to improve how we do things. So yeah,
questions. What have I said, that is stupid
or wrong or confusing or what have I not explained
properly? I did everything perfectly. Is that yes
. Megan? Um, no. But particular are
working with commercial. Right? Veronica's if you to
do to try to mash up some of the data
when they get involved, Does that make it a
prohibitive a situation graduate? That on the back?
So there's a good news bad news of when governments
go to companies that specialize in working with government to
make government data information available. The good news is
many governments lack the internal expertise to do this themselves
. And they not only do they not know how
to do it, they don't even know how to
hire somebody to do it. And so they're Granic
asses, the big vendor for video. If you're
looking at video online, like from the General Assembly
, are really just about any other government country.
It's coming through Caracas. Um, Gracchus has been
talking about every year or two, they announced they're
going to have a big new open data initiative.
All this video is going to be available, and
it never actually seems to happen. And so it's
become sort of a ghetto for open data like it's
where open data goes to die like people ignore it
there. It's never going to come out. There's
no way for it to work its way up into
proper open data. Uh, Secreta is one of
the major vendors what's known as open data repositories central
software, whereas one website you can go to,
like data Virginia dot gov and all the data that's
available can be registered there and made searchable. So
, for instance, on the federal level website that
I made some years ago ethics dot gov where you
can type in the name of anybody. And it
will tell you how they visited the White House.
If so, how many times have they donated?
If so, to whom have they registered as a
lobbyist? When who do they lobby for? It
searches all these data sets in one place. So
the secret is the major vendor in that space.
Uh uh. There isn't that problem with secret to
the good news is when you load your datasets into
that, they can come out again. Uh,
they remain accessible and, um, fungible and movable
and so on. Generally, I think it's a
really positive step. When governments are willing to work
with vendors like that. We need so many more
. There's so few vendors. My organization, in
fact, just awarded a large contract this week to
create open data repository software that is so easy to
launch and manage child repositories that we hope it will
get a bunch of people in the business that people
say. I guess I could be a government contractor
and sell government or software data repositories now because it's
just so easy to do with this software. We're
giving the software away because we think that we really
need competition in this space. So it's it's a
mixed bag is to answer your question. But I'm
glad to see the forward trend of governments trying to
release data, even if they're not doing it and
what I think is a great way just yet.
What else? So on your list of, uh
, what's needed or we need one of these schools
, which you mentioned so when you mentioned that we
need it, are you proposing that someone develop something
to scrape this and manage such a database? Are
you suggesting that administratively or legislatively, the state make
some changes that filter through the school systems that assemble
this information. What I have my own. Like
John GALT Ian fantasy of all developers say no more
scrapers. We're not writing any more scrapers. That's
it. If government isn't releasing data, that's it
. You can't get it. We're done. Um
, and that would be, uh, wow.
It's alarming hearing a klaxon go off while talking because
it's like either I've said something horribly wrong or we
need to evacuate the building. It's a good reality
check, though. I appreciate that and re evaluate
everything I just said, Um uh, yeah,
so I was not going to happen. But I
do like it is a common topic among my fellow
developers of like we have to stop. It's like
you're you're enabling somebody's bad behavior, like buying beers
for an alcoholic like No, we'll just scrape that
data for you. Um, what we really need
to do is for government to release this. God
help us. If the state doesn't have a list
of all at schools and that could be if the
state doesn't know all the elected officials are, maybe
the state doesn't know But I propose to you that
either a they have the data, in which case
publish it. Why not? What's the harm?
Make it available to everybody? People will do interesting
things with it. I bet localities could use it
or B if they don't have a list of all
the schools. And that's the real problem. Not
that the data is not available, but the state
doesn't know what the schools are. And I don't
know which outcome is more alarming that they have a
list of schools and they won't give it to us
or they just don't know what their schools are.
So for all of these data sets, I would
like government to release these. I think it would
be useful to the private sector I know would be
useful to government quick follow up. So that means
that for schools, for example, there must be
private databases that are sold to places like Zillow dot
com, where you can find what schools. So
that's what's happening now is that some of this data
is available, but it's expensive with, as with
the example of geo coding, I could have paid
$1200 to get latitude and why couldn't like that wasn't
actually going to happen. But theoretically, one could
paid$1200 to get all of those addresses. Geo
coded in practice. There's a lot of innovation that
simply doesn't happen as a result. So uh,
yeah, so sometimes the data does exist for exactly
for things like Zillow, but it's it's tough to
know that it's accurate. It's tough. No,
that's up to date. And then it's only available
to Zillow. So you're Yeah, the cost of
work the government has already done. The private sector
has to bear repeatedly, and this is incredibly inefficient
, and I think it's also in the abstract.
But I think it's also it's unfair to taxpayers who
, returning to my list of ineffective arguments, have
already paid for this data. Why do we need
to pay for it again when it's already available?
Mhm. Well, they are related to the sort
of quasi commercial activities of the state. The lottery
comes to mind. Maybe there's some other ones to
lotteries actually, on my list, like, why
not put up the list of all the lottery numbers
? I don't know what somebody's going to do with
a list of every Like that data is not useful
the next day, you know, like either you
want or you didn't. But but yeah, I
want that available. Yeah, like as a amateur
statistician, which is the most dangerous kind of statistician
, by the way, Uh, I would love
to analyze winning lottery numbers, Virginia. That's,
uh, if it's not totally random, that's alarming
, right? Like, yeah, I think there's
a lot of data like that. That would be
really interesting. Yeah. Any other questions? Yes
, sir. Okay. Yeah. It's crazy passion
. I love it. Uh, specific area.
You government board open data that you really right.
Really? I'm really excited right now, and yet
I've done nothing on it. So maybe I'm some
cognitive dissonance here of transportation data. There's a gold
mine and transportation data. So the Virginia Department of
Transportation on there, I think it's 511 Virginia dot
org Virginia 51 I don't know any other website where
you can see at all times. Uh, what
are the incidents occurring right now? What lanes are
closed? What? Construction is planned? Uh,
where has there been an accident? Where the traffic
cameras so that I can look at the traffic in
real time. They provide all of that, but
only on their website. And so I wrote some
code automatically pull it off their website. Within a
few days, they blocked me. Why? This
is really useful. I would love. I don't
want to have to visit Virginia 511 dot org,
which looks terrible on the phone before every trip.
I just want Google Maps to be able to grab
their data and tell me you're getting off on 11
. Get off 81 getting 11. Just trust me
. You're gonna be happier if you do that.
I shouldn't have to review it and plan my own
trip. It's 2015, like our software should be
able to react to this. So there's a lot
of ways in which transportation that's that's a really pedestrian
. Well, I shouldn't potentially has a literal meaning
here, but has a really boring, uh,
mundane example of the value of it. But the
lack of data sharing on a local, state and
federal level between transportation agencies on both mass transit and
transportation generally is appalling and wasteful, and it is
really holding back a lot of innovation. So I'm
excited about that. And a close runner up is
healthcare data. The amount of money waiting to be
saved in health data is vast. Doctors ought to
be able to see that patient who has come in
four times with asthma attacks in the past year.
She lives down wind from a paper mill and her
asthma attacks coincide with incidents we see from EPA data
of excess pollution coming from their which we can now
see kills 10 people a year in this one town
because we can correlate those data. So then doctors
can say, Oh, I see here from asthma
attack I also see that where you live is downwind
and your attack Yes, you need to move the
little girl. Are you going to die? Tough
advice for a nine year old possibly hard to act
upon, but certainly variable, useful public policy data
. So there's a lot of ways that health data
is also waiting to be used for individual value for
community value for hospital value. In particular, as
we see the Affordable Care Act do more to bend
the cost curve, healthcare doctors and hospitals are gonna
have a strong incentive, a financial incentive to keep
patients well. And so they're about to be very
excited about data like this that will allow them to
drive down their costs by keeping people healthier. Because
at the moment we're not blaming an individual doctors here
, but the industry, On the whole, it's
like the power industry. The power industry is supposed
to tell people to use less power. That's all
they sell. Why do they want people to confess
crazy? There's no other industry in which would make
them tell people and, like put systems in place
to use less of their product. Well, tobacco
, I suppose there's another one kind of alcohol.
Um, but as with health care, like for
a doctor, I mean, unless you're a monster
, this isn't individually true. But there's a perverse
incentive you want your patients to be sick because that's
your business model. So we need to find ways
to help doctors see ways that they can make individual
and all of their patients better in the affordable character
, starting to force that. So they're about to
be pretty excited about that with health care data.
Sorry, saw another hand somewhere, So you talked
about how different levels of government and the same levels
of government don't communicate with each other about with with
regard to sharing their data. What is their response
when you speak to only love? Yeah, So
their response to because you're basically being government needs to
change. We need to change how we do.
You know, when I talked to audiences that are
mostly government, a reliable laugh, you're gonna have
to trust me because this will not be funny to
interview. But a reliable laugh line is when I
say Here's what data looks like in government or open
data looks like in government. You walking down and
knocking on Norma Jean's door and saying, Hey,
Norma Jean, listen, I know you're the database
administrator. Could you export some data for me?
Here's a thumb drive that would be great. Or
it's going to like the L Drive on the network
and going through the crazy, calcified network structure that
somebody actually put together in 1999. I didn't think
it would still be in use to find the data
that you need. There's a belief among open data
types in the private sector. The government is just
sitting on, like these amazing databases of all of
their data that they can access and they just won't
share it. And it's a it's a mess in
there. And so when I talk about improving the
provision of data within government for data people in government
, that's often when their heads started like, Oh
, thank God, somebody's thinking of us because from
their perspective, it can be really frustrating to see
all this talk about improving data for the public.
And they're thinking, Yeah, but what about me
? Our system is terrible. How do we make
my job easier? And so it's usually received enthusiastically
, and I T folks might object, but their
objection is Usually we don't have the resources. Yeah
, because we regard I t. As like these
Scooby people who are supposed to live in the basement
, who we pay not wildly well, but just
enough to appease the technology gods because it's some add
on that we need instead of recognizing that it's 2015
. It's the underpinning of a lot of governments and
technology, and society is mediated through digital technology,
and so it is no longer a bolt on.
It's It's how government needs to work. And so
that's That's when I start to win over the people
in government. Yes, thank you, Thank you
. We are important. But I think that's usually
why it's received pretty well by folks, folks within
government. And there's a big move in the federal
government to change this. After the disaster of healthcare
dot gov, which was outsourced for a fortune famously
to several companies who did an awful job, there
was a realization within the federal government. You know
, that that site was bailed out by a dozen
people, a few of whom I was pleased to
work with when I was in the White House a
few years ago who were called up, many of
whom were no longer in federal service, were called
up, saying, We need you, your country
needs you. Please report to a hotel room in
Columbia, Maryland, and you're going to live here
indefinitely and fix this website. The website was fixed
by a dozen people in a hotel suite on sleepless
nights when they said the whole assistant we used to
build this this whole junkie website is out. We're
doing it our way, and they updated the website
daily. They constantly fixed, and it got so
much better in a month. It took years to
screw up. They fixed in no time. So
this attitude has been, um, made the way
the government is trying to work now with this new
organization called 18 F Because it's the corner of 18
F Street. It's sort of an internal i t
procurement agency for the federal government so agencies can work
with brilliant I t. People who direct directly for
the federal government who are regarding it. Working for
18 F is a form of federal service like they're
not going into the military. They're working for 18
F. They're going to make half, maybe 10%
of what they could make in San Francisco. But
by God, they're going to serve their country and
their working 18 F for a couple years and show
that there are better ways to do it in this
country. And I hope that starts trickling down to
a state level at a local level. And I
sure hope that Virginia emulates this and creates an internal
I T agency because our current approach of outsourcing everything
to is it Northrop Grumman. Is that who has
the contract? It's a disaster. It's gone terribly
for everybody. And I think we need to yank
that contract and hire competent people. It's a dream
. I'll take one more question before I let you
. Poor people go, but I will hang out
and talk to anybody for as long as people want
to talk to me. Yes, scary. But
the question that I have is how do we continue
this conversation? I'm relatively new, higher things,
but I am alright. Salad. We need to
have it happen. My background, agriculture. And
so I want inter agency operations to the ailment back
two years or my farmer's got home. And so
I need to figure out a way to be able
to continue this conversation and to be take one take
home message, help me try to craft that.
Sure, So ag data is huge and it is
really huge. There's a I think it's only a
four or five year old company. Now that Monsanto
bought for a billion dollars last year, this company
combined open agriculture data soil, the aforementioned soil data
, soil data, uh, weather data and crop
data, all public data. They combine to this
in some very clever fashion fashions, rather, and
they now have an insurance. Is crop insurance.
Basically, they'll ensure your crops if you grow what
they say will grow, well, where to grow
them, and you plant when they say to plant
and harvest when they say to plan because their data
is so precise on a field by field basis,
they'll say in your Northfield, your alfalfa and you
plant it on the 18th. We're not gonna give
you crop insurance if you know anything of the alfalfa
, Go Planet 18th and the results are excellent crop
. The price of the crop insurance goes way down
because the results are much better. So Monsanto bought
them for a billion bucks. It's entirely open data
power company. It's been a huge success story,
so I am psyched about AG data and farmers.
For any of you, all without a background in
that, like they got GPS and their tractors,
now like they're planting, is incredibly precise. It
is. It is absolutely a science, uh,
and quite modern. They'll they'll even instant message in
the field to each other between the tractors and and
coordinate the iPads that might have mounted up in their
tractor. And it's a pretty cool system. So
my take home for you is this, he said
, trying to think of the take home, Uh
, I encourage you to think of one way in
which there is data that either doesn't exist as open
data, but probably exists somewhere that should exist,
whether it be on a federal or on a state
level and encourages it to think on those levels in
particular. If you can identify what that is and
come up with a use case here, our farmers
I've talked to you got a one pager. I
talked to this farmer who said this this farmer who
said this at Virginia Tech. We're prepared to do
this thing and then you approach both state and federal
act departments say, Listen, here's the thing that
would be possible if you would just published this data
, and often it's available, but it's available as
bad information or it's PDFs. You can't do anything
with that. Um, I think you probably got
a better reception on a state level. It's a
little hard to be heard on a federal level.
I don't know who? I don't think there's a
chief data officer within the Department of Agricultural federally.
Uh, my hope there's a c t o.
But often C T O s are too bogged down
with system administration to deal with this. Um,
I I don't have No, I have no useful
connection there. But if you can gin up something
like that, that one case that can start to
crack that not, uh, of getting the data
exposed. And from there, it's a little easier
to build out. You know, once I get
that business data, that's all right. You know
, for Virginia, we've got the business data released
. Let's get licensing data. Oh, get license
data. Great. Well, let's get records of
professional admonishments, and you can build out from there
. I think the same thing works just as well
in an egg or anywhere else. Well, I'm
going to hang out here as long as anybody wants
to chat. I'm really grateful to you all coming
out. I understand. Uh, it's not the
most exciting thing to do on a Friday night to
come out here talked about open data, but I
am very excited to spend Friday night uh, we're
talking about open data, and I appreciate having found
some people who are likewise engaged at the end.