Tag Archives: open data

The Rise of the Open City: the current state of affairs

I’ve been following with great interest the number of cities partaking in open data initiatives. With the online announcement yesterday of a motion going before Calgary’s City Council, things are again on the move. So what is the count at now? This little table tries to capture who’s done what so far. If I’m missing something please do let me know – I will try to update this from time to time.

City

Date of initial activity

Action

Note

Website

Washington, DC October 12th, 2008 Created a data portal on city website and launched apps for democracy Action was taken by the CIO, no city motion passed. Currently launching a second apps for democracy contest. http://data.octo.dc.gov/
Vancouver, BC May 21st, 2009 Vancouver City Council Passes the Open Motion Open Data website is in the works, release date unknown. N/A
San Francisco, CA June 16th, 2009 City of SF posts a craigslist request looking for developers to help create a data.gov like site for the city No motion passed, there is an OpenSF blog where current activities and ideas are shared. N/A
Nanaimo, BC June 22nd, 2009 City launches an open data website No motion passed http://www.nanaimo.ca/datafeeds/
New York City, NY June 25th, 2009 A bill is being circulated by Council Member Gale Brewer Has announced a “Big Apps” competition for apps that use 80 soon to be released city data sets. N/A
Calgary, AB July 27th, 2009 City of Calgary tables an Open Motion to be debated N/A N/A
Toronto, ON 2010 Announces (April 7th, 2009) intention of creating open data website Mayor David Miller announces Toronto will create an open data website by fall of 2009 at Mesh 09 conference N/A
Ottawa, ON I’ve heard there is movement in Ottawa, have not found any information

Open Data at the City of Vancouver – An Update 16/7/2009

matrix

For those interested in the progress around the Open Motion (or Open3 as city staff have started to call it) I’ve a little update.

Last week during a visit to city hall to talk about the motion I was shown a preview of the website the city has created to distribute and share data sets. For those unsure what such a website would look like, the baseline and example I’m measuring the city against is, of course, the Washington DC website. At the moment the city’s prospective website looks more like the (also impressive) beta site the City of Nanaimo set up after ChangeCamp – a little simpler and with a lot fewer data sets, but it is the first step.

As an aside kudos to the City of Nanaimo team which has been pushing open data and especially geo-data for quite some time as this must read Time magazine piece (yes, a must read Time magazine piece) will attest.

Anyway… back to Vancouver. The fact that the city has a beta website with a (very) modest amount of data ready to launch is a testament to the hard work and effort of the City’s IT staff. Rarely in my work with government’s have I seen something move so quickly and so needless to say… I’m quite excited. At the moment, I don’t know when the beta data site will go live – there are still a few remaining issues being worked on – but as soon as it launches I will be writing a blog post.

In the interim, big kudos should also go to the City’s Archives who posted a number of videos from the archives online and created it’s own YouTube Channel. They received so much traffic over the videos that the servers almost ground to a halt. Awesome, I say. It just goes to show how much interest there is out there.

Also exciting is that my post on how open data makes garbage collection sexy has inspired two local hackers (Luke and Kevin) to scrape the city’s garbage calendar and hand created digital versions of the city’s garbage maps to create the app I spec’ed out in the blog post. (I’ll have more on that, including links, in a few weeks) Luke also suggested I start recording other app ideas that come to me so over at the Vancouver Data Google group which was created on the fly by local coders in the audience during my and Andrea’s presentation at Open Web Vancouver. I’ve asked people to share their ideas for applications (mobile or desktop) that they’d like to see created with open data.

Sooooo… if there is an app you’d like to see created please post it to the google group or send me an email or write it in the comments below. No guarantees that it will be created but I’m hoping to help organize some hack-a-thons (or as my city friends prefer… code sprints). Having some ideas for people to sink their teeth into is always helpful.

How Open Data even makes Garbage collection sexier, easier and cheaper

So presently the City of Vancouver only shares its garbage schedule (which it divides into north and south) as a PDF file. This is a pity as it means that no one can build any apps around it. Imagine a website or Iphone app that mashed up google maps with a constantly up to date city garbage pick up schedule. With such a application one could:

  • Simply punch in your address and find out your garbage zone (no more guessing if you live on the border of a zone)
  • Get an email or SMS notification 15 minutes, 1 hour, 12 hours (whenever!) before pick up
  • Download the garbage schedule into your calendar using XML, HTML or ICAL

Garbage-App

Let’s face it, everyone could do with a garbage day reminder. I can imagine that there are 1000’s of Vancouverites who’d sign up for an email notification.

Maybe this seems all very simply, nice but unimportant. But this is more than just creating convenience. What are the implications for such an application?

For citizens:

  • Let’s face it, for many of us, it would be a convenient and nice to get a reminder of when the garbage is going to be picked up
  • It would improve citizen’s appreciation for city services
  • It might also increase the likelihood citizen will recycle as the notification would enable them to better plan and prepare for garbage pick up

For the city & taxpayers

  • Every day 100s of Vancouverites forget to put out there garbage out for pick up. If garbage isn’t picked up, it piles up. This creates several new risks and costs to the city including: increasing likelihood of rodents and health hazards and an increased risk that some residents will dispose of their garbage inappropriately/illegally
  • When garbage is not put out an expensive city asset (the garbage truck, fuel, garbage men) all pass by unused. This means that taxpayers money is not used as efficiently as it could.
  • Moreover,  when garbage isn’t put out there will be twice as much at the house the next week. Multiply that by 100’s of houses and the very quickly the city must deploy extra garbage trucks to deal with this unusually large volume of garbage – costing city taxpayers.

What would be the total value/savings of such an application? Hard to gauge. But add up, time saved by citizens better able to plan around garbage pick up, a small reduction in illegal garbage disposal, a small increase in recycling, slight increase in garbage pick up efficiency and I’m sure the total value would be in the low millions, and the direct savings for the city in the low $100,000 per year. That is not insignificant – especially if all the city had to do was free the data and allow an intrepid hacker to code up the website.

Of course it doesn’t end there. A reliable source tells me the city collects far more data about garbage collection. For example when the driver can’t pick up the garbage can, they make an entry on their device as to why (e.g., it is under a tree). This entry is sent via a wireless connection back to a City database, and includes the highly precise coordinates of the truck at that moment. Then when a resident calls in to find out why the crew did not pick up the garbage can from their residence, the operator can bring up the address on a map and pinpoint the problem.

Such data could also be on the website in an option with something like “Why didn’t my garbage get picked up?” By sharing this data the city could reduce its call volume (and thus costs). With open data, the possibilities, savings and convenience is endless.

Feeding the next economy – Give us a stimulus that stimulates, not placates

Last December – as the debates over the stimulus packages were just beginning, I wrote a piece on why the wrong stimulus today could fail us tomorrow. Well, today has become tomorrow, and we are failing.

A stimulus package should be an investment. It should create new industries and markets, it should find help create efficiencies and improve productivity, in short, in should help the economy grow in a sustainable manner. In the last depression the government accomplished this by funding infrastructure necessary for the 20th century economy, things like roads and highways for cars and transportation, power stations and grids for cities and industry, university buildings for education. Today, we already have much of that infrastructure and – while some of it needs to be renewed – we need to be focusing on what infrastructure is needed for the next economy – the digital economy – that will carry us out of this recession.

So what powers the digital economy? It isn’t coal, steel or cars and power (although these things are necessary), it’s data and connectivity.

Data is the plankton of the new economy. It seems plentiful, tiny and insignificant. But a whole ecosystem of companies, large and small are emerging to feed off of it and support our next economy. People often fail to recognize that the largest company already created by the new economy – Google – is a data company.  Google is effective, rich and powerful not because it sells ads… but because it generates petaflops of data everyday from billions of search queries. This allows it to know more about our society, and sometimes us individually – the merchandise we like, the services we want, the spam we’ll receive, even if the likelihood we’ll get sick in 4 months – than we know about yourself. Give people access to data and they will use it to become more efficient (freeing up more money to reinvest) and to create new services and opportunities (creating new jobs and profits).

Look no further than the City of Washington DC. It created a publicly available database of city collected and created data and asked local individuals and companies to use it. The result? A $50,000 dollar investment in changing processes and offering prize money has so far yielded $2.3M in value. That’s a 46 times return on investment in one year.

Now imagine that at a national level. Imagine Statistics Canada making all its data available freely (since taxpayers have already paid for its creation). As I outlined in a talk to StatsCan last year, not only would this make them a more important ministry, it could foster billions in savings, investments and new jobs all for a tiny sum. Let’s pick a truly excessive number, say $100M (.2% of the stimulus package) and imagine that’s this would be the cost for Statistics Canada to free all its data and provide in formats usable for webpages, cellphones and applications. Even if such a stimulus were only 20% as effective as Washington’s open data project it would still yield $460M in one year of new (not saved) jobs, and improved economic efficiencies and competitiveness. Over a decade, billions in new wealth would be created.

Compare this to our current course of action. To date much of our stimulus has been spent propping up (not creating) industries that are in death spirals such as logging, newspapers, and the auto-sector. The money spent isn’t about creating new or better jobs, it is simply being spent to keep jobs. Indeed, Andrew Coyne calculates that each auto-job saved cost us just under 2 million dollars. It will take years, if not decades, for such an investment to pay off, if it ever does. Worse still, the Canadian economy will be no more efficient or profitable as a result. While we are at it we might as well be giving Canadians money to buy land-line telephones to stimulate the telecommunications sector.

Oh, and it case you are wondering, the Americans already give out much of their government data for free and they are starting to give away more and more. This is a competitive race we are already losing, and only falling further and further behind.

Canada needs a better stimulus, one that is low on carbon and fat on data. Sadly, I fear our current government lacks the vision and creativity to give us what we need to prepare for the 21st century. So far we are off to an ominous start.

Open data in local education: broader lessons for government, citizens and NGOs

Last months I remember reading a couple of news stories about a provincial government ministry in Canada that was forced to become less transparent.

Forced?

Yes, this was not a voluntary move. A specific group of people pressured the government, wanting it to remove data it had made public as well as make it harder for the public to repurpose and make use of the data. So what happened? And what lessons should governments, NGOs and citizens take away from this incident?

The story revolves around the Ontario Ministry of Education which earlier this year created a website that mashed up performance data (e.g. literacy and math scores) with demographic information (e.g. percentage of pupils from low-income households and percentage of gifted students). The real problem – according to a group representing teachers, parents and stakeholders – occurred when the Ministry enabled a feature that allowed the website’s users to compare up schools to one another.

The group, called People for Education, protested that the government was encouraging a “shopping-mentality” in the public school system.

Of course, many parents already shop for schools. I remember, as a kid, hearing about how houses on one side a street, but within the catchment area of my high school, were more expensive than houses on the other side of the same street, but within the catchment area of another school. Presently however, this type of shopping is reserved for the wealthy and connected (e.g. the privileged). Preventing people from comparing schools online won’t eliminate or even discourage this activity, it will simply preference those who are able to do it, further reinforcing inequity.

The real problem however, is that the skills and analysis involved in school shopping are the same as those required in accessing and being engaged in, the performance of one’s local school. Parents, and taxpayers in general, have a right to know their childrens school’s performance – especially in comparison to similar schools. If parents don’t have information to analyze and compare, how can they know what systemic issues they should ask their childrens teachers about? More importantly, how can they know what issues to press their local school board about?

Ironically, People for Education states on its “About Us” page that it works towards a vision of a strong public education system by a) doing research; b) providing clear, accessible information to the public and c) engaging people to become actively involved in education issues in their own community.

And yet, asking the government to remove the comparison feature runs counter to all three of its activities. Limiting how the Ministry’s data can be used (and as we’ll see later, suggesting that this data shouldn’t be shared):

  • prevents parents, and other analysts such as professors or politicians, from doing their own research
  • runs counter to the goal of providing clear and accessible information to the public. Indeed, it makes information harder to access.
  • makes it harder for parents to know how they should get involved and what issues they should champion to improve their local school

What is interesting about this story is that it reveals the core values and underlying motivation of different actors. In this case People for Education – which I believe to be a well intentioned a positive contributor to the issue of education – is nonetheless revealed to have a conservative side to it.

It fears a world where citizens and parents are equipped with information and knowledge about schools. On the one hand it may fear the types of behaviours this could foster (such as school shopping). However, it may also fear a weakening of its monopoly as “expert” and advocate on educational issues. If parents can look at the data directly, and form their own analysis and conclusions, they may find that they don’t agree with People for Education. Open data would allow those it represents to self-organize, challenging the hierarchy and authority of the organization.

For whatever reason, we see an NGO bending over backwards to advocate for an outcome that runs directly counter to the very vision and activities it was founded to serve. More ironically, this result in some paradoxical messaging as an organization that champions Ontario’s school system essentially arguing that it doesn’t trust the products of that system – the citizens of Ontario – to use the information and tools provided by the Ministry that was responsible for their education. It is an unsustainable position – particularly for a group that was originally founded as a bottom up, grass-roots organization.

So what lessons are there here?

For government:

A key mistake made by the Ontario Ministry of Education is that it didn’t open up the data enough. While the website allowed users to look at school performance data they could only do this on the Ministry’s website using the Ministry’s tools and interface. Had the data been available as an API or in downloadable format someone else could have taken the data and created the system for comparing schools. People for Education were mostly upset that the Ministry’s website encouraged a “shopping-mentality.” Had the Ministry simply shared the data then People for Education could build their own interface using criteria and tools they though relevant. The Fraser Institute or a multitude of other organizations could build their own as well, and people could have used the tools and websites they found most useful and relevant. Let People for Education go head to head with the Fraser Institute and whoever else. This is not a battle the government need fight.

Lesson: Always provide the data – a goal that is hard to argue against – but sometimes, leave it to others to conduct the analysis. A marketplace of ideas will emerge, and citizens can choose what works best for them.

For NGOs in general

First, understand what open data means for your cause. One of the news articles had this highly disturbing quote from the Executive Director of People for Education:

Among her complaints about the type of information available, Ms. Kidder took issue with the ministry’s contention the Web site merely consolidated information already available to the public. “You can’t walk into your child’s school and say ‘What’s the average income of parents at this school?’ ” she said. “It’s not true at all [that this is public information].”

This is a shocking statement. In actuality, all the data assembled by the Ministry is publically available. It was just that, until now, it had remained scattered and isolated. Just because it was hard to find (and thus reserved for an elite few) or located on the school property (and thus easy for parents to locate) does not mean it didn’t exist or wasn’t public.

Lesson: Transparency is the new objectivity. People increasingly don’t trust anyone – governments, the media, or even NGOs. They want to see the analysis themselves, not take your word for it. Be prepared for this world.

Second, be careful about taking positions that will deny your supporters – and those you represent – tools with which to educate themselves. Organizations that are perceived as trying to constrain the flow of information so as to retain influence and control risk imploding. I won’t repeat this lesson in detail but Clay Shirky’s case study about the Vatican, written up in Here Comes Everybody, is a powerful example.

For educators in particular

In the past, educators have been deeply concerned with ranking systems. This is understandable. Ranking systems are often a blunt tool. Comparing apples to oranges can be foolish – but then, sometimes it is helpful. The question is to know when it is helpful and ensure it is used accordingly.

The fact is, ranking is an outcome of data. The two simply cannot be separated. The moment there is data, there is ranking. A ranking by school size, number of teachers, or amount of gym equipment is not different than a ranking of class sizes, literacy rates, disciplinary trends, or graduation rates. What matters is not the rank, but the conclusions, meaning and significance we apply to these rankings. Here, the role for groups like People for Education could be profound.

This is because we can’t be in favour of transparency and accessible information on the one hand and against ranking on the other. The two come hand in hand. What we can be opposed to are poor ranking systems.

Every profession gets assessed, and teaching should be no different. The challenge is that there is much more to teaching than what gets reflected in the data collected. This means that groups like People for Education shouldn’t be against transparency and open data – they should be trying to complexify and nuance the discussion. Once the data is publicly available anyone can create a ranking system of their own choosing – but this gives us an opportunity to have a public discussion about it. One way to do this is to create one’s own tools for measuring schools. You don’t like the Ministry of Education’s system? Create your own. Use it to talk to parents about the right questions to ask and to promote the qualitative ways to evaluate their childrens’ schools performance. It’s an open world. But that doesn’t mean it needs to be feared – it is rife with opportunity.

Open Cities: Popularity lessons for municipal politicians

Last Thursday I posted the Vancouver City motion that is being introduced today.

Prior to the posting the motion several of my friends wondered if the subject of open data, open cities and open source were niche issues, ones that wouldn’t attract the attention or care of the media, not to mention citizens. I’m not sure that this is, as of yet, a mainstream issue, but there is clear, vocal, engaged and growing constituency – that is surprisingly broad – supporting it.

For politicians (who are often looking for media attention), open-advocates (who are looking for ways to get politicians attention) and others (who usually care about and want access to, some of the data the city collects) there are real wins to be had by putting forward a motion such as this.

To begin with, let’s look at the media and broader attention this motion has garnered to date:

First, a search of twitter for the terms Vancouver and Open shows hundreds upon hundreds of tweet from around the world celebrating the proposal. Over the weekends, I tried to track down the various tweets relating to the motion and they number at least 500, and possibly even exceed 1000. What is most interesting is that some tweets included people saying – “that they wished they lived in Vancouver.”

As an aside, have no doubt, City Hall sees this initiative in part as an effort to attract and retain talent. Paul Graham, who created a multimillion dollar software company and is now a venture capitalist summed it up best “Great [programmers] also generally insist on using open source software. Not just because it’s better, but because it gives them more control… This is part of what makes them good: when something’s broken, they need to fix it. You want them to feel this way about the software they’re writing for you.” Vancouver is not broken – but it could always be improved, and  twitter confirms a suspicion I have: that programmers and creative workers in all industries are attracted to places that are open because it allows them to participate in improving where they live. Having a city that is attractive to great software programmers is a strategic imperative for Vancouver. Where there are great software programmers there will be big software companies and start ups.

Blogs, of course, have also started to get active. As sampling includes locals in the tech sector, such as David Asher of Mozilla, Duane Nickull of Adobe, as well as others interested in geographic data. Academics/public thinkers also took note on their blogs.

Then, the online tech magazines began to write about it too. ReadWriteWeb wrote this piece, ZDnet had this piece and my original blog post went to orange on Slashdot (a popular tech news aggregator).

Of course, traditional media was in the mix too. The Straight’s tech blog was onto the story very early with this piece, a national newspaper, the Globe and Mail, had this piece by Frances Bula (which has an unfortunate sensationalist title which has nothing to do with the content) and finally, today, the Vancouver Sun published this piece.

Still more interesting will be to see the number of supportive letters/emails and the diversity of their sources. I’ve already heard supportive letters coming from local technology companies, large international tech companies, local gardening groups and a real estate consulting firm. Each of these diverse actors sees ways to use city data to help lower the costs of their business, conduct better analysis or facilitate their charitable work.

In short, issues surrounding the open city – open data, open source software and open standards – are less and less restricted to the domain of a few technology enthusiasts. There is a growing and increasingly vocal constituency in support.

Update May 20th, 2009 more media links:

The Libertarian Western Standard wrote this positive piece (apparently this is the Vancouver City only good initiative).

I did a CBC radio interview on the afternoon of May 19th during the show On the Coast with Stephen Quinn. The CBC also published this piece on its news site.

From here to open – How the City of Toronto began Opening up

Toronto the open

For myself, the biggest buzz at ChangeCamp Toronto was that the city showed up with lots of IT staff (much of it quite senior) who were trying to better understand how they could enable others to use their data and help citizens identify and solve problems. In fact the City of Toronto ran what I believe will be seen later as the most enduring sessions in which they asked what data should they start making available immediately (as APIs).

For those not in the know, think of an API as a plug that rather than delivering electricity instead delivers access to a database.

The exciting outcome is that web designers, coders and companies can then use this data to better deliver services, coordinate activities in neighborhoods, make government more transparent, or analyze problems. For example, imagine if all the information regarding restaurants health violations were not hidden deep within a government website (in a PDF format that is not easily searchable by google) but were available on every restaurant review website? Or if road closures were available in a data stream so a google maps application could show which road were closed on any given day – and email you if they were in your neighborhood.

This is the future that cities like Toronto are moving towards. But why Toronto? How did it arrive at this place? How is it that the City of Toronto sent staff to ChangeCamp Toronto?

The emergence of open in Toronto

I’ve tried to map this evolution. I may have missed steps and encourage people to email me or post comments if I have.

evolution of open data TO

The first step was taken when people like David Crow created a forum – Barcamp – around which some of Toronto’s vibrant tech and social tech community began to organize itself. This not only brought the community together but it also enabled unconferences to gain traction as a fun and effective approach to addressing an issue.

Then, in late 2006 the Toronto Transit Commission (TTC) issued an Request For Proposals (RFP) for a redesign of its website. Many in the tech community – who had no interest in doing the redesign – were horrified at the RFP. It was obvious that given the specifications the new website would not achieve its potential. A community self-organized around redesigning the RFP. Others took note and, because they cared about the TTC, wanted to also talk about simple non-website changes the TTC could make to improve services. TransitCamp was this born and – with enormous trepidation, some TTC officials showed up (all of whom should be loudly applauded). The result? The tech and social tech community in Toronto was engaged in civic matters and their activities were beginning to make it onto the city government’s radar.

Other Camps carried on through 2007 and 2008 (think OpenCities), building momentum in the city. Then, in November of 2008 – a breakthrough. The City of Toronto hosted an internal Web 2.0 conference and invited Mark Surman – executive director of the Mozilla Foundation and long time participant in the Toronto social tech space – to deliver the keynote entitled “A City that Thinks like the Web“. After the talk, the Mayor of Toronto stood up and said:

” … I’ve been emailing people about your challenges. Open data for Google Transit is coming by next June, and I don’t see what we shouldn’t open source the software Toronto creates.” He also said “I promise the City will listen” if Torontonians set up a site like FixMyStreet.com

You can hear the Mayor Miller’s full response here:

In short, the Mayor promised to begin talking about opening up (and open sourcing) the city. Freeing up Ryan Merkley and the City of Toronto IT team to attend ChangeCamp

Lessons for ChangeCamp Vancouver

It remains unclear to me whether ChangeCamp is the right venue for tackling this opportunity in Vancouver.

We in Vancouver are not as far along the arc as Toronto is. We do, however, have some advantages. The map is more obvious to us and some of us have good relationships with key staff in the city. However, this process takes time. To replicate the success in Toronto, governments here on the west coast need not only be at ChangeCamp, they need to be running sessions and deeply engaged. For this to occur cultures need to be shifted, new ideas need to percolate within government institutions and agencies and relationships need to be built. All this will take time.