Tag Archives: opendata

Canada’s Opaque Transparency – An Open Data Failure

Yesterday, at the Prospectors and Developers Association of Canada’s (PDAC) Canada Minister of Natural Resource, Joe Oliver, announced with great fanfare a new initiative to compel mining companies to disclose payments of over $100,000′s to foreign and domestic governments.

On the surface this looks like a win for transparency, particularly for a sector that is of great importance to Canada: mining.

And this issue matters since not only do extractive industries represent an important part of Canada’s economy, but the sector has been dogged with controversy. Indeed the Toronto Star just uncovered today a report commissioned (and buried) by the Prospectors and Developers Association of Canada (PDAC) that showed Canadian mining companies have the worst record when it comes to environmental standard and human rights.

Forcing mining companies to account for their payments to foreign and domestic governments won’t solve every problem, but it can help curb corruption. Indeed the issue was seen as so important that at the last G8 summit, the leaders agreed that companies should be compelled to disclose these payments.

Happily, there is a legitimate global movement to make government payments by extractive industry companies more transparent. It is called the Extractive Industries Transparency Iniative (EITI). It has set a series of standards for disclosing such payments so that they are easier to track across borders. In fact EITI is seen as so important it is actually the only organization mentioned by name in the last G8 summit communique. This is the same EITI program about which last year the Minister’s press secretary boasted:

Since 2007 Canada has also been a supporting country of the Extractive Industries Transparency Initiative, and is now the second largest financial donor to the initiative, providing $12.65 million to the World Bank’s Extractive Industries Transparency Initiative Multi-donor Trust Fund…

Which brings us to Minister Oliver’s important announcement.

Did the government announce that it was joining 42 other countries, including its G8 partners the United States, the United Kingdom, France, Germany and Italy to join the standard it has been a major funder of?

No. It did not.

Apparently EITI is good enough to fund so that others can implement it. When it comes to actually doing what is effective… the government balked. Canada, apparently, is going to adhere to its own “unique” approach.

And it gets worse.

Read the Minister’s statement more closely, particularly this line:

“We want to make it as easy as possible, so we will not create a central database. Instead, we would require that reports be posted to company websites, with the government and public notified.”

So unlike EITI, which offers a centralized repository where records can quickly be downloaded and compared, Canada’s “compliance” will involve each company to maintain their own records “somewhere” and will require anyone interested if actually figuring out what is going on to go and track down each one individually.

We call this secrecy by obscurity. It makes a mockery of the notion of transparency.

We have a global infrastructure designed to make disclosure cheap, easy and effective. Infrastructure our own government has poured $12.7M into it. And we turn around and ignore it all.

Canada claims it wants to be a leader in open data. But if it can’t even get something basic like this right… such claims sounds increasingly silly here at home, among our G8 partners and, well, among the rest of the world.

Addendum: It gets worse still. Few people have noticed yet, but Canada recently (and quietly) stopped reporting the names of corporate directors in the public database of the country’s firms. This is a major step backwards and makes those who benefit from one of the most important benefits society can confer – limited liability – invisible to the public who confers that right. This is a major step backwards. Read this wonderful Economist article on why. More on this to come.

Great Hacks from the Open Data in Vancouver

Last weekend I helped host an Open Data Day in Vancouver. With the generous support of Domain7, who gave us a place to host talks and hack, over 30 Vancouverites braved the sleet and snow to spend the day sharing ideas and working on projects.

We had opening comments from Andy Yan – whose may be the most prolific user of Open Data in Vancouver, possibly Canada. I encourage you to check out his work here. We were also incredibly lucky to have Jeni Tennison – the Technical Director of the Open Data Institute – onsite to talk to participants about the ODI.

After the opening talks, people simply shared what they hoped to work on and people just found projects to contribute to. Minimal organization was involved… and here a taste of the awesome projects that got worked on! Lots of ideas here for other communities.

1. Open Data Licenses Resource: JSON + search + compatibility check = Awesome.

Kent Mewhort, who recently moved to Vancouver from Ottawa (via the Congo) updated his ongoing CLIPol project by adding some of the recently published licenses. If you’ve not seen CLIPol it is… awesome. It allows you to easily understand and compare the restrictions and rights of many open government licenses.

CLIPol Data

Better still CLIPol also lets you to see how compatible a license is (see example here). Possibly the best tool of all is one that allows you to determine what license you can apply to your re-mixed work in a way that is compliant with the original licenses (check out that tool here - screenshot below).

CLIPol compatibility

CLIPol is just such a fantastic tool – can’t recommend it enough and encourage people to add more licenses to it.

2. Vancouver in MineCraft

I have previously written about how Minecraft is being used to help in public consultations and urban planning - I love how the game becomes a simple tool that enables anyone to shape the environment.

So I was crazy excited I heard that Ryan Smith (aka Goldfish) had used the City of Vancouver’s open elevation data to recreate much of the city in Minecraft.

Below is a photo of Ryan presenting at the end of the day. The projection behind him shows Stanley park, near Siwash Rock. The flat feature at the bottom is the sea wall. Indeed Ryan notes that the sea wall makes for one of the clearest features since it creates almost perfectly flat structure along the city’s coast.

Mincraft Data

3. Vancouver’s Capital Budget Visualized in Where Does my Money Go

It is hard to imagine a project going better. I’m going to do a separate blog post on it.

This is a project I’ve always wanted to do – create a bubble tree visualization with Where Does my Money Go. Fortunately two developers – Alexandre Dufournet and Luc Lussier – who had never hacked on open data jumped on the idea. With help from City of Vancouver’s staff who were on site, I found a PDF of the capital budget which we then scraped.


The site is not actually live, but for developers who are interested in seeing this work (hint, hint City of Vancouver staff) you can grab their code from github here.

4. Monitoring Vancouver’s Bike Accident Data – Year 3

Eric Promislow has been coming to Open Data Hack-a-thons ever since Luke Closs and I started organizing them in 2009. During the first Open Data Day in 2011 you can read in my wrap up post about a bike accident monitoring website Eric created that day which Eric would eventual name Bent Frame. Well, Bent Frame has been live ever since and getting bigger. (Eric blogs about it here)

Each open data day, Eric updates Bent Frame with new data from ICBC – the province’s insurance monopoly. With over 6 years of data now in Eric is starting to be able to analyze trends – particularly around the decline of bike accidents along many roads with bike lanes, and an increase in accidents where the bike lanes end.

Bike Data

I initially had conversations with ICBC to persuade them to share their data with Eric and they’ve been in touch with him ever since, passing along the data on a regular basis. It is a real example of how an active citizen can change an organization’s policies around sharing important data that can help inform public policy debates.

5. ProactiveDisclosure.ca – Making government information easier to search

Kevin McArthur is the kind of security guy most governments dreads having around but should actually love (example his recent post on e-voting).  He continued to hack on one of his side projects: proactivedisclosure.ca. The site is a sort of front end for open data sets, making it easier to do searches based on people or companies. Thus, want to find all the open data about a specific minister… proactive disclosure organizes it for you.

Proactive Data

Kevin and a small team of players uploaded more data into their site and allowed it to consume unstructured data. Very cool stuff.

6. Better Open Data Search

Herb Lainchbury – another fantastic open data advocate – worked on a project in which he tried to rethink what an open data search engine would look like. This is a topic that I think matters A LOT. There is simply not a lot of good ways to find data that you are interested in.

Herb’s awesome insight was invert the traditional way of thinking about data search. He created a search engine that didn’t search for the data set keywords or titles, but rather searched the meta data exclusively.

One interesting side outcome of this approach is that it made related data sets easier and, made locating identical data sets but from different years a snap. As Herb notes the meta data becomes a sort of “finger print” that makes it easy to see when it has been duplicated. (Quick aside rant: I loath it when governments releases 20 data files of the same data set – say crime data – with each file representing a different year and then claiming that it is 20 unique data sets in their catalogue. No. It is one data set. You just have 20 years of it. Sigh).

7. School Performance Chart

Two local video game programers – Louie Dinh and Raymond Huang – with no experience in open data looked around the BC Government Open Data catalogue and noticed the data on test scores. Since they attended school here in British Columbia they thought it might be interesting to chart the test scores to see how their own schools had preformed over time.

They were able to set up a site which graphed how a number of elementary schools had performed over time by looking at the standardized test scores.

Test SCore Data

This is just a great example of data as a gateway to learning. Here a simple hackathon project become a bridge for two citizens to dive into a area of public policy and learn more about it. No one is claiming that there chart is definitive, rather it is the start of a learning process around what matters and what doesn’t and what can be measured and what can’t in education.

Congratulations to everyone who participated in the day – thank you for making it such an amazing success!

Open Data Day 2014 – Five Fun Events Around the World

With over 110 Events happening world wide it is impossible to talk about every Open Data Day event. But looking almost every event on the wiki I’ve been deeply moved and inspired by the various efforts, goals and aspirations of the people who have organized these events.

In order to help others understand why Open Data Day matters as well as what can happen on it, here are five open data day events that I’m stumbled across that are doing something particularly fun or interesting.

1. Capetown & Johannesburg, South Africa

Their Description:

Coders, data wranglers and data investigators will pair up to look at one of three openly available datasets we have on hand, and work out the most interesting questions they can ask of it in less than half an hour. After 30 minutes, it’s all change – another desk, another partnership, another exciting data set to turn into a story.

What we’re hoping is that you’ll learn tips and tricks for getting data, querying it, creating quick visualizations and turning it to stories that people want to know about. You’ll learn from different people with a variety of skills, hopefully that you wouldn’t normally work with. And we’re also hoping it will be four hours of fun.

Why I love it: I love the focus on learning. With the participation of hacks and hackers the goal is clearly to help journalists and citizens learn new skills, not so they can do something with the data sets available on open data day, but so they can better play with data sets in the future to pursue stories or help a community. The point of speed data dating is thus not to build a product, the product is the skills and networks developed and, with luck the future stories and analyses that will be told by those who participated.

2. Buenos Aires, Argentina

Their description:

On February 22nd we will go out to the street and play with local data and some street artists to create beautiful visualizations.

Why I love it: Street art open data? What a great way to try to raise awareness of the importance of data literacy and transparency. In addition, how awesome is it to move outside the digital realm and use data to create artifacts that are not necessarily digital. And if there are artists involved? Jer would be so happy to read about this.

3. Greenfield, MA, United States

Their description:

We’re convening a small group to work with the Franklin Regional Council of Governments on a user-friendly way to map private wells in Western Massachusetts…

…Why is it important to map wells?

  • Only about 5% of private wells in Massachusetts are geolocated.
  • Many towns in Western Mass rely 100% on private wells.

Not knowing where our wells are can (and does) lead to water contaminated by nearby septic systems, dumping, and pollutant storage. Aside from the obvious health concerns, there are also financial implications from remediation costs and lowered property values.

Why I love it: Wow, WOW, WOW!!! This is maybe one of the coolest open data day events I’ve ever seen. Here you have a small community focusing on a problem that is real and tangible to them. Moreover, open data could have a direct and meaningful impact on the issue. I love the focus. I love that rallying point. I love the high impact with low resources (their building has minimal heat – so they are advising people to layer up). I wish these crew all the best success and hope to see an update.

4. Nagoya, Japan

Their Description:

Now, highlight of this year is “data of Nagoya Castle!”

The nearly 300 maps and survey drawings of the Nagoya Castle will be made open prior to Open Data Day. The Nagoya Castle office is cooperating with us and has decided that we can use their data for “International Open Data Day.” The references to the image are here.

Why I love it: Well – fill disclosure, my understanding of this event is through the prism of Google translate. But if I understood correctly… there are a few open data events in Japan that have a strong focus on local history which I find totally fascinating. At this event in Nagoya they are bringing in a professor who is an expert in open data as well as expert in the Nagoya castle to talk about the data that is being made open. In addition they are organizing and actual physical tour of the caste. Open Data meets local history buffs!

5. Cairo, Egypt

Their Description:

We will be organizing an online and decentralized event in Cairo, Egypt for the Open Data Day. There are numerous suggested tracks depending on the participants set of expertise:

For translators (المترجمون)

  • Open Data Handbook: The handbook discusses the legal, social and technical aspects of open data. It can be used by anyone but is especially designed for those seeking to open up data. It discusses the why, what and how of open data – why to go open, what open is, and the how to ‘open’ data. Translate it into Arabic here
  • Translate any of the School of Data short tutorials, for example, What is Data?Telling a Story with DataFinding Data, or Any other course/modules

For bloggers (المدونون)

Write blog posts about Open Data related topics and case studies, and don’t forget to use the following hashtag, #ODD2014. Possible ideas for blog posts:

  • Write about the concept of Frictionless Data
  • Case studies how you searched for, extracted and used governmental data
  • Listing of local organizations working or promoting Open Data or advocating for more Transparent and Open Governments

For Developers (مطوري البرامج)

  • Scrape data from capmas and put it into Open Format
  • Scrape data (Budget or the The Financial Monthly Bulletin) from the Minister of Finance and upload it to OpenSpending.org,
  • Create a tool to scrap the traffic data from bey2ollak and put it in an open format.

For Data Wranglers (هواة جمع البيانات)

Why I love it: I love that there are calls to action for a variety of people – including those who have no coding skills at all. How genius is it to organize an event to localize/translate the Open Data handbook? This is something a large number of people could do – and better still can help make open data accessible to a still larger pool of people.

And for the other roles the suggestion of projects – particular with a focus on the national budget and government operations data (capmas) suggests there is a strong civil society presence within the open data community. Will be super interested to see what progress they make and if there is broader interest in their work.

Open Data Day in Vancouver – Registration is live

I’m happy to share that this February 22nd there will be an Open Data Day event in Vancouver. The details are as follows:

The event will take place at Domain7 – who have kindly volunteered their space – on Saturday, February 22, from 10:00 AM to 4:30 PM. We have limited space and want to coordinate lunch so if you want to attend please register here.

We have an exciting day planned – a few opening talks and networking and then a day of hacking, learning and sharing. More details to follow shortly! Very much hope to see you there!

The dangerous mystique of the “open data” business

I’m frequently asked by people about how they can start an “open data business.” Let me first say that I love that the question gets asked. I love that people are interested in Open Data. I love that people want to learn more, they want to play, they want to think of ways of creating a company. These are, in part, signs of how far the open data discussion has come – people see it as a resource that they would like to leverage.

It is, also, the wrong question.

This is not to say there are not businesses that use open data. Indeed, a vast number of companies use open data (anyone company using census data for even a tiny part of their business qualifies). Nor am I denying there aren’t businesses built primarily with open data – the Open Data 500 list demonstrates there are. Plus I get introduced almost daily to businesses that are: both Ajah and OpenCorporates come to mind (I have donated advice, but have no financial connection, with either).

The Trap

But from a founder (or, I suppose, investor) perspective there are dangers to thinking about “open data” as a unique business space.

The danger is in failing to understand there is virtually nothing that distinguishes an open data business from any other business. Any business needs to solve a real (or sadly, at times imagined) problem, it needs to find clients (e.g. people willing to pay for that solution), and it needs to execute on a number of other things at least competently (HR, marketing, management, cashflow, etc…).

The danger with putting the words “open data” before the word “business” is that it risks making people think Open Data businesses are somehow unique. They are not. If there is a gapping chasm between the question of “what can I do with software” and “how can I create a viable software company” there is an equally large gap between “what can I do with open data” and “how can I create a viable company using open data.” And the questions you need to ask yourself to figure out that latter question (many of which are nicely laid out in this book) are independent of whether it is a software, hardware, crafts or open data business.

Indeed open source software space gives us a nice analogy. I suspect few people decide to create an open source software company – they decide to create a company and the software license is a reflection of their strategic options. I think it is the same with open data. You don’t start a company saying “let’s use open data.” You start a company to solve a problem, of which using or publishing open data may be the only, or the most strategic, way of doing this.

The Opportunity 

Some readers may be surprised to see me write this. I am, and continue to be an advocate of open data. But open data is not some magic pixie dust that causes normal business logic to disappear. And it is not that I think people are saying that per se, it is just that I want them to understand that the 99% of the problems that needs to be solved in an “open data business” lie in the third word of that string, and that while the first two do confer some unique advantages and disadvantages, these are relatively trivial.

The real opportunity of open data lies not in the way it creates a new unique type of business, but that it offers a new set of cheap building blocks by which to try to solve problems. In other words it increases the diversity and, lowers the cost of, inputs.

Here again the world of software is instructive. The Economist’s recent survey on Tech Start Ups talks of a Cambrian Explosion because of the availability of “Cheap and ubiquitous building blocks for digital products…” many of which are (and many of which are not) open source. The cheap availability of these building blocks is allowing for a range of experimentation that was previously not possible, or at least, prohibitively expensive.

Open data – whether as an input for software products and services, for analysis (journalistic or corporate) or for scientific research – is cheap (in theory free) and increasingly plentiful. It has the possibility of thus being the equivalent of the cheap code that is powering a great deal of experimentation in the world of software. As an open data advocate the possibility of this increased experimentation has me excited.


So if you are thinking about starting an open data business – that is great! I’m excited to hear that and I am keen to help and be supportive. But focus on that third word – business. That’s the one that really matters.

From a data perspective you should be asking yourself – what real tangible pain does doing something with this data set help me solve that was previously only possible with a more expensive input (e.g. proprietary data) or not possible at all. The second is to think of the impact of using open data on your strategy. Where does it leave you more vulnerable (too copy cats or the whims of the data publisher) and where does it leave you stronger (if the data is commoditized then the axis of competition will lie in other parts of the business).

I hope this is a helpful nuance to the issue of open data businesses, and some helpful input for those looking at open data and thinking about to find business opportunities in it.

Open Data Day Google+ Hang Out

With just about a month to go until Open Data Day things are going well. There are quite a few cities that have been added to the open data day wiki.

This year we thought we would try something new. On January 21st we are going to host a  Get Ready For Open Data Day 2014! Google hangout.

The goal of the hangout is to help people thinking about organizing an event in their city get a sense of what others doing, ask questions about what has worked in the past, and just learn more about what is possible on the day. 

We’ll be hosting a 30-60 minute event on Tuesday, January 21 (at 11:00 am EST/ 8:00 PST/ 16:00 GMT /17:00 CEST) with myself, Heather Leson and Beatrice Martini focused on:

  1. What is Open Day Day – History
  2. Planning tips
  3. Open Q&A

There is likely a limit to how many people we can host on the hangout so please let us know if you’d like to participate.

And if you are interested in connecting with others – especially those who have run open data day events before – please consider joining the mailing list!

Open Data for Development Challenge on Jan 27-28

This just came across my email via Michael Roberts who has been doing great work in this space.

Mail Attachment

Open Data for Development Challenge
January 27–28, 2014 — Montreal, Canada

Do you want to share your creative ideas and cutting-edge expertise, and make a difference in the world?
Do you want to help Canadians and the world understand how development aid is spent and what its impact is?
Do you want to be challenged and have fun at the same time?

If so, take the Open Data for Development Challenge!

This unique 36-hour ”codathon” organized by Foreign Affairs, Trade and Development Canada will bring together Canadian and international technical experts and policy makers to generate new tools and ideas in the fields of open data and aid transparency and contribute to innovative solutions to the world’s pressing development challenges.

The event will feature keynote speakers Aleem Walji, Director of the World Bank’s Innovation Labs, and Mark Surman, Executive Director of the Mozilla Foundation. It will have two related dimensions:

  • Technical challenges that involve building applications to make existing open aid and development-related data more useful. Proposed topics include building a data viewer compatible with multilingual data, creating a publishing tool suitable for use by mid-sized Canadian non-profit organizations, developing and testing applications for open contracting, and taking a deep dive into the procurement data of the World Bank Group. There is room for challenges proposed by the community. Proposals should be submitted through the event website no later than January 8th. Challenges will be published prior to the event, along with key datasets and other related information, to enable participants to prepare for the event.
  • Policy discussions on how open data and open government can enable development results. This would include the use of big data in development programming, the innovative ways in which data can be mapped and visualized for development, and the impact of open data on developing countries.

The international aid transparency community will be encouraged to take promising tools and ideas from the event forward for further research and development.

An overview of the draft program is attached. The event will be in English and French, with interpretation provided in the plenary sessions and panel discussions.

We invite you to register, at no cost, at this website as soon as possible and no later than January 10. A message confirming your registration and providing additional information about the venue and accommodation will be sent to confirmed participants. Please wait for this confirmation before making any travel arrangements. Participants are asked to make their own accommodation arrangements. A limited number of guest rooms will be available to event participants at a preferential rate.

To find out more about the Open Data for Development Challenge, please go to DFATD’s website.

Open Data Day 2014 is Coming Feb 22 – Time to Join the Fun!

So, with much help from various community members (who reminded me that we need to get this rolling – looking at you Heather Leson), I pleased to say we are starting to gear up for Open Data Day 2014 on February 22nd, 2014.

From its humble beginnings of a conversation between a few friends who were interested in promoting and playing with open data, last year Open Data Day had locally organized events take place in over 100 cities around the world. Check out this video of open data day in Kathmandu last year.

Why makes Open Data Day work? Mostly you. It is a global excuse for people in communities like yours to come together and organize an event that meets their needs. Whether that is a hackathon, a showcase and fair, lectures, workshops for local NGOs and businesses, training on data, or meetings with local politicians – people are free to organize around whatever they think their community needs. You can read more about how Open Data Day works on our website.

Want to join in on the fun? I thought you’d never ask. Listed below are some different ways you can help make Open Data Day 2014 a success in your community!

A) How can I let EVERYONE know about open data day

I love the enthusiasm. Here’s a tweet you can send:

#OpenData Day is community powered in a timezone near you.  http://opendataday.org/ #ODD2014

Yes, our hashtag is #ODD2014. Cause we are odd. And cause we love open data.

B) I’d like to participate!

Great! If you are interested in participating in check out the Open Data Day wiki. We’ve just unlocked the pages so cities haven’t been added yet but feel free to add your city to the list, and put down your name as interested in participating. You can even check to see who organized the event last year to see if they are interested in doing it again.

C) Forget about participating, I want to coordinate an Open Data Day event in my city.

Whoa! Very exciting! Here’s a short checklist of what to do:

  • If you didn’t organize one last year, check to see if anyone in your city did. It would be good to connect with them first.
  • Read the Open Data Day website. Basically, pick up on our vibe: we want Open Data Day to work for everyone, from novices who know little about data to experts like Kaggle participants and uber geeks like Bruce Schneier. These events have always been welcoming and encouraging – it is part of the design challenge.
  • Okay, now add your city to the list, let people know where it will be taking place (or that you are working on securing space), let them know a rough agenda, what to expect, and how they can contribute.
  • Add yourself to the 2014 Open Data Day map. (Hint: Wikipedia lists Lat/Long in the information side bar for each cities wiki page: “Coordinates: 43°42′N 79°24′W”)
  • Join the Open Data Day mailing list. Organizers tend to share best practices and tips here. It’s not serious, really just a help and support group.
  • Check out resources like this and this about how to organize a successful event.
  • Start spreading the news!

D) I want to help more! How can Open Data Day work more smoothly everywhere?

Okay, for the truly hardcore you right, we need help. Open Data day has grown. This means we’ve outgrown a whole bunch of our infrastructure… like our webpage! Everyone involved in this is a volunteer so… we have some extra heavy lifting we need help with. This includes:

a. Website template update: The current Open Data Day template was generously donated by Mark Dunkley (thank you!!!). We’d love to have it scale a little better and refresh the content. You can see the code on github here. Email me if you are interested. Skills required: css, design

b. Translation: Can you help translate the ODD site into your language? You can submit the requests on github or send a document to heather.leson at okfn dot org with the content. She’ll do the github stuff if that’s beyond you.

c. Map: Leaflet and layers helpers wanted! We’d like a map geek to help correct geolocation and keep the 2014 map fresh with accurate geo for all the locations. Github repo is here and the event list is here.

What’s next?

I’m really looking forward to this year… I’ve lots more thoughts I’ll be sharing shortly.

Plus, I can’t wait to hear from you!

The Importance of Open Data Critiques – thoughts and context

Over at the Programmable City website Rob Kitchin has a thoughtful blog post on open data critiques. It is very much worth reading and wider discussion. Specifically, there are two competing things worth noting. First, it is important for the open data community – and advocates in particular – to acknowledge the responsibility we have in debates about open data. Second, I’d like to examine some of the critiques raised and discuss those I think misfire and those that deserve deeper dives.

Open Data as Dominant Discourse

During my 2011 keynote at Open Government Data camp I talked about how the open data movement was at an inflection point:

For years we have been on the outside, yelling that open data matters. But now we are being invited inside.

Two years later the transition is more than complete. If you have any doubts, consider this picture:OD as DCOnce you have these people talking about things like a G8 Open Data Charter you are no longer on the fringes. Not even remotely.

It also means understanding the challenges around open data has never been more important. We – open data advocates – are now complicit it what many of the above (mostly) men decide to do around open data. Hence the importance of Rob’s post. Previously those with power were dismissive of open data – you had to scream to get their attention. Today, those same actors want to act now and go far. Point them (or the institutions they represent) in the wrong direction and/or frame an issue incorrectly and you could have a serious problem on your hands. Consequently, the responsibility of advocates has never been greater. This is even more the case as open data has spread. Local variations matter. What works in Vancouver may not always be appropriate in Nairobi or London.

I shouldn’t have to say this but I will, because it matters so much: Read the critiques. They matter. They will make you better, smarter, and above all, more responsible.

The Four Critiques – a break down

Reading the critiques and agreeing with them is, of course, not the same thing. Rob cites four critiques of open data: funding and sustainability, politics of the benign and empowering the empowered, utility and usability, and neoliberalisation and marketisation of public services. Some of these I think miss the real concerns and risks around open data, others represent genuine concerns that everyone should have at the forefront of their thinking. Let me briefly touch on each one.

Funding and sustainability

This one strikes me as the least effective criticism. Outside the World Bank I’ve not heard of many examples where government effectively sell their data to make money. I would be very interested in examples to the contrary – it would make for a great list and would enlighten the discussion – although not, I suspect in ways that would make either side of the discussion happy.

The little research that has been done into this subject has suggested that charging for government data almost never yields much money, and often actually serves as a loss creating mechanism. Indeed a 2001 KPMG study of Canadian geospatial data found government almost never made money from data sales if purchases by other levels of government were not included. Again in Canada, Statistics Canada argued for years that it couldn’t “afford” to make its data open (free) as it needed the revenue. However, it turned out that the annual sum generated by these sales was around $2M dollars. This is hardly a major contributor to its bottom line. And of course, this does not count the money that had to go towards salaries and systems for tracking buyers and users, chasing down invoices, etc…

The disappointing line in the critique however was this:

de Vries et al. (2011) reported that the average apps developer made only $3,000 per year from apps sales, with 80 percent of paid Android apps being downloaded fewer than 100 times.  In addition, they noted that even successful apps, such as MyCityWay which had been downloaded 40 million times, were not yet generating profits.

Ugh. First, apps are not what is going to make open data interesting or sexy. I suspect they will make up maybe 5% of the ecosystem. The real value is going to be in analysis and enhancing other services. It may also be in the costs it eliminates (and thus capital and time it frees up, not in the companies it creates), something I outlined in Don’t Measure the Growth, Measure the Destruction.

Moreover, this is the internet. The average doesn’t mean anything. The average webpage probably gets 2 page views per day. That hardly means there aren’t lots of very successful webpages. The distribution is not a bell curve, its a long tail, so it is hard to see what the average tells us other than the cost of experimentation is very, very low. It tells us very little about if there are, or will be successful uses of open data.

Politics of the benign and empowering the empowered

The is the most important critique and it needs to be engaged. There are definitely cases where data can serve to further marginalize at risk communities. In addition, there are data sets that for reasons of security and privacy, should not be made open. I’m not interested in publishing the locations of women’s shelters or worse, the list of families taking refuge in them. Nor do I believe that open data will always serve to challenge the status quo or create greater equality. Even at its most reductionist – if one believes that information is power, then greater ability to access and make us of information makes one more powerful – this means that winners and losers will be created by the creation of new information.

There are however, two things that give me some hope in this space. The first is that, when it comes to open data, the axis of competition among providers usually centers around accessibility. For example, the Socrata platform (an provider of open data portals to government) invests heavily in creating tools that make government data accessible and usable to the broadest possible audience. This is not a claim that all communities are being engaged (far from it) and that a great deal more work cannot be done, but there is a desire to show greater use which drives some data providers to try to find ways to engage new communities.

The second is that if we want to create data literate society – and I think we do, for reasons of good citizenship, social justice and economic competitiveness – you need the data first for people to learn and play with. One of my most popular blog posts is Learning from Libraries: The Literacy Challenge of Open Data in which I point out that one of the best ways to help people become data literate is to give them more interesting data to play with. My point is that we didn’t build libraries after everyone knew how to read, we built them beforehand with the goal of having them as a place that could facilitate learning and education. Of course libraries also often have strong teaching components to them, and we definitely need more of this. Figuring out who to engage, and how it can be done most effectively is something I’m deeply interested in.

There are also things that often depress me. I struggle to think of technologies that did not empower the empowered – at least initially. From the cell phone to the car to the printing press to open source software, all these inventions have had helped billions of people, but they did not distribute themselves evenly, especially at first. So the question cannot be reduced to – will open data empower the empowered, but to what degree, and where and with whom. I’ve seen plenty of evidence where data has enabled small groups of people to protect their communities or make more transparent the impact (or lack there of) of a government regulation. Open data expands the number of people who can use government information for their own ends – this, I believe is a good thing – but that does not mean we shouldn’t be constantly looking for ways to ensure that it does not reinforce structural inequity. Achieving perfect distribution of the benefits of a new technology, or even public policy, is almost impossible. So we cannot make perfect the enemy of the good. However, that does not hide the fact that there are real risk – and responsibilities as advocates – that need to be considered here. This is an issue that will need to be constantly engaged.

Utility and Usability

Some of the issues around usability I’ve addressed above in the accessibility piece – for some portals (that genuinely want users) the axis of evolution is pointed in the right direction with governments and companies (like Socrata) trying to embed more tools on the website to make the data more usable.

I also agree with the central concern (not a critique) of this section, which is that rather than creating a virtuous circle, poorly thought out and launched open data portals will create a negative “doomloops” in which poor quality data begets little interest which begets less data. However, the concern, in my mind, focuses on to narrow a problem.

One of the big reasons I’ve been an advocate of open data was a desire not just to help citizens, non-profits and companies gain access to information that could help them with their missions, but to change the way government deals with its data so that it can share it internally more effectively. I often cite a public servant I know who had a summer intern spend 3 weeks surfing the national statistical agency website to find data they knew existed but could not find because of terrible design and search. A poor open data site is not just a sign that the public can’t access or effectively use government data, it usually suggests that the governments employees can’t access or effectively use their own data. This is often deeply frustrating to many public servants.

Thus, the most important outcome created by the open data movement may have been making governments realize that data represents an asset class that of which they have had little understanding (outside, sadly, the intelligence sector, which has been all too aware of this) and little policy and governance (outside, say, the GIS space and some personal records categories). Getting governments to think about data as a platform (yes, I’m a fan of government as a platform for external use, but above all for internal use) is, in my mind, one way we can both enable public servants to get better access to information while simultaneously attacking the huge vendors (like SAP and Oracle) whose $100 million dollar implementations often silo off data, rarely produce the results promised and are so obnoxiously expensive it boggles the mind (Clay Johnson has some wonderful examples of the roughly 50% of large IT projects that fail).

They key to all this is that open data can’t be something you slap on top of a big IT stack. I try to explain this in It’s the Icing Not the Cake, another popular blog post about why Washington DC was able to effectively launch an open data program so quickly (which was, apparently, so effective at bringing transparency to procurement data the subsequent mayor rolled it back). The point is, that governments need to start thinking in terms of platforms if – over the long term – open data is going to work. And it needs to start thinking of itself as the primary consumer of the data that is being served on that platform. Steve Yegge’s brilliant and sharp witted rant on how Google doesn’t get platforms is an absolute must read in this regard for any government official – the good news is you are not alone in not finding this easy. Google struggles with it as well.

My main point. Let’s not play at  the edges and merely define this challenge as one of usability. It is much, much bigger problem than that. It is a big, deep, culture-changing BHAG problem that needs tackling. If we get it wrong, then the big government vendors and he inertia of bureaucracy win. We get it right and we potentially could save taxpayers millions while enabling a more nimble, effective and responsive government.

Neoliberalisation and Marketisation of Government

If you not read Jo Bates article “Co-optation and contestation in the shaping of the UK’s Open Government Data Initiative” I highly recommend it. There are a number of arguments in the article I’m not sure I agree with (and feel are softened by her conclusion – so do read it all first). For example, the notion that open data has been co-opted into an “ideologically framed mould that champions the superiority of markets over social provision” strikes me as lacking nuance. One of the things open data can do is create a public recognition of a publicly held data set and the need to protect these against being privatized. Of course, what I suspect is that both things could be true simultaneously – there can be increased recognition of the importance of a public asset while also recognizing the increased social goods and market potential in leveraging said asset.

However, there is one thing Bates is absolutely correct about. Open data does not come into an empty playing field. It will be used by actors – on both the left and right – to advance their cause. So I too am uncomfortable with those that believe open data is going to somehow depoliticize government or politics – indeed I made a similar argument in a piece in Slate on the politics of data. As I try to point out you can only create a perverse, gerrymandered electoral district that looks like this…

gerrymandered in chicago… if you’ve got pretty good demographic data about target communities you want to engage (or avoid). Data – and even open data – doesn’t magically make things better. There are instances where open data can, I believe, create positive outcomes by shifting incentives in appropriate ways… but similarly, it can help all sorts of actors find ways to satisfy their own goals, which may not be aligned with your – or even society at large’s – goals.

This makes voices like Bates deeply important since they will challenge those of us interested in open data to be constantly evaluating the language we use, the coalitions we form and the priorities that get made, in ways that I think are profoundly important. Indeed, if you get to the end of Bates article there are a list of recommendations that I don’t think anyone I work with around open data would find objectionable, quite the opposite, they would agree are completely critical.


I’m so grateful to Rob for posting this piece. It is has helped me put into words some thoughts I’ve had, both about the open data criticisms as well as the important role the critiques play. I try hard to be critical advocate of open data – one who engages the risks and challenges posed by open data. I’m not perfect, and balancing these two goals – advocacy with a critical view – is not easy, but I hope this shines some window into the ways I’m trying to balance it and possible helps others do more of it as well.

The promise and challenges of open government – Toronto Star OpEd

As some readers many know it was recently announced that I’ve been asked by Ontario Premier Wynn and Government Services Minister John Milloy to be part of the Government of Ontario’s task force on Open Government.

The task force will look at best practices around the world as well as engage a number of stakeholders and conduct a series of public consultations across Ontario to make a number of recommendations around opening up the Ontario government.

I have an opinion piece in the Toronto Star today titled The Promise and Challenges of Open Government where I try (in a few words) to outline some of the challenges the task force faces as well as some of the opportunities I hope it can capitalize on.

The promise and challenges of open government

Last week, Premier Kathleen Wynne announced the launch of Ontario’s Open Government initiative, including an engagement task force (upon which I sit).

The premier’s announcement comes on the heels of a number of “open government” initiatives launched in recent years. President Barack Obama’s first act in 2009 was to sign the Memorandum on Transparency and Open Government. Since then numerous city, state and provincial governments across North America are finding new ways to share information. Internationally, 60 countries belong to the Open Government Partnership, a coalition of states and non-profits that seeks to improve accountability, transparency, technology and innovation and citizen participation.

Some of this is, to be blunt, mere fad. But there is a real sense among many politicians and the public that governments need to find new ways to be more responsive to a growing and more diverse set of citizen needs, while improving accountability.

Technology has a certainly been – in part – a driver, if only because it shifts expectations. Today a Google search takes about 30 milliseconds, with many users searching for mere minutes before locating what they are looking for. In contrast, access to information requests can take weeks, or months to complete. In an age of computers, government processes often seem more informed by the photocopier – clinging to complex systems for sorting, copying and sharing information – than using computer systems that make it easy to share information by design.

There is also growing recognition that government data and information can empower people both inside and outside government. In British Columbia, the province’s open data portal is widely used by students – many of whom previously used U.S. data as it was the only free source. Now the province benefits from an emerging workforce that uses local data while studying everything from the environment to demography to education. Meanwhile the largest user of B.C.’s open data portal are public servants, who are able to research and create policy while drawing on better information, all without endless meetings to ask for permission to use other departments’ data. The savings from fewer meetings alone is likely significant.

The benefits of better leveraging government data can affect us all. Take the relatively mundane but important issue of transit. Every day hundreds of thousands of Ontarians check Google Maps or locally developed applications for transit information. The accumulated minutes not spent waiting for transit has likely saved citizens millions of hours. Few probably realize however that it is because local governments “opened” transit data that it has become so accessible on our computers and phones.

Finally, there are a number of new ways to think about how to “talk” to Ontarians. It is possible that traditional public consultations could be improved. But there is also an opportunity to think more broadly about how the government interacts with citizens. Projects like Wikipedia demonstrate how many small contributions can create powerful resources and public assets. Could such a model apply to government?

All of these opportunities are exciting – and the province is right to explore them. But important policy questions remain. For example: how do we safeguard the data government collects to minimize political interference? The country lost a critical resource when the federal government destroyed the reliability of the long form census by making it voluntary. If crowdsourcing and other new forms of public engagement can be adopted for government, how do we manage privacy concerns and preserve equality of opportunity? And how will such changes affect public representation? Canada’s political system has been marked by increasing centralization of power over the past several decades – will new technologies and approaches further this trend? Or could they be shaped to arrest it? These are not simple questions.

It is also easy to dismiss these efforts. This will neither be the first nor the last time people talk about open government. Indeed, there is a wonderfully cynical episode of Yes, Minister from 1980 titled “Open Government.” More recently, various revelations about surveillance and national governments’ desire to snoop in on our every email and phone call reveals much about what is both opaque and to be feared about our governments. Such cynicism is both healthy and necessary. It is also a reason why we should demand more.

Open government is not something we will ever fully achieve. But I do hope that it can serve as an objective and a constantly critical lens for thinking about what we should demand. I can’t speak for the other panelists of the task force, but that will be how I approach my work.

David Eaves is a public policy entrepreneur, open government activist and negotiation expert. He is a member of the Ontario government’s new Engagement Task Force.