The State of Open Data 2011

What is the state of the open data movement? Yesterday, during my opening keynote at the Open Government Data Camp (held this year in Warsaw, Poland) I sought to follow up on my talk from last year’s conference. Here’s my take of where we are today (I’ll post/link to a video of the talk as soon as the Open Knowledge Foundation makes it available).

Successes of the Past Year: Crossing the Chasm

1. More Open Data Portals

One of the things that has been amazing to witness in 2011 is the veritable explosion of Open Data portals around the world. Today there are well over 50 government data catalogs with more and more being added. The most notable of these was probably the Kenyan Open Data catalog which shows how far, and wide, the open data movement has grown.

2. Better Understanding and More Demand

The things about all these portals is that they are the result of a larger shift. Specifically, more and more government officials are curious about what open data is. This is not to say that understanding has radically shifted, but many people in government (and in politics) now know the term, believe there is something interesting going on in this space, and want to learn more. Consequently, in a growing number of places there is less and less headwind against us. Rather than screaming from the rooftops, we are increasingly being invited in the front door.

3. More Experimentation

Finally, what’s also exciting is the increased experimentation in the open data space. The number of companies and organizations trying to engage open data users is growing. ScraperWiki, the DataHub, BuzzData, Socrata,, are some of the products and resources that have emerged out of the open data space. And the types of research and projects that are emerging – the tracking of the Icelandic volcano eruptions, the emergence of hacks and hackers, micro projects (like my own and the research showing that open data could be generating savings of £8.5 million a year to governments in the Greater Manchester area, is deeply encouraging.

The Current State: An Inflection Point

The exciting thing about open data is that increasingly we are helping people – public servants, politicians, business owners and citizens imagine a different future, one that is more open, efficient and engaging. Our impact is still limited, but the journey is still in its early days. More importantly, thanks to success (number 2 above) our role is changing. So what does this mean for the movement right now?

Externally to the movement, the work we are doing is only getting more relevant. We are in an era of institution failure. From the Tea Party to Occupy Wall St. there is a recognition that our institutions no longer sufficiently serve us. Open data can’t solve this problem, but it is part of the solution. The challenge of the old order and the institutions it fostered is that its organizing principle is built around the management (control) of processes, it’s been about the application of the industrial production model to government services. This means it can only move so fast, and because of its strong control orientation, can only allow for so much creativity (and adaption). Open data is about putting the free flow of information at the heart of government – both internally and externally – with the goal of increasing government’s metabolism and decentralizing societies’ capacity to respond to problems. Our role is not obvious to the people in those movements, and we should make it clearer.

Internally to the movement, we have another big challenge. We are at a critical inflection point. For years we have been on the outside, yelling that open data matters. But now we are being invited inside. Some of us want to rush in, keen to make advances, others want to hold back, worried about being co-opted. To succeed, it is essential we must become more skilled at walking this difficult line: engaging with governments and helping them make the right decisions, while not being co-opted or sacrificing our principles. Choosing to not engage would, in my opinion, be to abscond from our responsibility as citizens and open data activists. This is a difficult transition, but it will be made easier if we at least acknowledge it, and support one another in it.

Our Core Challenges: What’s next

Looking across the open data space, my own feeling is that there are three core challenges that are facing the open data movement that threaten to compromise all the successes we’ve currently enjoyed.

1. The Compliance Trap

One key risk for open data is that all our work ends up being framed as a transparency initiative and thus making data available is reduced to being a compliance issue for government departments. If this is how our universe is framed I suspect in 5-10 years governments, eager to save money and cut some services, will choose to cut open data portals as a cost saving initiative.

Our goal is not to become a compliance issue. Our goal is to make governments understand that they are data management organizations and that they need to manage their data assets with the same rigour with which they manage physical assets like roads and bridges. We are as much about data governance as we are open data. This means we need to have a vision for government, one where data becomes a layer of the government architecture. Our goal is to make data platform one that not only citizens outside of government can build on, but one that government reconstructs its policy apparatus as well as its IT systems at top of. Achieving this will ensure that open data gets hardwired right into government and so cannot be easily shut down.

2. Data Schemas

This year, in the lead up to the Open Data Camp, the Open Knowledge Foundation created a map of open data portals from around the world. This was fun to look at, and I think should be the last time we do it.

We are getting to a point where the number of data portals is becoming less and less relevant. Getting more portals isn’t going to enable open data to scale more. What is going to allow us to scale is establishing common schemas for data sets that enable them to work across jurisdictions. The single most widely used open government data set is transit data, which because it has been standardized by the GTFS is available across hundreds of jurisdictions. This standardization has not only put the data into google maps (generating millions of uses everyday) but has also led to an explosion of transit apps around the world. Common standards will let us scale. We cannot forget this.

So let’s stop mapping open data portals, and start mapping datasets that adhere to common schemas. Given that open data is increasingly looked upon favourably by governments, creating these schemas is, I believe, now the central challenge to the open data movement.

3. Broadening the Movement

I’m impressed by the hundreds and hundreds of people here at the Open Data Camp in Warsaw. It is fun to be able to recognize so many of the faces here, the problem is that I can recognize too many of them. We need to grow this movement. There is a risk that we will become complacent, that we’ll enjoy the movement we’ve created and, more importantly, our roles within it. If that happens we are in trouble. Despite our successes we are far from reaching critical mass.

The simple question I have for us is: Where is the United Way, Google, Microsoft, the Salvation Army, Oxfam, and Greenpeace? We’ll know were are making progress when companies – large and small – as well as non-profits – start understanding how open government data can change their world for the better and so want to help us advance the cause.

Each of us needs to go out and start engaging these types of organizations and helping them see this new world and the potential it creates for them to make money or advance their own issues. The more we can embed ourselves into other’s networks, the more allies we will recruit and the stronger we will be.


17 thoughts on “The State of Open Data 2011

  1. Sarah J

    Hello, totally agree with your last comment re broadening the movement. For example (and this is completely my own view), international aid funders are starting to look at IATI as an ‘answer’ to their open data commitment. But, the data has to come from us, the recipient INGOs. And there’s a whole raft of questions that haven’t been answered yet. Not least of which is, how is the data we publish going to benefit the people we work with – communities in developing countries. Until there’s some good answers, I’m not sure you’ll get the likes of Oxfam rushing to sign up to open data. But I’m well open to challenges and suggestions on this view!

  2. Andrew

    David, this is good read – thank you.  I was hoping to find a write-up of your thoughts from the International Conference of Information Commissioners (in Ottawa 4-5 October).  Partly because I was speaking the session parallel to yours so I missed the opportunity to hear you live, and partly because I think it would be useful to have on the record your thoughts on what happens when an open data advocate meets the FOI crowd.

  3. Pingback: Newsbits #3 | Open Geodata

  4. Pingback: sillygwailo sent a spelling edit. | Editz

  5. Pingback: The State of Open Data », News, Augmented

  6. Claudia Schwegmann

    Too many people still have no idea what open data is about. Those who have heard the term open data may hold a host of misconceptions and fears. To move ahead in open data, we cannot wait for important stakeholders to come and ask for advice. Organisations that don’t have internal champions may take years to discover open data. So I absolutely agree that we need to go out and talk to those organisations – particularly those that should have a keen and immediate interest in open data: transparency international, global integrity, greenpeace, NGO networks like CIVICUS, sector specific alliances like the land coalition, the climate alliance, global health networks, UNESCO, media networks, networks of parliamentarians, etc. We also need to engage much more with NGOs about their own data. OpenAid initiated three open aid data events in September 2011 to reach out to the wider aid community and we are planning to doing more in 2012. Any collaboration on this is most welcome.

  7. Pingback: L’Open Data en 2011 » Digital-In

  8. Pingback: Lo stato degli open data 2011 « Open data in Italia

  9. Pingback: Lo stato dell’Open Data nel 2011. « ilNichilista

  10. lawrence serewicz

    I am not sure we can say that our insitutions have failed. We still have the bins emptied, schools are open, and hospitals still function as do the roads and air travel.  They were doing that before Open Data and they will do that after it.

    I was taken by the following idea: “Open data is about putting the free flow of information at the heart of government – both internally and externally – with the goal of increasing government’s metabolism and decentralizing societies’ capacity to respond to problems.”

    In this sense, it is trying to get government to be more efficient and effective, but is that what we want? In one sense, democracy is about slowing government down, making it explain itself, and keeping it from reacting instantly and intensively to events that do not require it. For example, we want the government to react quickly to a disaster, to help people, but we do not want it to react quickly in censoring or inhibiting freedoms.

    I fear that open data may limit itself by staking ideological or political positions when the open data movement is not about politics or ideology, except so far as open data can be considered an ideology.  Let’s focus on the data, how it applies, and how it helps people, as people, in their day to day lives rather than consider whether insitutions have failed and open data opens a brave utopia of an efficient, effective government response to the individual.

  11. Pingback: Solving the Common Standards problem in the Open Data Space |

  12. Pingback: The State of Open Data 2011 : Datos Abiertos

  13. Pingback: Calculating the Value of Canada’s Open Data Portal: A Mini-Case Study |

  14. Pingback: Government In The Lab | Government In The Lab

  15. Pingback: Un état des lieux de l’Open Data » OWNI, News, Augmented

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s