Tag Archives: transparency

My Canadian Open Government Consultation Submission

Attached below is my submission to the Open Government Consultation conducted by Treasury Board over the last couple of weeks. There appear to be a remarkable number of submission that were made by citizens, which you can explore on the Treasury Board website. In addition, Tracey Lauriault has tracked some of the submissions on her website.

I actually wish the submissions on the Government website were both searchable and could be downloaded in there entirety. That way we could re-organize them, visualize them, search and parse them as well as play with the submissions so as to make the enormous number of answers easier to navigate and read. I can imagine a lot of creative ways people could re-format all that text and make it much more accessible and fun.

Finally, for reference, in addition to my submission I wrote this blog post a couple months ago suggesting goals the government set for itself as part of its Open Government Partnership commitments. Happily, since writing that post, the government has moved on a number of those recommendations.

So, below is my response to the government’s questions (in bold):

What could be done to make it easier for you to find and use government data provided online?

First, I want to recognize that a tremendous amount of work has been done to get the present website and number of data sets up online.

FINDING DATA:

My advice on making data easier to engage Socrata to create the front end. Socrata has an enormous amount of experience in how to share government data effectively. Consider http://data.oregon.gov here is a site that is clean, easy to navigate and offers a number of ways to access and engage the governments data.

More specifically, what works includes:

1. Effective search: a simple search mechanism returns all results
2. Good filters: Because the data is categorized by type (Internal vs. external, charts, maps, calendars, etc…) it is much easier to filter. One thing not seen on Socrata that would be helpful would be the ability to sort by ministry.
3. Preview: Once I choose a data set I’m given a preview of what it looks like, this enables me to assess whether or not it is useful
4. Social: Here there is a ton on offer
– I’m able to sort data sets by popularity – being able to see what others find interesting is, in of itself interesting.
– Being able to easily share data sets via email, or twitter and facebook means I’m more likely to find something interesting because friends will tell me about it
– Data sets can also be commented upon so I can see what others think of the data, if they think it is useful or not, and what for or not.
– Finally, it would be nice if citizens could add meta data, to make it easier for others to do keyword searches. If the government was worried about the wrong meta data being added, one could always offer a search with crowd sourced meta data included or excluded
5. Tools: Finally, there are a large number of tools that make it easier to quickly play with and make use of the data, regardless of one’s skills as a developer. This makes the data much more accessible to the general public.

USING DATA

Finding data is part of the problem, being able to USE the data is a much bigger issue.

Here the single most useful thing would be to offer API’s into government data. My own personal hope is that one day there will be a large number of systems both within and outside of government that will integrate government data right into their applications. For example, as I blogged about here – https://eaves.ca/2011/02/18/sharing-critical-information-with-public-lessons-for-governments/ – product recall data would be fantastic to have as an API so that major retailers could simply query the API every time they scan inventory in a warehouse or at the point of sale, any product that appears on the list could then be automatically removed. Internally, Borders and Customs could also query the API when scanning exports to ensure that nothing exported is recalled.

Second, if companies and non-profits are going to invest in using open data, they need assurances that both they are legally allowed to use the data and that the data isn’t going to suddenly disappear on them. This means, a robust license that is clear about reuse. The government would be wise to adopt the OGL or even improve on it. Better still helping establish a standardized open data license for Canada and ideally internationally could help reduce some legal uncertainty for more conservative actors.

More importantly, and missing from Socrata’s sites, would be a way of identifying data sets on the security of their longevity. For example, data sets that are required by legislation – such as the NPRI – are the least likely to disappear, whereas data sets the the long form census which have no legal protection could be seen as at higher risk.

 

How would you use or manipulate this data?

I’m already involved in a number of projects that use and share government data. Among those are Emitter.ca – which maps and shares NPRI pollution data and Recollect.net, which shares garbage calendar information.

While I’ve seen dramatically different uses of data, for me personally, I’m interested mostly in using data for thinking and writing about public policy issues. Indeed, much has been made of the use of data in “apps” but I think it is worth noting that the single biggest use of data will be in analysis – government officials, citizens, academics and others using the data to better understand the world around them and lobby for change.

This all said, there are some data sets that are of particular usefulness to people, these include:

1. Data sets on sensitive issues, this includes health, inspection and performance data (Say surgery outcomes for specific hospitals, or restaurant inspection data, crime and procurement data are often in great demand).
2. Dynamic real-time Data: Data that is frequently updated (such a border, passport renewal or emergency room wait times). This data is shared in the right way can often help people adjust schedules and plans or reallocate resources more effectively. Obviously this requires an API.
3.Geodata: Because GIS standards are very mature it is easy to “mashup” geo data to create new maps or offer new services. These common standards means that geo data from different sources will work together or can be easily compared. This is in sharp contrast to say budget data, where there are few common standards around naming and organizing the data, making it harder to share and compare.

What could be done to make it easier for you to find government information online?

It is absolutely essential that all government records be machine readable.

Some of the most deplorable moment in open government occur when the government shares documents with the press, citizens or parliamentary officers in paper form. The first and most important thing to make government information easier to find online is to ensure that it is machine readable and searchable by words. If it does not meet this criteria I increasingly question whether or not it can be declared open.

As part of the Open Government Partnership commitments it would be great for the government to commit to guarantee that every request for information made of it would include a digital version of the document that can be searched.

Second, the government should commit that every document it publishes be available online. For example, I remember in 2009 being told that if I wanted a copy of the Health Canada report “Human Health in a Changing Climate:A Canadian Assessment of Vulnerabilities and Adaptive Capacity” I had to request of CD, which was then mailed to me which had a PDF copy of the report on it. Why was the report not simply available for download? Because the Minister had ordered it not to appear on the website. Instead, I as a taxpayer and to see more of my tax dollars wasted for someone to receive my mail, process it, then mail me a custom printed cd. Enabling ministers to create barriers to access government information, simply because they do not like the contents, is an affront to the use of tax payer dollars and our right to access information.

Finally, Allow Government Scientists to speak directly to the media about their research.

It has become a reoccurring embarrassment. Scientists who work for Canada publish an internationally recognized ground break paper that provides some insight about the environment or geography of Canada and journalists must talk to government scientists from other countries in order to get the details. Why? Because the Canadian government blocks access. Canadians have a right to hear the perspectives of scientists their tax dollars paid for – and enjoy the opportunity to get as well informed as the government on these issues.

Thus, lift the ban that blocks government scientists from speaking with the media.

 

Do you have suggestions on how the Government of Canada could improve how it consults with Canadians?

1. Honour Consultation Processes that have started

The process of public consultation is insulted when the government itself intervenes to bring the process into disrepute. The first thing the government could do to improve how it consults is not sabotage processes that already ongoing. The recent letter from Natural Resources Minister Joe Oliver regarding the public consultation on the Northern Gateway Pipelines has damaged Canadians confidence in the governments willingness to engage in and make effective use of public consultations.

2. Focus on collecting and sharing relevant data

It would be excellent if the government shared relevant data from its data portal on the public consultation webpage. For example, in the United States, the government shares a data set with the number and location of spills generated by Enbridge pipelines, similar data for Canada would be ideal to share on a consultation. Also useful would be economic figures, job figures for the impacted regions, perhaps also data from nearby parks (visitations, acres of land, kml/shape boundary files). Indeed, data about the pipeline route itself that could be downloaded and viewed in Google earth would be interesting. In short, there are all sorts of ways in which open data could help power public consultations.

3. Consultations should be ongoing

It would be great to see a 311 like application for the federal government. Something that when loaded up, would use GPS to identify the services, infrastructure or other resources near the user that is operated by the federal government and allow the user to give feedback right then and there. Such “ongoing” public feedback could then be used as data when a formal public consultation process is kicked off.

 

Are there approaches used by other governments that you believe the Government of Canada could/should model?

1. The UK governments expense disclosure and release of the COINS database more generally is probably the most radical act of government transparency to date. Given the government’s interest in budget cuts this is one area that might be of great interest to pursue.

2. For critical data sets, those that are either required by legislation or essential to the operation of a ministry or the government generally, it would be best to model the city of Chicago or Washington DC and foster the creation of a data warehouse where this data could be easily shared both internally and externally (as privacy and security permits). These cities are leading governments in this space because they have tackled both the technical challenges (getting the data on a platform where it can be shared easily) and around governance (tackling the problem of managing data sets from various departments on a shared piece of infrastructure).

 

Are there any other comments or suggestions you would like to make pertaining to the Government of Canada’s Open Government initiative?

Some additional ideas:

Redefine Public as Digital: Pass an Online Information Act

a) Any document it produces should be available digitally, in a machine readable format. The sham that the government can produce 3000-10,000 printed pages about Afghan detainees or the F-35 and claim it is publicly disclosing information must end.

b) Any data collected for legislative reasons must be made available – in machine readable formats – via a government open data portal.

c) Any information that is ATIPable must be made available in a digital format. And that any excess costs of generating that information can be born by the requester, up until a certain date (say 2015) at which point the excess costs will be born by the ministry responsible. There is no reason why, in a digital world, there should be any cost to extracting information – indeed, I fear a world where the government can’t cheaply locate and copy its own information for an ATIP request as it would suggest it can’t get that information for its own operations.

Use Open Data to drive efficiency in Government Services: Require the provinces to share health data – particularly hospital performance – as part of its next funding agreement within the Canada Health Act.

Comparing hospitals to one another is always a difficult task, and open data is not a panacea. However, more data about hospitals is rarely harmful and there are a number of issues on which it would be downright beneficial. The most obvious of these would be deaths caused by infection. The number of deaths that occur due to infections in Canadian hospitals is a growing problem (sigh, if only open data could help ban the antibacterial wipes that are helping propagate them). Having open data that allows for league tables to show the scope and location of the problem will likely cause many hospitals to rethink processes and, I suspect, save lives.

Open data can supply some of the competitive pressure that is often lacking in a public healthcare system. It could also better educate Canadians about their options within that system, as well as make them more aware of its benefits.

Reduce Fraud: Creating a Death List

In an era where online identity is a problem it is surprising to me that I’m unable to locate a database of expired social insurance numbers. Being able to query a list of social security numbers that belong to dead people might be a simple way to prevent fraud. Interestingly, the United States has just such a list available for free online. (Side fact: Known as the Social Security Death Index this database is also beloved by genealogist who use it to trace ancestry).

Open Budget and Actual Spending Data

For almost a year the UK government has published all spending data, month by month, for each government ministry (down to the £500 in some, £25,000 in others). More over, as an increasing number of local governments are required to share their spending data it has lead to savings, as government begin to learn what other ministries and governments are paying for similar services.

Create a steering group of leading Provincial and Municipal CIOs to create common schema for core data about the country.

While open data is good, open data organized the same way for different departments and provinces is even better. When data is organized the same way it makes it easier to citizens to compare one jurisdiction against another, and for software solutions and online services to emerge that use that data to enhance the lives of Canadians. The Federal Government should use its convening authority to bring together some of the countries leading government CIOs to establish common data schemas for things like crime, healthcare, procurement, and budget data. The list of what could be worked on is virtually endless, but those four areas all represent data sets that are frequently requested, so might make for a good starting point.

Open Government Advocacy: The Danger of Letting Narrative Trump Fact

So I loath making this the first post of the new year, but here we go.

Today Canada.com published a story “Tony Clement vows innovative new open government, but critics point to poor record.” In it,  Jason Fekete the journalist responsible for the story, quotes a Democracy Watch spokesperson who sadly gets the facts completely wrong despite the fact that I warned Democracy Watch about their error a month ago after their press release caused similar errors to appear in a CBC story. I’ll outline why this is problem later in the post. Bur first the error.

In the article Fekete reports

Democracy Watch said it will appeal to the international open government group to reject Canada’s entry because the federal government failed to keep one of its required commitments to consult Canadians. Ottawa announced its online consultation one day after the watchdog complained about it.

This is, in fact, not true. To date, the government has not failed to meet its commitment. As I pointed out in an earlier blog post (to which Democracy Watch responded as is aware) Democracy Watch accuses the government of failing because it believed consultations needed to be conducted before a November OGP meeting in Brazil. Unfortunately, the meeting in which Governments will be sharing their plans (and thus need to complete their consultations) will be taking place in Brazil in April. The OPG clearly states this on their website (under section 5). There was a meeting in November, so the confusion was understandable.

Of course, just to be safe, I did what the CBC and Postmedia should have done. I emailed the OGP secretariat to check. Within minutes they confirmed to me that the April meeting is the deadline for consultations and developing plans. What is more interesting to me is the no one from Democracy Watch, the CBC or Post Media bothered to simply email or call the OGP secretariat and confirm these facts. For the CBC and Postmedia this is a matter of laziness. For Democracy Watch, I’m not sure what is driving this willed blindness. Ultimately, I suspect that once they went public with their narrative, backing down would be seen as weakness and then government secrecy would win!

This is dangerous for two reasons.

The first is, it isn’t true. Government secrecy doesn’t win if Democracy Watch got its facts mixed up. I agree that this government has a lot to answer for around its willingness to disclose government documents. Be it documents around the procurement of the F-35 or the treatment of Afghan prisoners there are many cases where the lack of transparency has been blatant and, I believe, counter to the principles of democracy and open government. Conceding that the Government is still on track for its Open Government Partnership objectives does not diminish that fact. The only thing that misrepresenting the facts does is cause conservative leaning voters who believe in government transparency (an important constituency) to tune out of the debate and believe that Democracy Watch is simply out to score points against the government, not fulfill its mission.

The bigger reason I think it is dangerous is that it undermines the very thing that makes the Open Government Partnership an effective tool of open government advocates. I want to be clear. The Open Government Partnership is, in part, designed to empower advocates and help them compel their government’s to be more open. Used correctly it could be powerful. The fact that Canadian government signed on to the International Aid Transparency Initiative (IATI) and the Extractive Industries Transparency Initiative is a result, in part, of the fact that other OGP countries were signing on. We were able to use peer pressure to create an upward spiral. We can also use the timelines of the OGP to ensure the government carries out the pledges that it has made. And of course, there is an Open Government consultation that is currently taking place (please participate) that the government will have to share the results of with its partners – potentially giving us a way to verify that our input is being taken seriously. Indeed, participating OPG countries may even try to out do one another to demonstrate how seriously they are taking this input.

But when this tool is misused it gives the government license to ignore and write off critics. As someone who wants to use the OGP commitments as a carrot and stick to hold our government to account, stories like those I linked to above hurt our capacity to be effective.

This government does not have a great record of transparency. At the same time, there is a legitimate effort to create open government goals as part of the OGP, let’s let the process run its course (and criticize when they actually violate the process) and use the tools that are at our disposal constructively to maximize impact, rather than try to snare a quick headline that in the long term, could damage the open government movements credibility.

I certainly wouldn’t encourage Democracy Watch to petition the OGP to ask Canada to leave the partnership. I suspect the secretariat would be confused by the request. The deadline has not passed, indeed, most OGP countries are in the middle of their consultations right now.

 

 

The State of Open Data 2011

What is the state of the open data movement? Yesterday, during my opening keynote at the Open Government Data Camp (held this year in Warsaw, Poland) I sought to follow up on my talk from last year’s conference. Here’s my take of where we are today (I’ll post/link to a video of the talk as soon as the Open Knowledge Foundation makes it available).

Successes of the Past Year: Crossing the Chasm

1. More Open Data Portals

One of the things that has been amazing to witness in 2011 is the veritable explosion of Open Data portals around the world. Today there are well over 50 government data catalogs with more and more being added. The most notable of these was probably the Kenyan Open Data catalog which shows how far, and wide, the open data movement has grown.

2. Better Understanding and More Demand

The things about all these portals is that they are the result of a larger shift. Specifically, more and more government officials are curious about what open data is. This is not to say that understanding has radically shifted, but many people in government (and in politics) now know the term, believe there is something interesting going on in this space, and want to learn more. Consequently, in a growing number of places there is less and less headwind against us. Rather than screaming from the rooftops, we are increasingly being invited in the front door.

3. More Experimentation

Finally, what’s also exciting is the increased experimentation in the open data space. The number of companies and organizations trying to engage open data users is growing. ScraperWiki, the DataHub, BuzzData, Socrata, Visua.ly, are some of the products and resources that have emerged out of the open data space. And the types of research and projects that are emerging – the tracking of the Icelandic volcano eruptions, the emergence of hacks and hackers, micro projects (like my own Recollect.net) and the research showing that open data could be generating savings of £8.5 million a year to governments in the Greater Manchester area, is deeply encouraging.

The Current State: An Inflection Point

The exciting thing about open data is that increasingly we are helping people – public servants, politicians, business owners and citizens imagine a different future, one that is more open, efficient and engaging. Our impact is still limited, but the journey is still in its early days. More importantly, thanks to success (number 2 above) our role is changing. So what does this mean for the movement right now?

Externally to the movement, the work we are doing is only getting more relevant. We are in an era of institution failure. From the Tea Party to Occupy Wall St. there is a recognition that our institutions no longer sufficiently serve us. Open data can’t solve this problem, but it is part of the solution. The challenge of the old order and the institutions it fostered is that its organizing principle is built around the management (control) of processes, it’s been about the application of the industrial production model to government services. This means it can only move so fast, and because of its strong control orientation, can only allow for so much creativity (and adaption). Open data is about putting the free flow of information at the heart of government – both internally and externally – with the goal of increasing government’s metabolism and decentralizing societies’ capacity to respond to problems. Our role is not obvious to the people in those movements, and we should make it clearer.

Internally to the movement, we have another big challenge. We are at a critical inflection point. For years we have been on the outside, yelling that open data matters. But now we are being invited inside. Some of us want to rush in, keen to make advances, others want to hold back, worried about being co-opted. To succeed, it is essential we must become more skilled at walking this difficult line: engaging with governments and helping them make the right decisions, while not being co-opted or sacrificing our principles. Choosing to not engage would, in my opinion, be to abscond from our responsibility as citizens and open data activists. This is a difficult transition, but it will be made easier if we at least acknowledge it, and support one another in it.

Our Core Challenges: What’s next

Looking across the open data space, my own feeling is that there are three core challenges that are facing the open data movement that threaten to compromise all the successes we’ve currently enjoyed.

1. The Compliance Trap

One key risk for open data is that all our work ends up being framed as a transparency initiative and thus making data available is reduced to being a compliance issue for government departments. If this is how our universe is framed I suspect in 5-10 years governments, eager to save money and cut some services, will choose to cut open data portals as a cost saving initiative.

Our goal is not to become a compliance issue. Our goal is to make governments understand that they are data management organizations and that they need to manage their data assets with the same rigour with which they manage physical assets like roads and bridges. We are as much about data governance as we are open data. This means we need to have a vision for government, one where data becomes a layer of the government architecture. Our goal is to make data platform one that not only citizens outside of government can build on, but one that government reconstructs its policy apparatus as well as its IT systems at top of. Achieving this will ensure that open data gets hardwired right into government and so cannot be easily shut down.

2. Data Schemas

This year, in the lead up to the Open Data Camp, the Open Knowledge Foundation created a map of open data portals from around the world. This was fun to look at, and I think should be the last time we do it.

We are getting to a point where the number of data portals is becoming less and less relevant. Getting more portals isn’t going to enable open data to scale more. What is going to allow us to scale is establishing common schemas for data sets that enable them to work across jurisdictions. The single most widely used open government data set is transit data, which because it has been standardized by the GTFS is available across hundreds of jurisdictions. This standardization has not only put the data into google maps (generating millions of uses everyday) but has also led to an explosion of transit apps around the world. Common standards will let us scale. We cannot forget this.

So let’s stop mapping open data portals, and start mapping datasets that adhere to common schemas. Given that open data is increasingly looked upon favourably by governments, creating these schemas is, I believe, now the central challenge to the open data movement.

3. Broadening the Movement

I’m impressed by the hundreds and hundreds of people here at the Open Data Camp in Warsaw. It is fun to be able to recognize so many of the faces here, the problem is that I can recognize too many of them. We need to grow this movement. There is a risk that we will become complacent, that we’ll enjoy the movement we’ve created and, more importantly, our roles within it. If that happens we are in trouble. Despite our successes we are far from reaching critical mass.

The simple question I have for us is: Where is the United Way, Google, Microsoft, the Salvation Army, Oxfam, and Greenpeace? We’ll know were are making progress when companies – large and small – as well as non-profits – start understanding how open government data can change their world for the better and so want to help us advance the cause.

Each of us needs to go out and start engaging these types of organizations and helping them see this new world and the potential it creates for them to make money or advance their own issues. The more we can embed ourselves into other’s networks, the more allies we will recruit and the stronger we will be.

 

CIDA announces Open Data portal: What it means to Canadians

For those who missed it, the Canadian International Development Agency (CIDA) has announced it is launching an open data portal.

This is exciting news. On Monday I was interviewed about the initiative by Embassy Magazine which published the resulting article (behind their paywall) here.

As (I hope) the interview conveys, I’m cautiously optimistic about the Minister’s announcement. I’m conservative in my reaction only because we don’t actually know what the Minister has announced. At the moment the CIDA open data page is, quite literally, a blank slate. I feel positive because pretty much anything that gets more information about Canada’s aid budget available online is a step in the right direction. I’m cautious however, because the text from the Minister’s speech leads me to believe that she is using the term “open data” to describe something that may, in fact, not be open data.

Donors and partner countries must be accountable to their citizens, absolutely, but both must also be accountable to each other.

Transparency underpins these accountabilities.

With this in mind, today I am pleased to announce the Open Data Portal on the CIDA website that will make our searchable database of roughly 3,000 projects quick and simple to access.

The Open Data portal will put our country strategies, evaluations, audits and annual statistical and results reports within easy reach.

One of the core elements of the definition of “open data” is that it be machine readable. I need to actually get the “data” (e.g an excel spreadsheet, or database I can download and/or access) so that I can play with it, mash it up, analyze it, etc… It isn’t clear that this is on offer. The minister’s announcements talks about a database that allows you to search, and quickly download, reports on the 3000 projects that CIDA funds or operates. A report however, is not data. It may cite data, it may (and hopefully does) even contain data in charts or tables, but if what we are getting is access to reports then this is not an open data portal.

What I hope is happening – and what I advocated for in an oped in the Toronto Star – is that the Minister is launching a true open data portal which will share actual data – not analysis – with Canadians. More importantly, I hope this means Canada will be joining the efforts of Publish What you Fund, as it pushes donor organizations to share their aid data in a single common structure, so that budgets, contributions, projects, timelines, geography and other information about aid can be compared across countries, agencies, and organizations.

Open data, and especially in a internationally recognized standardized format, matters because no one is going to read all 10,000 reports about all 3000 projects CIDA funds. However, if we had access to the data, in a structured manner, there are those at non-profits, in universities and colleges and in the media (among other places) that could map the projects, compare budgets and results more clearly, compare our efforts against those of other countries, and do their own analysis to say, find duplication and overlap. I don’t, for a second, believe that 99.9% of Canadians will use CIDA’s open data portal, but the .1% who do will be able to create products that can inform the rest of us, and allow us to better understand Canada’s role in the world. In other words, Open Data portal could be empowering and educating to a broad number of people. Access to 10,000 reports, while a good step, simply won’t be able to create a similar outcome on any scale. The difference is, quite frankly, dramatic.

So let’s wait and see. I’m excited that the Minister of International Cooperation is using the language of Open Data – it means that she and her staff understand it has currency. What I also hope is that they understand its meaning – so far we have no data on whether they do or do not, and I remain cautiously optimistic, they should, after all, realize the significance of the language they are using. Either way, they have set high expectations among those of us who think about, talk about and work in, this area. As a Canadian, I’m hoping those expectations get fulfilled.

Back to Reality: The Politics of Government Transparency & Open Data

A number of my friends and advocates in the open government, transparency and open data communities have argued that online government transparency initiatives will be permanent since, the theory goes, no government will ever want to bear the political cost of rolling it back and being perceived as “more opaque.” I myself have, at times, let this argument go unchallenged or even run with it.

This week’s US budget negotiations between Congress and the White House should lay that theory to rest. Permanently.

The budget agreement that has emerged from most recent round of negotiations – which is likely to be passed by congress –  slashes funding to an array of Obama transparency initiatives such as USASpending, the ITDashboard, and data.gov from $34M to $8M. Agree or disagree, Republicans are apparently all too happy to kill initiatives which make the spending and activities of the US government more transparent as well as create a number of economic opportunities around open data. Why? Because they believe it has no political consequences.

So unsurprisingly, it turns out that political transparency initiatives – even when they are online – are as bound to the realities of traditional politics as dot.com’s were bound by the realities of traditional economics. It’s not enough to get a policy created or an initiative launched – it needs to have a community, a group of interested supporters, to nurture and protect it. Otherwise, it will be at risk.

Back in 2009, in the lead up to the drafting and launching of Vancouver’s Open Data motion I talked about creating an open-government bargain. Specifically, I argued that:

..in an open city, a bargain must exists between a government and its citizens. To make open data a success and to engage the community a city must listen, engage, ask for help, and of course, fulfill its promise to open data as quickly as possible. But this bargain runs both ways. The city must to its part, but so too must the local tech community. They must participate, be patient (cities move slower than tech companies), offer help and, most importantly, make the data come alive for each other, policy makers and citizens through applications and shared analysis.

Some friends countered that open data and transparency should simply exist because it is the right thing to do. I don’t disagree – and I wish we lived in a world where the existence of this ideal was sufficient enough to guarantee these initiatives. But it isn’t sufficient. It’s easy to kill something that no one uses (or in the case of data.gov, that hasn’t been given enough time to generate a vibrant user base). It’s much, much harder to kill something that has a community that uses it, especially if that community and the products it creates are valued by society more generally. This is why open data needs users, it needs developers, think tanks and above all, the media, to take interest in it and to leverage it to create content. It’s also why I’ve tried to create projects like Emitter.ca, recollect.net, taxicity and others, because the more value we create with open data for everyone, the more secure government transparency policies will be.

It’s use it or risk losing it. I wish this weren’t the case, but it’s the best defense I can think of.

The Curious Case of Media Opposing Government Transparency

My gosh there is a lot going on. Republicans – REPUBLICANS(!) who were in charge of America’s prison system are warning Canada not to follow the Conservatives plan on prisons, the Prime Minister has renamed the government, after himself and my friends at Samara had in Toronto the Guardian’s Emily Bell to talk wikileaks and data journalism (wish I could have been there).

It’s all very interesting… and there is a media story here in British Columbia that’s been brewing where a number of journalists have become upset about a government that has become “too” transparent.

It’s an important case as it highlights some of the tensions that will be emerging in different places as governments rethink how they share information.

The case involves BC Ferries, a crown corporation that runs ferries along critical routes around the province. For many years the company was not subject to the province’s Freedom of Information legislation. However, a few months ago the government stated the crown corporation would need to comply with the act. This has not pleased the corporation’s president.

To comply with the act BC Ferries has created an FOI tracker website on which it posts the text of FOI requests received. Once the records are processed they are posted online and some relevant listservs. As a result they can be read by an audience (that cares).

Broadly, journalists, are up in arms for two reasons. One bad, the other even worse.

The terrible reasons was raised by Chad Skelton (who’s a great reporter for whom I have a lot of respect and whose column should be read regularly).

Skelton argues that BC Ferries deserves part of the blame for stories with errors as the process lead news agencies to rush (carelessly) in order to beat each other in releasing the story. This is a disappointing position. It’s the news media’s job to get the facts right. (It’s also worth noting here that Skelton’s own media organizations did not make the mistakes in question). Claiming that BC Ferries is even partly responsible seems beyond problematic since they are in no way involved in the fact and error checking processes. We trust the media (and assess it) to get facts right in fast moving situations… why should this be different?

More interesting is the critique that this model of transparency undermines the ability of journalists to get a scoup and thus undermines the business model of traditional media.

What makes this so interesting is that is neither true nor, more importantly, relevant.

First, it’s not the job of government to support the business model of the media. The goal of government should be to be as transparent as possible about its operations. This can, and should, include its FOI requests. Indeed, one thing I like about this process is that an FOI request that is made but isn’t addressed starts to linger on the site – and that the organization can be held to account, publicly, for the delay. More importantly, however, I’m confident that the media will find new ways to exploit the process and that, while painful, new business models will emerge.

Second, the media is not the only user of FOI. It strikes me as problematic to expect that the FOI system should somehow be tailored to meet needs alone. Individuals, non-profits, businesses, opposition politicians and others all use the FOI process. Indeed, the policy strengthens many of these use cases since, as mentioned above,  delays in processing will be visible and open the organization up to greater pressure and scrutiny. Why are all the use cases of these other institutions somehow secondary to those of journalists and the media? Indeed, the most important use case – that of the citizen – is better served. Isn’t that the most important outcome?

Third, this form of transparency could make for better media. One of my favourite quotes (which I got via Tim O’Reilly) comes from Clayton Christensen in a 2005 Harvard Business Review article:

“When attractive profits disappear at one stage in the value chain because a product becomes modular and commoditized, the opportunity to earn attractive profits with proprietary products will usually emerge at an adjacent stage.”

So BC Ferries has effectively commoditized FOI requests. That simply means that value will shift elsewhere. One place it could shift to is analysis. And wouldn’t that be a good thing to have the media compete on? Rather than simply who got the fact fastest (a somewhat silly model in the age of the internet) readers instead started to reward the organization with the best insights? Indeed, it makes me think that on superficial issues, like say, the salary of an employee, it may be hard for one individual or organization to scoop another. But most often the value of these stories is also pretty low. On a more significant story, one that requires research and digging and a knowledge of the issue, it’s unclear that transparency around FOI requests will allow others to compete. More interestingly, some media organizations, now that they have access to all FOI requests, might start analyzing them for deeper more significant patterns or trends that might reveal more significant problems that the current scattered approach to FOI might never reveal.

What’s also been interesting is the reaction stories by journalists complaining about this issue have been received. It fits nicely in with the piece I wrote a while ago (and now published as part of a journalism textbook) about Journalism in an Open Era. The fact is, the public trust of opaque institutions is in decline – and the media is itself a pretty opaque institution. Consider these three separate comments people wrote after the stories I’ve linked to above:

“I wonder over the years how many nuggets of information reporters got through FOI but the public never heard about because they didn’t deem it “newsworthy”. Or worse, that it was newsworthy but didn’t follow their storyline.” (found here)

“And the media whining about losing scoops — well, tough beans. If they post it all online and give it to everyone, they are serving the public –the media isn’t the public, and never has been.” (found here)

“The media’s track record, in general, for owning up to its blunders continues to be abysmal. Front page screw-ups are fixed several days (or weeks) later with a little “setting it straight” box buried at the bottom of P. 2 — and you think that’s good enough. If the media were more open and honest about fixing its mistakes, I might cut you a little slack over the BC Ferries’ policy of making your life difficult. But whining about it is going to be counterproductive, as you can see from most of the comments so far.” (found here)

While some comments were supportive of the articles, the majority have not been. Suggesting that at the minimum that the public does not share the media’s view that this new policy is a “controversial.”

This is not, of course, to say that BC Ferries implemented its policy because it sought to do the right thing. I’m sure it’s president would love for their to be fewer requests and impede the efforts of journalists. I just happen to think he will fail. Dismally. More concerning is the fact that FOI requests are not archived on the site and are removed after a few months. This is what should get the media, the public and yes, the Information and Privacy Commissioner, up in arms.

Open Knowledge Foundation Open Data Advocate

My colleagues over at the Open Knowledge Foundation have been thinking about recruiting an Open Data Advocate, someone who can coordinate a number of the activities they are up to in the open data space. I offered to think about what the role should entail and how that person could be effective. Consequently, in the interests of transparency, fleshing out my thinking and seeing if there might be feed back (feel free to comment openly, or email me personally if you wish to keep it private) I’m laying out my thinking below.

Context

These are exciting times for open government data advocates. Over the past few years a number of countries, cities and international organizations have launched open data portals and implemented open data policies. Many, many more are contemplating joining the fray. What makes this exciting is that some established players (e.g. United States, UK, World Bank) are continue to push forward and will, I suspect, be refining and augmenting their services in the coming months. At the same time there are still a number of laggards (e.g. Canada federally, Southern Europe, Asia) in which mobilizing local communities, engaging with public servants and providing policy support is still the order of the day.

This makes the role of an Open Data Advocate complex. Obviously, helping pull the laggards along is an important task. Alternatively (or in addition) they may need to also be thinking longer term. Where is open data going, what will second and third generation open data portals need to look like (and what policy infrastructure will be needed to support them).

These are two different goals and so either choosing, or balancing, between them will not be easy.

Key Challenges

Some of the key challenges spring quite obviously from that context. But there are also other challenges, I believe to be looming as well. So what do I suspect are the key challenges around open data over the next 1-5 years?

  1. Getting the laggards up and running
  2. Getting governments to use standardized licenses that are truly open (be it the PDDL, CC-0 or one of the other available licenses out there
  3. Cultivating/fostering an eco-system of external data users
  4. Cultivating/fostering an eco-system of internal government user (and vendors) for open data (this is what will really make open data sustainable)
  5. Pushing jurisdictions and vendors towards adopting standard structures for similar types of data (e.g. wouldn’t it be nice if restaurant inspection data from different jurisdictions were structured similarly?)
  6. Raising awareness about abuses of, and the politicization of, data. (e.g. this story about crime data out of New York which has not received nearly enough press)

The Tasks/Leverage Points

There are some basic things that the role will require including:

  1. Overseeing the Working Group on Open Government Data
  2. Managing opengovernmentdata.org
  3. Helping organize the Open Government Data Camp 2011, 2012 and beyond

But what the role will really have to do is figure out the key leverage points that can begin to shift the key challenges listed above in the right direction. The above mentioned tasks may be helpful in doing that… but they may not be. Success is going to be determined but figuring how to shift systems (government, vendor, non-profit, etc…) to advance the cause of open data. This will be no small task.

My sense is that some of these leverage points might include:

  1. Organizing open data hackathons – ideally ones that begin to involve key vendors (both to encourage API development, but also to get them using open data)
  2. Leveraging assets like Civic Commons to get open data policies up on online so that jurisdictions entertaining the issue can copy them
  3. Building open data communities in key countries around the world – particularly in key countries in such as Brasil and India where a combination of solid democratic institutions and a sizable developer community could help trigger changes that will have ramifications beyond their borders (I suspect there are also some key smaller countries – need to think more on that)
  4. I’m sure this list could be enhanced…

Metrics/Deliverables

Obviously resolving the above defined challenges in 1-5 years is probably not realistic. Indeed, resolving many of those issues is probably impossible – it will be a case of ensuring each time we peel back one layer of the onion we are well positioned to tackle the next layer.

Given this, some key metrics by which the Open Knowledge Foundation should evaluate the person in this role might be:

At a high level, possible some metrics might include:

  • Number of open data portals world wide? (number using CKAN?)
  • Number of groups, individuals, cities participating in Opendata hackathons
  • Number of applications/uses of open data
  • Awareness of CKAN and its mission in the public, developer space, government officials, media?
  • Number of government vendors offering open data as part of their solution

More additional deliverables, could include:

  • Running two Global OpenData Hackathons a year?
  • Developing an OKFN consulting arm specializing in open data services/implementation
  • Create an open data implementation policy “in a box” support materials for implementing an open data strategy in government
  • Develop a global network of OKFN chapters to push their local and national governments, share best practices
  • Run opendata bootcamps for public servants and/or activists
  • Create a local open data hackathon in a box kit (to enable local events)
  • Create a local “how to be an open data activist” site
  • Conduct some research on the benefits of open data  to advance the policy debate
  • Create a stronger feedback loop on CKAN’s benefits and weaknesses
  • Create a vehicle to connect VC’s and/or money with open data drive companies and app developers (or at least assess what barriers remain to use open data in business processes).

Okay, I’ll stop there, but if you have thoughts please send them or comment below. Hope this stimulates some thinking among fellow open data geeks.

World Bank Discussion on Open Data – lessons for developers, governments and others

Yesterday the World Bank formally launched its Apps For Development competition and Google announced that in addition to integrating the World Bank’s (large and growing) data catalog into searches, it will now do it in 34 languages.

What is fascinating about this announcement and the recent changes at the bank is it appears to be very serious about open data and even more serious about open development. The repercussions of this shift, especially if the bank starts demanding that its national partners also disclose data, could be significant.

This of course, means there is lots to talk about. So, as part of the overall launch of the competition and in an effort to open up the workings of the World Bank, the organization hosted its first Open Forum in which a panel of guests talked about open development and open data. The bank was kind enough to invite me and so I ducted out of GTEC a pinch early and flew down to DC to meet some of the amazing people behind the world bank’s changes and discuss the future of open data and what it means for open development.

Embedded below is the video of the event.

As a little backgrounder here are some links to the bios of the different panelists and people who cycled through the event.

Our host: Molly Wood of CNET.

Andrew McLaughlin, Deputy Chief Technology Officer, The White House (formerly head of Global Public Policy and Government Affairs for Google) (twitter feed)

Stuart Gill, World Bank expert, Disaster Mitigation and Response for LAC

David Eaves, Open Government Writer and Activist

Rakesh Rajani, Founder, Twaweza, an initiative focused on transparency and accountability in East Africa (twitter)

Aleem Walji, Manager, Innovation Practice, World Bank Institute (twitter)

Rethinking Freedom of Information Requests: from Bugzilla to AccessZilla

Last week I gave a talk at the Conference for Parliamentarians hosted by the Information Commission as part of Right to Know Week.

During the panel I noted that, if we are interested in improving response times for Freedom of Information (FOI) requests (or, in Canada, Access to Information (ATIP) requests) why doesn’t the Office of the Information Commissioner use a bugzilla type software to track requests?

Such a system would have a number of serious advantages, including:

  1. Requests would be public (although the identity of the requester could remain anonymous), this means if numerous people request the same document they could bandwagon onto a single request
  2. Requests would be searchable – this would make it easier to find documents already released and requests already completed
  3. You could track performance in real time – you could see how quickly different ministries, individuals, groups, etc… respond to FOI/ATIP requests, you could even sort performance by keywords, requester or time of the year
  4. You could see who specifically is holding up a request

In short such a system would bring a lot of transparency to the process itself and, I suspect, would provide a powerful incentive for ministries and individuals to improve their performance in responding to requests.

For those unfamiliar with Bugzilla it is an open source software application used by a number of projects to track “bugs” and feature requests in the software. So, for example, if you notice the software has a bug, you register it in Bugzilla, and then, if you are lucky and/or if the bug is really important, so intrepid developer will come along and develop a patch for it. Posted below, for example, is a bug I submitted for Thunderbird, an email client developed by Mozilla. It’s not as intuitive as it could be but you can get the general sense of things: when I submitted the bug (2010-01-09), who developed the patch (David Bienvenu), it’s current status (Fixed), etc…

ATIPPER

Interestingly, an FOI or ATIP request really isn’t that different than a “bug” in a software program. In many ways, bugzilla is just a complex and collaborative “to do” list manager. I could imagine it wouldn’t be that hard to reskin it so that it could be used to manage and monitor access to information requests. Indeed, I suspect there might even be a community of volunteers who would be willing to work with the Office of the Information Commissioner to help make it happen.

Below I’ve done a mock up of what I think revamped Bugzilla, (renamed AccessZilla) might look like. I’m put numbers next to some of the features so that I can explain in detail about them below.

ATIPPER-OIC1

So what are some of the features I’ve included?

1. Status: Now an ATIP request can be marked with a status, these might be as simple as submitted, in process, under review, fixed and verified fixed (meaning the submitter has confirmed they’ve received it). This alone would allow the Information Commissioner, the submitter, and the public to track how long an individual request (or an aggregate of requests) stay in each part of the process.

2.Keywords: Wouldn’t it be nice to search of other FOR/ATIP requests with similar keywords? Perhaps someone has submitted a request for a document that is similar to your own, but not something you knew existed or had thought of… Keywords could be a powerful way to find government documents.

3. Individual accountability: Now you can see who is monitoring the request on behalf of the Office of the information commissioner and who is the ATIP officer within the Ministry. If the rules permitted then potential the public servants involved in the document might have their names attached here as well (or maybe this option will only be available to those who log on as ATIP officers.

4. Logs: You would be able to see the last time the request was modified. This might include getting the documents ready, expressing concern about privacy or confidentiality, or simply asking for clarification about the request.

5. Related requests: Like keywords, but more sophisticated. Why not have the software look at the words and people involved in the request and suggest other, completed requests, that it thinks might similar in type and therefor of interest to the user. Seems obvious.

6. Simple and reusable resolution: Once the ATIP officer has the documentation, they can simply upload it as an attachment to the request. This way not only can the original user quickly download the document, but anyone subsequent user who stumbles upon the request during a search could download the documents. Better still any public servant who has unclassified documents that might relate to the request can simply upload them directly as well.

7. Search: This feels pretty obvious… it would certainly make citizens life much easier and be the basic ante for any government that claims to be interested in transparency and accountability.

8. Visualizing it (not shown): The nice thing about all of these features is that the data coming out of them could be visualized. We could generate realt time charts showing average response time by ministry, list of respondees by speed from slowest to fastest, even something as mundane as most searched keywords. The point being that with visualizations is that a governments performance around transparency and accountability becomes more accessible to the general public.

It may be that there is much better software out there for doing this (like JIRA), I’m definitely open do suggestions. What I like about bugzilla is that it can be hosted, it’s free and its open source. Mostly however, software like this creates an opportunity for the Office of the Information Commissioner in Canada, and access to information managers around the world, to alter the incentives for governments to complete FOI/ATIP requests as well as make it easier for citizens to find out information about their government. It could be a fascinating project to reskin bugzilla (or some other software platform) to do this. Maybe even a Information Commissioners from around the world could pool their funds to sponsor such a reskinning of bugzilla…

The Census weak link: What the Liberals, Bloc & NDP should do

Public and the media condemnation of the government’s decision to end the long form census has been universal (see more fun quotes below). Now the political opposition has started to mobilize. The focus of the opposition has been on the secretive nature of the decision and the failure to consult any stakeholders. While this is problematic I’m not sure it is the most ironic and sensitive point. There is actually a much more juicy Achilles heel in this decision, one that might garner press attention.

In explaining the decision (such as in the quote to the Canadian Press below), Minister Clement has repeatedly claimed that MP offices has received numerous complaints about the long form census form:

“Every MP has had complaints like that so this year we decided to at least try another method that could be a sound method that would beat the issue of concern of degradation of data, and deal with the issue of coercion and too much intrusiveness”

The statement suggests broad base support but, ironically, anecdotal and, in theory, untestable.

The fun thing is, it doesn’t have to be that way. The opposition could bring more (and hard) information to this process and expose how the Conservatives are using a lack of information to at best mislead and at worse, lie, to ordinary Canadians. Better still, if the opposition parties have been organized, they should possess that data.

How is this? Well, I’d like to see the Liberals, Bloc and NDP ask each of their MPs to search and count every email and letter from constituents over the past 4 (or some sensible number) years that involved a complaint about the census.

My suspicion is that there are no more than 100 such letters (and, maybe even a lot less, like 10). You could then pick a couple of issues of your choice on which you have received 1000s of letters and which the conservatives have taken no action and put a simple chart up showing how they react to made up issues, but ignore real priorities of ordinary Canadians.

Of course, maybe there has been some massive, secret letter campaign targeted at Conservative only MPs but I think most Canadians will want to know how many letters the Minister and conservative MPs have actually received (sadly I don’t think you can FOI this) and most will suspect it is about as many as other MPs have.

Let’s show that better data leads to better decisions and that a little transparency can go a long way to exposing those who seek to mislead us. The minister says MPs get lots of letters – lets find out how many. And in comparison to what. Moreover, let’s show that those who seek to restrict the gathering of good information are generally those most inclined to use a lack of data to mislead the public and drive agendas that are not in the public interest. Canadians are sensible people, they want a smart government that makes good decisions. This, much more than secrecy, will rub them the wrong way.

The narrative isn’t as neat – but I suspect it could be turned into something more fun and more impactful.

Other fun articles…

Also, the universal condemnation of the Conservatives decision to end the mandatory long term census continues. My newest favourite is an article in the Globe where Derek Cook, Calgary’s the research and social planner at the City of Calgary and in the riding of the Prime Minister Harpers states that “If we don’t have that data at the neighbourhood level, we’re crippled.” Now, in addition to business groups, ngos, think tanks, cities, university researchers we are also seeing even more local and focused organizations such as those representing French Canadian communities, Inuit people and others starting to call for a reversal as well.