Category Archives: open data

Making StatsCan Data Free: Assessing the Cost

Regular readers of my blog will know that I’ve advocated that StatsCan’s data – and particularly its Census data – should be made open (e.g. free, unlicensed, and downloadable in multiple formats). Presently, despite the fact that Canadian tax dollars pay to collect (a sadly diminishing amount, and quality of,) data, it is not open.

The main defense I hear to why StatsCan’s data should not be free is because the department depends on the revenue the data generates.

So exactly how much revenue are we talking about? Thanks to the help of some former public servants I’ve been able to go over the publicly available numbers. The basic assessment – which I encourage people to verify and challenge – turns out not to be a huge a number.

The most interesting figure in StatsCan’s finances is the revenue it generates from its online database (e.g. data downloaded from its website). So how much revenue is it? Well in 2007/2008, it was $559,000.

That’s it. For $559,000 in lost government revenue Canadians could potentially have unlimited access to the Statscan census database their tax dollars paid to collect and organize. I suspect this is a tiny fraction of the value (and tax revenue) that might be generated by economic activity if this data were free.

Worse, the $559,000 is not profit. From what I can tell it is only revenue. Consequently, it doesn’t factor in collection costs StatsCan has to absorb to run and maintain a checkout system on its website, collect credit card info, bill people, etc… I’m willing to bet almost anything that the cost of these functions either exceed $559,000 a year, or come pretty close. So the net cost of making the data free could end up being a less.

StatsCan makes another $763,000 selling Statistics Canada publications (these are 243 data releases of the 29 major economic indicators StatsCan measures and the 5 census releases it does annually – in short these are non-customized reports). So for $1,422,000 Canadians could get access to both the online data statscan has and the reports the organization generates. This is such laughably (or depressingly) small number it begs the question – why are we debating this? (again this is revenue, not profit, so the cost could be much lower)

Of course, the figure that you’ll often hear cited is $100M in revenue. So what accounts for the roughly 100x difference between the above number and the alleged revenue? Well, in 2007/08 StatsCan did make $103,155,000 but this was from value added (e.g. customized) reports. This is very, very different product than the basic data that is available on its website. My sources tell me this is not related to downloaded data.

I think we should concede that if the entire StatsCan’s database were made open and free it would impact some of this revenue. But this would also be a good thing. Why is this? Let’s break it down:

  1. Increase Capacity and Data Literacy: By making a great deal of data open and free, StatsCan would make it easier for competitors to enter the market place. More companies and individuals could analyze the country’s census and other data, and so too could more “ordinary” Canadians than ever would be able to access the database (again, that their tax dollars paid to create). This might include groups like senior high school and university students, non-profits and everyday citizens who wanted to know more about their country. So yes, Statscan would have more competitors, but the country might also benefit from having a more data literate population (and thus potential consumers).
  2. Increase Accessibility of Canadian Data to Marginalized Groups: An increase in the country’s analysis capacity would drop the price for such work. This would make it cheaper and easier for more marginal groups to benefit from this data – charities, religious groups, NGO’s, community organizations, individuals, etc…
  3. Improve Competitiveness: It would also be good for Canadian competitiveness, companies would have to spend less to understand and sell into the Canadian market. This would lower the cost of doing business in Canada – helpful to consumers and the Canadian economy.
  4. StatsCan would not lose all or even most of its business: Those at StatsCan who fear the organization would be overwhelmed by a more open world should remember, not all the data can be shared. Some data – particularly economic data gathered from companies – is sensitive and confidential. As a result there will be some data that StatsCan retains exclusive access to, and thus a monopoly over analysis. More importantly, I suspect that were Statscan data made open the demand for data analysis would grow, so arguably new capacity might end up being devoted to new demand, not existing demand.
  5. It will Reduce the Cost of Government: Finally, the crazy thing about StatsCan is that it sells its data and services to other Ministries and layers of government. This means that governments are paying people to move tax payer money between government ministries and jurisdictions. This is a needless administrative costs that drives up everybody’s taxes and poorly allocates scarce government resources (especially at the local level). Assuming every town and city in Canada pays $50 – 1000 dollars to access statscan data may not seem like much, but in reality, we are really paying that, plus their and StatsCan’s staff time to manage all these transactions, enforce compliance, etc… all of which is probably, far, far more.

So in summary, the cost to Canada of releasing this data will likely be pretty marginal, while the benefits could be enormous.

At best, if costs half a million dollars in forgone revenue. Given the improved access and enormous benefits, this is a pittance to pay.

At worst, StatsCana would lose maybe 20-30 million – this is a real nightmare scenario that assumes much greater competition in the marketplace (again, a lot of assumptions in this scenario). Of course the improved access to data would lead to economic benefits that would far, far, surpass this lost revenue, so the net benefit for the country would be big, but the cost to StatsCan would be real. Obviously, it would be nice if this decline in revenue was offset by improved funding for StatsCan (something a government that was genuinely concerned about Canadian economic competitiveness would jump at doing). However, given the current struggles Statscan faces on the revenue front (cuts across the board) I could see how a worse case scenario would be nerve wracking to the department’s senior public servants, who are also still reeling from the Long Form Census debacle.

Ultimately, however, I think the worse case scenario is unlikely. Moreover, in either scenario the benefits are significant.

Bonus Material:

Possibly the most disconcerting part of the financial reports on StatsCan on Treasury Board’s website was the stakeholder consultation associated with access to statscan’s database. It claimed that:

Usability and client satisfaction survey were conducted with a sample of clients in early 2005. Declared level of satisfaction with service was very high.

This is stunning. I’ve never talked to anyone who has had a satisfactory experience on StatsCan’s website (in contrast to their phone support – which everyone loves). I refer to the statscan site where the place where what you want is always one click away.

I’m willing to bet a great deal that the consultations were with existing long term customers – the type of people that have experience using the website. My suspicion is that if a broader consultation was conducted with potential users (university students, community groups, people like me and you, etc…) the numbers would tank. I dare you to try to use their website. It is virtually unnavigable.

Indeed, had made its website and data more accessible I suspect it the department would engage Canadians and have more stakeholders. This would have been the single most powerful thing it could have done to protect itself from cuts and decisions like the Long Form fiasco.

I know this post may anger a number of people at Statscan. I’m genuinely sorry. I know the staff work hard, are dedicated and are exceedingly skilled and professional. This type of feedback is never flattering – particularly in public. It is because you are so important to the unity, economy and quality of life in our country that it is imperative we hold you to the highest possible bar – not just in the quality of that data your collect (there you already excel) but in the way you serve and engage Canadians. In this, I hope that you get the support you need and deserve.

Three stories of change from the International Open Data Hackathon

Over the past few weeks people have been in touch with me about what happened in their city during the open data hackathon. I wanted to share some of their stories so that people can see the potential around the event.

Here are a few that really struck me:

If you get a moment a Ton Zijlstra’s blog post about the open data hackathon in Enschede, in the Netherlands. It pretty much sums up everything we wanted to have happen during the hackathon:

  • Data sets released: Because of the hackathon the City of Enshede got motivated and released 25 data sets for participants to work with. This alone made me happy as this was a big part of why we wanted to do the hackathon – get governments to act!
  • Good cross section of participation: Local entrepreneurs, students and civil servants, including a civil servant with an IT background on hand all day to help out and a departmental head dropping by to see what was going on
  • Education: Interested government officials from neighboring cities dropped by to learn more
  • Tangible Outputs: As a result of the hackathon’s efforts two prototypes were built, a map overview of all building permit requests and the underlying plans (wish we had this in Vancouver) and a map overview of local business bankruptcies
  • Connectivity: They had a video session with the groups in Helsinki, Vienna to share lessons about event and show and tell the prototypes.

Meanwhile from Bangalore, I got the email from the local organizer Vasanth B:

We have not found a place to host our app yet. Unfortunate as it may seem. We are hoping to get it up in another 3 days. wanted to thank you for coming up with this novel concept. We all are convinced that open data is crucial and hence we will create a website which will be a one stop place to get the data of our country’s parliament!
I will send you the link of our site soon. Once again thanks to this event, we learned a lot and hope to be part of this in the coming days.
It’s great to see people:
  • Civic Engagement: Here is a group of developers that hadn’t thought much about Open Data but became interested because of the event and have developed a passion for using their skills to help make democratic information more available.
  • Tangible Outcome: They created an app that allows you to see public statements made by the leaders of India’s political parties at the national and state level. (Demo can be seen here)
And in Thailand, Keng organized an amazing hackathon in two weeks. Here one of the big outputs was scraping the Thailand’s Member of House of Representative Website. What was great about this output is:
  • Created Open Data: In many jurisdictions there is little available machine readable open data. The great thing about the work of the Bangkok team is that they now made it possible for others to create applications using data from Thailand’s House of Representatives
  • Learned new skills/tools: After the hackathon KenG sent the creators of Scraperwiki a really nice note explaining how great a tool it was. The fact that a bunch of people got familiar with scraperwiki is itself a big win as each time someone uses it, they create more open data for others to leverage. Indeed, Justin Houk, who participated in the Open Data Hackathon on the other side of the world in Portland Oregon, has written a great blog post explaining why they used scraperwiki.
Finally, in Oxford, Tim Davies has this excellent recap of what occurred at the Hackathon there with a number of great lessons learned. Again, some of what I loved there was:
  • Civic Engagement: As with Enschede, developers mainly worked on things that they thought would make their community better. Hackathons are about getting people involved in and better understanding their community.
  • More tangible outcomes(!): See Tim’s list…
I also got a great email from Iain Emsley who described exactly why Open Data can lead to public engagement.
I started on playing with Arts Council of England funding data from this region for last year but we got so enthused that a few of us downloaded the entire dataset of 5 years worth of funding! Anyhow, just thought I’d ping you with the URL of the stuff that we started playing with and I went off and started redeveloping .

Glad you organised it and looking forward to future days. I’m even thinking of trying to organise a literature hackday now…

Again this is not all the events that happened, there was lots more activity, just some highlights that I read and wanted to share.

To see a list of many of the artifacts produced during the hackathon take a look at the Open Data Hackathon wiki.

An Open Data Inspired Holiday Gift to Montrealers

It turns out that Santa, with the help of some terribly two clever elves over at Montreal Ouvert has created an Open Data inspired present for Montrealers.

What, you must ask could it be?

It’s PatinerMontreal.ca

It’s a genius little website created by two Montreal developers – James McKinney and Dan Mireault – that scrapes the City of Montreal’s data on ice rink status to display the location and condition of all the outdoor ice rinks in the city.

What more could a winter bound montrealer ask for? Well… actually… how about being able to download it as an Android app to use on your smart phone. Yes, you can do that too thanks to another Montreal software developer: Mudar Noufal.

Here’s a screen shot of the slick web version (more on the project below the fold)

Creating this unbelievably useful application was no small feat. It turns out that the City of Montreal publishes the state of the outdoor hockey rinks every day in PDF format. While it is nice that the city puts this information up on the web, sharing it via PDF is probably the most inaccessible way of meeting this goal. To create this site the developers have to “scrape” the data out of these PDF files every day. Creating the software to do this is not only tedious, it can also be frustrating and laborious. In reality, this data was created with tax dollars and is encouraging the use of city assets. Making it difficult to access is unnecessary and counterproductive.

This is because if you can get the data, the things you can create (like PatinerMontreal.ca) can be gorgeous and far superior to anything the city offers. The City’s PDFs conveys a lot of information in a difficult to decipher format – text. Visualizing this information and making it searchable allows the user to quickly see where rinks or located in the city, what types of rinks (skating versus hockey) are located where, and the status of said rinks (newly iced or not).

My hope – and the hope of Montreal Ouvert – is that projects like this show the City of Montreal (and other cities across Canada) the power of getting data out of PDFs and shared in a machine readable format on an open data portal. If Montreal had an Open Data portal (like Vancouver, Nanaimo, Edmonton, Toronto, Ottawa, and others) this application would have been much easier to create and Montrealers would enjoy the benefit of being able to better use the services their tax dollars works so hard to create.

Congratulations to James, Dan and Mudar on such a fantastic project.

Happy Holidays to Montreal Ouvert.

Happy Holidays Montreal. Hope you enjoy (and use) this gift.

Visualizing Firefox Plugins Memory Consumption

A few months ago the Mozilla Labs and the Metrics Team, together with the growing Mozilla Research initiative, launched an Open Data Visualization Competition.

Using data collected from Test Pilot users (people who agreed to share anonymous usage data with Mozilla and test pilot new features) Mozilla asked its community to think of creative visual answers to the question: “How do people use Firefox?”

As an open data geek and Mozilla supporter the temptation to try to do something was too great. So I teamed up with my old data partner Diederik Van Liere and we set out to create a visualization. Our goals were simple:

  • have fun
  • focus on something interesting
  • create something that would be useful to Firefox developers and/or users
  • advance the cause for creating a Firefox open data portal

What follows is the result.

It turns out that – in our minds – the most interesting data set revolved around plugin memory consumption. Sure this sounds boring… but plugins (like Adobe reader, Quicktime or Flash) or are an important part of the browser experience – with them we engage in a larger, richer and more diverse set of content.  Plugins, however, also impact memory consumption and, consequently, browser performance. Indeed, some plugins can really slow down Firefox (or any browser). If consumers had a better idea of how much performance would be impacted they might be more selective about which plugins they download, and developers might be more aggressive in trying to make their plugins more efficient.

Presently, if you run Firefox you can go to the Plugin Check page to see if your plugins are up to date. We thought: Wouldn’t it be great if that page ALSO showed you memory consumption rates? Maybe something like this (note the Memory Consumption column, it doesn’t exist on the real webpage, and you can see a larger version of this image here):

Firefox data visualization v2

Please understand (and we are quite proud of this). All of the data in this mockup is real. Memory consumptions are estimates we derived by analyzing the Test Pilot data.

How, you might ask did we (Diederik) do that?

GEEK OUT EXPLANATION: Well, we (Diederik) built a dataset of about 25,000 different testpilot users and parsed the data to see which plugins were installed and how much memory was consumed around the time of initialization. This data was analyzed using ordinary least squares regression where the dependent variable is memory consumption and the different plugins are the explanatory variables. We only included results that are highly significant.

The following table shows our total results (you can download a bigger version here).

Plugin_memory_consumption_chart v2

Clearly, not all plugins are created equal.

Our point here isn’t that we have created the definitive way of assessing plugin impact on the browser, our point is that creating a solid methodology for doing so is likely witihin Mozilla’s grasp. More importantly, doing this could help improve the browsing experience. Indeed, it would probably be even wiser to do something like this for Add-ons, which is where I’m guessing the real lag time around the browsing experience is created.

Also, with such a small data set we were only able to calculate the memory usage for a limited number of plugins and generally those that are more obscure. Our methodology required having several data points from people who are and who aren’t using a given plugin and so with many popular plugins we didn’t have enough data from people who weren’t using it… a problem however, that would likely be easily solved with access to more data.

Finally, I hope this contest and our submission helps make the case for why Mozilla needs an open data portal. Mozilla collects and incredible amount of data of which it does not have the resources to analyze internally. Making it available to the community would do to data what Mozilla has done to code – enable others create value that could affect the product and help advance the open web. I had a great meeting earlier this week with a number of the Mozilla people about this issue, I hope that we can continue to make progress.

An Open Letter on Open Government to the Access to Information, Privacy & Ethics Parliamentary Committee

The other week I received an invitation from the Canadian Standing Parliamentary Committee on Access to Information, Privacy & Ethics to come and testify about open government and open data on February 1st.

The Committee has talked a great deal about its efforts to engage in a study of open government and since February 1st is quite a bit away and I’d like to be helpful before my testimony, I thought I draft up some thoughts and suggestion for the committee’s strategy. I know these are unsolicited but I hope they are helpful and, if not, that they at least spark some helpful thoughts.

1. Establish a common understanding of the current state of affairs

First off, the biggest risk at the moment is that the Committee’s work might actually slow down efforts of the government to launch an open data strategy. The Committee’s work, and the drafting of its report, is bound to take several months, it would be a shame if the government were to hold back launching any initiatives in anticipation of this report.

Consequently, my hope is that the committee, at is earliest possible convenience, request to speak to the Chief Information Officer of the Government of Canada to get an update regarding the current status of any open government and open data initiatives, should they exist. This would a) create a common understanding regarding the current state of affairs for both committee members and witnesses; b) allow subsequent testimony and recommendations to take into consideration the work already done and c) allow the committee to structure its work so as to not slow down any current efforts that might be already underway.

2. Transform the Committee into a Government 2.0 Taskforce – similar to the Australian effort

Frankly, my favourite approach in this space has been the British. Two Government’s, one Labour, one Conservative have aggressive pursued an open data and open government strategy. This, would be my hope for Canada. However, it does not appear that is is presently the case. So, another model should be adopted. Fortunately, such a model exists.

Last year, under the leadership of Nicholas Gruen, the Australian government launched a Government 2.0 taskforce on which I had the pleasure of serving on the International Reference Group. The Australian Taskforce was non-partisan and was made up of policy and technical experts and entrepreneurs from government, business, academia, and cultural institutions. More importantly, the overwhelming majority of its recommendations were adopted.

To replicate its success in Canada I believe the Committee should copy the best parts of the Australian taskforce. The topic of Canadians access to their government is of central importance to all Canadians – to non-profits, to business interests, to public servants and, of course, to everyday citizens. Rather than non-partisan, I would suggest that a Canadian taskforce should be pan-partisan – which the Committee already is. However, like the Australian Taskforce it should include a number of policy and technical experts from outside government. This fill committee would this represent both a political cross-section and substantive knowledge in the emerging field of government 2.0. It could thus, as a whole, effectively and quickly draft recommendations to Parliament.

Best of all, because of step #1, this work could proceed in parallel to any projects (if any) already initiated by the government and possibly even inform such work by providing interim updates.

I concede such an approach may be too radical, but I hope it is at least a starting point for an interesting approach.

3. Lead by Example

There is one arena where politicians need not wait on the government to make plans: Parliament itself. Over the past year, while in conversations with the Parliamentary IT staff as well as the Speaker of the House, I have worked to have Parliament make more data about its own operations open. Starting in January, the Parliamentary website will begin releasing the Hansard in XML – this will make it much easier for software developers like the creators of Openparliament.ca as and howdtheyvote.ca to run their sites and for students, researchers and reporters to search and analyze our country’s most important public discussions. In short, by making the Hansard more accessible the Speaker and his IT staff are making parliament more accessible. But this is only the beginning of what parliamentarians could do to make for a truly Open Parliament. The House and Senate’s schedules and agendas, along with committee calendars should all be open. So to should both chambers seating arrangement. Member’s photos and bios should be shared with an unrestricted license as should the videos of parliament.

Leadership in this space would send a powerful message to both the government and the public service that Canada’s politicians are serious about making government more open and accessible to those who elect it. In addition, it could also influence provincial legislature’s and even municipal governments, prompting them to do the same and so enhance our democracy at every level.

4. Finally, understand your task: You are creating a Knowledge Government for a Knowledge Society

One reason I advise the Committee to take on external members is because, laudably, many admit this topic is new to them. But I also want the committee members to understand the gravity of their task. Open Government, Open Data and/or Government 2.0 are important first steps in a much larger project.

What you are really wrestling with here is what government is going to look like in an knowledge economy and a knowledge society. How is going to function with knowledge workers as employees? And, most importantly, how is it going to engage with knowledge citizens, many of whom can and want to make real contributions beyond the taxes they pay and don’t need government to self-organize?

In short, what is a knowledge based government going to look like?

At the centre of that question is how we manage and share information. The basic building block of a knowledge driven society.

Look around, and you can see how the digital world is transforming how we do everything. Few of us can imagine living today without access to the internet and the abundance of information it brings to us. Indeed, we have already become so used to the internet we forget how much it has radically changed whole swaths of our life and economy from the travel and music industry to the post to political fund-raising and to journalism.

If today our government still broadly looks and feels like an institution shaped by the printing press it is because, well it is. Deputy Ministers and Ministers still receive giant briefing binders filled with paper. This is a reflection of how we deal within information and knowledge in government, we move it around (for good reasons) in siloes, operating as though networks, advance search, and other innovations don’t exist (even though they already do).

How our government deals with information is at the heart of your task. I’m not saying you have to re-invent government or dismantle all the silos and ministries. Quite the contrary, I believe small changes can be made that will yield significant benefits, efficiencies and savings while enhancing our democracy. But you will be confronting decades, if not centuries of tradition, culture and process in an institution that is about to go through the biggest change since the invention of the printing press. You don’t have to do it all, but even some small first steps will not come easily. I share this because I want you going into the task with eyes wide open.

At the very least we aren’t going first, our cousins both across the Atlantic, the Pacific and our southern border have already taken the plunge. But this should add urgency to our task. We cannot afford to stand by while others renew their democratic institutions while simultaneously enhancing an emerging and critical pillar of a new knowledge economy and knowledge society.

Opendataday & the International Hackathon: What happened. What happens next.

I’m floored.

As many of you know, 5 weeks I had a conversation with a group of open data geeks (like me, likely like you) in Ottawa and Sao Paulo and we agreed to see if we could prompt an international opendata hackathon. At the time we thought there would be our three cities and maybe three of four more. At no point did we think that there would be 1000s of people in over 73 cities on 5 continents who would dedicate the time and energy to helping foster both a local and international community of open data hackers, advocates and citizens. Nor did we know that the wonderful people like those with Random Hacks of Kindness would embrace us and help make this event such a success.

opendataday-1024x515

All of this of course was the results of 100s of people in communities all over the world, working on their own, hustling to set things up and to get people engaged. If you participated, as an organizer, and a hacker, as a gardener of the wiki, or as someone who just wanted to help – congratulations. We are amazed. We hope you are amazed.

odhd-map

If you are out there I’ve a few thoughts on what we’d like to do right away:

  1. Congratulate yourself.
  2. People have only just begun to share the cool work they started. I’m hoping that more of you will share it so that everyone can be inspired by your work. I’m also hoping that these projects will continue to evolve.
  3. Let us know who you are (if you are comfortable with that). A number of you have told me you‘d like to do this again. Part of what made Saturday amazing was how much happened without any of us having to connect directly. That is the power of the internet. And keeping these events simple and loosely joined will always be a goal for us, but I know I’d like to thank more of you personally and be able to connect more so as to make communicating easier.
  4. Finally, we are thinking that another event will be fun to do is something like 6 months. But in the meantime we hope that you, like us, will try to keep the flame burning in your city by hosting the occasional local event. I know I will be endeavoring to do so in Vancouver.

Longer term:

5. I hope we can develop tools and resources to enable participants to engage with politicians and public servants on the importance of open data. The projects we hack on are powerful examples of what can be, but we also need to become more effective at explaining why open data matters in a language everyone understands. I’m hoping we’ll have resources to help us with this important task.

In the meantime, we’ll be figuring out what to do next. We’d love your help, to hear your thoughts and frustrations and your ideas. Please reach out.

Dave, Edward, Mary Beth, Daniel, Daniela and Pedro

Some favourite shots:

Screen-shot-2010-12-05-at-11.07.46-PM

Screen-shot-2010-12-05-at-11.08.45-PM

 

Vancouver International Open Data Hackathon Event Agenda/Invite

Tomorrow is December 4th. The International Open Data Hackathon will be taking place around the world. Here in Vancouver, we’ll be contributing as well.
Here are some details:
Goals/Important points:
  • This is about having fun and working on something that makes you feel good. If, at any point you aren’t feeling that… then start doing something that does or come talk to me. :)
  • Our main goal is to create artifacts that will help strengthen our democracy, be fun to use, or just make life a little better – we want more open data, let’s show the world why it matters.
  • Our other goal is to build community, both here in Vancouver, and around the world, so let’s help one another, both in city, and those elsewhere…
  • Remember, this is not just for programmers. Any project will need a variety of skills.
  • If you don’t think you can do anything helpful, trust me, that is not the case.
Location:
  • W2/Storyeum 151 W. Cordova (map) in Vancouver
  • Phone: 604-689-9896
Schedule:
  • 9:30-10am People can start arriving any time after 9:30am
  • 10:00-10:30am We’ll be starting at 10am. We’ll begin with brief introductions and give those with ideas an opportunity to share them
  • 10:30 Sort into teams. If you haven’t already chosen a project to work on for the day… now is the time.
  • 10:30-12:00pm Hacking. Lots here to do for designers, developers, citizens to get materials organized, write copy or code, etc…
  • 12:00pm Check in. 5 minutes for teams to share progress, challenges, ask for suggestions
  • 12:30-3:30pm More hacking goodness
  • 3:30 project update/presentations

Vancouver Hack Space will be opening its doors after the hackathon for people who want to keep hacking over there. Great people at VHS so it should be good…

What to bring & expect:
  • A laptop.  If you absolutely can’t bring a laptop, please come anyway, there will be things to do.
  • We have a number of ideas that can be worked on. If you have one… great! If you are looking for cool people to work with… you’re coming to the right place.
  • Ideas that no one wants to work on will be removed.
  • We will ask people to vote with their feet, gathering into self forming teams for each project.

Wikileaks and the coming conflict between closed and open

I’ve been thinking about wikileaks ever since the story broke. Most of the stories – like those written by good friends like Taylor Owen and Scott Gilmore are pieces very much worth reading but I think miss the point about wikileaks and/or assess it on their own terms and thus fail to understand what wikileaks is actually about and what it is trying to do. We need to be clear in our understanding, and thus the choices we are about to confront.

However, before you read anything I write there are smarter people out there – two in particular – who have said things that I’m not reading anywhere else. The first is Jay Rosen (key excerpt below) whose 15 minutes Pressthink late night video on the subject is brilliant and the second is by zunguzungu piece Julian Assange and the Computer Conspiracy; “To destroy this invisible government” (key excerpt further below) is a cool and calculated dissection of wikileaks goals and its intentions. I’ve some thoughts below, but these two pieces are, in my mind, the most important things you can read on the subject and strongly inform my own piece (much, much further below). I know that this is all very long, and that many of you won’t have the patience, but I hope that what I’ve written and shared below is compelling enough to hold your attention, I certainly think it is important enough.

Jay Rosen:

While we have what purports to be a “watchdog press” we also have, laid out in front of us, the clear record of the watchdog press’s failure to do what is says it can do, which is to provide a check on power when it tries to conceal its deeds and its purpose. So I think it is a mistake to reckon with Wikileaks without including in the frame the spectacular failures of the watchdog press over the last 10, 20, 40 years, but especially recently. And so, without this legitimacy crisis in mainstream American journalism, the leakers might not be so inclined to trust Julian Assange and a shadowy organization like Wikileaks. When the United States is able to go to war behind a phony case, when something like that happens and the Congress is fooled and a fake case is presented to the United Nations and war follows and 100,000s of people die and the stated rationale turns out to be false, the legitimacy crisis extends from the Bush government itself to the American state as a whole and the American press and the international system because all of them failed at one of the most important things that government by consent can do: which is reason giving. I think these kind of huge cataclysmic events within the legitimacy regime lie in the background of the Wikileaks case, because if wasn’t for those things Wikileaks wouldn’t have the supporters it has, the leakers wouldn’t collaborate the way that they do and the moral force behind exposing what this government is doing just wouldn’t be there.

This is one of the things that makes it hard for our journalists to grapple with Wikileaks. On the one hand they are getting amazing revelations. I mean the diplomatic cables tell stories of what it is like to be inside the government and inside international diplomacy that anyone who tries to understand government would want to know. And so it is easy to understand why the big news organizations like the New York Times and The Guardian are collaborating with Wikileaks. On the other hand they are very nervous about it because it doesn’t obey the laws of the state and it isn’t a creature of a given nation and it is inserting itself in-between the sources and the press. But I think the main reason why Wikileaks causes so much insecurity with our journalists is because they haven’t fully faced the fact that the watchdog press they treasure so much died under George W. Bush. It failed. And instead of rushing to analyze this failure and prevent it from happening ever again – instead of a truth and reconciliation commission-style effort that could look at “how did this happen” – mostly what our journalists did, with a few exceptions, is they moved on to the next story. The watchdog press died. And what we have is Wikileaks instead. Is that good or is that bad? I don’t know, because I’m still trying to understand what it is.

Zunguzungu:

But, to summarize, he (Assange) begins by describing a state like the US as essentially an authoritarian conspiracy, and then reasons that the practical strategy for combating that conspiracy is to degrade its ability to conspire, to hinder its ability to “think” as a conspiratorial mind. The metaphor of a computing network is mostly implicit, but utterly crucial: he seeks to oppose the power of the state by treating it like a computer and tossing sand in its diodes…

…The more secretive or unjust an organization is, the more leaks induce fear and paranoia in its leadership and planning coterie. This must result in minimization of efficient internal communications mechanisms (an increase in cognitive “secrecy tax”) and consequent system-wide cognitive decline resulting in decreased ability to hold onto power as the environment demands adaption. Hence in a world where leaking is easy, secretive or unjust systems are nonlinearly hit relative to open, just systems. Since unjust systems, by their nature induce opponents, and in many places barely have the upper hand, mass leaking leaves them exquisitely vulnerable to those who seek to replace them with more open forms of governance.

– zunguzungu

Almost all the media about wikileaks has, to date, focused on the revelations about what our government actually thinks versus what it states publicly. The bigger the gap between internal truth and external positions, the bigger the story.

This is, of course, interesting stuff. But less discussed and more interesting is our collective reaction to wikileaks. Wikileaks is drawing a line, exposing a fissure in the open community between those who believe in overturning current “system(s)” (government and international) and those who believe that the current system can function but simply needs greater transparency and reform.

This is why placing pieces like Taylor Owen and Scott Gilmore‘s against zunguzungu’s is so interesting. Ultimately both Owen and Gilmore believe in the core of the current system – Scott explicitly so, arguing how secrecy in the current system allows for human right injustices to be tackled. Implicit in this, of course, is the message that this is how they should be tackled. Consequently they both see wikileaks as a failure as they (correctly) argue that its radical transparency will lead to a more closed and ineffective governments. Assange would likely counter that Scott’s effort address systems and not cause and may even reinforce the international structures that help foster hunan rights abuses. Consequently Assange’s core value of transparency, which at a basic level Owen and Gilmore would normally identify with, becomes a problem.

This is interesting. Owen and Scott believe in reform, they want the world to be a better place and fight (hard) to make it so. I love them both for it. But they aren’t up for a complete assault on the world’s core operating rules and structures. In a way this ultimately groups them (and possibly me – this is not a critique of Scott and Taylor whose concerns I think are well founded) on the same side of a dividing line as people like Tom Flanagan (the former adviser to the Canadian Prime Minister who half-jokingly called for Assange to be assassinated) and Joe Liberman (who called on companies that host material related to wikileaks to sever their ties with them). I want to be clear, they do not believe Assange should be assassinated but they (and possibly myself) do seem to agree that his tactics are a direct threat to the functioning of system that I think they are arguing needs to be reformed but preserved – and so see wikileaks as counterproductive.

My point here is that I want to make explicit the choices wikileaks is forcing us to make. Status quo with incremental non-structural reform versus whole hog structural change. Owen and Gilmore can label wikileaks a failure but in accepting that analysis we have to recognize that they view it from a position that believes in incremental reform. This means you believe in some other vehicle. And here, I think we have some tough questions to ask ourselves. What indeed is that vehicle?

This is why I think Jay Rosen’s piece is so damn important. One of the key ingredients for change has been the existence of the “watchdog” press. But, as he puts it (repeated from above):

I think it is a mistake to reckon with Wikileaks without including in the frame the spectacular failures of the watchdog press over the last 10, 20, 40 years, but especially recently. And so, without this legitimacy crisis in mainstream American journalism, the leakers might not be so inclined to trust Julian Assange and a shadowy organization like Wikileaks. When the United States is able to go to war behind a phony case, when something like that happens and the Congress is fooled and a fake case is presented to the United Nations and war follows and 100,000s of people die and the stated rationale turns out to be false, the legitimacy crisis extends from the Bush government itself to the American state as a whole and the American press and the international system because all of them failed at one of the most important things that government by consent can do: which is reason giving.

the logical conclusion of Rosen’s thesis is a direct challenge to those of us who are privileged enough to benefit from the current system. As ugly and imperfect as the current system may be Liberman, Flanagan, Owen and Gilmore and, to be explicit, myself, benefit from that system. We benefit from the status quo. Significantly. Dismantling the world we know carries with it significant risks, both for global stability, but also personally. So if we believe that Assange has the wrong strategy and tactics we need to make the case, both to ourselves, to his supporters, to those who leak to wiki leaks and to those on the short end of the stick in the international system about how it is the reform will work and how it is that secrecy and power will be managed for the public good.

In this regard the release of wikileak documents is not a terrorist event, but it is as much an attack on the international system as 9/11 was. It is a clear effort to destabilize and paralyze the international system. It also comes at a time when confidence in our institutions is sliding – indeed Rosen argues that this eroding confidence feeds wikileaks.

So what matters is how we react. To carry forward (the dangerous) 9/11 analogy, we cannot repeat the mistakes of the Bush administration. Then our response corrupted the very system we sought to defend, further eroded the confidence in institutions that needed support and enhanced our enemies – we attacked human rights, civil liberties, freedom of speech and prosecuted a war that killed 100,000s of innocent lives on the premise of manufactured evidence.

Consequently, our response to the current crises can’t be to close up governments and increase secrecy. This will strengthen the hands of those who run wikileaks and cause more public servants and citizens to fear the institutions wikileaks and look for alternatives… many of whoe will side with wikileaks and help imped the capacity of the most important institution in our society to respond to everyday challenges.

As a believer in open government and open data the only working option to us to do the opposite. To continue to open up these institutions as the only acceptable and viable path to making them more credible. This is not to say that ALL information should be made open. Any institution needs some private place to debate ideas and test unpopular theses. But at the moment our governments – more through design and evolution than conspiracy – enjoy far more privacy and secrecy than the need. Having a real and meaningful debate about how to change that is our best response. In my country, I don’t see that debate happening. In the United States, I see it moving forward, but now it has more urgency. Needless to say, I think all of this gives new weigh to my own testimony I’ll be making before the parliamentary Standing Committee on Access to Information, Privacy and Ethics.

I still hope the emerging conflict between open and closed can be won without having to resort to the types of tactics adopted by wikileaks. But for those of use who believe it, we had better start making the case persuasively. The responses of people like Flanagan and Liberman remind me of Bush after 9/11 “you are either with us, or with the terrorists.” Whether intentionally or unintentionally, an analogous response will create a world in which power and information are further removed from the public and will lead to the type of destabilizing change Assange wants.

I’m bound to write more on this – especially around wikileaks, open data and transparency that I think some authors unhelpfully conflate but this post is already long enough and I’m sure most people haven’t even reached a place where they’ll be reading this.

Opening up parliament and getting government IT right

Last week I received two invitations to present.

The first was an invitation to present to the Parliamentary Standing Committee on Access to Information, Privacy and Ethics. They are preparing a report on Open Government and would like me to make a short presentation and then answer questions for a couple of hours. This is a ways out but obviously I’m treating it with a significant amount of seriousness – so if you have thoughts or comments on things you think I should share, please feel free to ping me or comment below.

(Speaking of parliament… as an aside, I want again to let developers there know that through some engagement I’ve been having with the parliamentary IT staff they’ve informed me they will be releasing a number of data sets in January including the Hansard.)

Second is, next week, I’ll be at the United Nations as part of the Expert Group Meeting on the 2012 e-Government Survey: Towards a More Citizen-Centric Approach. My main goal here is to stop getting governments to compare themselves to one another on how “successful” they are in delivering services and information online. With a few notable exceptions, most government websites are at best functional at worst, unnavigable.  Consequently, comparing themselves to one another allows them to feel like all is okay, when really they are collectively trapped in a world of design mediocrity.

Yes, they aren’t pretty words, but someone has to say them.

So any thoughts on this subject are welcome as well.

More soon on the hackathon and the census.

International Open Data Hackathon – 63 cities, 25 countries, 5 continents

and counting. Never could any of us have imagined that there would be so many stepping forward to organize an event in their cities.

The clear implication is that Open Data matters. To a lot of people.

To a lot of us.

If you are in the media, a politician or the civil service: pay attention. There are a growing number of people – not just computer programmers and hackers, but ordinary citizens – who’ve come to love and want to help build sites and applications like fixmystreet, wheredoesmymoneygo, emitter.ca or datamasher.

If you are planning to participate in a hackathon – I hope you’ll read the next part (and help continue grow the wiki).

I think, for the day – December 4th – we all really have three shared goals.

1. Have fun

2. Help foster local supportive and diverse communities of people who advocate for open data

3. Help raise awareness of open data, why it matters, by building sites and applications

I’m confident that the first two will happen, but as I said in an earlier post… it is important that we have artifacts at the end of the day to share with the world.

In pursuit of that goal, I continue to believe that one of the easiest things we can do is localize cool projects that have happened in other jursidictions. For example, a team in Bangalore, India as well as a team in Vancouver & Victoria, Canada are contemplating porting openparliament.ca to their respective jurisdictions. It’s a great way to get a huge win and a new, useful, site up and running in a (relatively) short period of time.

I write this because I’m thinking there must be tons of interesting and engaging open data applications out there. If you run such an application… (I’m especially looking at you Sunlight Foundation, Open Knowledge Foundation, MySociety & others…) and you think people in other jurisdictions might want to localize them for their country, state or city… then I’d like you to consider doing the following:

Post to the Apps page of the wiki:

  • the project name,
  • link to the source code repository,
  • any documentation,
  • the various tasks you think will be involved in localizing it
  • things that non-coders can do to advance the project (like research, documentation, graphics, copy text for websites, etc…)
  • and some (very) rough senses of scope and timelines

(Note, I’m hoping to throw a template up shortly, but sadly, right now, I’m hoping on a red-eye flight so can’t do it… with luck tomorrow sometime I’ll delete this text and have added an example like openparliament.ca. For now Victoria and Vancouver have the beginnings of what I’m thinking of on their wiki pages)

Nothing would be cooler than having open data apps ported around world, helping spread citizen engagement, democratic accountability and fun with them.

I know there are some emails flying around about connecting cities for demos as suggested on the opendata hackathon website. Hope to have more on that soon as well.

Also, if you do think that media or local officials will attend and you’d like to brief them on opendata, I have some people at the world bank who’ve been helping launch and expand their open data portal who might be willing to help engage and explain why it is important to such people. Could be nice to have the additional help. Up to you. But feel free to let me know if there is interest.

Finally, if you are running a hackathon, please reach out and say hi. I’d love to hear from you.

Excited.

Very, very excited.