Yearly Archives: 2010

Three stories of change from the International Open Data Hackathon

Over the past few weeks people have been in touch with me about what happened in their city during the open data hackathon. I wanted to share some of their stories so that people can see the potential around the event.

Here are a few that really struck me:

If you get a moment a Ton Zijlstra’s blog post about the open data hackathon in Enschede, in the Netherlands. It pretty much sums up everything we wanted to have happen during the hackathon:

  • Data sets released: Because of the hackathon the City of Enshede got motivated and released 25 data sets for participants to work with. This alone made me happy as this was a big part of why we wanted to do the hackathon – get governments to act!
  • Good cross section of participation: Local entrepreneurs, students and civil servants, including a civil servant with an IT background on hand all day to help out and a departmental head dropping by to see what was going on
  • Education: Interested government officials from neighboring cities dropped by to learn more
  • Tangible Outputs: As a result of the hackathon’s efforts two prototypes were built, a map overview of all building permit requests and the underlying plans (wish we had this in Vancouver) and a map overview of local business bankruptcies
  • Connectivity: They had a video session with the groups in Helsinki, Vienna to share lessons about event and show and tell the prototypes.

Meanwhile from Bangalore, I got the email from the local organizer Vasanth B:

We have not found a place to host our app yet. Unfortunate as it may seem. We are hoping to get it up in another 3 days. wanted to thank you for coming up with this novel concept. We all are convinced that open data is crucial and hence we will create a website which will be a one stop place to get the data of our country’s parliament!
I will send you the link of our site soon. Once again thanks to this event, we learned a lot and hope to be part of this in the coming days.
It’s great to see people:
  • Civic Engagement: Here is a group of developers that hadn’t thought much about Open Data but became interested because of the event and have developed a passion for using their skills to help make democratic information more available.
  • Tangible Outcome: They created an app that allows you to see public statements made by the leaders of India’s political parties at the national and state level. (Demo can be seen here)
And in Thailand, Keng organized an amazing hackathon in two weeks. Here one of the big outputs was scraping the Thailand’s Member of House of Representative Website. What was great about this output is:
  • Created Open Data: In many jurisdictions there is little available machine readable open data. The great thing about the work of the Bangkok team is that they now made it possible for others to create applications using data from Thailand’s House of Representatives
  • Learned new skills/tools: After the hackathon KenG sent the creators of Scraperwiki a really nice note explaining how great a tool it was. The fact that a bunch of people got familiar with scraperwiki is itself a big win as each time someone uses it, they create more open data for others to leverage. Indeed, Justin Houk, who participated in the Open Data Hackathon on the other side of the world in Portland Oregon, has written a great blog post explaining why they used scraperwiki.
Finally, in Oxford, Tim Davies has this excellent recap of what occurred at the Hackathon there with a number of great lessons learned. Again, some of what I loved there was:
  • Civic Engagement: As with Enschede, developers mainly worked on things that they thought would make their community better. Hackathons are about getting people involved in and better understanding their community.
  • More tangible outcomes(!): See Tim’s list…
I also got a great email from Iain Emsley who described exactly why Open Data can lead to public engagement.
I started on playing with Arts Council of England funding data from this region for last year but we got so enthused that a few of us downloaded the entire dataset of 5 years worth of funding! Anyhow, just thought I’d ping you with the URL of the stuff that we started playing with and I went off and started redeveloping .

Glad you organised it and looking forward to future days. I’m even thinking of trying to organise a literature hackday now…

Again this is not all the events that happened, there was lots more activity, just some highlights that I read and wanted to share.

To see a list of many of the artifacts produced during the hackathon take a look at the Open Data Hackathon wiki.

Some theories on why Canadians are the #1 user of YouTube (it's not all good)

In theory I’m on break – trying to recharge my batteries, summit mount inbox zero and finish off a couple of papers I owe various good people – but a few people have sent me links to this story (same content here at the CBC), about how Canadians are embrace the web like few others citizens of the world.

Naturally I’m thrilled and unsurprised. Canadians live in a large country and connectivity has always been something that has driven us. Indeed the country as we know it only exists because of a deal on connectivity – my own province of British Columbia agreed to enter the Dominion only if a transcontinental railway was built to connect it with the rest of the emerging country. Connectivity is in our blood.

There is, however, I suspect another reason why Canadians have taken to the web and it has to do with our monopolies and content regulation.

The article notes that Canada is the number one viewer of YouTube videos:

“In Canada, YouTube per capita consumption of video is No. 1 in the world, it’s just absolutely crazy in terms of how passionate Canadians are about YouTube,” said Chris O’Neill, Canada’s country director for Google.

I wonder, however, if this is because of Canada’s proximity to and familiarity with American created content, but our limited access to seeing said content. The CRTC restricts Canadians access to US channels (and as a result, TV shows). Consequently, much like I argued that the continued success of Blockbuster in Canada is not a sign of effective corporate management but poor innovation strategy and telecommunication regulation Canadians may be flooding to YouTube because they can’t access the content they want through more traditional channels.

If true (and I concede I don’t know what Canadians are watching on YouTube) then on the brightside, this is good news for Canadian consumers are able to get what they want access to, regardless of how the government tries to shape their tastes. Indeed, I suspect that American content isn’t the only thing driving YouTube traffic, as a country of immigrants I’m sure that new (and longstanding) Canadians of a range of backgrounds use YouTube to stay on top of culture, shows and other content from their countries of origin. If all this is helping Canadians become more web savvy and appreciative of the benefits of an open web – then all the better!

On the flip side, this could be a sign that a whole series of Canadian companies (and the jobs they create) are imperiled because they refuse to innovate as quickly as Canadians would like. This isn’t a reason to preserve them, but it is a reason for us to start demanding more from the executives of these companies.

An Open Data Inspired Holiday Gift to Montrealers

It turns out that Santa, with the help of some terribly two clever elves over at Montreal Ouvert has created an Open Data inspired present for Montrealers.

What, you must ask could it be?

It’s PatinerMontreal.ca

It’s a genius little website created by two Montreal developers – James McKinney and Dan Mireault – that scrapes the City of Montreal’s data on ice rink status to display the location and condition of all the outdoor ice rinks in the city.

What more could a winter bound montrealer ask for? Well… actually… how about being able to download it as an Android app to use on your smart phone. Yes, you can do that too thanks to another Montreal software developer: Mudar Noufal.

Here’s a screen shot of the slick web version (more on the project below the fold)

Creating this unbelievably useful application was no small feat. It turns out that the City of Montreal publishes the state of the outdoor hockey rinks every day in PDF format. While it is nice that the city puts this information up on the web, sharing it via PDF is probably the most inaccessible way of meeting this goal. To create this site the developers have to “scrape” the data out of these PDF files every day. Creating the software to do this is not only tedious, it can also be frustrating and laborious. In reality, this data was created with tax dollars and is encouraging the use of city assets. Making it difficult to access is unnecessary and counterproductive.

This is because if you can get the data, the things you can create (like PatinerMontreal.ca) can be gorgeous and far superior to anything the city offers. The City’s PDFs conveys a lot of information in a difficult to decipher format – text. Visualizing this information and making it searchable allows the user to quickly see where rinks or located in the city, what types of rinks (skating versus hockey) are located where, and the status of said rinks (newly iced or not).

My hope – and the hope of Montreal Ouvert – is that projects like this show the City of Montreal (and other cities across Canada) the power of getting data out of PDFs and shared in a machine readable format on an open data portal. If Montreal had an Open Data portal (like Vancouver, Nanaimo, Edmonton, Toronto, Ottawa, and others) this application would have been much easier to create and Montrealers would enjoy the benefit of being able to better use the services their tax dollars works so hard to create.

Congratulations to James, Dan and Mudar on such a fantastic project.

Happy Holidays to Montreal Ouvert.

Happy Holidays Montreal. Hope you enjoy (and use) this gift.

The False choice: Bilingualism vs. Open Government (and accountability)

Last week a disturbing headline crossed my computer screen:

B.C. RCMP zaps old news releases from its website

2,500 releases deleted because they weren’t translated into French

1) The worst of all possible outcomes

This is a terrible outcome for accountability and open government. When we erase history we diminish accountability and erode our capacity to learn. As of today, Canadians have a poorer sense of what the RCMP has stood for, what it has claimed and what it has tried to do in British Columbia.

Consider this. The Vancouver Airport is a bilingual designated detachment. As of today, all press releases that were not translated were pulled down. This means that any press release related to the national scandal that erupted after Robert Dziekański – the polish immigrant who was tasered five times by the (RCMP) – is now no longer online. Given the shockingly poor performance the RCMP had in managing (and telling the truth about) this issue – this concerns me.

Indeed, I can’t think that anyone thinks this is a good idea.

The BC RCMP does not appear to think it is a good idea. Consider their press officer’s line: “We didn’t have a choice, we weren’t compliant.”

I don’t think there are any BC residents who believe they are better served by this policy.

Nor do I think my fellow francophone citizens believe they are better served by this decision. Now no one – either francophone or anglophone can find these press releases online. (More on this below)

I would be appalled if a similar outcome occurred in Quebec or a francophone community in Manitoba. If the RCMP pulled down all French press releases because they didn’t happen to have English translations, I’d be outraged – even if I didn’t speak French.

That’s because the one thing worse than not having the document in both official languages, is not having access to the document at all. (And having it hidden in some binder in a barracks that I have to call or visit doesn’t event hint of being accessible in the 21st century).

Indeed, I’m willing to bet almost anything that Graham Fraser, the Official Languages Commissioner – who is himself a former journalist – would be deeply troubled by this decision.

2) Guided by Yesterday, Not Preparing for Tomorrow

Of course, what should really anger the Official Languages Commissioner is an attempt to pit open and accountable government against bilingualism. This is a false choice.

I suspect that the current narrative in government is that translating these documents is too expensive. If one relies on government translators, this is probably true. The point is, we no longer have to.

My friend and colleague Luke C. pinged me after I tweeted this story saying “I’d help them automate translating those news releases into french using myGengo. Would be easy.”

Yes, mygengo would make it cheap at 5 cents a word (or 15 if you really want to overkill it). But even smarter would be to approach Google. Google translate – especially between French and English – has become shockingly good. Perfect… no. Of course, this is what the smart and practical people on the ground at the RCMP were doing until the higher ups got scared by a French CBC story that was critical of the practice. A practice that was ended even though it did not violate any policies.

The problem is there isn’t going to be more money to do translation – not in a world of multi-billion dollar deficits and in a province that boasts 63,000 french speakers. But Google translate? It is going to keep getting better and better. Indeed, the more it translates, the better it gets. If the RCMP (or Canadian government) started putting more documents through Google translate and correcting them it would become still more accurate. The best part is… it’s free. I’m willing to bet that if you ran all 2500 of the press releases through Google translate right now… 99% of them would come out legible and of a standard that would be good enough to share. (again, not perfect, but serviceable). Perhaps the CBC won’t be perfectly happy. But I’m not sure the current outcome makes them happy either. And at least we’ll be building a future in which they will be happy tomorrow.

The point here is that this decision reaffirms a false binary: one based on a 20th century assumptions where translations were expensive and laborious. It holds us back and makes our government less effective and more expensive. But worse, it ignores an option that embraces a world of possibilities – the reality of tomorrow. By continuing to automatically translate these documents today we’d continue to learn how to use and integrate this technology now, and push it to get better, faster. Such a choice would serve the interests of both open and accountable governments as well as bilingualism.

Sadly, no one at the head office of the RCMP – or in the federal government – appears to have that vision. So today we are a little more language, information and government poor.

Three asides:

1) I find it fascinating that the media can get mailed a press release that isn’t translated but the public is not allowed to access it on a website until it is – this is a really interesting form of discrimination, one that supports a specific business model and has zero grounding in the law, and indeed may even be illegal given that the media has no special status in Canadian law.

2) Still more fascinating is how the RCMP appears to be completely unresponsive to news stories about inappropriate behavior in its ranks, like say the illegal funding of false research to defend the war on drugs, but one story about language politics causes the organization to change practices that aren’t even in violation of its policies. It us sad to see more evidence still that the RCMP is one of the most broken agencies in the Federal government.

3) Thank you to Vancouver Sun Reporter Chad Skelton for updating me on the Google aspect of this story.

The best moment in Canadian democracy in 2010?: the census debate

Over at Samara, my friend Alison Loat is asking people to answer the question “What was the best moment in Canadian democracy in 2010?” In what I think was a good decision, they’ve defined the terms pretty broadly, stating:

The moment could be one that took place inside or outside of Parliament or other legislative chambers.  It could have happened at the federal, provincial, territorial or municipal level.  It could include any number of things, such as an election with a historic turnout, a stimulating public debate, a rally or protest, a critical piece of news analysis, the creation of a new digital application, or an important Parliamentary motion or decision.

If you’ve got an idea I encourage you to hear over there and write it up and submit it! The Samara people are great and are up to good work, so definitely worth checking out.

I’ve got one answer the question myself – what follows is my write up. I think I may even have one more in me… but here’s my first effort:

The Census Debate as Canada’s 2010 democratic moment.

In a functioning democracy disagreement is necessary and healthy. But at its core there most be some basic agreement – some shared understanding of who we are, as a people and as a society. This shared understanding not only serves as the basic facts that must inform our debates but also the basis of our shared identity that keeps us together even when we disagree.

This is why the census is so important, and why it is my choice for the best moment in Canadian democracy for 2010. The census binds us together by creating a shared understanding of who we are. Even the most marginalized Canadians stand up and are counted and thus can be reflected and heard in our national discourse.

That’s why at a time when Canadian political coverage tries to cleave the country’s citizens into different, competing groups – rural versus urban, French versus English, left versus right – I think the best moment in Canadian Democracy was seeing over 500 groups including all levels of government, non-profits from across the country, business organizations, rural communities, and virtually all the major religious organizations come together and challenge the government with one voice.

What a great democratic moment that so many organizations, that often disagree on so many issues, can collectively agree on a core shared interest: that a functioning democracy and an effective government is built on a foundation of some basic information about who we are. Even more so when the government tried to make the decision in secret, announcing it quietly on a friday, during a long weekend in the middle of summer.

The decision and the process surrounding it may be one of the year’s darkest moments for Canadian democracy but the country’s reaction was definitely one of our brightest.

What Governments can Learn about Citizen Engagement from Air Canada

Yes. You read that title right.

I’m aware that airlines are not known for their customer responsiveness. Ask anyone whose been trapped on a plane on the tarmac for 14 hours. You know you’ve really dropped the ball when Congress (which agrees on almost nothing) passes a customer bill of rights explicitly for your industry.

Air Canada, however, increasingly seems to be the exception to this rule. Their recent response to online customer feedback is instructive of why this is the case. For governments interested in engaging citizens online and improving services, Air Canada is an interesting case study.

The Background

Earlier this year, with great fanfare, Air Canada announced it was changing how it managed its frequent flyer reward system. Traditional, it had given out upgrade certificates which allowed customers who’d flown a certain number of flights the air-canada-logoability to upgrade themselves into business class for free. Obviously the people who use these certificates are some of Air Canada’s more loyal customers (to get certificates you have to be flying a fair amount). The big change was that rather than simple giving customers certificates after flying a certain number of miles, customers would earn “points” which they could allocate towards flights.

This was supposed to be a good news story because a) it meant that users had greater flexibility around how they upgraded themselves and b) the whole system was digitized so that travelers wouldn’t have to carry certificates around with them (this was the most demanded feature by users).

The Challenge

In addition to the regular emails and website announcement an Air Canada representative also posted details about the new changes on a popular air traveler forum called Flyertalk.com. (Note: Here is the first great lesson – don’t expect customers or citizens to come to you… go to where they hang out, especially your most hard core stakeholders).

flyertalk_logoVery quickly these important stakeholders (customers) began running the numbers and started discovering various flaws and problems. Some noticed that the top tier customers were getting a lesser deal than ordinary customers. Others began to sniff out how the new program essentially meant their benefits were being cut. In short, the very incentives the rewards program was supposed to create were being undermined. Indeed the conversation thread extended to over 113 pages. With roughly 15 comments per page, that meant around 1500 comments about the service.

This, of course, is what happens with customers, stakeholder and citizens in a digital world. They can get together. They can analyze your service. And they will notice any flaws or changes that do not seem above board or are worse than what previously existed.

So here, on Flyertalk, Air Canada has some of its most sophisticated and important customers – the people that will talk to everyone about Air travel rewards programs, starting to revolt against its new service which was supposed to be a big improvement. This was (more than) a little bit of a crisis.

The Best Practice

First, Air Canada was smart because it didn’t argue with anyone. It didn’t have communication people trying to explain to people how they were wrong.

Instead it was patient. It appeared silent. But in reality it was doing something more important, it was listening.

Remember many of these users know the benefits program better than most Air Canada employees. And it has real impact on their decisions, so they are going to analyze it up and down.

Second, When it finally did respond, Air Canada did several things right.

It responded in Flyertalk.com – again going to where the conversation was. (It subsequently sent around an email to all its members).

It noted that it had been listening and learning from its customers.

More than just listen, Air Canada had taken its customers feedback and used it to revise its air travel rewards program.

And, most importantly, the tone it took was serious, but engaging. Look at the first few sentences:

Thanks to everyone for the comments that have been posted here the last few days, and especially those who took the time to post some very valuable, constructive feedback. While it’s not our intent to address every issue raised on this forum on the changes to the 2011 Top Tier program, some very valid points were raised which we agree should be addressed to the best of our ability. These modifications are our attempt to do just that.

Governments, this is a textbook case on how to listen to citizens. They use your services. They know how they work. The single biggest take away here is, when they complain and construct logical arguments about why a service doesn’t make sense use that feedback to revise the service and make it better. People don’t want to hear why you can’t make it better – they want you to make it better. More importantly, these types of users are the ones who know your service the best and who talk to everyone about it. They are your alpha users – leverage them!

Again, to recap. What I saw Air Canada do that was positive was:

  • Engage their stakeholders where their stakeholders hang out (e.g. not on the Air Canada website)
  • Listen to what their stakeholders had to say
  • Use that feedback to improve the service
  • Communicate with customers in a direct and frank manner

Air Canada is doing more than just getting this type of engagement right. Their twitter account posts actual useful information, not just marketing glop and spin. I’m not sure who is doing social media for them, but definitely worth watching.

There’s a lot here for organizations to learn from. Moreover, for a company that used to be a crown corp I think that should mean there is hope for your government too – even if they presently ban access to facebook, twitter or say, my blog.

Big thank you to Mike B. for pointing out this cool case study to me.

Visualizing Firefox Plugins Memory Consumption

A few months ago the Mozilla Labs and the Metrics Team, together with the growing Mozilla Research initiative, launched an Open Data Visualization Competition.

Using data collected from Test Pilot users (people who agreed to share anonymous usage data with Mozilla and test pilot new features) Mozilla asked its community to think of creative visual answers to the question: “How do people use Firefox?”

As an open data geek and Mozilla supporter the temptation to try to do something was too great. So I teamed up with my old data partner Diederik Van Liere and we set out to create a visualization. Our goals were simple:

  • have fun
  • focus on something interesting
  • create something that would be useful to Firefox developers and/or users
  • advance the cause for creating a Firefox open data portal

What follows is the result.

It turns out that – in our minds – the most interesting data set revolved around plugin memory consumption. Sure this sounds boring… but plugins (like Adobe reader, Quicktime or Flash) or are an important part of the browser experience – with them we engage in a larger, richer and more diverse set of content.  Plugins, however, also impact memory consumption and, consequently, browser performance. Indeed, some plugins can really slow down Firefox (or any browser). If consumers had a better idea of how much performance would be impacted they might be more selective about which plugins they download, and developers might be more aggressive in trying to make their plugins more efficient.

Presently, if you run Firefox you can go to the Plugin Check page to see if your plugins are up to date. We thought: Wouldn’t it be great if that page ALSO showed you memory consumption rates? Maybe something like this (note the Memory Consumption column, it doesn’t exist on the real webpage, and you can see a larger version of this image here):

Firefox data visualization v2

Please understand (and we are quite proud of this). All of the data in this mockup is real. Memory consumptions are estimates we derived by analyzing the Test Pilot data.

How, you might ask did we (Diederik) do that?

GEEK OUT EXPLANATION: Well, we (Diederik) built a dataset of about 25,000 different testpilot users and parsed the data to see which plugins were installed and how much memory was consumed around the time of initialization. This data was analyzed using ordinary least squares regression where the dependent variable is memory consumption and the different plugins are the explanatory variables. We only included results that are highly significant.

The following table shows our total results (you can download a bigger version here).

Plugin_memory_consumption_chart v2

Clearly, not all plugins are created equal.

Our point here isn’t that we have created the definitive way of assessing plugin impact on the browser, our point is that creating a solid methodology for doing so is likely witihin Mozilla’s grasp. More importantly, doing this could help improve the browsing experience. Indeed, it would probably be even wiser to do something like this for Add-ons, which is where I’m guessing the real lag time around the browsing experience is created.

Also, with such a small data set we were only able to calculate the memory usage for a limited number of plugins and generally those that are more obscure. Our methodology required having several data points from people who are and who aren’t using a given plugin and so with many popular plugins we didn’t have enough data from people who weren’t using it… a problem however, that would likely be easily solved with access to more data.

Finally, I hope this contest and our submission helps make the case for why Mozilla needs an open data portal. Mozilla collects and incredible amount of data of which it does not have the resources to analyze internally. Making it available to the community would do to data what Mozilla has done to code – enable others create value that could affect the product and help advance the open web. I had a great meeting earlier this week with a number of the Mozilla people about this issue, I hope that we can continue to make progress.