Monthly Archives: July 2011

Using Data to Make Firefox Better: A mini-case study for your organization

I love Mozilla. Any reader of this blog knows it. I believe in its mission, I find the organization totally fascinating and its processes engrossing. So much so I spend a lot of time thinking about it – and hopefully, finding ways to contribute.

I’m also a big believer in data. I believe in the power of evidence-based public policy (hence my passion about the long-form census) and in the ability of data to help organizations develop better products, and people make smarter decisions.

Happily, a few months ago I was able to merge these two passions: analyzing data in an effort to help Mozilla understand how to improve Firefox. It was fun. But more importantly, the process says a lot about the potential for innovation open to organizations that cultivate an engaged user community.

So what happened?

In November 2010, Mozilla launched a visualization competition that asked: How do People Use Firefox? As part of the competition, they shared anonymous data collected from Test Pilot users (people who agreed to share anonymous usage data with Mozilla). Working with my friend (and quant genius) Diederik Van Liere, we analyzed the impact of add-on memory consumption on browser performance to find out which add-ons use the most memory and thus are most likely slowing down the browser (and frustrating users!). (You can read about our submission here).

But doing the analysis wasn’t enough. We wanted Mozilla engineers to know we thought that users should be shown the results – so they could make more informed choices about which add-ons they download. Our hope was to put pressure on add-on developers to make sure they weren’t ruining Firefox for their users. To do that we visualized the data by making a mock up of their website – with our data inserted.

FF-memory-visualizations2.001

For our efforts, we won an honourable mention. But winning a prize is far, far less cool than actually changing behaviour or encouraging an actual change. So last week, during a trip to Mozilla’s offices in Mountain View, I was thrilled when one of the engineers pointed out that the add-on site now has a page where they list add-ons that most slow down Firefox’s start up time.

Slow-Performing-Add-ons-Add-ons-for-Firefox_1310962746129

(Sidebar: Anyone else find it ironic that “FastestFox: Browse Faster” is #5?)

This is awesome! Better still, in April, Mozilla launched an add-on performance improvement initiative to help reduce the negative impact add-ons can have on Firefox. I have no idea if our submission to the visualization competition helped kick-start this project; I’m sure there were many smart people at Mozilla already thinking about this. Maybe it was already underway? But I like to believe our ideas helped push their thinking – or, at least, validated some of their ideas. And of course, I hope it continues to. I still believe that the above-cited data shouldn’t be hidden on a webpage well off the beaten path, but should be located right next to every add-on. That’s the best way to create the right feedback loops, and is in line with Mozilla’s manifesto – empowering users.

Some lessons (for Mozilla, companies, non-profits and governments)

First lesson. Innovation comes from everywhere. So why aren’t you tapping into it? Diederik and I are all too happy to dedicate some cycles to thinking about ways to make Firefox better. If you run an organization that has a community of interested people larger than your employee base (I’m looking at you, governments), why aren’t you finding targeted ways to engage them, not in endless brainstorming exercises, but in innovation challenges?

Second, get strategic about using data. A lot of people (including myself) talk about open data. Open data is good. But it can’t hurt to be strategic about it as well. I tried to argue for this in the government and healthcare space with this blog post. Data-driven decisions can be made in lots of places; what you need to ask yourself is: What data are you collecting about your product and processes? What, of that data, could you share, to empower your employees, users, suppliers, customers, whoever, to make better decisions? My sense is that the companies (and governments) of the future are going to be those that react both quickly and intelligently to emerging challenges and opportunities. One key to being competitive will be to have better data to inform decisions. (Again, this is the same reason why, over the next two decades, you can expect my country to start making worse and worse decisions about social policy and the economy – they simply won’t know what is going on).

Third, if you are going to share, get a data portal. In fact, Mozilla needs an open data portal (there is a blog post that is coming). Mozilla has always relied on volunteer contributors to help write Firefox and submit patches to bugs. The same is true for analyzing its products and processes. An open data portal would enable more people to help find ways to keep Firefox competitive. Of course, this is also true for governments and non-profits (to help find efficiencies and new services) and for companies.

Finally, reward good behaviour. If contributors submit something you end up using… let them know! Maybe the idea Diederik and I submitted never informed anything the add-on group was doing; maybe it did. But if it did… why not let us know? We are so pumped about the work they are doing, we’d love to hear more about it. Finding out by accident seems like a lost opportunity to engage interested stakeholders. Moreover, back at the time, Diederik was thinking about his next steps – now he works for the Wikimedia Foundation. But it made me realize how an innovation challenge could be a great way to spot talent.

The Audacity of Shaw: How Canada's Internet just got Worse

It is really, really, really hard to believe. But as bad as internet access is in Canada, it just got worse.

Yesterday, Shaw Communications, a Canadian telecommunications company and internet service provider (ISP) that works mostly in Western Canada announced they are launching Movie Club, a new service to compete with Netflix.

On the surface this sounds like a good thing. More offerings should mean more competition, more choice and lower prices. All things that would benefit consumers.

Look only slightly closer and you learn the very opposite is going on.

This is because, as the article points out:

“…subscribers to Movie Club — who initially can watch on their TV or computer, with phones and tablets planned to come on line later — can view content without it counting against their data plan.

“There should be some advantage to you being a customer,” Bissonnette said.”

The very reason the internet has been such an amazing part of our lives is that every service that is delivered on it is treated equally. You don’t pay more to look at the Vancouver Sun’s website than you do to look at eaves.ca or CNN or to any other website in the world. For policy and technology geeks this principle of equality of access is referred to as net neutrality. The idea is that ISPs (like Shaw) should not restrict or give favourable access to content, sites, or services on the internet.

But this is precisely what Shaw is doing with its new service.

This is because ISPs in Canada charge what are called “overages.” This means if you use the internet a lot, say you watch a lot of videos, at a certain point you will exceed a “cap” and Shaw charges you extra, beyond your fixed monthly fee. If, for example, you use Netflix (which is awesome and cheap, for $8 a month you get unlimited access to a huge quantity of content) you will obviously be watching a large number of videos, and the likelihood of exceeding the cap is quite high.

What Shaw has announced is that if you use their service – Movie Club – none of the videos you watch will count against your cap. In other words they are favouring their service over that of others.

So why should you care? Because, in short, Shaw is making the internet suck. It wants to turn your internet from the awesome experience where you have unlimited choice and can try any service that is out there, into the experience of cable, where your choice is limited to the channels they choose to offer you. Today they’ll favour their movie service as opposed to (the much better) Netflix service. But tomorrow they may decide… hey you are using Skype instead of our telephone service, people who use “our skype” will get cheaper access than people who use skype. Shaw is effectively applying a tax on new innovative and disruptively cheap service on the internet so that you don’t use them. They are determining – through pricing – what you can and cannot do with your computer while elsewhere in the world, people will be using cool new disruptive services that give them better access to more fun content, for cheaper. Welcome to the sucky world of Canada’s internet.

Doubling down on Audacity: The Timing

Of course what makes this all the more obscene is that Shaw has announced this service at the very moment the CRTC – the body that regulates Canada’s Internet Service Providers – is holding hearings on Usage Based Billings. One of the reasons Canada’s internet providers say that have to charge “overages” for those who use the internet a lot is because of there isn’t enough bandwidth. But how is it that there is enough bandwidth for their own services?

As Steve Anderson of the OpenMedia – a consumer advocacy group – shared with me yesterday “It’s a huge abuse of power.” and that “The launch of this service at the time when the CRTC is holding a hearing on pricing regulation should be seen as a slap in the face to the the CRTC, and the four hundred and ninety one thousand Canadians that signed the Stop The Meter petition.”

My own feeling is the solution is pretty simple. We need to get the ISPs out of the business of delivering content. Period. Their job should be to deliver bandwidth, and nothing else. You do that, you’ll have them competing over speed and price very, very quickly. Until then the incentive of ISPs isn’t to offer good internet service, it’s to do the opposite, it’s to encourage (or force) users to use the services they offer over the internet.

For myself, I’m a Shaw customer and a Netflix customer. Until now I’ve had nothing to complain about with either. Now, apparently I have to choose between the two. I can tell you right now who is going to win. Over the next few months I’m going to be moving my internet service to another provider. Maybe I’ll still get cable TV from Shaw, I don’t know, but my internet service is going to a company that gives me the freedom to choose the services I want and that doesn’t ding me with fees that apparently, I’m being charged under false pretenses. I’ll be telling by family members, friends and pretty much everyone I know, to do the same.

Shaw, I’m sorry it had to end this way. But as a consumer, it’s the only responsible thing to do.

It's the icing, not the cake: key lesson on open data for governments

At the 2010 GTEC conference I did a panel with David Strigel, the Program Manager of the Citywide Data Warehouse (CityDW) at the District of Columbia Government. During the introductory remarks David recounted the history of Washington DC’s journey to open data.

Interestingly, that journey began not with open data, but with an internal problem. Back around 2003 the city had a hypothesis that towing away abandoned cars would reduce crime rates in the immediate vicinity, thereby saving more money in the long term than the cost of towing. In order to access the program’s effectiveness city staff needed to “mash-up” longitudinal crime data against service request data – specifically, requests to remove abandoned cars. Alas, the data sets were managed by different departments, so this was tricky task. As a result the city’s IT department negotiated bilateral agreements with both departments to host their datasets in a single location. Thus the DC Data Warehouse was born.

Happily, the data demonstrated the program was cost effective. Building on this success the IT department began negotiating more bilateral agreements with different departments to host their data centrally. In return for giving up stewardship of the data the departments retained governance rights but reduced their costs and the IT group provided them with additional, more advanced, analytics. Over time the city’s data warehouse became vast. As a result, when DC decided to open up its data it was, relatively speaking, easy to do. The data was centrally located, was already being shared and used as a platform internally. Extending this platform externally (while not trivial) was a natural step.

In short, the deep problem that needed to solved wasn’t open data. Its was an information management. Getting the information management and governance policies right was essential for DC to move quickly. Moreover, this problem strikes at the heart of what it means to be government. Knowing what data you have, where it is, and under a governance structure that allows it to be shared internally (as well as externally) is a problem every government is going to face if it wants to be efficient, relevant and innovative in the 21st century. In other words, information management is the cake. Open data – which I believe is essential – is however the sweet icing you smother on top of that dense cake you’ve put in place.

Okay, with that said two points that flow from this.

First: Sometime, governments that “do” open data start off by focusing on the icing. The emphasis in on getting data out there, and then after the fact, figuring out  governance model that will make sense. This is a viable strategy, but it does have real risks. When sharing data isn’t at the core function but rather a feature tacked on at the end, the policy and technical infrastructure may be pretty creaky. In addition, developers may not want to innovate on top of your data platform because they may (rightly) question the level of commitment. One reason DC’s data catalog works is because it has internal users. This gives the data stability and a sense of permanence. On the upside, the icing is politically sexier, so it may help marshal resources to help drive a broader rethink of data governance. Either way, at some point, you’ve got to tackle the cake, otherwise, things are going to get messy. Remember it took DC 7 years to develop its cake before it put icing on it. But that was making it from scratch. Today thanks to new services (armies of consultants on this), tools (eg. Socrata) and models (e.g. like Washington, DC) you can make that cake following a recipe and even use cake mix. As David Strigel pointed out, today, he could do it in a fraction of the time.

Second: More darkly, one lesson to draw from DC is that the capacity of a government to do open data may be a pretty good proxy for their ability to share information and coordinate across different departments. If your government can’t do open data in a relatively quick time period, it may mean they simply don’t have the infrastructure in place to share data internally all that effectively either. In a world where government productivity needs to rise in order to deal with budget deficits, that could be worrying.

The End of the World and Journalism in the Era of Open

For those not in the United Kingdom a massive scandal has erupted around allegations that one of the country’s tabloids – the News of the World ( a subsidiary of Rupert Murdoch’s News Corporation) – was illegally hacking into and listening in on the voicemails of not only the royal family members and celebrities but also murder victims and family members of soldiers killed in Afghanistan.

The fall out from the scandal, among other things, has caused the 168 year old newspaper to be unceremoniously closed, prompted an enormous investigation into the actions of editors and executives at the newspaper, forced the resignation (and arrest) of Andy Coulson – former News of the World editor and director of communications for the Prime Minister – and thrown into doubt Rupert Murdoch’s bid to gain complete control over the British satellite television network BskyB.

For those wanting to know more I encourage you to head over to the Guardian, which broke the story and has done some of the best reporting on it. Also, possibly the best piece of analysis I’ve read on the whole sordid affair is this post from reuters which essentially points out that by shutting down News of the World, Newscorp may shrewdly ensure that all incriminating documents can (legally) be destroyed. Evil genius stuff.

But why bring this all up here at eaves.ca?

Because I think this is an example of a trend in media that I’ve been arguing has been going on for some time.

Contrary to what news people would have you believe, my sense is that most people don’t trust newspapers – no more so then they trust governments. Starting in 1983 Ipsos MORI and the British Medical Association have asked UK citizens who they trust. The results for politicians are grim. The interesting thing is, they are no better for journalists (although TV news anchors do okay). Don’t believe me? Take a look at the data tables from Ipsos MORI. Or look at the chart Benne Dezzle over at Viceland created out of the data.

There is no doubt people value the products of governments and the media – but this data suggests they don’t trust the people creating them, which I really think is a roundabout way of saying: they don’t trust the system that creates the news.

I spend a lot of my time arguing that government’s need to be more transparent, and that this (contrary to what many public servants feel) will make them more, not less, effective. Back in 2009, in reaction to the concern that the print media was dying, I wrote a blog post saying the same was true for journalism. Thanks, in part, to Jay Rosen listing it as part of his flying seminar on the future of news, it became widely read and ended up as getting reprinted along with Taylor Owen and I’s article Missing the Link, in the journalism textbook The New Journalist. Part of what I think is going in the UK is a manifestation of the blog post, so if you haven’t read it, I think now is as good a time as any.

The fact is, newsrooms are frequently as opaque (both in process and, sometimes, in motivations) as governments are. People may are willing to rely on them, and they’ll use them if their outputs are good, but they’ll turn on them, and quickly, if they come to understand that the process stinks. This is true of any organization and news media doesn’t get a special pass because of the job it plays – indeed the opposite may be true. But more profoundly I think it is interesting how what many people consider to be two of the key pillars to western democracy are staffed by people who are among the least trusted in our society. Maybe that’s okay. But maybe it’s not. But if we think we need better forms of government – which many people seem to feel we do – it may also be that we believe we need better ways of generating, managing and engaging in the accountability of that government.

Of course, I don’t want to overplay the situation here. News of the World doomed itself because it broke the law. More importantly, it did so in a truly offensive way: hacking into the cell phone of a murder victim who was an everyday person. Admitedly, when the victims were celebrities, royals and politicians, it percolated as a relatively contained scandal. But if we believe that transparency is the sunlight that causes governments to be less corrupt – or at least forces politicians to recognize their decisions will be more scrutinized – maybe a little transparency might have caused the executives and editors at News Corp to behave a little better as well. I’m not sure what a more open media organization might look like – although wikipedia does an interesting job – but from both a brand protection and values based decision making perspective a little transparency could be the right incentive to ensure that the journalists, editors and executives in a news system few of us seem to trust, behave a little better. And that might cause them to earn more of the trust I think many deserve.

 

 

 

 

 

 

Lots of Open Data Action in Canada

A lot of movement on the open data (and not so open data) front in Canada.

Canadian International Development Agency (CIDA) Open Data Portal Launched

IATI-imagesSome readers may remember that last week I wrote a post about the imminent launch of CIDA’s open data portal. The site is now live and has a healthy amount of data on it. It is a solid start to what I hope will become a robust site. I’m a big believer – and supporter of the excellent advocacy efforts of the good people at Engineers Without Borders – that the open data portal would be greatly enhanced if CIDA started publishing its data in compliance with the emerging international standard of the International Aid Transparency Initiative as these 20 leading countries and organizations have.

If anyone creates anything using this data, I’d love to see it. One simple start might be to try using the Open Knowledge Foundation’s open source Where Does my Money Go code, to visualize some of the spending data. I’d be happy to chat with anyone interested in doing this, you can also check out the email group to find some people experienced in playing with the code base.

Improved License on the CIDA open data portal and data.gc.ca

One thing I’ve noticed with the launch of the CIDA open data portal was how the license was remarkably better than the license at data.gc.ca – which struck me as odd, since I know the feds like to be consistent about these types of things. Turns out that the data.gc.ca license has been updated as well and the two are identical. This is good news as some of the issues that were broken with the previous license have been fixed. But not all. The best license out there remains the license at data.gov (that’s a trick question, because data.gov has no license, it is all public domain! Tricky eh…? Nice!) but if you are going to have a license, the UK Open Government License used by at data.gov.uk is more elegant, freer and satisfies a number of the concerns I cite above and have heard people raise.

So this new data.gc.ca license is a step in the right direction, but still behind the open gov leaders (teaching lawyers new tricks sadly takes a long time, especially in government).

Great site, but not so open data: WellBeing Toronto

Interestingly, the City of Toronto has launched a fabulous new website called Well Being Toronto. It is definitely worth checking out. The main problem of course is that while it is interesting to look at, the underlying data is, sadly, not open. You can’t play with the data, such as mash it up with your own (or another jurisdiction’s) data. This is disappointing as I believe a number of non-profits in Toronto would likely find the underlying data quite helpful/important. I have, however, been told that the underlying data will be made open. It is something I hope to check in on again in a few months as I fear that it may never get prioritized, so it may be up to Torontonians to whold the Mayor and council’s feet to the fire to ensure it gets done.

Parliamentary Budget Office (PBO) launches (non-open) data website

It seems the PBO is also getting in on the data action with the launch of a beta site that allows you to “see” budgets from the last few years. I know that the Parliamentary Budget Office has been starved of resources, so they deserve to be congratulated for taking this first, important step. Also interesting is that the data has no license on the website, which could make it the most liberally licensed open data portal in the country. The site does have big downsides. First, the data can only be “looked” at, there is no obvious (simple) way to download it and start playing with it. More oddly still the PBO requires that users register with their email address to view the data. This seems beyond odd and actually, down right creepy, to me. First, parliament’s budget should be free and open and one should not need to hand over an email address to access it. Second, the email addresses collected appear to serve no purpose (unless the PBO intends to start spamming us), other than to tempt bad people to hack their site so they can steal a list of email addresses.

Why not create an Open311 add-on for Ushahidi?

This is not a complicated post. Just a simple idea: Why not create an Open311 add-on for Ushahidi?

So what do I mean by that, and why should we care?

Many readers will be familiar with Ushahidi, non-profit that develops open source mapping software that enables users to collect and visualize data in interactive maps. It’s history is now fairly famous, as the Wikipedia article about it outlines: “Ushahidi.com’ (Swahili for “testimony” or “witness”) is a website created in the aftermath of Kenya’s disputed 2007 presidential election (see 2007–2008 Kenyan crisis) that collected eyewitness reports of violence sent in by email and text-message and placed them on a Google map.[2]“Ushahidi’s mapping software also proved to be an important resource in a number of crises since the Kenyan election, most notably during the Haitian earthquake. Here is a great 2 minute video on How how Ushahidi works.

ushahidi-redBut mapping of this type isn’t only important during emergencies. Indeed it is essential for the day to day operations of many governments, particularly at the local level. While many citizens in developed economies may be are unaware of it, their cities are constantly mapping what is going on around them. Broken infrastructure such as leaky pipes, water mains, clogged gutters, potholes, along with social issues such as crime, homelessness, business and liquor license locations are constantly being updated. More importantly, citizens are often the source of this information – their complaints are the sources of data that end up driving these maps. The gathering of this data generally falls under the rubric of what is termed 311 systems – since in many cities you can call 311 to either tell the city about a problem (e.g. a noise complaint, service request or inform them about broken infrastructure) or to request information about pretty much any of the city’s activities.

This matters because 311 systems have generally been expensive and cumbersome to run. The beautiful thing about Ushahidi is that:

  1. it works: it has a proven track record of enabling citizens in developing countries to share data using even the simplest of devices both with one another and agencies (like humanitarian organizations)
  2. it scales: Haiti and Kenya are pretty big places, and they generated a fair degree of traffic. Ushahidi can handle it.
  3. it is lightweight: Ushahidi technical footprint (yeap making that up right now) is relatively light. The infrastructure required to run it is not overly complicated
  4. it is relatively inexpensive: as a result of (3) it is also relatively cheap to run, being both lightweight and leveraging a lot of open source software
  5. Oh, and did I mention IT WORKS.

This is pretty much the spec you would want to meet if you were setting up a 311 system in a city with very few resources but interested in starting to gather data about both citizen demands and/or trying to monitor newly invested in infrastructure. Of course to transform Ushahidi into a process for mapping 311 type issues you’d need some sort of spec to understand what that would look like. Fortunately Open311 already does just that and is supported by some of the large 311 providers system providers – such as Lagan and Motorola – as well as some of the disruptive – such as SeeClickFix. Indeed there is an Open311 API specification that any developer could use as the basis for the add-on to Ushahidi.

Already I think many cities – even those in developing countries – could probably afford SeeClickFix, so there may already be a solution at the right price point in this space. But maybe not, I don’t know. More importantly, an Open311 module for Ushahidi could get local governments, or better still, local tech developers in developing economies, interested in and contributing to the Ushahidi code base, further strengthening the project. And while the code would be globally accessible, innovation and implementation could continue to happen at the local level, helping drive the local economy and boosting know how. The model here, in my mind, is OpenMRS, which has spawned a number of small tech startups across Africa that manage the implementation and servicing of a number of OpenMRS installations at medical clinics and countries in the region.

I think this is a potentially powerful idea for stakeholders in local governments and startups (especially in developing economies) and our friends at Ushahidi. I can see that my friend Philip Ashlock at Open311 had a similar thought a while ago, so the Open311 people are clearly interested. It could be that the right ingredients are already in place to make some magic happen.