Tag Archives: opendata

International Open Data Hackathon, Dec 3rd. It's coming together.

So a number of things have started to really come together for this Saturday Dec 3rd. I’ve noticed a number of new cities being tweeted about (hello Kuala Lumpur & Oakland!) and others adding themselves to the wiki. Indeed, we seem to be above 40 cities. It is hard to know how many people will be showing up in each given city, but in Vancouver I know that we already over 20 registered, while in Ottawa they are well above 40. If other cities have similar numbers it’s a great testament to the size of the community out there interested in playing with open government data.

A few thoughts to share with people as we get ready for the big day.

1. Leverage existing projects.

I’ve mentioned a few times that there are some great existing projects out there that can be easily leveraged.

In that vein I’ve noticed the good people at the Open Knowledge Foundation, who are behind OpenSpending (the project that powers WherDoesMyMoneyGo.org) have not only made their software easier to use but have put up some helpful instructions for creating your own instance up on the wiki. One hope I have for Saturday is that a number of different places might be able to visualize local budgets in much easier to understand ways. OpenSpending has the potential of being an enormously helpful tool for communities trying to understand their budget – hopefully we can provide some great examples and feedback for its creators.

In addition, the folks at MySociety have provided some helpful advice on the wiki for those interested in spinning up a version of MapIt for their country.

2. Get Data Now, Not on Saturday!

Here in Vancouver, my friend Luke C asked if we could get bicycle accident data for the city or province as he wanted to play around with it and maybe visualize it on December 3rd. It just so happened I had a contact at the Insurance Company of British Columbia (ICBC) which insures every vehicle in the province. I reached out and, after going through their request process, now have the data set to share with Luke.

The key piece here: now is the time to check and see if data you are interested in is available, see investigate what is out there, and request it from various stakeholders if it is not.

3. Share Your Code, Share your Data

Indeed, one advantage of having the BC bicycle accident data early is that I can start sharing it with people immediately. I’ve already uploaded the data set (all 6400 lines) onto BuzzData’s site here so others can download it, clone it, and share their own work on it. That way, even if Luke and I get separated, he’s still got something to hack on!

So please do let people know where they can find data you are hacking on, as well as project you’re hacking on. The Open Data Day Projects 2011 wiki page currently sits empty (as should be expected). But take a swing by the page 2010 project page, notice how it is quite full… I’d love to see us replicate this success. I’m hoping people link to not just their projects, but also Github repos, scraperwiki creations, BuzzData accounts and other places.

If you have a project and you think people in open data day hackathons in other cities might be interested, put it in the project page and tweet about it using the #odhd hashtag. You may discover there are people out there who feel as passionately about your project as you do!

4. Let’s Get Connected

Speaking of sharing, my friend Edward O-G, who is organizing the hackathon in Ottawa, did a great job last year setting up some infrastructure so people from different hackathons could video conference with one another. This year I think we’ll try using Google hangouts on google+. However, there is a non-trivial risk that this will not scale super well.

So…

Edward also suggested (brilliantly) that people create YouTube videos of whatever they create during the hackathon or in the days and weeks that follow. Please post those links to the Open Data Day Projects 2011 wiki page as well. There were a few projects last year that had youtube videos and they were very helpful, particularly when a project isn’t quite ready for prime time. It gives us a taste of what will be available. It also becomes something we can point people to.

5. Have Fun, Do What Is Interesting

Remember, Open Data Day is about meeting people, learning about open data, and working on something that you feel passionate about. This is all very decentralized and informal – no one is going to come and save your hackathon… it is up to you! So make sure you find something you think is worth caring about and work on it. Share your idea, and your passion, with others, that’s what makes this fun.

Can’t wait to hear what people are up to. Please feel free to email or tweet at me what you’re working on. I’d love to hear about it and blog about them.

Here in Vancouver, the open data hackathon will be happening at the offices of FoodTree, which has some of its own developers working on Open Food data.(If you haven’t signed up yet, definitely do so here).

Statistics Canada Data to become OpenData – Background, Winners and Next Steps

14 Replies

As some of you learned last night, Embassy Magazine broke the story that all of Statistics Canada’s online data will not only be made free, but released under the Government of Canada’s Open Data License Agreement (updated and reviewed earlier this week) that allows for commercial re-use.

This decision has been in the works for months, and while it does not appear to have been formally announced, Embassy Magazine does appear to have managed to get a Statistics Canada spokesperson to confirm it is true. I have a few thoughts about this story: Some background, who wins from this decision, and most importantly, some hope for what it will, and won’t lead to next.

Background

In the embassy article, the spokesperson claimed this decision had been in the works for years, something that is probably technically true. Such a decision – or something akin to it – has likely been contemplated a number of times. And there have been a number of trials and projects that have allowed for some data to be made accessible albeit under fairly restrictive licenses.

But it is less clear that the culture of open data has arrived at StatsCan, and less clear to me that this decision was internally driven. I’ve met many a Statscan employee who encountered enormous resistance while advocating for data open. I remember pressing the issue during a talk at one of the department’s middle managers conference in November of 2008 and seeing half the room nod vigorously in agreement, while the other half crossed it arms in strong disapproval.

Consequently, with the federal government increasingly interested in open data, coupled with a desire to have a good news story coming out of statscan after last summer census debacle, and with many decisions in Ottawa happening centrally, I suspect this decision occurred outside the department. This does not diminish its positive impact, but it does mean that a number of the next steps, many of which will require StatsCan to adapt its role, may not happen as quickly as some will hope, as the organization may take some time to come to terms with the new reality and the culture shift it will entail.

This may be compounded by the fact that there may be tougher news on the horizon for StatsCan. With every department required to have submitted proposal to cut their budgets by either 5% and 10%, and with StatsCan having already seen a number of its programs cut, there may be fewer resources in the organization to take advantage of the opportunity making its data open creates, or even just adjust to what has happened.

Winners (briefly)

The winners from this decision are of course, consumers of statscan’s data. Indirectly, this includes all of us, since provincial and local governments are big consumers of statscan data and so now – assuming it is structured in such a manner – they will have easier (and cheaper) access to it. This is also true of large companies and non-profits which have used statscan data to locate stores, target services and generally allocate resources more efficiently. The opportunity now opens for smaller players to also benefit.

Indeed, this is the real hope. That a whole new category of winners emerges. That the barrier to use for software developers, entrepreneurs, students, academics, smaller companies and non-profits will be lowered in a manner that will enable a larger community to make use of the data and therefor create economic or social goods.

Such a community, however, will take time to evolve, and will benefit from support.

And finally, I think StatsCan is a winner. This decision brings it more profoundly into the digital age. It opens up new possibilities and, frankly, pushes a culture change that I believe is long over due. I suspect times are tough at StatsCan – although not as a result of this decision – this decision creates room to rethink how the department works and thinks.

Next Steps

The first thing everybody will be waiting for is to see exactly what data gets shared, in what structure and to what detail. Indeed this question arose a number of times on twitter with people posting tweets such as “Cool. This is all sorts of awesome. Are geo boundary files included too, like Census Tracts and postcodes?” We shall see. My hope is yes and I think the odds are good. But I could be wrong, at which point all this could turn into the most over hyped data story of the year. (Which actually matters now that data analysts are one of the fastest growing categories of jobs in North America).

Second, open data creates an opportunity for a new and more relevant role for StatsCan to a broader set of Canadians. Someone from StatsCan should talk to the data group at the World Bank around their transformation after they launched their open data portal (I’d be happy to make the introduction). That data portal now accounts for a significant portion of all the Bank’s web traffic, and the group is going through a dramatic transformation, realizing they are no longer curators of data for bank staff and a small elite group of clients around the world but curators of economic data for the world. I’m told a new, while the change has not been easy, a broader set of users have brought a new sense of purpose and identity. The same could be true of StatsCan. Rather than just an organization that serves the government of Canada and a select groups of clients, StatsCan could become the curators of data for all Canadians. This is a much more ambitious, but I’d argue more democratized and important goal.

And it is here that I hope other next steps will unfold. In the United States, (which has had free census data for as long as anyone I talked to can remember) whenever new data is released the census bureau runs workshops around the country, educating people on how to use and work with its data. StatsCan and a number of other partners already do some of this, but my hope is that there will be much, much more of it. We need a society that is significantly more data literate, and StatsCan along with the universities, colleges and schools could have a powerful role in cultivating this. Tracey Lauriault over at the DataLibre blog has been a fantastic advocate of such an approach.

I also hope that StatsCan will take its role as data curator for the country very seriously and think of new ways that its products can foster economic and social development. Offering APIs into its data sets would be a logical next step, something that would allow developers to embed census data right into their applications and ensure the data was always up to date. No one is expecting this to happen right away, but it was another question that arose on twitter after the story broke, so one can see that new types of users will be interested in new, and more efficient ways, of accessing the data.

But I think most importantly, the next step will need to come from us citizens. This announcement marks a major change in how StatsCan works. We need to be supportive, particularly at a time of budget cuts. While we are grateful for open data, it would be a shame if the institution that makes it all possible was reduced to a shell of its former self. Good quality data – and analysis to inform public policy – is essential to a modern economy, society, and government. Now that we will have free access to what our tax dollars have already paid for, let’s make sure that it stays that way, by both ensure it continues to be available, and that there continues to be a quality institution capable of collecting and analyzing it.

(sorry for typos – it’s 4am, will revise in the morning)

If you are in Vancouver Vote Open Data, Vote Vision

1 Reply

If you are a Vancouver resident tomorrow is election day. I’m hoping if you are a resident and a reader of this blog, you’ll consider voting for Vision Vancouver.

As many of you know just over two years ago the city launched Vancouver’s Open Data portal – the first of its kind in Canada and the second municipal open data portal in the World.

This didn’t happen by chance. It took leadership from politicians who were a) willing to see and grasp a good idea long before it was widely celebrated, and b) able to drive it through city hall. It is a testament to those leaders – and the city staff who worked on it – that the city went from talking about the idea to implementing it in less than 3 months.

I know this is not an issue that everyone cares about. Personally, I believe open data is going to play a critical role in helping us rethink how government works, enabling it to be more effective and efficient while also empowering citizens to better understand and contribute to policy discussions. All that isn’t going to capture people’s imagination as much as ensuring the garbage gets collected regularly and on time (that’s important too!), but I do think the open data issue is a proxy, something that shows which leaders are willing to engage in innovative approaches and work with new technologies.

So if you are in Vancouver and you care about this issue, please vote Vision tomorrow. This is the party that made open data happen in Vancouver – and cleared the way for it in Canada. (location of voting stations is available here)

International Open Data Hackathon Updates and Apps

Open Data Day – a project I'd like to be doing

3 Replies

As some readers and International Open Data Hackathon participants know, I’m really keen on developers reusing each others code. All too often, in hackathons, we like to build something from scratch (which can be fun) but I’ve always liked the idea of hackathons either spurring genuine projects that others can reuse, or using a hackathon as an excuse to find a project they’d like support and contribute to.

That’s why I’ve been really encouraging people to find open source projects out there that they’d find interesting and that will support others efforts. This is a big reason I’ve been thinking about MapIt and the Open Knowledge Foundation’s Where does my Money Go project.

In Vancouver, one project I’m eventually hoping we can contribute to is Adopt-a-Hydrant. A project out of Code for America, the great thing about Adopt-a-Hydrant is that it can be adapted to become an adopt an anything app. It’s the end goal of a project I’m hoping to plan out and start on during the Hackathon.

Here in Vancouver, I’ve been talking with the Parks Board around getting a database of all the cities trees opened up. Interestingly this dataset does not include location data (Lat/Long) for each tree… So what would initially be great is to build a mobile phone app that will show you a list of trees near the address the user is currently at, and then allow the user to use their phone’s GPS to add the lat/long data to the database. That way we can help augment the city’s database. Once you begin to add lat long data then you could map trees in Adopt-a-Hydrant and create an Adopt-a-Tree app. Citizens could then sign up to adopt a tree, offer to take care of it, maybe notify the parks board if something is wrong.

I consider this a fairly ambitious project, but it could end up engaging a number of stakeholders – students, arborists, retirees, and others – that don’t normally engage in open data.

I know that the crew organizing a hackathon in Hamilton, Ontario are also looking to create an instance of Adopt-a-Hydrant, which is awesome. We need to both track what worked and what didn’t work so that the kinks in Adopt-a-hydrant can be worked out. More users and developers like us will help refine it further.

If you are planning a hackathon for the Dec 3rd International Open Data Hackathon, please be sure to update the wiki, join the mailing list, and if you have a project your are planning on working on, please email the list, or me directly, I’d love to blog about it!

Weaving Foreign Ministries into the Digital Era: Three ideas

1 Reply

Last week I was in Ottawa giving a talk at the Department of Foreign Affairs talking about how technology, new media and open innovation will impact the department’s it work internally, across Ottawa and around the world.

While there is lots to share, here are three ideas I’ve been stewing on:

Keep more citizens safe when abroad – better danger zone notification

Some people believe that open data isn’t relevant to departments like Foreign Affairs or the State Department. Nothing could be further than the truth.

One challenge the department has is getting Canadians to register with them when they visit or live in a country labeled by the department as problematic for traveling in its travel reports (sample here). As you can suspect, few Canadians register with the embassy as they are likely not aware of the program or travel a lot and simply don’t get around to it.

There are other ways of tackling this problem that might yield broader participation.

Why not turn the Travel Report system into an open data with an API? I’d tackle this by approaching a company like TripIt. Every time I book an airplane ticket or a hotel I simply forward TripIt the reservation, which they scan and turn into events that then automatically appear my calendar. Since they scan my travel plans they also know which country, city and hotel I’m staying in… they also know where I live and could easily ask me for my citizenship. Working with companies like TripIt (or Travelocity, Expedia, etc…) DFAIT could co-design an API into the departments travel report data that would be useful to them. Specifically, I could imagine that if TripIt could query all my trips against those reports then any time they notice I’m traveling somewhere the Foreign Ministry has labelled “exercise a high-degree of caution” or worse trip TripIt could ask me if I’d be willing to let them forward my itinerary to the department. That way I could registry my travel automatically, making the service more convenient for me, and getting the department more information that it believes to be critical as well.

Of course, it might be wise to work with the State Department so that their travel advisories used a similarly structured API (since I can assume TripIt will be more interested in the larger US market than the Canadian market) But facilitating that conversation would be nothing but wins for the department.

More bang for buck in election monitoring

One question that arose during my talk came from an official interested in elections monitoring. In my mind, one thing the department should be considering is a fund to help local democracy groups spin up installations of Ushahidi in countries with fragile democracies that are gearing up for elections. For those unfamiliar with Ushahidi it is a platform developed after the disputed 2007 presidential election in Kenya that plotted eyewitness reports of violence sent in by email and text-message on a google map.

Today it is used to track a number of issues – but problems with elections remain one of its core purposes. The department should think about grants that would help spin up a Ushahidi install to enable citizens of the country register concerns and allegations around fraud, violence, intimidation, etc… It could then verify and inspect issues that are flagged by the countries citizens. This would allow the department to deploy its resources more effectively and ensure that its work was speaking to concerns raised by citizens.

A Developer version of DART?

One of the most popular programs the Canadian government has around international issues is the Disaster Assistance Response Team (DART). In particular, Canadians have often been big fans of DART’s work in purifying water after the boxing day tsunami in Asia as well as its work in Haiti. Maybe the department could have a digital DART team, a group of developers that, in an emergency could help spin up Ushahidi, Fixmystreet, or OpenMRS installations to provide some quick but critical shared infrastructure for Canadians, other countries’ response teams and for non-profits. During periods of non-crisis the team could work on these projects or supporting groups like CrisisCommons or OpenStreetMaps, helping contribute to open source projects that can be instrumental in a humanitarian crisis.

Using Open Data to drive good policy outcomes – Vancouver’s Rental Database

And Now… Another Message on Open Innovation for Realtors

7 Replies

Over the past few months I’ve given a number of talks on open data and open innovation to groups of realtors around the country. During these talks I have cautioned that the more the real estate industry tries to protect (e.g. not share) its data, the more it risks making access to data (control) be the source of competition as opposed to accessibility with the data (allowing others to create value-added services).

Consequently, recreating already existing data sets will become the goal of competitors if working with the real estate industries data to innovate new services is not possible or prohibitively expensive. Competition on this axis has, I believe, two possible outcomes: One is a winner take all world where the person with the biggest data set wins, or put another way, either the current monopoly maintains its grip or a new monopoly takes over. The other is that there ceases to be, in a meaningful way, a single data repository on real estate and market place data gets broken into lots of different silos. This implications of this outcome are less clear, but there is a risk it could be bad for consumers as, essentially, market information would be fragmented. In either case, both outcomes carry significant risks for organized real estate.

Despite this, many realtors don’t believe it is likely to happen, because they don’t believe their data can be duplicated, no matter how much I try to tell them otherwise.

But…. whoops! Look what happened!

Today, as if to hammer home what I believe is the inevitable, the Globe and Mail has an article today on a new service that allows one to get an estimate on the assessed value of one’s home by accessing an alternative data set (one essentially created by the banks). It is basically a data set that is outside of any owned by organized real estate (that found in MLS). Look! Someone has recreated what was previously seen as a data set that could not be replicated!

Of course the counter is: “It isn’t as good.” Well, two things here. First, it may not have to be. If it offers 80% of the accuracy and 20% of the cost then it will probably be good enough for at least part of the market. And once it is established, I’m confident the owners of the website will find ways to make their service better and the data more accurate.

The real estate industry has an opportunity to shape its future or be shaped by the future. The market (and the competition bureau) isn’t going to give them for forever to make up their minds.

As Canada Searches for its Open Government Partnership Commitments: A Proposal

6 Replies

Just before its launch in New York on September 20th, the Canadian Government agreed to be a signatory of the Open Government Partnership (OGP). Composed of over 40 countries the OGP signatories are required to create a list of commitments they promise to implement. Because Canada signed on just before the deadline it has not – to date – submitted its commitments. As a result, there is a fantastic window for the government to do something interesting with this opportunity.

So what should we do? Here are the top 5 suggestions I propose for Canada’s OGP Commitments:

Brief Background on Criteria:

Before diving in, it is worth letting readers know that there are some criteria for making commitments. Specifically, any commitment must tackle at least one of the five “core” challenges: improve public services, increase public integrity, more effectively manage public resources, create safer communities, and increase corporate accountability.

In addition, each recommendation should reflect at least one of the core OGP principles, which are: transparency, citizen participation, accountability, and technology and innovation.

The Top Ten

Having reviewed several other countries commitments and being familiar with both what Canada has already done and what it could do, attached are 10 commitments I would like to see our government make to the OGP.

1. Be open about developing the commitments

Obviously there are a number of commitments the government is going to make since they are actions or programs that government was going to launch anyways. In addition, there will be some that will be new ideas that public servants or politicians have been looking for an opportunity to champion and now have an excuse. This is all fine and part of the traditional way government works.

But wouldn’t it be nice if – as part of the open government partnership – we asked citizens what they thought the commitments should be? That would make the process nicely consistent with the principles and goals of the OGP.

Thus the government should launch a two week crowd sourced idea generator, much like it did during the Digital Economy consultations. This is not suggestion that the ideas submitted must become part of the commitments, but they should inform the choices. This would be a wonderful opportunity to hear what Canadians have to say. In addition, the government could add some of its own proposal into the mix and see what type of response they get from Canadians.

2. Redefine Public as Digital: Pass an Online Information Act

At this year’s open government data camp in Warsaw, the always excellent Tom Steinberg noted that creating a transparent government and putting in place the information foundations of a digital economy will be impossible unless access to government data is not a gift from government (that can be taken away) but a right every citizen has. At the same time Andrew Rasiej of Tech President advocated that we must redefine public as digital. A paper print out in a small office in the middle of nowhere, does not make for “public disclosure” in the 21st century. It’s bad for democracy, it’s bad for transparency, and it is grossly inefficient for government.

Thus, the government should agree to pass a Online Information Act, perhaps modeled on that proposed in the US Senate, that

a) Any document it produces should be available digitally, in a machine readable format. The sham that the government can produce 3000-10,000 printed pages about Afghan detainees or the F-35 and claim it is publicly disclosing information must end.

b) Any data collected for legislative reasons must be made available – in machine readable formats – via a government open data portal.

c) Any information that is ATIPable must be made available in a digital format. And that any excess costs of generating that information can be born by the requester, up until a certain date (say 2015) at which point the excess costs will be born by the ministry responsible. There is no reason why, in a digital world, there should be any cost to extracting information – indeed, I fear a world where the government can’t cheaply locate and copy its own information for an ATIP request as it would suggest it can’t get that information for its own operations.

3. Sign the Extractive Industries Transparency Initiative

As a leader in the field of resource extraction it is critical that Canada push for the highest standards in a sector that all too often sees money that should be destined for the public good get diverted into the hands of a few well connected individuals. Canada’s reputation internationally has suffered as our extractive resource sector is seen as engaging in a number of problematic practices such as bribing public officials – this runs counter to the Prime Minister’s efforts to promote democracy.

As a result, Canada should sign, with out delay, the Extractive Industries Transparency Initiative, much like the United States did in September. This can help signal our desire for a transparent extractive industry, one in which we play a significant role.

4. Sign on to the International Aid Transparency Initiative

Canada has already taken significant steps to publishing its aid data online, in machine readable formats. This should be applauded. The next step is to do so in a way that conforms with international standards so that this data can be assessed against the work of other donors.

The International Aid Transparency Initiative (IATI) offers an opportunity to increase transparency in foreign aid, better enable the public to understand its aid budget, compare the country’s effectiveness against others and identify duplication (and thus poorly used resources) among donors. Canada should agree to implement IATI immediately. In addition, it should request that the organizations it funds also disclose their work in ways that are compliant with IATI.

5. Use Open Data to drive efficiency in Government Services: Require the provinces to share health data – particularly hospital performance – as part of its next funding agreement within the Canada Health Act.

Comparing hospitals to one another is always a difficult task, and open data is not a panacea. However, more data about hospitals is rarely harmful and there are a number of issues on which it would be downright beneficial. The most obvious of these would be deaths caused by infection. The number of deaths that occur due to infections in Canadian hospitals is a growing problem (sigh, if only open data could help ban the antibacterial wipes that are helping propagate them). Having open data that allows for league tables to show the scope and location of the problem will likely cause many hospitals to rethink processes and, I suspect, save lives.

Open data can supply some of the competitive pressure that is often lacking in a public healthcare system. It could also better educate Canadians about their options within that system, as well as make them more aware of its benefits.

6. Reduce Fraud: Find Fraud by Creating a Death List

In an era where online identity is a problem it is surprising to me that I’m unable to locate a database of expired social insurance numbers. Being able to querry a list of social security numbers that belong to dead people might be a simple way to prevent fraud. Interestingly, the United States has just such a list available for free online. (Side fact: Known as the Social Security Death Index this database is also beloved by genealogist who use it to trace ancestry).

7. Save lives by publishing a API of recall data

The only time the public finds out about a product recall is after someone has died. This is a terribly tragic, not to mention grossly inefficient, outcome. Indeed, the current approach is a classic example of using 21st century technology to deliver a service in a 19th century manner. If the government is interested in using the OGP to improve government services it should stop just issuing recall press releases and also create an open data feed of recalled products. I expand on this idea here.

If the government were doubly smart it would work with major retailers – particularly in the food industry – to ensure that they regularly tap into this data. In an ideal world any time Save-on-Foods, Walmart, Safeway, or any other retailers scans product in their inventory it would immediately check it against the recall database, allowing bad food to be pulled out of production before it hits the shelves. In addition, customers who use loyalty cards could be called or emailed to be informed that they had bought a product that had been recalled. This would likely be much more effective than hoping the media picks the story up.

8. Open Budget and Actual Spending Data

For almost a year the UK government has published all spending data, month by month, for each government ministry (down to the £500 in some, £25,000 in others). More over, as an increasing number of local governments are required to share their spending data it has lead to savings, as government begin to learn what other ministries and governments are paying for similar services.

Another bonus is that it becomes possible to talk about the budget in new and interesting ways. This BEAUTIFUL graphic was published in the Guardian, while still complicated it is much easier to understand than any government document about the budget I have ever seen.

9. Allow Government Scientists to speak directly to the media about their research.

It has become a reoccurring embarrassment. Scientists who work for Canada publish an internationally recognized ground break paper that provides some insight about the environment or geography of Canada and journalists must talk to government scientists from other countries in order to get the details. Why? Because the Canadian government blocks access. Canadians have a right to hear the perspectives of scientists their tax dollars paid for – and enjoy the opportunity to get as well informed as the government on these issues.

Thus, lift the ban that blocks government scientists from speaking with the media.

10. Create a steering group of leading Provincial and Municipal CIOs to create common schema for core data about the country.

While open data is good, open data organized the same way for different departments and provinces is even better. When data is organized the same way it makes it easier to citizens to compare one jurisdiction against another, and for software solutions and online services to emerge that use that data to enhance the lives of Canadians. The Federal Government should use its convening authority to bring together some of the countries leading government CIOs to establish common data schemas for things like crime, healthcare, procurement, and budget data. The list of what could be worked on is virtually endless, but those four areas all represent data sets that are frequently requested, so might make for a good starting point.

The State of Open Data 2011

17 Replies

What is the state of the open data movement? Yesterday, during my opening keynote at the Open Government Data Camp (held this year in Warsaw, Poland) I sought to follow up on my talk from last year’s conference. Here’s my take of where we are today (I’ll post/link to a video of the talk as soon as the Open Knowledge Foundation makes it available).

Successes of the Past Year: Crossing the Chasm

1. More Open Data Portals

One of the things that has been amazing to witness in 2011 is the veritable explosion of Open Data portals around the world. Today there are well over 50 government data catalogs with more and more being added. The most notable of these was probably the Kenyan Open Data catalog which shows how far, and wide, the open data movement has grown.

2. Better Understanding and More Demand

The things about all these portals is that they are the result of a larger shift. Specifically, more and more government officials are curious about what open data is. This is not to say that understanding has radically shifted, but many people in government (and in politics) now know the term, believe there is something interesting going on in this space, and want to learn more. Consequently, in a growing number of places there is less and less headwind against us. Rather than screaming from the rooftops, we are increasingly being invited in the front door.

3. More Experimentation

Finally, what’s also exciting is the increased experimentation in the open data space. The number of companies and organizations trying to engage open data users is growing. ScraperWiki, the DataHub, BuzzData, Socrata, Visua.ly, are some of the products and resources that have emerged out of the open data space. And the types of research and projects that are emerging – the tracking of the Icelandic volcano eruptions, the emergence of hacks and hackers, micro projects (like my own Recollect.net) and the research showing that open data could be generating savings of £8.5 million a year to governments in the Greater Manchester area, is deeply encouraging.

The Current State: An Inflection Point

The exciting thing about open data is that increasingly we are helping people – public servants, politicians, business owners and citizens imagine a different future, one that is more open, efficient and engaging. Our impact is still limited, but the journey is still in its early days. More importantly, thanks to success (number 2 above) our role is changing. So what does this mean for the movement right now?

Externally to the movement, the work we are doing is only getting more relevant. We are in an era of institution failure. From the Tea Party to Occupy Wall St. there is a recognition that our institutions no longer sufficiently serve us. Open data can’t solve this problem, but it is part of the solution. The challenge of the old order and the institutions it fostered is that its organizing principle is built around the management (control) of processes, it’s been about the application of the industrial production model to government services. This means it can only move so fast, and because of its strong control orientation, can only allow for so much creativity (and adaption). Open data is about putting the free flow of information at the heart of government – both internally and externally – with the goal of increasing government’s metabolism and decentralizing societies’ capacity to respond to problems. Our role is not obvious to the people in those movements, and we should make it clearer.

Internally to the movement, we have another big challenge. We are at a critical inflection point. For years we have been on the outside, yelling that open data matters. But now we are being invited inside. Some of us want to rush in, keen to make advances, others want to hold back, worried about being co-opted. To succeed, it is essential we must become more skilled at walking this difficult line: engaging with governments and helping them make the right decisions, while not being co-opted or sacrificing our principles. Choosing to not engage would, in my opinion, be to abscond from our responsibility as citizens and open data activists. This is a difficult transition, but it will be made easier if we at least acknowledge it, and support one another in it.

Our Core Challenges: What’s next

Looking across the open data space, my own feeling is that there are three core challenges that are facing the open data movement that threaten to compromise all the successes we’ve currently enjoyed.

1. The Compliance Trap

One key risk for open data is that all our work ends up being framed as a transparency initiative and thus making data available is reduced to being a compliance issue for government departments. If this is how our universe is framed I suspect in 5-10 years governments, eager to save money and cut some services, will choose to cut open data portals as a cost saving initiative.

Our goal is not to become a compliance issue. Our goal is to make governments understand that they are data management organizations and that they need to manage their data assets with the same rigour with which they manage physical assets like roads and bridges. We are as much about data governance as we are open data. This means we need to have a vision for government, one where data becomes a layer of the government architecture. Our goal is to make data platform one that not only citizens outside of government can build on, but one that government reconstructs its policy apparatus as well as its IT systems at top of. Achieving this will ensure that open data gets hardwired right into government and so cannot be easily shut down.

2. Data Schemas

This year, in the lead up to the Open Data Camp, the Open Knowledge Foundation created a map of open data portals from around the world. This was fun to look at, and I think should be the last time we do it.

We are getting to a point where the number of data portals is becoming less and less relevant. Getting more portals isn’t going to enable open data to scale more. What is going to allow us to scale is establishing common schemas for data sets that enable them to work across jurisdictions. The single most widely used open government data set is transit data, which because it has been standardized by the GTFS is available across hundreds of jurisdictions. This standardization has not only put the data into google maps (generating millions of uses everyday) but has also led to an explosion of transit apps around the world. Common standards will let us scale. We cannot forget this.

So let’s stop mapping open data portals, and start mapping datasets that adhere to common schemas. Given that open data is increasingly looked upon favourably by governments, creating these schemas is, I believe, now the central challenge to the open data movement.

3. Broadening the Movement

I’m impressed by the hundreds and hundreds of people here at the Open Data Camp in Warsaw. It is fun to be able to recognize so many of the faces here, the problem is that I can recognize too many of them. We need to grow this movement. There is a risk that we will become complacent, that we’ll enjoy the movement we’ve created and, more importantly, our roles within it. If that happens we are in trouble. Despite our successes we are far from reaching critical mass.

The simple question I have for us is: Where is the United Way, Google, Microsoft, the Salvation Army, Oxfam, and Greenpeace? We’ll know were are making progress when companies – large and small – as well as non-profits – start understanding how open government data can change their world for the better and so want to help us advance the cause.

Each of us needs to go out and start engaging these types of organizations and helping them see this new world and the potential it creates for them to make money or advance their own issues. The more we can embed ourselves into other’s networks, the more allies we will recruit and the stronger we will be.

eaves.ca

if writing is a muscle, this is my gym