Category Archives: public policy

StreetMix for testing bike lanes – Burrard St. Bridge Example

I’m MCing the Code for America Summit at the moment, so short on time to write a post, but I’m just LOVING StreetMix so much I had to give it a shout out. If you are a councillor, urban planner or community activist, StreetMix is a site you HAVE to check out.

What does it do? I basically allows you to create or edit and street you want. It is so simple to use it takes about 1 minute to master. At that point, you can build, copy and redesign any street in the world.

Here, for example I’ve recreated the Burrard St. Bridge in Vancouver as it exists today, with bike lanes and below, as it existed before the addition bike lane.

Burrard Bridge new

Burrard Bridge old

The 311 Open Data Competition is now Live on Kaggle

As I shared the other week, I’ve been working on a data competition with Kaggle and SeeClickFix involving 311 data from four cities: Chicago, New Haven, Oakland and Richmond.

So first things first – the competition is now live. Indeed, there are already 19 teams and 56 submissions that have been made. Fortunately, time is on your side, there are 56 days to go.

As I mentioned in my previous post on the subject, I have real hopes that this competition can help test a hypothesis I have about the possibility of an algorithmic open commons:

There is, however, for me, a potentially bigger goal. To date, as far as I know, predictive algorithms of 311 data have only ever been attempted within a city, not across cities. At a minimum it has not been attempted in a way in which the results are public and become a public asset.

So while the specific problem  this contest addresses is relatively humble, I’d see it as a creating a larger opportunity for academics, researchers, data scientists, and curious participants to figure out if can we develop predictive algorithms that work for multiple cities. Because if we can, then these algorithms could be a shared common asset. Each algorithm would become a tool for not just one housing non-profit, or city program but a tool for all sufficiently similar non-profits or city programs.

Of course I’m also discovering there are other benefits that arise out of these competitions.

This last weekend there was a mini-sub competition/hackathon involving a subset of the data. It was amazing to watch from afar. First, I was floored by how much cooperation there was, even between competitors and especially after the competition closed. Take a look at the forums, they are probably make one of the more compelling cases that open data can help foster more people to want to learn how to manipulate and engage with data. Here are contestants sharing their approaches and ideas with one another – just like you’d want them to. I’d known that Kaggle had a interesting community and that learning played an important role in it, but “riding along” in a mini competition has caused me to look again at the competitions through a purely educational lens. It is amazing how much people both wanted to learn and share.

As in the current competition, the team at the hackathon also ran a competition around visualizing the data. And there were some great visualization of the data that came out of it, as well as another example of where people were trying to learn and share. Here are two of my favourites:

map2

I love this visualization by Christoph Molnar because it reveals the different in request locations in each city. In some they are really dense, whereas in others they are much (more) evenly distributed. Super interesting to me.

Most pressing issues in each city

I also love the simplicity of this image created by miswift. There might have been other things I’d done, like colour coded similar problems to make them easier to compare across cities. But I still love it.

Congratulations to all the winners from this weekends event, and I hope readers will consider participating in the current competition.

Why Journalists Should Support Putting Access to Information Requests Online Immediately

Here’s a headline you don’t often expect to see: “Open-Government Laws Fuel Hedge-Fund Profits.”

It’s a fascinating article that opens with a story about SAC Capital Advisors LP – a hedge fund. Last December SAC Capital used Freedom of Information Laws (FOIA) to request preliminary results on a Vertex Pharmaceuticals drug being tested by the US Food and Drug Administration. The request revealed there were no “adverse event reports,” increasing the odds the drug might be approved. SAC Capital used this information – according to the Wall Street Journal – to snatch up 15,000 shares and 25,000 options of Vertex. In December – when the request was made – the stock traded around $40. Eight months later it peaked at $89 and still trades today at around $75. Thus, clever usage of government access to information request potentially netted the company a cool ROI of 100% in 9 months and a profit of roughly 1.2 million dollars (assuming they sold around $80).

This is an interesting story. And I fear it says a lot about the future of access to information laws.

This is because it contrasts sharply with the vision of access to information the media likes to portray: Namely, that access requests are a tool used mainly by hardened journalists trying to uncover dirt about a government. This is absolutely the case… and an important use case. But it is not the only usage of access laws. Nor was it the only intended use of the law. Indeed, it is not even the main usage of the law.

In my work on open data I frequently get pulled into conversations about access to information laws and their future. I find these conversations are aggressively dominated by media representatives (e.g. reporters) who dislike alternative views. Indeed, the one-sided nature of the conversation – with some journalists simply assuming they are the main and privileged interpreters of the public interest around access laws – is deeply unhealthy. Access to information laws are an important piece of legislation. Improving and sustaining them requires a coalition of actors (particularly including citizens), not just journalists. Telling others that their interests are secondary is not a great way to build an effective coalition. Worse, I fear the dominance of a single group means the conversation is often shaped by a narrow view of the legislation and with a specific set of (media company) interests in mind.

For example, many governments – including government agencies in my own province of British Columbia – have posted responses to many access to information requests publicly. This enrages (and I use that word specifically) many journalists who see it as a threat. How can they get a scoop if anyone can see government responses to their requests at the same time? This has led journalists to demand – sometimes successfully – that the requestor have exclusive access to government responses for a period of time. Oy vey. This is dangerous.

For certain types of stories I can see how complete transparency of request responses could destroy a scoop. But most stories – particularly investigative stories – require sources and context and understanding. Such advantages, I suspect, are hard to replicate and are the real source of competitive advantage (and if they aren’t… shouldn’t they be?).

It also suggests that a savvy public – and the media community – won’t be able to figure out who always seems to be making the right requests and reward them accordingly. But let’s put issues of a reputation economy and the complexity of reporting on a story aside.

First, it is worth noting that it is actually in the public interest to have more reporters cover a story and share a piece of news – especially about the government. Second, access to information laws were not created to give specific journalists scoops – they were designed to maximize the public’s capacity to access government information. Protecting a media company’s business model is not the role of access laws. It isn’t even in the spirit of the law.

Third, and worst, this entire debate fails to discuss the risks of such an approach. Which brings me back to the Wall Street Journal article.

I have, for years, warned that if public publication of access to information requests results are delayed so that one party (say, a journalist) has exclusive access for a period of time, then the system will also be used by others in pursuit of interests that might not be in the public good. Specifically, it creates a strong incentive for companies and investors to start mining government to get “exclusive” rights to government information they can put to use in advancing their agenda – making money.

As the SAC Capital Case outlined above underscores, information is power. And if you have exclusive access to that information, you have an advantage over others. That advantage may be a scoop on a government spending scandal, but it can also be a stock tip about a company whose drug is going to clear a regulatory hurdle, or an indication that a juicy government contract is about to be signed, or that a weapons technology is likely to be shelved by the defence department. In other words – and what I have pointed out to my journalist friends – exclusivity in access to information risks transforming the whole system into a giant insider information generation machine. Great for journalists? Maybe. (I’ve my doubts – see above.) But great for companies? The Wall Street Journal article shows us it already is. Exclusivity would make it worse.

Indeed, in the United States, the private sector is already an enormous generator of access requests. Indeed one company, that serves as a clearing house for requests, accounts for 10% of requests on its own:

The precise number of requests from investors is impossible to tally because many come from third-party organizations that send requests on behalf of undisclosed clients—a thriving industry unto itself. One of them, FOI Services Inc., accounted for about 10% of the 50,000 information requests sent to the FDA during the period examined by the Journal. Marlene Bobka, a senior vice president at Washington-based FOI Services, says a “huge, huge reason people use our firm is to blind their requests.”

Imagine what would happen if those making requests had formal exclusive rights? The secondary market in government information could become huge. And again, not in a way that advances the public interest.

In fact, given the above-quoted paragraph, I’m puzzled by the fact that journalists don’t demand that every access to information request be made public immediately. All told, the resources of the private sector (to say nothing of the tens of thousands of requests made by citizens or NGOs) dwarf those of media companies. Private companies may start (or already are) making significantly more requests than journalists ever could. Free-riding on their work could probably be a full time job and a successful career for at least a dozen data journalists. In addition, by not duplicating this work, it frees up media companies’ capacity to focus on the most important problems that are in the public good.

All of this is to say… I fear for a world where many of the journalists I know – by demanding changes that are in their narrow self-interest – could help create a system that, as far as I can tell, could be deeply adverse to the public interest.

I’m sure I’m about to get yelled at (again). But when it comes to access to information requests, we are probably going to be better off in a world where they are truly digitized. That means requests can be made online (something that is somewhat arriving in Canada) and – equally importantly – where results are also published online for all to see. At the very minimum, it is a conversation that is worth having.

New Zealand: The World’s Lab for Progressive Tech Legislation?

Cross posted with TechPresident.

One of the nice advantage of having a large world with lots of diverse states is the range of experiments it offers us. Countries (or regions within them) can try out ideas, and if they work, others can copy them!

For example, in the world of drug policy, Portugal effectively decriminalized virtually all drugs. The result has been dramatic. And much of it positive. Some of the changes include a decline in both HIV diagnoses amongst drug users by 17% and drug use among adolescents (13-15 yrs). For those interested you can read more about this in a fantastic report by the Cato Institute written by Glenn Greenwald back in 2009 before he started exposing the unconstitutional and dangerous activities of the NSA. Now some 15 years later there have been increasing demands to decriminalize and even legalize drugs, especially in Latin America. But even the United States is changing, with both the states of Washington and Colorado opting to legalize marijuana. The lessons of Portugal have helped make the case, not by penetrating the public’s imagination per se, but by showing policy elites that decriminalization not only works but it saves lives and saves money. Little Portugal may one day be remembered for changing the world.

I wonder if we might see a similar paper written about New Zealand ten years from now about technology policy. It may be that a number of Kiwis will counter the arguments in this post by exposing all the reasons why I’m wrong (which I’d welcome!) but at a glance, New Zealand would probably be the place I’d send a public servant or politician wanting to know more about how to do technology policy right.

So why is that?

First, for those who missed it, this summer New Zealand banned software patents. This is a stunning and entirely sensible accomplishment. Software patents, and the legal morass and drag on innovation they create, are an enormous problem. The idea that Amazon can patent “1-click” (e.g. the idea that you pre-store someone’s credit card information so they can buy an item with a single click) is, well, a joke. This is a grand innovation that should be protected for years?

And yet, I can’t think of single other OECD member country that is likely to pass similar legislation. This means that it will be up to New Zealand to show that the software world will survive just fine without patents and the economy will not suddenly explode into flames. I also struggle to think of an OECD country where one of the most significant industry groups – the Institute of IT Professionals appeared – would not only both support such a measure but help push its passage:

The nearly unanimous passage of the Bill was also greeted by Institute of IT Professionals (IITP) chief executive Paul Matthews, who congratulated [Commerce Minister] Foss for listening to the IT industry and ensuring that software patents were excluded.

Did I mention that the bill passed almost unanimously?

Second, New Zealanders are further up the learning curve around the dangerous willingness their government – and foreign governments – have for illegally surveilling them online.

The arrest of Kim Dotcom over MegaUpload has sparked some investigations into how closely the country’s police and intelligence services follow the law. (For an excellent timeline of the Kim Dotcom saga, check out this link). This is because Kim Dotcom was illegally spied on by New Zealand’s intelligence services and police force, at the behest of the United States, which is now seeking to extradite him. The arrest and subsequent fall out has piqued public interest and lead to investigations including the Kitteridge report (PDF) which revealed that “as many as 88 individuals have been unlawfully spied on” by the country’s Government Communications Security Bureau.

I wonder if the Snowden documents and subsequent furor probably surprised New Zealanders less than many of their counterparts in other countries since it was less a bombshell than another data point on a trend line.

I don’t want to overplay the impact of the Kim Dotcom scandal. It has not, as far as I can tell, lead to a complete overhaul of the rules that govern intelligence gathering and online security. That said, I suspect, it has created a political climate that amy be more (healthily) distrustful of government intelligence services and the intelligence services of the United States. As a result, it is likely that politicians have been more sensitive to this matter for a year or two longer than elsewhere and that public servants are more accustomed at policies through the lens of its impact on rights and privacy of citizens than in many other countries.

Finally, (and this is somewhat related to the first point) New Zealand has, from what I can tell, a remarkably strong open source community. I’m not sure why this is the case, but suspect that people like Nat Torkington – and open source and open data advocate in New Zealand – and others like him play a role in it. More interestingly, this community has had influence across the political spectrum. The centre left labour party deserves much of the credit for the patent reform while the centre-right New Zealand National Party has embraced both open data. The country was among the first to embrace open source as a viable option when procuring software and in 2003 the government developed an official open source policy to help clear the path for greater use of open source software. This contrasts sharply with my experience in Canada where, as late as 2008, open source was still seen by many government officials as a dangerous (some might say cancerous?) option that needed to be banned and/or killed.

All this is to say that in both the public (e.g. civil society and the private sector) and within government there is greater expertise around thinking about open source solutions and so an ability to ask different questions about intellectual property and definitions of the public good. While I recognize that this exists in many countries now, it has existed longer in New Zealand than in most, which suggests that it enjoys greater acceptance in senior ranks and there is greater experience in thinking about and engaging these perspectives.

I share all this for two reasons:

First, I would keep my eye on New Zealand. This is clearly a place where something is happening in a way that may not be possible in other OECD countries. The small size of its economy (and so relative lack of importance to the major proprietary software vendors) combined with a sufficient policy agreement both among the public and elites enables the country to overcome both internal and external lobbying and pressure that would likely sink similar initiatives elsewhere. And while New Zealand’s influence may be limited, don’t underestimate the power of example. Portugal also has limited influence, but its example has helped show the world that the US -ed narrative on the “war on drugs” can be countered. In many ways this is often how it has to happen. Innovation, particularly in policy, often comes from the margins.

Second, if a policy maker, public servant or politician comes to me and asks me who to talk to around digital policy, I increasingly find myself looking at New Zealand as the place that is the most compelling. I have similar advice for PhD students. Indeed, if what I’m arguing is true, we need research to describe, better than I have, the conditions that lead to this outcome as well as the impact these policies are having on the economy, government and society. Sadly, I have no names to give to those I suggest this idea to, but I figure they’ll find someone in the government to talk to, since, as a bonus to all this, I’ve always found New Zealanders to be exceedingly friendly.

So keep an eye on New Zealand, it could be the place where some of the most progressive technology policies first get experimented with. It would be a shame if no one noticed.

(Again If some New Zealanders want to tell me I’m wrong, please do. Obviously, you know your country better than I do).

Announcing the 311 Data Challenge, soon to be launched on Kaggle

The Kaggle – SeeClickFix – Eaves.ca 311 Data Challenge. Coming Soon.

I’m pleased to share that, in conjunction with SeeClickFix and Kaggle I’ll be sponsoring a predictive data competition using 311 data from four different cities. My hope is that – if we can demonstrate that there are some predictive and socially valuable insights to be gained from this data – we might be able to persuade cities to try to work together to share data insights and help everyone become more efficient, address social inequities and address other city problems 311 data might enable us to explore.

Here’s the backstory and some details in anticipation of the formal launch:

The Story

Several months back Anthony Goldbloom, the founder and CEO of Kaggle – a predictive data competition firm – approached me asking if I could think of something interesting that could be done in the municipal space around open data. Anthony generously offered to waive all of Kaggle’s normal fees if I could come up with a compelling contest.

After playing around with some ideas I reached out to Ben Berkowitz, co-founder of SeeClickFix (one of the world’s largest implementers of the Open311 standard) and asked him if we could persuade some of the cities they work for to share their data for a competition.

Thanks to the hard work of Will Cukierski at Kaggle as well as the team at SeeClickFix we were ultimately able to generate a consistent data set with 300,000 lines of data involving 311 issues spanning 4 cities across the United States.

In addition, while we hoped many of who might choose to participate in a municipal open data challenge would do so out curiosity or desire to better understand how cities work, both myself and SeeClickFix agreed to collectively put up $5000 in prize money to help raise awareness about the competition and hopefully stoke some media (as well as broader participant) interest.

The Goal

The goal of the competition will be to predict the number of votes, comments and views an issue is likely to generate. To be clear, this is not a prediction that is going to radically alter how cities work, but it could be a genuinely useful to communications departments, helping them predict problems that are particularly thorny or worthy proactively communicating to residents about. In addition – and this remains unclear – my own hope is that it could help us understand discrepancies in how different socio-economic or other groups use online 311 and so enable city officials to more effectively respond to complaints from marginalized communities.

In addition there will be a smaller competition around visualization the data.

The Bigger Goal

There is, however, for me, a potentially bigger goal. To date, as far as I know, predictive algorithms of 311 data have only ever been attempted within a city, not across cities. At a minimum it has not been attempted in a way in which the results are public and become a public asset.

So while the specific problem  this contest addresses is relatively humble, I’d see it as a creating a larger opportunity for academics, researchers, data scientists, and curious participants to figure out if can we develop predictive algorithms that work for multiple cities. Because if we can, then these algorithms could be a shared common asset. Each algorithm would become a tool for not just one housing non-profit, or city program but a tool for all sufficiently similar non-profits or city programs. This could be exceptionally promising – as well as potentially reveal new behavioral or incentive risks that would need to be thought about.

Of course, discovering that every city is unique and that work is not easily transferable, or that predictive models cluster by city size, or by weather, or by some other variable is also valuable, as this would help us understand what types of investments can be made in civic analytics and what the limits of a potential commons might be.

So be sure to keep an eye on the Kaggle page (I’ll link to it) as this contest will be launching soon.

Beyond Property Rights: Thinking About Moral Definitions of Openness

“The more you move to the right the more radical you are. Because everywhere on the left you actually have to educate people about the law, which is currently unfair to the user, before you even introduce them to the alternatives. You aren’t even challenging the injustice in the law! On the right you are operating at a level that is liberated from identity and accountability. You are hacking identity.” – Sunil Abraham

I have a new piece up on TechPresident titled: Beyond Property Rights: Thinking About Moral Definitions of Openness.

This piece, as the really fun map I recreated is based on a conversation with Sunil Abraham (@sunil_abraham), the Executive Director of the Centre for Internet and Society in Bangalore.

If you find this map interesting… check the piece out here.

map of open

 

OGP Rules of the Game – Tactical Mistake or Strategic Necessity?

The other week Martin Tisne, the UK Policy Director at the Omidyar Network, as well as one of the key architects of the Open Government Partnership (OGP), posted a blog post expressing concern that Civil Society participants have misunderstood the OGP. Specifically Tisne is concerned that by focusing on entrance into the OGP rather than on the process which requires them to fulfill commitments towards greater transparency, NGOs are making a tactical mistake.

There is a tremendous amount of good insight in Tisne’s piece and it deserves to be widely read (and has been). There are however, important reasons civil society members spend as much time fretting about entrance into the OGP rather than purely on the process. And contrary to Tisne, I don’t think this is a tactical mistake – it is, in fact, both a tactically and strategically sound choice. Most importantly of all it is a reflection of how power is structured and distributed within the OGP.

For most activists fostering change is about a developing a set of carrots and sticks that can be used to cajole a reluctant actor into making the change you seek. One big carrot is participation in the OGP. This is good. It urges governments to make commitments and sign on to a process. However, it also has a serious impact on civil society’s power in the process. This is because it puts one major carrot – participation – at the beginning of the process while placing the stick – an assessment of how well a government is adhering to its commitments – at the end.

We shouldn’t underestimate the benefit participation confers on many governments. The OGP brand can become a sort of shield that protects a government against all sorts of accusations of opacity. “Of course we are transparent, we participate in the OGP” is an easy line for minister to counter to an uncomfortable question. And that is not the only way participation can diminish civil society’s power. Because a government’s necessarily requires civil society cooperation (they sign off on the commitments), it binds the two together. This means that, in some basic way, civil society has endorsed a – yet to be implemented – government plan. That can provide enormous political cover. In addition, OGP members may cause some citizens (e.g. potential transparency supporters and activists) to adopt a “wait and see” approach to judging their government, or to assume that a reliable process is in place and so they can focus on other issues. Rather than maintain or intensify pressure on a government, the OGP, in the short term, may diminish the power of civil society.

The aforementioned stick in the OGP process is the independent reporting mechanism. And it arrives at the end of the process, a couple of years after the country has joined the OGP. The hope is it provides an objective assessment that civil society members can use to shame and drive for change where the assessment is critical. The challenge, and the reason I suspect many civil society members remain nervous, is that this mechanism remains mostly untested. The OGP carrot and stick model becomes even more challenging if either a) the timeline for fulfilling commitments falls onto the term of the next government or b) a transparency issue arises that runs counter to the OGPs values but falls outside the government’s action plan.  This is what happened in South Africa and so calling for ejection from the OGP became rational (and even necessary) since both the short term carrot (OGP participation) and long term stick – are review of the implementation plan – provided civil society with no leverage or power against a law that distinctly ran counter to the OGPs principles.

Consequently, the threat of striping a government of its OGP membership is not only a rational choice for many civil society members, in some cases it may be one of the few sticks available to them during a period in the process when other forms of influence have been made less effective. Threats of ejection is this not only a rational choice, but possible the only choice.

Indeed, OGP architects should take heart of the fact that civil society members are relatively hawkish about who gets to enter the OGP. As previously mentioned, OGP membership itself denotes a degree credibility – particularly to an unaware public. Civil society members bound to the OGP are potentially more invested in protecting the credibility and brand of the OGP than either the member governments of the OGP secretariat is. This is because, try as the OGP might to not compare countries to one another, civil society members know the company you keep matters.

This is not to say that the OGP should only be a high achievers club. I think the public understands there are differences in capacity, and the entrance of a country like Libya that is making a difficult transition, is broadly seen as positive. However, the participation of an authoritarian government, or even a democracy infamous for jailing journalists, significant corruption and little transparency – damages the the OGP brand for all participants, and particularly for civil society members participating in the process. I can only imagine the Executive Director of a civil society group grimacing as someone asks incredulously: “you are part of a transparency group that includes (insert country with poor record of your choice)?” Civil society actors that are the most invested in protecting the OGP’s brand, if only to ensure that the IRM has credibility when it is finally launched in their country. As such, protesting the potential entry of a country is not a tactical mistake, but a highly strategic decision.

I say this not because Martin is wrong, especially about his four points – civil society participation, OGP stretch goals, relevance check and the IRM – these are indeed critical to the bedrock of the OGP. And I remain exceedingly hopeful about the OGP, although a great deal hinges on the IRM and the degree with which it empowers local civil society actors. Rather I think it bears reminding all involved that we need to continuously have explicit and productive conversations about power, and how it is structured and where it flows, when it comes to the OGP process, as this reveals a lot about why actors act the way they do, and could provide insights in how we can make the OGP more effective.

The Uncertain Future of Open Data in the Government of Canada

It is possible to state that presently, open data is at its high water mark in the Government of Canada. Data.gc.ca has been refreshed, more importantly, the government has signed the Open Data Charter committing it to making data “open” by default, and a rash of new data sets have been made available.

In other words there is a lot of momentum in the right direction. So what could go wrong.

The answer…? Everything.

The reason is the upcoming cabinet shuffle.

I confess that Minister Clement and I have not agreed on all things. I believe – like the evidence shows us – that needle injection sites such as Insite make communities safer, save lives and make it easier for drug users to get help. As Health Minister, Clement did not. I argued strongly against dismantling of the mandatory long form census, noting its demise would make our government dumber and, ultimately, more expensive. As Industry Minister, Minister Clement was responsible for the end of a reliable long form census.

However, when it comes to open data, Minister Clement has been a powerful voice in a government that has, on many occasions, looked for ways to make access to information harder, not easier. Indeed, open data advocates have been lucky to have had two deeply supportive ministers, Clement and, prior to him, Stockwell Day (who also felt strongly about this issue and was incredibly responsive to many of my concerns when I shared them). This run, though, may be ending.

With the Government in trouble there is wide spread acceptance that a major cabinet re-shuffle will be in order. While Minister Clement has been laying a lot of groundwork for the upcoming negotiations with the public sector unions and a rethink of the public service could be more effective and accountable, he may not be sticking around to see this work (that I’m sure the government sees as essential) through to the end. Nor may he want to. Treasury Board remains a relatively inward facing ministry and it would not surprise me if both the Minister, and the PMO, were interested in moving him to a portfolio that was more outward and public facing. Only a notable few politicians dream of wrestling with public servants and figuring out how to reform the public service. (Indeed Reg Alcock is the only one I can think of).

If the Minister is moved it will be a real test for the sustainability of open data at the federal level. Between the Open Data charter, the expertise and team built up within Treasury Board and hopefully some educational work Minister Clement has done within his own caucus, ideally there is enough momentum and infrastructure in place that the open data file will carry on. This is very much what I hope to be the case.

But much may depend on who is made President of the Treasury Board.  If that role changes open data advocates may find themselves busy not doing new things, but rather safe guarding gains already made.

 

Some thoughts on the relaunched data.gc.ca

Yesterday, I talked about what I thought was the real story that got missed in the fanfare surrounding the relaunch of data.gc.ca. Today I’ll talk about the new data.gc.ca itself.

Before I begin, there is an important disclaimer to share (to be open!). Earlier this year Treasury Board asked me to chair five public consultations across Canada to gather feedback on both its open data program and data.gc.ca in particular. As such, I solicited peoples suggestions on how data.gc.ca could be improved – as well as shared my own – but I was not involved in the creation of data.gc.ca. Indeed the first time I saw the site was on Tuesday when it launched. My role was merely to gather feedback. For those curious you can read the report I wrote here

There is, I’m happy to say, much to commend about the new open data portal. Of course, aesthetically, it is much easier on the eye, but this is really trivial compared to a number of other changes.

The most important shift relates to the desire of the site to foster community. Users can now register with the site as well as rate and comment on data sets. There are also places like the Developers’ Corner which contains documentation that potential users might find helpful and a sort of app store where government agencies and citizens can posts applications they have created. This shift mirrors the evolution of data.govdata.gov.uk and DataBC which started out as data repositories but sought to foster and nurture a community of data users. The critical piece here is that simply creating the functionality will probably not be sufficient, in the US, UK and BC it has required dedicated community managers/engagers to help foster such a community. At present it is unclear if that exists behind the website at data.gc.ca.

The other two noteworthy improvements to the site are an improved search and the availability of API’s. While not perfect, the improved search is nonetheless helpful as previously it was basically impossible to find anything on the site. Today a search for “border time” and a border wait time data set is the top result. However, search for “border wait times” and “Biogeochemical exploration using Douglas-fir tree tops in the Mabel Lake area, southern British Columbia (NTS 82L09 and 10)” becomes the top hit with actual border wait time data set pushed down to fifth. That said the search is still a vast improvement and this alone could be a boon to policy wonks, researchers and developers who elect to make use of the site.

The introduction of APIs is another interesting development. For the uninitiated an API (application programming interface) provides continuous access to updated data, so rather than downloading a file, it is more like you are plugging into a socket that delivers data, rather than electricity. The aforementioned border wait time data set is a fantastic example. It is less of a “data set” than of a “data stream” providing the most recent updates of border wait times, like what you would see on the big signs across the highway as you approach the border. By providing it through the open data site it would not, for example, be impossible for Google Maps to scan this data set daily, understand how border wait times fluctuate and incorporate these delays in its predicted travel times. Indeed, it could even querry the API  in real time and tell you how long it will take to drive from Vancouver to Seattle, with border delays taken into account. The opportunity for developers and, equally intriguing, government employees and contractors, to build applications a top of these APIs is, in my mind, quite exciting. It is a much, much cheaper and flexible approach than how a lot of government software is currently built.

I also welcome the addition of the ability to search Access to Information (ATIP) requests summaries. That said, I’d like for there to be more than just the summaries, that actually responses would be nice, particularly given that ATIP requests likely represent information people have identified as important. In addition, the tool for exploring government expenditures is interesting, but it is weirdly more notable because, as far as I can tell, none of the data displayed in the tool can be downloaded, meaning it is not very open.

Finally, I will briefly note that the license is another welcome change. For more on that I recommend checking out Teresa Scassa’s blog post on it. Contrary to my above disclaimer I have been more active on this side of things, and hope to have more to share on that another time.

I’m sure, as I and others explore the site in the coming days we will discover more to like and dislike about it, but it is a helpful step forward and another signal that open data is, slowly, being baked into the public service as a core service.

 

The Real News Story about the Relaunch of data.gc.ca

As many of my open data friends know, yesterday the government launched its new open data portal to great fanfare. While there is much to talk about there – something I will dive into tomorrow – that was not the only thing that happened yesterday.

Indeed, I did a lot of media yesterday between flights and only after it was over did I notice that virtually all the questions focused on the relaunch of data.gc.ca. Yet it is increasingly clear that for me, the much, much bigger story of the portal relaunch was the Prime Minister announcing that Canada would adopt the Open Data Charter.

In other words, Canada just announced that it is moving towards making all government data open by default. Moreover, it even made commitments to make specific “high value” data sets open in the next couple of years.

As an aside, I don’t think the Prime Minister’s office has ever mentioned open data – as far as I can remember, so that was interesting in of itself. But what is still more interesting is what the Prime Minister committed Canada to. The open data charter commits the government to make data open by default as well as four other principles including:

  • Quality and Quantity
  • Useable by All
  • Releasing Data for Improved Governance
  • Releasing Data for Innovation

In some ways Canada has effectively agreed to implement the equivalent to Presidential Executive Order on Open Data the White House announced last month (and that I analyzed in this blog post). Indeed, the charter is more aggressive than the executive order since it goes on to layout the need to open up not just future data, but also current “high value” data sets. Included among these are data sets the Open Knowledge Foundation has been seeking to get opened via its open data census, as well as some data sets I and many others have argued should be made open, such as the company/business register. Other suggested high value data sets include data on crime, school performance, energy and environment pollution levels, energy consumption, government contracts, national budgets, health prescription data and many, many others. Also included on the list… postcodes – something we are presently struggling with here in Canada.

But the charter wasn’t all the government committed to. The final G8 communique contained many interesting tidbits that again, highlighted commitments to open up data and adhere to international data schemas.

Among these were:

  • Corporate Registry Data: There was a very interesting section on “Transparency of companies and legal arrangements” which is essentially on sharing data about who owns companies. As an advisory board member to OpenCorporates, this was music to my ears. However, the federal government already does this, the much, much bigger problem is with the provinces, like BC and Quebec that make it difficult or expensive to access this data.
  • Extractive Industries Transparency Initiative: A commitment that “Canada will launch consultations with stakeholders across Canada with a view to developing an equivalent mandatory reporting regime for extractive companies within the next two years.” This is something I fought to get included into our OGP commitment two years ago but failed to succeed at. Again, I’m thrilled to see this appear in the communique and look forward to the government’s action.
  • International Aid Transparency Initiative (IATI) and Busan Common Standard on Aid Transparency,: A commitment to make aid data more transparent and downloadable by 2015. Indeed, with all the G8 countries agreed to taking this step it may be possible to get greater transparency around who is spending what money, where on aid. This could help identify duplication as well as in assessments around effectiveness. Given how precious aid dollars are, this is a very welcome development. (h/t Michael Roberts of Acclar.org)

So lots of commitments, some on the more vague side (the open data charter) but some very explicit and precise. And that is the real story of yesterday, not that the country has a new open data portal, but that a lot more data is likely going to get put into that portal over then next 2-5 years. And a tsunami of data could end up in it over the next 10-25 years. Indeed, so much data, that I suspect a portal will no longer be a logical way to share it all.

And therein lies the deeper business and government story in all this. As I mentioned in my analysis of the White House Executive Order that made open data default, the big change here is in procurement. If implemented, this could have a dramatic impact on vendors and suppliers of equipement and computers that collect and store data for the government. Many vendors try to find ways to make their data difficult to export and share so as to lock the government in to their solution. Again, if (and this is a big if) the charter is implemented it will hopefully require a lot of companies to rethink what they offer to government. This is a potentially huge story as it could disrupt incumbents and lead to either big reductions in the costs of procurement (if done right) or big increases and the establishment of the same, or new, impossible to work with incumbents (if done incorrectly).

There is potentially a tremendous amount at stake in how the government handles the procurement side of all this, because whether it realizes it or not, it may have just completely shaken up the IT industry that serves it.

 

Postscript: One thing I found interesting about the G8 communique was how many times commitments about open data and open data sets occurred in the section that had nothing to do with open data. Will be interesting if that is a trend that continues at the next G8 meeting. Indeed, I wouldn’t be surprised is a specific open data section disappears and instead these references just become part of various issue related commitments.