Canada’s Opaque Transparency – An Open Data Failure

Yesterday, at the Prospectors and Developers Association of Canada’s (PDAC) Canada Minister of Natural Resource, Joe Oliver, announced with great fanfare a new initiative to compel mining companies to disclose payments of over $100,000′s to foreign and domestic governments.

On the surface this looks like a win for transparency, particularly for a sector that is of great importance to Canada: mining.

And this issue matters since not only do extractive industries represent an important part of Canada’s economy, but the sector has been dogged with controversy. Indeed the Toronto Star just uncovered today a report commissioned (and buried) by the Prospectors and Developers Association of Canada (PDAC) that showed Canadian mining companies have the worst record when it comes to environmental standard and human rights.

Forcing mining companies to account for their payments to foreign and domestic governments won’t solve every problem, but it can help curb corruption. Indeed the issue was seen as so important that at the last G8 summit, the leaders agreed that companies should be compelled to disclose these payments.

Happily, there is a legitimate global movement to make government payments by extractive industry companies more transparent. It is called the Extractive Industries Transparency Iniative (EITI). It has set a series of standards for disclosing such payments so that they are easier to track across borders. In fact EITI is seen as so important it is actually the only organization mentioned by name in the last G8 summit communique. This is the same EITI program about which last year the Minister’s press secretary boasted:

Since 2007 Canada has also been a supporting country of the Extractive Industries Transparency Initiative, and is now the second largest financial donor to the initiative, providing $12.65 million to the World Bank’s Extractive Industries Transparency Initiative Multi-donor Trust Fund…

Which brings us to Minister Oliver’s important announcement.

Did the government announce that it was joining 42 other countries, including its G8 partners the United States, the United Kingdom, France, Germany and Italy to join the standard it has been a major funder of?

No. It did not.

Apparently EITI is good enough to fund so that others can implement it. When it comes to actually doing what is effective… the government balked. Canada, apparently, is going to adhere to its own “unique” approach.

And it gets worse.

Read the Minister’s statement more closely, particularly this line:

“We want to make it as easy as possible, so we will not create a central database. Instead, we would require that reports be posted to company websites, with the government and public notified.”

So unlike EITI, which offers a centralized repository where records can quickly be downloaded and compared, Canada’s “compliance” will involve each company to maintain their own records “somewhere” and will require anyone interested if actually figuring out what is going on to go and track down each one individually.

We call this secrecy by obscurity. It makes a mockery of the notion of transparency.

We have a global infrastructure designed to make disclosure cheap, easy and effective. Infrastructure our own government has poured $12.7M into it. And we turn around and ignore it all.

Canada claims it wants to be a leader in open data. But if it can’t even get something basic like this right… such claims sounds increasingly silly here at home, among our G8 partners and, well, among the rest of the world.

Addendum: It gets worse still. Few people have noticed yet, but Canada recently (and quietly) stopped reporting the names of corporate directors in the public database of the country’s firms. This is a major step backwards and makes those who benefit from one of the most important benefits society can confer – limited liability – invisible to the public who confers that right. This is a major step backwards. Read this wonderful Economist article on why. More on this to come.

Great Hacks from the Open Data in Vancouver

Last weekend I helped host an Open Data Day in Vancouver. With the generous support of Domain7, who gave us a place to host talks and hack, over 30 Vancouverites braved the sleet and snow to spend the day sharing ideas and working on projects.

We had opening comments from Andy Yan – whose may be the most prolific user of Open Data in Vancouver, possibly Canada. I encourage you to check out his work here. We were also incredibly lucky to have Jeni Tennison – the Technical Director of the Open Data Institute – onsite to talk to participants about the ODI.

After the opening talks, people simply shared what they hoped to work on and people just found projects to contribute to. Minimal organization was involved… and here a taste of the awesome projects that got worked on! Lots of ideas here for other communities.

1. Open Data Licenses Resource: JSON + search + compatibility check = Awesome.

Kent Mewhort, who recently moved to Vancouver from Ottawa (via the Congo) updated his ongoing CLIPol project by adding some of the recently published licenses. If you’ve not seen CLIPol it is… awesome. It allows you to easily understand and compare the restrictions and rights of many open government licenses.

CLIPol Data

Better still CLIPol also lets you to see how compatible a license is (see example here). Possibly the best tool of all is one that allows you to determine what license you can apply to your re-mixed work in a way that is compliant with the original licenses (check out that tool here - screenshot below).

CLIPol compatibility

CLIPol is just such a fantastic tool – can’t recommend it enough and encourage people to add more licenses to it.

2. Vancouver in MineCraft

I have previously written about how Minecraft is being used to help in public consultations and urban planning - I love how the game becomes a simple tool that enables anyone to shape the environment.

So I was crazy excited I heard that Ryan Smith (aka Goldfish) had used the City of Vancouver’s open elevation data to recreate much of the city in Minecraft.

Below is a photo of Ryan presenting at the end of the day. The projection behind him shows Stanley park, near Siwash Rock. The flat feature at the bottom is the sea wall. Indeed Ryan notes that the sea wall makes for one of the clearest features since it creates almost perfectly flat structure along the city’s coast.

Mincraft Data

3. Vancouver’s Capital Budget Visualized in Where Does my Money Go

It is hard to imagine a project going better. I’m going to do a separate blog post on it.

This is a project I’ve always wanted to do – create a bubble tree visualization with Where Does my Money Go. Fortunately two developers – Alexandre Dufournet and Luc Lussier – who had never hacked on open data jumped on the idea. With help from City of Vancouver’s staff who were on site, I found a PDF of the capital budget which we then scraped.

WDMYG Data

The site is not actually live, but for developers who are interested in seeing this work (hint, hint City of Vancouver staff) you can grab their code from github here.

4. Monitoring Vancouver’s Bike Accident Data – Year 3

Eric Promislow has been coming to Open Data Hack-a-thons ever since Luke Closs and I started organizing them in 2009. During the first Open Data Day in 2011 you can read in my wrap up post about a bike accident monitoring website Eric created that day which Eric would eventual name Bent Frame. Well, Bent Frame has been live ever since and getting bigger. (Eric blogs about it here)

Each open data day, Eric updates Bent Frame with new data from ICBC – the province’s insurance monopoly. With over 6 years of data now in Eric is starting to be able to analyze trends – particularly around the decline of bike accidents along many roads with bike lanes, and an increase in accidents where the bike lanes end.


Bike Data

I initially had conversations with ICBC to persuade them to share their data with Eric and they’ve been in touch with him ever since, passing along the data on a regular basis. It is a real example of how an active citizen can change an organization’s policies around sharing important data that can help inform public policy debates.

5. ProactiveDisclosure.ca – Making government information easier to search

Kevin McArthur is the kind of security guy most governments dreads having around but should actually love (example his recent post on e-voting).  He continued to hack on one of his side projects: proactivedisclosure.ca. The site is a sort of front end for open data sets, making it easier to do searches based on people or companies. Thus, want to find all the open data about a specific minister… proactive disclosure organizes it for you.

Proactive Data

Kevin and a small team of players uploaded more data into their site and allowed it to consume unstructured data. Very cool stuff.

6. Better Open Data Search

Herb Lainchbury – another fantastic open data advocate – worked on a project in which he tried to rethink what an open data search engine would look like. This is a topic that I think matters A LOT. There is simply not a lot of good ways to find data that you are interested in.

Herb’s awesome insight was invert the traditional way of thinking about data search. He created a search engine that didn’t search for the data set keywords or titles, but rather searched the meta data exclusively.

One interesting side outcome of this approach is that it made related data sets easier and, made locating identical data sets but from different years a snap. As Herb notes the meta data becomes a sort of “finger print” that makes it easy to see when it has been duplicated. (Quick aside rant: I loath it when governments releases 20 data files of the same data set – say crime data – with each file representing a different year and then claiming that it is 20 unique data sets in their catalogue. No. It is one data set. You just have 20 years of it. Sigh).

7. School Performance Chart

Two local video game programers – Louie Dinh and Raymond Huang – with no experience in open data looked around the BC Government Open Data catalogue and noticed the data on test scores. Since they attended school here in British Columbia they thought it might be interesting to chart the test scores to see how their own schools had preformed over time.

They were able to set up a site which graphed how a number of elementary schools had performed over time by looking at the standardized test scores.

Test SCore Data

This is just a great example of data as a gateway to learning. Here a simple hackathon project become a bridge for two citizens to dive into a area of public policy and learn more about it. No one is claiming that there chart is definitive, rather it is the start of a learning process around what matters and what doesn’t and what can be measured and what can’t in education.

Congratulations to everyone who participated in the day – thank you for making it such an amazing success!

Open Data Day 2014 – Five Fun Events Around the World

With over 110 Events happening world wide it is impossible to talk about every Open Data Day event. But looking almost every event on the wiki I’ve been deeply moved and inspired by the various efforts, goals and aspirations of the people who have organized these events.

In order to help others understand why Open Data Day matters as well as what can happen on it, here are five open data day events that I’m stumbled across that are doing something particularly fun or interesting.

1. Capetown & Johannesburg, South Africa

Their Description:

Coders, data wranglers and data investigators will pair up to look at one of three openly available datasets we have on hand, and work out the most interesting questions they can ask of it in less than half an hour. After 30 minutes, it’s all change – another desk, another partnership, another exciting data set to turn into a story.

What we’re hoping is that you’ll learn tips and tricks for getting data, querying it, creating quick visualizations and turning it to stories that people want to know about. You’ll learn from different people with a variety of skills, hopefully that you wouldn’t normally work with. And we’re also hoping it will be four hours of fun.

Why I love it: I love the focus on learning. With the participation of hacks and hackers the goal is clearly to help journalists and citizens learn new skills, not so they can do something with the data sets available on open data day, but so they can better play with data sets in the future to pursue stories or help a community. The point of speed data dating is thus not to build a product, the product is the skills and networks developed and, with luck the future stories and analyses that will be told by those who participated.

2. Buenos Aires, Argentina

Their description:

On February 22nd we will go out to the street and play with local data and some street artists to create beautiful visualizations.

Why I love it: Street art open data? What a great way to try to raise awareness of the importance of data literacy and transparency. In addition, how awesome is it to move outside the digital realm and use data to create artifacts that are not necessarily digital. And if there are artists involved? Jer would be so happy to read about this.

3. Greenfield, MA, United States

Their description:

We’re convening a small group to work with the Franklin Regional Council of Governments on a user-friendly way to map private wells in Western Massachusetts…

…Why is it important to map wells?

  • Only about 5% of private wells in Massachusetts are geolocated.
  • Many towns in Western Mass rely 100% on private wells.

Not knowing where our wells are can (and does) lead to water contaminated by nearby septic systems, dumping, and pollutant storage. Aside from the obvious health concerns, there are also financial implications from remediation costs and lowered property values.

Why I love it: Wow, WOW, WOW!!! This is maybe one of the coolest open data day events I’ve ever seen. Here you have a small community focusing on a problem that is real and tangible to them. Moreover, open data could have a direct and meaningful impact on the issue. I love the focus. I love that rallying point. I love the high impact with low resources (their building has minimal heat – so they are advising people to layer up). I wish these crew all the best success and hope to see an update.

4. Nagoya, Japan

Their Description:

Now, highlight of this year is “data of Nagoya Castle!”

The nearly 300 maps and survey drawings of the Nagoya Castle will be made open prior to Open Data Day. The Nagoya Castle office is cooperating with us and has decided that we can use their data for “International Open Data Day.” The references to the image are here.

Why I love it: Well – fill disclosure, my understanding of this event is through the prism of Google translate. But if I understood correctly… there are a few open data events in Japan that have a strong focus on local history which I find totally fascinating. At this event in Nagoya they are bringing in a professor who is an expert in open data as well as expert in the Nagoya castle to talk about the data that is being made open. In addition they are organizing and actual physical tour of the caste. Open Data meets local history buffs!

5. Cairo, Egypt

Their Description:

We will be organizing an online and decentralized event in Cairo, Egypt for the Open Data Day. There are numerous suggested tracks depending on the participants set of expertise:

For translators (المترجمون)

  • Open Data Handbook: The handbook discusses the legal, social and technical aspects of open data. It can be used by anyone but is especially designed for those seeking to open up data. It discusses the why, what and how of open data – why to go open, what open is, and the how to ‘open’ data. Translate it into Arabic here
  • Translate any of the School of Data short tutorials, for example, What is Data?Telling a Story with DataFinding Data, or Any other course/modules

For bloggers (المدونون)

Write blog posts about Open Data related topics and case studies, and don’t forget to use the following hashtag, #ODD2014. Possible ideas for blog posts:

  • Write about the concept of Frictionless Data
  • Case studies how you searched for, extracted and used governmental data
  • Listing of local organizations working or promoting Open Data or advocating for more Transparent and Open Governments

For Developers (مطوري البرامج)

  • Scrape data from capmas and put it into Open Format
  • Scrape data (Budget or the The Financial Monthly Bulletin) from the Minister of Finance and upload it to OpenSpending.org,
  • Create a tool to scrap the traffic data from bey2ollak and put it in an open format.

For Data Wranglers (هواة جمع البيانات)

Why I love it: I love that there are calls to action for a variety of people – including those who have no coding skills at all. How genius is it to organize an event to localize/translate the Open Data handbook? This is something a large number of people could do – and better still can help make open data accessible to a still larger pool of people.

And for the other roles the suggestion of projects – particular with a focus on the national budget and government operations data (capmas) suggests there is a strong civil society presence within the open data community. Will be super interested to see what progress they make and if there is broader interest in their work.

Open Data Day in 110 cities Worldwide! Here’s 6 things to do

It is, as always, with a fair amount of wonder that I watch the open data day wiki grow each year. This year there are 100 self organized events taking place worldwide (at last count). It is an impressive number. This includes events in places like Buenos Aires (which is doing open data street art), Ilorin Nigeria, Kampala Uganda, Barcelona Spain and Aomori Japan to name just a few.

Many of these events have speakers lined up, projects to hack on, seminars and workshops organized. Others are straightforward hackathons and/or unconferences.

If you are heading to an Open Data Day event be sure to take a look at the local events page on the wiki or local website to see what is planned. But if you are looking to get inspired… here are 6+ ideas for what to do on open data day.

1. Use Where Does My Money Go to create a simple visualization of a government budget.

WDMMG Ontario

I love where does my money go. It is just so simple and makes budgets much easier to understand. I persuaded the Ontario Government to use it when it released its budget a few months ago.

The nice part is… it is open source. Check out the code here. No reason why a small team couldn’t fork it and enter in new numbers to visualize the local, regional, state or national budget near where you live!

2. Don’t fixate on an app – fixate on your community

Metro_YVR_Schools2008_20131-533x411Here in Vancouver I’m inviting Andy Yan of the local non-proft BTAWorks to give some opening remarks at our open data day. Why? Because Andy is a legendary user of local open data. And he doesn’t build apps – instead he uses data to better explain Vancouver to Vancouverites.

Andy uses simple mapping software, and even sometimes just plain old excel to take data and analyze it. He’s written about property values in Vancouver (on several occasions), enrolment patters in the city’s public schools, the distribution of coffee houses and books stores in the city and, perhaps most notably, an analysis of the number of empty condos in a city with expensive real estate.

My point with Andy’s work is that you don’t need to create an app to use open data. Indeed, it may not even be a good idea. Rather there is lots to be done just analyzing data or figuring out different ways it may have meaning for the people who live in your city. If you can enable more people to understand an phenomenon or problem that is, in of itself, quite valuable.

3. Find Your People

While Open Data Day has become a “thing” it isn’t an organization or a group. It is just an excuse with some simple rules for a world wide community to mobilize itself at a very local level.

That said there ARE lots of amazing groups out there. And Open Data Day is the perfect day to meet some of them and get involved in the amazing work they are doing. Indeed, groups that use open data have a variety of flavours – so there is often something for everyone. Some of these groups include Hacks and Hackers (journalists and software developers), Open Knowledge Foundation (open data activists), Code for All (civic hackers), Mozilla (open web), the Sunlight Foundation (government transparency), to name but a few. There are also lots of more local and national based groups, as well as industry specific groups such as the Extractive Industries Transparency Initiative and the International Aid Transparency Initiative. Often someone at your local event will already be a member of one of these groups – it is a great time to learn more and maybe even join!

4. How to be useful even if you don’t code or can’t use excel

Last year I wrote about three ways anyone can participate in projects on open data day. There is nothing that stops you from creating a keynote/powerpoint that mocks up an idea you have for an analysis or app, or from helping a group document its work so that others can replicate it, or from just going out and finding some useful data that you think might be in the public interest and working with others to get it scraped and made usable.

Indeed what I love about all these examples is that they are paths to learning more. The straight fact is, I can’t code and I have only limited proficiency with something like excel. But I’ve always found ways to make myself useful and in doing so… I have learned so much.

Open Data Day is less about doing than it is about learning, networking and organizing. Ultimately there is nothing stopping you from self organizing a workshop about anything you care about, an issue, a data set, a tool, and asking someone to teach you about and others to learn with you!

5. Go New School – Mobilize a Community to Request a Data Set

Many governments that have open data portals have a “request a data set” option on the page. What an opportunity. If there is a data set that would help bring transparency and accountability to your community… say around environmental pollution, local budget data, contracts and procurement data or arrest rates in a community then why not mobilize local non-profits, stakeholders and even citizens to mount a campaign to “make it open.”

A public campaign that notes the number of people who have “requested” a data set could apply pressure on a government to fulfill its promise to make more data sets open. More importantly, if might giver the community important data by which to challenge public policy or advocate for change.

6. Go Old School – Submit an Access to Information Request

Before there was open data there was… access to information requests. This slow, often cumbersome way of getting government information is critically important. This is because – in places where Access to Information laws exist – they are the only legally backed way to secure government information (and data).

Just learning how to do an access to information can be novel since so few ordinary citizens ever do one. But it many jurisdictions it is your right to be able to do so.

For activists and software developers who are particularly interested in this area of governance/law I encourage you to check out the Alaveteli project started by the amazing team at MySociety. It attempts to bring to add some #opengov tech to the Access to Information world.

I hope this are helpful ideas. More importantly, I hope wherever you are that you have an amazing time on open data day.

 

 

 

So… before there was access to information, there was

What’s in my yogurt

Would you like to know more?

 

 

 

Open Data Day in Vancouver – Registration is live

I’m happy to share that this February 22nd there will be an Open Data Day event in Vancouver. The details are as follows:

The event will take place at Domain7 – who have kindly volunteered their space – on Saturday, February 22, from 10:00 AM to 4:30 PM. We have limited space and want to coordinate lunch so if you want to attend please register here.

We have an exciting day planned – a few opening talks and networking and then a day of hacking, learning and sharing. More details to follow shortly! Very much hope to see you there!

The dangerous mystique of the “open data” business

I’m frequently asked by people about how they can start an “open data business.” Let me first say that I love that the question gets asked. I love that people are interested in Open Data. I love that people want to learn more, they want to play, they want to think of ways of creating a company. These are, in part, signs of how far the open data discussion has come – people see it as a resource that they would like to leverage.

It is, also, the wrong question.

This is not to say there are not businesses that use open data. Indeed, a vast number of companies use open data (anyone company using census data for even a tiny part of their business qualifies). Nor am I denying there aren’t businesses built primarily with open data – the Open Data 500 list demonstrates there are. Plus I get introduced almost daily to businesses that are: both Ajah and OpenCorporates come to mind (I have donated advice, but have no financial connection, with either).

The Trap

But from a founder (or, I suppose, investor) perspective there are dangers to thinking about “open data” as a unique business space.

The danger is in failing to understand there is virtually nothing that distinguishes an open data business from any other business. Any business needs to solve a real (or sadly, at times imagined) problem, it needs to find clients (e.g. people willing to pay for that solution), and it needs to execute on a number of other things at least competently (HR, marketing, management, cashflow, etc…).

The danger with putting the words “open data” before the word “business” is that it risks making people think Open Data businesses are somehow unique. They are not. If there is a gapping chasm between the question of “what can I do with software” and “how can I create a viable software company” there is an equally large gap between “what can I do with open data” and “how can I create a viable company using open data.” And the questions you need to ask yourself to figure out that latter question (many of which are nicely laid out in this book) are independent of whether it is a software, hardware, crafts or open data business.

Indeed open source software space gives us a nice analogy. I suspect few people decide to create an open source software company – they decide to create a company and the software license is a reflection of their strategic options. I think it is the same with open data. You don’t start a company saying “let’s use open data.” You start a company to solve a problem, of which using or publishing open data may be the only, or the most strategic, way of doing this.

The Opportunity 

Some readers may be surprised to see me write this. I am, and continue to be an advocate of open data. But open data is not some magic pixie dust that causes normal business logic to disappear. And it is not that I think people are saying that per se, it is just that I want them to understand that the 99% of the problems that needs to be solved in an “open data business” lie in the third word of that string, and that while the first two do confer some unique advantages and disadvantages, these are relatively trivial.

The real opportunity of open data lies not in the way it creates a new unique type of business, but that it offers a new set of cheap building blocks by which to try to solve problems. In other words it increases the diversity and, lowers the cost of, inputs.

Here again the world of software is instructive. The Economist’s recent survey on Tech Start Ups talks of a Cambrian Explosion because of the availability of “Cheap and ubiquitous building blocks for digital products…” many of which are (and many of which are not) open source. The cheap availability of these building blocks is allowing for a range of experimentation that was previously not possible, or at least, prohibitively expensive.

Open data – whether as an input for software products and services, for analysis (journalistic or corporate) or for scientific research – is cheap (in theory free) and increasingly plentiful. It has the possibility of thus being the equivalent of the cheap code that is powering a great deal of experimentation in the world of software. As an open data advocate the possibility of this increased experimentation has me excited.

Conclusion

So if you are thinking about starting an open data business – that is great! I’m excited to hear that and I am keen to help and be supportive. But focus on that third word – business. That’s the one that really matters.

From a data perspective you should be asking yourself – what real tangible pain does doing something with this data set help me solve that was previously only possible with a more expensive input (e.g. proprietary data) or not possible at all. The second is to think of the impact of using open data on your strategy. Where does it leave you more vulnerable (too copy cats or the whims of the data publisher) and where does it leave you stronger (if the data is commoditized then the axis of competition will lie in other parts of the business).

I hope this is a helpful nuance to the issue of open data businesses, and some helpful input for those looking at open data and thinking about to find business opportunities in it.

Open Data Advice for Librarian Coders

For the purposes of this post we will ignore how remarkably badass sounding the word librarian becomes when “coder” is added to the end of it.

I recently had a librarian who had just picked up some coding skills email me and ask how they could get into Open Data (if, by the way, you were interested in someone with those skills, I’d be happy to connect you to them). Here is an edited version of my response:

I actually think that librarians are exactly what the open data space needs. It is interesting to think about the consulting services and organizational problems that exist in a world of almost infinite data – and particularly in a world with large amounts of open data.

Open Data portals are are getting more and more data sets – but I’m not sure anyone has meaningfully figured out how you organize that data in meaningful way? Or at least make them searchable in a way that is meaningful to a cross section of people as opposed to a narrow highly specialized specific type of user (e.g. an academic or domain expert)? Nor has anyone – as far as I can tell – cracked the problem (or developed the human hacks necessary) around acquiring and organizing meta data (e.g. actually getting it, and then making it understandable to a range of audiences without losing its nuance).

Certainly when I facilitated consultations on open data in a range of cities across Canada the issue of searchability and finding data came up over and over again. It was a very strong theme.

These are serious problems. And they are probably going to get much much worse as more data becomes available.