Category Archives: cool links

Great Hacks from the Open Data in Vancouver

Last weekend I helped host an Open Data Day in Vancouver. With the generous support of Domain7, who gave us a place to host talks and hack, over 30 Vancouverites braved the sleet and snow to spend the day sharing ideas and working on projects.

We had opening comments from Andy Yan – whose may be the most prolific user of Open Data in Vancouver, possibly Canada. I encourage you to check out his work here. We were also incredibly lucky to have Jeni Tennison – the Technical Director of the Open Data Institute – onsite to talk to participants about the ODI.

After the opening talks, people simply shared what they hoped to work on and people just found projects to contribute to. Minimal organization was involved… and here a taste of the awesome projects that got worked on! Lots of ideas here for other communities.

1. Open Data Licenses Resource: JSON + search + compatibility check = Awesome.

Kent Mewhort, who recently moved to Vancouver from Ottawa (via the Congo) updated his ongoing CLIPol project by adding some of the recently published licenses. If you’ve not seen CLIPol it is… awesome. It allows you to easily understand and compare the restrictions and rights of many open government licenses.

CLIPol Data

Better still CLIPol also lets you to see how compatible a license is (see example here). Possibly the best tool of all is one that allows you to determine what license you can apply to your re-mixed work in a way that is compliant with the original licenses (check out that tool here – screenshot below).

CLIPol compatibility

CLIPol is just such a fantastic tool – can’t recommend it enough and encourage people to add more licenses to it.

2. Vancouver in MineCraft

I have previously written about how Minecraft is being used to help in public consultations and urban planning – I love how the game becomes a simple tool that enables anyone to shape the environment.

So I was crazy excited I heard that Ryan Smith (aka Goldfish) had used the City of Vancouver’s open elevation data to recreate much of the city in Minecraft.

Below is a photo of Ryan presenting at the end of the day. The projection behind him shows Stanley park, near Siwash Rock. The flat feature at the bottom is the sea wall. Indeed Ryan notes that the sea wall makes for one of the clearest features since it creates almost perfectly flat structure along the city’s coast.

Mincraft Data

3. Vancouver’s Capital Budget Visualized in Where Does my Money Go

It is hard to imagine a project going better. I’m going to do a separate blog post on it.

This is a project I’ve always wanted to do – create a bubble tree visualization with Where Does my Money Go. Fortunately two developers – Alexandre Dufournet and Luc Lussier – who had never hacked on open data jumped on the idea. With help from City of Vancouver’s staff who were on site, I found a PDF of the capital budget which we then scraped.

WDMYG Data

The site is not actually live, but for developers who are interested in seeing this work (hint, hint City of Vancouver staff) you can grab their code from github here.

4. Monitoring Vancouver’s Bike Accident Data – Year 3

Eric Promislow has been coming to Open Data Hack-a-thons ever since Luke Closs and I started organizing them in 2009. During the first Open Data Day in 2011 you can read in my wrap up post about a bike accident monitoring website Eric created that day which Eric would eventual name Bent Frame. Well, Bent Frame has been live ever since and getting bigger. (Eric blogs about it here)

Each open data day, Eric updates Bent Frame with new data from ICBC – the province’s insurance monopoly. With over 6 years of data now in Eric is starting to be able to analyze trends – particularly around the decline of bike accidents along many roads with bike lanes, and an increase in accidents where the bike lanes end.


Bike Data

I initially had conversations with ICBC to persuade them to share their data with Eric and they’ve been in touch with him ever since, passing along the data on a regular basis. It is a real example of how an active citizen can change an organization’s policies around sharing important data that can help inform public policy debates.

5. ProactiveDisclosure.ca – Making government information easier to search

Kevin McArthur is the kind of security guy most governments dreads having around but should actually love (example his recent post on e-voting).  He continued to hack on one of his side projects: proactivedisclosure.ca. The site is a sort of front end for open data sets, making it easier to do searches based on people or companies. Thus, want to find all the open data about a specific minister… proactive disclosure organizes it for you.

Proactive Data

Kevin and a small team of players uploaded more data into their site and allowed it to consume unstructured data. Very cool stuff.

6. Better Open Data Search

Herb Lainchbury – another fantastic open data advocate – worked on a project in which he tried to rethink what an open data search engine would look like. This is a topic that I think matters A LOT. There is simply not a lot of good ways to find data that you are interested in.

Herb’s awesome insight was invert the traditional way of thinking about data search. He created a search engine that didn’t search for the data set keywords or titles, but rather searched the meta data exclusively.

One interesting side outcome of this approach is that it made related data sets easier and, made locating identical data sets but from different years a snap. As Herb notes the meta data becomes a sort of “finger print” that makes it easy to see when it has been duplicated. (Quick aside rant: I loath it when governments releases 20 data files of the same data set – say crime data – with each file representing a different year and then claiming that it is 20 unique data sets in their catalogue. No. It is one data set. You just have 20 years of it. Sigh).

7. School Performance Chart

Two local video game programers – Louie Dinh and Raymond Huang – with no experience in open data looked around the BC Government Open Data catalogue and noticed the data on test scores. Since they attended school here in British Columbia they thought it might be interesting to chart the test scores to see how their own schools had preformed over time.

They were able to set up a site which graphed how a number of elementary schools had performed over time by looking at the standardized test scores.

Test SCore Data

This is just a great example of data as a gateway to learning. Here a simple hackathon project become a bridge for two citizens to dive into a area of public policy and learn more about it. No one is claiming that there chart is definitive, rather it is the start of a learning process around what matters and what doesn’t and what can be measured and what can’t in education.

Congratulations to everyone who participated in the day – thank you for making it such an amazing success!

Open Data Day 2014 – Five Fun Events Around the World

With over 110 Events happening world wide it is impossible to talk about every Open Data Day event. But looking almost every event on the wiki I’ve been deeply moved and inspired by the various efforts, goals and aspirations of the people who have organized these events.

In order to help others understand why Open Data Day matters as well as what can happen on it, here are five open data day events that I’m stumbled across that are doing something particularly fun or interesting.

1. Capetown & Johannesburg, South Africa

Their Description:

Coders, data wranglers and data investigators will pair up to look at one of three openly available datasets we have on hand, and work out the most interesting questions they can ask of it in less than half an hour. After 30 minutes, it’s all change – another desk, another partnership, another exciting data set to turn into a story.

What we’re hoping is that you’ll learn tips and tricks for getting data, querying it, creating quick visualizations and turning it to stories that people want to know about. You’ll learn from different people with a variety of skills, hopefully that you wouldn’t normally work with. And we’re also hoping it will be four hours of fun.

Why I love it: I love the focus on learning. With the participation of hacks and hackers the goal is clearly to help journalists and citizens learn new skills, not so they can do something with the data sets available on open data day, but so they can better play with data sets in the future to pursue stories or help a community. The point of speed data dating is thus not to build a product, the product is the skills and networks developed and, with luck the future stories and analyses that will be told by those who participated.

2. Buenos Aires, Argentina

Their description:

On February 22nd we will go out to the street and play with local data and some street artists to create beautiful visualizations.

Why I love it: Street art open data? What a great way to try to raise awareness of the importance of data literacy and transparency. In addition, how awesome is it to move outside the digital realm and use data to create artifacts that are not necessarily digital. And if there are artists involved? Jer would be so happy to read about this.

3. Greenfield, MA, United States

Their description:

We’re convening a small group to work with the Franklin Regional Council of Governments on a user-friendly way to map private wells in Western Massachusetts…

…Why is it important to map wells?

  • Only about 5% of private wells in Massachusetts are geolocated.
  • Many towns in Western Mass rely 100% on private wells.

Not knowing where our wells are can (and does) lead to water contaminated by nearby septic systems, dumping, and pollutant storage. Aside from the obvious health concerns, there are also financial implications from remediation costs and lowered property values.

Why I love it: Wow, WOW, WOW!!! This is maybe one of the coolest open data day events I’ve ever seen. Here you have a small community focusing on a problem that is real and tangible to them. Moreover, open data could have a direct and meaningful impact on the issue. I love the focus. I love that rallying point. I love the high impact with low resources (their building has minimal heat – so they are advising people to layer up). I wish these crew all the best success and hope to see an update.

4. Nagoya, Japan

Their Description:

Now, highlight of this year is “data of Nagoya Castle!”

The nearly 300 maps and survey drawings of the Nagoya Castle will be made open prior to Open Data Day. The Nagoya Castle office is cooperating with us and has decided that we can use their data for “International Open Data Day.” The references to the image are here.

Why I love it: Well – fill disclosure, my understanding of this event is through the prism of Google translate. But if I understood correctly… there are a few open data events in Japan that have a strong focus on local history which I find totally fascinating. At this event in Nagoya they are bringing in a professor who is an expert in open data as well as expert in the Nagoya castle to talk about the data that is being made open. In addition they are organizing and actual physical tour of the caste. Open Data meets local history buffs!

5. Cairo, Egypt

Their Description:

We will be organizing an online and decentralized event in Cairo, Egypt for the Open Data Day. There are numerous suggested tracks depending on the participants set of expertise:

For translators (المترجمون)

  • Open Data Handbook: The handbook discusses the legal, social and technical aspects of open data. It can be used by anyone but is especially designed for those seeking to open up data. It discusses the why, what and how of open data – why to go open, what open is, and the how to ‘open’ data. Translate it into Arabic here
  • Translate any of the School of Data short tutorials, for example, What is Data?Telling a Story with DataFinding Data, or Any other course/modules

For bloggers (المدونون)

Write blog posts about Open Data related topics and case studies, and don’t forget to use the following hashtag, #ODD2014. Possible ideas for blog posts:

  • Write about the concept of Frictionless Data
  • Case studies how you searched for, extracted and used governmental data
  • Listing of local organizations working or promoting Open Data or advocating for more Transparent and Open Governments

For Developers (مطوري البرامج)

  • Scrape data from capmas and put it into Open Format
  • Scrape data (Budget or the The Financial Monthly Bulletin) from the Minister of Finance and upload it to OpenSpending.org,
  • Create a tool to scrap the traffic data from bey2ollak and put it in an open format.

For Data Wranglers (هواة جمع البيانات)

Why I love it: I love that there are calls to action for a variety of people – including those who have no coding skills at all. How genius is it to organize an event to localize/translate the Open Data handbook? This is something a large number of people could do – and better still can help make open data accessible to a still larger pool of people.

And for the other roles the suggestion of projects – particular with a focus on the national budget and government operations data (capmas) suggests there is a strong civil society presence within the open data community. Will be super interested to see what progress they make and if there is broader interest in their work.

Open Data for Development Challenge on Jan 27-28

This just came across my email via Michael Roberts who has been doing great work in this space.

Mail Attachment

Open Data for Development Challenge
January 27–28, 2014 — Montreal, Canada

Do you want to share your creative ideas and cutting-edge expertise, and make a difference in the world?
Do you want to help Canadians and the world understand how development aid is spent and what its impact is?
Do you want to be challenged and have fun at the same time?

If so, take the Open Data for Development Challenge!

This unique 36-hour ”codathon” organized by Foreign Affairs, Trade and Development Canada will bring together Canadian and international technical experts and policy makers to generate new tools and ideas in the fields of open data and aid transparency and contribute to innovative solutions to the world’s pressing development challenges.

The event will feature keynote speakers Aleem Walji, Director of the World Bank’s Innovation Labs, and Mark Surman, Executive Director of the Mozilla Foundation. It will have two related dimensions:

  • Technical challenges that involve building applications to make existing open aid and development-related data more useful. Proposed topics include building a data viewer compatible with multilingual data, creating a publishing tool suitable for use by mid-sized Canadian non-profit organizations, developing and testing applications for open contracting, and taking a deep dive into the procurement data of the World Bank Group. There is room for challenges proposed by the community. Proposals should be submitted through the event website no later than January 8th. Challenges will be published prior to the event, along with key datasets and other related information, to enable participants to prepare for the event.
  • Policy discussions on how open data and open government can enable development results. This would include the use of big data in development programming, the innovative ways in which data can be mapped and visualized for development, and the impact of open data on developing countries.

The international aid transparency community will be encouraged to take promising tools and ideas from the event forward for further research and development.

An overview of the draft program is attached. The event will be in English and French, with interpretation provided in the plenary sessions and panel discussions.

We invite you to register, at no cost, at this website as soon as possible and no later than January 10. A message confirming your registration and providing additional information about the venue and accommodation will be sent to confirmed participants. Please wait for this confirmation before making any travel arrangements. Participants are asked to make their own accommodation arrangements. A limited number of guest rooms will be available to event participants at a preferential rate.

To find out more about the Open Data for Development Challenge, please go to DFATD’s website.

Open Data Day 2014 is Coming Feb 22 – Time to Join the Fun!

So, with much help from various community members (who reminded me that we need to get this rolling – looking at you Heather Leson), I pleased to say we are starting to gear up for Open Data Day 2014 on February 22nd, 2014.

From its humble beginnings of a conversation between a few friends who were interested in promoting and playing with open data, last year Open Data Day had locally organized events take place in over 100 cities around the world. Check out this video of open data day in Kathmandu last year.

Why makes Open Data Day work? Mostly you. It is a global excuse for people in communities like yours to come together and organize an event that meets their needs. Whether that is a hackathon, a showcase and fair, lectures, workshops for local NGOs and businesses, training on data, or meetings with local politicians – people are free to organize around whatever they think their community needs. You can read more about how Open Data Day works on our website.

Want to join in on the fun? I thought you’d never ask. Listed below are some different ways you can help make Open Data Day 2014 a success in your community!

A) How can I let EVERYONE know about open data day

I love the enthusiasm. Here’s a tweet you can send:

#OpenData Day is community powered in a timezone near you.  http://opendataday.org/ #ODD2014

Yes, our hashtag is #ODD2014. Cause we are odd. And cause we love open data.

B) I’d like to participate!

Great! If you are interested in participating in check out the Open Data Day wiki. We’ve just unlocked the pages so cities haven’t been added yet but feel free to add your city to the list, and put down your name as interested in participating. You can even check to see who organized the event last year to see if they are interested in doing it again.

C) Forget about participating, I want to coordinate an Open Data Day event in my city.

Whoa! Very exciting! Here’s a short checklist of what to do:

  • If you didn’t organize one last year, check to see if anyone in your city did. It would be good to connect with them first.
  • Read the Open Data Day website. Basically, pick up on our vibe: we want Open Data Day to work for everyone, from novices who know little about data to experts like Kaggle participants and uber geeks like Bruce Schneier. These events have always been welcoming and encouraging – it is part of the design challenge.
  • Okay, now add your city to the list, let people know where it will be taking place (or that you are working on securing space), let them know a rough agenda, what to expect, and how they can contribute.
  • Add yourself to the 2014 Open Data Day map. (Hint: Wikipedia lists Lat/Long in the information side bar for each cities wiki page: “Coordinates: 43°42′N 79°24′W”)
  • Join the Open Data Day mailing list. Organizers tend to share best practices and tips here. It’s not serious, really just a help and support group.
  • Check out resources like this and this about how to organize a successful event.
  • Start spreading the news!

D) I want to help more! How can Open Data Day work more smoothly everywhere?

Okay, for the truly hardcore you right, we need help. Open Data day has grown. This means we’ve outgrown a whole bunch of our infrastructure… like our webpage! Everyone involved in this is a volunteer so… we have some extra heavy lifting we need help with. This includes:

a. Website template update: The current Open Data Day template was generously donated by Mark Dunkley (thank you!!!). We’d love to have it scale a little better and refresh the content. You can see the code on github here. Email me if you are interested. Skills required: css, design

b. Translation: Can you help translate the ODD site into your language? You can submit the requests on github or send a document to heather.leson at okfn dot org with the content. She’ll do the github stuff if that’s beyond you.

c. Map: Leaflet and layers helpers wanted! We’d like a map geek to help correct geolocation and keep the 2014 map fresh with accurate geo for all the locations. Github repo is here and the event list is here.

What’s next?

I’m really looking forward to this year… I’ve lots more thoughts I’ll be sharing shortly.

Plus, I can’t wait to hear from you!

Access to Information, Technology and Open Data – Keynote for the Commissioners

On October 11th I was invited by Elizabeth Denham, the Access to Information and Privacy Commissioner for British Columbia to give a keynote at the Privacy and Access 20/20 Conference in Vancouver to an audience that included the various provincial and federal Information Commissioners.

Below is my keynote, I’ve tried to sync the slides up as well as possible. For those who want to skip to juicier parts:

  • 7:08 – thoughts about the technology dependence of RTI legislation
  • 12:16 –  the problematic approach to RTI implementation that results from these unsaid assumptions
  • 28:25 – the need and opportunity to bring open data and RTI advocates together

Some acronyms used:

StreetMix for testing bike lanes – Burrard St. Bridge Example

I’m MCing the Code for America Summit at the moment, so short on time to write a post, but I’m just LOVING StreetMix so much I had to give it a shout out. If you are a councillor, urban planner or community activist, StreetMix is a site you HAVE to check out.

What does it do? I basically allows you to create or edit and street you want. It is so simple to use it takes about 1 minute to master. At that point, you can build, copy and redesign any street in the world.

Here, for example I’ve recreated the Burrard St. Bridge in Vancouver as it exists today, with bike lanes and below, as it existed before the addition bike lane.

Burrard Bridge new

Burrard Bridge old

The 311 Open Data Competition is now Live on Kaggle

As I shared the other week, I’ve been working on a data competition with Kaggle and SeeClickFix involving 311 data from four cities: Chicago, New Haven, Oakland and Richmond.

So first things first – the competition is now live. Indeed, there are already 19 teams and 56 submissions that have been made. Fortunately, time is on your side, there are 56 days to go.

As I mentioned in my previous post on the subject, I have real hopes that this competition can help test a hypothesis I have about the possibility of an algorithmic open commons:

There is, however, for me, a potentially bigger goal. To date, as far as I know, predictive algorithms of 311 data have only ever been attempted within a city, not across cities. At a minimum it has not been attempted in a way in which the results are public and become a public asset.

So while the specific problem  this contest addresses is relatively humble, I’d see it as a creating a larger opportunity for academics, researchers, data scientists, and curious participants to figure out if can we develop predictive algorithms that work for multiple cities. Because if we can, then these algorithms could be a shared common asset. Each algorithm would become a tool for not just one housing non-profit, or city program but a tool for all sufficiently similar non-profits or city programs.

Of course I’m also discovering there are other benefits that arise out of these competitions.

This last weekend there was a mini-sub competition/hackathon involving a subset of the data. It was amazing to watch from afar. First, I was floored by how much cooperation there was, even between competitors and especially after the competition closed. Take a look at the forums, they are probably make one of the more compelling cases that open data can help foster more people to want to learn how to manipulate and engage with data. Here are contestants sharing their approaches and ideas with one another – just like you’d want them to. I’d known that Kaggle had a interesting community and that learning played an important role in it, but “riding along” in a mini competition has caused me to look again at the competitions through a purely educational lens. It is amazing how much people both wanted to learn and share.

As in the current competition, the team at the hackathon also ran a competition around visualizing the data. And there were some great visualization of the data that came out of it, as well as another example of where people were trying to learn and share. Here are two of my favourites:

map2

I love this visualization by Christoph Molnar because it reveals the different in request locations in each city. In some they are really dense, whereas in others they are much (more) evenly distributed. Super interesting to me.

Most pressing issues in each city

I also love the simplicity of this image created by miswift. There might have been other things I’d done, like colour coded similar problems to make them easier to compare across cities. But I still love it.

Congratulations to all the winners from this weekends event, and I hope readers will consider participating in the current competition.