Category Archives: technology

Open Data Day: Lessons for Hacktivists

This piece is cross-posted on TechPresident where I post articles on the intersection of politics, technology and transparency and serve as an editor.

Three years ago, after a chance encounter with Daniela Silva and Pedro Markun of Sao Paulo and a meeting with Edward Ocampo-Gooding and Mary Beth Baker in Ottawa, with whom I shared a passion about open data, we agreed to simultaneously host events in our three cities on the same day. It would be a hackathon, and because it would take place in at least two countries … we liberally called it “international” inviting others to join us.

In that first year we had about six cities conduct events on every continent save Australia. Now in its third year, Open Data Day events is far bigger than we ever dared imagine. More interesting still is its impact, both expected and unexpected.

Reflecting on it all, I thought it might be worthwhile to share a little bit about the impact I think Open Data has, and some lessons hacktvists may find interesting to draw upon.

Build Community

From the beginning our goal was simple. It wasn’t code, or even opening data per se. It was about community.

We wanted to foster a friendly event where anyone would feel welcome to participate. The team in Ottawa had, in particular, done great work in reaching out an engaging new people in previous hackathons, they’d have a wealth of non-software developers and even full families attend their events. This was as much a hackathon as it was a community event.

So Open Data Day was always meant to serve as a catalyst for community and network creation. Yes, creating, adding to or working on a project was strongly encouraged, but the real output to help open data advocates find and connect with one another, as well as grow the movement by engaging new people. An effective community was always going to be the core ingredient for petitioning a government to make data open, teaching students or policy makers how to use data or have a group of developers launch a project. While we wanted to create cool software, visualizations and analysis, what I wanted even more was to foster local leaders, champions, social glue and community hubs.

And that’s what we got.

One of the strengths of Open Data Day is its incredible decentralized nature. I blog about it to encourage people to organize and help moderate a mailing list, but beyond that everything is done by local volunteers. From a capacity building perspective, it is an incredible event to watch unfold. I’d like to think Open Data Day has played a helpful role in connecting local stakeholders and even knitting them together with regional, national and international peers.

While I believe that a beautiful piece of code can be critical in making policy makers, the public or others see the world in a different light I also think that building community and developing allies who can share that story of that code with leaders and the public at large is, depending on the breadth of your goals, equally important.

What Works for Communities Can Work for Governments

Almost immediately upon launch of Open Data Day, government officials began showing up at the hackathons. Some came unofficially, others officially, and in some places, the hackathons were invited to take place at city hall. Because we set a venue Open Data Day created a predictable publicized space where governments that were curious, reluctant, eager or cautious could come and engage at a speed that worked for them.

When you build a safe place for a broader community – if your event has more of the feel of a public consultation or community meet-up than a gathering of subversive technologists – you can create space for governments. More importantly, the one thing that characterizes most open data hackathons I’ve witnessed or investigated,is that the participants share a desire to make their community better. That is something most government officials can easily wrap their heads around.

Thus, in a way that I don’t think anyone planned, Open Data Day events have become a place where governments want to understand more of what is possible, learn about the community that is interested in data and wrestle with how open data can change the way they work. What began with a few pioneering cities three years ago has evolved where now places like the Victoria Palace in Romania (the seat of government) and the White House in Washington DC are hosting Open Data Day events.

This type of engagement creates new opportunities, and new challenges, but that is exactly the type of progress many movements would like to see.

Plant a Flag

Setting a date and having people around the world step up and embrace had another interesting impact. It created a deadline not just for community organizers who were organizing their local open data day event, but also for governments that want to engage these communities.

Indeed in the past it has been amazing to see how many governments now see Open Data Day as a deadline for launching open data portals or releasing additional data sets. This is a fantastic outcome as it creates subtle pressure on governments to act.

This year was no exception. I heard of governments around the world releasing data sets in anticipation of Open Data Day. In particular there was a flurry of activity from European governments and agencies. Indeed the European Union chose to launch its open data portal in time for Open Data Day. But there was also much activity across the continent. This included that launch of data portals for the Building Performance Institute Europe, the Italian Senate, the city of Venice , the city ofTrento, the region of Puglia in the south of Italy, and the province of Bolzano. But the practice has become widespread. I was particularly happy to see a city a few kilometers from my home town do the same: the City of Victoria, BC launched its Open Data portal for the day.

These announcements show the amplifying effect of getting organized across geographies. The advocacy work of a group in one country helps reinforce a message that benefits advocates in other countries. This is pretty basic stuff — political advocacy 101 if you will — but it is a reminder of why congregating around a single date, and meeting in person still have value for those who are unpersuaded.

Ultimately Open Data Day is important because it serves the needs of local advocates and activists wherever they are. That means keeping the event coherent so that it can continue to be a global event that people understand, yet flexible so that it can satisfy local needs and desires. For example, the event in Manila focused on poverty while in Washington DC it focused on building community and capacity. These types of events are not, in of themselves, going to completely solve a problem, but I do believe that they are part of a broader political effort that activates a community, engages government and creates pressure for change.

How Hackers Will Blow Up The World: China, Cyber-Warfare and the Cuban Missile Crisis

I have a piece on TechPresident I really enjoyed writing about how certain technologies – as they become weaponized – can in turn become highly destabilizing to global stability. The current rash of Cyber-Warfare, or Cyber-Spying or Cyber-crime (depending on the seriousness and intent with which you rate it) could be one such destabilizing technology.

Here’s a long excerpt:

This would certainly not be the first time technology altered a balance of military power and destabilized global political orders everyone thought was robust. One reason the world plunged into global war in 1914 after a relatively minor terrorist attack — the assassination of Arch-Duke Ferdinand — was because the hot new technology of the day, the speedy railway, caused strategists to believe it would confer a decisive advantage on those who mobilized first. The advent of nuclear intercontinental ballistic missiles of the 1950s had a similar effect, with fears that a first strike “decapitation attack” against Moscow from Turkey, or against Washington from Cuba, could preempt a counter attack.

Cyber warfare may be evolving into a similarly destabilizing type of technology. Prior to the 21st century, cyber attacks were relatively localized affairs. People imagined the main threats of a cyber attack being with virtual thefts from banks, identify theft against individuals and even industrial piracy. Serious problems to be sure, but not end-of-the-world stuff. Even when targeted against the state, cyber attacks rarely pose an existential threat to a country. The loss of state secrets, the compromising of some officials could, cumulatively, be corrosive on a state’s ability to defend itself or advance its interests, but it was unlikely even a combination of operations would shake a mature state to its core.

Two things have changed….

You can read the full piece here. Always love feedback.

Open Data Day 2013 in Vancouver

Better late than never, I’m going to do a few posts this week recapping a number of ideas and thoughts from Open Data Day 2013. As is most appropriate, I’m going to start the week with a recap of Vancouver – the Open Data Day event I attended and helped organize along with my friend Luke Closs and the very helpful and supportive staff at the City of Vancouver – in particular Linda Low and Kevin Bowers. I’ve got further thoughts about the day in general, its impact and some other ideas I’ll share in subsequent posts.

Vancouver! 2013!

What made Open Data Day in Vancouver great for me was that we had a range of things happen that really balanced time between “creating” (e.g. hacking) and engagement. As a result I’m diving this blog post into three parts: setting the scene, engagement and outcomes as well as sharing some lessons and best practices. My hope is that the post will make for fun reading for regular readers, but the lessons will prove helpful to future open data day organizers and/or plain old hackathon organizers.

Setting the scene

City hall photo by Roland

Despite competing with a somewhat rare sunny day in Vancouver at this time of year we had over 80 participants show up at City Hall, who graciously agreed to host the room. Backgrounds varied – we had environmentalists, college and university students, GIS types, open street mappers, journalists, statisticians among others. In addition, the city’s administration made a big commitment to be on hand. Over 20 staff were present, including one of the Deputy City Managers and a councillor – Andrea Reimer – who has been most active and supportive on the Open Data file. In addition we had a surprise guest – Federal Minister Tony Clement, who is the minister responsible for Open Data with the national government.

Our event was fairly loosely organized. We had a general agenda but didn’t over script anything. While Luke and I are comfortable with relatively flexible agenda for the day, we knew the unstructured nature of the event was a departure for some from the city and were grateful that they trusted us both with the format and with the fast, loose and informal way we sought to run the day.

Here are some high level lessons:

  • When Possible host it Somewhere Meaningful. City Hall is always nice. Many attendees will have never engaged with local government before, this creates a space for them to learn and care about municipal government. It also reaffirms the broader culture changing and social/change oriented goals of an open data hackathon. Besides, who isn’t welcome at City Hall? It sets a great tone about who can come – which is pretty much anyone.
  • Give the Politicians Some Room. Don’t be afraid to celebrate politicians and leaders who have championed the cause. Hackathons are about experimenting, not talking, so Andrea’s five minute talk at the beginning was the right mix of sincere engagement, enthusiasm and length.
  • Don’t Only Attract Software Developers. Vancouver’s event was in part a success because of the diversity of participants. As you’ll see below, there are stories that emerged because we tried hard to engage non-software developers as well.
  • Transit. This may sound obvious but… if you want to attract lots of people make it convenient to get to. We right off the subway line – City Hall is pretty central.
  • RSVP. We did one thing wrong and one thing right. On the wrong side, we should have allowed more people to register recognizing that not everyone would show. We’d turned people away and ended up having room for them because of the no shows. On the right side, we blasted our list to see if anyone wasn’t going to come at the last minute and, got some responses, which allowed us to re-allocate those seats.

Engagement: Open Dataing

Speed Dating with Minister Clement

The one thing both Luke and I were committed to doing was getting participants and city staff to talk to one another. Building off an idea Luke witnessed in Ottawa we did 45 minutes of Open “Dataing.” During this period city staff from about eight different departments, as well as Minister Clement, staked out a part of the room. We then had people cluster – in groups of about 5-7 – around a department whose data or mission was of interest to them. They then had 10 minutes to learn about what data was available, ask questions about data they’d like, projects they were thinking of working on, or learn more about the operations of the city. In essence, they got to have a 10 minute speed date on data with a city official.

After 10 minutes blew a whistle and people went and clustered around a new official and did it all over again.  It was a blast.

Key Lessons:

  • Open Dataing is about getting everyone engaged. Helping public servants see what citizens are interested in and how they can see technology working for them, it’s also about getting participants to learn about what is available, what’s possible, and what are some of the real constraints faced by city staff
  • I wish we’d had signs for the various departments being represented, would have made it easier for people to find and gravitate towards the issues that mattered most to them.
  • This type of activity is great early on, it’s a way to get people talking and sharing ideas. In addition, we did the whole thing standing, that way no one could get too comfortable and it ensure that things kept flowing.

Outcomes

After the dating, we did a brief run through of projects people wanted to work on and basically stepped back and hacked. So what got worked on? There were other projects that were worked on, but here are the ones that saw presentations at the end of the day. There are some real gems.

Bike Parking App 

The Bike Rack app came out of some challenges about data gathering that Councillor Reimer shared with me. I suggested the project and a team of students from UBC’s Magic Lab turned it into something real.

The city recently released a data set of the locations of all city owned bike racks. However, the city has little data about if these locations are useful. So the app that got hacked is very simple. When you arrive at your destination on bike you load it up and it searches for the nearest bike rack. For the user, this can be helpful. However, the app would also track that lat/long of where the search was conducted. This would let city planners know where people are when looking for bike rack so they would start to have some data about underserved locations. Very helpful. In addition, the app could have a crowdsourced function for marking the location of private bike racks (managed by businesses) as well as an option to lat/long where and when your bike was stolen. All this data could be used to help promote cycling, as well as help the city serve cyclists more effectively.

Homelessness Dashboard

Luke showing off his rental dashboard

My friend Luke connected a Raspberry Pi device to one of the giant TVs in the room where we were working to share the work he had done to create a dashboard based on the city’s rental standards database.  Luke’s work even got featured on the Atlantic’s website.

Crime mapping – lots of quality questions

One great project involved no software at all. We had some real data crunchers in attendance and they started diving into the Vancouver Police Department’s data with a critical eye. I wish the VPD could have been there since their conclusions were not pretty. The truth is, the Vancouver Police department makes very little data open, and what it does make, is not very good. What was great to see were some very experienced statisticians explain why. I hope to be able to share the deck they created.

Air quality egg

A few weeks prior to Open Data Day I shared that I’d be launching a project – with the help of the Centre for Digital Media – around measuring air quality in Vancouver. Well, the air quality egg we’d ordered arrived and our team started exploring what it would, and might not, allow us to do. The results were very exciting. Open Data Day basically gave us time to determine that the technical hurdles we were worried about are surmountable – so we will be moving forward.

Over the coming months we’ll be crafting a website that uses Air Quality Eggs to measure the air quality in various neighborhoods in Vancouver. We have a number of other community partners that are hoping on board. By Clean Air Day  we’d love to have 50-100 air quality eggs scattered across various neighborhoods in greater Vancouver. If you are interested in sponsoring one (they are about $150) please contact me.

Youth Oriented events RSS feed

One intrepid participant, also not a developer, got the city to agree to create and share an RSS feed of youth oriented events. This was important to them as they were concerned with youth issues – so a great example of a community organizer getting a data resource from the city. Next steps – trying to get other organizations with youth organized events to agree to share their program data in a similar data schema. What a great project.

Provincial Crowdsourced Road Kills and Poaching Maps

I was very excited to have to participants from the David Suzuki Foundation at the hackathon. They worked on creating some maps that would allow people to crowdsource map road kills, and instances of poaching, across the province of British Columbia. I love that they had a chance to explore the technical side of this problem, particularly as they may be well placed to resolve the community building side that would be essential to making a project like this a success.

Neighbourhood quality Heat Maps

Another team took various data sets from the Vancouver Open Data portal to generate heat maps of data quality (proximity to certain services and other variables). This prompted a robust conversation about the methodologies used to assess quality as well as how to account for services vs. population density. Exactly the types of conversations we want to foster!

Figuring our how to translate all of the data.vancouver.ca datasets

Another participant  Jim DeLaHunt – put in some infrastructure that would make it easier to translate Vancouver’s open data into multiple languages in order to make it more accessible. He spent the day trying to identify what data in which data sets was structured versus unstructured human readable text. And… much to his credit he created a wiki page, the  Vancouver Open Data language census to update people on his work so far.

Live Bus Data Mapped

Transit mapping

Another team played with the local transit authority’s real time bus data location API. It was pretty cool to see the dots moving across a google map in real time.

Their goals were mostly just to experiment, play and learn, but I know that apps like this have been sold into coffee shops in Boston, where they let customers know how far away the next bus.

Open Street Map

A Open Street Mapper participant spent the data getting address data merged with OSM.

Key Lessons:

  • Don’t try to over manage the event – give people space and time to create
  • Even if the energy feels low after a long day – definitely share out what people worked on, even if they didn’t finish what they wanted. There is lots to be learned from what others are doing and many new ideas get generated. It was also great for city staff to see what is possible
  • Get people to share github repos and other links while on site. Too many of the above projects lack links!

Thank you again for everyone who made Open Data Day in Vancouver a success! Looking forward to next year!

Three Ways Anyone Can Contribute to Open Data Day

With well over 90 cities now scheduled to partake in Open Data Day and with several events expecting 50+ and even 100+ participants I wanted to outline some thoughts to help people who are thinking about participating but not sure what to expect or if they have anything helpful to offer.

First things first.

You can help. You have something to offer.

Second. You are not alone. Many people are not sure what to expect nor what they will do. If you come with a sense of play, curiosity and a willingness to work with others, you will find something interesting to do. And, of course, you will have amazing local organizer who, from what I’ve seen on the wiki and the web, have worked very hard to find a great venue, put together great agendas and reach out to other great people at the event you’ll be attending.

So here are two things even someone who knows next to nothing can do and five projects to consider exploring:

1. Mock Up Your Idea

Not everyone can code. I can’t code! But coding is only one part of a series of activities that can lead to an interesting project. The first… is thinking of an interesting project.

If you have an idea, one of the simplest and best things you can do is “spec” the idea out. Open up Keynote, Powerpoint, or Google Present and create a fake “screenshot” of what you think your app should look like, describe what it would do, and outline why someone would use it. This is probably the simplest most straightforward thing you can do and the most helpful to get others to undestand and want to be part of your idea. I’ve done this a few times, and on several occasions, developers have taken the idea and created it!

The most important thing is to create something. That way others can read about it, build on it or critique/improve it. If you don’t create something, it can’t grow!

You can see an example here where I used keynote to edit a website to add a feature I thought it needed! (plus a whole blog post describing it)

2. Be an Astronaut. Document what you do.

During a recent hackathon I gave an opening address where I urged participants to “be an astronaut.” Not because astronauts are cool (they are). But because of the ultimate (somewhat depressing) responsibility astronauts have – they document and share everything they do so that the next person doesn’t suffer the same fate as them if anything goes wrong.

Hackathons are an effort to create a fun environment to encourage experimentation and foster learning, but that doesn’t mean people have to leanr the same lesson over and over again.

One of the most helpful thing I’ve seen people do during a hackathon is keep a running blog of what the developers were doing. They documented the steps they took, the ideas they researched and the scripts/libraries/etc… they used.

If the project was interesting, these “documentaries” can be helpful to get others involved. If the project is a disaster then, like an astronaut, your piece can be a warning to others so they don’t repeat your mistakes and try alternatives. Even a failure can inspire others to try something that ends up working. Indeed, that is how a lot of innovation takes place.

3. Identify some data worth scraping or share a new data set

While I love open data, there remains a lot of juicy, good government data that is, well… not open. So locating a juicy data set that you think could help shed new perspectives on a problem or help people take action is itself quite valuable. For example, I’m hoping to find a way to “scrape” (e.g. create a re-usable copy) of the City of Vancouver’s budget as well as the provincial budget.

But we don’t have to just focus on governments. I love the call to action in Oakland for its Open Data Day where they are asking non-profits:

“Do you work for or run an Oakland based nonprofit organization? Do you have data problems? Do you have data needs? Because if you do, we can help!”

I’ve always wanted to find ways to get non-profits involved in open data and just thinking about data analysis, so love that open data day can be a vehicle for this.

 

International #OpenDataDay: Now at 90 Cities (and… the White House)

Okay. We are 10 days away from International Open Data Day this February 23rd, 2013. There is now so much going on, I’ve been excited to see the different projects people are working on. Indeed there is so much happening, I thought I’d share just a tiny fraction of it in a little blog post to highlight the variety.

Again if you haven’t yet – please do see if there is an event near you and let the organizer know you are keen to come participate! As you see if you read below, this event is for everyone.

And if you are going – be sure to thank your local organizer. With roughly 90 or more events now scheduled world wide this is and remains a locally organized event. It is the organizers on the ground, who book the rooms, rally people and think of projects that make this day magical.

The White House joins International Open Data Day

Yes. You read that right. As you can read read here:

“We’re inviting a small group to join us in Washington, DC on February 22, 2013 for the White House Open Data Day Hackathon”

So if you are in the US and interested in participating, get on over to their website and apply. How cool would it be to hack on data at he White House?

More Organization Release Data in Anticipation of Open Data Day

One of the by products of open data day that we’ve been particularly happy about has been the reaction of governments and other organizations to release data in anticipation of the day to give developers, designers, data crunchers and every day citizens a new data set to play with.

Yesterday the  Building Performance Institute Europe (BPIE), a European not-for-profit think-do-tank made its online knowledge assets “open data ready” by launching an open data portal with facts and figures related to buildings and with a particular focus on the delivery of energy efficiency retrofits to existing buildings through addressing technical and financial barriers. This includes things like building stock performance (energy consumption, envelope performance, energy sources) and building stock inventories reflecting floor area, construction year, ownership profiles as well as national policies and regulation.

This could be of interest to people concerned with climate change and construction. I know there is a team in Vancouver and British Columbia that might find this data interesting, if only for benchmarking.

Global

Few people realize just how global Open Data is… here is a small sampling of some of the locations and how organized they are:

Getting it Done in Ghana

If you want to see what a tightly organized Open Data looks like, check out the agenda in Accra, Ghana.

Thinking about Poverty in the Philippines

There are a bunch of cool things happening in Manilla on Open Data Day but I love that one of them is focused on anti-poverty and the engagement with local NGOs:

“National Anti-Poverty Commission (NAPC) Open Data initiative (http://maps.napc.gov.ph). We welcome suggestions and comments to further improve our work.”

I know in Vancouver I’ll be talking to people about homelessness (a big priority here) and hope we’ll get some non-profits in the sector looking participating as well – particularly given the recent release of the city’s Rental Standards batabase (which lists outstanding infractions).

Lots of Community in Kathmandu

In Kathmandu they got stalls for a number of organizations related to open data, the open web and development such asOpen DRI, Open Data Nepal, IATI, Mozilla & Wikimedia, the OGP (LIG) and others. I also love that they are doing a course on how to edit a wiki. The focus on education is something we see everywhere… People come to Open Data Day to above all else, learn. 

School of Data in Amsterdam

Speaking of learning, one thing we’ve tried hard to emphasize is that Open Data day is not just for hackers. It is for anyone interested in community, learning and data. One group that has epitomized that has been the team in Amsterdam who are running a number of workshops, including some pretty wonkish ones such as exploring tax evasion working on the Open Data Census.

Building Community in Edinburgh (and every where)

I love how Edinburgh is focusing on getting people to talk about data, problems and code that can help one another. In many open data day events this is typical – as much time is spent learning, understanding and talking about how we can (and should or shouldn’t) use data to help with local problems. People are trying to figure out what this tool – open data – is and is not helpful for… all while connecting people in the community. Awesomeness.

Google Translate Required

And man, I don’t know what is going happening in Taipei (first open data event in Taiwan!!) but they have two tracks going on, so it has got to be serious! And it is hard to believe that in the first two years there were no events in Japan and this year there will be at least five. Something is happening there.

It makes me doubly happy when I see events where the wiki and comments are all in the local language – it reminds me of how locally driven the event is.

Hacking Open Data and Education – Open Science coutse

Billy Meinke of Creative Commons has posted that in Mountainview, “the Science Program at Creative Commons is teaming up with the Open Knowledge Foundation and members of the Open Science Community to facilitate the building of an open online course, an Introduction to Open Science.”

Participation in this event IS NOT LIMITED TO MOUNTAINVIEW. So check out their website if you want to participate.

Exciting.

Code Across America

If you live in the US and you don’t see an event in your community (or even if you do) also know that Code for America is running Code Across America that weekend. We love Code for America and they love open data, so I hope there is some cross pollination at some of these sites!

And much, much more…

This is just a small part of what will be happening. I’m going to be blogging some more on open data day.

I hope you’ll come participate!

CivicOpen: New Name, Old Idea

The other day Zac Townsend published a piece, “Introducing the idea of an open-source suite for municipal governments,” laying out the case for why cities should collaboratively create open source software that can be shared among them.

I think it is a great idea. And I’m thrilled to hear that more people are excited about exploring this model, and think any such discussion would be helped with having some broader context, and more importantly, because any series of posts on this subject that fails to look at previous efforts is, well, doomed to repeat the failures of the past.

Context

I wrote several blog posts making the case for it in 2009. (Rather than CivicOpen, I called it Muniforge.) These post, I’m told, helped influence the creation of CivicCommons, a failed effort to do something similar (and upon which I sat on an advisory board).

Back when I published my posts I thought I was introducing the idea of shared software development in the civic space. It was only shortly after that l learned of Kuali – a simliar effort that was occurring the university sector – and was so enamoured with it I wrote a piece about how we should clone its governance structure and create a simliar organization for the civic space (something that the Kuali leadership told me they would be happy to facilitate). I also tried tried to expose anyone I could to Kuali. I had a Kuali representative speak at the first Code for America Summit and have talked about it from time to time while helping teach the Code for America Fellows at the beginning of each CfA year. I also introduced them to anyone I would meet in municipal IT and helped spur conference calls between them and people in San Francisco, Boston and other cities so they could understand the model better.

But even in 2009 I was not introducing the idea of shared municipal open source projects. While discovering that Kuali existed was great (and perhaps I can be forgiven for my oversight, as they are in the educational and not municipal space) I completely failed to locate the Association des développeurs et utilisateurs de logiciels libres pour les administrations et les collectivités territoriales (ADULLACT) a French project created seven(!) years prior to my piece that does exactly what I described in my Muniforge posts and what Zac describes in his post. (The English translation of ADULLACT’s name is the Association of Developers and Users of Free Software for Governments and Local Authorities; there is no English Wikipedia page that I could see, but a good French version is here.) I know little about why ADULLACT has survived, but their continued existence suggests that they are doing something right – ideas and processes that should inform any post about what a good structure for CivicOpen or another initiative might want to draw upon.

Of course, in addition to these, there are several other groups that have tried – some with little, others with greater, success – to talk about or create open source projects that span cities. Indeed, last year, Berkeley Unviersity’s Smart Cities program proclaiming the Open Source city arrived. In that piece Berkeley references OpenPlans, which has for years tried to do something similar (and was indeed one of the instigators behind CivicCommons – a failed effort to get cities to share code. Here you can read Philip Ashlock, who was then at OpenPlans, talk about the desirability of cities creating and sharing open source code. In addition to CivicCommons, there is CityMart, a private sector actor that seeks to connect cities with solutions, including those that are open source; in essence it could be a catalyst for building municipal open source communities, but, as far as I can tell, it isn’t. (update) There was also the US Federal Government’s Open Code Initiative, which now seems defunct, and Mark Headd of Code for America tells me Google “Government Open Code Collaborative” but to which I can find no information on.

To understand why some models have failed and why some models have succeeded, Andrew Oram’s article in Journal of Information Technology and Politics is another good starting point. There was also some research into these ideas shared at the Major Cities of Europe IT Users Group back in 2009 by James Fogarty and Willoughby that can be read here; this includes several mini-case studies from several cities and a crude, but good, cost-benefit analysis.

And there are other efforts that are more random. Like  The Intelligent/Smart Cities Open Source Community which for “anyone interested on intelligent / smart cities development and looks for applications and solutions which have been successfully implemented in other cities, mainly open source applications.”

Don’t be ahistorical

I share all this for several reasons.

  1. I want Zac to succeed in figuring out a model that works.
  2. I’d love to help.
  3. To note that there has been a lot of thought into this idea already. I myself thought I was breaking ground when I wrote my Muniforge piece back in 2009. I was not. There were a ton of lessons I could have incorporated into that piece that I did not, and previous successes and failures I could have linked to, but didn’t (until at least discovering Kuali).

I get nervous when I see posts – like that on the Ash Centre’s blog – that don’t cite previous efforts and that feel, to put it bluntly, ahistorical. I think laying out a model is a great idea. But we have a lot of data and stories about what works and doesn’t work. To not draw on these examples (or even mention or link to them) seems to be a recipe for repeating the mistakes of the past. There are reasons CivicCommons failed, and why Kuali and ADDULACT have succeeded. I’ve interviewed a number of people at the former (and sadly no one at the latter) and this feels like table stakes before venturing down this path. It also feels like a good way of modelling what you eventually want a municipal/civic open source community to do – build and learn from the social code as well as business and organizational models of those that have failed and succeeded before you. That is the core of what the best and most successful open source (and, frankly many successful non-open source) projects do.

What I’ve learned is that the biggest challenge are not technical, but cultural and institutional. Many cities have policies, explicit or implicit that prevent them from using open source software, to say nothing of co-creating open source software. Indeed, after helping draft the Open Motion adopted by the City of Vancouver, I helped the city revise their procurement policies to address these obstacles. Indeed, drawing on the example mentioned in Zac’s post, you will struggle to find many small and medium sized municipalities that use Linux, or even that let employees install Firefox on computers. Worse, many municipal IT staff have been trained that Open Source is unstable, unreliable and involves questionable people. It is a slow process to reverse these opinions.

Another challenge that needs to be addressed is that many city IT departments have been hollowed out and don’t have the capacity to write much code. For many cities IT is often more about operations and selecting who to procure from, not writing software. So a CivicOpen/Muniforge/CivicCommons/ADULLACT approach will represent a departure into an arena where many cities have little capacity and experience. Many will be reluctant to built this capacity.

There are many more concerns of course and, despite them, I continue to think the idea is worth pursuing.

I also fear this post will be read as a critique. That is not my intent. Zac is an ally and I want to help. Above all, I share the above because   the good news is this isn’t an introduction. There is a rich history of ideas and efforts from which to learn and build upon. We don’t need do this on our own or invent anew. There is a community of us who are thinking about these things and have lessons to share, so let’s share and make this work!

A note to the Ash Centre

As an aside, I’d have loved to linked to this at the bottom of Zac’s post on the Havard Kennedy School Ash Centre website, but webmaster has opted to not allow comments. Even more odd is that it does not show any dates on their posts. Again, fearing I sound critical but just wanted to be constructive, I believe it is important for anyone, but academic institution especially to have dates listed on articles so that we can better understand the timing and context in which they were written. In addition, (and I understand this sentiment may not be shared) but a centre focused on Democratic Governance and Innovation should allow for some form of feedback or interaction, at least some way people can respond to and/or build on the ideas they publish.

International Open Data Day Feb 23rd: Vancouver Edition

So International Open Data is rapidly approaching! All around the world people are organizing local events to bring together developers, designers, policy wonks, non-profits, government officials, journalists, everyday citizens and others to play, chart, analyze, educate and/or build apps with open data.

For those of us who started International Open Data Day, it was never designed to be just a hackathon. Rather we’ve always wanted it to be an event that anyone interested in data, and interested in open data about their community in particular, could come to. So if you live in Vancouver and that is you… please sign up here. If you live somewhere else, check out the wiki as there are events happening all around the world.

So what do we have planned in Vancouver? And what will be some fun projects to work on?

Open Data Dating!

Well, this year, generously, the event will be taking place at City Hall and we are expecting some staff to be on hand. Following the lead from the excellent organizers in Ottawa were going to run some open data dating. Specifically, we’ll have city staff share with us what is some of the data they have, how they are using it, and answer questions participants may have. These conversations often spark ideas on behalf of both staff and participants about useful analysis or apps that could be created, or important data that should be collected.

Budget Visualization

Stéphane Guidoin from Montreal is trying to get people at Open Data Events across Canada (and possibly around the world) to input their city’s budget data into Where Does My Money Go so we can toy with creating some visualizations of city budgets. Not sure yet that we can get the budget data, but think it could be scrapped – but am nonetheless hopeful.

Homelessness and Rental Properties

One of the big priorities of city government is homelessness. The city is gearing up to launch a database of infractions affecting rental properties. Councillor Reimer – who is been a strong supporter of open data and addressing homelessness – will be on hand and has several ideas about how this data could be used to help the city, and residents, better understand the nature of some of the challenges around housing and foster support for more and better housing.

Air Quality Egg Hacking

My colleagues at the Centre for Digital Media at the Great Northern Way Campus have recently procured an Air Quality Egg and are hoping to explore how they can hack this hardware. We been dreaming up a scenario where we deploy may 10-30 of these around the lower mainland to get realtime measurements of the air quality. For the Centre for Digital Media, we’d love it if we can create a dashboard for the measurements from these eggs, as it would enable residents to compare air quality from neighborhood to neighborhood.

Biking Apps

Since I’m now just throwing ideas out there… (Cause that is what happens on open data day), another idea Councillor Reimer shared involves the newly released bike rack data from the City of Vancouver (which this fine gentleman mapped in very short order). An app that does nothing more than load, locate you, and point you to the direction of the nearest bike rack could be helpful to bikers. But if the location of the user could be anonymously shared with the city… it would be hugely valuable to them as it would provide some insight about where people are when they are looking for bike racks. This could allow the city to deploy its bike rack infrastructure more efficiently and save on (re)installation/moving costs.

And lots, lots more…

Obviously, there is lots more that happens, people network, brainstorm projects of their own, join established projects… or just learn about what is possible. Hopefully these ideas give you some insight into what is possible. That said, if you think you don’t have the right skills, please come anyways… we’ll find you a way to participate.

Presently the City of Montreal just tipped over 100 participants for their open data day. Ottawa regularly has similar participation levels. I confess that I’ve been a little slow in getting the word out, so please do consider coming and… pass the word along!

 

Again, one can sign up here.

The event will be taking place at City Hall on February 23rd, from 9:30am until 5:30pm.

 

International Open Data Hackathon Wiki now live – 14 cities added in 24 hours

Just a brief update for those interested in participating in or organizing an event for International Open Data Day on February 23rd, 2013.

The Wiki

The Open Data Day wiki (sans logo, so a little rough around the edges) is now live and ready for action.

The wiki is where organizers can list the city in which they’ll be putting together an event and where interested participants can find local events and let people know that they’ll be attending (to give organizers a sense of numbers) and what projects, workshops, discussions or other activities they are interested in participating in.

It has been awesome to see that in the first 24 hours of the wiki being up how 14 different people have stepped forward and listed their city as a place where they will be hosting an event. Better still, I’ve been contacted by numerous people saying they would like to host an event who have not yet added their city to list, so if you have not yet done it, please do so! The wiki is super easy to edit. If you are thinking of organizing event you can use my city, Vancouver, as a template for how to organize your city’s wiki page. But feel free to innovate and do you own thing!

Translations

A number of people have asked me about translating the opendata day website. We have not yet translated the version (except to portuguese which will be posted soon). So if you are interested in making a translation please do so. I’m happy to share that it is really easy!

The html for the site is on github. For those who know how (and it is VERY easy to learn) please feel free to fork the code and write a version in your language. I will approve it! Already THackday Portugal has done an updated portuguese version of the new site.

One warning, there is a good chance that I’ll edit (not heavily – but a little) the page once or twice before the event. I will let people know if I make changes.

Super excited about the response so far and hope the wiki helps people see what is going on and coordinate activities more effectively.

The Beneficial Impact of Newspaper Paywalls on Users

There continues to be fierce debate about the cost/benefits of newspaper paywalls, a debate Mathew Ingram has been helping drive with a great deal of depth and with excellent links.

It is interesting to watch Ingram take on, and have to rebut, the problematic thinking that seems to so frequently comes out of the Columbia Journalism Review which, sadly, as America’s most important journal on the industry and trainer of next generation journalist, is probably the most conservative voice in the debate. That said, while its contributions are defensive and disappointing, they are understandable. And well, it makes for fascinating reading of how people deal with an industry in decline/collapse/[insert whatever word you prefer here]. Someone should package these and make it mandatory stuff for MBA types.

Personally, I’m somewhat indifferent. If paywalls can save newspapers (every indication suggests that they cannot) then great! If they can’t… okay. The sky will not fall and the world will not end. We will have to figure out some new models for thinking about how we dispense news and discuss issues. Or, perhaps better still, might rethink what “news” and “media” is and means (psst… already happening).

I don’t want to join the holy war around paywalls, so I’m not trying to add to that debate directly – you should read Ingram and the others yourself.

What I do want to contribute is what I’m experiencing as a consumer. And what I can say is that I’ve found paywalls to be profoundly beneficial to me as an avid and significant news consumer, but in ways that I’m pretty sure the news industry is going to find disturbing and disappointing.

tweets-on-paper

So, what do I mean by that?

Well, in Canada several newspapers have thrown up paywalls – including the Globe and Mail (the main national newspaper) as well as the main “traditional” newspaper in my market – the Vancouver Sun. Each offers something like 20 free articles a month – akin to what the New York Times does. I also assume (but confess I don’t actually know) that if you arrive at an article via facebook or twitter, it does not count towards that total.

Now, every morning when I visit the home page of the Globe and the Sun and I see an article that might be interesting, I hover my mouse over it and ask myself the same question I suspect tens of thousands of others ask in that moment: “Is reading this article worth burning one my of my 20 free articles this month?” (or, framed another way: “is this article, sponge worthy?”)

And you know what? About 90% of the time, more actually, the answer is no.

And you know what else?

I’m grateful that this is the choice I’m being forced to make.

The internet is filled with news and articles that are wonderfully distracting and that I should probably not spend time reading. It turns out that by “internet” I also mean “pieces of news and articles in the pages of the Globe and the Sun.

What pay walls are reminding me of is that time is my most valuable (or scarce) resource, not access to content. By putting a price on their content the Globe, the Sun and everyone else with a paywall is simultaneously helping me put a “value” on my time. And that is a real service.

Interestingly, it makes getting a subscription is even less interesting to me. A subscription is basically a license for the Globe or Sun to eat up an even bigger chunk my time (which is scarce) with content (which is not scarce), much of which I really don’t need. This is not to say that there is not good content in either newspaper. There is. Sometimes some VERY good content. But about 99% of their content is either not relevant (or basically a form of entertainment) or worse, a kind of unhelpful distraction – what Clay Johnson would call “junk information” in his book, The Information Diet: A Case for Conscious Consumption (my review here). So why would I pay to get access behind a paywall when only 1% of the content is valuable to me and the other 99% is likely an unwanted time suck? Is the 1% worth it… so far… no. Indeed, if you are going to do paywalls, newspapers had better REALLY up their game.

And, well, if I just want to burn time and read fun things… well, the internet is the best thing in the world for that! There is almost unlimited quantities of high quality articles about things I don’t urgently need to know scattered all over it!

So I’m loving the paywall because it has made me more judicious about what I read. But that happens to mean less from these newspapers. In the end, if many people are like me (and, I’m a pretty avid news reader, so not really representational) I can’t see this boding well for the industry. Maybe some big papers, like the Guardian and the New York Times can survive on this model. I can’t see local newspapers, with their value chain and costs, surviving.

Indeed the other thing I’m paying for… the editorial piece, also becomes a starker choice to me. I may be an avid news reader, but the editor of these papers is not my editor – their audience is someone else (probably a boomer). Twitter continues to be my main news editor, particularly my “thought-leaders” list of people I respect (but don’t always agree with) give me a constant stream of interesting content.

I also want to make clear, I’m also not a “it must be free” kind of guy. I mean… I like free. But I don’t worship at its altar.

I pay for books all the time. I pay for my subscription to the Economist. There I feel about 60% of the content is really good. Of course – bigger caveat – I only get the digital edition, and I really care (e.g. pay a premium for) the fact that they read every edition to me. This means I can listen while exercising (running) or walking through airports. As a result the Economist doesn’t compete for my time in the same way the Globe or Sun is – but that is clever of them! They worked themselves into my less valuable “workout” and “commuting time”, not my more valuable “productivity” time. It means I also digest almost the entire thing every week.

So my sense is that pay walls can be a good thing for thoughtful users. I’m sure it will drive many to lower quality free content, but for many others it will help them think more judicious about how they spend their time. Frankly, not reading half the stuff I read and spending that time writing, reading something else, or spending time with my son would probably be a big boost to both my productivity and quality of life.

But that doesn’t solve any problems for newspapers.

Ontario's Open Data Policy: The Good, The Bad, The Ugly and the (Missed?) Opportunity

Yesterday the province of Ontario launched its Open Data portal. This is great news and is the culmination of a lot of work by a number of good people. The real work behind getting open data program launched is, by and large, invisible to the public, but it is essential – and so congratulations are in order for those who helped out.

Clearly this open data portal is in its early stages – something the province is upfront about. As a result, I’m less concerned with the number of data sets on the site (which however, needs to, and should, grow over time). Hopefully the good people in the government of Ontario have some surprises for us around interesting data sets.

Nor am I concerned about the layout of the site (which needs to, and should, improve over time – for example, once you start browsing the data you end up on this URL and there is no obvious path back to the open data landing page, it makes navigating the site hard).

In fact, unlike some I find any shortcomings in the website downright encouraging. Hopefully it means that speed, iteration and an attitude to ship early has won out over media obsessive, rigid, risk adverse approach governments all to often take. Time will tell if my optimism is warranted.

What I do want to focus on is the license since this is a core piece of infrastructure to an open data initiative. Indeed, it is the license that determines whether the data is actually open or closed. And I think we should be less forgiving of errors in this regard than in the past. It was one thing if you launched in the early days of open data two or four years ago. But we aren’t in early days anymore. There over 200 government open data portal around the world. We’ve crossed the chasm people. Not getting the license right is not a “beta” mistake any more. It’s just a mistake.

So what can we say about the Ontario Open Data license?

First, the Good

There is lots of good things to be said about it. It clearly keys off the UK’s Open Government License much like BC’s license did as does the proposed Canadian Open Government License. This means that above all, it is written in  plain english and is easily understood. In addition, the general format is familiar to many people interested in open data.

The other good thing about the license (pointed out to me by the always sharp Jason Birch) is that it’s attribution clause is softer than the UK, BC or even the proposed Federal Government license. Ontario uses the term “should” whereas the others use the term “must.”

Sadly, this one improvement pales in comparison to some of the problems and, most importantly the potentially lost opportunity I urgently highlight at the bottom of this post.

The Bad

While this license does have many good qualities initiated by the UK, it does suffer from some major flaws. The most notable comes in this line:

Ontario does not guarantee the continued supply of the Datasets, updates or corrections or that they are or will be accurate, useful, complete, current or free and clear of any possible third party copyright, moral right, other intellectual property right or other claim.

Basically this line kills the possibility that any business, non-profit or charity will ever use this data in any real sense. Hobbyests, geeks, academics will of course use it but this provision is deeply flawed.

Why?

Well, let me explain what it means. This says that the government cannot be held accountable to only release data it has the right to release. For example: say the government has software that tracks road repair data and it starts to release it and, happily all sorts of companies and app developers use it to help predict traffic and do other useful things. But then, one day the vendor that provided that road repair tracking software suddenly discovers in the fine print of the contract that they, not the government, own that data! Well! All those companies, non-profits and app developers are suddenly using proprietary data, not (open) government data. And the vendor would be entirely in its rights to go either sue them, or demand a license fee in exchange of letting them continue to use the data.

Now, I understand why the government is doing this. It doesn’t want to be liable if such a mistake is made. But, of course, if they don’t want to absorbe the risk, that risk doesn’t magically disappear, it transfers to the data user. But of course they have no way of managing that risk! Those users don’t know what the contracts say and what the obligations are, the party best positioned to figure that out is the government! Essentially this line transfers a risk to the party (in this case the user) that is least able to manage it. You are left asking yourself, what business, charity or non-profit is going to invest hundreds of thousands of dollars (or more) and people time to build a product, service or analysis around an asset (government data) that it might suddenly discover it doesn’t have the right to use?

The government is the only organization that can clear the rights. If it is unwilling to do so, then I think we need to question whether this is actually open data.

The Ugly

But of course the really ugly part of the license (which caused me to go on a bit of a twitter rant) comes early. Here it is:

If you do any of the above you must ensure that the following conditions are met:

  • your use of the Datasets causes no harm to others.

Wowzers.

This clause is so deeply problematic it is hard to know where to begin.

First, what is the definition of harm? If I use open data from the  Ontario government to rate hospitals and the some hospitals are sub-standard am I “harming” the hospital? Its workers? The community? The Ministry of Health?

So then who decides what the definition is? Well, since the Government of Ontario is the licensor of the data… it would seem to suggest that they do. Whatever the standing government of the data wants to decree is a “harm” suddenly becomes legit. Basically this clause could be used to strip many users – particularly those interested in using the data as a tool for accountability – of their right to use the data, simply because it makes the licensor (e.g. the government) uncomfortable.

A brief history lesson here for the lawyers who inserted this clause. Back in in March of 2011 when the Federal Government launched data.gc.ca they had a similar clause in their license. It read as follows:

“You shall not use the data made available through the GC Open Data Portal in any way which, in the opinion of Canada, may bring disrepute to or prejudice the reputation of Canada.”

While the language is a little more blunt, its effect was the same. After the press conference launching the site I sat down with Stockwell Day (who was the Minister responsible at the time) for 45 minutes and walked him through the various problems with their license.

After our conversations, guess how long it took for that clause to be removed from the license? 3 hours.

If this license is going to be taken seriously, that clause is going to have to go, otherwise, it risks being a laughing stock and a case study of what not to do in Open Government workshops around the world.

(An aside: What was particularly nice was the Minister Day personally called my cell phone to let me know that he’d removed that clause a few hours after our conversation. I’ve disagreed with Day on many, many, many things, but was deeply impressed by his knowledge of the open data file and his commitment to its ideals. Certainly his ability to change the license represents one of the fastest changes to policy I’ve ever witnessed.)

The (Missed?) Opportunity

What is ultimately disappointing about the Ontario license however is that it was never needed. Why every jurisdiction feels the need to invent its own license is always beyond me. What, beyond the softening of the attribution clause, has the Ontario license added to the Open Data world. Not much that I can see. And, as I’ve noted above, it many ways it is a step back.

You know data users would really like? A common license. That would make it MUCH easier to user data from the federal government, the government of Ontario and the Toronto City government all at the same time and not worry about compatibility issues and whether you are telling the end user the right thing or not. In this regard the addition of another license is a major step backwards. Yes, let me repeat that for other jurisdictions thinking about doing open data: The addition of another new license is a major step backwards.

Given that the Federal Government has proposed a new Open Government License that is virtually identical to this license but has less problematic language, why not simply use it? It would make the lives of the people who this license is supposed to enable  – the policy wonks, the innovators, the app developers, the data geeks – lives so much easier.

That opportunity still exists. The Government of Ontario could still elect to work with the Feds around a common license. Indeed given that the Ontario Open Data portal says they are asking for advice on how to improve this program, I implore, indeed beg, that you consider doing that. It would be wonderful if we could move to a single license in this country, and if a partnership between the Federal Government and Ontario might give such an initiative real momentum and weight. If not, into the balkanized abyss of a thousand licenses we wil stumble.