Tag Archives: open data

International Open Data Hackathon 2011: Better Tools, More Data, Bigger Fun

Last year, with only a month of notice, a small group passionate people announced we’d like to do an international open data hackathon and invited the world to participate.

We were thinking small but fun. Maybe 5 or 6 cities.

We got it wrong.

In the end people from over 75 cities around the world offered to host an event. Better still we definitively heard from people in over 40. It was an exciting day.

Last week, after locating a few of the city organizers email addresses, I asked them if we should do it again. Every one of them came back and said: yes.

So it is official. This time we have 2 months notice. December 3rd will be Open Data Day.

I want to be clear, our goal isn’t to be bigger this year. That might be nice if it happens. But maybe we’ll only have 6-7 cities. I don’t know. What I do want is for people to have fun, to learn, and to engage those who are still wrestling with the opportunities around open data. There is a world of possibilities out there. Can we seize on some of them?

Why.

Great question.

First off. We’ve got more data. Thanks to more and more enlightened governments in more and more places, there’s a greater amount of data to play with. Whether it is Switzerland, Kenya, or Chicago there’s never been more data available to use.

Second, we’ve got better tools. With a number of governments using Socrata there are more API’s out there for us to leverage. Scrapperwiki has gotten better and new tools like Buzzdata, TheDataHub and Google’s Fusion Tables are emerging every day.

And finally, there is growing interest in making “openess” a core part of how we measure governments. Open data has a role to play in driving this debate. Done right, we could make the first Saturday in December “Open Data Day.” A chance to explain, demo and invite to play, the policy makers, citizens, businesses and non-profits who don’t yet understand the potential. Let’s raise the world’s data literacy and have some fun. I can’t think of a better way than with another global open data hackathon – an maker’s fair like opportunity for people to celebrate open data by creating visualizations, writing up analyses, building apps or doing what ever they want with data.

Of course, like last time, hopefully we can make the world a little better as well. (more on that coming soon)

How.

The basic premises for the event would be simple, relying on 5 basic principles.

1. Together. It can be as big or as small, as long or as short, as you’d like it, but we’ll be doing it together on Saturday, December 3rd, 2011.

2. It should be open. Around the world I’ve seen hackathons filled with different types of people, exchanging ideas, trying out new technologies and starting new projects. Let’s be open to new ideas and new people. Chris Thorpe in the UK has done amazing work getting young and diverse group hacking. I love Nat Torkington’s words on the subject. Our movement is stronger when it is broader.

3. Anyone can organize a local event. If you are keen help organize one in your city and/or just participate add your name to the relevant city on this wiki page. Where ever possible, try to keep it to one per city, let’s build some community and get new people together. Which city or cities you share with is up to you as it how you do it. But let’s share.

4. You can work on anything that involves open data. That could be a local or global app, a visualization, proposing a standard for common data sets, scraping data from a government website to make it available for others in buzzdata.

It would be great to have a few projects people can work on around the world – building stuff that is core infrastructure to future projects. That’s why I’m hoping someone in each country will create a local version of MySociety’s Mapit web service for their country. It will give us one common project, and raise the profile of a great organization and a great project.

We also hope to be working with Random Hacks of Kindness, who’ve always been so supportive, ideally supplying data that they will need to run their applications.

5. Let’s share ideas across cities on the day. Each city’s hackathon should do at least one demo, brainstorm, proposal, or anything that it shares in an interactive way with at members of a hackathon in at least one other city. This could be via video stream, skype, by chat… anything but let’s get to know one another and share the cool projects or ideas we are hacking on. There are some significant challenges to making this work: timezones, languages, culture, technology… but who cares, we are problem solvers, let’s figure out a way to make it work.

Like last year, let’s not try to boil the ocean. Let’s have a bunch of events, where people care enough to organize them, and try to link them together with a simple short connection/presentation.Above all let’s raise some awareness, build something and have some fun.

What next?

1. If you are interested, sign up on the wiki. We’ll move to something more substantive once we have the numbers.

2. Reach out and connect with others in your city on the wiki. Start thinking about the logistics. And be inclusive. Someone new shows up, let them help too.

3. Share with me your thoughts. What’s got you excited about it? If you love this idea, let me know, and blog/tweet/status update about it. Conversely, tell me what’s wrong with any or all of the above. What’s got you worried? I want to feel positive about this, but I also want to know how we can make it better.

4. Localization. If there is bandwidth locally, I’d love for people to translate this blog post and repost it locally. (let me know as I’ll try cross posting it here, or at least link to it). It is important that this not be an english language only event.

5. If people want a place to chat with other about this, feel free to post comments below. Also the Open Knowledge Foundation’s Open Data Day mailing list will be the place where people can share news and help one another out.

Once again, I hope this will sound like fun to a few committed people. Let me know what you think.

The Science of Community Management: DjangoCon Keynote

At OSCON this year, Jono Bacon, argued that we are entering a era of renaissance in open source community management – that increasingly we don’t just have to share stories but that repeatable, scientific approaches are increasingly available to us. In short, the art of community management is shifting to a science.

With an enormous debt to Jono, I contend we are already there. Indeed the tools for enable a science of community management have existed for at least 5 years. All that is needed is an effort to implement them.

A few weeks ago the organizers of DjangoCon were kind enough to invite me to give the keynote at their conference in Portland and I made these ideas the centerpiece of my talk.

Embedded below is the result: a talk that that starts slowly, but that grew with passion and engagement as it progressed. I really want to thank the audience for the excellent Q&A and for engaging with me and the ideas as much as they did. As someone from outside their community, I’m grateful.

My hope in the next few weeks is to write this talk up in a series of blog posts or something more significant, and, hopefully, to redo this video in slideshare (although I’m going to have to get my hands on the audio of this). I’ll also be giving a version of this talk at the Drupal Pacific Northwest Summit in a few weeks. Feedback, as always, is not only welcome, but gratefully received. None of this happens in a vacuum, it is always your insights that help me get better, smarter and more on target.

Big thanks to Dierderik Van Liere and Lauren Bacon for inspiration and help as well as Mike Beltzner, Daniel Einspanjer, David Ascher and Dan Mosedale (among many others) at Mozilla who’ve been supportive and a big assistance.

In the meantime, I hope this is enjoyable, challenging and spurs good thoughts.

Interview with Charles Leadbeater – Monday September 19th

I’m excited to share that I’ll be interviewing British public policy and open innovation expert Charles Leadbeater on September 19th as part of a SIG’s webinar series. For readers not familiar with Charles Leadbeater, he is the author of We-Think and numerous other chapters, pamphlets and articles, ranging in focus from social innovation, to entrepreneurship to public sector reform. He served as an adviser to Tony Blair and has a long standing relationship with the British think tank Demos.

Our conversation will initially focus on open innovation, but I’m sure will range all over, touching on the impact of open source methodologies on the private, non-profit and public sector, the future of government services and, of course, the challenges and opportunities around open data.

If you are interested in participating in the webinar you can register here. There is a small fee I’m told is being charged to recover some of the costs for running the event.

If you are participating and have a question you’d like to see asked, or a theme or topic you’d like to see covered, please feel free to comment below or, if you prefer more discretion, send me an email.

Shared IT Services across the Canadian Government – three opportunities

Earlier this week the Canadian Federal Government announced it will be creating Shared Services Canada which will absorb the resources and functions associated with the delivery of email, data centres and network services from 44 departments.

These types of shared services projects are always fraught with danger. While they sometimes are successful, they are often disasters. Highly disruptive with little to show for results (and eventually get unwound). However, I suspect there is a significant amount of savings that can be made and I remain optimistic. With luck the analogy here is the work outgoing US CIO Vivek Kundra accomplished as he has sought to close down and consolidate 800 data centres across the US which is yielding some serious savings.

So here’s what I’m hoping Shared Services Canada will mean:

1) A bigger opportunity for Open Source

What I’m still more hopeful about – although not overly optimistic – is the role that open source solutions could play in the solutions Shared Services Canada will implement. Over on the Drupal site, one contributor claims government officials have been told to hold off buying web content management systems as the government prepares to buy a single solution for across all departments.

If the government is serious about lowering its costs it absolutely must rethink its procurement models so that open source solutions can at least be made a viable option. If not this whole exercise will mean the government may save money, but it will be the we move from 5 expensive solutions to one expensive solution variety.

On the upside some of that work has clearly taken place. Already there are several federal government websites running on Drupal such as this Ministry of Public Works website, the NRCAN and DND intranet. Moreover, there are real efforts in the open source community to accommodate government. In the United States OpenPublic has fostered a version of Drupal designed for government’s needs.

Open source solutions have the added bonus of allowing you the option of using more local talent, which, if stimulus is part of the goal, would be wise. Also, any open source solutions fostered by the federal government could be picked up by the provinces, creating further savings to tax payers. As a bonus, you can also fire incompetent implementors, something that needs to happen a little more often in government IT.

2) More accountability

Ministers Ambrose and Clement are laser focused on finding savings – pretty much every ministry needs to find 5 or 10% savings across the board. I also know both speak passionately about managing tax payers dollars: “Canadians work hard for their money and expect our Government to manage taxpayers dollars responsibly, Shared Services Canada will have a mandate to streamline IT, save money, and end waste and duplication.”

Great. I agree. So one of Shared Service Canada’s first act should be to follow in the footsteps of another Vivek Kundra initiative and recreate his incredibly successful IT Dashboard. Indeed it was by using the dashboard Vivek was able to “cut the time in half to deliver meaningful [IT system] functionality and critical services, and reduced total budgeted [Federal government IT] costs by over $3 billion.” Now that some serious savings. It’s a great example of how transparency can drive effective organizational change.

And here’s the kicker. The White House open sourced the IT Dashboard (the code can be downloaded here). So while it will require some work adapting it, the software is there and a lot of the heavy work has been done. Again, if we are serious about this, the path forward is straightforward.

3) More open data

Speaking of transparency… one place shared services could really come in handy is creating some data warehouses for hosting critical government data sets (ideally in the cloud). I suspect there are a number of important datasets that are used by public servants across ministries, and so getting them on a robust platform that is accessible would make a lot of sense. This of course, would also be an ideal opportunity to engage in a massive open data project. It might be easier to create policy for making the data managed by Shared Service Canada “open.” Indeed, this blog post covers some of the reasons why now is the time to think about that issue.

So congratulations on the big move everyone and I hope these suggestions are helpful. Certainly we’ll be watching with interest – we can’t have a 21st century government unless we have 21st century infrastructure, and you’re now the group responsible for it.

Calgary Launches Business Plan and Budget App

So this is interesting. The City of Calgary has launched a Business Plan & Budget app for free from iTunes.

It’s a smart move as it creates an easy, “one button” option for citizens to participate in and learn about the city’s financial planning process. You can read (a tiny bit) more at the City of Calgary’s blog.

Looking more closely at the app, it doesn’t offer a huge amount but don’t dismiss it too quickly. Consolidating all the information into a single place and making it available to people on the go is a great starting point. Secondly, it is worth remembering that this is just a starting point – there is obviously lots to be learned about how to engage citizens online – especially using mobile technology. If this is done right, Calgary will be learning these lessons first, which means their 2nd and 3rd generation versions of the app and the process will be more sophisticated while others are left catching up (think of Apple and the iPad).

So while the app is fairly light on features today… I can imagine a future where it becomes significantly more engaging and comprehensive, using open data on the data and city services to show maps of where and how money is spent, as well as post reminders for in person meet ups, tours of facilities, and dial in townhall meetings. The best way to get to these more advanced features is to experiment with getting the lighter features right today. The challenge for Calgary on this front is that it seems to have no plans for sharing much data with the public (that I’ve heard of), it’s open data portal has few offerings and its design is sorely lacking. Ultimately, if you want to consult citizens on planning and the budget it might be nice to go beyond surveys and share more raw data and information with them, it’s a piece of the puzzle I think will be essential. This is something no city seems to be tackling with any gusto and, along with crime data, is emerging as a serious litmus test of a city’s intention to be transparent.

The possibilities that Calgary’s consultation app presents are exciting – and again it is early days – so it will be interesting if developers in Calgary and elsewhere can begin to figuring out how to easily extend and enhance this type of approach. Moreover, it’s nice to see a city venturing out and experimenting with this technology, I hope other cities will not just watch, but start experiments of their own, it’s the best way to learn.

 

The Curious Case of Media Opposing Government Transparency

My gosh there is a lot going on. Republicans – REPUBLICANS(!) who were in charge of America’s prison system are warning Canada not to follow the Conservatives plan on prisons, the Prime Minister has renamed the government, after himself and my friends at Samara had in Toronto the Guardian’s Emily Bell to talk wikileaks and data journalism (wish I could have been there).

It’s all very interesting… and there is a media story here in British Columbia that’s been brewing where a number of journalists have become upset about a government that has become “too” transparent.

It’s an important case as it highlights some of the tensions that will be emerging in different places as governments rethink how they share information.

The case involves BC Ferries, a crown corporation that runs ferries along critical routes around the province. For many years the company was not subject to the province’s Freedom of Information legislation. However, a few months ago the government stated the crown corporation would need to comply with the act. This has not pleased the corporation’s president.

To comply with the act BC Ferries has created an FOI tracker website on which it posts the text of FOI requests received. Once the records are processed they are posted online and some relevant listservs. As a result they can be read by an audience (that cares).

Broadly, journalists, are up in arms for two reasons. One bad, the other even worse.

The terrible reasons was raised by Chad Skelton (who’s a great reporter for whom I have a lot of respect and whose column should be read regularly).

Skelton argues that BC Ferries deserves part of the blame for stories with errors as the process lead news agencies to rush (carelessly) in order to beat each other in releasing the story. This is a disappointing position. It’s the news media’s job to get the facts right. (It’s also worth noting here that Skelton’s own media organizations did not make the mistakes in question). Claiming that BC Ferries is even partly responsible seems beyond problematic since they are in no way involved in the fact and error checking processes. We trust the media (and assess it) to get facts right in fast moving situations… why should this be different?

More interesting is the critique that this model of transparency undermines the ability of journalists to get a scoup and thus undermines the business model of traditional media.

What makes this so interesting is that is neither true nor, more importantly, relevant.

First, it’s not the job of government to support the business model of the media. The goal of government should be to be as transparent as possible about its operations. This can, and should, include its FOI requests. Indeed, one thing I like about this process is that an FOI request that is made but isn’t addressed starts to linger on the site – and that the organization can be held to account, publicly, for the delay. More importantly, however, I’m confident that the media will find new ways to exploit the process and that, while painful, new business models will emerge.

Second, the media is not the only user of FOI. It strikes me as problematic to expect that the FOI system should somehow be tailored to meet needs alone. Individuals, non-profits, businesses, opposition politicians and others all use the FOI process. Indeed, the policy strengthens many of these use cases since, as mentioned above,  delays in processing will be visible and open the organization up to greater pressure and scrutiny. Why are all the use cases of these other institutions somehow secondary to those of journalists and the media? Indeed, the most important use case – that of the citizen – is better served. Isn’t that the most important outcome?

Third, this form of transparency could make for better media. One of my favourite quotes (which I got via Tim O’Reilly) comes from Clayton Christensen in a 2005 Harvard Business Review article:

“When attractive profits disappear at one stage in the value chain because a product becomes modular and commoditized, the opportunity to earn attractive profits with proprietary products will usually emerge at an adjacent stage.”

So BC Ferries has effectively commoditized FOI requests. That simply means that value will shift elsewhere. One place it could shift to is analysis. And wouldn’t that be a good thing to have the media compete on? Rather than simply who got the fact fastest (a somewhat silly model in the age of the internet) readers instead started to reward the organization with the best insights? Indeed, it makes me think that on superficial issues, like say, the salary of an employee, it may be hard for one individual or organization to scoop another. But most often the value of these stories is also pretty low. On a more significant story, one that requires research and digging and a knowledge of the issue, it’s unclear that transparency around FOI requests will allow others to compete. More interestingly, some media organizations, now that they have access to all FOI requests, might start analyzing them for deeper more significant patterns or trends that might reveal more significant problems that the current scattered approach to FOI might never reveal.

What’s also been interesting is the reaction stories by journalists complaining about this issue have been received. It fits nicely in with the piece I wrote a while ago (and now published as part of a journalism textbook) about Journalism in an Open Era. The fact is, the public trust of opaque institutions is in decline – and the media is itself a pretty opaque institution. Consider these three separate comments people wrote after the stories I’ve linked to above:

“I wonder over the years how many nuggets of information reporters got through FOI but the public never heard about because they didn’t deem it “newsworthy”. Or worse, that it was newsworthy but didn’t follow their storyline.” (found here)

“And the media whining about losing scoops — well, tough beans. If they post it all online and give it to everyone, they are serving the public –the media isn’t the public, and never has been.” (found here)

“The media’s track record, in general, for owning up to its blunders continues to be abysmal. Front page screw-ups are fixed several days (or weeks) later with a little “setting it straight” box buried at the bottom of P. 2 — and you think that’s good enough. If the media were more open and honest about fixing its mistakes, I might cut you a little slack over the BC Ferries’ policy of making your life difficult. But whining about it is going to be counterproductive, as you can see from most of the comments so far.” (found here)

While some comments were supportive of the articles, the majority have not been. Suggesting that at the minimum that the public does not share the media’s view that this new policy is a “controversial.”

This is not, of course, to say that BC Ferries implemented its policy because it sought to do the right thing. I’m sure it’s president would love for their to be fewer requests and impede the efforts of journalists. I just happen to think he will fail. Dismally. More concerning is the fact that FOI requests are not archived on the site and are removed after a few months. This is what should get the media, the public and yes, the Information and Privacy Commissioner, up in arms.

Canada ranks last in freedom of information

For those who missed it over the weekend it turns out Canada ranks last in freedom of information study that looked at the world’s western Parliamentary democracies. What makes it all the more astounding is that a decade ago Canada was considered a leader.

Consider two from the Information Commissioner, Suzanne Legault quotes pulled from the piece:

Only about 16 per cent of the 35,000 requests filed last year resulted in the full disclosure of information, compared with 40 per cent a decade ago, she noted.

And delays in the release of records continue to grow, with just 56 per cent of requests completed in the legislated 30-day period last year, compared with almost 70 per cent at the start of the decade.

These are appalling numbers.

The sad thing is… don’t expect things to get better. Why?

Firstly, the current government seems completely uninterested in access to information, transparency and proactive disclosure, despite these being core planks of its election platform and core values of the reform movement that re-launched Canadian conservatism. Indeed, reforming and improving access to information is the only unfulfilled original campaign promise of the Conservatives – and there appears to be no interest in touching it. Quite the opposite – that political staff now intervene to block and restrict Access to Information Requests – contravening the legislation and policy – is now a well known and documented fact.

Second, this issue is of secondary importance to the public. While everyone will say they care about access to information and open government, then number of people (while growing) still remains small. These types of reports and issues are of secondary importance. This isn’t to say they don’t matter. They do – but generally after something bigger and nastier has come to light and the public begins to smell rot. Then studies like this become the type of thing that hurts a government – it gives legitimacy and language to a sentiment people widely feel.

Third, the public seems confused about who they distrust more – the fact is, however bad the current government is on this issue, the Liberal brand is still badly tarnished on this issue of transparent government due to the scandals from almost a decade ago. Sadly, this means that there will be less burden on this government to act since – every time the issue of transparency and open government arise – rather than act, Government leaders simply point out the other parties failings.

So as the world moves on while Canada remains stuck, its government becoming more opaque, distant and less accountable to the people that elect it.

Interestingly , this also has a real cost to Canada’s influence in the world. It means something when the world turns to you as an expert – as we once were on access to information – minister’s are consulted by other world leaders, your public servants are given access to information loops they might otherwise not know about, there is a general respect, a soft power, that comes from being an acknowledged leader. Today, this is gone.

Indeed, it is worth noting that of the countries survey in the above mentioned study, only Canada and Ireland do not have open data portals which allow for proactive disclosure.

It’s a sign of the times.