Tag Archives: the long tail

Open Data: An Example of the Long Tail of Public Policy at Work

VancouverGraffiti_AnalysisAs many readers know, Vancouver passed what has locally been termed the Open3 motion a year ago and has had a open data portal up and running for several months.

Around the world much of the focus of open data initiatives have focused on the development of applications like Vancouver’s Vantrash, Washington DC’s Stumble Safely or Toronto’s Childcare locator. But the other use of data portals is to actually better understand and analyze phenomena in a city – all of which can potentially lead to a broader diversity of perspectives, better public policy and a more informed public and/or decision makers.

I was thus pleased to find out about another example of what I’ve been calling the Long Tail of Public Policy when I received an email from Victor Ngo, a student at the University of British Columbia who just completed his 2nd year in the Human Geography program with an Urban Studies focus (He’s also a co-op student looking for a summer job – nudge to the City of Vancouver).

It turns out that last month, he and two classmates did a project on graffiti occurrence and its relationship to land use, crime rates, and socio-economic variables. As Victor shared with me:

It was a group project I did with two other members in March/April. It was for an introductory GIS class and given our knowledge, our analysis was certainly not as robust and refined as it could have been. But having been responsible for GIS analysis part of the project, I’m proud of what we accomplished.

The “Graffiti sites” shapefile was very instrumental to my project. I’m a big fan of the site and I’ll be using it more in the future as I continue my studies.

So here we have University students in Vancouver using real city data to work on projects that could provide some insights, all while learning. This is another small example of why open data matters. This is the future of public policy development. Today Victor may be a student, less certain about the quality of his work (don’t underestimate yourself, Victor) but tomorrow he could be working for government, a think tank, a consulting firm, an insurance company or a citizen advocacy group. But wherever he is, the open data portal will be a resource he will want to turn to.

With Victor’s permission I’ve uploaded his report, Graffiti in the Urban Everyday – Comparing Graffiti Occurrence with Crime Rates, Land Use, and Socio-Economic Indicators in Vancouver, to my site so anyone can download it. Victor has said he’d love to get people’s feedback on it.

And what was the main drawback of using the open data? There wasn’t enough of it.

…one thing I would have liked was better crime statistics, in particular, the data for the actual location of crime occurrence. It would have certainly made our analysis more refined. The weekly Crime Maps that the VPD publishes is an example of what I mean:


You’re able to see the actual location where the crime was committed. We had to tabulate data from summary tables found at:


To translate: essentially the city releases this information in a non-machine-readable format, meaning that citizens, public servants at other levels of government and (I’m willing to wager) City of Vancouver public servants outside the police department have to recreate the data in a digital format. What a colossal waste of time and energy. Why not just share the data in a structured digital way? The city already makes it public, why not make it useful as well? This is what Washington DC (search crime) and San Francisco have done.

I hope that more apps get created in Vancouver, but as a public policy geek, I’m also hoping that more reports like these (and the one Bing Thom architects published on the future of Vancouver also using data from the open data catalog) get published. Ultimately, more people learning, thinking, writing and seeking solutions to our challenges will create a smarter, more vibrant and more successful city. Isn’t that what you’d want your city government (or any government, really…) to do?

ChangeCamp Vancouver

This weekend ChangeCamp comes to Vancouver. If you are interested definitely sign up early.

I’ll be there of course. But better still Shari Wallace (Director of IT for the City of Vancouver) and I will be running a session together from 3-4 pm to brainstorm what data the City of Vancouver should prioritize on opening up. It’s an opportunity for coders to suggest what might help them build the local apps they’ve always wanted to build.

That, and numerous other sessions will try to help us dive deeper into The Long Tail of Public Policy

So what is ChangeCamp and where will it be?

Saturday June 20th 2009 |  8:30 am – 5:30 pm

555 Seymour Street, Vancouver, BC (BCIT Downtown Campus)

$20 in advance | $25 at the door

Vancouver ChangeCamp is a participatory web-enabled face-to-face event that brings together citizens, technologists, designers, academics, social entrepreneurs, policy wonks, political players, change-makers and government employees to answer these questions:

  • How can we help government become more open and responsive?
  • How do we as citizens organize to get better outcomes ourselves?

The event is a partly structured unconference. One track of the conference will introduce the kinds of projects that harness new ideas and tools for social change. Other tracks at the conference will be participant-driven, with the agenda created collaboratively at the start of the event, allowing participants to share their experiences and expertise.

Hope to see you there!

Wedding Open Source to Government Service Delivery

One of the challenges I’m most interested in is how we can wed “open” systems to government hierarchies. In a lecture series I’ve developed for Health Canada I’ve developed a way of explaining how we do this already with our 911 service.

To begin, I like using 911 as an example because people are familiar and comfortable with it. More importantly, virtually everyone agrees that it is not only an essential piece of modern government service but also among the most effective.

What is interesting is that 911, unlike many government programs, relies on constant citizen input.  It is a system that has been architected to be participatory. Indeed it only works because it is participatory – without citizen input the system falls apart. Specifically, it aggregates, very effectively, the long-tail 0f knowledge within a community to deliver, with pin point accuracy, an essential service to the location it is needed at a time it is needed.

I’ve visualized in this slide below (explanation below the fold)

long tail public policy

Imagine the white curve represents all of the police, fire and ambulance interventions in a city. Many of the most critical interventions are ones the police force and ambulance service determine themselves (shaded blue). For example, the police are involved in an investigation that results in a big arrest, or the ambulance parks outside an Eagles reunion concert knowing that some of the boomers in attendance will be “over-served” and will suffer a heart attack.

However, while investigations and predictable events may account for some police/fire/ambulatory actions (and possibly those that receive the most press attention) the vast majority of arrests, fire fights and medical interventions result from plain old 911 calls made by ordinary citizens (shaded red). True, many of these are false alarms, or are resolved with minimal effort (a fire extinguisher deals with the problem, or minor amount of drugs are confiscated but no arrests made). But the sheer quantity of these calls means that while the average quality may be low, they still account for the bulk of successful (however defined) interventions. Viewed in this light 911 is a knowledge aggregator, collecting knowledge from citizens to determine where police cars, fire trucks and ambulances need to go.

Thus to find a system that leverages citizens knowledge and is architected for participation we don’t need to invent something new – there are existing systems, like 911, that we can learn from.

With this in mind, two important lessons about 911 leap out at me:

1) It is a self-interested system: While many 911 callers are concerned citizens calling about someone else I suspect the majority of calls – and the most accurate calls – are initiated by those directly or immediately impacted by a situation. People who have been robbed, are suffering from a heart attack, or who have a fire in their kitchen are highly incented to call 911. Consequently, the system leverages our self interest, although it also allows for good Samaritans to contribute as well.

2) It is narrowly focused in its construct: 911 doesn’t ask callers or permit callers to talk about the nature of justice, the history of fire, or the research evidence supporting a given medical condition. It seeks a very narrow set of data points: the nature of the problem and its location. This is helpful to both emergency response officials and citizens. It limits the quantity of data for the former and helps minimize the demands on the latter.

These, I believe, are the secret ingredients to citizen engagement of the future. A passive type of engagement that seeks specific, painless information/preferences/knoweldge from citizens to augment or redistribute services more effectively.

It isn’t sexy, but it works. Indeed we have 20 years of evidence showing us how well it works with regards to one of our most important services.

ChangeCamp: Pulling people and creativity out of the public policy long tail

ChangeCamp is a free participatory web-enabled face-to-face event that brings together citizens, technologists, designers, academics, policy wonks, political players, change-makers and government employees to answer one question: How do we re-imagine government and governance in the age of participation?

What is ChangeCamp? It is the application of “the long tail” to public policy.

It is a long held and false assumption that ordinary citizens don’t care about public policy. The statement isn’t, in of itself, false. Many, many, many people truly don’t care that much. They want to live their lives focusing on other things – pursuing other hobbies or interests – but there are many of us who do care. Public policy geeks, fans, followers, advocates, etc… we are everywhere, we’ve just been hidden in a long tail that saw the market place and capacity for developing and delivering public policy restricted to a few large institutions. The single most important lesson I learnt from my time with Canada25 is that it doesn’t have to be that way.

Did Canada25 get a new generation of Canadians, aged 20-35 engaged in public policy? I don’t know.

What I do know is, that at the very minimum, we harnessed and enormous, dispersed desire of many Canadians to participate in, and help shape, the public policy debates affecting the country. Most importantly, we did this by doing three things:

  1. we aggregated together the people who cared about public policy, we gave them peers, friends and a sense of community.
  2. we provided a vehicle through which to channel their energy
  3. by combining 1 and 2, and by using simple technology and a low cost approach – we dramatically lowered the barriers (and csots) to entry for credible participating in these national debates

Today, the technology to enable and aggregate people their ideas, to connect them with peers and to create community, is still more powerful. Our capacity to challenge, push, help, cooperate, leverage and compete with the large institutional public policy actors has never been greater. This, for me, is the goal of ChangeCamp. What concrete tools can we build, what information can we demand be opened up, what new relationships can we build to re-imagine how we – the citizens who care – participate in the creation of public policy and the effective delivery of public services. Not to compete or replace the traditional institutional actors, but to ensure more and better ideas are heard and increasingly effective and efficient services are created.

Long tail of public policy

Individually, none of us may have the collective power of a government ministry or even the resources of most think tanks. But collectively, linked together by technology and powered by our energy and spare capital, the long tail of policy geeks and ordinary citizens is bigger, nimbler, more creative and faster than anything else. Do I know that the long tail of policy can be set free? No. But ChangeCamp seems like a fun place to start experimenting, brainstorming and sharing ways we can make this country better.

Communities within Slideshare

So my presentation on Community Management as a Core Competency of Open Source recently passed the 7000 views mark. I admit that I find it somewhat incredible that one can create a lecture that gets viewed this many times. But still more interesting is seeing how the content and community around Slidecast has evolved.

Presently my presentation is the 74th most viewed slidecast on Slideshare – it just got run over by “What Teachers Make” which blew by it on its way up the charts (and rightly so, it really is quite good even if it doesn’t have sound). I am also pleased to note thought that my bit recently passed “Paris Hilton Photo Collector” and is moving in on the Beautiful Women of Japan and 50 Funny Cats.

I’ve been visiting slideshare a fair bit since I first posted to it 8 months ago. What has been interesting is that as it has gets more popular it seems that both the slidecasts get better (for example “What Teachers Make”), and that the more interesting slidecasts are polling better (note “What Teachers Make” meteoric rise). Indeed my own presenetation’s relative rise against cats and sexy women is further example of this trend. A year ago it seemed at least a fifth involved scantily clad women – but the days of those slideshows doing well appears to be in relative decline (they are still there, don’t get me wrong). It just appears that once  you are this deep in the long tail, lots of other content is more interesting to people.

All this is to say that yet again Andrew Keen should be roundly ignored, but then the very fact you are here probably means you were already ignoring him…

canadian history – long live the long tail?

So I’ve just started Chris Anderson’s audiobook version of The Long Tail and am loving it. No surprise here since I’ve already heard him lecture on it and so knew what I was getting into. But what has really peaked my interest is how Canadian history – that subject that everyone thinks the public has little to no appetite for, may be a perfect long tail example.

For those not familiar with The Long Tail thesis, Wikipedia describes it as follows:

“…products that have low sales volume can collectively make up a market share that rivals or exceeds the relatively few current bestsellers and blockbusters. Anderson cites earlier research on the relationship between Amazon sales and Amazon sales ranking and found a large proportion of Amazon.com’s book sales come from obscure books that are not available in brick-and-mortar stores.”

In other words, although most large publishing houses only look to publish the book that will make the top 10 best seller list (the green part of the graph), there is a huge market for those books that will only sell one or two copies every three months (the yellow part of the graph), but will do so over and over again over for a long period of time. All that is necessary to make this viable is a cheap distribution channel.

The point here is that there is still demand for lots of old goods, it is just that the relative demand – compared to the current blockbusters – is so tiny that no one notices it. Which brings me to books on Canadian history.

Peter C. Newman is a national treasure. When was the last time you looked at that man’s astounding catalog of books?  (This is not even a full list!) But did you realize that 90% of his books are no longer in print? And yet, many are just as relevant, and well researched today as when they were published 20 or even 35 years ago. The good news is that the Long Tail suggests Peter Newman’s work is still in demand. Indeed Canadian history more generally may not be a best seller but a constant churning demand is out there. One that, if fed, could fuel still greater interest.

The bad news is that most of Newman’s works are not publicized, or even published, anymore. This is what Lessig calls orphaned works: pieces still under copyright, but not in print and essential unavailable. This means that the potentially enourmous, but slow moving demand of The Long Tail, is not being met.

While discussing this problem over scotch in the wee hours of this morning we agreed that it would be great if Canadians, in complete violation of copyright opted to dictate the oldest of Newman’s works into their computers and publish the voice recordings online as free audiobook versions of his work? This would certainly create a cheap distribution channel for his works.

Would this make them bestsellers? No, but it would make them cheap and easy to disseminate. It would definitely open up his work to a whole new audience: the ipod generation. Maybe Peter C. Newman would even give us his blessing…