Category Archives: Uncategorized

Open Contracting Workshop in Montreal on Friday

This was sent to me by Michael Roberts who has been doing great work in the Open Data and Open Standards space in Canada. Please check this out:

What: Open Contracting Data Standard: Stakeholder Workshop

When: 31st January 2014 – 9am – 3pm

Where: Hilton Montreal Bonaventure 

Hashtag (always required): #opencontractingdata

Register at:

About the Open Contracting Data Standards project:

Over the course of 2014 the Open Contracting Partnership (OCP), the World Wide Web Foundation and wide range of stakeholders will be working to develop the first version of a global open data standard for publishing information on public contracts. The Open Contracting Partnership (OCP) believes that increased disclosure and participation in public contracting will make contracting more competitive and fair, will improve contract performance, and will better secure development outcomes. The development of an open contracting data standard is a vital step in joining up data across sectors and silos, allowing it to become truly socially useful. It will result in increased transparency around contracting processes and will empower citizens to be able to hold governments to account.

Stakeholder workshop:

We invite you to join us on 31st January 2014 for a stakeholder workshop to:

  • Explore the goals and potential for an Open Contracting Data Standard;
  • Identify opportunities for involvement in the development of the standard;
  • Shape core activities on the 2014 standard development road map;

Outline agenda (tbc):

  • 8.30am – 9.00 am: Welcome and coffee
  • 9.00am – 10.30am: The Open Contracting Road Map 
  • Including the history of the Open Contracting initiative; an introduction to the data standard project; and an exploration of data standard development so far.
  • 10.30am – 10.45am: Coffee break
  • 10.45am – 11.45am:  Shaping the vision & identifying stakeholders
  • Participative small group discussions focussed on outlining short and long-term visions for an open contracting data standard, and identifying the roles for key stakeholders to play in the development of the standard.
  • 11.45am – 12:30pm: Ways of working and key issues: an open development approach
  • Introducing the collaborative tools available for engaging with the development of the data standard, and identifying the key issues to be addressed in the coming months (the basis for task-groups in the afternoon).
  • 12.30pm – 1.30pm: Lunch
  • 1.30pm – 3.00pmTask groups
  • Small group work on specific issues, including the future governance of a standard, shared identifiers (e.g. organisational identifiers), data formats, demand side workshops, supply side research and related standards.

Who is the meeting for?

This meeting is designed to provide an opportunity for anyone interested in the Open Contracting Data Standard work to learn more about it. There will be discussions tailored to both providers of contracting data, and users of data, as well as discussions focussed at connecting the Open Contracting Data Standard with related open data projects, including IATI, Open Spending and open data on companies.

Santa Claus, Big Data and Asymmetric Learning

Any sufficiently advanced technology is indistinguishable from magic.

– Arthur C. Clarke’s Third Law of Prediction

This Christmas I had a wonderfully simple experience of why asymmetric rates of learning matter so much, and a simple way to explain it to friends and colleagues.

I have a young son. This is his first Christmas where he’s really aware of the whole Christmas thing: that there is a Santa Claus, there is a tree, people are being extra nice to one another. He’s loving it.

Naturally, part of the ritual is a trip to visit Santa Claus and so the other day, he embarked on his first visit with the big guy. Here’s a short version of the transcript:

Santa: “Hello Alec, would you like to talk to Santa?”

Alec: (with somewhat shy smile…) “Yes.” 

Santa: “So Alec, do you like choo-choo trains?”

Alec: (smile, eyes wide) “Yes.”

Santa: “Do you like Thomas the choo-choo train?”

Alec: (practically giggling, eyes super wide) “Yes!”

Reflecting on this conversation, all I can think is… no wonder kids believe in Santa Claus. I mean, here’s Alec’s first interaction with Santa ever and the guy, with no prompting, knows his name, knows one of his favourite things in the world, and even a specific type of toy related to that thing. Combine this with an appealing user interface (costumes, Christmas theme, and visitors treated like they’re very special) and of course it appears to actually be like magic. Alec must have thought Santa knew him personally.

It is easy to pretend this is something that only happens to two year-olds, but pretty much everyone I’ve met has, at one point, felt like LinkedIn or Facebook has been uncanny (for better or worse) in predicting a preference or connection.

The reality is there is just a massive asymmetry in learning. Most kids will have a single “Santa” interaction (or learning opportunity) a season – at most they’ll have two or three. Whereas a Santa in a shopping mall is going to meet thousands of kids (and thus have thousands of learning opportunities). Between these interactions and some basic research, any Santa worth his salt is going to pick up pretty quickly on what boys and girls tend to like and develop some quickly testable hypotheses about what any kid wants. Compared to the kids he meets, Santa is swimming in a world of big data: Lots of interactions and learning opportunities that allow him to appear magically aware.

The point here is that, as users, it is important for us to remember these things are not magic, but are driven by some pretty understandable tools. In addition, while we can’t always diminish the asymmetry in the rate of learning between users and the services they use, just understanding the dynamic can help demystify a service in a way that can be empowering to the user. I’m not going to ruin the magic of Santa’s tricks with my son, at least not for a few years – but I’m quite happy (and feel we are collectively responsible) to do so when it comes to online services.

Finally, it does show the power that increasing the rate of transactions can have on how quickly a system can learn. I’m spending more and more time trying to think about how systems – be they governments, non-profits or companies – can capture interactions and learning moments to help them become more effective, without pretending that it is magical – or infantilizing their users.

The Importance of Open Data Critiques – thoughts and context

Over at the Programmable City website Rob Kitchin has a thoughtful blog post on open data critiques. It is very much worth reading and wider discussion. Specifically, there are two competing things worth noting. First, it is important for the open data community – and advocates in particular – to acknowledge the responsibility we have in debates about open data. Second, I’d like to examine some of the critiques raised and discuss those I think misfire and those that deserve deeper dives.

Open Data as Dominant Discourse

During my 2011 keynote at Open Government Data camp I talked about how the open data movement was at an inflection point:

For years we have been on the outside, yelling that open data matters. But now we are being invited inside.

Two years later the transition is more than complete. If you have any doubts, consider this picture:OD as DCOnce you have these people talking about things like a G8 Open Data Charter you are no longer on the fringes. Not even remotely.

It also means understanding the challenges around open data has never been more important. We – open data advocates – are now complicit it what many of the above (mostly) men decide to do around open data. Hence the importance of Rob’s post. Previously those with power were dismissive of open data – you had to scream to get their attention. Today, those same actors want to act now and go far. Point them (or the institutions they represent) in the wrong direction and/or frame an issue incorrectly and you could have a serious problem on your hands. Consequently, the responsibility of advocates has never been greater. This is even more the case as open data has spread. Local variations matter. What works in Vancouver may not always be appropriate in Nairobi or London.

I shouldn’t have to say this but I will, because it matters so much: Read the critiques. They matter. They will make you better, smarter, and above all, more responsible.

The Four Critiques – a break down

Reading the critiques and agreeing with them is, of course, not the same thing. Rob cites four critiques of open data: funding and sustainability, politics of the benign and empowering the empowered, utility and usability, and neoliberalisation and marketisation of public services. Some of these I think miss the real concerns and risks around open data, others represent genuine concerns that everyone should have at the forefront of their thinking. Let me briefly touch on each one.

Funding and sustainability

This one strikes me as the least effective criticism. Outside the World Bank I’ve not heard of many examples where government effectively sell their data to make money. I would be very interested in examples to the contrary – it would make for a great list and would enlighten the discussion – although not, I suspect in ways that would make either side of the discussion happy.

The little research that has been done into this subject has suggested that charging for government data almost never yields much money, and often actually serves as a loss creating mechanism. Indeed a 2001 KPMG study of Canadian geospatial data found government almost never made money from data sales if purchases by other levels of government were not included. Again in Canada, Statistics Canada argued for years that it couldn’t “afford” to make its data open (free) as it needed the revenue. However, it turned out that the annual sum generated by these sales was around $2M dollars. This is hardly a major contributor to its bottom line. And of course, this does not count the money that had to go towards salaries and systems for tracking buyers and users, chasing down invoices, etc…

The disappointing line in the critique however was this:

de Vries et al. (2011) reported that the average apps developer made only $3,000 per year from apps sales, with 80 percent of paid Android apps being downloaded fewer than 100 times.  In addition, they noted that even successful apps, such as MyCityWay which had been downloaded 40 million times, were not yet generating profits.

Ugh. First, apps are not what is going to make open data interesting or sexy. I suspect they will make up maybe 5% of the ecosystem. The real value is going to be in analysis and enhancing other services. It may also be in the costs it eliminates (and thus capital and time it frees up, not in the companies it creates), something I outlined in Don’t Measure the Growth, Measure the Destruction.

Moreover, this is the internet. The average doesn’t mean anything. The average webpage probably gets 2 page views per day. That hardly means there aren’t lots of very successful webpages. The distribution is not a bell curve, its a long tail, so it is hard to see what the average tells us other than the cost of experimentation is very, very low. It tells us very little about if there are, or will be successful uses of open data.

Politics of the benign and empowering the empowered

The is the most important critique and it needs to be engaged. There are definitely cases where data can serve to further marginalize at risk communities. In addition, there are data sets that for reasons of security and privacy, should not be made open. I’m not interested in publishing the locations of women’s shelters or worse, the list of families taking refuge in them. Nor do I believe that open data will always serve to challenge the status quo or create greater equality. Even at its most reductionist – if one believes that information is power, then greater ability to access and make us of information makes one more powerful – this means that winners and losers will be created by the creation of new information.

There are however, two things that give me some hope in this space. The first is that, when it comes to open data, the axis of competition among providers usually centers around accessibility. For example, the Socrata platform (an provider of open data portals to government) invests heavily in creating tools that make government data accessible and usable to the broadest possible audience. This is not a claim that all communities are being engaged (far from it) and that a great deal more work cannot be done, but there is a desire to show greater use which drives some data providers to try to find ways to engage new communities.

The second is that if we want to create data literate society – and I think we do, for reasons of good citizenship, social justice and economic competitiveness – you need the data first for people to learn and play with. One of my most popular blog posts is Learning from Libraries: The Literacy Challenge of Open Data in which I point out that one of the best ways to help people become data literate is to give them more interesting data to play with. My point is that we didn’t build libraries after everyone knew how to read, we built them beforehand with the goal of having them as a place that could facilitate learning and education. Of course libraries also often have strong teaching components to them, and we definitely need more of this. Figuring out who to engage, and how it can be done most effectively is something I’m deeply interested in.

There are also things that often depress me. I struggle to think of technologies that did not empower the empowered – at least initially. From the cell phone to the car to the printing press to open source software, all these inventions have had helped billions of people, but they did not distribute themselves evenly, especially at first. So the question cannot be reduced to – will open data empower the empowered, but to what degree, and where and with whom. I’ve seen plenty of evidence where data has enabled small groups of people to protect their communities or make more transparent the impact (or lack there of) of a government regulation. Open data expands the number of people who can use government information for their own ends – this, I believe is a good thing – but that does not mean we shouldn’t be constantly looking for ways to ensure that it does not reinforce structural inequity. Achieving perfect distribution of the benefits of a new technology, or even public policy, is almost impossible. So we cannot make perfect the enemy of the good. However, that does not hide the fact that there are real risk – and responsibilities as advocates – that need to be considered here. This is an issue that will need to be constantly engaged.

Utility and Usability

Some of the issues around usability I’ve addressed above in the accessibility piece – for some portals (that genuinely want users) the axis of evolution is pointed in the right direction with governments and companies (like Socrata) trying to embed more tools on the website to make the data more usable.

I also agree with the central concern (not a critique) of this section, which is that rather than creating a virtuous circle, poorly thought out and launched open data portals will create a negative “doomloops” in which poor quality data begets little interest which begets less data. However, the concern, in my mind, focuses on to narrow a problem.

One of the big reasons I’ve been an advocate of open data was a desire not just to help citizens, non-profits and companies gain access to information that could help them with their missions, but to change the way government deals with its data so that it can share it internally more effectively. I often cite a public servant I know who had a summer intern spend 3 weeks surfing the national statistical agency website to find data they knew existed but could not find because of terrible design and search. A poor open data site is not just a sign that the public can’t access or effectively use government data, it usually suggests that the governments employees can’t access or effectively use their own data. This is often deeply frustrating to many public servants.

Thus, the most important outcome created by the open data movement may have been making governments realize that data represents an asset class that of which they have had little understanding (outside, sadly, the intelligence sector, which has been all too aware of this) and little policy and governance (outside, say, the GIS space and some personal records categories). Getting governments to think about data as a platform (yes, I’m a fan of government as a platform for external use, but above all for internal use) is, in my mind, one way we can both enable public servants to get better access to information while simultaneously attacking the huge vendors (like SAP and Oracle) whose $100 million dollar implementations often silo off data, rarely produce the results promised and are so obnoxiously expensive it boggles the mind (Clay Johnson has some wonderful examples of the roughly 50% of large IT projects that fail).

They key to all this is that open data can’t be something you slap on top of a big IT stack. I try to explain this in It’s the Icing Not the Cake, another popular blog post about why Washington DC was able to effectively launch an open data program so quickly (which was, apparently, so effective at bringing transparency to procurement data the subsequent mayor rolled it back). The point is, that governments need to start thinking in terms of platforms if – over the long term – open data is going to work. And it needs to start thinking of itself as the primary consumer of the data that is being served on that platform. Steve Yegge’s brilliant and sharp witted rant on how Google doesn’t get platforms is an absolute must read in this regard for any government official – the good news is you are not alone in not finding this easy. Google struggles with it as well.

My main point. Let’s not play at  the edges and merely define this challenge as one of usability. It is much, much bigger problem than that. It is a big, deep, culture-changing BHAG problem that needs tackling. If we get it wrong, then the big government vendors and he inertia of bureaucracy win. We get it right and we potentially could save taxpayers millions while enabling a more nimble, effective and responsive government.

Neoliberalisation and Marketisation of Government

If you not read Jo Bates article “Co-optation and contestation in the shaping of the UK’s Open Government Data Initiative” I highly recommend it. There are a number of arguments in the article I’m not sure I agree with (and feel are softened by her conclusion – so do read it all first). For example, the notion that open data has been co-opted into an “ideologically framed mould that champions the superiority of markets over social provision” strikes me as lacking nuance. One of the things open data can do is create a public recognition of a publicly held data set and the need to protect these against being privatized. Of course, what I suspect is that both things could be true simultaneously – there can be increased recognition of the importance of a public asset while also recognizing the increased social goods and market potential in leveraging said asset.

However, there is one thing Bates is absolutely correct about. Open data does not come into an empty playing field. It will be used by actors – on both the left and right – to advance their cause. So I too am uncomfortable with those that believe open data is going to somehow depoliticize government or politics – indeed I made a similar argument in a piece in Slate on the politics of data. As I try to point out you can only create a perverse, gerrymandered electoral district that looks like this…

gerrymandered in chicago… if you’ve got pretty good demographic data about target communities you want to engage (or avoid). Data – and even open data – doesn’t magically make things better. There are instances where open data can, I believe, create positive outcomes by shifting incentives in appropriate ways… but similarly, it can help all sorts of actors find ways to satisfy their own goals, which may not be aligned with your – or even society at large’s – goals.

This makes voices like Bates deeply important since they will challenge those of us interested in open data to be constantly evaluating the language we use, the coalitions we form and the priorities that get made, in ways that I think are profoundly important. Indeed, if you get to the end of Bates article there are a list of recommendations that I don’t think anyone I work with around open data would find objectionable, quite the opposite, they would agree are completely critical.


I’m so grateful to Rob for posting this piece. It is has helped me put into words some thoughts I’ve had, both about the open data criticisms as well as the important role the critiques play. I try hard to be critical advocate of open data – one who engages the risks and challenges posed by open data. I’m not perfect, and balancing these two goals – advocacy with a critical view – is not easy, but I hope this shines some window into the ways I’m trying to balance it and possible helps others do more of it as well.

On Vaccines, Incentives, Open Data and Public Policy

I know. Some mom coming out in favor of vaccines shouldn’t be breaking news. There’s nothing edgy about siding with most parents, nearly all the world’s governments and the vast majority of medical researchers and practitioners. But more of us need to do it.

And so begins JJ Keith – reknown blogger of all things parenting (I’m told) – titled I’m Coming Out… as Pro-Vaccine in the Huffington Post. Two days after getting published the piece has received a staggering 35,000+ facebook “likes” (a number, I suspect, that dwarfs the total likes of every “formal” public policy piece written in the last 48 hours combined – of course her’s IS a piece about public policy). And, it is a wonderfully written piece – filled with a dark sense of humour and concern that the topic deserves.

Keith’s main point is that more parents need to come out in favour of vaccines. That the central problem with the anti-vaccine movements is that they are loud and the rest of… well… are not. And I agree. It would be great if more of us stood up and talked about the importance of vaccines and engaged in this so called debate. Indeed, it may be the best of a number of not so great public policy options.

I’d love to think that there are simple policy measures that could fix what is, quite frankly, a matter of life and death. Requiring parents to produce proof of vaccination when entering their kids in public school feels like an obvious option. Greater transparency into vaccination rates in schools could also potentially spur parents to select schools that are “safer” from a health perspective. But not all those incentives in either case always push in the right direction.

So first, why does this matter?

When parents fail to vaccinate their children they don’t just put their own kids at risk of contracting measles, polio and other terrible diseases. Sadly, they put at risk newborns (who cannot be vaccinated) and – more critically – a chunk of the population who legitimately cannot be vaccinated or interestingly, who do get vaccinated but for whom the vaccination does not work.

This is why epidemiologists refer to “herd immunity” (it’s always nice when discussing public policy to refer to humanity as a “herd”). Since vaccines don’t work on everyone, enough people need to be vaccinated to prevent the disease from spreading reliably. The percentages required is usually north of 80% or 90% although I’m sure it varies a little based on the communicability of the disease.

Thus, what we actually have here is a free rider problem. If everyone vaccinates, then a few people opting out are probably safe if “the herd” remains sufficiently immune to the diseases. But drop below 80% and suddenly a tipping point is reached and things can get scary. Very scary. Frighteningly, there are whole (small) schools districts in California that fall below 50% immunized. And there are normal sized school districts that sit in the 60% and 70% range.

Indeed, thanks to the California Department of Public Health (CDPH), I was able to download the vaccination rates for every preschool in the state and poke through the raw assessment data myself (Sidenote: Dear CDPH administrators, please format your excel spreadsheets without merging title columns that make it impossible for users to sort the data, super frustrating). Shockingly two preschools, Kolbe Academy in Napa and the Waldorf School of Santa Barbara, had a zero kids – that is not one child – with all the required vaccines. Moreover, at Kolbe every single kid had a personal (not medical) exemption while at the Waldorf all but two had personal exemptions (meaning there parents didn’t want their kids to have vaccines) and the other two had simple not gotten around to it. Indeed at least 230 preschools in the state had kids with “personal exemptions” from vaccines that caused total immunization rates to fall below 80%.

These preschools are, in essence time bombs.

And some have already gone off! Just Googling “Whoop Cough Waldorf” (a chain of preschools that seems to dominate my poor immunization list) revealed a story about The East Bay Waldorf School – where only 44% of students are immunized – that had to be shut down by the local health authority because of an outbreak of Whooping Cough which is… easily prevent by a vaccine.

So a simple solution would be to require all kids who go to public schools to be vaccinated. This however, is already the rule in places like Santa Clara, and it while in part effective, it also drives  all the non-immunized kids into the same private schools, further diminishing the herd immunization effect and creating an ideal breeding ground for a dangerous disease.

Disturbingly, this is only desirable insofar as having a few outbreaks in schools like this cause parents to rethink their approach. But this is a terrible, terrible price to pay. It recently happened in Texas where a church that preached a deep skepticism of medicine and vaccines suffered a measles outbreak. Luckily no one died, but the church very quickly set up vaccination stations on its site. It is amazing how quickly one returns to fold of science and medicine once kids start contracting fatal diseases.

I also wouldn’t be surprised me if someone, using the data from the California Dept of Public Heath website created an web application that did a risk assessment of every school based on a number of factors of which vaccination rates would loom large. On the one hand it might cause parents to demand that other kids who attend the school also be vaccinated. On the flip side it might just accelerate the divisions between the anti-immunization camp and everyone else, creating even more dense clusters of schools where non of the kids are immunized.

In the end, short of a series of minor epidemics where enough kids die to scare non-conforming parents into getting their kids immunized, I fear that JJ Keith is doing the best possible thing: trying to build a coalition that will confront those spreading misinformation about vaccinations. Indeed, maybe a good first step would be organizing a boycott of Jenny McCarthy – a celebrity and “The View” co-hostess who has been one of the worst offenders in spreading misinformation about vaccines. Maybe we could call schools where more than 20% of the kids are not immunized because of their parents personal preference “McCarthy Schools.” And every time there is a measles breakout we can label it a “McCarthy Outbreaks” or worse, a “McCarthy death.” It’s strong language, but society maybe need to get tough to rebuild the social pressure needed to stomp out this problem… the other tools at our disposal are not, I fear, that strong.

Hello World & Berkman Affiliation

I’m back!

After a restful 1-2 month break from blogging, I’m re-emerging. Both energy and ideas wise. Can’t wait to share some of my new thoughts, as well as revisit some old ones.

I’ve also some fun news. This academic year I’ll be an affiliate at the Berkman Centre for Internet and Society. Indeed, I’m in Boston at the moment taking part in their open house and the launch of their year. I’ll continue to reside in Vancouver but hope to be in and out of Boston throughout the year, so if you are passing through, let me know. I’m also already enjoying the many fruitful conversations with the Berkman community and am drawing energy and ideas from them. Really looking forward to a year where I can start thinking of new ways in which I’d like to grow my intellectual capital.

A big part of the year for me is going to be about that, shedding topics that I feel I’ve talked enough about and beginning to explore new ones. Part of that will, hopefully, be reflected in my writing.

Also, Karen Fung helpfully pointed out that the link for the “my life” venn diagram on my about me page had broken, so I’ve taken it as a sign to update it. Many friends and colleagues have found it to be helpful when trying to understand or (worse!) describe some of what I do. It’s hardly a complete representation of my professional life, but it is a fun effort to try to visualize a good chunk of it. (BTW, if you decide to create your own, please send me a link, would love to see it)

My Life


Finally, I’d like to say thank to those that wrote me emails asking if everything was okay as I took a blogging hiatus. All is good! After almost 7 years of straight blogging, plus writing with TechPresident… I’m Just trying to recenter myself, manage my life (both as I grow, and as I manage new responsibilities, like my 21 month old son!) and think about what I’m doing and where I want to go.

Can’t wait to get back to writing. I hope some of what I say continues to be of interest, or of value to you too.

Some good articles on surveillance

There are a number of very good articles floating around as the NSA debacle begins to sink in. Two in particular come to mind, the first is a great long read, the second and third are just about the range of implications that will emerge over time of living in a surveillance state that continues to have many functioning democratic institutions:

The Ecuadorian Library

Lawyers eye NSA data as treasure trove for evidence in murder, divorce cases

It’s time Google came to grips with how it enables the surveillance state

Again, as outraged as Americans may be, we non-american have already had to come to grips with the fact that a drone strike can be called against us any time, any where, with no judicial oversight. And now we are coming to grips with the fact that the US government is monitoring everything we do online and building the capacity to store it, indefinitely.

I don’t think America has really grasped how far it has fallen from the “shining city upon a hill whose beacon light guides freedom-loving people everywhere.” This isn’t to say any one thinks that other players are better, but America sold the world not on a relative debate (hey, we’re better than the Russians) but on the promise of an ideal.

More on all of this on Tuesday but wanted to share the reads.


Some of you may have noticed I haven’t posted in a bit.

I’ve not stopped blogging! I’m just taking a break to recharge the batteries and do some thinking. The travel this year has been intense so I think I may have over done it a bit. Trying to catch up on some reading. I’ll be back soon.


What? No op-eds?

So, some of you may have noticed that it’s been a bit since Taylor and/or I have published an op-ed. There is a good reason for this. You can read all about it here.

It is a piece we’ve been wanting to write for quite some time – it is nice to finally have found a home for it. Yesterday we shipped off the final draft to the wonderful people at the LRC who – along with a few of our friends – have been very supportive and helpful.

Obviously when the LRC publishes its September edition and the piece is out I’ll publish a link.