Tag Archives: policy

New Zealand: The World’s Lab for Progressive Tech Legislation?

6 Replies

Cross posted with TechPresident.

One of the nice advantage of having a large world with lots of diverse states is the range of experiments it offers us. Countries (or regions within them) can try out ideas, and if they work, others can copy them!

For example, in the world of drug policy, Portugal effectively decriminalized virtually all drugs. The result has been dramatic. And much of it positive. Some of the changes include a decline in both HIV diagnoses amongst drug users by 17% and drug use among adolescents (13-15 yrs). For those interested you can read more about this in a fantastic report by the Cato Institute written by Glenn Greenwald back in 2009 before he started exposing the unconstitutional and dangerous activities of the NSA. Now some 15 years later there have been increasing demands to decriminalize and even legalize drugs, especially in Latin America. But even the United States is changing, with both the states of Washington and Colorado opting to legalize marijuana. The lessons of Portugal have helped make the case, not by penetrating the public’s imagination per se, but by showing policy elites that decriminalization not only works but it saves lives and saves money. Little Portugal may one day be remembered for changing the world.

I wonder if we might see a similar paper written about New Zealand ten years from now about technology policy. It may be that a number of Kiwis will counter the arguments in this post by exposing all the reasons why I’m wrong (which I’d welcome!) but at a glance, New Zealand would probably be the place I’d send a public servant or politician wanting to know more about how to do technology policy right.

So why is that?

First, for those who missed it, this summer New Zealand banned software patents. This is a stunning and entirely sensible accomplishment. Software patents, and the legal morass and drag on innovation they create, are an enormous problem. The idea that Amazon can patent “1-click” (e.g. the idea that you pre-store someone’s credit card information so they can buy an item with a single click) is, well, a joke. This is a grand innovation that should be protected for years?

And yet, I can’t think of single other OECD member country that is likely to pass similar legislation. This means that it will be up to New Zealand to show that the software world will survive just fine without patents and the economy will not suddenly explode into flames. I also struggle to think of an OECD country where one of the most significant industry groups – the Institute of IT Professionals appeared – would not only both support such a measure but help push its passage:

The nearly unanimous passage of the Bill was also greeted by Institute of IT Professionals (IITP) chief executive Paul Matthews, who congratulated [Commerce Minister] Foss for listening to the IT industry and ensuring that software patents were excluded.

Did I mention that the bill passed almost unanimously?

Second, New Zealanders are further up the learning curve around the dangerous willingness their government – and foreign governments – have for illegally surveilling them online.

The arrest of Kim Dotcom over MegaUpload has sparked some investigations into how closely the country’s police and intelligence services follow the law. (For an excellent timeline of the Kim Dotcom saga, check out this link). This is because Kim Dotcom was illegally spied on by New Zealand’s intelligence services and police force, at the behest of the United States, which is now seeking to extradite him. The arrest and subsequent fall out has piqued public interest and lead to investigations including the Kitteridge report (PDF) which revealed that “as many as 88 individuals have been unlawfully spied on” by the country’s Government Communications Security Bureau.

I wonder if the Snowden documents and subsequent furor probably surprised New Zealanders less than many of their counterparts in other countries since it was less a bombshell than another data point on a trend line.

I don’t want to overplay the impact of the Kim Dotcom scandal. It has not, as far as I can tell, lead to a complete overhaul of the rules that govern intelligence gathering and online security. That said, I suspect, it has created a political climate that amy be more (healthily) distrustful of government intelligence services and the intelligence services of the United States. As a result, it is likely that politicians have been more sensitive to this matter for a year or two longer than elsewhere and that public servants are more accustomed at policies through the lens of its impact on rights and privacy of citizens than in many other countries.

Finally, (and this is somewhat related to the first point) New Zealand has, from what I can tell, a remarkably strong open source community. I’m not sure why this is the case, but suspect that people like Nat Torkington – and open source and open data advocate in New Zealand – and others like him play a role in it. More interestingly, this community has had influence across the political spectrum. The centre left labour party deserves much of the credit for the patent reform while the centre-right New Zealand National Party has embraced both open data. The country was among the first to embrace open source as a viable option when procuring software and in 2003 the government developed an official open source policy to help clear the path for greater use of open source software. This contrasts sharply with my experience in Canada where, as late as 2008, open source was still seen by many government officials as a dangerous (some might say cancerous?) option that needed to be banned and/or killed.

All this is to say that in both the public (e.g. civil society and the private sector) and within government there is greater expertise around thinking about open source solutions and so an ability to ask different questions about intellectual property and definitions of the public good. While I recognize that this exists in many countries now, it has existed longer in New Zealand than in most, which suggests that it enjoys greater acceptance in senior ranks and there is greater experience in thinking about and engaging these perspectives.

I share all this for two reasons:

First, I would keep my eye on New Zealand. This is clearly a place where something is happening in a way that may not be possible in other OECD countries. The small size of its economy (and so relative lack of importance to the major proprietary software vendors) combined with a sufficient policy agreement both among the public and elites enables the country to overcome both internal and external lobbying and pressure that would likely sink similar initiatives elsewhere. And while New Zealand’s influence may be limited, don’t underestimate the power of example. Portugal also has limited influence, but its example has helped show the world that the US -ed narrative on the “war on drugs” can be countered. In many ways this is often how it has to happen. Innovation, particularly in policy, often comes from the margins.

Second, if a policy maker, public servant or politician comes to me and asks me who to talk to around digital policy, I increasingly find myself looking at New Zealand as the place that is the most compelling. I have similar advice for PhD students. Indeed, if what I’m arguing is true, we need research to describe, better than I have, the conditions that lead to this outcome as well as the impact these policies are having on the economy, government and society. Sadly, I have no names to give to those I suggest this idea to, but I figure they’ll find someone in the government to talk to, since, as a bonus to all this, I’ve always found New Zealanders to be exceedingly friendly.

So keep an eye on New Zealand, it could be the place where some of the most progressive technology policies first get experimented with. It would be a shame if no one noticed.

(Again If some New Zealanders want to tell me I’m wrong, please do. Obviously, you know your country better than I do).

The End of the World: The State vs. the Internet

23 Replies

Last weekend at FooCamp, I co-hosted a session titled “The End of the World: Will the Internet Destroy the State, or Will the State Destroy the Internet?” What follows are the ideas I opened with during my intro to the session and some additional thoughts I’ve had and that others shared during the conversation. To avoid some confusion, I’d also like to clarify a) I don’t claim that these questions have never been raised before, I mostly hope that this framing can generate useful thought and debate; and b) that I don’t believe these are the only two or three possible outcomes; it was just a interesting way of framing some poles so as to generate good conversation.

Introduction

A while back, I thought I saw a tweet from Evgeny Morozov that said something to the effect: “You don’t just go from printing press to Renaissance to iPad; there are revolutions and wars in between you can’t ignore.” Since I can’t find the tweet, maybe he didn’t say it or I imagined it… but it sparked a line of thinking.

Technology and Change

Most often, when people think of the printing press, they think of its impact on the Catholic Church – about how it enabled Martin Luther’s complaints to go viral and how the localization of the Bible cut out the need of the middle man the priest to connect and engage with God. But if the printing press undermined the Catholic Church, it had the opposite impact on the state. To be fair, heads of state took a beating (see French Revolution et al.), but the state itself was nimbler and made good use of the technology. Indeed, it is worth noting that the modern notion of the nation state was not conceivable without the printing press. The press transformed the state – scaling up its capacity to demand control over loyalty from citizens and mobilize resources which, in turn, had an impact on how states related (and fought) with one another.

In his seminal book Imagined Communities, Benedict Anderson outlined how the printing press allowed the state to standardize language and history. In other words, someone growing up in Marseilles 100 years before the printing press probably had a very different sense of history and spoke a markedly different dialect of French than someone living in Paris during the same period. But the printing press (and more specifically, those who controlled it) allowed a dominant discourse to emerge (in this case, likely the Parisian one). Think standardized dictionaries, school textbooks and curricula, to say nothing of history and entertainment. This caused people who might never have met to share a common imagined history, language and discourse. Do not underestimate the impact this had on people’s identity. As this wonderful quote from the book states: “Ultimately it is this fraternity that makes it possible, over the past two centuries, for so many millions of people, not so much to kill, as willingly to die for such limited imaginings.” In other words, states could now fully dispense with feudal middle managers and harness the power of larger swaths of population directly – a population that might never actually meet, but could nonetheless feel connected to one another. The printing press thus helped create the modern nation state by providing a form of tribalism at scale: what we now call nationalism. This was, in turn, an important ingredient for the wars that dominated the late 19th and early 20th century – think World War I and World War II. This isn’t to say without the printing press, you don’t get war – we know that isn’t true – but the type of total war between 20th century nation states does have a direct line to the printing press.

So yes, the techno-utopian world of: printing press -> Renaissance -> iPad is not particularly accurate.

What you do get is: printing press -> Renaissance -> state evolution -> destabilization of international order -> significant bloodshed -> re-stabilization of international system -> iPad.

I raise all this because if this is the impact the printing press had on the state, it begs a new question: What will be the impact of the internet on the state? Will the internet be a technology the state can harness to extract more loyalty from its citizens… or will the internet destroy the imagined communities that make the state possible, replaced by a more nimble, disruptive organization better able to survive the internet era?

Some Scenarios

Note: again, these scenarios aren’t absolutes or the only possibilities, they are designed to raise questions and provoke thinking.

The State Destroys the Internet

One possibility is that the state is as adaptive as capitalism. I’m always amazed at how capitalism has evolved over the centuries. From mercantilism to free market to social market to state capitalism, as a meme it readily adapts  to new environments. One possibility is that the state is the same – sufficiently flexible to adapt to new conditions. Consequently, one can imagine that the state grabs sufficient control of the internet to turn it into a tool that at best enhances – and at worst, doesn’t threaten – citizens’ connection to it. Iran, with its attempt to build a state-managed internal network that will allow it to closely monitor its citizens’ every move, is a scary example of the former. China – with its great firewall – may be an example of the latter. But one not need pick on non-western states.

And a networked world will provide states – especially democratic ones – with lots of reasons to seize greater control of their citizens’ lives. From organized crime, to  terrorism, to identity theft, governments find lots of reasons to monitor their citizens. This is to say nothing of advanced persistent threats which create a state of continual online warfare – or sort of modern day phoney phishy war – between China, the United States, Iran and others. This may be the ultimate justification.

Indeed, as a result of these threats, the United States already has an extensive system for using the internet to monitor its own citizens and even my own country – Canada – tried to pass a law last year to significantly ramp up the monitoring of citizens online. The UK, of course, has just proposed a law whose monitoring provisions would make any authoritarian government squeal with glee. And just last week we found out that the UK government is preparing to cut a blank check for internet service providers to pay for installing the monitoring systems to record what its citizens do online.

Have no doubts, this is about the state trying to ensure the internet serves – or at least doesn’t threaten – its interests.

This is sadly, the easiest future to imagine since it conforms with the world we already know – one where states are ascendant. However, this future represents, in many ways, a linear projection of the future – and our world, especially our networked world, rarely behaves in a linear fashion. So we should be careful about confusing familiarity with probability.

The Internet Destroys the State

Another possibility is that the internet undermines our connection with the state. Online we become increasingly engaged with epistemic communities – be it social, like someone’s World of Warcraft guild, or professional, such as an association with a scientific community. Meanwhile, in the physical world, local communities – possibly at the regional level – become ascendant. In both cases, regulations and rules created by the state feel increasingly like an impediment to conducting our day to day lives, commerce and broader goals. Frustration flares, and increasingly someone in Florida feels less and less connection with someone in Washington state – and the common sense of identity, the imagined community, created by the state begins to erode.

This is, of course, hard for many people to imagine – especially Americans. But for many people in the world – including Canadians – the unity of the state is not a carefree assumption. There have been three referenda on breaking up Canada in my lifetime. More to the point, this process probably wouldn’t start in places where the state is strongest (such as in North America); rather, it would start in places where it is weakest. Think Somalia, Egypt (at the moment) or Belgium (which has basically functioned for two years without a government and no one seemed to really notice). Maybe this isn’t a world with no state – but lots of little states (which I think breaks with our mold of what we imagine the state to be to a certain degree) or maybe some new organizing mechanism, one which leverages local community identities, but can co-exist with a network of diffused but important transnational identities. Or maybe the organizing unit gets bigger, so that greater resources can be called upon to manage ne,w network-based threats.

I, like most people find this world harder to imagine. This is because so many of our assumptions suddenly disappear. If not the state, then what? Who or what protects and manages the internet infrastructure? What about other types of threats – corporate interests, organized and cyber-crime, etc.? This is true paradigm-shifting stuff (apologies for use of the word,) and frankly, I still find myself too stuck in my Newtonian world and the rules make it hard to imagine or even know what quantum mechanics will be like. Again, I want to separate imagining the future with its probability. The two are not always connected, and this is why thinking about this future, as uncomfortable and alienating as it may be, is probably an important exercise.

McWorld – The Internet Rewards the Corporation

One of the big assumptions I often find about people who write/talk about the internet is that it almost always assumes that the individual is the fundamental unit of analysis. There are good reasons for this – using social media, an individual’s capacity to be disruptive has generally increased. And, as Clay Shirky has outlined, the need for coordinating institutions and managers has greatly diminished. Indeed, Shirky’s blog post on the collapse of complex business models is (in addition to being a wonderful piece) a fantastic description of how a disruptive technology can undermine the capacity of larger complex players in a system and benefit smaller, simpler stakeholders. Of course, the smaller stakeholder in our system may not be the individual – it may be an actor that is smaller, nimbler than the state, that can foster an imagined community, and can adopt various forms of marshaling resources for self-organization to hierarchical management. Maybe it is the corporation.

During the conversation at FooCamp, Tim O’Reilly pressed this point with great effect. It could be that the corporation is actually the entity best positioned to adapt to the internet age. Small enough to leverage networks, big enough to generate a community that is actually loyal and engaged.

Indeed, it is easy to imagine a feedback loop that accelerates the ascendance of the corporation. If our imagined communities of nation states cannot withstand a world of multiple narratives and so become weaker, corporations would benefit not just from a greater capacity to adapt, but the great counterbalance to their power – state regulation and borders – might simultaneously erode. A world where more and more power – through information, money and human capital – gets concentrated in corporations is not hard to imagine. Indeed there are many who believe this is already our world. Of course, if the places (generally government bodies) where corporate conflicts – particularly those across sectors – cannot be mediated peacefully then corporations may turn much more aggressive. The need to be bigger, to marshal more resources, to have a security division to defend corporate interests, could lead to a growth in corporations as entities we barely imagine today. It’s a scary future, but not one that hasn’t been imagined several times in SciFi novels, and not one I would put beyond the realm of imagination.

The End of the World

The larger point of all this is that new technologies do change the way we imagine our communities. A second and third order impact of the printing press was its critical role in creating the modern nation-state. The bigger question is, what will be the second and third order impacts of the internet – on our communities (real and imagined), our identity and where power gets concentrated?

As different as the outcomes above are, they share one important thing in common. None represent the status quo. In each case, the nature of the state, and its relationship with citizens, shifts. Consequently, I find it hard to imagine a future where the internet does not continue to put a real strain on how we organize ourselves, and in turn the systems we have built to manage this organization. Consequently, it is not hard to imagine that as more and more of those institutions – including potentially the state itself – come under strain, it could very likely push systems – like the international state system – that are presently stable into a place of instability. It is worth noting that after the printing press, one of the first real nation states – France – wreaked havoc on Europe for almost a half century, using its enhanced resources to conquer pretty much everyone in its path.

While I am fascinated by technology and believe it can be harnessed to do good, I like to think that I am not – as Evgeny labels them – a techno-utopian. We need to remember that, looking back on our history, the second and third order effects of some technologies can be highly destabilizing, which carries with it real risks of generating significant bloodshed and conflict. Hence the title of this blog post and the FooCamp session: The End of the World.

This is not a call for a renewed Luddite manifesto. Quite the opposite – we are on a treadmill we cannot get off. Our technologies have improved our lives, but they also create new problems that, very often social innovations and other technologies will be needed to solve. Rather, I want to raise this because I believe it to be important that still more people – particularly those in the valley and other technology hubs (and not just military strategists) – be thinking critically about what the potential second and third order effects of the internet, the web and the tools they are creating, so that they can contribute to the thinking around potential technological, social and institutional responses that could hopefully mitigate against the worst outcomes.

I hope this helps prompt further thinking and discussion.

 

Public Policy: The Big Opportunity For Health Record Data

3 Replies

A few weeks ago Colin Hansen – a politician in the governing party in British Columbia (BC) – penned an op-ed in the Vancouver Sun entitled Unlocking our data to save lives. It’s a paper both the current government and opposition should read, as it is filled with some very promising ideas.

In it, he notes that BC has one of the best collections of health data anywhere in the world and that, data mining these records could yield patterns – like longitudinal adverse affects when drugs are combined or the correlations between diseases – that could save billions as well as improve health care outcomes.

He recommends that the province find ways to share this data with researchers and academics in ways that ensure the privacy of individuals are preserved. While I agree with the idea, one thing we’ve learned in the last 5 years is that, as good as academics are, the wider public is often much better in identifying patterns in large data sets. So I think we should think bolder. Much, much bolder.

Two years ago California based Heritage Provider Network, a company that runs hospitals, launched a $3 Million predictive health contest that will reward the team who, in three years, creates the algorithm that best predicts how many days a patient will spend in a hospital in the next year. Heritage believes that armed with such an algorithm, they can create strategies to reach patients before emergencies occur and thus reduce the number of hospital stays. As they put it: “This will result in increasing the health of patients while decreasing the cost of care.”

Of course, the algorithm that Heritage acquires through this contest will be proprietary. They will own it and I can choose who to share it with. But a similar contest run by BC (or say, the VA in the United States) could create a public asset. Why would we care if others made their healthcare system more efficient, as long as we got to as well. We could create a public good, as opposed to Heritage’s private asset. More importantly, we need not offer a prize of $3 million dollars. Several contests with prizes of $10,000 would likely yield a number of exciting results. Thus for very little money with might help revolutionize BC, and possibly Canada’s and even the world’s healthcare systems. It is an exciting opportunity.

Of course, the big concern in all of this is privacy. The Globe and Mail featured an article in response to Hansen’s oped (shockingly but unsurprisingly, it failed to link back to – why do newspaper behave that way?) that focused heavily on the privacy concerns but was pretty vague about the details. At no point was a specific concern by the privacy commissioner raised or cited. For example, the article could have talked about the real concern in this space, what is called de-anonymization. This is when an analyst can take records – like health records – that have been anonymized to protect individual’s identity and use alternative sources to figure out who’s records belong to who. In the cases where this occurs it is usually only only a handful of people whose records are identified, but even such limited de-anonymization is unacceptable. You can read more on this here.

As far as I can tell, no one has de-anonymized the Heritage Health Prize data. But we can take even more precautions. I recently connected with Rob James – a local epidemiologist who is excited about how opening up anonymized health care records could save lives and money. He shared with me an approach taking by the US census bureau which is even more radical than de-anonymization. As outlined in this (highly technical) research paper by Jennifer C. Huckett and Michael D. Larsen, the approach involves creating a parallel data set that has none of the features of the original but maintains all the relationships between the data points. Since it is the relationships, not the data, that is often important a great deal of research can take place with much lower risks. As Rob points out, there is a reasonably mature academic literature on these types of privacy protecting strategies.

The simple fact is, healthcare spending in Canada is on the rise. In many provinces it will eclipse 50% of all spending in the next few years. This path is unsustainable. Spending in the US is even worse. We need to get smarter and more efficient. Data mining is perhaps the most straightforward and accessible strategy at our disposal.

So the question is this: does BC want to be a leader in healthcare research and outcomes in an area the whole world is going to be interested in? The foundation – creating a high value data set – is already in place. The unknown is if can we foster a policy infrastructure and public mandate that allows us to think and act in big ways. It would be great if government officials, the privacy commissioner and some civil liberties representatives started to dialogue to find some common ground.  The benefits to British Columbians – and potentially to a much wider population – could be enormous, both in money and, more importantly, lives, saved.

Neo-Progressive Watch: Rahm Emanuel vs. Teachers Union

3 Replies

Anyone who read Obama’s book, The Audacity of Hope will have been struck with the amount of time the then aspiring presidential candidate spent writing about public education policy. More notably, he seemed to acknowledge that any effort at education reform was, at some point, going to butt heads with teachers unions and that new approaches were either going to have to be negotiated or imposed. It was a point of tension that wasn’t much talked about in the reviews I read. But it always struck me as interesting that here was Obama, a next generation progressive, railing against the conservatism of what is possible the original pillar of the progressive movement: public education.

All of this has, of course, been decidedly forgotten given both the bigger problems the president has faced and by the fact that he’s been basically disinterested in monkeying around in public education policy since taking office. That’s why it is still more fascinating to see what his disciples are doing as they get involved in levels of government that are in more direct contact with this policy area. Here, none is more interesting to watch than Rahm Emanuel.

This Saturday my friend Amy L. pointed me to a New York Times article outlining the most recent battle between Rahm Emanuel and the teacher’s union. My own take is that the specifics of the article are irrelevant, what matters is the broad theme. In short, Rahm Emanuel is on a short timeline. He needs to produce results immediately since local elections both happen more frequently and one is much, much closer to the citizen. That said, he doesn’t have to deliver uniform results, progress, in of itself may be sufficient. Indeed, a little experimentation is profoundly good given it can tease out faster and cheaper ways to deliver said results.

In contrast, the teacher’s union faces few of the pressures experienced by Rahm. It can afford to move at a slower pace and, more importantly, wants a uniform level of treatment across the entire system. Indeed, its entire structure is built around the guarantee of uniform treatment for its members. This uniformity is a value that evolved parallel to but not of progressive thinking. It is an artifact of industrial production that gets confused with progressive thought because of the common temporal lineage.

This skirmish offers a window into the major battle that is going to dominate the our politics in about a decade. I increasingly suspect we are moving into a world where the possibilities for education, thanks to the web and social networks, is going to be completely altered. What we deem is possible, what parents demand, and the skills that are seen as essential, are all going to shift. Our educational system, its schools, the school boards and, of course, the unions, are still bound in a world of mass production – shifting students from room to room to prepare them for the labour and production jobs of the 20th century. No matter how gifted the teachers (and there are many who are exceedingly gifted) they remain bound by the structure of the system the education system, the school boards, and the unions, have built and enforce.

Of course, what is going to be in demand are students that can thrive in the world of mass collaboration and peer production in the 21st century -behaviours that are generally viewed as “cheating” in the current model. And parents who are successful in 21st century jobs are going to be the first to ensure their children get the “right” kind of education. Which is going to put them at odds with the current education system.

This is all this is to say that the real question crisis is: how quickly will educational systems be able to adapt? Here both the school boards and the unions play an enormous role, but it is the unions that, it would appear, may be a constraining factor. If they find that having Rahm engage schools directly feels like a threat, I suspect they are going to find the next 20 years a rough, rough ride. Something akin to how the newspapers have felt regarding the arrival of the internet and craigslist.

What terrifies me most, is that unless we can devise a system where teachers are measured and so good results can be both rewarded and shared… and where parents and students have more choices around education, then families (that can afford to) are going to vote with their feet. In fact, you already see it in my home town.

The myth in Vancouver is that high property values are driving families – and thus children – out of the city. But this is patently not true. The fantastic guys over at Bing Thom Architects wrote a report on student populations in Vancouver. According to their research, in the last 10 years the estimated number of elementary and secondary aged children in Vancouver has risen by 3% (around 2,513 new students). And yet, the number of students enrolled in public education facilities has declined by 5.46%. (around 3,092 students). In fact, the Vancouver School Boards numbers seem to indicate the decline may be more pronounced.

In the meantime the number of private/independent schools has exploded by 43% going from 39 to 68 with enrollment increases of 13.8%. (Yes that does leave a surplus of students unaccounted for, I suspect they are also in private/independent schools, but outside of the City of Vancouver’s boundaries). As a public school graduate myself, one who had truly fantastic teachers but who also benefited from enormous choice (IB, French Immersion) the numbers of the past decade are very interesting to immerse oneself in.

Correct or incorrect, it would seem parents are opting for schools that offer a range of choices around education. Of course, it is only the parents who can afford to do this that are doing it. But that makes the outcome worse, not better. With or without the unions, education is going to get radically rethought. It would be nice if it was the public sector that lead that revolution, or at least was on the vanguard of it. But if our public sector managers and teachers are caught arguing over how to adjust the status quo by increments, it is hard to see how our education policy is going to make a quantum leap into the 21st century.

It's the icing, not the cake: key lesson on open data for governments

12 Replies

At the 2010 GTEC conference I did a panel with David Strigel, the Program Manager of the Citywide Data Warehouse (CityDW) at the District of Columbia Government. During the introductory remarks David recounted the history of Washington DC’s journey to open data.

Interestingly, that journey began not with open data, but with an internal problem. Back around 2003 the city had a hypothesis that towing away abandoned cars would reduce crime rates in the immediate vicinity, thereby saving more money in the long term than the cost of towing. In order to access the program’s effectiveness city staff needed to “mash-up” longitudinal crime data against service request data – specifically, requests to remove abandoned cars. Alas, the data sets were managed by different departments, so this was tricky task. As a result the city’s IT department negotiated bilateral agreements with both departments to host their datasets in a single location. Thus the DC Data Warehouse was born.

Happily, the data demonstrated the program was cost effective. Building on this success the IT department began negotiating more bilateral agreements with different departments to host their data centrally. In return for giving up stewardship of the data the departments retained governance rights but reduced their costs and the IT group provided them with additional, more advanced, analytics. Over time the city’s data warehouse became vast. As a result, when DC decided to open up its data it was, relatively speaking, easy to do. The data was centrally located, was already being shared and used as a platform internally. Extending this platform externally (while not trivial) was a natural step.

In short, the deep problem that needed to solved wasn’t open data. Its was an information management. Getting the information management and governance policies right was essential for DC to move quickly. Moreover, this problem strikes at the heart of what it means to be government. Knowing what data you have, where it is, and under a governance structure that allows it to be shared internally (as well as externally) is a problem every government is going to face if it wants to be efficient, relevant and innovative in the 21st century. In other words, information management is the cake. Open data – which I believe is essential – is however the sweet icing you smother on top of that dense cake you’ve put in place.

Okay, with that said two points that flow from this.

First: Sometime, governments that “do” open data start off by focusing on the icing. The emphasis in on getting data out there, and then after the fact, figuring out  governance model that will make sense. This is a viable strategy, but it does have real risks. When sharing data isn’t at the core function but rather a feature tacked on at the end, the policy and technical infrastructure may be pretty creaky. In addition, developers may not want to innovate on top of your data platform because they may (rightly) question the level of commitment. One reason DC’s data catalog works is because it has internal users. This gives the data stability and a sense of permanence. On the upside, the icing is politically sexier, so it may help marshal resources to help drive a broader rethink of data governance. Either way, at some point, you’ve got to tackle the cake, otherwise, things are going to get messy. Remember it took DC 7 years to develop its cake before it put icing on it. But that was making it from scratch. Today thanks to new services (armies of consultants on this), tools (eg. Socrata) and models (e.g. like Washington, DC) you can make that cake following a recipe and even use cake mix. As David Strigel pointed out, today, he could do it in a fraction of the time.

Second: More darkly, one lesson to draw from DC is that the capacity of a government to do open data may be a pretty good proxy for their ability to share information and coordinate across different departments. If your government can’t do open data in a relatively quick time period, it may mean they simply don’t have the infrastructure in place to share data internally all that effectively either. In a world where government productivity needs to rise in order to deal with budget deficits, that could be worrying.