At the 2010 GTEC conference I did a panel with David Strigel, the Program Manager of the Citywide Data Warehouse (CityDW) at the District of Columbia Government. During the introductory remarks David recounted the history of Washington DC’s journey to open data.
Interestingly, that journey began not with open data, but with an internal problem. Back around 2003 the city had a hypothesis that towing away abandoned cars would reduce crime rates in the immediate vicinity, thereby saving more money in the long term than the cost of towing. In order to access the program’s effectiveness city staff needed to “mash-up” longitudinal crime data against service request data – specifically, requests to remove abandoned cars. Alas, the data sets were managed by different departments, so this was tricky task. As a result the city’s IT department negotiated bilateral agreements with both departments to host their datasets in a single location. Thus the DC Data Warehouse was born.
Happily, the data demonstrated the program was cost effective. Building on this success the IT department began negotiating more bilateral agreements with different departments to host their data centrally. In return for giving up stewardship of the data the departments retained governance rights but reduced their costs and the IT group provided them with additional, more advanced, analytics. Over time the city’s data warehouse became vast. As a result, when DC decided to open up its data it was, relatively speaking, easy to do. The data was centrally located, was already being shared and used as a platform internally. Extending this platform externally (while not trivial) was a natural step.
In short, the deep problem that needed to solved wasn’t open data. Its was an information management. Getting the information management and governance policies right was essential for DC to move quickly. Moreover, this problem strikes at the heart of what it means to be government. Knowing what data you have, where it is, and under a governance structure that allows it to be shared internally (as well as externally) is a problem every government is going to face if it wants to be efficient, relevant and innovative in the 21st century. In other words, information management is the cake. Open data – which I believe is essential – is however the sweet icing you smother on top of that dense cake you’ve put in place.
Okay, with that said two points that flow from this.
First: Sometime, governments that “do” open data start off by focusing on the icing. The emphasis in on getting data out there, and then after the fact, figuring out governance model that will make sense. This is a viable strategy, but it does have real risks. When sharing data isn’t at the core function but rather a feature tacked on at the end, the policy and technical infrastructure may be pretty creaky. In addition, developers may not want to innovate on top of your data platform because they may (rightly) question the level of commitment. One reason DC’s data catalog works is because it has internal users. This gives the data stability and a sense of permanence. On the upside, the icing is politically sexier, so it may help marshal resources to help drive a broader rethink of data governance. Either way, at some point, you’ve got to tackle the cake, otherwise, things are going to get messy. Remember it took DC 7 years to develop its cake before it put icing on it. But that was making it from scratch. Today thanks to new services (armies of consultants on this), tools (eg. Socrata) and models (e.g. like Washington, DC) you can make that cake following a recipe and even use cake mix. As David Strigel pointed out, today, he could do it in a fraction of the time.
Second: More darkly, one lesson to draw from DC is that the capacity of a government to do open data may be a pretty good proxy for their ability to share information and coordinate across different departments. If your government can’t do open data in a relatively quick time period, it may mean they simply don’t have the infrastructure in place to share data internally all that effectively either. In a world where government productivity needs to rise in order to deal with budget deficits, that could be worrying.
eaves.ca 
David – great point at the end there.
My sense given the inevitable belt tightening is that departments and agencies are going to have to work hard at establishing the proper governance and policy frameworks around their data because they simply won’t be able to afford the costs of failed coordination anymore. Some departments, like my own, had the foresight to initiate the work already, giving us a distinct advantage over other agencies who are undoubtedly scrambling (or will be after their employees return from their vacations) to demonstrate that their organization is moving in the right direction.
That being said, the core challenge of that coordination will be the internal opposition, as you know, the people whose salaries are earned in the mysterious ether will undoubtedly claim that such coordination cannot be done because they know that that coordination undermines their power base. These people are ultimately in a position to slow the change by virtue of their positions on decision making pipelines.
This is where leadership becomes so important, if we truly want to achieve our goals in this area we need (at the risk of sounding cliché) leadership at all levels to combat those who have dug in their heels against the coming wave.
Cheers
David,
Good thoughts for a well-cooked recipe.
I like specially the idea showing a correlation of the time it takes a government to open data and the capacity for internal data sharing… Could even act as a measure indicator.
I do agree that open data is just the icing but I do also think that if a government wants open data succeed it will end up building the cake, no matter which comes first.
Cheers
In complete agreement that information management and data sharing–not open data–are the underlying drivers that will make these efforts sustainable. However, I also think open data has lit a fire under previously slow-moving attempts at data sharing.
As to whether lack of adoption of open data indicates poor internal infrastructure: poor infrastructure is a fact of life for most governments. Lack of enterprise architecture, multiple and redundant financial, HR, and case management systems, and numerous and disorganized offline processes are the rule, not the exception.
Rather than blaming the infrastructure for not being able to do open data, governments may be able to use the strategic value of open data as leverage to begin planning for and investing in better (cloud-based, anyone?) infrastructure.
To make a great cake you need a few things:
1 good ingredients
2 good equipment
3 great recipes
4 experience at using the above 3
While most governments have great ingredients (usually) they seriously are missing #2 and #3 in most cases, and therefore can’t possible get to #4.
I agree with your insights about open data being icing. I have worked for several years and have observed that while government most deseprately wants to modernize, they also have a cultural barrier around who has to be involved. Incompetence, status quo and sheer lack of initiative hamper lots of good ideas.
I also like to use the analogy of Open Data as the icing, but in a different context.
The Cake (multi-layered) represents gov’t organization with each layer being the people (culture), processes (workflows) and technology (IT infrastructure).
Today, Open Data is seen as merely the icing, sitting on top of the cake. Nice to have, but not essential. Sweet, but not required. It’s isolated from the “cake”, meaning open data doesn’t really have anything to do (yet) with the culture or workflows, perhaps a bit with technology (open data catalogues, data feeds, possible some citizen apps, etc.)
In the future I believe gov’t org’s will have a better alignment across each layer of the cake, allowing the “sweetness of open data” to penetrate to the core of government business (vs. being on the “Edge” or a nice-to-have as it’s perceived today).
When fully aligned, technology, processes & people can then take full advantage of the promise that Open Data holds, which is where Open Data becomes an asset at the core of government culture, with workflows and processes to both share out data and Gather Back meta-data or corrected data from the citizens, plus technology that supports that feedback loop. I talked about this at Open Gov West and recent OpenGovCanada Webinar (slideshare link: http://su.pr/22Ksg5)
That’s when Open Data and Gov’t become a perfect harmony between the icing and the cake.
Pingback: Cakes, Icing & Open Data | Port 25
Pingback: Information management is critical | GreenCityStreets Wiki
Pingback: Public Service Renewal: the weekly round-up « In the shuffle
Pingback: Shared IT Services across the Canadian Government – three opportunities | eaves.ca
Pingback: The US Government’s Digital Strategy: The New Benchmark and Some Lessons | eaves.ca
Pingback: Information management is critical | GreenCityStreets
Pingback: Information management is critical
Pingback: Information management is critical - Andy Nash