Tag Archives: hacker

The State of Open Data 2011

What is the state of the open data movement? Yesterday, during my opening keynote at the Open Government Data Camp (held this year in Warsaw, Poland) I sought to follow up on my talk from last year’s conference. Here’s my take of where we are today (I’ll post/link to a video of the talk as soon as the Open Knowledge Foundation makes it available).

Successes of the Past Year: Crossing the Chasm

1. More Open Data Portals

One of the things that has been amazing to witness in 2011 is the veritable explosion of Open Data portals around the world. Today there are well over 50 government data catalogs with more and more being added. The most notable of these was probably the Kenyan Open Data catalog which shows how far, and wide, the open data movement has grown.

2. Better Understanding and More Demand

The things about all these portals is that they are the result of a larger shift. Specifically, more and more government officials are curious about what open data is. This is not to say that understanding has radically shifted, but many people in government (and in politics) now know the term, believe there is something interesting going on in this space, and want to learn more. Consequently, in a growing number of places there is less and less headwind against us. Rather than screaming from the rooftops, we are increasingly being invited in the front door.

3. More Experimentation

Finally, what’s also exciting is the increased experimentation in the open data space. The number of companies and organizations trying to engage open data users is growing. ScraperWiki, the DataHub, BuzzData, Socrata, Visua.ly, are some of the products and resources that have emerged out of the open data space. And the types of research and projects that are emerging – the tracking of the Icelandic volcano eruptions, the emergence of hacks and hackers, micro projects (like my own Recollect.net) and the research showing that open data could be generating savings of £8.5 million a year to governments in the Greater Manchester area, is deeply encouraging.

The Current State: An Inflection Point

The exciting thing about open data is that increasingly we are helping people – public servants, politicians, business owners and citizens imagine a different future, one that is more open, efficient and engaging. Our impact is still limited, but the journey is still in its early days. More importantly, thanks to success (number 2 above) our role is changing. So what does this mean for the movement right now?

Externally to the movement, the work we are doing is only getting more relevant. We are in an era of institution failure. From the Tea Party to Occupy Wall St. there is a recognition that our institutions no longer sufficiently serve us. Open data can’t solve this problem, but it is part of the solution. The challenge of the old order and the institutions it fostered is that its organizing principle is built around the management (control) of processes, it’s been about the application of the industrial production model to government services. This means it can only move so fast, and because of its strong control orientation, can only allow for so much creativity (and adaption). Open data is about putting the free flow of information at the heart of government – both internally and externally – with the goal of increasing government’s metabolism and decentralizing societies’ capacity to respond to problems. Our role is not obvious to the people in those movements, and we should make it clearer.

Internally to the movement, we have another big challenge. We are at a critical inflection point. For years we have been on the outside, yelling that open data matters. But now we are being invited inside. Some of us want to rush in, keen to make advances, others want to hold back, worried about being co-opted. To succeed, it is essential we must become more skilled at walking this difficult line: engaging with governments and helping them make the right decisions, while not being co-opted or sacrificing our principles. Choosing to not engage would, in my opinion, be to abscond from our responsibility as citizens and open data activists. This is a difficult transition, but it will be made easier if we at least acknowledge it, and support one another in it.

Our Core Challenges: What’s next

Looking across the open data space, my own feeling is that there are three core challenges that are facing the open data movement that threaten to compromise all the successes we’ve currently enjoyed.

1. The Compliance Trap

One key risk for open data is that all our work ends up being framed as a transparency initiative and thus making data available is reduced to being a compliance issue for government departments. If this is how our universe is framed I suspect in 5-10 years governments, eager to save money and cut some services, will choose to cut open data portals as a cost saving initiative.

Our goal is not to become a compliance issue. Our goal is to make governments understand that they are data management organizations and that they need to manage their data assets with the same rigour with which they manage physical assets like roads and bridges. We are as much about data governance as we are open data. This means we need to have a vision for government, one where data becomes a layer of the government architecture. Our goal is to make data platform one that not only citizens outside of government can build on, but one that government reconstructs its policy apparatus as well as its IT systems at top of. Achieving this will ensure that open data gets hardwired right into government and so cannot be easily shut down.

2. Data Schemas

This year, in the lead up to the Open Data Camp, the Open Knowledge Foundation created a map of open data portals from around the world. This was fun to look at, and I think should be the last time we do it.

We are getting to a point where the number of data portals is becoming less and less relevant. Getting more portals isn’t going to enable open data to scale more. What is going to allow us to scale is establishing common schemas for data sets that enable them to work across jurisdictions. The single most widely used open government data set is transit data, which because it has been standardized by the GTFS is available across hundreds of jurisdictions. This standardization has not only put the data into google maps (generating millions of uses everyday) but has also led to an explosion of transit apps around the world. Common standards will let us scale. We cannot forget this.

So let’s stop mapping open data portals, and start mapping datasets that adhere to common schemas. Given that open data is increasingly looked upon favourably by governments, creating these schemas is, I believe, now the central challenge to the open data movement.

3. Broadening the Movement

I’m impressed by the hundreds and hundreds of people here at the Open Data Camp in Warsaw. It is fun to be able to recognize so many of the faces here, the problem is that I can recognize too many of them. We need to grow this movement. There is a risk that we will become complacent, that we’ll enjoy the movement we’ve created and, more importantly, our roles within it. If that happens we are in trouble. Despite our successes we are far from reaching critical mass.

The simple question I have for us is: Where is the United Way, Google, Microsoft, the Salvation Army, Oxfam, and Greenpeace? We’ll know were are making progress when companies – large and small – as well as non-profits – start understanding how open government data can change their world for the better and so want to help us advance the cause.

Each of us needs to go out and start engaging these types of organizations and helping them see this new world and the potential it creates for them to make money or advance their own issues. The more we can embed ourselves into other’s networks, the more allies we will recruit and the stronger we will be.

 

Hackers are Social Activists mashup – Eric Raymond vs. Bill Moyes

Bullhorn5If the Open Web is a social movement then are Hackers Social Activists?

From the reaction of a number of people to that post the answer at first seems unclear. Some of those who responded shared their discomfort with the idea of being labeled a social activist. Notions of people chaining themselves to something, yelling into loudspeakers or even adopting the approach of archetype hacker activist Richard Stallman’s makes many open source supporters uneasy.

I agree – but being an activist doesn’t necessarily mean these things.

I’ve been struggling to explain why when, while re-reading the Cathedral and the Bazaar essay “Revenge of the Hackers” I was stuck by how Raymond’s description of the rise of Open Source parallels the stages of activism outlined in the “Movement Action Plan” – a strategic model for social movements developed by Bill Moyer, a US social change activist (basic outline at bottom of this post).

The parallels are not perfect, nor are they absolutely linear (but then Moyer concedes that they never are). Moreover, the language used by Bill Moyer would probably make any coder (or even non-coders like me) feel out of place or downright uncomfortable (does anyone in an open-source community talk powerholders?).  But mapping Eric Raymond against Bill Moyes is instructive. It highlights quite effectively that hackers are a movement, they are social activists (even when thy don’t consider themselves to be), and that when mobilized they can achieve big hairy audacious goals more effectively than many other groups.

I share this not because I want hackers to change who they are or what they do, but, like Eric Raymond, to help them gain a better awareness of who they are and what they are doing (accomplishing!). There are others who want to join this movement – and so understanding that there is a movement, and your role and place in it, is the first step in figuring out how to help others join in. If we want to invite, enable and foster the next million mozillians, we need to at least understand our place within the ecosystem of the open web social movement.

Title and Quotes from the 8 stage Movement Action Plan (see below) by Bill Moyers

Quotes from Eric S. Raymond’s “Revenge of the Hackers”

Stage 1: Business as UsualIn this first stage there are many conditions that grossly violate widely held, cherished human values such as freedom, democracy, security, and justice, and the best interests of society as a whole.

The opposition feels hopeless because it seems that the situation will continue indefinitely, and they feel powerless to change it.

vs.

Stage 1: A hackers life was not an easy one…I had been in the hacker culture, living through its various phases, for twenty years. Twenty years of repeatedly watching brilliant ideas, promising starts, and superior technologies crushed by slick marketing. Twenty years of watching hackers dream and sweat and build, too often only to watch the likes of the bad old IBM or the bad new Microsoft walk away with the real-world prizes. Twenty years of living in a ghetto — a fairly comfortable ghetto full of interesting friends, but still one walled in by a vast and intangible barrier of mainstream prejudice inscribed “ONLY FLAKES LIVE HERE”.
Stage 2: Normal Channels FailProve that the official doctrine and policies of powerholders and institutions violate society’s values and the public trust.

Positive results are not expected now

Except for the rare media coverage of opponents’ activities, the problem is still neither in the public spotlight nor on society’s agenda of contested issues.

vs.

Stage 2: Market fails to sustain open standardsNetscape had been targeted for destruction. Microsoft rightly feared that the open Web standards embodied by Netscape’s browser might lead to an erosion of its lucrative PC desktop monopoly. The weight of Microsoft’s billions, and shady tactics that would later trigger an antitrust lawsuit, were deployed to crush Netscape.For Netscape, the issue was less browser-related income (never more than a small fraction of their revenues) than maintaining a safe space for their much more valuable server business. If Internet Explorer achieved market dominance, Microsoft would be able to bend the Web’s protocols away from open standards and into proprietary channels that only Microsoft’s servers would be able to service.
Stage 3: Conditions RipenThe “take-off” of a new social movement requires preconditions that build up over many years. These conditions include broad historic developments, a growing discontented population of victims and allies, and a budding autonomous grassroots opposition, all of which encourage discontent with the present conditions, raise expectations that they can change, and provide the means to do it.The growing numbers of discontented local people across the country quietly start new autonomous local groups, which as a whole form a “new wave” of grassroots opposition, which is independent from the established Professional Opposition Organizations.

vs.

Stage 3: Condition Ripen
On new Groups and approaches:My first encounter with Linux came in late 1993. I had already been involved in the hacker culture for fifteen years and I thought I understood the hacker culture — and its limitations — pretty well.Encountering Linux came as a shock. I still carried in my head the unexamined assumption that hacker amateurs, gifted though they might be, could not possibly muster the resources or skill necessary to produce a usable multitasking operating system. The HURD developers, after all, had been evidently failing at this for a decade.

But where they had failed, Linus Torvalds and his community had succeeded. And they did not merely fulfill the minimum requirements of stability and functioning Unix interfaces. No. They blew right past that criterion with exuberance and flair, providing hundreds of megabytes of programs, documents, and other resources. Full suites of Internet tools, desktop-publishing software, graphics support, editors, games…you name it. The hacker tradition I had been observing for two decades seemed suddenly alive in a vibrant new way.

On a sense of awareness and identity:

…Around me was a community which had evolved the most effective software-development method ever and didn’t know it!. That is, an effective practice had evolved as a set of customs, transmitted by imitation and example, without the theory or language to explain why the practice worked.

Lacking that theory and that language hampered us in two ways. First: we couldn’t think systematically about how to improve our own methods. Second: we couldn’t explain or sell the method to anyone else… The Cathedral and the Bazaar was less a revelation of novelty than an opportunity to celebrate the new language and the consciousness that went with it. That standing ovations were not so much for my work as for the hacker culture itself — and rightly so.

Stage 4: Take Off!New social movements surprise and shock everyone when they burst into the public spotlight on the evening TV news and in newspaper headlines. Overnight, a previously unrecognized social problem becomes a social issue that everyone is talking about. It starts with a highly publicized, shocking incident, a “trigger event”…

vs.

Stage 4: Take Off!On netscape going Open-Source:

On January 22nd 1998 Netscape announced it would release the sources of the Netscape client line to the Internet… This was the event that commentators in the computer trade press would later call “the shot heard ’round the world’… For the first time in the history of the hacker culture, a Fortune 500 and darling of Wall Street had bet its future on the belief that our way was right.

Stage 5: Activist FailureThe general populace experiences dissonance, not knowing who or what to believe. While many agree with the movement’s challenges, they also fear siding with dissidents and losing the security of the powerholders and status quo. The alternatives are unclear to them. The general citizenry is about evenly divided, 50 percent to 50 percent, between the powerholders and the movement. Movement violence, rebelliousness, and seeming anti-Americanism turn people off and tend to frighten them into supporting the powerholders’ policies, police actions, and status quo.

Powerholders continue a hardline strategy, including escalating their policies to prove that they are in charge and that both the movement and public have no effect.

The overall goal is to help activists become empowered and move on to Stage Six by becoming strategists by using a framework such as MAP, forming political and personal support groups

vs.

Stage 5: Crossing the ChasmOn developing a grand strategy:

At the Mozilla launch I met with several key people in the Silicon Valley and national Linux community. While helping Netscape was clearly a short-term priority, everybody I spoke with already understood the need for a longer-term strategy.

On abandoning failed activist approaches:

It seemed clear that the term `free software’ had done our movement tremendous damage over the years. Part of this stemmed from the fact that the word `free’ has two different meanings in the English language, one suggesting a price of zero and one related to the idea of liberty. Richard Stallman, whose Free Software Foundation (FSF) has long championed the term, says “Think free speech, not free beer” but the ambiguity of the term nevertheless created serious problems — especially since most “free software” is also distributed free of charge.

Most of the damage, though, came from something worse — the strong association of the term `free software’ with hostility to intellectual property rights, communism, and other ideas hardly likely to endear it to an MIS manager.

Under the pressure of the Netscape release the FSF’s actual position didn’t matter. What mattered was that its evangelism (associating `free software’ with these negative stereotypes in the minds of the trade press and the corporate world) had backfired.

Stage 6: Win Majority of Public The majority stage is a long process of eroding the social, political, and economic supports that enable the powerholders to continue their policies. It is a slow process of social transformation that create a new social and political consensus, reversing those of normal times.

The movement’s chief goal, therefore, is to nurture, support, and empower grassroots activists and groups. Finally, activists also need to have a grand strategy for waging Stage Six majority movements to win positive social changes against the strong opposition of the powerholders.

vs.

Stage 6: Win Majority of Decision MakersThe following six months were a study in increasingly surreal contrasts. On the one hand, I was giving talks on open source to Fortune 100 corporate strategists and technology investors; for the first time in my life, I got to fly first class and saw the inside of a stretch limousine. On the other hand, I was doing guerrilla street theater with grass-roots hackers — as in the riotously funny Windows Refund Day demonstration of March 15 1999, when a band of Bay-area Linux users actually marched on the Microsoft offices in the glare of full media coverage, demanding refunds under the terms of the Microsoft End-User License for the unused Windows software that had been bundled with their machines.
Stage 7: SuccessStage Seven begins when the long process of building opposition reaches a new plateau in which the new social consensus turns the tide of power against the powerholders and begins an endgame process leading to the movement’s success. The Stage Seven process can take three forms: dramatic showdown, quiet showdown, or attrition.

vs.

Stage 7: Marketplace Supports Open Standards
In the early part of 1999 big independent software vendors began porting their business applications to Linux, following the lead set earlier by the major database vendors. In late July, the biggest of them all, Computer Associates, announced that it would be supporting Linux. And preliminary results from an August 1999 survey of 2000 IT managers revealed that 49% consider Linux an “important or essential” element of their enterprise computing strategies. Another survey by IDC described what it called “an amazing level of growth’’ since 1998, when the market research couldn’t find statistically significant use of Linux; 13% of the respondents now employ it in business operations.1999 also saw a wave of wildly successful Linux IPOs by Red Hat Linux, VA Linux Systems, and other Linux companies.
Stage 8: Moving On Stage 8: Moving On

On a separate note, I think it is fascinating the overlap between Bill Moyes Movement Action Plan – which is about activism – and Moore’s “Crossing the Chasm” – which is about technology in the market place. The two things are describing the same event (the adoption of something new) in different spaces (community/society vs. marketplace) and each highlights a point on the exact same place of an S-curve where the threat failure is greatest: the Chasm for Moore, and Activisim Failure for Moyers. Happily, I think Open-Source has crossed the threshold, but other significant challenges, challenges both Moore and Moyer have insights into, await.

Bill Moyes Movement Action Plan:

Movement Action Plan