My friend Diederik and I are starting to work more closely with some open source projects about how to help "open" communities (be they software projects or cities) become more efficient.
One of the claims of open source is that many eyes make all bugs shallow. However, this claim is only relevant if there is a mechanism for registering and tackling the bugs. If a thousand people point out a problem, one may find that one is overwhelmed with problems - some of which may be critical, some of which are duplicates and some of which are not problems at all, but mistakes, misunderstandings or feature requests. Indeed, in recent conversations with open source community leaders, one of the biggest challenges and time sinks in a project is sorting through bugs and identifying those that are both legitimate and "new." Cities, particularly those with 311 systems that act similar to "bug tracking" software in open source projects, have a similar challenge. They essentially have to ensure that each new complaint is both legitimate, and geuninely "new" (and not a duplicate complaint - eg. are there 2 potholes at Broadway and 8th vs. two people have called in to complain about the same pothole).
The other month Diederik published the graph below that used bug submission data for Mozilla Firefox tracked in Bugzilla to demonstrate how, over time, bug submitters on average do become more efficient (blue line). However, what is interesting is that despite the improved average quality the variability in the efficacy of individual bug submitters remained high (red line). The graph makes it appear as though the variability increases as submitters become more experienced but this is not the case, towards the left there were simply many more bug submitters and they averaged each other out creating the illusion of less variability. As you move to the right the number of bug submitters with these levels of experience are quite few, sometimes only 1-2 per data point, so the variability simply becomes more apparent.
Consequently, the group encircled by purple oval are very experienced and yet continue to submit bugs the community ultimately chooses to either ignore or deems not worth fixing. Sorting through, testing and evaluating these bugs suck up precious time and resource.
We are presently looking at more data to assess if we can come up with a profile for what makes for a bug submitter who falls into this group (as opposed to be "average" or exceedingly effective). If one could screen for such bug submitters, then a community might be able to better educate them and/or provide more effective tools and thus improve their performance. In more radical cases - if the net cost of their participation was too great - one could even screen them out of the bug submission process. If one could improve the performance of this purple oval group by even 25% there would be a significant improvement in the average (blue line). We are looking forward to talk and share more about this in the near future.
As a secondary point, I feel it is important to note that we are still in the early days of open source development model. My sense is there are still improvements - largely through more effective community management - that can yield dramatic (as opposed to incremental) boosts in productivity for open source projects. This separates them again from proprietary models which - as far as I can tell - can at the moment at best hope for incremental improvements in productivity. Thus, for those evaluating the costs of open versus closed processes, it might be worth considering the fact that the two approaches may be (and, in my estimation, are) evolving at very different rates.
(If someone from a city government is reading this and you have data regarding 311 reports - we would be interested in analyzing your data to see if similar results bear out - plus it may enable us to help you manage you call volume more effectively.)
