My friend Diederik and I are starting to work more closely with some open source projects about how to help “open” communities (be they software projects or cities) become more efficient.
One of the claims of open source is that many eyes make all bugs shallow. However, this claim is only relevant if there is a mechanism for registering and tackling the bugs. If a thousand people point out a problem, one may find that one is overwhelmed with problems – some of which may be critical, some of which are duplicates and some of which are not problems at all, but mistakes, misunderstandings or feature requests. Indeed, in recent conversations with open source community leaders, one of the biggest challenges and time sinks in a project is sorting through bugs and identifying those that are both legitimate and “new.” Cities, particularly those with 311 systems that act similar to “bug tracking” software in open source projects, have a similar challenge. They essentially have to ensure that each new complaint is both legitimate, and geuninely “new” (and not a duplicate complaint – eg. are there 2 potholes at Broadway and 8th vs. two people have called in to complain about the same pothole).
The other month Diederik published the graph below that used bug submission data for Mozilla Firefox tracked in Bugzilla to demonstrate how, over time, bug submitters on average do become more efficient (blue line). However, what is interesting is that despite the improved average quality the variability in the efficacy of individual bug submitters remained high (red line). The graph makes it appear as though the variability increases as submitters become more experienced but this is not the case, towards the left there were simply many more bug submitters and they averaged each other out creating the illusion of less variability. As you move to the right the number of bug submitters with these levels of experience are quite few, sometimes only 1-2 per data point, so the variability simply becomes more apparent.
Consequently, the group encircled by purple oval are very experienced and yet continue to submit bugs the community ultimately chooses to either ignore or deems not worth fixing. Sorting through, testing and evaluating these bugs suck up precious time and resource.
We are presently looking at more data to assess if we can come up with a profile for what makes for a bug submitter who falls into this group (as opposed to be “average” or exceedingly effective). If one could screen for such bug submitters, then a community might be able to better educate them and/or provide more effective tools and thus improve their performance. In more radical cases – if the net cost of their participation was too great – one could even screen them out of the bug submission process. If one could improve the performance of this purple oval group by even 25% there would be a significant improvement in the average (blue line). We are looking forward to talk and share more about this in the near future.
As a secondary point, I feel it is important to note that we are still in the early days of open source development model. My sense is there are still improvements – largely through more effective community management – that can yield dramatic (as opposed to incremental) boosts in productivity for open source projects. This separates them again from proprietary models which – as far as I can tell – can at the moment at best hope for incremental improvements in productivity. Thus, for those evaluating the costs of open versus closed processes, it might be worth considering the fact that the two approaches may be (and, in my estimation, are) evolving at very different rates.
(If someone from a city government is reading this and you have data regarding 311 reports – we would be interested in analyzing your data to see if similar results bear out – plus it may enable us to help you manage you call volume more effectively.)
eaves.ca 
I am very surprised by this graph. I would have guessed based on personal experience and discussions with others that the probability of getting a bug fixed if you report it to mozilla was less than 20%. I think the graph basically says: “The mozilla project uses bugzilla as a todo list and most mozilla project members get about half of their todos done.” That is, these are not real bugs.In my opinion you are looking at this problem from the wrong side. The important question is how many bug reports are not filed because the process is too clumsy and the remote chance of a fix too discouraging. I contend that the number of citizens reporting a pothole is a direct measurement of the effectiveness of government: many people believe the government listens and responds. Zero reports is collapse.
It might be interesting to know how 9-11 deals with multiple reports of the same incident. I remember calling in and being told that someone had already reported the same accident, and that was maybe 1 minute from the collision time.
Pingback: Tweets that mention Making Open Source Communities (and Open Cities) More Efficient | eaves.ca -- Topsy.com
Pingback: uberVU - social comments
John – I really enjoyed your comment – particularly the last line. It is definitely agree that fewer bug reports could be a strong indicator of a lack of confidence in the system/software/project and that that is a valuable way to look at the problem. Diederik and I are working on some methods to try to ascertain how many people don't submit bugs because of this – but it is hard to do, as there is (by definition) no base data or information to start with (if you don't submit a bug, there is definitely no bug submission we can query). Indeed, if you had thoughts about how we could measure or infer this, I would be very interested in chatting.I'm not sure that the last sentence of your first paragraph stands though. I do think that, to a certain degree bugzilla does function as a to do list for some at Mozilla, but the employees may be the people at the upper variance (80-90%) complete. I think what this chart in part tells us is that it may be hard to ascertain if a bug is actually new (e.g. search results don't turn up effective results, its hard to do, or it is hard to learn to do) and that as a result there are many, many, many duplicate submissions.
It's true that the way most projects use bug trackers, not every “bug” is a bug. The term “issue tracker” is usually more accurate. A bug can be a feature request, a documentation typo, or an error that only happens on a certain page. I think calling it a “todo list” is oversimplifying, but it's certainly the case the bugs vary widely in relevance and work involved.And it's also CERTAINLY the case that the Bugzilla is not suited for mere mortals. Then again, the most likely outcome of a newbie submitting a bug is that it's a dupe.
Of course we could poll users to get an understanding of why they don't report bugs. But I think we know the answer: the cost of reporting exceeds the benefit by a wide margin. Unfortunately fixing the bugzilla UI is not enough. Let's compare a pothole on Broadway and 7th to a software bug. Most software bug reports are similar to “I hit a bump on the way home”. The developer can't reproduce the problem and other users can't figure out if they have the same bug. Sometimes we get a bit more information “I hit a hole in the road on the way home”. Not a pothole that we know how to fix, but a 'hole' which could be anything from a pothole to a cliff. Even when users do communicate in the same vocabulary, we still need a test case, the equivalent of “Broadway and 7th”. Often this is very difficult to create. If these problems are not enough, software shares the problem of expectations and responsibility with the world of potholes: if I hit a bump on the way home, is it 'normal'? Why should I report it rather than any of the other drivers on the road? Doesn't the road crew just drive around and fix these? I guess potholes and software bugs are reported to friends and neighbors often and we get a sense of how to proceed from the response. “yea, I hit too” or “well pay attention dummy” or “We should call the highway department”. A community can have a lot of impact on the probability of a next step, but only if the repair report is easy and effective.Let's go back to your study. I bet if you looked at FIXED bugs, a very high percentage would have test cases; conversely if you look at bugs that don't get fixed, few have test cases. So if true, fixing the bugzilla UI would just increase the number of bugs that don't get fixed.And now I shift to report to you on a bug on your site: this box is too small to re-read my comment ;-)
Pingback: Tweets that mention Making Open Source Communities (and Open Cities) More Efficient | eaves.ca -- Topsy.com
To stretch your analogy way too far…I wonder if I'm in the oval; it feels like the road crews are concentrated downtown (since that's what they get paid to do), but I keep reporting bugs about Marine Drive because that's where my commute is. Unfortunately people who decide what to fix never wander down this way, and I can't figure out where to submit my plans for fixing the problem.(This isn't actually true of the real Marine Drive, of course – for that you'd need to look for a farm access road.)
John is spot on about lack of reproduceable test cases leading to useless bug reports, and the cost/benefit issue being a barrier to reporting bugs in the first place. I don't think these issues have much to do with the source being open or closed. It's more about the ability and willingness of the software owners (which could be a large community) to diagnose and fix problems. One way Microsoft addressed this around 1999 was to instrument beta versions of Office and Windows to automatically generate anonymized bug report data and make it available for uploading along with the end user's text comments. The product team ended up with a full trace of user activity leading up to the crash, and the end user only had to click a checkbox to permit sending the info. No more baffling questions — “What's your OS version? Oh, let me explain what that means. Now what type of hardware are you running? Oh, let me explain….” This tactic coupled with improving internet connectivity (dial-up modems –> DSL) made it dramatically easier to report bugs. Bug report rates and fix rates jumped upwards significantly for those versions of Windows and Office. (I don't recall the numbers, but it was a huge shift.) Nowadays, of course, it's commonplace to do this. And I bet if you looked at Firefox bug reports you'd see a significantly higher fix rate for those with app crash data attached.In the world of pothole reporting, the analogy is uploading a geotagged photo rather than trying to explain the problem in words. The photo wins: very low cost, very high utility for diagnosis.So provide tools that make reporting and diagnosis cheap.As for the really experienced bug reporters, you may be seeing people uncovering bugs that are extremely hard to diagnose reliably (e.g. Heisenbugs), or issues that are really architectural or design flaws the product owners don't want to fix for whatever reason. Look at Ruby on Rails or Linux bug reports and you'll see pages of discussion between developers arguing about whether an issue is a “bug” or “by design”. That's a huge cost, but I don't know how you make it go away. I'd rather have smart people poking holes at my product than ignoring me.It would be interesting to see the red line plotted as individual data points. Showing it as a continuous line is confusing.Thanks for writing this. Thought-provoking!
Pingback: Links 24/11/2009: KDE Icon on TV, Ubuntu Netbook Remix Reviewed | Boycott Novell
I echo Oshama's thought that unfixed bugs were not necessarily a *problem*. I consider myself an experienced bug reporter, and yet there are lots of bugs that I have filed that probably won't get fixed. Why? Because I have this habit of occasionally submitting feature requests, and sometimes the feature requests are so uh advanced that it just doesn't make sense to add that feature (at least now). However, if you add it, that maybe gets people thinking about it, and how they maybe could implement it.I made six requests for enhancements to Thunderbird back in 2002. None of them have been implemented, but I expect that some of them *will* be implemented someday (based on discussions with some of the Thunderbird team).
Interesting discussion. It would be interesting to rerun the graph about bug efficiency in the post when duplicates are filtered out. The SE research community has been doing increasing work in the area of easing bug reporting and triage, more triage than reporting. For example, automatically directing bugs to individuals with expertise in the area of the bug. Duplicate detectors are increasing in efficiency with the aim to auto-link together reports since reports often provide different perspectives on a problem and a vote only on a problem provides data to the eventual bug fixer.It would be great to gather data about when bugs are not filed but someone wanted to. Perhaps even simple checkboxes to let someone indicate who got to a Bugzilla that they abandoned the entry might get some data. It might be interesting to show recent bugs filed in a news box on the top-level of a project and make it easier for users to vote. Gail MurphyUBC
Interesting discussion. It would be interesting to rerun the graph about bug efficiency in the post when duplicates are filtered out. The SE research community has been doing increasing work in the area of easing bug reporting and triage, more triage than reporting. For example, automatically directing bugs to individuals with expertise in the area of the bug. Duplicate detectors are increasing in efficiency with the aim to auto-link together reports since reports often provide different perspectives on a problem and a vote only on a problem provides data to the eventual bug fixer.It would be great to gather data about when bugs are not filed but someone wanted to. Perhaps even simple checkboxes to let someone indicate who got to a Bugzilla that they abandoned the entry might get some data. It might be interesting to show recent bugs filed in a news box on the top-level of a project and make it easier for users to vote. Gail MurphyUBC
Pingback: Your eyes are like limpid pools… a LOT of limpid pools. | Noise to Signal