It would be interesting to rerun the graph about bug efficiency in the post when duplicates are filtered out. The SE research community has been doing increasing work in the area of easing bug reporting and triage, more triage than reporting. For example, automatically directing bugs to individuals with expertise in the area of the bug. Duplicate detectors are increasing in efficiency with the aim to auto-link together reports since reports often provide different perspectives on a problem and a vote only on a problem provides data to the eventual bug fixer.

It would be great to gather data about when bugs are not filed but someone wanted to. Perhaps even simple checkboxes to let someone indicate who got to a Bugzilla that they abandoned the entry might get some data. It might be interesting to show recent bugs filed in a news box on the top-level of a project and make it easier for users to vote.

Gail Murphy
UBC

I made six requests for enhancements to Thunderbird back in 2002. None of them have been implemented, but I expect that some of them *will* be implemented someday (based on discussions with some of the Thunderbird team).

I don't think these issues have much to do with the source being open or closed. It's more about the ability and willingness of the software owners (which could be a large community) to diagnose and fix problems.

One way Microsoft addressed this around 1999 was to instrument beta versions of Office and Windows to automatically generate anonymized bug report data and make it available for uploading along with the end user's text comments. The product team ended up with a full trace of user activity leading up to the crash, and the end user only had to click a checkbox to permit sending the info. No more baffling questions — “What's your OS version? Oh, let me explain what that means. Now what type of hardware are you running? Oh, let me explain….”

This tactic coupled with improving internet connectivity (dial-up modems –> DSL) made it dramatically easier to report bugs. Bug report rates and fix rates jumped upwards significantly for those versions of Windows and Office. (I don't recall the numbers, but it was a huge shift.)

Nowadays, of course, it's commonplace to do this. And I bet if you looked at Firefox bug reports you'd see a significantly higher fix rate for those with app crash data attached.

In the world of pothole reporting, the analogy is uploading a geotagged photo rather than trying to explain the problem in words. The photo wins: very low cost, very high utility for diagnosis.

So provide tools that make reporting and diagnosis cheap.

As for the really experienced bug reporters, you may be seeing people uncovering bugs that are extremely hard to diagnose reliably (e.g. Heisenbugs), or issues that are really architectural or design flaws the product owners don't want to fix for whatever reason. Look at Ruby on Rails or Linux bug reports and you'll see pages of discussion between developers arguing about whether an issue is a “bug” or “by design”. That's a huge cost, but I don't know how you make it go away. I'd rather have smart people poking holes at my product than ignoring me.

It would be interesting to see the red line plotted as individual data points. Showing it as a continuous line is confusing.

Thanks for writing this. Thought-provoking!

I wonder if I'm in the oval; it feels like the road crews are concentrated downtown (since that's what they get paid to do), but I keep reporting bugs about Marine Drive because that's where my commute is. Unfortunately people who decide what to fix never wander down this way, and I can't figure out where to submit my plans for fixing the problem.

(This isn't actually true of the real Marine Drive, of course – for that you'd need to look for a farm access road.)

Let's compare a pothole on Broadway and 7th to a software bug. Most software bug reports are similar to “I hit a bump on the way home”. The developer can't reproduce the problem and other users can't figure out if they have the same bug. Sometimes we get a bit more information “I hit a hole in the road on the way home”. Not a pothole that we know how to fix, but a 'hole' which could be anything from a pothole to a cliff. Even when users do communicate in the same vocabulary, we still need a test case, the equivalent of “Broadway and 7th”. Often this is very difficult to create.

If these problems are not enough, software shares the problem of expectations and responsibility with the world of potholes: if I hit a bump on the way home, is it 'normal'? Why should I report it rather than any of the other drivers on the road? Doesn't the road crew just drive around and fix these? I guess potholes and software bugs are reported to friends and neighbors often and we get a sense of how to proceed from the response. “yea, I hit too” or “well pay attention dummy” or “We should call the highway department”. A community can have a lot of impact on the probability of a next step, but only if the repair report is easy and effective.

Let's go back to your study. I bet if you looked at FIXED bugs, a very high percentage would have test cases; conversely if you look at bugs that don't get fixed, few have test cases. So if true, fixing the bugzilla UI would just increase the number of bugs that don't get fixed.

And now I shift to report to you on a bug on your site: this box is too small to re-read my comment ;-)

And it's also CERTAINLY the case that the Bugzilla is not suited for mere mortals. Then again, the most likely outcome of a newbie submitting a bug is that it's a dupe.

I'm not sure that the last sentence of your first paragraph stands though. I do think that, to a certain degree bugzilla does function as a to do list for some at Mozilla, but the employees may be the people at the upper variance (80-90%) complete.

I think what this chart in part tells us is that it may be hard to ascertain if a bug is actually new (e.g. search results don't turn up effective results, its hard to do, or it is hard to learn to do) and that as a result there are many, many, many duplicate submissions.