Useful quality metrics to publish?
"We" are having planning and discussion about how to handle Sun's Java implementation as open source. I've seen several articles and blog postings from the folk directly involved in the discussions, and it's all very interesting. What I'm most puzzling over is, what should the quality team do or publish etc in this environment?
In my past reading it was most enlightening to learn that one of the major advantages of an open source project is -- the public has more ability to gauge the quality of that project, than they can with the typical closed source project.
Now, in the book I was reading the distinction they were making is a typical closed source project where all you get is a binary, versus the typical open source where you compile it yourself or install prepackaged stuff from some repository off the net. Of course Java has always been in a middle area between those two extremes, making the source available, and more recently being more open with the source, accepting contributions, etc.
In the typical closed source project you only have the vendors reassurances that they've even tested the product, much less that the product works properly etc. With the typical open source project you have the ability, even if you don't use it, to examine the source code, the build procedures, even to do your own support for your own uses. And, of course, Java has always been in the middle area where you could examine the source, and more recently the JIUL allowed for internal redistribution of modified JDK builds.
In my case I'm leading the discussions in the quality team to plan what the quality team ought to do as part of the open source Java project. One issue is there are few examples, because in the majority of open source projects there is not a dedicated quality team, and instead either there's no formalized testing or else the developers follow a test driven development process and think they don't need formalized testing beyond their unit tests.
With the few examples available to us of an open source project with a dedicated testing team ... let me ask you, our public, your opinion. I am making zero commitment to do anything you say, but I do value your input and advice.
What information / metrics / reports / plans / tests / etc would you find helpful to see to help you gauge the quality of Sun's Java implementation?