Konstantin I. Boudnik's Blog: September 2005 Archive

Posted by cos on September 30, 2005 at 09:05 PM | Permalink | Comments (0)

Hi there!

Surprisingly, my last post was rated #1 by Google for 'java quality' search and lasted in this position for a few days. My friends were wondering how much I had paid to gain this honorable position. Honestly: I didn't pay a penny for it and I only have to thank those of you, who spent time reading it. So, thank you! I also hope not to disappoint you this time.

Moving closer to the promises given in this blog, I will discuss some of not totally innovative, but still interesting techniques of improving quality development effectiveness.

Firstly, I'd like to talk about code coverage methodology (sometimes referred as a test coverage). It is a very efficient quality indicator, but relatively simple to collect and analyze. I will not spend much time describing this technique here, considering the variety of good information sources. If you're willing to educate yourself about the topic, you might want to check a paper like this

One of the common misunderstandings of this interesting technology is as follows: you shouldn't consider coverage numbers as the final point of your quality development, but rather the starting one.

I mentioned it once and I would like to emphasize it again: code coverage merely shows an amount of tested code. However, it doesn't address quality of the test coverage nor its effectiveness; it doesn't guarantee that the most important pieces of the source code are covered. But nevertheless, code coverage is a valuable measure, and I'm not in a position to 'misunderestimate (C)' it.

Coverage data gathering in a pure Java environment is quite a straight forward task and there's a variety of tools for this job (sometime I'll talk about mixed one, i.e. where Java code coexists with native). So far, you might want to: check this

However, you still have to take care about your build's instrumentation and about a storage for all produced information. And that one might be a very significant chunk of data sometimes. Ideally, you might want to use a database server, which seems to do the job right.

Processing of collected data and visualizing of results might be not that easy at times and would require surrounding frameworks and/or infrastructure.

once in a month, release engineering prepares JDK's instrumented build for coverage testing (it might not be required in your particular case, but, of course, I don't know your situation)
components' quality engineering teams run their test sets and gather results, which can be collected in a central storage during the execution stage or right after it

collected results might later visualized on-demand through Web-interface or another device. It is really helpful - even for manual code inspection - to present a product's source code in different colors, e.g. red for non-covered areas of code; green otherwise. Having executions counter associated with test(s) per basic block of code might be very handy as well. Also, you might consider a possibility to show a version control's data, i.e. check-in information, pointing to the latest code update or something. I hope you already got the idea. However, let me illustrate my point with that tiny snapshot (A friend of mine Roman Shaposhnick has created this tool a while ago for Sun's compilers work. After some minor modifications it was applied to Java quality as well):

ExplorerPanel.java

org/openide/explorer/ExplorerPanel.java code coverage

1.2 psk	1:		private final class PropL extends Object implements PropertyChangeListener {
	2:	12728 ->	PropL() {}
1.1 ma	3:
1.1 ma	4:		public void propertyChange(PropertyChangeEvent evt) {
	5:	12728 ->	if(evt.getSource() != manager) {
	6:		return;
	7:		}
1.3 rvs	8:
	9:	12728 ->	if (ExplorerManager.PROP_EXPLORED_CONTEXT.equals(evt.getPropertyName())) {
	10:	12728 ->	updateTitle ();
1.1 ma	11:		return;
1.1 ma	12:		}
1.1 ma	13:		}
1.1 ma	14:		}

coverage data is getting used for managerial reports, exit criteria preparation, et cetera coverage statistic are used in a process of quality metrics preparation. Metric subsystems are working with coverage storage facility through some kind of API, i.e. JDBC

And please keep in mind: most of the quantative metrics of quality development are about trends in the first place. So, instead of doing code coverage once in a release time, you might find it useful to collect data on a regular basis and build a tendency of the coverage's improvements or otherwise:-)

Sorta disclaimer, though: my opinion might appear as arguable. Well, leave me a comment if you disagree with it and I'll try to address your concern in my future posts. See you soon!

Java. Quality. Metrics

Posted by cos on September 23, 2005 at 09:16 PM | Permalink | Comments (3)

It isn't, perhaps, a secret, that software test development and quality are like a snow ball rolling down a hill: and as it reaches further towards the end of the slope, harder to stop... and think. Think about what is done right; what missed, and how I can do this better, if only I had another chance.

We all have heard (or know from a real experience) about a number of testing types: functional, unit, white box and black box, and so on, and so on. But what really drives most of the test development efforts? Ok, ok, I know - everyone wants to find this ugly bug sitting next to the last one :-) But how we can know if that bug is uglier than others? What is the criteria for this? What tools have to be brought into the process? What additional testing techniques we need to introduce? And last, but not least - where the efforts of test development engineers have to be directed to reduce the cost of quality e.g. more bugs found at the earlier stages of the development?

Now, multiply most of above by a number of platforms, your product is running on and you will start seeing the picture similar to what we see ourselves in the Java Standard Edition Quality Organization.

In the following series of posts I plan to talk about these and some other issues of the software testing and quality measurements. I will share our practices of static analysis (do not confuse it with FindBugs application - static analysis is somehow different from this technique); what stages of test development and test execution we're having in JavaSE production cycle; what tools and techniques we use to increase ROI and free our engineers to do something more sophisticated and exciting than merely manual test execution.

I have to admit, that I really like Test Driven Development (TDD). It sounds so cool to develop all your tests first and then simply make your software to pass these tests. I wish that all problems of the software quality would be that sound and being solved that clearly. Unfortunately, it isn't so. And there is a lot of issues, which couldn't be foreseeing and test framed in advance. And as more complex a system becomes the less efficient this approach will be. As time and development goes, a quality department has to introduce more complicated methodologies and trickier techniques.

To be more specific, I'd like to quickly illustrate my point here: only in Java Virtual Machine (VM) testing we are running more than a million test cases in a few different testing cycles, e.g. nightly, pre-integration, weekly, regression, et cetera. We do separate stress testing and BigApps (or real world applications) testing. We support about two dozen (sic!) platforms. Did I mention, that we walk dogs too?

Now, let me move to the point and talk about real things. No wonder, that to control such a crowd you have to be really creative. And we do. We use some distributed execution environments. One of them is home grown Distributed Tasks Framework (DTF; patents pending) and is 100% pure Java application based on Jini. Another one is Grid Engine from Sun Microsystems. Both of these applications are quite similar in their functionality. However, Grid Engine is officially supported product and I gave up on further DTF maintenance. Now we use it just for scheduling and executing tasks on Wintel platform

Another thing, I'd like to mention here, is multi platform test harness Tonga (patents pending) which supports distributed testing scenarios and can do a lot of other things. I hope to see this great product in open source some day. You can read more about test frameworks and test harnesses at my colleague David Herron's blog

And it is obviously important to have an efficient solution of test results analysis. Ideal system would have low level of false positives; automated detection of regressions and known bugs; run-to-run comparison to find any trends; and many others qualities. Of course, the system like this relies on bugs (or issues) tracking system of some kind. For generic approach you can pick some of well-known systems like Bugzilla. However, larger companies are often go with their own products. Some day I will talk more about result analysis applications we're using in our process.

Measurements of tests effectiveness becomes a high priority issue at the certain point of a product development. And test coverage is one of reasonable ways to measure it. Indeed, it is cool to see that your test coverage had increased by 12% since last release. It makes you proud and confident in the quality of your product. However, it might be not enough to rely on such metric only. E.g. a product's last release includes 27 new features. Overall size of source code introduced is 270K lines. At the same time, the test coverage increased 12% and became 68% overall. Hmm, I wonder if this is good or bad? Or how exactly good is it? Shall we celebrate over such an achievement or do something about it? So, test coverage isn't the single one and shouldn't be treated like the only panacea of software quality measurements. The situation just get's more complex when a product consists of native and Java source code.

On this optimistic note I will close my today's post, so that something else will be left for further discussion.

Sorta disclaimer, though: this is my very first blogging experience ever and so bear with my over enthusiasm and leave me a comment or suggestions of topics you would like to hear or dicuss further.. See you soon!

Powered by
Movable Type 3.01D

java.net RSS Feeds