David Herron's Blog

Java testing and "test first development"

My first posting drew a question "does Sun use 'test first development' practices". So, let's talk about that a bit.

I can only speak for the Java team as that's where I work. You have to remember that Java is a 10+ year old project with a lot of history. "History" can mean, as it does in this case, that procedures we've been following for 10+ years often interferes with adopting the latest-and-greatest of development methodologies. Maybe.

I'm going to remain at the high-level picture of what we do.

There's generally 4 areas where Java tests come from:

1. Conformance (JCK): The conformance tests (a.k.a. JCK) are derived from minute-level examination of the available spec's. The Java TCK (a.k.a. JCK) is used by any licensee (including Sun) to verify that their Java implementation meets the standard definition of what 'Java' is. The JCK is that definition, in a way.
2. Regression (DEV): The regression tests are developed by the Java developers as they fix bugs. Sometimes the regression test is derived from the sample code submitted in the bug report (if any), and other times it's a simple implementation the developer uses to verify the change. The regression suite is meant to find regressions, of course.
3. Unit (DEV): This is a relatively new process, and is the "test first" practice asked about. The developers are required to come up with some tests to exercise each feature they develop, and before they can commit the feature (that is, do a "putback") their feature has to pass those unit tests.
4. Functional (SQE): The SQE team concentrates on "functional testing", which is trying to look at Java as an application developer. We take the specifications for new features and develop tests which fall outside the above categories.

It's the Unit tests written by the developers that is the closest to the query if Sun does "Test First Development" during the development cycle?

What I'm most familiar with is the functional testing SQE does, as that's the team I've worked in for 7 years. Before joining the SQE team I'd spent 8+ years in different software development jobs, and it was the functional test suites I'd worked on here which gave the most "interesting" software development challenges I'd seen.

As a typical application developer the scope of what you're developing is pretty limited. You have some functionality to develop - e.g. I used to write email applications, and for example there's only a certain amount of functionality to an address book module. Your customer has limited flexibility, because you're giving them a user interface and they can only do what you allow them to do. The small number of options limits the combinatorial expansion of the number of things to test for, etc.

However the customer in our case is the Java developers in the field writing applications using Java. Those application developers have free reign to do whatever they want, only limited by their ability to understand the Java platform and writing software to use it. The immensely larger number of options our customers have mean for an immensely bigger combinatorial expansion on the number of things to test.

That means that when an SQE team member is writing tests for a feature - we have to understand the feature from many, many angles. Most of Java's features can be used in lots of ways, and are never used in isolation from other features.

This last point makes for an interesting distinction between unit testing and functional testing. Unit testing concentrates on individual pieces of functionality, and verifying that each piece works. But you could have a zillion pieces, all individually tested and validated. However the product could still have bad quality unless you verify the individual functions all work together. This is where functional testing comes in, to bring the functionality together and test it in a way that more closely resembles the real world use of Java than you would capture in a Unit test.

Comments

• Of the Java tests only the JCK is available, only for Java 5.0 and only with a read only licence. Are there any moves to open it up a bit more? In particular to help writing thorough sets of test cases for bug reporting.
  

  Posted by: tackline on July 14, 2005 at 09:02 AM

• I can't answer that question, I'm afraid. (if the tests will be more widely available)

  Speaking for myself, I'm really interested in finding a way where "we" (the customers of Java, and the developers of Java) could collaborate on test cases. In the analysis I've done so far, the functional testing (SQE) is going to be most effective when the tests are most like what a developer in the field would write in their applications. And, who best to provide that kind of information than the developers in the field? Just an idea for consideration.

  Posted by: robogeek on July 14, 2005 at 09:55 AM

This work is licensed under a Creative Commons License.