Developer scenario testing
Today I want to talk about something I've been thinking about a lot lately. I work in the Java SE Quality Engineering team. We do a lot of the testing that goes into ensuring the quality of Sun's Java. We see the new features in Java before anybody else does. As one of my colleagues says "we are the first customers of Java". Our job is to create "functional" test suites for new features in Java, the functional test suites being a different style of testing than either the unit tests or conformance tests others in the Java SE team do. I've covered some of this in previous blog postings.
The thing I've been thinking about is the testing that we cannot do. That is, we cannot test your applications in your environment.
We try and put ourselves in your shoes, to think about what you might do with the features, and write tests from that viewpoint. But, try as we might, we are not you, we do not have your environment, we do not have your applications, we do not have your mindset, and we can only go so far.
What's most important is your experience with Java when you develop and run applications. Even more important is that your applications continue to run well even when we make a major Java release. e.g. the Mustang project is wending towards its finish date (and, no, we have not disclosed the Beta/RC/FCS dates as yet), and it will be really great if your applications were to execute just as well or better on 1.6 as they do on 1.5.
As I said, we do the best we can. The team as a whole tests Java in several different models (conformance testing, unit testing, performance testing, stress testing and functional testing). But, as I said, we are not you and we cannot have covered all the functional combinations that you have implemented in your applications. There is an effectively infinite number of ways to combine Java API calls and features, so it is clear that you will have come up with ways to use Java that we cannot have dreamed of.
This is where you come in. You can test your application in your environment on the Mustang early access releases today. We really want to know about problems you see with your application on Mustang. The earlier we know about those problems the greater a chance there is we can fix the problems before Mustang is finalized.
We've been providing the weekly snapshot builds on mustang.dev.java.net for several months now. It's easy to download those builds and try them out.
So, "why". Why do this testing for Sun? Let me suggest that you do it for yourself. You will eventually want to run your application on 1.6 anyway, so why not try it out today and see how well it works?