Skip to main content

Performance Testing is not Scalability Testing

Posted by johnsmart on June 26, 2008 at 7:28 AM PDT

People often confuse performance and scalability testing, but they are actually quite different activities. Performance testing involves ensuring that your application responds to requests within an acceptable timeframe. Of course, defining what "acceptable" is is a fine art, and perceived performance (what the user actually sees) is often more important than real performance.

Performance testing may aim at testing the overall performance of the system as a whole. Or it can be a safeguard activity, designed to catch sub-optimal queries before they reach production, for example. In both cases, well-designed integration tests, that simulate typical user interactions, can serve as the basis of these performance tests. For simple uses, simple timers, or the JUnit 4.4 @Timeout annotation, can suffice. However, the integration tests should be designed well enough to accept variable data, to avoid database caching and other such issues. Tests using JUnit 4.4 Theories can be a good candidate for this sort of reuse.

Load testing, on the other hand, is about concurrent access - how does your application repond when lots of users start to use the system at the same time. Does it still respond to each user within a reasonable time? Does it degrade gracefully or come to a grinding halt? Load testing is harder and more technical than pure performance testing, as a lot more variables are involved.

Tools like JMeter, the Grinder and TestMaker are quite capable of executing JUnit tests in parallel, to simulate load. Now, if these tests are integration tests that reproduce a typical user interaction, you can obtain a quite reasonable set of load tests, at least for the Java layers of your application. These tests tend to be more stable than web-based tests, and can serve both as performance and regression tests.

Tools like Selenium can test your application from the web layer down. This is great for testing the whole application, including the web pages, and for regression testing. You can even embed Selenium tests in a JUnit test case and use them for load testing as well. This is fine, but the user interface of a web application is typically less stable than the service and DAO layers, and so the test scripts will probably require more maintenance.

Just my 2¢.


John, you talked about Performance testing and Load Testing, not about Scalability. I would recommend you to talk about Scalability or change the name of the entry. Load testing isn't Scalability. Remember that we can talk about data growth, not just CPU or network load.