Skip to main content

Switching to TestNG

Posted by fabriziogiudici on June 17, 2011 at 9:09 AM PDT

I don't recall when I was introduced to TestNG for the first time - for sure, it's a matter of a few years ago. I was quickly convinced that TestNG is a superior tool than JUnit for a number of reasons (which I'm not going to full enumerate: among others, it's easier to write parameterized tests with TestNG and there's embedded support for parallel testing; for a comprehensive comparison, just Google around). Sure, as time passed by JUnit caught up with the initial gap, but in the end TestNG is still richer and offers many more ways to be customized.

But in 2009 I was still working with JUnit and writing some extensions. Actually, I was still writing my tests with JUnit one week ago. Why? Because my builds were Ant-based and frankly I was reluctant to patch them. I had a mixture of JSE, JME, JEE and NetBeans Platform projects, each with its own Ant build scripts. At that time they were already growing in complexity (e.g. because of the ongoing integration of static analysis tools) and in the end you can survive with JUnit. Furthermore, I was unsure about the NetBeans support (there was and is a plugin, that often doesn't catch up quickly with IDE updates). But it's sad to decide to "survive" with a tool and give up with a better one.

After moving to Maven, everything is easier. In fact, Surefire (the Maven plugin for tests) can work with both JUnit and TestNG. It has been only a matter of time (more urgent things were in the pipeline), but a few days ago I started converting my projects from JUnit to TestNG (for instance, blueBill Mobile has been entirely ported to TestNG).

It's just a matter of replacing JUnit with TestNG in POM references and performing the following substitutions in the code:

  • import org.junit.After with import org.testng.annotations.AfterMethod;
  • import org.junit.Before with import org.testng.annotations.BeforeMethod;
  • import org.junit.AfterClass with import org.testng.annotations.AfterClass;
  • import org.junit.BeforeClass with import org.testng.annotations.BeforeClass;
  • import org.junit.Test with import org.testng.annotations.Test;
  • import static org.junit.Assert.* with import static org.hamcrest.MatcherAssert.*;
  • @Before with @BeforeMethod;
  • @After with @AfterMethod;
  • @Test(timeout with @Test(timeOut
  • the single-value annotation parameter expected with expectedException={...}
  • eventually you'll need to import static org.testng.Assert.fail;

This, of course, for plain tests. If you have used some customization, such as parameterization, you'll need more work. But in most case it's pretty simple. NetBean 7.0 works with TestNG by means of its integration with Surefire, so not a problem (I must say, though, that today I let my Hudson run most of my tests, or I run tests with the command line).

I can give you an example on how I could easily write a first customization to TestNG. In the past years, I've trained myself in efficiently scanning listings and log files, but I manage in doing this task really quickly only when the code (or the log file) is organized as I expect. In particular, I need visual cues for delimitating scopes (e.g. comments for delimitating methods, or breaking lines for log files). Sure, you can explicitly log the test name calling the logger at the beginning of each method (as I did until a few days ago), but it's boring. Furthermore, SureFire doesn't write in the log that a test has failed, reporting instead all the failures at the end of the suite. In the end, I want to read in my Hudson something like:

09:14:38.524 [main                ] INFO  it.tidalwave.util.test.TestLogger                  -
09:14:38.525 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - ******************************************************************************************
09:14:38.525 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - TEST "showNewsFeed must properly populate the view when the cached news feed is available"
09:14:38.525 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - ******************************************************************************************
09:14:38.572 [TestNGInvoker-showNe] INFO  i.t.b.mobile.news.spi.DefaultNewsViewController    - showNewsFeed()
09:14:38.579 [TestNGInvoker-showNe] INFO  i.t.b.m.news.spi.DefaultNewsViewControllerTest     - >>>> mock NewsService answering notifyCachedFeedAvailable(RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}])
09:14:38.579 [TestNGInvoker-showNe] DEBUG i.t.b.mobile.news.spi.DefaultNewsViewController    - notifyCachedFeedAvailable()
09:14:38.579 [TestNGInvoker-showNe] DEBUG i.tidalwave.bluebill.mobile.role.util.RoleRegister - getLookup()
09:14:38.580 [TestNGInvoker-showNe] TRACE i.t.n.capabilitiesprovider.ThreadLookupBinder      - Bound ProxyLookup(class=class org.openide.util.lookup.ProxyLookup)->[org.openide.util.lookup.SingletonLookup@13647278, org.openide.util.Lookup$Empty@4ce66f56] to thread Thread[TestNGInvoker-showNe,5,main]
09:14:38.580 [TestNGInvoker-showNe] DEBUG i.t.b.mobile.news.spi.DefaultNewsViewController    - populateAndUnlockView(RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}])
09:14:38.586 [TestNGInvoker-showNe] DEBUG i.t.n.capabilitiesprovider.ThreadLookupBinder      - createLookup(ProxyLookup(class=class it.tidalwave.netbeans.util.test.MockLookup)->[org.openide.util.lookup.SingletonLookup@6477eb97, it.tidalwave.netbeans.util.test.MockLookup$1@2c979e8b, org.openide.util.lookup.SingletonLookup@7d0c3a08], RssFeed@7a22ce00[id - {}], ...)
09:14:38.586 [TestNGInvoker-showNe] TRACE i.t.n.capabilitiesprovider.ThreadLookupBinder      - >>>> lookups for RssFeed@7a22ce00[id - {}]: [org.openide.util.Lookup$Empty@4ce66f56]
09:14:38.586 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> createLookup() - for RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f took 1 msec.
09:14:38.586 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f.as(Composite) took 1 msec.
09:14:38.587 [TestNGInvoker-showNe] TRACE i.t.n.capabilitiesprovider.ThreadLookupBinder      - Unbound ProxyLookup(class=class org.openide.util.lookup.ProxyLookup)->[org.openide.util.lookup.SingletonLookup@13647278, org.openide.util.Lookup$Empty@4ce66f56] to thread Thread[TestNGInvoker-showNe,5,main]
09:14:39.590 [TestNGInvoker-showNe] DEBUG i.t.n.capabilitiesprovider.ThreadLookupBinder      - createLookup(ProxyLookup(class=class it.tidalwave.netbeans.util.test.MockLookup)->[org.openide.util.lookup.SingletonLookup@6477eb97, it.tidalwave.netbeans.util.test.MockLookup$1@2c979e8b, org.openide.util.lookup.SingletonLookup@7d0c3a08], RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}], ...)
09:14:39.591 [TestNGInvoker-showNe] TRACE i.t.n.capabilitiesprovider.ThreadLookupBinder      - >>>> lookups for RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}]: [org.openide.util.lookup.SingletonLookup@7d5b1773]
09:14:39.592 [TestNGInvoker-showNe] DEBUG i.t.n.capabilitiesprovider.ThreadLookupBinder      - createLookup(ProxyLookup(class=class org.openide.util.lookup.ProxyLookup)->[org.openide.util.lookup.SingletonLookup@13647278, org.openide.util.Lookup$Empty@4ce66f56], DefaultPresentationModel@6760bf50[[RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}]]], ...)
09:14:39.593 [TestNGInvoker-showNe] TRACE i.t.n.capabilitiesprovider.ThreadLookupBinder      - >>>> lookups for DefaultPresentationModel@6760bf50[[RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059@228b677f[id - {}]]]: [ProxyLookup(class=class org.openide.util.lookup.ProxyLookup)->[org.openide.util.lookup.SingletonLookup@7d5b1773]]
09:14:39.593 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> createLookup() - for DefaultPresentationModel@6760bf50 took 3 msec.
09:14:39.593 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059) took 3 msec.
09:14:39.594 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(ActionProvider) took 0 msec.
09:14:39.594 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(Readable) took 0 msec.
09:14:39.594 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(RssFeed$$EnhancerByMockitoWithCGLIB$$e2eb059) took 0 msec.
09:14:39.595 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(ActionProvider) took 0 msec.
09:14:39.595 [TestNGInvoker-showNe] TRACE it.tidalwave.netbeans.util.AsLookupSupport         - >>>> DefaultPresentationModel@6760bf50.as(Readable) took 0 msec.
09:14:39.597 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - TEST PASSED
09:14:39.599 [main                ] INFO  it.tidalwave.util.test.TestLogger                  -
09:14:39.599 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - ***********************************************************************************
09:14:39.599 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - TEST "showNewsFeed must properly populate the view when the news feed is available"
09:14:39.599 [main                ] INFO  it.tidalwave.util.test.TestLogger                  - ***********************************************************************************

Now, this is what I needed to do with TestNG: first add a short configuration to the SureFire plugin:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <properties>
                        <property>
                            <name>listener</name>
                            <value>it.tidalwave.util.test.TestLogger</value>
                        </property>
                    </properties>
                </configuration>
            </plugin>

and this is the implementation of it.tidalwave.util.test.TestLogger:

public class TestLogger extends TestListenerAdapter
  {
    private static final Logger log = LoggerFactory.getLogger(TestLogger.class);
    
    private static final String SEPARATOR = "************************************************************************************************";
    
    @Override
    public void onTestStart (final @Nonnull ITestResult result) 
      {
        final String separator = SEPARATOR.substring(0, Math.min(result.getName().length() + 7, SEPARATOR.length()));
        log.info("");
        log.info(separator);
        log.info("TEST "{}"", result.getName().replace('_', ' '));
        log.info(separator);
      }

    @Override
    public void onTestFailure (final @Nonnull ITestResult result) 
      {
        log.info("TEST FAILED");
      }

    @Override
    public void onTestSkipped (final @Nonnull ITestResult result) 
      {
        log.info("TEST SKIPPED");
      }

    @Override
    public void onTestSuccess (final @Nonnull ITestResult result)
      {
        log.info("TEST PASSED");
      }
  }
TestLogger is in a library that is included in my tests; thanks to POM inheritance you can have a single place where to configure everything and your project modules will inherit it.
Yes, in JUnit you can use the @RunWith annotation that allows to use your customized "test runner", that could do something similar. But the test runner is, well you betcha, also responsible for running the test. You should extend some existing code to preserve the existing functions and integrate the logging feature. Things would get more complicated when you need to use another "test runner" because a certain technology demands so. In TestNG it's way less intrusive: you add a listener and, as you can imagine, you can have multiple listeners working independently.

Related Topics >>
Blog Links >>