Skip to main content

Static analyzers comparison

Posted by cos on October 27, 2006 at 5:19 PM PDT
   The following is the evaluation of two static analysis solutions
   from Coverity and KlocWork.

    Background of the research

   The purpose of the conducted research was to find a software
   application capable of doing a source code static analysis. Any
   organization creating and maintaining a source code base above a
   couple of hundreds of lines of code has to take serious measures to
   guarantee overall code cleanness and absence of nasty design and
   architectural flaws.

   The traditional testing approaches are one of possible ways to
   achieve it; yet another one is the static analysis concept, which
   allows to find many hidden problems without doing any test
   development and execution. Besides, the development of tests to
   cover all possible paths of an application execution might be a
   non-trivial task by itself. Thus, static analyzers have to comply
   with a number of characteristics to be flawlessly embed into
   existing software development and quality control processes. Please
   find some of them below: * type of found bugs * level of false
   positive alerts * analysis performance (e.g. time of complete
   analysis should be measurable) * non-invasive integration into
   build systems * incremental analysis capabilities * extension API
   (e.g. to create more custom checkers) * platforms metric * user
   experience * resource requirements * miscellanies feature * price
   (base price, support fees, other expenses) * support of Solaris
   (Sparc/x86), Linux (i386/amd64), and Windows (i386/amd64) and a
   variety of compilers (Sun, GCC, Microsoft, etc.)

    Side-by-side comparison analysis

   The analysis has been performed over exactly the same workspace
   (Runtime baseline, about a couple of months old, i.e. end of June).

   Firstly, the tools have reported slightly different numbers of
   lines in the code. Coverity has ~455K (including 93K of comments
   and 70K of blank lines) and Klocwork only ~300K lines. Also, count
   of files is different, e.g. 1,639 and 1807 respectively.

      Types of defects

   Although both tools are utilizing pretty much the same techniques,
   their results are somehow different. Please be advised, that we
   were running both sessions on 'out-of-the-box' configurations of
   the analyzers.

        Coverity bugs

   Total: 319 (about 160 bugs are more like warnings, e.g. Variable is
   never been used and similar. The decent compiler should be able to
   catch these on a fly).

   +--------------------------------------------------------------------------------------------+
   |       Type        |                              Description                               |
   |-------------------+------------------------------------------------------------------------|
   |                   |Calling strncpy with a maximum size argument of (100) on |
   |BUFFER_SIZE_WARNING|destination buffer of size (100) can possibly leave the destination |
   |                   |buffer lacking null-termination                                         |
   |-------------------+------------------------------------------------------------------------|
   |CHECKED_RETURN     |Return value is never checked                                           |
   |-------------------+------------------------------------------------------------------------|
   |CTOR_DTOR_LEAK     |A field is allocated in the constructor, and there is no destructor     |
   |-------------------+------------------------------------------------------------------------|
   |DEADCODE           |Code isn't reachable                                                    |
   |-------------------+------------------------------------------------------------------------|
   |FORWARD_NULL       |Variable tracked as NULL was passed to a function that dereferences it  |
   |-------------------+------------------------------------------------------------------------|
   |NEGATIVE_RETURNS   |Unsigned tracked variable used as a loop boundary                       |
   |-------------------+------------------------------------------------------------------------|
   |NULL_RETURNS       |Dereferencing possibly NULL                                             |
   |-------------------+------------------------------------------------------------------------|
   |OVERRUN_STATIC     |Overrun of static array with index variable                             |
   |-------------------+------------------------------------------------------------------------|
   |PASS_BY_VALUE      |Parameter passed by value                                               |
   |-------------------+------------------------------------------------------------------------|
   |RESOURCE_LEAK      |Storage not saved                                                       |
   |-------------------+------------------------------------------------------------------------|
   |RETURN_LOCAL       |Returning pointer which is assigned to local variable                   |
   |-------------------+------------------------------------------------------------------------|
   |REVERSE_INULL      |Pointer is dereferenced before NULL check                               |
   |-------------------+------------------------------------------------------------------------|
   |STRING_NULL        |Unterminated string passed to a string null sink                        |
   |-------------------+------------------------------------------------------------------------|
   |TAINTED_SCALAR     |TAINTED variable used                                                   |
   |-------------------+------------------------------------------------------------------------|
   |UNINIT             |Using uninitialized value                                               |
   |-------------------+------------------------------------------------------------------------|
   |UNUSED_VALUE       |Pointer returned from a function is never used                          |
   |-------------------+------------------------------------------------------------------------|
   |USE_AFTER_FREE     |Dereferencing freed pointer                                             |
   +--------------------------------------------------------------------------------------------+

        Bugs found by KlocWork

   I have better details for this tool, because of lesser restrictions
   of their temp. license.

   +-------------------------------------------------------------------------------------+
   |                                Type                                 | ## |Percentage|
   |---------------------------------------------------------------------+----+----------|
   |Memory Leak                                                          |  11|      0.65|
   |---------------------------------------------------------------------+----+----------|
   |Result of function that can return NULL may be dereferenced          | 177|     10.47|
   |---------------------------------------------------------------------+----+----------|
   |Function returns reference to local variable                         |  11|      0.65|
   |---------------------------------------------------------------------+----+----------|
   |Result of function that may return NULL will be dereferenced         |1196|     70.77|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer may be dereferenced                                     |  24|      1.42|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable                                               |  92|      5.44|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable - possible                                    |  27|       1.6|
   |---------------------------------------------------------------------+----+----------|
   |Memyyyory Leak - possible                                            |  30|      1.78|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable in Constructor - possible                     |  53|      3.14|
   |---------------------------------------------------------------------+----+----------|
   |Pointer will be dereferenced after it was positively checked for NULL|  19|      1.12|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer will be dereferenced                                    |  13|      0.77|
   |---------------------------------------------------------------------+----+----------|
   |Suspicious dereference of pointer before NULL check                  |  16|      0.95|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer may be passed to function that may dereference it       |   1|      0.06|
   |---------------------------------------------------------------------+----+----------|
   |Suspicious dereference of pointer in function call before NULL check |   4|      0.24|
   |---------------------------------------------------------------------+----+----------|
   |Use of Freed Memory                                                  |   2|      0.12|
   +-------------------------------------------------------------------------------------+

   Potential security defects
   |---------------------------------------------------------------------+----+----------|
   |Buffer Overflow - Array Index Out of Bounds                          | 113|     96.58|
   |---------------------------------------------------------------------+----+----------|
   |Use of Unvalidated Integer as Array Index by Function Call           |   1|      0.85|
   |---------------------------------------------------------------------+----+----------|
   |Use of Unvalidated Integer in Loop Condition                         |   3|      2.56|
   +-------------------------------------------------------------------------------------+

   For the sake of convenience, I split the bugs above into a few categories:

   +-------------------------------------------------------------------------------+
   |                       Type of defects                       |Coverity|KlocWork|
   |-------------------------------------------------------------+--------+--------|
   |Resource leaks                                               |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |NULL dereferencing                                           |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Unused values/Unchecked returns                              |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Return locals                                                |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Security (buffer overflow/un-validated array's indeces/other)|   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Use after free                                               |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Deadcode                                                     |   x    |        |
   |-------------------------------------------------------------+--------+--------|
   |Static overrun                                               |   x    |        |
   |-------------------------------------------------------------+--------+--------|
   |Passing by value                                             |   x    |        |
   +-------------------------------------------------------------------------------+

   Overall, KlocWork seems to find more potential defects, although it
   also reports a lot (>80%) of potentially bad, but just probable
   defects, which requires more attention in post-analysis phases. I
   got some explanations from KlocWork's team, saying that all defects
   are falling into two different kinds: MUST and POSSIBLE. And
   checkers can be configured, so one might choose to turn off all
   POSSIBLE defects.


      Noise level/False positives

   According to the companies, their tools are having about the same
   levels of false positives, e.g. 10-15% respectively. However,
   Coverity claims that after some configurations, the level of false
   positives drops significantly down to 3-5%. Plus, Coverity has a
   way of modeling primitives' semantics which allows to adapt the
   analysis for some models, not supported out of the box.

   On a similar note, I got the following comment from KlocWork's
   team: "...Upon installation, however, Klocwork undergoes the same
   tuning process that our competitors do with their software. Our
   false positives rate drops significantly after Klocwork is tuned
   for your code...


      Analysis time

   +-------------------------+
   |Product |Time of analysis|
   |--------+----------------|
   |Coverity|        05:35:54|
   |--------+----------------|
   |KlocWork|        ~2 hours|
   +-------------------------+

   I have to note, however, that the KlocWork team was smarly utilizing the
   power of Sun's Z880 server by running the build and consequent analysis as
   six parallel processes. Therefore, I'm not sure what would be the time of
   Coverity's analysis, being ran in the same mode.

      Build techniques

   Coverity has a way of redefining chunks of analyzed code. E.g. one
   can specify macros to redefine something in the code or explicitly
   suppress analysis of some pieces. From the later explanation I
   gathered, that KlocWork has similar capabilities, with an extensive
   knowledgebase documentation on how to redefine how a function operates and
   even implement for third-party libraries for which no source code is
   available.

   Both tools are utilizing non-invasive build techniques. Instead of
   introducing any changes into the client's build environment, they
   are monitoring build processes. Coverity is redirecting any calls
   to the standard compilers to their own frontends to collect all
   needed statistics/info. As the result of the build, a normal code
   is produced along with all data, required for further analysis.

   KlocWork has a utility called kwinject. They also have other
   utilities and means that help to be more versatile when integrating
   into the diverse types of environments.

   Overall, both systems can be reasonably easy integrated into
   existing development/quality life cycles.

      Incremental analysis

   Both tools have an ability to do incremental code analysis. E.g. if
   only a few files were changed in the workspace, then only those
   files will be analyzed. Such consequent analysis has to take much
   less time.

   Overall, in both cases these can be use in daily development
   activities for pre-integration checks or for ad-hoc analysis of a
   new and existing code.

      Extention APIs

   Both tools have ways of accessing the internal data gathered at the
   stage of the analysis.  E.g. with provided C API, one can work
   directly with static callgraph and data flow information and create
   custom checkers.

   Overall, I think this is a very important thing for us as we need
   to think of Impact analysis technique, better than BSP (to know more about BSP please check this.

      Platforms metric

   +------------------------------------------------------------------------+-----+
   |Product |Windows flavors|Linuses|Solaris|sparc|x86|64-bit|Java          |C/C++|
   |--------+---------------+-------+-------+-----+---+------+--------------+-----|
   |Coverity|       x       |   x   |   x   |  x  | x |  x   | coming soon  |  x  |
   |--------+---------------+-------+-------+-----+---+------+--------------+-----|
   |KlocWork|       x       |   x   |   x   |  x  | x |  x   |     x        |  x  |
   +------------------------------------------------------------------------+-----+

   Despite Java support's current absense, Coverity is coming up with
   it later in October, 2006.  We were invited to participate in the
   product's beta testing in a few weeks.


      User experience

   Coverity has a very convenient source code browser, which allows
   looking into any part of a product's code. The browser creates
   hyperlinks to all known types and structures. Also, it allows to
   generate some reports per build, find difference between two
   builds, et cetera.

   KlocWork UI has a very intensive set of features to hide unwanted
   defect records. They aren't deleted, but just aren't shown in the
   reports. There are 10 different sub-categories of the defects'
   severety taxonomy, which helps to see only groups of defects one
   wants to inspect. It also has an interface to produce custom
   reports on top of 10+ standard ones.

   Both tools have quite efficient help systems with a lot of typical
   defects, examples, and explanations.

   Overall, Coverity's UI is more ascetic and somehow slower, but more
   tuned for its purpose.

      Resource requirements

   Both tools are quite moderate in terms of hardware utilization and
   are doing pretty well on the V890 machine.

      Miscellanious features

   Coverity has a number of concurrency checks, e.g. double locks, a
   lock without freeing, a double free, and inconsistent local
   order.


Additional configurations are available for sleep functions
   and those that need to be holding an exclusive lock when
   called. Also, it supports different threading models.

   KlocWork mostly lacks concurrency analysis features and does only
   some very basic stuff in this area. I've been recently told by KlocWork's
   team that their latest version of the software (7.5) has a number of
   concurrent checkers implemented. Besides, it has some kind of CASE tool
   packaged along with a static analyzer, designated to help in finding and
   fixing the code design's flaws.  However, this topic is beyond this
   analysis' boundaries.

      Pricing models

   Coverity and KlocWork have similar pricing model, although they are
   slightly different in the overall costs. Basically, Coverity
   charges for volume of the code only; KlocWork.charges for this and
   for user licenses. The latter might cost you if you're planning to
   use for the relatively big team.

      More information

   could be found at http://www.coverity.com/ and
   http://www.klocwork.com/ and/or requested directly from their sales
   teams.

      Word of caution.

   All tests mentioned above were performed on the versions of the
   products, which were current when the article has been written,
   e.g. June-August 2006

I wrote this article a couple of months back, however it took me awhile to get an agreement from all interested parties and make some corrections in my original text. Please feel free to let me know if you have any comments, ideas, suggestions, etc.

Take care,

Cos

P.S. The original formatting of the article was preserved, because java.net's blogging system is simply lacks of many essentials.

Related Topics >>