 |
October 2006 Archives
Static analyzers comparison
Posted by cos on October 27, 2006 at 05:19 PM | Permalink
| Comments (5)
The following is the evaluation of two static analysis solutions
from Coverity and KlocWork.
Background of the research
The purpose of the conducted research was to find a software
application capable of doing a source code static analysis. Any
organization creating and maintaining a source code base above a
couple of hundreds of lines of code has to take serious measures to
guarantee overall code cleanness and absence of nasty design and
architectural flaws.
The traditional testing approaches are one of possible ways to
achieve it; yet another one is the static analysis concept, which
allows to find many hidden problems without doing any test
development and execution. Besides, the development of tests to
cover all possible paths of an application execution might be a
non-trivial task by itself. Thus, static analyzers have to comply
with a number of characteristics to be flawlessly embed into
existing software development and quality control processes. Please
find some of them below: * type of found bugs * level of false
positive alerts * analysis performance (e.g. time of complete
analysis should be measurable) * non-invasive integration into
build systems * incremental analysis capabilities * extension API
(e.g. to create more custom checkers) * platforms metric * user
experience * resource requirements * miscellanies feature * price
(base price, support fees, other expenses) * support of Solaris
(Sparc/x86), Linux (i386/amd64), and Windows (i386/amd64) and a
variety of compilers (Sun, GCC, Microsoft, etc.)
Side-by-side comparison analysis
The analysis has been performed over exactly the same workspace
(Runtime baseline, about a couple of months old, i.e. end of June).
Firstly, the tools have reported slightly different numbers of
lines in the code. Coverity has ~455K (including 93K of comments
and 70K of blank lines) and Klocwork only ~300K lines. Also, count
of files is different, e.g. 1,639 and 1807 respectively.
Types of defects
Although both tools are utilizing pretty much the same techniques,
their results are somehow different. Please be advised, that we
were running both sessions on 'out-of-the-box' configurations of
the analyzers.
Coverity bugs
Total: 319 (about 160 bugs are more like warnings, e.g. Variable is
never been used and similar. The decent compiler should be able to
catch these on a fly).
+--------------------------------------------------------------------------------------------+
| Type | Description |
|-------------------+------------------------------------------------------------------------|
| |Calling strncpy with a maximum size argument of (100) on |
|BUFFER_SIZE_WARNING|destination buffer of size (100) can possibly leave the destination |
| |buffer lacking null-termination |
|-------------------+------------------------------------------------------------------------|
|CHECKED_RETURN |Return value is never checked |
|-------------------+------------------------------------------------------------------------|
|CTOR_DTOR_LEAK |A field is allocated in the constructor, and there is no destructor |
|-------------------+------------------------------------------------------------------------|
|DEADCODE |Code isn't reachable |
|-------------------+------------------------------------------------------------------------|
|FORWARD_NULL |Variable tracked as NULL was passed to a function that dereferences it |
|-------------------+------------------------------------------------------------------------|
|NEGATIVE_RETURNS |Unsigned tracked variable used as a loop boundary |
|-------------------+------------------------------------------------------------------------|
|NULL_RETURNS |Dereferencing possibly NULL |
|-------------------+------------------------------------------------------------------------|
|OVERRUN_STATIC |Overrun of static array with index variable |
|-------------------+------------------------------------------------------------------------|
|PASS_BY_VALUE |Parameter passed by value |
|-------------------+------------------------------------------------------------------------|
|RESOURCE_LEAK |Storage not saved |
|-------------------+------------------------------------------------------------------------|
|RETURN_LOCAL |Returning pointer which is assigned to local variable |
|-------------------+------------------------------------------------------------------------|
|REVERSE_INULL |Pointer is dereferenced before NULL check |
|-------------------+------------------------------------------------------------------------|
|STRING_NULL |Unterminated string passed to a string null sink |
|-------------------+------------------------------------------------------------------------|
|TAINTED_SCALAR |TAINTED variable used |
|-------------------+------------------------------------------------------------------------|
|UNINIT |Using uninitialized value |
|-------------------+------------------------------------------------------------------------|
|UNUSED_VALUE |Pointer returned from a function is never used |
|-------------------+------------------------------------------------------------------------|
|USE_AFTER_FREE |Dereferencing freed pointer |
+--------------------------------------------------------------------------------------------+
Bugs found by KlocWork
I have better details for this tool, because of lesser restrictions
of their temp. license.
+-------------------------------------------------------------------------------------+
| Type | ## |Percentage|
|---------------------------------------------------------------------+----+----------|
|Memory Leak | 11| 0.65|
|---------------------------------------------------------------------+----+----------|
|Result of function that can return NULL may be dereferenced | 177| 10.47|
|---------------------------------------------------------------------+----+----------|
|Function returns reference to local variable | 11| 0.65|
|---------------------------------------------------------------------+----+----------|
|Result of function that may return NULL will be dereferenced |1196| 70.77|
|---------------------------------------------------------------------+----+----------|
|Null pointer may be dereferenced | 24| 1.42|
|---------------------------------------------------------------------+----+----------|
|Uninitialized Variable | 92| 5.44|
|---------------------------------------------------------------------+----+----------|
|Uninitialized Variable - possible | 27| 1.6|
|---------------------------------------------------------------------+----+----------|
|Memyyyory Leak - possible | 30| 1.78|
|---------------------------------------------------------------------+----+----------|
|Uninitialized Variable in Constructor - possible | 53| 3.14|
|---------------------------------------------------------------------+----+----------|
|Pointer will be dereferenced after it was positively checked for NULL| 19| 1.12|
|---------------------------------------------------------------------+----+----------|
|Null pointer will be dereferenced | 13| 0.77|
|---------------------------------------------------------------------+----+----------|
|Suspicious dereference of pointer before NULL check | 16| 0.95|
|---------------------------------------------------------------------+----+----------|
|Null pointer may be passed to function that may dereference it | 1| 0.06|
|---------------------------------------------------------------------+----+----------|
|Suspicious dereference of pointer in function call before NULL check | 4| 0.24|
|---------------------------------------------------------------------+----+----------|
|Use of Freed Memory | 2| 0.12|
+-------------------------------------------------------------------------------------+
Potential security defects
|---------------------------------------------------------------------+----+----------|
|Buffer Overflow - Array Index Out of Bounds | 113| 96.58|
|---------------------------------------------------------------------+----+----------|
|Use of Unvalidated Integer as Array Index by Function Call | 1| 0.85|
|---------------------------------------------------------------------+----+----------|
|Use of Unvalidated Integer in Loop Condition | 3| 2.56|
+-------------------------------------------------------------------------------------+
For the sake of convenience, I split the bugs above into a few categories:
+-------------------------------------------------------------------------------+
| Type of defects |Coverity|KlocWork|
|-------------------------------------------------------------+--------+--------|
|Resource leaks | x | x |
|-------------------------------------------------------------+--------+--------|
|NULL dereferencing | x | x |
|-------------------------------------------------------------+--------+--------|
|Unused values/Unchecked returns | x | x |
|-------------------------------------------------------------+--------+--------|
|Return locals | x | x |
|-------------------------------------------------------------+--------+--------|
|Security (buffer overflow/un-validated array's indeces/other)| x | x |
|-------------------------------------------------------------+--------+--------|
|Use after free | x | x |
|-------------------------------------------------------------+--------+--------|
|Deadcode | x | |
|-------------------------------------------------------------+--------+--------|
|Static overrun | x | |
|-------------------------------------------------------------+--------+--------|
|Passing by value | x | |
+-------------------------------------------------------------------------------+
Overall, KlocWork seems to find more potential defects, although it
also reports a lot (>80%) of potentially bad, but just probable
defects, which requires more attention in post-analysis phases. I
got some explanations from KlocWork's team, saying that all defects
are falling into two different kinds: MUST and POSSIBLE. And
checkers can be configured, so one might choose to turn off all
POSSIBLE defects.
Noise level/False positives
According to the companies, their tools are having about the same
levels of false positives, e.g. 10-15% respectively. However,
Coverity claims that after some configurations, the level of false
positives drops significantly down to 3-5%. Plus, Coverity has a
way of modeling primitives' semantics which allows to adapt the
analysis for some models, not supported out of the box.
On a similar note, I got the following comment from KlocWork's
team: "...Upon installation, however, Klocwork undergoes the same
tuning process that our competitors do with their software. Our
false positives rate drops significantly after Klocwork is tuned
for your code...
Analysis time
+-------------------------+
|Product |Time of analysis|
|--------+----------------|
|Coverity| 05:35:54|
|--------+----------------|
|KlocWork| ~2 hours|
+-------------------------+
I have to note, however, that the KlocWork team was smarly utilizing the
power of Sun's Z880 server by running the build and consequent analysis as
six parallel processes. Therefore, I'm not sure what would be the time of
Coverity's analysis, being ran in the same mode.
Build techniques
Coverity has a way of redefining chunks of analyzed code. E.g. one
can specify macros to redefine something in the code or explicitly
suppress analysis of some pieces. From the later explanation I
gathered, that KlocWork has similar capabilities, with an extensive
knowledgebase documentation on how to redefine how a function operates and
even implement for third-party libraries for which no source code is
available.
Both tools are utilizing non-invasive build techniques. Instead of
introducing any changes into the client's build environment, they
are monitoring build processes. Coverity is redirecting any calls
to the standard compilers to their own frontends to collect all
needed statistics/info. As the result of the build, a normal code
is produced along with all data, required for further analysis.
KlocWork has a utility called kwinject. They also have other
utilities and means that help to be more versatile when integrating
into the diverse types of environments.
Overall, both systems can be reasonably easy integrated into
existing development/quality life cycles.
Incremental analysis
Both tools have an ability to do incremental code analysis. E.g. if
only a few files were changed in the workspace, then only those
files will be analyzed. Such consequent analysis has to take much
less time.
Overall, in both cases these can be use in daily development
activities for pre-integration checks or for ad-hoc analysis of a
new and existing code.
Extention APIs
Both tools have ways of accessing the internal data gathered at the
stage of the analysis. E.g. with provided C API, one can work
directly with static callgraph and data flow information and create
custom checkers.
Overall, I think this is a very important thing for us as we need
to think of Impact analysis technique, better than BSP (to know more about BSP please check this.
Platforms metric
+------------------------------------------------------------------------+-----+
|Product |Windows flavors|Linuses|Solaris|sparc|x86|64-bit|Java |C/C++|
|--------+---------------+-------+-------+-----+---+------+--------------+-----|
|Coverity| x | x | x | x | x | x | coming soon | x |
|--------+---------------+-------+-------+-----+---+------+--------------+-----|
|KlocWork| x | x | x | x | x | x | x | x |
+------------------------------------------------------------------------+-----+
Despite Java support's current absense, Coverity is coming up with
it later in October, 2006. We were invited to participate in the
product's beta testing in a few weeks.
User experience
Coverity has a very convenient source code browser, which allows
looking into any part of a product's code. The browser creates
hyperlinks to all known types and structures. Also, it allows to
generate some reports per build, find difference between two
builds, et cetera.
KlocWork UI has a very intensive set of features to hide unwanted
defect records. They aren't deleted, but just aren't shown in the
reports. There are 10 different sub-categories of the defects'
severety taxonomy, which helps to see only groups of defects one
wants to inspect. It also has an interface to produce custom
reports on top of 10+ standard ones.
Both tools have quite efficient help systems with a lot of typical
defects, examples, and explanations.
Overall, Coverity's UI is more ascetic and somehow slower, but more
tuned for its purpose.
Resource requirements
Both tools are quite moderate in terms of hardware utilization and
are doing pretty well on the V890 machine.
Miscellanious features
Coverity has a number of concurrency checks, e.g. double locks, a
lock without freeing, a double free, and inconsistent local
order.
Additional configurations are available for sleep functions
and those that need to be holding an exclusive lock when
called. Also, it supports different threading models.
KlocWork mostly lacks concurrency analysis features and does only
some very basic stuff in this area. I've been recently told by KlocWork's
team that their latest version of the software (7.5) has a number of
concurrent checkers implemented. Besides, it has some kind of CASE tool
packaged along with a static analyzer, designated to help in finding and
fixing the code design's flaws. However, this topic is beyond this
analysis' boundaries.
Pricing models
Coverity and KlocWork have similar pricing model, although they are
slightly different in the overall costs. Basically, Coverity
charges for volume of the code only; KlocWork.charges for this and
for user licenses. The latter might cost you if you're planning to
use for the relatively big team.
More information
could be found at http://www.coverity.com/ and
http://www.klocwork.com/ and/or requested directly from their sales
teams.
Word of caution.
All tests mentioned above were performed on the versions of the
products, which were current when the article has been written,
e.g. June-August 2006
I wrote this article a couple of months back, however it took me awhile to get an agreement from all interested parties and make some corrections in my original text. Please feel free to let me know if you have any comments, ideas, suggestions, etc.
Take care,
Cos
P.S. The original formatting of the article was preserved, because java.net's blogging system is simply lacks of many essentials.
|
java.net RSS Feeds
