<?xml version="1.0" encoding="utf-8"?>
<feed version="0.3" xmlns="http://purl.org/atom/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:lang="en">
<title>Konstantin I. Boudnik&apos;s Blog</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/" />
<modified>2007-03-30T00:53:37Z</modified>
<tagline></tagline>
<id>tag:weblogs.java.net,2008:/blog/cos/277</id>
<generator url="http://www.movabletype.org/" version="3.01D">Movable Type</generator>
<copyright>Copyright (c) 2007, cos</copyright>
<entry>
<title>Software reliability</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2007/03/software_reliab_1.html" />
<modified>2007-03-30T00:53:37Z</modified>
<issued>2007-03-30T00:53:22Z</issued>
<id>tag:weblogs.java.net,2007:/blog/cos/277.6949</id>
<created>2007-03-30T00:53:22Z</created>
<summary type="text/plain">I hope this article will help to scratch issues of software reliability. Brief discussion about established reliability practices and what has to be changed. </summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[<pre>
Seems like the process of bring Java under an open source license has
raised the question of Java platform quality even higher. In
particular the question of reliability has been discussed more and
more widely among my peers over last few months. Thus, I decided to
share a couple of thoughts on the topic. Hopefully, you'll like what
you'll about to see.

What realibility means for us.

According to IEEE definition, reliability is "The ability of a system
or component to perform its required functions under stated conditions
for a specified period of time." [1]

First of all, I'd like to emphasize words "required functions",
"stated conditions", and "specified period". Also I want to add
"repetitively". I believe later on it will be clear, why I've focused
on those.

The majority of software reliability studies are paying a good deal of
attention to the amount of time a system or component can perform
without a failure. Most of it is dealing with different kinds of
fault/time distributions, estimations of failure intensity, failure
likelihood probabilities, failure intervals, and such. I beg your
pardon for rather long citation: "...There is a lot of lore about
system testing, but it all boils down to guesswork. That is, it is
guesswork unless you can structure the problem and perform the testing
so that you can apply mathematical statistics.  If you can do this,
you can say some- thing like "No, we cannot be absolutely certain that
the software will never fail, but relative to a theoretically sound
and experimentally validated statistical model, we have done
sufficient testing to say with 95-percent confidence that the
probability of 1,OOO CPU hours of failure-free operation in a
probabilistically defined environment is at least 0.995."  When you do
this, you are applying software-reliability measurement." [2]

With no attempt to underestimate or undermine such studies ([3]), and
being in the agreement with absolute necessity of statistical modeling
and verification of the software testing, I want to talk about a wider
approach. It isn't perhaps a brand new one, but might be slightly
different from what you've seeing so far.

Different takes on quality.

I love to talk about quality, mostly because this is very vast topic
and one can sell some nonsense :-)

As I see this, there are two main quality approaches. I'll call them
hardware and software types. The main differences between those are
coming from the production cycle specifics of devices and
applications. Namely these are:
  - hardware production has much higher costs because of complex
    factory processes, complicated and costly equipment involved, et
    cetera. Thus, you'd better be careful with how a device's
    components are designed, produced, assembled together, and
    tested. It might cost a fortune to make changes in a silicon chip,
    a motherboard design, or a car once it's out

    Eventually, hardware development is addressed with more "respect"
    and precise planning because of high up front investment.

  - software, on the other hand, usually has more flexible life span,
    the targets are sometimes easily moved along the development
    process, requirement are changed, design documents might be
    somehow informal, specs changes might not be well tracked down to
    the real application defects, quality process gets stuck behind,
    and on, and on... At the end of the day a software application
    reaches its customers and they start finding bugs in it. Then an
    escalation is being arisen. And the product's sustaining team has
    to spend time to mirror customer's setup, repeat all the steps to
    reproduce a defect, etc. And consider yourself lucky if all of
    this can be done just in one interaction. However, if defect
    report wasn't detailed enough or the setup was a way too
    sophisticated you might spend months to nail down a particular
    problem. We all saw this many times, right?

Our take on the problem represents a mix between those two above as
follows. I wanted to bring the best parts of the hardware reliability
and bring it over to the software one wherever possible. Here's what I
see as necessary steps:
  1) Design and architectural reviews (many teams are doing this
     already)
  1a) Tracking correlations between architecture decisions, changes and
     discovered defects
  2) Mean-Time-To-Failure (MTTF) testing. A quality department can run
     some preferably standardized applications for a prolonged period
     of time to demonstrate the stability of the software
     platform. However a better approach would be to run scenario
     based MMTF tests.
  3) Employing statistical analysis of quality trends
  4) Enforcing static analysis valuations on the periodic basis
  5a) Scenario based MMTF testing. Normally, one can gather a few (may
     be a hundred or so) typical usage scenarios for a software
     application. The number likely to be much higher for a software
     platform like Java. These scenarios might be simulated or
     replicated with a test harness of choice and a specific set of
     existing or newly developed tests. Of course, you might not be
     able to simulate any of these real-life scenarios with 100%
     accuracy, but it's not always necessary. These scenarios then
     should be executed repeatedly and their pass/fail rate has to be
     tracked over time.
  5b) Scenarios completeness. Using a list of features, utilized
     during a scenario execution, and static analysis results one can
     tell which parts of a software application will be touched during
     a particular scenario's run. Using code coverage methods you can
     findout which parts of the scenario's functionality are covered
     or not. With something similar to BSP
     http://weblogs.java.net/blog/cos/archive/2005/12/java_quality_me_6.html
     you can leverage efforts of the improvements, but this is another
     story and it's been covered already.
  6) Quality trends monitoring. The proceedings of #5a should be
     included here.

When I'm communicating these steps to my peers and colleagues I'm
hearing a number of concerns. Typically, these are:
  - how #2 is connected to the reliability
  - #4 seems to be an over stretch
  - how you can be sure that #5a is the same as running heavy weight
    applications to verity your platform stability/reliability

Hopefully, I'll be able to answer these or other questions, you might
send to me as your comments.

  1) Why design and architectural review? Long story short, you can
     keep bad solutions away from your system. Proven practices
     usually guarantee lesser amount of last minutes changes at the
     development stage. Thus, the testing burden will be lower, as
     well as amount of regressions, customer escalation, etc.  
     What about 1a)? I don't know - it just sounds cool, I guess :-)

  2) Everybody seems to be doing this, so why don't we..? Seriously,
     this one of the reliability's aspects you want to count, because
     it backed up by well developed theory and years of practice, and
     this one is meaningful quantitative metric.

  3) Not sure why? Just read some of those books, will ya? ;-)

  4) Static analysis is capable of finding types of defects, which
     aren't likely to be discovered in the runtime. That happens
     because for complex systems you can't guarantee a coverage of
     Cartesian product of input and output states sets. However, some
     of the nasty bugs are tend to be hiding right in those dusty
     corners, which you or one of your customers only might hit once
     in a while. Thus, if you are running a designated static
     analyzers cleanly on every build of yours, you at least can
     demonstrate, that it doesn't leak memory or running out of file
     handles. Consider that reliable also means trustworthy.

     One might say, that you can track memory leaks with runtime
     monitoring. True. But how you'll going to find and fix them now?

  5a) Is giving you the determinism of testing repetitiveness which is
      likely to be missed with BigApps approach, discussed later. 
      5b) is complimentary to this one.

  6) You want to know if your development/quality processes are
     convergent, right?

And finally I'd like to mention several common reliability
approaches. Also I'm going to explain why I see these as
misconceptions.

1) Stress testing
   This one is most oftenly being messed up with reliability
   concept. The reason for this is perhaps clear, because one might
   expect from a reliable system to work in a wide variety of
   conditions and perform its functions well. 

   You can hear a word on the streets, that '...Microsoft Windows is
   unreliable." Hell, yes. It sure isn't if you'll try to debug a huge
   C++ project, process some statistical data, get a bunch of spam
   emails, and install 20+ security fixes from their update center at
   the same time. It will likely to crash and destroy some of your
   files, or it might hung nicely. Or you'll suffer some critical
   performance degradations. I can't tell for sure, 'cause I'm not one
   of those lucky Windows users. And I'm not trying to make fun of
   Windows - people are doing so with their computers on a daily basis
   much better than I can even dream of :-) My point is to tell, that
   the scenario above is a bit extreme and well beyond an average
   Windows' users capabilities or, perhaps, desire.

   However, normally your Visual C++ project debug session will go
   smoothly in probably 95% of cases (however, I once was doing some
   C# project, which crashed my development machine to BSoD on every
   load. But an after crash attempt to load it again was always a
   success. Weirdo...). Did you ever count how many times your email
   client worked well when you were sending your emails? Perhaps not,
   but I'm sure almost everyone has a story to tell about so badly
   corrupted was address book last time the Outlook crashed, right?

   Correctly data series processing and features usage information
   gathering can relatively easy demonstrate that the Outlook is
   reliable application. It has, say, 93.5% failure free behavior over
   every 10 hours of execution. But it hard to guarantee that this
   application will survive under some monstrous load conditions. 

2) BigApps testing
   The concept of BigApps testing consists of running some bulky
   commercial applications to derive MTTF for. usually, a software
   platform. Well, I see three fold problem here (I'm sure there more
   of these, but I'll let you to deduce them on your own:)
      1. Any BigApp run is as good as the typical utilization (or
	 usage scenario) of features of your platform by that
	 application.
      2. The correctness the exercised application itself might be
         questionable
      3. Results you'll see at the completion of a run should be
	 considered accountable to that particular application. If you
	 were running a PeopleSoft system for a week and have
	 demonstrated a MTTF=140 hours that is great... For PeopleSoft
	 marketing and PR team, but not that useful for your
	 development organization. That gives them a little of handy
	 info. Although, if a crash will occur the engineering team
	 can discover some really bad problem in the code and fix
	 it. Which is the rare case of non-zero sum game!

	 It might be cool marketing or sales tool to use on customers,
	 but it might be not as great for engineers.

I hope this article helped to scratch some surface of this
problem. Please let me know what do you think, pinpoint the lacks of
logic, or just yell at me if you think I'm wrong. Let's communicate
about this. May be we'll work something out that we all can use later
for the applications and products we develop for life or for an
enjoyment.

Cheers,
  Cos

--
[1] IEEE 982.2-1988 standard. Has been withdrawn in 2002
[2] John D. Musa. A. Frank Ackerman."Quantifying Software Validation:
    When to Stop Testing?"
[3] http://portal.acm.org/citation.cfm?id=22980&dl=#
</pre>]]>

</content>
</entry>
<entry>
<title>Wanna know where you&apos;re located</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/12/wanna_know_wher.html" />
<modified>2006-12-06T04:49:28Z</modified>
<issued>2006-12-06T04:49:19Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.6101</id>
<created>2006-12-06T04:49:19Z</created>
<summary type="text/plain">How to find out from where you are connecting to Internet</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Mobility</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
Want to know you current network&apos;s location - check this out http://ip-adress.com/

</content>
</entry>
<entry>
<title>Static analyzers comparison</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/10/static_analyzer.html" />
<modified>2007-03-28T22:21:41Z</modified>
<issued>2006-10-28T01:19:11Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.5813</id>
<created>2006-10-28T01:19:11Z</created>
<summary type="text/plain">Comparison of two popular static analyzers from Coverity and KlocWork
The analysis has been done over popular Sun&apos;s Java Hotspot VM</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Programming</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[<pre>
   The following is the evaluation of two static analysis solutions
   from Coverity and KlocWork.

    Background of the research

   The purpose of the conducted research was to find a software
   application capable of doing a source code static analysis. Any
   organization creating and maintaining a source code base above a
   couple of hundreds of lines of code has to take serious measures to
   guarantee overall code cleanness and absence of nasty design and
   architectural flaws.

   The traditional testing approaches are one of possible ways to
   achieve it; yet another one is the static analysis concept, which
   allows to find many hidden problems without doing any test
   development and execution. Besides, the development of tests to
   cover all possible paths of an application execution might be a
   non-trivial task by itself. Thus, static analyzers have to comply
   with a number of characteristics to be flawlessly embed into
   existing software development and quality control processes. Please
   find some of them below: * type of found bugs * level of false
   positive alerts * analysis performance (e.g. time of complete
   analysis should be measurable) * non-invasive integration into
   build systems * incremental analysis capabilities * extension API
   (e.g. to create more custom checkers) * platforms metric * user
   experience * resource requirements * miscellanies feature * price
   (base price, support fees, other expenses) * support of Solaris
   (Sparc/x86), Linux (i386/amd64), and Windows (i386/amd64) and a
   variety of compilers (Sun, GCC, Microsoft, etc.)

    Side-by-side comparison analysis

   The analysis has been performed over exactly the same workspace
   (Runtime baseline, about a couple of months old, i.e. end of June).

   Firstly, the tools have reported slightly different numbers of
   lines in the code. Coverity has ~455K (including 93K of comments
   and 70K of blank lines) and Klocwork only ~300K lines. Also, count
   of files is different, e.g. 1,639 and 1807 respectively.

      Types of defects

   Although both tools are utilizing pretty much the same techniques,
   their results are somehow different. Please be advised, that we
   were running both sessions on 'out-of-the-box' configurations of
   the analyzers.

        Coverity bugs

   Total: 319 (about 160 bugs are more like warnings, e.g. Variable is
   never been used and similar. The decent compiler should be able to
   catch these on a fly). 

   +--------------------------------------------------------------------------------------------+
   |       Type        |                              Description                               |
   |-------------------+------------------------------------------------------------------------|
   |                   |Calling strncpy with a maximum size argument of (100) on		|
   |BUFFER_SIZE_WARNING|destination buffer of size (100) can possibly leave the destination	|
   |                   |buffer lacking null-termination                                         |
   |-------------------+------------------------------------------------------------------------|
   |CHECKED_RETURN     |Return value is never checked                                           |
   |-------------------+------------------------------------------------------------------------|
   |CTOR_DTOR_LEAK     |A field is allocated in the constructor, and there is no destructor     |
   |-------------------+------------------------------------------------------------------------|
   |DEADCODE           |Code isn't reachable                                                    |
   |-------------------+------------------------------------------------------------------------|
   |FORWARD_NULL       |Variable tracked as NULL was passed to a function that dereferences it  |
   |-------------------+------------------------------------------------------------------------|
   |NEGATIVE_RETURNS   |Unsigned tracked variable used as a loop boundary                       |
   |-------------------+------------------------------------------------------------------------|
   |NULL_RETURNS       |Dereferencing possibly NULL                                             |
   |-------------------+------------------------------------------------------------------------|
   |OVERRUN_STATIC     |Overrun of static array with index variable                             |
   |-------------------+------------------------------------------------------------------------|
   |PASS_BY_VALUE      |Parameter passed by value                                               |
   |-------------------+------------------------------------------------------------------------|
   |RESOURCE_LEAK      |Storage not saved                                                       |
   |-------------------+------------------------------------------------------------------------|
   |RETURN_LOCAL       |Returning pointer which is assigned to local variable                   |
   |-------------------+------------------------------------------------------------------------|
   |REVERSE_INULL      |Pointer is dereferenced before NULL check                               |
   |-------------------+------------------------------------------------------------------------|
   |STRING_NULL        |Unterminated string passed to a string null sink                        |
   |-------------------+------------------------------------------------------------------------|
   |TAINTED_SCALAR     |TAINTED variable used                                                   |
   |-------------------+------------------------------------------------------------------------|
   |UNINIT             |Using uninitialized value                                               |
   |-------------------+------------------------------------------------------------------------|
   |UNUSED_VALUE       |Pointer returned from a function is never used                          |
   |-------------------+------------------------------------------------------------------------|
   |USE_AFTER_FREE     |Dereferencing freed pointer                                             |
   +--------------------------------------------------------------------------------------------+

        Bugs found by KlocWork

   I have better details for this tool, because of lesser restrictions
   of their temp. license.

   +-------------------------------------------------------------------------------------+
   |                                Type                                 | ## |Percentage|
   |---------------------------------------------------------------------+----+----------|
   |Memory Leak                                                          |  11|      0.65|
   |---------------------------------------------------------------------+----+----------|
   |Result of function that can return NULL may be dereferenced          | 177|     10.47|
   |---------------------------------------------------------------------+----+----------|
   |Function returns reference to local variable                         |  11|      0.65|
   |---------------------------------------------------------------------+----+----------|
   |Result of function that may return NULL will be dereferenced         |1196|     70.77|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer may be dereferenced                                     |  24|      1.42|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable                                               |  92|      5.44|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable - possible                                    |  27|       1.6|
   |---------------------------------------------------------------------+----+----------|
   |Memyyyory Leak - possible                                            |  30|      1.78|
   |---------------------------------------------------------------------+----+----------|
   |Uninitialized Variable in Constructor - possible                     |  53|      3.14|
   |---------------------------------------------------------------------+----+----------|
   |Pointer will be dereferenced after it was positively checked for NULL|  19|      1.12|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer will be dereferenced                                    |  13|      0.77|
   |---------------------------------------------------------------------+----+----------|
   |Suspicious dereference of pointer before NULL check                  |  16|      0.95|
   |---------------------------------------------------------------------+----+----------|
   |Null pointer may be passed to function that may dereference it       |   1|      0.06|
   |---------------------------------------------------------------------+----+----------|
   |Suspicious dereference of pointer in function call before NULL check |   4|      0.24|
   |---------------------------------------------------------------------+----+----------|
   |Use of Freed Memory                                                  |   2|      0.12|
   +-------------------------------------------------------------------------------------+

   Potential security defects
   |---------------------------------------------------------------------+----+----------|
   |Buffer Overflow - Array Index Out of Bounds                          | 113|     96.58|
   |---------------------------------------------------------------------+----+----------|
   |Use of Unvalidated Integer as Array Index by Function Call           |   1|      0.85|
   |---------------------------------------------------------------------+----+----------|
   |Use of Unvalidated Integer in Loop Condition                         |   3|      2.56|
   +-------------------------------------------------------------------------------------+

   For the sake of convenience, I split the bugs above into a few categories:

   +-------------------------------------------------------------------------------+
   |                       Type of defects                       |Coverity|KlocWork|
   |-------------------------------------------------------------+--------+--------|
   |Resource leaks                                               |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |NULL dereferencing                                           |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Unused values/Unchecked returns                              |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Return locals                                                |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Security (buffer overflow/un-validated array's indeces/other)|   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Use after free                                               |   x    |   x    |
   |-------------------------------------------------------------+--------+--------|
   |Deadcode                                                     |   x    |        |
   |-------------------------------------------------------------+--------+--------|
   |Static overrun                                               |   x    |        |
   |-------------------------------------------------------------+--------+--------|
   |Passing by value                                             |   x    |        |
   +-------------------------------------------------------------------------------+

   Overall, KlocWork seems to find more potential defects, although it
   also reports a lot (>80%) of potentially bad, but just probable 
   defects, which requires more attention in post-analysis phases. I
   got some explanations from KlocWork's team, saying that all defects
   are falling into two different kinds: MUST and POSSIBLE. And
   checkers can be configured, so one might choose to turn off all
   POSSIBLE defects.


      Noise level/False positives

   According to the companies, their tools are having about the same
   levels of false positives, e.g. 10-15% respectively. However,
   Coverity claims that after some configurations, the level of false
   positives drops significantly down to 3-5%. Plus, Coverity has a
   way of modeling primitives' semantics which allows to adapt the
   analysis for some models, not supported out of the box.

   On a similar note, I got the following comment from KlocWork's
   team: "...Upon installation, however, Klocwork undergoes the same
   tuning process that our competitors do with their software. Our
   false positives rate drops significantly after Klocwork is tuned
   for your code...


      Analysis time

   +-------------------------+
   |Product |Time of analysis|
   |--------+----------------|
   |Coverity|        05:35:54|
   |--------+----------------|
   |KlocWork|        ~2 hours|
   +-------------------------+

   I have to note, however, that the KlocWork team was smarly utilizing the
   power of Sun's Z880 server by running the build and consequent analysis as
   six parallel processes. Therefore, I'm not sure what would be the time of
   Coverity's analysis, being ran in the same mode.

      Build techniques

   Coverity has a way of redefining chunks of analyzed code. E.g. one
   can specify macros to redefine something in the code or explicitly
   suppress analysis of some pieces. From the later explanation I
   gathered, that KlocWork has similar capabilities, with an extensive
   knowledgebase documentation on how to redefine how a function operates and
   even implement for third-party libraries for which no source code is
   available.

   Both tools are utilizing non-invasive build techniques. Instead of
   introducing any changes into the client's build environment, they
   are monitoring build processes. Coverity is redirecting any calls
   to the standard compilers to their own frontends to collect all
   needed statistics/info. As the result of the build, a normal code
   is produced along with all data, required for further analysis.

   KlocWork has a utility called kwinject. They also have other
   utilities and means that help to be more versatile when integrating
   into the diverse types of environments.

   Overall, both systems can be reasonably easy integrated into
   existing development/quality life cycles.

      Incremental analysis

   Both tools have an ability to do incremental code analysis. E.g. if
   only a few files were changed in the workspace, then only those
   files will be analyzed. Such consequent analysis has to take much
   less time.

   Overall, in both cases these can be use in daily development
   activities for pre-integration checks or for ad-hoc analysis of a
   new and existing code.

      Extention APIs

   Both tools have ways of accessing the internal data gathered at the
   stage of the analysis.  E.g. with provided C API, one can work
   directly with static callgraph and data flow information and create
   custom checkers.

   Overall, I think this is a very important thing for us as we need
   to think of Impact analysis technique, better than BSP (to know more about BSP please <a href="http://weblogs.java.net/blog/cos/archive/2006/03/java_quality_me_7.html">check this</a>.

      Platforms metric

   +------------------------------------------------------------------------+-----+
   |Product |Windows flavors|Linuses|Solaris|sparc|x86|64-bit|Java          |C/C++|
   |--------+---------------+-------+-------+-----+---+------+--------------+-----|
   |Coverity|       x       |   x   |   x   |  x  | x |  x   | coming soon  |  x  |
   |--------+---------------+-------+-------+-----+---+------+--------------+-----|
   |KlocWork|       x       |   x   |   x   |  x  | x |  x   |     x        |  x  |
   +------------------------------------------------------------------------+-----+

   Despite Java support's current absense, Coverity is coming up with
   it later in October, 2006.  We were invited to participate in the
   product's beta testing in a few weeks.


      User experience

   Coverity has a very convenient source code browser, which allows
   looking into any part of a product's code. The browser creates
   hyperlinks to all known types and structures. Also, it allows to
   generate some reports per build, find difference between two
   builds, et cetera.

   KlocWork UI has a very intensive set of features to hide unwanted
   defect records. They aren't deleted, but just aren't shown in the
   reports. There are 10 different sub-categories of the defects'
   severety taxonomy, which helps to see only groups of defects one
   wants to inspect. It also has an interface to produce custom
   reports on top of 10+ standard ones.

   Both tools have quite efficient help systems with a lot of typical
   defects, examples, and explanations.

   Overall, Coverity's UI is more ascetic and somehow slower, but more
   tuned for its purpose.

      Resource requirements

   Both tools are quite moderate in terms of hardware utilization and
   are doing pretty well on the V890 machine.

      Miscellanious features

   Coverity has a number of concurrency checks, e.g. double locks, a
   lock without freeing, a double free, and inconsistent local
   order.


	 Additional configurations are available for sleep functions
   and those that need to be holding an exclusive lock when
   called. Also, it supports different threading models.

   KlocWork mostly lacks concurrency analysis features and does only
   some very basic stuff in this area. I've been recently told by KlocWork's
   team that their latest version of the software (7.5) has a number of
   concurrent checkers implemented. Besides, it has some kind of CASE tool
   packaged along with a static analyzer, designated to help in finding and
   fixing the code design's flaws.  However, this topic is beyond this
   analysis' boundaries.

      Pricing models

   Coverity and KlocWork have similar pricing model, although they are
   slightly different in the overall costs. Basically, Coverity
   charges for volume of the code only; KlocWork.charges for this and
   for user licenses. The latter might cost you if you're planning to
   use for the relatively big team.

      More information

   could be found at http://www.coverity.com/ and
   http://www.klocwork.com/ and/or requested directly from their sales
   teams.

      Word of caution.

   All tests mentioned above were performed on the versions of the
   products, which were current when the article has been written,
   e.g. June-August 2006

</pre>

I wrote this article a couple of months back, however it took me awhile to get an agreement from all interested parties and make some corrections in my original text. Please feel free to let me know if you have any comments, ideas, suggestions, etc.
<p>
Take care,<br>
  Cos
<p>

P.S. The original formatting of the article was preserved, because java.net's blogging system is simply lacks of many essentials.]]>

</content>
</entry>
<entry>
<title>Open quality metrics and processes</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/09/open_quality_me_1.html" />
<modified>2006-09-19T00:44:50Z</modified>
<issued>2006-09-19T00:44:39Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.5418</id>
<created>2006-09-19T00:44:39Z</created>
<summary type="text/plain">How the Q&amp;A gets done in open (and closed) source community?</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Despite the differences between business models, development
processes, functional areas of the application, and the languages,
these applications are written in, the quality approaches for them are
quite similar and final goal is the same. The engineering team has to
deliver quality application to the end users.<p>

Let's retrospect how it gets addressed by different teams. What techniques and tools are used by a few of well
adopted systems in both open and close source communities (to get this
information together I was using different sources; mostly from
available on the Web such as application's web sites, pod casts, etc.)
Also, I have to apologize in advance as the information you'll see
below isn't fully normalized and I'd to spent more time for the pretty
formatting, but I guess it will illustrate my point. Also, the choice of application might seem a bit awkward for you, but I could've done it intentionally<p>

<b>Open SuSe Linux:</b>
<ul>
  <li>stress testing
  <li>reliability testing (similar to the above + some boundary testing)
  <li>scalability testing (behavior on NUMA, etc.)
  <li>Tools: home grown automation tools, bugzilla for defects tracking
  <li>source code is available through participation in OpenSuse community
  <li>Number of currently reported bugs: 1238 or so it appears
</ul>

<b>Mozilla:</b>
<ul>
  <li>has some smoke tests http://www.mozilla.org/quality/smoketests/
  <li>overall Mozilla has a number of QA teams (per functional area,
  actually) and usually they have some tests and testing specs being
  published, so contributors can participate. More information
  http://www.mozilla.org/quality/
  <li>Tools: mostly manual testing; obviously Bugzilla for defects
  tracking 
  <li>Number of currently reported bugs (I've looken into Firefox only 9237)
</ul>
  
<b>LTP:</b> Community quality project for POSIX UNIX's
<ul>
  <li>a few commercial vendors are contributing (SuSe, IBM, etc.)
  <li>participants from all over the place (23+ by now)
  <li>Tools: lcov, gcov
  <li>Number of currently reported bugs: the web-site doesn't have much
  <li>information on it. Or it perhaps, my inability to find relevant
    information.
  <li>also you might want to check this atricle with more detail about
    Linux reliability testing
    http://www-128.ibm.com/developerworks/linux/library/l-rel/
</ul>

<b>JetBrains (Intellij IDEA)</b>:<br>
despite the tool is commercial it seems to utilize quite smart
model of providing the quality of the product. It is manyfold:
<ul>
  <li>developers are using new features immediately; thus as soon as
    new feature is being developed it gets into work. So, the
    programmers are eating their own dog food and polish their product
    as it goes. Also, this allows to polish work flow which is quite
    noticeable as the IDEA's user interface is very intuitive and
    flawless
  <li>Intellij IDEA has a system of EAP (Early Access Program) which
    allow IDEA's enthusiasts to get new build every week and to start
    using fresh builds and find new bugs without delays. All
    developers are communicating with EAP's participants directly,
    what eliminates any communication hardies
  <li>as for pure QA, they simply didn't have any designated QA forces
    until a couple of weeks ago and all quality activities were
    covered by development
  <li>Tools: JUnit, however most of the tests aren't classic unit, but
    rather functional tests; home grown continuous integration system
    (http://www.jetbrains.com/teamcity); defects tracking is done
    through JIRA
    (information is gathered from an insider. Thanks to Max Shafirov
    for his help)
</ul>
 
<b>Sun JDK</b>:
<ul>
  <li>has a well developed set of test suites for both JDK and virtual machine
  <li>has designated QA team of 100+ people
  <li>Tools: home grown tools for test harnesses, task dispatching,
    results analysis
  <li>internal defects tracking with external gateway and Webbugs
    system, which accepts bugs submission along with some test cases
    being sent in from outside
  <li>Source code is available through dev.java.net portal. Bugs could
    be submitted externally, however code is under Sun's own source
    license
  <li>overall number of bugs submitted so far: <put it>
</ul>

<b>Kaffe</b>:
<ul>
  <li>free project with free time contributions from around the
  world. No designated QA forces. Most of the quality activities are
  done by developers themselves. However they have a few smart ways of
  proven their product to be stable and sound. Here are the main
  approaches:
  <ol>
    <li>continuous integration: Tinderbox on regular basis
    <li>sneaking into other's build to see if there's problems:
       http://buildd.debian.org/build.php?pkg=kaffe
    <li>some in-house configuration scripts around HP's testdrive site
       (http://www.testdrive.hp.com/current.shtml). One can ftp his
       stuff in, login in and run something, than ftp the results
       back.
    <li>Kaffe cross-compilations to CPUs supported by Debian, which
       helps to weed out the odd breakage every now and then, and in
       general is a good test for intrusive patches, as they tend to
       trip up a lot of compiler warnings on non-mainstream platforms
    <li>There is also an archive of weekly snapshots created by Kiyo
       Inaba at ftp://ricohgwy.ricoh.co.jp/pub/Lang/Java/Kaffe/ , so
       he regularly notices and catches any breakage that occurs on
       NetBSD.
   </ol>
  <li>on the side of industrial type of testing, Kaffe runs regression
    testing and Mauve test suite. Also EMMA based coverage is enforced
    for Mauve (http://builder.classpath.org/~cpdev/coverage/)
  <li>Kaffe folks mentioned that they would love to use some of Sun's
    test suites, and I didn't have any comments at the time of our
    email conversation
  <li>Dalibor.Topic made quite valuable observation in his email (cited
    below) " Besides changes to the core VM, it involves picking the
    right projects to cooperate with, and driving public attention
    towards them, so that all boats are lifted, rather then everyone
    having to develop their own set of insular functionality. So, we
    are in touch with distributions, and in close touch with various
    projects we use code from, and participate in their (QA) efforts,
    as well, and encourage people to do the same. A lot of the QA work
    comes from the integrators in distribution using Kaffe to deliver
    packaged applications and libraries to their users."
  <li>the information above has been provided by robilad@kaffe.org and
    jim@kaffe.org. Thanks a bunch, folks!
</ul>

<b>Eclipse</b>:
<ul>
  <li>I'm expecting to add more information about Eclipse as soon as
    I'll get it from the team.
  <li>For now I was able to find 22,000+ open bugs filed against the
    all components sub-projects
</ul>

As we can see from above, all these team are mostly using quite
traditional approaches to provide a quality of their software systems
and applications. I would call these traditional ways of quality
delivery extensive or brutal force, as they are involving more tests
to be written and executed and/or more developer's eye-rolling.<p>

However, technologies such BSP (<put link here>) and some other
concepts of impact analysis (<check if KlocWork has this one or put
some links>) allows to sharply increase ROI of a typical quality
organization. Implementations of the impact analysis might differ, but
the general idea might be well expressed as following a couple of
rules of thumb:
<ol>
  <li>find pieces of source code/modules/subsystems critical for your
     application
  <li>focus your quality efforts on those
</ol>

There is yet another way to somehow guarantee the quality of an
application. You can do constant static analysis of your code, which
will give you some sense of security too. I'll be writing about
different static analyzers next time. Stay tuned to see info about
Coverity, Klocwork, FindBugs and some others. There was an article
recently in one of online technology news sites. The article was
comparing Klocwork vs. Coverity. Frankly speaking, I read it twice and
didn't get a clue of what was the difference between those two :-)<p>

Another lesson I learned from the above observation, is that despite
of popularity of the platform quality processes might be done equally
or less successful, especially in the world of open source software.<p>

Anyways, it's being rather a long post. And I'd like to draw a
conclusion now.<p>

Sun JDK is getting to open source quite soon. Considering all examples
above I could tell, that Sun JDK's quality process might be
implemented in one of already known models. However, "already known"
doesn't mean "perfect" or "sufficient".<p>

Thus, I would love to hear your ideas, suggestions, comments,
outbursts, etc. Please post them here or just send me an email to
kboudnik@gmail.com]]>

</content>
</entry>
<entry>
<title>Are lawyers up to open source movement?</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/06/are_lawyers_up_1.html" />
<modified>2006-06-09T19:38:16Z</modified>
<issued>2006-06-09T19:35:01Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4997</id>
<created>2006-06-09T19:35:01Z</created>
<summary type="text/plain">A brief note about recent Debian and Sun Java distrubution news.</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>

<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[<p>Hello there.</p>

<p>Looks like the open source market is getting more and more hot nowaday. This recent thread on <a href="http://lists.debian.org/debian-devel/2006/06/msg00256.html">Debian's list</a><br />
shows that when development community is trying to get things easier around, someone suspects that a lawyers can be in a position to cause a harm.</p>

<p>From that tiny snippet<br />
"...If Sun would commit to putting that interpretation in their license, that would be one thing. But as an aside as it is, with no legal standing, who is to say that other lawyers or a judge wouldn't decide it means something entirely different?..."<br />
I conclude, that the old saying is still correct "The law is like a shaft: it will point wherever you can turn it" :-)</p>

<p>Pardon my ignorance, but it seems like freedom of open source and official rulings are still as good mix as water and oil. Public wants one thing, but the governing entities might know better what is the best for that very public.</p>

<p>Take care,<br />
  Cos</p>]]>

</content>
</entry>
<entry>
<title>Java is getting open. What about Java quality?</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/05/java_is_getting.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-05-20T08:13:01Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4857</id>
<created>2006-05-20T08:13:01Z</created>
<summary type="text/plain">The brief brain-storm about future of Java Quality tools and frameworks.</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hello there!<br><br>

Last week happened to be quite busy for many Java developers,
activists, and supporters. JavaOne 2006 conference had a lot of
interesting pods, booths, talks and other kinds of presentations. A
leading development companies were bringing their innovations to share
the knowledge and expertise in the field.  <br><br>

I won't reiterate the same things you perhaps heard already: you can
find them <a href="http://java.sun.com/javaone/sf/index.jsp">here</a>
or <a
href="http://blogs.sun.com/roller/page/vmrobot?entry=javaone_2006_%D0%B4%D0%B5%D0%BD%D1%8C_1">custom
one</a> from our SPB team (in Russian) <br><br>

Among other great <a
href="http://www.tmcnet.com/usubmit/2006/05/18/1655749.htm">things</a>
there was one which firstly hit a crowd of attendees of <a
href="http://www.eweek.com/article2/0,1759,1962100,00.asp?kc=EWRSS03119TX1K0000594">Netbeans
conference</a> (there were a few very interesting talks, especially on
my lately favorite topic - Java ME development and tools; I'll talk
about this more next time) and then it'd been brought to the wider
audience of JavaOne Day One's keynotes. Right, I'm talking about
bringing Java to <a
href="http://www.vnunet.com/vnunet/news/2156205/sun-promises-open-source-java">open source</a>. I think, that has been expected for a long time and I
guess that this will bring some new blood into Java platform around
the world.  <br><br>

Ok, it's all hunky dory and rosy then. But also it brings some
concerns. Java platform is big and complicated application. It has
core things, like VM, libraries, a platform depending code, backward
compatibility promises, etc. Besides, Java community has to make sure,
that the quality of the platform won't suffer from this move.
<br><br>

I'm a strong believer of the following approach: if a standardized
technology is getting available for a community, then the testing
methodology has to accompany it as well. The only requirement ought to
be attached: test suites should pass. This will help to keep proper
level of compatibility, avoid unnecessary branching, and let
participants to keep better grip over the development process. Doesn't
sound too open to you? Great, share your thoughts with us.
<br><br>

However, any testing methodology assumes some frameworks to execute
the tests. E.g. Junit is required for unit tests; JCK suite has to be
passed under JavaTest for a JDK's certification, etc. So, the same is
true in our case: we need to supply the testing frameworks.  <br><br>

It sounds like Sun has to bring some chunks of the quality
infrastructure along. Does anyone get surprised, that I'm talking
about <a
href="http://weblogs.java.net/blog/cos/archive/2005/09/java_quality_me.html">Java
quality</a> again :-)? Back <a
href="http://weblogs.java.net/blog/cos/archive/2005/10/java_qualitys_o.html">then</a>
I'd mentioned some choices of the testing tools. David Herron'd added
more details on <a
href="http://weblogs.java.net/blog/robogeek/archive/2005/11/the_quality_tea.html">that
topic</a>. As the major contributor
to the DTF for the last few years, I want to see the project moving
ahead and not getting accidently abandoned. This scheduling framework
can save a lot of valuable engineering time (thus money) for any java
development/testing team.  <br><br>

Another great tool, which will be nicely going along with DTF is our
internal test harness called Tonga. DTF and Tonga are getting together
very well. In fact, they were designed and implemented with quite an
awareness of each other. This couple makes a great yet very flexible
and efficient testing infrastructure.  <br><br>

Obviously, we have a lot of other interesting in-house solutions,
which, I'm sure, would be a real value-add for the community. For
instance, our Java coverage framework, creatively called jcov, test
results processing and reporting tools, et cetera.  <br><br>

I'm not a lawyer or any of the high-flying Java execs, so don't take
my word to a court :-) However, I'd be pushing for this tools open
sourcing approach enthusiastically and will keep you posted on the
progress.

<p>Take care,<br>
Cos]]>

</content>
</entry>
<entry>
<title>Lost in translation</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/05/lost_in_transla.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-05-12T23:48:16Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4721</id>
<created>2006-05-12T23:48:16Z</created>
<summary type="text/plain">Этот блог не имеет ничего общего с проблемами программирования и компьютерными технологиями. Это заметка о сложностях перевода</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>

<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[<p>Недавно разглядывал газетку "Известия" - была такая уважаемая в советские времена газета. Почему была? Да потому что уважаемой я ее назвать больше не могу. Причин тому было несколько, последняя переполнила чашу моего терпения...</p>

<p>В переводе статьи о прогнозах Уоррена Баффетта (http://www.izvestia.ru/economic/article3092694/) автор перевел вот такую фразу "...an industry leader in the metal cutting tools business..." вот таким вот образом "...компания по производству лезвий из тяжелых металлов для кухонной промышленности..."</p>

<p>Это будет покруче "pure virgin wool" как "шерсти непорочной девственицы"</p>

<p>Круто, японцы просто отдыхают</p>]]>

</content>
</entry>
<entry>
<title>Smaller Java, but bigger market?</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/04/smaller_java_bu.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-04-21T20:47:17Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4556</id>
<created>2006-04-21T20:47:17Z</created>
<summary type="text/plain">The post on recent CTIA event: why didn&apos;t we see a lot of Java producers presented?</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>J2ME</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hi there<br>

This time it isn't about Java quality. However, it's related to Java
:-) Well, there is at least some difference, right?

<p>

I'd visited <a href="http://www.ctiawireless.com/">CTIA conference</a>
recently. Actually, this was my first time experience with that kind
of technological forum ever. It's totally different from development
conferences - it's so about money. It stinks of money, if you will. It
seems like all these business people at pumping a lot of cash into
almost any wireless-like technology or software application. It's
like the nineties of last century: if you had a word 'Internet'
somewhere on your business plan - you're likely to get some crazy
funding.

<p>
Anyways, at that even in Las Vegas, NV, I saw a lot of interesting wireless stuff presented by respectable companies
like RIM (gosh, I love their phones!), Nokia, LG, Samsung, Motorolla, NTT Do-Co-Mo, and many
others. The software side was represented by a variety
of service providers and SDK makers like QualComm. Even Microsoft
wasn't ashamed to bring their killer PocketPC smartphones (nice name,
isn't it? Perhaps all other's are dumbphones) to the expo. And it is a killer by the way it assassinates  those small gizmos.

<p>

But what hit me a lot is that I didn't see the main player of the 
JavaME's field: Sun Microsystems didn't appear at the
CTIA. And believe me - I was looking hard!

<br>
The company, which had created the most popular software platform in
the world; powering as many as 2.5B Java devices around the globe had
only one session in all three days of second biggest wireless
technology conference ever.

<br><br> The session was about
<a href="http://www.netbeans.org/kb/50/nbmobilityday.html">NetBeans
Mobility 5.0</a> However, it was taken away to the Harrah's casino (perhaps, hidden from
Microsoft's corporate spies or something): neither Sun Microsystems
nor NetBeans brands were listed in the conference catalog; neither
NetBeans or Sun's info hasn't been posted on the CTIA web site; clerks
at the info booths weren't aware about such a pod at
all.

<p> To give you some comparison ground: QualComm is supporting about
40+ OEM manufacturers. This gives us roughly 600 headset models
(according to one of their marketing folks). And they had
a big setup, a special all day long development seminar, etc. One more fact: there are as many as 200+ mil. of cell.phones owners in the US. So, do your math...

<p>So, I'm kinda wondering what is the message JavaME community will be
getting by this de-marsh? Does it simply mean, that Sun Microsystems
starts paying more and more attention to the mobility tools
development or something else?

<p>But on a bright note, I'd like to mention, that a 
number of software development companies are still looking toward
JavaME platform. I was talking to a few folks from that side of the
fence, and they seem to be quite excited about what they are able to
do with JavaME now. A particular interest is coming from wireless
social networking developers - Java platform is greatly simplifying
the user experience and reducing the development cycle for them.

<p>By the way, the wireless social networking is blooming nowadays. I met a very promising and proactive group of folks from <a
href="http://www.vcellvibes.com/">VCEL, Inc.</a> who did a very first 
truly wireless social networking platform in this country. And it's all using JavaME
(well, they've might be using Java for their server side too, instead
of scripting :-). They are supporting about 20+ MIDP2.0 headsets from
different manufacturers and providing a whole new way into people social
communication. It's so cool! And I'll write more on this particular
topic later...

<p>See yall soon,<br>Cos]]>

</content>
</entry>
<entry>
<title>New Java VM blog is coming...</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/03/new_java_vm_blo.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-03-31T18:04:37Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4404</id>
<created>2006-03-31T18:04:37Z</created>
<summary type="text/plain">Announcement about new Russian Java VM blog </summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Community</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hi there<p>

Again a message mostly for those, who is speaking or at least reading
Russian.

Our folks from Saint-Petersburg <a
href="http://weblogs.java.net/blog/cos/archive/2005/11/java_quality_me_4.html">I
was writing about this great city before</a> SQE team are starting
their own blog, designated for the Russia-speaking development crowd.<p>

The blog, I believe, will be a real first hand's value-add for
existing Java VM resources, about modern VM techniques, tools,
security features, co-existence with .NET, etc.<p>

There isn't much right now, but the folks will carry on fast, I think.<p>
You can find this new information resource at
<a href="http://blogs.sun.com/roller/page/vmrobot">at the official Sun
blogging site</a><p>

So, if you have colleagues or friends, who are suffering without such
information, written in Russian - please pass by my message or simply
the URL above.

Let's wish them good luck and God speed!<p>

--<br>
Thanks<br>
Cos]]>

</content>
</entry>
<entry>
<title>Java APIs comparison</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/03/java_apis_compa.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-03-29T05:57:37Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4400</id>
<created>2006-03-29T05:57:37Z</created>
<summary type="text/plain">Interesting approach of comparing two sets of APIs, e.g. JDK1.5 and JDK1.6</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hello folks<p>

We recently thought about an interesting approach of finding a
difference between APIs (public methods) of two versions of a Java
software. I was doing it for JDK1.5
(Tiger) and JDK1.6 (Mustang b76).
<p>

Sometime you might want to take a quick look and find out what exactly
has been changed in your (or somebody's else code) between two
versions. Why? Well, if you want to focus your test development on the
new API only it might be a very sound idea. I'd spoke about
some of other methods to help you to <a
href="http://weblogs.java.net/blog/cos/archive/2006/03/java_quality_me_7.html">achieve
better software quality</a> <br>

Well, there are some methods to do so. E.g. you can use "Since" <a
href="http://dsg.port.ac.uk/~mjeg/resources/javadiff/index.php">tags
from a Javadoc</a> Well, good if you have any...<p>

Or you can store your product API's snapshot like RefactorIt does. Boy, what if you didn't create this snapshot in a first
place and now you badly need one? "Sorry partner - no can do for ya..."<p>

Anyways, the idea was on the surface as usual:
<br> In one way or another
build a list of all public (protected/private/static/etc.) methods in
your software. You can use <a
href="http://www.gnu.org/software/grep/">grep</a> or <a
href="http://java.sun.com/j2se/1.5.0/docs/tooldocs/solaris/javap.html">javap</a>
along with grep for these purposes. Don't forget to do this for both
versions of your application - it's about the difference, right? :-)<p>

Now, you have these API lists, you just need to compare them. And again
it's up to you: you can use Unix diff command, Emacs ediff, or
vimdiff. I'd wrote that self-explanatory Perl script (feel free to
modify it, criticize it, or merely throw it away). Fix &open_file()'s loop if your API lists weren't created in the form of "filename:method signature" per line...

<pre>
#!/usr/bin/perl

if ($#ARGV != 1) {
    print "Please supply two lists of APIs you want to compare\n";
    exit 1;
}

my %diff = ();
my $left = &open_file ($ARGV[0]);  #Presumably, the earlier version
my $right = &open_file ($ARGV[1]); #Presumably, the latest version

foreach $file (keys %$right) {
    if (!exists $left->{$file}) {
	$diff{$file} =  \@{ $right->{$file} } ;
    } else {
	@l_stuff = @{ $left->{$file} };
	@r_stuff = @{ $right->{$file} };
	foreach $r ( @r_stuff )  {
	    if (&is_there($r, @l_stuff) != 1) {
		push @{ $diff{$file}}, $r;
	    }
	}
    }
}

foreach $d (keys %diff) {
    @extract = @{ $diff{$d} };
    print "$d\n   @extract\n";
}

#=========== Some subs
sub is_there () {
    my ($search, @list) = @_;

    for (@list) {
	return 1 if $_ eq $search;
    }
    return 0;
} 

sub open_file () {
    local ($name) = @_;
    open FILE, $name or
        die "Can't open specified file!$name\n";
    my %ret = ();
    
    while (<FILE>) {
        ($key, $value) = split(':', $_);
        push @{ $ret{$key} }, $value;
    }
    return \%ret;
}

</pre>

Of course, there's always an expert opinion. Or you can ask an author of an application, or eye-roll it yourself. But isn't that much simpler to load your computer with the stuff it can do better then a human being?<br>
<br>So, I'll let you to carry on from here...
<p>

CU,<br>
&#09;Cos]]>

</content>
</entry>
<entry>
<title>Java. Quality. Metrics (part 6)</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/03/java_quality_me_7.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-03-16T01:20:07Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4317</id>
<created>2006-03-16T01:20:07Z</created>
<summary type="text/plain">Talking about bugs prediction technology</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hi there.
<br><br>
In this short article I'll try to summarize what I was discussing for
the last couple of months.
<br><br>
So, let's briefly list key factors that are likely to affect our
judgment of software quality.

  - our code quality expectations (good enough quality, remember?)
  - coverage isn't everything
  - code complexity and a frequency of the changes
  - number of bugs filed against source code modules/files
  - testing methodologies
<br><br>
Alas, the last one doesn't sound like a beast, it might reduce the
effectiveness of defects discovery rate a lot. Obviously, it is a
choice of approaches of test failures analysis. The bulky one with a
weak algorithm of false positives detection pisses off engineers and
they begin ignoring most possibly important warnings and
notifications.
<br><br>
Anyways, I want to talk about a combination of the first three bullets
above.
<br><br>
About a year ago, a few of Sun's fellas were chatting about simpler
ways of delivering a better code. Static analysis and variety of
testing approaches were among the things on the table. At some point,
the bright idea of mixing both of those and adding some other flavors
had appeared. Afterwards, we came up with what was called Buggy Spots
Prediction (BSP).
<br><br>
The idea itself is as follows:
<ul>
<li>
  we're creating a static call graph (CFG) for any given source
    code, using a commercial or home-grown tools
</li><li>
  having this, we can calculate a few things about this graph,
    e.g. the frequency of calls to any particular method; the
    frequency of calls from a method to other methods with-in the
    code; basic-block based complexity of a code, etc. (currently, we
    calculate about five or seven of them, i.e. coverage per function,
    basic block per function, et cetera)
</li><li>
  when executing tests against the instrumented build, we can
    prepare a code coverage metric for it
</li><li>
  combine these two lists of modules - from CFG and from code
    coverage runs - by module names
</li><li>
  sort the resulting list ascending by in-call frequency and
    descending by coverage scores
</li><li>
  let's assume that most frequently called functions are, perhaps,
    most important from the quality standpoint. Well, their code is
    called more often, so any problems will immediately affect
    a top-level or at least quite important functionality.
</li><li>
  if such methods are having low coverage numbers and high
    complexity or high number of reported bugs, then it might be a
    good indication that the code has to be targeted by quality
    engineers and/or developers.
</li></ul>

So, all that is giving you a way of quickly selecting possibly buggy
spots, e.g. the pieces of the source code which are likely to become a
root of defects found by your customers. Why? Well, simply
because of the fact that the coverage is low in these areas and
existing tests aren't guaranteed an acceptable level of quality.
<br><br>
As any heuristic approach, this one might produce incorrect
results. However, our preliminary predictions are quite coherent to
the fact that most of externally reported defects were found in the
poorly covered but frequently called methods.
<br><br>
In organization with limited QE resource, a manager might want to
firstly address such hot spots. This will help achieve a good-enough
quality level and then concentrate on less important issues.
<br><br>
Yet another benefit is that the technique is a language
independent. Once you'll build a universal presentation for CFG and
code coverage information, you can use the same engine to measure
Java, C++, and programs written in other languages.
<br><br>
And, of course, our methodology doesn't replace a human knowledge of
the importance of product features. It helps engineers see a valuable
projection of static-to-runtime boundaries and helps focus on some
aspects of that complex matter.
<br><br>

And I just want to remind to you about Project Mustang (Java6)
Regressions Challenge. Please check <a
href="http://download.java.net/jdk/Mustang-Regressions.html">here for
more</a> details.
<br><br>
CU,<br>
Cos]]>

</content>
</entry>
<entry>
<title>Hidden problems of the blogging</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/03/blogging_hidden.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-03-06T18:34:29Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4256</id>
<created>2006-03-06T18:34:29Z</created>
<summary type="text/plain">About quality of blogs preparation process</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Tools</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hi there.
<br><br>
It's rather short message about a potential problem you might face as
well. 
<br><br>
Recently, I've been told that few of my blogs are having incorrect
links to other resources. Being very careful about posting any
materials, I was quite surprised and went to
http://weblogs.java.net/blog/cos/ to check it out.
<br><br>
In a short time I realized that a couple of the articles were typed
using <a
href="http://www.sun.com/software/star/staroffice/index.jsp">StarOffce's
document writer</a> and were saved in OpenDocument Test (ODT)
format. As the result, all quotas - &quot; - were replaced by fancy
typographical quota symbols. When they became a part of an URL they
were encoded into something like <pre>%E2%80%9C</pre> for left &quot;
and <pre>%E2%80%9D</pre> for the right one
<br><br>
That is well know fact, but is a good illustration of what kind of
weird bugs you might discover sometimes during your quality cycles.
<br><br>
Cheers,<br>
  Cos]]>

</content>
</entry>
<entry>
<title>Mustang (Java6) Beta is online now!</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2006/02/mustang_java6_b_1.html" />
<modified>2006-06-09T00:45:50Z</modified>
<issued>2006-02-15T20:13:04Z</issued>
<id>tag:weblogs.java.net,2006:/blog/cos/277.4127</id>
<created>2006-02-15T20:13:04Z</created>
<summary type="text/plain">Beta milestone of new Java6 platform has been release!</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>J2SE</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Whoa! Mustang is there! The long awaited, butts kicking Beta release of Java platforms is ready for downloads! And I'd like to join my colleagues and say a couple of words about this milestone. It's not something totally new for most of you. Everyone had a chance to take a look at <a href="https://mustang.dev.java.net/">snapshot releases</a> before. <br>

Feel free to grab it here:<br>
<a href="http://java.sun.com/javase/6">the main page of the Beta release</a>
<a href="http://java.sun.com/javase/6/download.jsp">if you want to just downloaded it</a><br>
Please refer to <a href="http://weblogs.java.net/blog/mreinhold/archive/2006/02/mustang_beta_bl.html">Mustang Beta Carnival</a> to read reviews of new features and improvements included in this release.<br><br>

And on my usual favorite note, I'd like to mention that our engineering and quality teams had made an every effort to guarantee the high-level quality of this Java release. Also, a number of new approached and techniques were proved to be efficient for better software quality development. I mentioned a few of them already and plan to shed some light on this topic soon.<br>

And I'm honored to tell you here, that Sun's stables keep only high-quality racer :-), like this very Mustang, we're presenting today for your judgment. <br><br>
Enjoy!<br>
  Cos
]]>

</content>
</entry>
<entry>
<title>Java. Quality. Metrics (part 5)</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2005/12/java_quality_me_6.html" />
<modified>2008-01-02T17:42:16Z</modified>
<issued>2005-12-24T01:55:46Z</issued>
<id>tag:weblogs.java.net,2005:/blog/cos/277.3824</id>
<created>2005-12-24T01:55:46Z</created>
<summary type="text/plain">A introduction to a cool technology we had developed with my colleagues. It can do a really good job, predicting buggy spots in your source code.</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hi there again.<br><br>

Getting back to my favorite topic about quality of life... I meant that pseudo-life, we all are trying to make. And if there's Something, what had once created all of us and everything else around - it did a way better job :-( But, I think, we have deserved a credit too: we don't have all the time in the eternity to finish our job by trial and errors approach. An average man leaves about 70 years; say 30 years of which she spent in diapers, at school or on medications in front of TV, purchased on retirement money. Which left us with roughly a 40 years term, split between personal and professional life (change the order if you want to :-)<br><br>

All my professional life is literally coupled with computers (I have six of them at home, including an old <a href="http://www.boudnik.org/~cos/images/lx.jpg">Sun's SparcClassic LX</a> and not including my Palm Pilot). And as more and more software is coming into our lives, I'm wondering if I can rely on it and where are the limits of this trust. Also, as a participant of Java software development cycle I want to do better job myself and help my colleagues to do same: namely spend less time to develop more robust software.<br><br>

This brought me to the point of quality indicators definition. What will be the right criteria to determine if a piece of software will break or not in customers hands. And <a href="http://weblogs.java.net/blog/cos/archive/2005/10/java_quality_me_2.html">again</a>, where is the right target to aim our testing efforts and wether or not they are efficient?<br><br>

After some consideration we humbly came up with the following list:
<ul>
<li>code complexity? Yes, this is the beast. If you have methods as long as 3+ screens it's bad. If you class hierarchy contains 5+ levels – it's not good either. Does it smell right, if you have 300+ lines of code/class?</li>
<li>a frequent code changes? Or even changes per feature or bug? Yeah, perhaps. If you do have about 5 commits per bugfix it doesn't sound right at all. However, I can imagine some other considerations.</li>
<li>poor code coverage? Yes, one of the kind. But even <a href="http://weblogs.java.net/blog/cos/archive/2005/11/java_quality_me_4.html">coverage numbers</a> can't save you from troubles. It isn't a panacea.</li>
<li>design and coding styles? Well, may be. Stylish code is easier to read, understand, but not necessarily to maintain. And if coding styles are easy to check, design style's flaws might not be that easy to find and eliminate. Patterns are helping here in particular, but you have to know where is the fine line between good design solution and simple yet sufficient implementation</li>
<li>code duplication? Not in a sense of reuse, but in a way of overuse of copy-n-paste "technique". Yes, ugly enough. As one of my readers had appointed - if one had introduced a bug in a code, which was copied a few time later then one will end up with a multiple copies of the same bug hidden all over the place :-( There's a number of automated ways to find and fix these rotten spots. One of them named PMD was mentioned by another reader of the article and can be <a href="http://pmd.sourceforge.net/">found here</a></li>
<li><i>Thanks to all of you sending comments or expressing interest otherwise. If you can imagine or know more of these from your practice – please send them in and I'll gladly add here any reasonable ones.</i></li>
</ul>

Well, assuming that now you got a data for all from above and then some. What and how you'll deduce from here? <br><br>

Let's talk about this after I get back from that nice winter break. How about open sourcing some ideas? Ok, I'll share (swear!) the principles of the technology we're working on for last few months.

<br><br>
Best of everything to all of you and Happy New Year to everybody. Also, Merry Christmas to those who believe in Santa Clause or whatever you can call that dude, who knew something about quality in a long term!
<br><br>
Cheers!<br>
  Cos
]]>

</content>
</entry>
<entry>
<title>Preparing your tests</title>
<link rel="alternate" type="text/html" href="http://weblogs.java.net/blog/cos/archive/2005/12/preparing_your_1.html" />
<modified>2008-01-02T17:42:16Z</modified>
<issued>2005-12-19T08:54:21Z</issued>
<id>tag:weblogs.java.net,2005:/blog/cos/277.3821</id>
<created>2005-12-19T08:54:21Z</created>
<summary type="text/plain">This topic is talking about an approach of test base preparations: building libraries, getting binary bits together, etc.</summary>
<author>
<name>cos</name>

<email>Konstantin.Boudnik@Sun.COM</email>
</author>
<dc:subject>Testing</dc:subject>
<content type="text/html" mode="escaped" xml:lang="en" xml:base="http://weblogs.java.net/blog/cos/">
<![CDATA[Hmm, I just realized, that it's been a while since my last posting here. So, hi there! And for a difference, this time it isn't about java quality :-)
<br><br>
Last month I was pretty much busy with our testbase build automation. It's not a secret though, that our server VM testbase has a lot of native test code. As David had 
<a href="http://weblogs.java.net/blog/robogeek/archive/2005/11/the_quality_tea.html">mentioned recently</a> 
we have quite a handful of platforms to take care of and it's getting even more troublesome when you need to carefully lay out a lot of native bits.
<br><br>
So, here some insider's information, and please don't tell any one where you got it: we need to build the native pieces for as much as eight platforms (I might've missed a one or two), including Solaris/Linux/Windows on AMD64, of course. Most of native build's pieces are getting built on a separate machines, e.g. you can't build Solaris for Sparc and Solaris for x86 on the same box, can ya? Well, technically speaking, you can do cross compilation in most of cases, but we aren't that limited with hardware resources. Any way, doing this kind of drill manually might be a boring, not cost-effective effort and one can definitely spend all this time for  something more 
<a href="http://nar.oxfordjournals.org/cgi/content/full/32/20/e162">interesting</a>
<br><br>
So, we decided to automate out build process. The whole thing was divided into two pretty isolated pieces: writing Ant managed build mechanism and setting up a tiny run-n-watch framework to control as many as 6-8 remote processes, spawned in a certain order. The Ant part has been taken care off by our genius engineers (sic! I mean it) in Saint-Petersburg office and I handled relatively simple execution part.
<br><br>
We do use two different distributed execution systems in our daily business: <a href="http://weblogs.java.net/blog/robogeek/archive/2005/11/a_patent_for_th.html">DTF</a> and <a href="http://weblogs.java.net/blog/cos/archive/2005/10/java_qualitys_o.html">Grid engine</a>. Latter takes care about Unix's and former handles Windows, because it's pure Java and doesn't care on what hardware it's running. As the result, I had to create two different sets of starters and watch-dogs to manage our build processes in a slightly different manner.
<br><br>
And I was kind of surprised, that I can write distributed watch dogs, using nothing but Bourn shell (yes, the matter of fact, you can actually program on it and create clear and maintainable code. Wow!). Of course, there's no inter-processes communication or multi-threading. I didn't use anything else more fancy than file-locking (I know, I know – I hate this myself). But it was sort of fun to do this. Overall, the watch dogs' system is quite simple:
<br>
<br> - there's a single driver script, which knows everything about build logic and sequences; takes care about any clinches or time out issues
<br> - it sends a set of smaller platform specific scripts to Grid'ed and DTF'ed machines
<br> - sets a number of watch-dog monitors to trace build progress on every platform
<br> - platform specific build drivers do an actually Ant build and are sending notification to the central Tinderbox-like monitor. This helps to ease visualized human control over the process.
<br> - top level script is rolling in a big loop until for all process to finish
<br> - upon the build is successfully completed, the main driver does all necessary adjustments for the fresh bits and run baseline testing against this new testbase. Voilà!
<br><br>

That was my pretty much first real experience of using Grid and it unveils some pit falls, perhaps (or it may be me knowing just too less about this system). For instance, you can't rely on relative paths in your application, because Grid copies a submitted shell script somewhere and executes it from there. Eventually, what was `dirname $0`/../logs/mylog when you were developing your script, will lead to nowhere.
<br>
Two other things: setting an 32- and 64-bits environment on Windows platform from a shell script. It took me a while to carefully write a shell equivalent of SetEnv.Cmd batch file, which a Visual Studio supplies.<br>
And I had to find a way around the fact that Linux's automounter sometimes has troubles mounting NFS' auto-maps.
<br><br>
However, on a pleasant note: I had spent a good chunk of my professional time between 2001 and 2004 developing DTF and now it's paying back :-) by nicely doing some leg work for me.
<br><br>
So, yesterday I was able to build new Server VM testbase out of the box without much of my involvement (still had to get done a couple of quick fixes, simply because I was lazy enough no to do any dry-runs beforehand). Hopefully, next time it will cost me as much as a single keystroke to start that main driver script.
<br><br>
What will be a conclusion, you might ask? Hmm, I don't know – I just liked the stuff and the way of getting complex things done with a set simple techniques and tricks :-)
<br><br>
Hope to see yall soon again,<br>
  Cos]]>

</content>
</entry>

</feed>