Skip to main content

Samplr: embedding the power of Visual VM in your application

Posted by jjviana on August 6, 2012 at 9:55 AM PDT

In this post I introduce Samplr: an open source, intelligent sampling profiler that can be embedded in any Java application for automatic identification of performance bottlenecks

Tools of the Trade

If you read the Developer Power story in the July/August issue of Oracle Java Magazine, you already know that I love VisualVM. It is the best tool I know for inspecting the inner workings of a running Java system.

The feature I use most in VisualVM is the sampling profiler. This is a very non-intrusive, easy to use profiler that works by taking regular samples of JVM running thread stacks and aggregating the results over time. Attaching this profiler to a live JVM is a quick and effective way to pinpoint hot spots in code. The sampling profiler is lightweight enough that it can be safely attached even to production systems.

Visual VM
VisualVM Sampler snapshot view

Working as a consultant, I have had numerous opportunities to use the VisualVM sampler to diagnose performance problems in live applications. When faced with a performance issue my usual approach is to try to reproduce the problem in a controlled test environment, and then use VisualVM sampling to locate the performance bottlenecks in code.

This approach does not always work. Sometimes the right kind of test environment is not available. Other times the test environment is available but its not an exact replica of the production system and hece the problem does not happen there. And even with rhe right kind of environment sometimes the conditions that trigger the problem in production are not fully known.But since the sampling profiler is so lightweight, it is often possible to just connect it to a live production system and observe its behaviour firsthand.

Sampling a production system is not devoid of challenges: it requires de cooperation of Ops people (whom are not always easily convinced that sampling is safe in production environments) and lots of patience as one have to sort through a lot of thread stack information in order to locate the threads processing the relevant requests.And if your production system is clustered (which is often true) it is not practical to connect VisualVM to every single cluster node.

After some time using this great tool, I started to wonder: wouldn't it be nice if it was possible to show the sampling profiler the location of request boundaries both in space (code locations) and time (begin and end) and then have it sample just the right threads at the right times? It would be even nicer if this functionality could be somehow embedded in an application so that slow requests would be automatically recognized and sampled in production system without the need of Ops help or manual intervention.

A few weeks ago I stopped daydreaming and created Samplr, an open source sampling profiler based on and compatible with VisualVM that can be easily embedded into any Java application.

Samplr architecture

Samplr is designed to be included as a library in any kind of Java application. Once initialized, it starts a supervisory thread that is notified by applications when requests start and when they finish. These notifications associate the request with a specific thread and time slot.

The following sequence diagram depicts the interaction between the Samplr threads and the request processing threads:

Samplr sequence diagram
Samplr example sequence

Once the monitoring thread identifies that a request is interesting, it wakes up a sampling thread that will take regular snapshots of the target thread stack. The definition of an "interesting" request depends upon how Samplr is configured. In the example above, Samplr is configured to sample only slow requests (requests that take longer than a predefined time threshold to complete).

Using Samplr

Samplr is distributed as a self contained JAR file which should be incorporated into the target application using the standard library inclusion methods.

The application then needs to initialize Samplr by creating an instance of RequestManager. There needs to be just one instance of this class per application, and it should be initialized with a set of parameters that tells it what requests to look for, how to sample them and how to store the results. An example initialization is provided below:

The code snippet above shows Samplr being initialized with instructions to sample requests that take longer than 30s (30000ms). It will record its results in the /home/glassfish/samplr-output directory. It will sample requests until they finish or for 3 minutes (300000ms), whatever happens first. More initialization examples can be found on the class RequestManagerTest.java.

Once RequestManager is initialized it needs to be notified about request boundaries (start an finish). Requests are represented by the Request class and can be anything: a method call, a web request being processed by a servlet or any other unit of work. You can subclass Request in order to provide contextual information that needs to be recorded alongside the thread sampling in order to help debugging. An example is provided below:

Notifying Samplr about request boundaries is very simple: just call RequestManager.requestStarting and RequestManager.requestFinished at the appropriate code locations and Samplr will do the rest:

Collecting and Analyzing results

Samplr currently supports saving results to a filesystem. Once the sampling finishes, the request information and sampling results are saved into the configured output directory. Each request is saved in a unique sub-directory with the structure depicted below:

samplr output

The most interesting file is request-sampling.nps: this file contains the saved thread sampling information in nps format. This file can be loaded and analyzed in either VisualVM or in the Netbeans IDE. The resulting visualization can be seen in the following screenshot:

Samplr nps files contain information only for a single request, which was flagged for sampling by the initialization criteria. This makes analyzing Samplr results much easier than analyzing VisualVM results, as all irrelevant information is already filtered out.

The files request-info.txt and sampling-info.txt contain contextual information about the sampled request that can further help understanding the results.

Current status and future directions

Samplr is very young, but is already being used in one production system, with great results. It has been designed for extensibility and has a lot of room for improvement. A promising improvement area is request boundary demarcation: currently request boundaries must be programmatically demarcated. Here are some ideas I plan to implement eventually in order to make using Samplr easier:

  • A Servlet Filter that automatically initializes Samplr and register requests, enabling automatic profiling for slow web requests.
  • An EJB 3.1 interceptor to do the same for EJB method invocations
  • A Java agent for transparent integration into running systems through bytecode instrumentation.

Samplr is free and open source. You are welcome to try it out and also to help make it better.

AttachmentSize
Samplr_sequence_diagram.png26.34 KB
samplr_output.png106.47 KB
samplr_jvisualvm.png144.86 KB
Related Topics >>

Comments

Hi, I really like your idea of in app profiling!!! I ...

Hi,

I really like your idea of in app profiling!!!
I had some trouble of compilning my application with javac (ant task): as soon your jar file was in its classpath, the compiler stopped after some seconds without any error message... and without generating any class files...

I got this solved by creating a new jar our of yours (only for the classpath while compining) which contains only your classes, and nothing from the netbean stuff you boundled in.

I had some ideas to make SAMPLR even better for desktop applications:
- make definitions for timing constants specific to individual requests (I have some requests, which should only take ms, and others (as application launch) could easily take 20 seconds...
- I like to generate the output folders only, if the sampling really has taken place, and not for EVERY request which was in tolerance
- If I restart the application, he should just continue with sampling at the next number.... or maybe something like "log rotate" would be nice... only keeping last N logs, and not all of them?
- I maybe like to add something like a maximal number of logs for every request. Normally it would be enough to have lets say 10 samplings from a slow request... having 10000 would not improve anything...

Are you open to collaborate on this?

Best Regards

-Marco

Hi Marco, Thanks for your feedback! Your ideas are very ...

Hi Marco,

Thanks for your feedback!
Your ideas are very interesting, and I think the code is modular enough that they can be easily implemented.

I have not had time to work on Samplr for a while.
If you want to contribute, please feel free to fork the project on GitHub:

https://github.com/jjviana/samplr

- Juliano

Nice article and cool apps. I will hands on and give you the ...

Nice article and cool apps. I will hands on and give you the feedback

Nice. Although it seems primarily for running times of apps, ...

Nice. Although it seems primarily for running times of apps, this Performance Monitor, Parfait takes measuring a bit further. Largely thanks to JSR-275 support, allowing globally unique and precise representation of the measured values across unit systems.