I've spent the past week reaquainting myself with JMeter to do some quick-and-dirty load testing for one of our web-based applications. Since I want to simulate many simultaneous users, I want to parameterize my test script to provide different login values for each user. One way of doing this with JMeter is to add a “User Parameters” element to my “Thread Group” and use the ”_StringFromFile” method to initialize variables from a file.

I found a few examples on how to use _StringFromFile by doing a Google search, including one from the JMeter Wiki, but for the life of me I just couldn’t get it to work right. When I ran my test, it appeared that the first line of my file was being repeatedly read rather then iterating through the file as advertised. After taking a nice long Thanksgiving break from this problem, I went back to Google and searched for some aid. What I found was promising; on Thu, 20 Mar 2003 a bug had been logged against the _StringFromFile method in JMeter:

The StringFromFile function only returns the first record from the file. This is because the file is (unintentionally) opened every time the function is called.

Patch follows which:

• checks if file is already open
• allows variable name to be omitted
• sets up variables in setParameters(), rather than execute()

Sure enough, when I examined the JMeter log file for my test script, I saw the following:

12/01/2003 10:19:00 AM INFO  - jmeter.engine: Running the test! 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Opened Data.txt 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Variable name: StringFromFile_ 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Opened Data.txt 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Variable name: StringFromFile_ 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Opened Data.txt 
12/01/2003 10:19:00 AM INFO  - jmeter.functions.StringFromFile: Variable name: StringFromFile_ 

My data file is being opened each time a variable is initialized, rather then each time the test is run. I am not crazy or inept. There actually is a bug in JMeter (gasp!).

The good news is that I was able to locate the patch referenced in the bug report, and I should be able to build a local copy of JMeter for my own use.

The bad news is that the _StringFromFile patch was submitted on March 21st, 2003, and as of today, December 1st, 2003, the bug still exists in both the latest release build (Jakarta JMeter 1.9.1) and the nightly build of JMeter. Perhaps the patch did not get applied to the source (or it was applied and overwritten later), or perhaps the fix doesn’t work (although it looks good to me).

So here’s my question: “Are Open Source tools worth the price?”

1. I’ve spent about a day and a half trying to use a documented function, only to learn that there is a known defect with that function
2. The “shipping product” contains the bug, even though the patch for the bug was available months ago
3. I have been able to locate a fix for the problem, and with some additional effort, perhaps a day or so, I should be able to build a version of JMeter that will meet my needs
4. In the process of tracking down this defect, I have learned a great deal about the tool which will help me use it much more effectively in the future, and it’s given me some pretty darn good ideas about enhancements that I’d like to add

Item number 3 is a key decision point for me. With Open Source, once a fix is known you can find it, and with some effort you can fix the tool to the point where it is useful for you. This is not the case with Closed Source products.

Item number 4 is also “pretty darn important” in my opinion. With many Open Source tools, extendibility is the point. Once you learn enough about a tool you will probably be able to make it even more useful for your particular circumstance. This will certainly cost your organization time and money, but if the tool is something that will really prove useful for your development process, then it's probably worth the cost to learn the internal details.

In my case, JMeter is definitely worth the price.