Volker Simonis's Blog

Community: NetBeans Archives

HotSpot development on Linux with NetBeans - Part 2

Here comes the second part of "HotSpot development on Linux with NetBeans". While the first part focused on building and running the different flavors (opt/debug, client/server JIT compiler, template/C++ interpreter) of the HotSpot VM on Linux/x86, this second part concludes with a short evaluation of NetBeans 6.0 as an development environment for HotSpot hacking.

Now that we've successfully built the OpenJDK and know how we can efficiently build and run the HotSpot, we are ready to start using NetBeans (NB for short) for HotSpot development. Be sure to use at least version 6.0 if you want to do C/C++ development because starting with 6.0, NetBeans comes with builtin and highly improved support for C/C++ projects that supersedes the old "cpp" and "cpplite" modules which formed the NetBeans C/C++ Developer Pack (CND) in the past. There's now a special C/C++ Bundle available for download, that's only 11mb in size and contains all the needed plugins for C/C++ development.

After installation, be sure you install the Mercurial plugin by selecting Tools ->Plugins ->Available Plugins ->Mercurial. You need to have Mercurial installed and available in your PATH for the Mercurial plugin to work (see Prerequisites - Mercurial). If your Mercurial executable isn't installed in a standard path or if you want to use a special Mercurial version with the NB Mercurial plugin you can configure this under Tools ->Options ->Versioning ->Mercurial ->Mercurial Executable Path.

You should note that if you are using NetBeans remotely (starting it on a remote host and setting the DISPLAY variable to your local machine), you'll probably want to set "-Dsun.java2d.pmoffscreen=false" in the NetBeans configuration file <NB_directory>/etc/netbeans.conf. Without this setting, NB was so slow for me that it was effectively unusable (e.g. opening the "File" menu took about 20 seconds - every time!).

The nice thing about using NB for HotSpot development is that it has builtin Mercurial support, so we don't need to use additional command line tools for version control. Although we could start right away with the available HotSpot sources from the previously cloned OpenJDK (see Cloning the OpenJDK sources) and have Mercurial support for it, we will instead clone a brand new HotSpot repository from within NB to work with.

To achieve this we select Versioning ->Mercurial ->Clone Other.... In the appearing "Clone External Repository" wizard, we enter http://hg.openjdk.java.net/jdk7/jdk7/hotspot as the "Repository URL" and a local "Parent Directory" and "Clone Name" (usually hotspot) for the new clone (don't forget to adjust the proxy settings if necessary!). After clicking "Finish", NB will clone the HotSpot sources to <Parent Directory>/<Clone Name>. For the rest of this blog I'll assume that the HotSpot sources have been cloned to /OpenJDK/jdk7/hotspot. Here's the content of the Mercurial Output window after a successful clone:

Mercurial Clone
---------------
adding changesets
adding manifests
adding file changes
...
2895 files updated, 0 files merged, 0 files removed, 0 files unresolved
The number of output lines is greater than 500; see message log for complete output
INFO Clone From: http://hg.openjdk.java.net/jdk7/jdk7/hotspot
INFO To:         /OpenJDK/jdk7/hotspot

INFO: End of Clone

Now that we have cloned a fresh repository, we should not forget to patch the sources with the patch file that has been previously described in order to work around some build problems.

With the newly cloned HotSpot sources, we are ready to create a NetBeans project for HotSpot. We more or less follow the description of "Creating a C/C++ Project From Existing Code " in the NB Quick Start tutorial, but as you'll see, there are quite some intricate issues you need to consider here, in order to get a working HotSpot project.

First we start the "New Project" wizard by going to File ->New Project. In the "New Project" wizard we select "C/C++ Project From Existing Code" as project type. In the next step, we select "Using an existing makefile" and enter /OpenJDK/jdk7/hotspot/make/Makefile as the project makefile. In the following "Build Actions" step, we have to specify the project's "Working Directory". The presetting /OpenJDK/jdk7/hotspot/make (i.e. the directory that contains the makefile we've just specified in the previous step) will not work here because of a NB bug (see Issue 125214). We have to use the HotSpot root directory (i.e. /OpenJDK/jdk7/hotspot in our case) in order to work around this problem.

Next we have to define the right "Build Command" which is a little bit tricky, because we've just selected the HotSpot root directory as our working directory. So the first thing we have to do in the "Build Command", will be to change into the make/ directory. As real build command we can use a command line similar to the ones we used in the Building the HotSpot section. And although the build output will be captured in the NB "Build Output Window", it may be still a good idea to additionally store it in a file. So here's a first version of the "Build Command":

cd make && LANG=C ALT_BOOTDIR=/share/software/jse/1.6.0/ ALT_OUTPUTDIR=../../hotspot_c2_debug \
make jvmg 2>&1 | tee ../../hotspot_c2_debug.log

But wait, there's another problem in NB with VERY long lines in the "Build Output Window" (see Issue 124796): if a line exceeds 32Kb, the output will stop and the build will freeze! Unfortunately, the HotSpot Makefile indeed produces a VERY HUGE line: the command line which is created by the makefiles to compile the Java sources of the Serviceability Agent may exceed 100Kb (depending on the absolute base path of your HotSpot sources). With the additional filter part which is needed to strip this huge line from the output, our build command looks as follows:

cd make && LANG=C ALT_BOOTDIR=/share/software/jse/1.6.0/ ALT_OUTPUTDIR=../../hotspot_c2_debug \
make jvmg 2>&1 | grep -v "javac \-source 1\.4" | tee ../../hotspot_c2_debug.log

The clean command is easier - we just have to call the "clean" target. Nevertheless we have to specify ALT_OUTPUTDIR= on the command line such that make knows which directory we want to clean:

cd make && ALT_OUTPUTDIR=../../hotspot_c2_debug make clean

We leave the "Build Result" field empty for now, because the directories which will contain the build results don't exist until now anyway and NB will complain about this fact. Later on, after we have build the project for the first time, it will be possible to adjust this setting easily.

The next step in the "New Project" wizard requires the specification of the folders which contain files that should be added to the project. Again, the "Working Directory" specified in a previous step is the predefined default setting for this entry. Because we have already set the "Working Directory" to point to the HotSpot root directory, this presetting is ok for now. Later on we can (and we will) add additional source folders for the files which were automatically generated by the HotSpot make to the project. We can also restrict the "File types" to ".c .cpp .h .hpp .ad", because these file types are probably the only ones we will be interested in (.ad is the extension for so called "Architecture Description" files which are not real C/C++ files, but contain C/C++ code which is used to automatically generate some other source files, so it is probably a good idea to have them in the project).

We will skip the "Code Assistance Configuration" for now and come back later to it. In the final project configuration step we'll have to choose a name for our project (i.e. hotspot_NB) and a directory (the "Project Location") where NetBeans will save its project folder. Because I don't like to clutter my version controlled directories with external files, I usually place the project directory parallel to the project's root directory (i.e. into /OpenJDK/jdk7 in this example). But that's up to the user. It may be for example a good idea (once the things have settled down) to place a generic NetBeans project directory (you can find such pre-configured projects for Solaris here) inside the HotSpot make directory. In such a case, NB could automatically detect and open the project during the cloning process (this is an option which can be activated with a check box in the Mercurial Cloning Wizard).

After we have created the new project, we can open the context menu of the project in the "Projects" view and select the "Build" action. This will start the HotSpot build and log all the build output to the "Build Output Window". The build command that was specified in the previous step will only build the server version of the HotSpot VM. If we want to build different versions of the VM (with C1 or C2 JIT compiler, with template or C++ interpreter, debug, optimized or product versions) we can create different project configurations for each version by selecting Set Configuration ->Manage Configurations... from the project's context menu. In the appearing "Project Properties" window, we press the Manage Configurations... button and get the "Configurations" window. Here we can rename our current configuration to "C2 debug" (because it builds a debug version of the HotSpot with the C2 server compiler) and make a copy of "C2 debug" which we rename to "C1 debug" (this will be the configuration for building a debug VM with the C1 client compiler). After we return to the "Project Properties" window by pressing OK, we can select the "C1 debug" configuration and adapt the build and clean commands accordingly (originally they are just copies of the corresponding commands from the "C2 debug" configuration). For building the "C1 debug" configuration, we could set them as follows:

cd make && LANG=C ALT_BOOTDIR=/share/software/jse/1.6.0/ ALT_OUTPUTDIR=../../hotspot_c1_debug \
make jvmg1 2>&1 | grep -v "javac \-source 1\.4" | tee ../../hotspot_c1_debug.log

cd make && ALT_OUTPUTDIR=../../hotspot_c1_debug make clean

With the knowledge from the Building the HotSpot section, it should be easy to create all the required, additional configurations accordingly. Notice however, that it may be a good idea to first complete the Code Assistance configuration described in the next section before cloning a configuration. This will save you a lot of time because most of the Code Assistance settings can be shared across configurations.

After we have successfully built the HotSpot project for the first time, it is necessary to refine the project settings. This is because the HotSpot is a quite unusual project in the sense that it has a different source structure than most open source C/C++ projects and therefore doesn't fit right away into a NetBeans project.

As you probably realized already, the HotSpot source tree contains sources for different operating systems and processor architectures. These different files often contain classes and functions with the same names for different platforms. This isn't a problem during the build because the build process figures out the current platform and only considers the needed files. But it considerably confuses the NB Code Assistance to have more than one class or method definition with the same name. Therefore, my advice is to remove the unused platform files from the project. This has the additional benefit of speeding up the parsing of a project at startup, because the project will contain considerably fewer files. If we build on Linux/x86, the following files and directories can be safely removed from the project (right click the corresponding file/directory items in the "Project View" and choose "Remove"):

hotspot/agent/src/os/solaris
hotspot/agent/src/os/win32
hotspot/agent/src/share/native/jvmdi
hotspot/src/cpu/sparc
hotspot/src/os/solaris
hotspot/src/os/windows
hotspot/src/os_cpu/solaris_sparc
hotspot/src/os_cpu/solaris_x86
hotspot/src/os_cpu/windows_x86
hotspot/src/share/vm/utilities/globalDefinitions_sparcWorks.hpp
hotspot/src/share/vm/utilities/globalDefinitions_visCPP.hpp

In the past, the JDK contained the two distinct subdirectories i486 and amd64 for 32-bit and 64-bit x86 architectures respectively. Now, these two architectures have been merged into the single subdirectory x86. However, because the x86 subdirectory still contains files which are relevant for only one of the two architectures, we can also remove the following files from the project on a 32-bit Linux system:

hotspot/src/cpu/x86/vm/assembler_x86_64.cpp
hotspot/src/cpu/x86/vm/assembler_x86_64.hpp
hotspot/src/cpu/x86/vm/assembler_x86_64.inline.hpp
hotspot/src/cpu/x86/vm/dump_x86_64.cpp
hotspot/src/cpu/x86/vm/interp_masm_x86_64.cpp
hotspot/src/cpu/x86/vm/interp_masm_x86_64.hpp
hotspot/src/cpu/x86/vm/interpreterRT_x86_64.cpp
hotspot/src/cpu/x86/vm/interpreter_x86_64.cpp
hotspot/src/cpu/x86/vm/jniFastGetField_x86_64.cpp
hotspot/src/cpu/x86/vm/runtime_x86_64.cpp
hotspot/src/cpu/x86/vm/sharedRuntime_x86_64.cpp
hotspot/src/cpu/x86/vm/stubGenerator_x86_64.cpp
hotspot/src/cpu/x86/vm/stubRoutines_x86_64.cpp
hotspot/src/cpu/x86/vm/stubRoutines_x86_64.hpp
hotspot/src/cpu/x86/vm/templateInterpreter_x86_64.cpp
hotspot/src/cpu/x86/vm/templateTable_x86_64.cpp
hotspot/src/cpu/x86/vm/templateTable_x86_64.hpp
hotspot/src/cpu/x86/vm/vm_version_x86_64.cpp
hotspot/src/cpu/x86/vm/vm_version_x86_64.hpp
hotspot/src/cpu/x86/vm/vtableStubs_x86_64.cpp
hotspot/src/cpu/x86/vm/x86_64.ad
hotspot/src/os_cpu/linux_x86/vm/assembler_linux_x86_64.cpp
hotspot/src/os_cpu/linux_x86/vm/linux_x86_64.ad
hotspot/src/os_cpu/linux_x86/vm/linux_x86_64.s

Another peculiarity of HotSpot is the fact that it creates source files during the build process. These files are placed into the sub-folder <os>_<arch>_compiler<1|2>/generated of the specified output folder (i.e. /OpenJDK/jdk7/hotspot_c2_debug/linux_i486_compiler2/generated in our example). Among others, these are mainly the files generated by the ADL compiler from the .ad file in the <os>_<arch>_compiler<1|2>/generated/adfiles subdirectory and the include files generated by MakeDeps from the "includeDB" files in the <os>_<arch>_compiler<1|2>/generated/incls subdirectory. (Notice that the "includeDB" technique is another undocumented, HotSpot specific trickery that's beyond the scope of this introduction. You can find a minimalistic introduction at the top of hotspot/src/share/vm/includeDB_core)

In the next step, we will therefore add the to two subdirectories generated/adfiles/ and generated/incls/ from within the output folder to the sources of our project by selecting Add Existing Items from Folders... from the project's context menu. Don't forget to use .c .cpp .h .hpp .incl as "File types" because the include files generated by MakeDeps all have .incl suffixes. We'll also have to tell NetBeans to treat files with a .incl suffix like usual C/C++ header files by going to Tools ->Options ->Advanced Options ->IDE Configuration ->System ->Object Types ->C and C++ Header Data Objects and add incl as extension in the Extension and MIME Types field.

Once we have added all the additional files to our project, we can start configuring the NetBeans Code Assistance. There are two main points that have to be done here: first we have to specify the directories where the Code Assistance parser should look for include files and second we have to define the preprocessor directives which should be considered during code parsing. For the first step we have to go to Project Properties ->Code Assistance ->C++ Compiler ->Include Directories. Unfortunately it is only possible to add one directory at a time from the file selection box and there are quite some directories we have to add:

hotspot/src/share/vm/adlc
hotspot/src/share/vm/asm
hotspot/src/share/vm/c1
hotspot/src/share/vm/ci
hotspot/src/share/vm/classfile
hotspot/src/share/vm/code
hotspot/src/share/vm/compiler
hotspot/src/share/vm/gc_implementation
hotspot/src/share/vm/gc_implementation/concurrentMarkSweep
hotspot/src/share/vm/gc_implementation/parNew
hotspot/src/share/vm/gc_implementation/parallelScavenge
hotspot/src/share/vm/gc_implementation/shared
hotspot/src/share/vm/gc_interface
hotspot/src/share/vm/interpreter
hotspot/src/share/vm/libadt
hotspot/src/share/vm/memory
hotspot/src/share/vm/oops
hotspot/src/share/vm/opto
hotspot/src/share/vm/prims
hotspot/src/share/vm/runtime
hotspot/src/share/vm/services
hotspot/src/share/vm/utilities
hotspot/src/cpu/x86/vm/
hotspot/src/os/linux/launcher/
hotspot/src/os/linux/vm/
hotspot/src/os_cpu/linux_x86/vm/
../hotspot_c2_debug/linux_i486_compiler2/generated/
../hotspot_c2_debug/linux_i486_compiler2/generated/adfiles/
../hotspot_c2_debug/linux_i486_compiler2/generated/jvmtifiles/

After we've added the required include directories, we have to tell the Code Assistance parser which preprocessor definitions it should use when parsing the project. For the time being, I just took a look at the compilation command line of some arbitrary HotSpot files, copied all the definitions they contained and inserted them into the Preprocessor Definitions field:

LINUX _GNU_SOURCE IA32 ASSERT DEBUG COMPILER2 COMPILER1 _REENTRANT VM_LITTLE_ENDIAN GAMMA

Because I was not sure if Code Assistance would parse .h as C or as C++ files, I doubled the values for Include Directories and Preprocessor Definitions in the C Compiler section. If you prefer typing instead of clicking, you can also edit the project's configurations file <NB_project_dir>/nbproject/configurations.xml by hand and simply duplicate the entries inside the <ccCompilerTool> tag for the <cCompilerTool> tag. As you can see, the enclosing <confs> tag contains all the relevant configuration settings for a project:

  <projectmakefile>hotspot_NB-Makefile.mk</projectmakefile>
  <confs>
    <conf name="C2_debug" type="0">
      <toolsSet>
        <compilerSet>GNU</compilerSet>
        <cRequired>true</cRequired>
        <cppRequired>true</cppRequired>
        <fortranRequired>false</fortranRequired>
        <platform>2</platform>
      </toolsSet>
      <makefileType>
        <makeTool>
          <buildCommandWorkingDir>../hotspot</buildCommandWorkingDir>
          <buildCommand>cd make && 
            LANG=C
            ALT_BOOTDIR=/share/software/jse/1.6.0/
            ALT_OUTPUTDIR=../../hotspot_c2_debug 
            make jvmg 2>&1 | 
            grep -v "javac \-source 1\.4"  | tee ../../hotspot_c2_debug.log
          </buildCommand>
          <cleanCommand>cd make && 
            ALT_OUTPUTDIR=../../hotspot_c2_debug make clean
          </cleanCommand>
          <executablePath>
            ../hotspot_c2_debug/linux_i486_compiler2/jvmg/gamma
          </executablePath>
          <cCompilerTool>
            <includeDirectories>
              <directoryPath>../hotspot/src/cpu/x86/vm</directoryPath>
              <directoryPath>../hotspot/src/os/linux/launcher</directoryPath>
              <directoryPath>../hotspot/src/os/linux/vm</directoryPath>
              ...
            </includeDirectories>
            <preprocessor>LINUX _GNU_SOURCE IA32 ASSERT DEBUG COMPILER2
              COMPILER1 _REENTRANT VM_LITTLE_ENDIAN GAMMA
            </preprocessor>
          </cCompilerTool>
          <ccCompilerTool>
            <includeDirectories>
              <directoryPath>../hotspot/src/cpu/x86/vm</directoryPath>
              <directoryPath>../hotspot/src/os/linux/launcher</directoryPath>
              <directoryPath>../hotspot/src/os/linux/vm</directoryPath>
              ...
            </includeDirectories>
            <preprocessor>LINUX _GNU_SOURCE IA32 ASSERT DEBUG COMPILER2
              COMPILER1 _REENTRANT VM_LITTLE_ENDIAN GAMMA
            </preprocessor>
          </ccCompilerTool>
        </makeTool>
        <requiredProjects>
        </requiredProjects>
      </makefileType>

Of course, manual changes to the project's configurations file should only be done if a project isn't active in a running NetBeans session! Notice that the parsing of a project like HotSpot may take a considerable amount of time - on my machine about 60 seconds. The progress of this operation is indicated by a small progress bar in the lower right corner of NB. Clicking on it will open a small window which logs the files that are currently processed by the Code Assistance parser. NetBeans parses a project every time the project is opened. Although this may take some time, I think this was the right decision, because this way there's no database file on disk that can get corrupted. If you should encounter problems with Code Assistance (see below), you'll just have to close and reopen the accountable project, to hopefully get it working again.

During my first configuration attempts, I sometimes got null pointer exceptions during code parsing (see issue 125611) and once Code Configuration didn't complete at all. However, once I figured out and set up the right Code Assistance settings (especially the right include paths and preprocessor directives), Code Assistance ran quite smooth. Especially the possibility of defining different project configurations with different preprocessor directives (opt vs. debug, template vs. C++ interpreter) and include paths and easily switching between these configurations while the source information is always accurate, is quite nice.

After I had finally finished the "Code Assistance" configuration I was keen on trying the debugging support in NetBeans. I first went to the Project Properties ->Make and changed the Build Results entry to point to the gamma executable in the output directory (../hotspot_c2_debug/linux_i486_compiler2/jvmg/gamma in this example). Notice that this will always be a relative path with respect to your project directory if you choose the executable with the file selection box. The only way to enter an absolute path name here is by entering it manually into the text field (you'll have to do this if you choose to use another Run Directory than the default project directory in the next step).

In the Running category of the Project Properties, I entered "-XX:+TraceBytecodes -XX:StopInterpreterAt=1 -version" for the Arguments and added JAVA_HOME and LD_LIBRARY_PATH to the Environment with the values as described in Running the HotSpot. If you don't set the Run Directory, the NB project directory will be used as a default. If you choose another Run Directory you need to set Build Results in the Make category such that it contains the absolute path to the executable, in order to run it successfully.

With these settings, it was possible to run the HotSpot by executing the Run Main Project action. This will open a fresh xterm window, dump the executed bytecodes (because of the -XX:+TraceBytecodes option) and finally print out the version of the VM and the Java Runtime Environment:

...
[4506] virtual void java.io.FileOutputStream.write(jobject, jint, jint)
[4506]   385353     0  fast_aload_0
[4506]   385354     1  aload_1
[4506]   385355     2  iload_2
[4506]   385356     3  iload_3
[4506]   385357     4  invokespecial 5376 <writeBytes> <([BII)V>
OpenJDK Runtime Environment (build 1.7.0-internal-debug-dXXXXXX_04_jan_2008_11_27-b00)
...
[4506] virtual void java.io.FileOutputStream.write(jobject, jint, jint)
[4506]   390343     0  fast_aload_0
[4506]   390344     1  aload_1
[4506]   390345     2  iload_2
[4506]   390346     3  iload_3
[4506]   390347     4  invokespecial 5376 <writeBytes> <([BII)V>
OpenJDK Server VM (build 12.0-b01-internal-jvmg, mixed mode)[4506]   390348     7  return
...
392702 bytecodes executed in 229.6s (0.002MHz)

As you can see, printing the OpenJDK version requires the execution of nearly 400.000 bytecodes, so think twice before before you heedlessly call java -version the next time:)

Now that we successfully run the HotSpot within NetBeans, we can set up a breakpoint and try the debugger. Please be sure to use at least gdb 6.6, otherwise the NetBeans gdb-plugin will abort the debugging session because of warnings which are thrown by older versions of gdb when debugging the HotSpot. Notice that you can not specify which gdb version to use in NetBeans. Instead you have to set the executable path (under Tools ->Options ->C/C++ ->Build Tools ->Current Path) such that the desired version of gdb will be detected first by NB.

The -XX:StopInterpreterAt=<n> option can be used to stop the HotSpot interpreter before the execution of the specified bytecode number. It is implemented by continuously counting each executed bytecode until the n-th bytecode is reached at which point the VM will call the global, empty breakpoint() function. This breakpoint() function is a hook for the debugger which can be used to intercept the execution of the specified bytecode by defining a debugger, breakpoint for it. We choose Run ->New Breakpoint... and enter breakpoint as the Function Name. Then we execute Debug Main Project to debug the project. This time, only the first bytecode will be printed in the output window and the HotSpot will stop in the breakpoint() function:

VM option '+TraceBytecodes'
VM option 'StopInterpreterAt=1'

[5027] static void java.lang.Object.<clinit>()
[5027]        1     0  invokestatic 2304 <registerNatives> <()V> 

Unfortunately, that's basically all we can currently do with the debugger support in NetBeans. If we have a look at the stack in the Call Stack window, we can see something similar to:

#0  breakpoint ()
    at /OpenJDK/jdk7/hotspot/src/os/linux/vm/os_linux.cpp:394
#1  0x0665d5f5 in os::breakpoint ()
    at /OpenJDK/jdk7/hotspot/src/os/linux/vm/os_linux.cpp:389
#2  0x40246164 in ?? ()
#3  0xbfffb658 in ?? ()
#4  0x43316097 in ?? ()
#5  0xbfffb67c in ?? ()
#6  0x43375ab8 in ?? ()

The unknown frames are neither a debugger nor a NetBeans problem. It's just the result of the fact that the HotSpot is more than often in generated code. This is true for both interpreters, the template interpreter as well as for the C++ interpreter to a certain degree, and of course for compiled (JITed) code. Unfortunately, the NB debugger support can not handle assembler (e.g. there is no stepi command, no register window and no assembler code view). Moreover, signal handling is not implemented (crucial for HotSpot debugging) and the thread support doesn't work either. Generally speaking, the debugging support in NetBeans is quite basic right now and only useful for debugging not deeper than to the C/C++ level.

Probably I should have read the NetBeans GDB Debugger Milestones before starting. It lists all the missing features for future milestones. By private communication with some NB developers (thanks to Leonid Lenyashin and Sergey Grinev) I also got the information that the dbx-engine already has disassembler support, stepi, a register view and it works well with multiple threads. However, the dbx-engine is currently only available in Sun Studio which at the time of this writing is still NetBeans 5.5.1 based (although there should be a new NB 6 based express release in couple of weeks). I did some quick experiments and the dbx-engine looks indeed promising, but dbx had problems with the debug information generated by g++, so I gave up. Perhaps I'll try again with the next Sun Studio release that can read my NetBeans 6 project files.

I also had the chance to ask Gordon Prieur, a Sun staff engineer and project lead for Sun Studio IDE some questions about the status of the NetBeans gdb debugging engine. Following you can find his answers:

Congratulation! If you really read all this blog from the beginning up to here please drop me a note. Hopefully you managed to successfully build and run the HotSpot on Linux!

Although this tutorial got quite lengthy in the end, building and running an own version of OpenJDK and the HotSpot VM should be not to hard for any developer with average Linux experience. Depending on the Linux distribution, there will probably always be the need to install some newer and sometimes even some older version (e.g. gcc) of some packages to meet the build requirements of the OpenJDK - but that should be manageable.

Regarding NetBeans, there's no clear recommendation from my side. If you're already familiar with an IDE like NetBeans, it my be worth while going through all the project configuration hassle to get the integrated Mercurial support and a reasonably well working Code Assistance - but you will still have to use the command line for debugging. On the other side, if you're already using Emacs with Cscope and DVC there's probably no killer argument for switching to NetBeans for HotSpot development.

[1] Kelly O'Hair's Build Cheat Sheet
[2] Kelly O'Hair's Glossary for JDK Builds
[3] OpenJDK Build Readme
[4] NetBeans C/C++ Support Quick Start Tutorial
[5] Interview with Gordon Prieur about Sun Studio 12

HotSpot development on Linux with NetBeans - Part 1

Here comes yet another step-by-step tutorial which explains how to fetch the OpenJDK sources, compile them and work with them inside the NetBeans IDE. It focuses on building and running the different flavors (opt/debug, client/server JIT compiler, template/C++ interpreter) of the HotSpot VM on Linux/x86 and concludes with a short evaluation of NetBeans 6.0 as an development environment for HotSpot hacking.

Because it was two big for a single blog entry, it is split up into two parts: this first part explains how to build and run the HotSpot while the NetBeans integration is described in the second part.

If you're interested in building on Windows you can consult Ted Nedwards blog. A description of how to build different Java-only parts of the JDK with NetBeans 6 can be found here (note that the current NetBeans "world" project under jdk/make/netbeans which is supposed to build the entire JDK (including the HotSpot) doesn't work anymore after the Mercurial switch and the resulting directory restructuring).

I started to develop on a Suse Enterprise Linux 9.3 server with 4 Intel Xeon CPUs at 3GHz with 4GB of memory. It had gcc 3.3.3 and gnumake 3.80 installed by default which both suffice for OpenJDK development.

The following subsections will detail how I installed various required software packages. I usually compile and install new software into /share/software and link the resulting executables to /usr/local/bin which comes first in my PATH environment variable.

Also my box had Python 2.4 installed, I decided to install a fresh, 2.5 version of Python. You probably don't have to repeat this step because Mercurial should work perfectly fine with Python 2.4.

> cd /share/software
> tar -xzf Python-2.5.1.tgz
> cd Python-2.5.1/
> mkdir /share/software/Python-2.5.1_bin
> configure --prefix=/share/software/Python-2.5.1_bin
> make
> make install
> ln -s /share/software/Python-2.5.1_bin/bin/* /usr/local/bin/

After this step I downloaded, compiled and installed Mercurial. As mentioned above, you'll be probably fine if you use your default Python installation for this step:

> cd /share/software
> tar -xzf mercurial-0.9.5.tar.gz
> cd mercurial-0.9.5/
> make install-bin PYTHON=/share/software/Python-2.5.1_bin/bin/python PREFIX=/share/software/hg
> ln -s /share/software/hg/lib/python2.5/site-packages/* /share/software/Python-2.5.1_bin/lib/python2.5/site-packages/
> ln -s /share/software/hg/bin/hg/* /usr/local/bin/
> cat > ~/.hgrc
[ui]
username = Volker H. Simonis
^D
> hg debuginstall

The last command hg debuginstall, should complete without error and should produce the following output:

Checking encoding (UTF-8)...
Checking extensions...
Checking templates...
Checking patch...
Checking merge helper...
Checking commit editor...
Checking username...
No problems detected

After I had a working Mercurial, I could use it to get the Forest extension which isn't strictly needed but which will simplify the download of the OpenJDK sources. Note that you'll have to set the http_proxy environment variable to point to your http proxy server if you're behind a firewall (e.g. http_proxy=http://proxy:8080).

cd /share/software
hg clone http://www.terminus.org/hg/hgforest
ln -s /share/software/hgforest/forest.py /share/software/hg/lib/python2.5/site-packages/hgext/

Now comes the big moment. I fired up a hg fclone command to clone the OpenJDK sources. If everything works fine, this should download about 28.000 files to your local machine. The output should look as follows (stripped-down..):

> cd /share/software
> mkdir OpenJDK
> cd OpenJDK
> hg fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7
[.]
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 26 changes to 26 files
26 files updated, 0 files merged, 0 files removed, 0 files unresolved

[corba]

... ... ...

[langtools]
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2974 changes to 2974 files
2974 files updated, 0 files merged, 0 files removed, 0 files unresolved

There may still be a problem here, if you're behind a firewall. Instead of getting the sources, you may get the following error:

> hg fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7
[.]
abort: error: Name or service not known

This is because of a bug in fclone which doesn't honor the setting of the http_proxy environment variable (although hg alone does use it). Fortunately, this can be fixed easily by setting the http proxy in the ~/.hgrc configuration file like this:

[http_proxy]
host=proxy:8080

That's it! Now we should have the sources. In order to build them, we still need to install some third party libraries like Freetype, Findbugs and Cups and of course the binary encumbrances which are bundled in the appropriate binary plugs file. You may skip any of the following installation steps if your Linux distribution already has the development packages installed for the corresponding library.

Let's start with Freetype:

> cd /share/software/OpenJDK
> tar -xzf freetype-2.3.5.tar.gz
> cd freetype-2.3.5/
> vi include/freetype/config/ftoption.h
uncomment /* #define TT_CONFIG_OPTION_BYTECODE_INTERPRETER */
> mkdir ../freetype-2.3.5_bin
> ./configure --prefix=/share/software/OpenJDK/freetype-2.3.5_bin/
> make
> make install

In order to make the Freetype library accessible for the OpenJDK build, we can set the following environment variables:

> export ALT_FREETYPE_LIB_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/lib
> export ALT_FREETYPE_HEADERS_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/include

Another possibility to make a library available during the OpenJDK build is to set the corresponding variable on the make command line. This is the approach that I'll use later on in the build section. Nevertheless, I'll also list the export statements after the installation of each library for the sake of completeness.

The same thing for Findbugs ..

> cd /share/software/OpenJDK
> tar -xzf findbugs-1.3.1.tar.gz
> export FINDBUGS_HOME=/share/software/OpenJDK/findbugs-1.3.1

.. and Cups:

> cd /share/software/OpenJDK
> tar -xzf cups-1.3.5-source.tar.gz
> cd cups-1.3.5/
> mkdir ../cups-1.3.5_bin
> ./configure --prefix=/share/software/OpenJDK/cups-1.3.5_bin 
> make
> make install
> export ALT_CUPS_HEADERS_PATH=/share/software/OpenJDK/cups-1.3.5_bin/include

Finally we need to get the Binary plugs and a boot JDK (at least Java 6). I already had Java 6 and Ant installed in /share/software/Java/1.6.0 and /share/software/Ant/1.6.4 respectively, so I only had to download and install the Binary plugs and make all of them known to the OpenJDK build:

> cd /share/software/OpenJDK
> java -jar jdk-7-ea-plug-b24-linux-i586-04_dec_2007.jar
> export ALT_BINARY_PLUGS_PATH=/share/software/OpenJDK/jdk-7-ea-plug-b24-linux-i586-04_dec_2007/openjdk-binary-plugs
> export ALT_BOOTDIR=/share/software/Java/1.6.0
> export ANT_HOME=/share/software/Ant/1.6.4/

With this step we ultimately finished the necessary preparations and can now happily proceed to build the OpenJDK!

Before we start the build, we should first run the sanity check to see if our settings and the available tools and libraries are sufficient for the build. Because I don't like to clutter my environment, I'll set all the needed environment variables on the command line, such that they only affect the current command as follows:

> cd /share/software/OpenJDK/jdk7
> LANG=C \
  FINDBUGS_HOME=/share/software/OpenJDK/findbugs-1.3.1 \
  ANT_HOME=/share/software/Ant/1.6.4/ \
  ALT_CUPS_HEADERS_PATH=/share/software/OpenJDK/cups-1.3.5_bin/include \
  ALT_BOOTDIR=/share/software/Java/1.6.0 \
  ALT_BINARY_PLUGS_PATH=/share/software/OpenJDK/jdk-7-ea-plug-b24-linux-i586-04_dec_2007/openjdk-binary-plugs \
  ALT_FREETYPE_LIB_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/lib \
  ALT_FREETYPE_HEADERS_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/include \
  make sanity

You shouldn't proceed forward, until the sanity check completes without any errors or warnings. The checks performed by the sanity check are still quite weak and a passed sanity check is no guarantee for a successful build. The sanity check for example just verifies that ALT_BINARY_PLUGS_PATH points to a valid directory, but not if that directory really contains the binary plugins!

Internally, SUN apparently still uses gcc 3.2.2 to build the JDK and with gcc 3.2.2 there seem to be no warnings during the build so they decided to use the -Werror option on Linux which instructs gcc to treat every compiler warning as error. If you however want to build with a gcc version higher than 3.2.2 (and you'll probably want to do this on a newer Linux distribution) you'll either have to use precompiled headers (USE_PRECOMPILED_HEADER=true) or comment the line WARNINGS_ARE_ERRORS = -Werror in the file hotspot/build/linux/makefiles/gcc.make (see Bug 6469784).

Using precompiled headers is probably the easier way to go for first time users and it has the additional benefit of speeding up the build considerably. But it only helps with warnings related to inlining and it has the disadvantage of hiding problems with the includeDB (see this mail thread for a discussion of the topic). If you want to use gcc 4.3 or higher, you'll probably have to disable the treatment of warnings as errors for a successful build (see for example here).

Building the corba subdirectory will fail if you haven't set ALT_JDK_IMPORT_PATH. This is because of a known bug in corba/make/common/shared/Defs.gmk (see this mail thread). You can fix the problem by inserting the following lines into corba/make/common/shared/Defs.gmk, just before the line that includes Compiler.gmk:

ifdef ALT_LANGTOOLS_DIST
  LANGTOOLS_DIST :=$(call FullPath,$(ALT_LANGTOOLS_DIST))
else
  LANGTOOLS_DIST =
endif

After this last patch, we can finally start the build. I'll focus here on debug builds because you're probably a developer if you read this and as a developer you're probably interested in a debug build (after all you could download a product build, so it would not be worth the work).

> cd /share/software/OpenJDK/jdk7
> LANG=C \
  FINDBUGS_HOME=/share/software/OpenJDK/findbugs-1.3.1 \
  ANT_HOME=/share/software/Ant/1.6.4/ \
  ALT_CUPS_HEADERS_PATH=/share/software/OpenJDK/cups-1.3.5_bin/include \
  ALT_BOOTDIR=/share/software/Java/1.6.0/ \
  ALT_BINARY_PLUGS_PATH=/share/software/OpenJDK/jdk-7-ea-plug-b24-linux-i586-04_dec_2007/openjdk-binary-plugs \
  ALT_FREETYPE_LIB_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/lib \
  ALT_FREETYPE_HEADERS_PATH=/share/software/OpenJDK/freetype-2.3.5_bin/include \
  HOTSPOT_BUILD_JOBS=5 \
  ALT_PARALLEL_COMPILE_JOBS=5 \
  USE_PRECOMPILED_HEADER=true \
  SKIP_DEBUG_BUILD=false \
  SKIP_FASTDEBUG_BUILD=true \
  DEBUG_NAME=debug \
  ALT_OUTPUTDIR=/share/software/OpenJDK/jdk7/build/openjdk_full_debug \
  make 2>&1 | tee /share/software/OpenJDK/jdk7/build/openjdk_full_debug.log

Note that we don't use the make_debug target because there's a bug in the top-level Makefile that ignores the ALT_OUTPUTDIR if that target will be used (see this mail thread). You should be also aware of the fact, that the build will always create an empty directory named <ALT_OUTPUTDIR>-fastdebug which can be ignored and removed.

It is also advisable to save the build output in a file. This can be achieved by redirecting the whole output to tee as shown in the call to make above. tee is a utility that duplicates its input to the standard output and to an additional file. This file can be consulted later on if there have been build problems or if we just want to know how a file has been built and where the resulting object files have been placed to.

Following the above pattern it is also possible to build a product or a fastdebug build. You just have to set SKIP_DEBUG_BUILD=true SKIP_FASTDEBUG_BUILD=false DEBUG_NAME=fastdebug for a fastdebug build and SKIP_DEBUG_BUILD=true SKIP_FASTDEBUG_BUILD=true for a product build.

Notice that ALT_PARALLEL_COMPILE_JOBS is currently only honored by the corba and the jdk subprojects while the hotspot subproject uses HOTSPOT_BUILD_JOBS as indicator to do a parallel build. Unfortunately, due to another bug, neither ALT_PARALLEL_COMPILE_JOBS nor HOTSPOT_BUILD_JOBS is handed over from the top-level makefile to the hotspot makefile. However, this can be easily fixed by adding the following lines to /make/hotspot-rules.gmk, just before the hotspot-build target:

ifneq ($(PARALLEL_COMPILE_JOBS), 1)
  HOTSPOT_BUILD_ARGUMENTS += HOTSPOT_BUILD_JOBS=$(PARALLEL_COMPILE_JOBS)
endif

With this change, setting ALT_PARALLEL_COMPILE_JOBS on the make command line will be enough to trigger a parallel hotspot build, which should be considerable faster on a multi-processor machine (a good default setting for ALT_PARALLEL_COMPILE_JOBS is hard to predict, but 1.5xNrOfCPUs should be a good starting point).

Sooner or later (depending on your machine and the right setting of ALT_PARALLEL_COMPILE_JOBS:) the build should finish (hopefully without any error). Among others, this will create the following subdirectories in the build location that was specified with ALT_OUTPUTDIR:

bin
corba
j2sdk-image
lib
hotspot
include
j2re-image
langtools

corba, hotspot and langtools contains the build results of the corresponding subprojects. bin, include and lib contain the binaries, include files and libraries that are contained in the corresponding directories of a Java SDK or RE distribution. Finally, j2sdk-image and j2re-image contain a complete image of a Java SDK or RE distribution, assembled from the subdirectories of the build subdirectory. If everything went fine, we can now call bin/java (or j2sdk-image/bin/java or j2re-image/bin/java which is all the same) to verify our build:

> /share/software/OpenJDK/jdk7/build/openjdk_full_debug/bin/java -version
openjdk version "1.7.0-internal-debug"
OpenJDK Runtime Environment (build 1.7.0-internal-debug-dXXXXXX_04_jan_2008_11_27-b00)
OpenJDK Server VM (build 12.0-b01-jvmg, mixed mode)

That looks really nice, isn't it!!! We managed to built a complete debug version of the OpenJDK from scratch!

Now that we've successfully built the OpenJDK and started hacking the HotSpot VM, we probably don't want to go through all this hassle just to verify that a small VM change compiles and works correctly. Luckily, the HotSpot developers at SUN didn't wanted either, so they provided an elegant way to rebuild the HotSpot part of the VM and test it.

And here is how it works. Go to the hotspot/make directory and execute the following make command:

> LANG=C \
  ALT_BOOTDIR=/share/software/Java/1.6.0/ \
  HOTSPOT_BUILD_JOBS=5 \
  ALT_OUTPUTDIR=../../build/hotspot_debug \
  make jvmg jvmg1 2>&1 | tee ../../build/hotspot_debug.log

As you can see, there are considerably fewer variables needed to build the VM (in fact the only real dependency is the boot JDK specified with ALT_BOOTDIR). By selecting the corresponding build target it is possible to build debug builds (jvmg and jvmg1 targets), fastdebug builds (fastdebug and fastdebug1 targets), optimized builds (optimized and optimized1 targets) and product builds (product and product1 targets). A "1"-suffix in the target name indicates that the client version (the one with the C1 JIT compiler) will be build while a target name without suffix will build the server version (the one with the C2 JIT compiler) of the corresponding VM. A fastdebug build is an optimized build with assertions (C/C++ style asserts in the VM code) enabled. An optimized build has no assertions, while a product build is an optimized build without assertions and -DPRODUCT defined (this may for example disable non-product switches in the resulting VM).

With this information in mind, you'll probably easily guess that the last make command builds the debug version of the client and the server VM. The results will be placed in the corresponding <os>_<arch>_compiler1/ and <os>_<arch>_compiler2/ subdirectories of the output directory with the little anomaly that debug build will be placed in the jvmg subdirectory (i.e. linux_i486_compiler1/jvmg/ and linux_i486_compiler2/jvmg/ in this example).

These directories not only contain the HotSpot VM as a shared library (libjvm.so) but also a small executable called gamma. I don't really know the origin of this name (perhaps someone of the geeks can comment on this?), but it is a really convenient possibility to test the newly created VM:

> build/hotspot_debug/linux_i486_compiler1/jvmg/gamma -version
build/hotspot_debug/linux_i486_compiler1/jvmg/gamma: \
 error while loading shared libraries: libjvm.so: \
 cannot open shared object file: No such file or directory

> LD_LIBRARY_PATH=build/hotspot_debug/linux_i486_compiler1/jvmg \
  build/hotspot_debug/linux_i486_compiler1/jvmg/gamma -version
JAVA_HOME must point to a valid JDK/JRE to run gamma
Error: could not find libjava.so
Error: could not find Java 2 Runtime Environment.

> LD_LIBRARY_PATH=build/hotspot_debug/linux_i486_compiler1/jvmg \
  JAVA_HOME=build/openjdk_full_debug/j2sdk-image \
  build/hotspot_debug/linux_i486_compiler1/jvmg/gamma -version
openjdk version "1.7.0-internal-debug"
OpenJDK Runtime Environment (build 1.7.0-internal-debug-dXXXXXX_04_jan_2008_11_27-b00)
OpenJDK Client VM (build 12.0-b01-internal-jvmg, mixed mode)

As you can see, we just have to put the directory which contains the desired libjvm.so (client or server) into the LD_LIBRARY_PATH and define JAVA_HOME to point to a valid JDK or JRE (e.g. the one we built in the first step). gamma is a simple launcher intended for internal engineering test. It is build from hotspot/src/os/linux/launcher/java.c and hotspot/src/os/linux/launcher/java_md.c (search for launcher.c in the build log to see the details). hotspot/src/os/linux/launcher/java.c and hotspot/src/os/linux/launcher/java_md.c are stripped down versions of the real Java launcher sources jdk/src/share/bin/java.c and jdk/src/solaris/bin/java_md.c from the jdk workspace. The gamma launcher misses some of the logic of the default Java launcher which finds the corresponding VM and JDK automatically. Therefore it is necessary to signal their location by setting the LD_LIBRAY_PATH and JAVA_HOME environment variables. The big advantage however is the fact that it builds within the hotspot project without any dependency on the jdk workspace and comes in quite handy for fast development and testing.

Just as a side note: you probably wondered why I wrote jdk/src/solaris/bin/java_md.c in the previous paragraph, for the location of java_md.c in the jdk workspace. The solaris part was not a typo. The jdk workspace isn't that well structured like the hotspot workspace which divides platform and architecture dependant files into the corresponding os, cpu and os_cpu subdirectories. Instead, in the jdk workspace there's just a windows directory for the Windows native code and a solaris directory which contains all the Unix native code (separated by ifdefs if necessary). (In fact there is a linux subdirectory, but it only contains the linux man pages and no code). This may identify as serious problem for the various porting projects which attempt to port the OpenJDK to other Unix-like operation systems.

Currently, the gamma launcher is somewhat outdated (hotspot/src/os/linux/launcher/java.c is based on the 1.6.0-b28 JDK version of jdk/src/share/bin/java.c as stated in the file) but it is still sufficient to test the HotSpot VM. Hopefully it will be updated, as the JDK7 development moves forward.

If you want to build the C++-Interpreter instead of the default template interpreter, you have to additionally set CC_INTERP=true on the build command line. Currently the C++-Interpreter only works for the 32-bit x86 debug build and for the 32-bit opt and debug builds on SPARC (see my previous blog entry for how to get the 32-bit x86 opt and the 64-bit SPARC versions running). Notice that you'll also have to disable the the treatment of warnings as errors if you built the C++-Interpreter, because its sources generate some warnings.

For your convenience I created a patch file (in fact I run hg diff > linux32.pach in the hotspot directory) which fixes all the problems mentioned so far. To apply it, just download it and call patch as follows:

> cd hotspot/
> patch -p1 < linux32.patch
patching file build/linux/makefiles/gcc.make
patching file src/cpu/x86/vm/cppInterpreter_x86.cpp
patching file src/share/vm/interpreter/bytecodeInterpreter.cpp

You should now be able to build a debug version of the C++-Interpreter enabled HotSpot with the following command:

> LANG=C \
  ALT_BOOTDIR=/share/software/Java/1.6.0/ \
  HOTSPOT_BUILD_JOBS=5 \
  ALT_OUTPUTDIR=../../build/hotspot_CC_INTERP_debug \
  CC_INTERP=true \
  make jvmg jvmg1 2>&1 | tee ../../build/hotspot_CC_INTERP_debug.log

Notice that building an interpreter-only VM isn't currently supported out of the box with the current top-level HotSpot makefiles. Previously, the CORE targets (debugcore, jvmgcore, fastdebugcore, optimizedcore, profiledcore and productcore) could be used for this purpose and building the HotSpot that way, gave you a pure, interpreter-only VM. However now, as Tom Rodriguez explains "..core is just a system without a compiler but still including the classes needed by the compiler like nmethod so it's slightly less minimal ... and the core makefile targets should still work". These core targets are defined in the platform-dependant makefiles hotspot/build/<os>/Makefile only, and not in hotspot/make/Makefile so you'll probably have to hack hotspot/make/Makefile to build them.

See HotSpot development on Linux with NetBeans - Part 2

[1] Kelly O'Hair's Build Cheat Sheet
[2] Kelly O'Hair's Glossary for JDK Builds
[3] OpenJDK Build Readme
[4] NetBeans C/C++ Support Quick Start Tutorial
[5] Interview with Gordon Prieur about Sun Studio 12

This work is licensed under a Creative Commons License.