Skip to main content

Compute Server: how it works

Posted by nidaley on March 30, 2006 at 12:33 PM PST

The recently created Compute Server project aims to enable Java developers to easily and efficiently use the Sun Grid Compute Utility as a platform for the distributed execution of parallel jobs. The following diagram describes currently "How It Works" from 50,000 feet. I thought it might be helpful to walk through this diagram and describe each step in more depth.

In the above diagram, steps 1 - 3 describe activities performed in NetBeans 5.0 using support provided by the NetBeans Compute Server plug-in:

  1. write task, generator, and output processing code: The project creation wizard in NetBeans contains new Compute Server project types that generate template Java classes required by the Compute Server programming model. Briefly, the 3 important classes you may need to implement for our programming model are:
    • com.sun.computeserver.task.Task: You must implement this interface. A task is an independent unit of work in your job. Task objects can create 3 types of objects: feedback, output, and new tasks. Briefly, feedback objects go back to the generator (see the next bullet) and output objects gets placed in your job's result ZIP file.
    • com.sun.computeserver.task.Generator: A generator is essentially an iterator over your Task objects, and typically creates them on the fly. In addition, your generator receives the feedback objects that are created by your tasks and may create output objects that then get placed in your job's result ZIP file. Writing a generator is not necessary in all cases, such as when retrieving task data from a database.
    • Output Processor: An output processor is an application (has a main method that takes 1 argument -- the computation results ZIP file), that you run on your machine once your job is complete and you have downloaded your job results ZIP file. Using a Compute Server utility class (com.sun.computeserver.output.JobOutputProcessor), you can retrieve and process the objects that were created on Sun Grid by your tasks and generator.
  2. build and debug code: The plug-in contains a Compute Server execution environment that can run on your local machine. This execution environment mirrors the Compute Server execution environment on Sun Grid, with the exception that only one task is executed at a time. Debugging your project is performed in the same manner you debug any other Java application project within NetBeans.
  3. package code and data: The plug-in supports the packaging of your code (and optionally data from your database) into a ZIP file that you can then upload to Sun Grid. Many options on what gets packaged, such as native and Java libraries, are configurable through various project property sheets or context menus.

Steps 4 - 6 involve interaction and execution of your job on the Sun Grid Compute Utility:

  1. manually submit job to Sun Grid: You connect to the Sun Grid Compute Utility web portal, login and upload the packaged ZIP file created by the plug-in.
  2. job is provisioned on Sun Grid: Once you start your job on Sun Grid, your generator class gets instantiated on 1 node and then repeatedly called to retrieve your Task objects. Your tasks then get run in parallel on different nodes. Feedback objects created by your tasks flow back to your generator. Output objects created by your tasks flow back to your job's result ZIP file. New task objects created by your tasks will be executed on other nodes.
  3. manually retrieve output from Sun Grid: Once your job completes, you connect to the Sun Grid Compute Utility web portal, login and download your job's result ZIP file.

Finally, step 7 is performed back in NetBeans using the plug-in:

  1. process output using your output processing code: With your job's result ZIP file in-hand, the plug-in will run your output processor against the ZIP file. Besides accessing the object created by your tasks and generator during your job's execution on Sun Grid, you can also access your job's statistics.

Hopefully that gives you a taste for how the NetBeans Compute Server plug-in can greatly simplify the development, debugging, and packing of your jobs for Sun Grid.

Go check it out! And, of course, send us your feedback.

The Final Word...

The price of reliability is the pursuit of the utmost simplicity.

- C.A.R. Hoare

Related Topics >>