Skip to main content

FlashGridding with Java Technology (TS-3714)

Posted by dhushon on May 18, 2006 at 6:59 PM PDT

I'm sitting here watching Van and his ComputeCycles.org team describe the benefits of Java and Jini in the construction of grid technologies that bridge heterogeneous networks (including firewalls), systems / operating environments and deal with Deutsch's Fallacies of Networking to assist in massively parallel processing (master-worker pattern) - and cannot help but say, pretty cool...

In a nutshell

ComputeCycle is looking to provide a solution to massively parallel tasks that can handle the complexities of a fully distributed grid. Fully distributed, in my mind is not relying on a single farm of processors, such as Sun Grid provides today, but instead looking to allow farms like Sun Grid, in house cycle stealing, Internet cycle stealing to deliver a coherent computing platform. Whereas the ComputeServer project provides a master-worker framework for processing against the Sun Grid, the ComputeCycle team is looking to expand to support workload management across a federation of providers, using Jini and true distributed service semantics.

Componentry

Using GlassFish for integration with NetBeans... Jini gives them JERI, Reggie and JavaSpaces, and then their own frameworks.... this pattern seems to repeat itself frequently ;). And interestingly Groovy... Michael Henderson "Groovy killed the XML File". To help with respect to the configuration across the environment... it's syntax is remarkably close to what Jini Services need.

Security Model

"KeyRing" based authentication - authorized/verified access to keypairs using keytool files (SSL Encryption of all communications... intra and extra grid network)

Enforce the use of a SecurityManager

Master Session has a session space... delivers isolation to valid key holders, provides a separate protection domain

Dynamic Configuration, Activation and Provisioning

Groovy enabled ConfigurationProvider (Jini provider)

2 phase configuration... 1) generic worker service, 2) flavors applied(provisioned) during grid session activation

Distributed Resource Management

Jini leasing model & session model

sessions have expirations and are managed by a "GridManager", but the sessions reside in JavaSpaces (Outrigger) w/ their enforceable isolation.

Federate structure supported w/ a GlobalSesison as the root... this enables very dynamic coupling of sessions to the resources, and session master to the workers

Lookup Service(Reggie).... to find resources

Grid Manager.... service controls - activate, grow/shring, recover, checkpoint, shutdown, all global signals, knowledgeable of all participating workers through a Session TupleSpace (which the Grid Manager instantiates and then gives a reference back to the Session Master.

The workers monitor the space for work to be done, and does a transactional take.

Failure/Fault Management:

workers are naturally fault tolerant, but how do we maintain the 2n/n+1 availability .. this starts to put the requirements for mahalo (a distributed transaction management technology) to allow for transactional updates made for the replication needed to establish high-availability.

One typical question is one of dynamic work binding - how do we let non Java/Jini tools to submit work? They use GlassFish (heavyweight/permanent services) and JNDI to enable the vending of lookup proxies through web services/https in order to allow for firewall penetration.

I think that the approach is very promising, and would encourage anyone interested to take a look!

Related Topics >>