Shoal Dynamic Clustering

The Shoal Framework provides a cluster communication and event mechanisms. Shoal's  core service (GMS), enables an application to dynamically become a member of  a predefined cluster, and as such, is subscribed to cluster events, such as :

- Member join, planned shutdown, failures
- Recovery member selection
- Automated delegated recovery initiation

A cluster member is also able to broadcast messages to an individual member or all members of the cluster(see Messaging). In addition to messaging, members can share data using GMS's Shared Cache (see DistributedStateCache).

The main goal for the shoal project to provide a clustering framework for the GlassFish project, however it does not preclude it from being used for purposes, such as the foundation for a grid deployment, or a deployment of a highly available service, etc (I would be interested in hearing about other use cases).

Shoal/GMS utilizes the JxtaManagement component (a JXTA based group service provider) for dynamic cluster configuration, formation, and monitoring.  This component is broken out as follows :

• NetworkManager.  Given a instance and group name, NetworkManager uses a SHA-1 hash to encode the cluster GroupID, and NodeID, in addition it also defines a set of predefined communication identifiers which are used for formation, monitoring and messaging.  In addition, an application may pass additional configuration parameters, such as bootstrapping addresses to facilitate cross sub-net and firewall communication.
• SystemAdvertisement. An extensible XML document describing system characteristics (HW/SW configuration. CPU load would be a nice extension). This information is exchanged during cluster formation, and monitoring.  It is envisioned that such information would serve at the foundation of a Grid  framework.
• MasterNode. Is a lightweight protocol allowing a set of nodes to discover one another, and autonomously elect a master for the cluster.  The protocol is resilient to multi node collisions and employs an autonomous mechanism to avoid network chatter to resolve collisions.
• ClusterView. Maintains an ordered view of the Cluster
  
• HealthMonitor.  Is a  lightweight protocol allowing a set of nodes to monitor the health of a cluster. The HealthMonitor relies on a tunable heart beat, which is acted upon by the MasterNode to notify the group of failures, and by other members to elect a new master if the master node fails.
• ClusterManger. Manages lifecycle of this SPI
  

By using JXTA as the foundation for the ClusterManager, shoal is able to :

• Use logical name cluster and node addressing (requires unique naming)
  
• Achieve near Zero configuration for cross sub-net and firewall connectivity
• Inherit dynamic transport selection without application intervention (multicast vs. unicast)
• Inherit dynamic route repair
  
• Enables mobility, same name different physical addresses
• Automatic rerouting on/to available interfaces on multi-homed nodes
  
• Inherit traffic scoping to cluster members
• Establish secure end-to-end channels

for additional blogs on the project, see blog entries by Shreedhar, Bernard, Traversat and Masood Mortazvi