Joining Multiple Coherence Clusters

"I do not recall distinctly when it began, but it was months ago."

"Nyarlathotep" - H.P. Lovecraft

At my previous job I was given the following problem to consider: is it possible for a Java application (with just a single JVM) to connect to more than one Coherence cluster? In this context, the word "connect" means that the application actually joins the clusters in question, rather than simply works through a proxy such as provided by Coherence*Extend. I was able to determine that the answer was "yes", however the solution was not as straightforward as one might think. It's not possible to just load two Coherence instances into the JVM by calling a constructor with the necessary parameters, as Coherence makes extensive use of system properties and default configuration files in lieu of defining constructors for a client to call. In addition Coherence uses the Singleton pattern in conjuction with several key classes. That said, there is a solution and that's what this post is all about.

Part I: Setting up the clusters

Before discussing my solution, we'll need two running Coherence clusters to work with. To get these up and running all that is required (other than Coherence) are some configuration files (which we'll need later on for the client as well). Here's a look at what we will be working with:

server
|-- coherence.jar
|-- cthulhu-cache.xml
|-- yogSothoth-cache.xml

The coherence.jar file is the standard distribution jar from Oracle (version 3.4 in my case). The other two files are the cache configuration files for each of the clusters we will be setting up. Here is the configuration for the first cluster:

<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
  <caching-scheme-mapping>
    <cache-mapping>
      <cache-name>cthulhuCache</cache-name>
      <scheme-name>cthulhuCache</scheme-name>
    </cache-mapping>
  </caching-scheme-mapping>
  <caching-schemes>
   <distributed-scheme>
      <scheme-name>cthulhuCache</scheme-nameme>
      <service-name>cthulhuCache</service-name>
      <autostart>true</autostart>
      <backing-map-scheme>
        <read-write-backing-map-scheme>
          <internal-cache-scheme>
            <local-scheme>
            </local-scheme>
          </internal-cache-scheme>
          <read-only>false</read-only>
        </read-write-backing-map-scheme>
        <autostart>true</autostart>
      </backing-map-scheme>
    </distributed-scheme>
  </caching-schemes>
</cache-config>

Some things worth noting are:

• There is just one cache service, cthulhuCache. It is distributed (moot in this case as we will be running with just one node) and will store all data in memory.
  
• The cache is set to autostart once it has been referenced.
  

The other cluster's configuration is very similar; below are shown the only changes:

<caching-scheme-mapping>
    <cache-mapping>
      <cache-name>yogsothothCache</cache-name>
      <scheme-name>yogsothothCache</scheme-name>
    </cache-mapping>
  </caching-scheme-mapping>
  <caching-schemes>
   <distributed-scheme>
      <scheme-name>yogsothothCache</scheme-name>
      <service-name>yogsothothCache</service-name>

To actually start up a cluster and add data to its cache we are going to utilize the self contained console application that comes included with the coherence.jar file. For our first cluster we issue the following command to get it started:

codefhtagn/server: java -Dtangosol.coherence.cacheconfig=cthulhu-cache.xml \
> -Dtangosol.coherence.cluster=Cthulhu \
> -Dtangosol.coherence.clusteraddress=224.1.1.1 \
> -Dtangosol.coherence.clusterport=12345 \
> -jar coherence.jar

• The first system property, tangosol.coherence.cacheconfig, specifies the cache configuration for this cluster.
  
• The remaining three properties are used to set up a triplet (cluster name, multicast address, port) that will uniquely identify this cluster.
  

After entering the above command the following appears (note the highlighted lines showing the cluster identity property values):

2011-05-12 09:09:18.415/0.276 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/codefhtagn/server/coherence.jar!/tangosol-coherence.xml"
2011-05-12 09:09:18.418/0.279 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/codefhtagn/server/coherence.jar!/tangosol-coherence-override-dev.xml"
2011-05-12 09:09:18.418/0.279 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2011-05-12 09:09:18.421/0.282 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.4.2/411
 Grid Edition: Development mode
Copyright (c) 2000-2009 Oracle. All rights reserved.

2011-05-12 09:09:18.857/0.718 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2011-05-12 09:09:22.062/3.923 Oracle Coherence GE 3.4.2/411 <Info> (thread=Cluster, member=n/a): Created a new cluster "Cthulhu" with Member(Id=1, Timestamp=2011-05-12 09:09:18.789, Address=192.168.1.101:8088, MachineId=26981, Location=process:5741, Role=CoherenceConsole, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) UID=0xC0A801650000012FE4F826C569651F98
SafeCluster: Name=Cthulhu

Group{Address=224.1.1.1, Port=12345, TTL=4}

MasterMemberSet
  (
  ThisMember=Member(Id=1, Timestamp=2011-05-12 09:09:18.789, Address=192.168.1.101:8088, MachineId=26981, Location=process:5741, Role=CoherenceConsole)
  OldestMember=Member(Id=1, Timestamp=2011-05-12 09:09:18.789, Address=192.168.1.101:8088, MachineId=26981, Location=process:5741, Role=CoherenceConsole)
  ActualMemberSet=MemberSet(Size=1, BitSetCount=2
    Member(Id=1, Timestamp=2011-05-12 09:09:18.789, Address=192.168.1.101:8088, MachineId=26981, Location=process:5741, Role=CoherenceConsole)
    )
  RecycleMillis=120000
  RecycleSet=MemberSet(Size=0, BitSetCount=0
    )
  )

Services
  (
  TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.1.101:8088}, Connections=[]}
  ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.4, OldestMemberId=1}
  )

Map (?): 

At this point we are now operating in interactive mode with the console application. The next steps are to first switch from the default (unnamed) cache to our cthlhuCache cache (this will trigger reading of the configuration file we specified), add an entry to the cache, and then (just to be safe) verify the cache was updated.

Map (?): cache cthulhuCache
2011-05-11 18:06:09.844/874.448 Oracle Coherence GE 3.4.2/411 <Info> (thread=main, member=1): Loaded cache configuration from file "/codefhtagn/server/cthulhu-cache.xml"
2011-05-11 18:06:09.976/874.580 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:cthulhuCache, member=1): Service cthulhuCache joined the cluster with senior service member 1
<distributed-scheme>
  <scheme-name>cthulhuCache</scheme-name>
  <service-name>cthulhuCache</service-name>
  <autostart>true</autostart>
  <backing-map-scheme>
    <read-write-backing-map-scheme>
      <internal-cache-scheme>
        <local-scheme/>
      </internal-cache-scheme>
      <read-only>false</read-only>
    </read-write-backing-map-scheme>
    <autostart>true</autostart>
  </backing-map-scheme>
</distributed-scheme>

Map (cthulhuCache): 

Map (cthulhuCache): put status sleeping
null

Map (cthulhuCache): get status
sleeping

Map (cthulhuCache): 

We are now done setting up the first cluster; we will leave this window open since if we close it we will terminate the cluser. After opening a new terminal window we can start up the second cluster by giving similar commands, all we need to change is the values for the various system properties.

codefhtagn/server: java -Dtangosol.coherence.cacheconfig=yogSothoth-cache.xml \
> -Dtangosol.coherence.cluster=YogSothoth \
> -Dtangosol.coherence.clusteraddress=224.2.2.2 \
> -Dtangosol.coherence.clusterport=23456 \
> -jar coherence.jar

Here's an excerpt from the output:

SafeCluster: Name=YogSothoth

Group{Address=224.2.2.2, Port=23456, TTL=4}

As with the first cluster, accessing our yogsothothCache cache and placing a value in it is easy:

Map (?): cache yogsothothCache
2011-05-14 10:52:58.857/33.832 Oracle Coherence GE 3.4.2/411 <Info> (thread=main, member=1): Loaded cache configuration from file "/codefhtagn/server/yogSothoth-cache.xml"
2011-05-14 10:52:58.955/33.930 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:yogsothothCache, member=1): Service yogsothothCache joined the cluster with senior service member 1
<distributed-scheme>
  <scheme-name>yogsothothCache</scheme-name>
  <service-name>yogsothothCache</service-name>
  <autostart>true</autostart>
  <backing-map-scheme>
    <read-write-backing-map-scheme>
      <internal-cache-scheme>
        <local-scheme/>
      </internal-cache-scheme>
      <read-only>false</read-only>
    </read-write-backing-map-scheme>
    <autostart>true</autostart>
  </backing-map-scheme>
</distributed-scheme>

Map (yogsothothCache): put key 31415
null

Map (yogsothothCache): get key
31415

Map (yogsothothCache):

Our work setting up the clusters is now done. To recap:

• Each cluster has a distinct identifying triplet: (name, multicast address, port).
  
• Each cluster has its own distinct distributed cache with one (key,value) entry.
  
• The (key,value) entry is unique for each cluster (this will prove useful in confirming that our client really is connecting to each of these clusters).
  

Part II: The Application

Joining one cluster

Before tackling the issue of joining two clusters, let's take a look what's involved in getting an application to join a single cluster. We won't do anything fancy, however our approach will illustrate some of the challenges to be faced in joining more than one cluster. The files that we will be using are structured as follows:

client
|-- coherence.jar
|-- cthulhu-cache.xml
|-- cultist
    |-- Invocation.java

The coherence.jar and cthulhu-cache.xml files are identical to the ones used to get the first cluster up and running. The remaining file, Invocation.java, is our application which will connect to the Cthulhu cluster and print out the value (from the cthulhuCache) for the status key :

package cultist;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;

public class Invocation{

  public static void main(String... args) {
    NamedCache cache = CacheFactory.getCache("cthulhuCache");

    // Retrieve a single value and print to console
    System.out.println("Value for status is: " + cache.get("status"));
  }
}

Compiling this is easy:

codefhtagn/client: javac -cp coherence.jar cultist/Invocation.java 

Running this is slightly more verbose as we need to provide the same system properties that were used to start the Cthulhu cluster as well as one additional property, tangosol.coherence.distributed.localstorage, which is used to designate that this cluster node should not participate in storing data for the cluster that it is joining. All told the command looks like:

codefhtagn/client: java -cp .:coherence.jar \
> -Dtangosol.coherence.cacheconfig=cthulhu-cache.xml \
> -Dtangosol.coherence.cluster=Cthulhu \
> -Dtangosol.coherence.clusteraddress=224.1.1.1 \
> -Dtangosol.coherence.clusterport=12345 \
> -Dtangosol.coherence.distributed.localstorage=false \
> cultist/Invocation

Below is the output from this command, the highlighted lines show that yes, it did join the Cthulhu cluster as well as retrieve the correct status value:

2011-05-12 12:57:20.686/0.272 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/codefhtagn/client/coherence.jar!/tangosol-coherence.xml"
2011-05-12 12:57:20.689/0.275 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/codefhtagn/client/coherence.jar!/tangosol-coherence-override-dev.xml"
2011-05-12 12:57:20.690/0.276 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2011-05-12 12:57:20.693/0.279 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.4.2/411
 Grid Edition: Development mode
Copyright (c) 2000-2009 Oracle. All rights reserved.

2011-05-12 12:57:20.848/0.434 Oracle Coherence GE 3.4.2/411 <Info> (thread=main, member=n/a): Loaded cache configuration from file "/codefhtagn/client/cthulhu-cache.xml"
2011-05-12 12:57:21.136/0.722 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2011-05-12 12:57:21.344/0.930 Oracle Coherence GE 3.4.2/411 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2011-05-12 12:57:21.143, Address=192.168.1.101:8089, MachineId=26981, Location=process:6022, Role=CultistCthulhuCultist, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) joined cluster "Cthulhu" with senior Member(Id=1, Timestamp=2011-05-12 12:56:47.979, Address=192.168.1.101:8088, MachineId=26981, Location=process:6021, Role=CoherenceConsole, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4)
2011-05-12 12:57:21.351/0.937 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Member 1 joined Service cthulhuCache with senior member 1
2011-05-12 12:57:21.471/1.058 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:cthulhuCache, member=2): Service cthulhuCache joined the cluster with senior service member 1
2011-05-12 12:57:21.483/1.069 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:cthulhuCache, member=2): Service cthulhuCache: received ServiceConfigSync containing 259 entries
Value for status is: sleeping 

The problem with scaling this approach to join two clusters is that there is no way to give a single system property multiple values at one time. Sure, we could cheat and declare a new property whose value was a comma seperated list, but then we'd need to read that value, parse it, and then somehow pass those values to multiple Coherence instances in the JVM. Oh, and we'd also need to overcome Coherence's usage of the Singleton pattern.

Joining two clusters

It turns out that there is an easier way to approach the problem. Rather than focus on the system properties, let's instead tackle the Singleton issue first. Recall that that when the JVM identifies a class it relies on a triplet of information:

• the class name.
  
• the package for the class.
  
• the class loader instance used to load that class.
  

In short, if a class Foo in package bar is loaded first by class loader bazOne and then again by class loader bazTwo, then you end up with two instance of Foo.class in the JVM.

Given that working with class loaders in Java can be confusing, I'm going to defer giving details for later. Instead, let's look at the finished application. Below is the directory structure for the application files. In addition to an updated client directory, there are several new directories that I'll talk about later. For completeness I've also included the files that were used to set up the clusters.

.
|-- client
|   |-- cultist
|   |   |-- CthulhuCultist.java
|   |   |-- CultistFactory.java
|   |   |-- Invocation.java
|   |   |-- YogSothothCultist.java
|   |-- util
|       |-- InvertedClassLoader.java
|-- clientImplCthulhu
|   |-- configCthulhu
|   |   |-- cthulhu-cache.xml
|   |   |-- tangosol-coherence-override.xml
|   |-- cultist
|       |-- CthulhuCultistImpl.java
|-- clientImplYogSothoth
|   |-- configYogSothoth
|   |   |-- tangosol-coherence-override.xml
|   |   |-- yogSothoth-cache.xml
|   |-- cultist
|       |-- YogSothothCultistImpl.java
|-- coherence.jar
|-- server
    |-- coherence.jar
    |-- cthulhu-cache.xml
    |-- yogSothoth-cache.xml

For now, let's just look at just the updated Invocation class:

package cultist;

public class Invocation{

  public static void main(String... args) {
    CthulhuCultist cthulhuCultist = 
      CultistFactory.getCultist(CthulhuCultist.class, CthulhuCultistImpl.class, 
        "clientImplCthulhu", "configCthulhu");
    YogSothothCultist yogSothothCultist = 
      CultistFactory.getCultist(YogSothothCultist.class, YogSothothCultistImpl.class, 
        "clientImplYogSothoth", "configYogSothoth");
    System.out.println("Value for status: " + cthulhuCultist.call("status"));
    System.out.println("Value for key: " + yogSothothCultist.summon("key"));
  }

}

Unlike the original Invocation, this version does not reference the cache (or anything else involving Coherence) directly. Instead, it relies on two classes (interfaces actually), CthulhuCultist and YogSothtohCultist, that are obtained from a CultistFactory. Each of these interfaces defines a method that, when called, interacts with the appropriate cluster's cache (more details later). To compile and run this code is not too complicated:

codefhtagn: javac -cp client:clientImplCthulhu:clientImplYogSothoth:coherence.jar \
client/util/InvertedClassLoader.java \
client/cultist/CthulhuCultist.java \
client/cultist/YogSothothCultist.java \
clientImplCthulhu/cultist/CthulhuCultistImpl.java \
clientImplYogSothoth/cultist/YogSothothCultistImpl.java \
client/cultist/CultistFactory.java \
client/cultist/Invocation.java

codefhtagn: java -Dtangosol.coherence.distributed.localstorage=false \
-cp .:client:clientImplCthulhu:clientImplYogSothoth \
cultist/Invocation

Before looking at the output, some things to note:

• The compile time classpath includes the coherence.jar file as well as the three separate folders holding our source code.
  
• The runtime classpath does not include the coherence.jar file.
  
• The runtime classpath does include the root folder so that the coherence.jar file (as well as the other top level folders) can be loaded as resources dynamically at runtime.
  
• The only system property set is tangosol.coherence.distributed.localstorage.
  

As far as output goes:

2011-05-14 10:53:07.707/0.308 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/codefhtagn/coherence.jar!/tangosol-coherence.xml"
2011-05-14 10:53:07.711/0.312 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/codefhtagn/coherence.jar!/tangosol-coherence-override-dev.xml"
2011-05-14 10:53:07.713/0.314 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "file:/codefhtagn/clientImplCthulhu/configCthulhu/tangosol-coherence-override.xml"
2011-05-14 10:53:07.718/0.319 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.4.2/411
 Grid Edition: Development mode
Copyright (c) 2000-2009 Oracle. All rights reserved.

2011-05-14 10:53:07.892/0.493 Oracle Coherence GE 3.4.2/411 <Info> (thread=main, member=n/a): Loaded cache configuration from resource "file:/codefhtagn/clientImplCthulhu/configCthulhu/cthulhu-cache.xml"
2011-05-14 10:53:08.245/0.846 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2011-05-14 10:53:08.454/1.055 Oracle Coherence GE 3.4.2/411 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2011-05-14 10:53:08.255, Address=192.168.1.122:8090, MachineId=27002, Location=process:8744, Role=CultistInvocation, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) joined cluster "Cthulhu" with senior Member(Id=1, Timestamp=2011-05-14 10:52:20.648, Address=192.168.1.122:8088, MachineId=27002, Location=process:8742, Role=CoherenceConsole, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4)
2011-05-14 10:53:08.461/1.062 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Member 1 joined Service cthulhuCache with senior member 1
2011-05-14 10:53:08.600/1.201 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:cthulhuCache, member=2): Service cthulhuCache joined the cluster with senior service member 1
2011-05-14 10:53:08.610/1.211 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:cthulhuCache, member=2): Service cthulhuCache: received ServiceConfigSync containing 259 entries
2011-05-14 10:53:08.750/1.351 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/codefhtagn/coherence.jar!/tangosol-coherence.xml"
2011-05-14 10:53:08.754/1.355 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/codefhtagn/coherence.jar!/tangosol-coherence-override-dev.xml"
2011-05-14 10:53:08.755/1.356 Oracle Coherence 3.4.2/411 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "file:/codefhtagn/clientImplYogSothoth/configYogSothoth/tangosol-coherence-override.xml"
2011-05-14 10:53:08.758/1.359 Oracle Coherence 3.4.2/411 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.4.2/411
 Grid Edition: Development mode
Copyright (c) 2000-2009 Oracle. All rights reserved.

2011-05-14 10:53:08.879/1.480 Oracle Coherence GE 3.4.2/411 <Info> (thread=main, member=n/a): Loaded cache configuration from resource "file:/codefhtagn/clientImplYogSothoth/configYogSothoth/yogSothoth-cache.xml"
2011-05-14 10:53:09.198/1.799 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2011-05-14 10:53:09.405/2.006 Oracle Coherence GE 3.4.2/411 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2011-05-14 10:53:09.206, Address=192.168.1.122:8091, MachineId=27002, Location=process:8744, Role=CultistInvocation, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) joined cluster "YogSothoth" with senior Member(Id=1, Timestamp=2011-05-14 10:52:25.646, Address=192.168.1.122:8089, MachineId=27002, Location=process:8743, Role=CoherenceConsole, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4)
2011-05-14 10:53:09.411/2.012 Oracle Coherence GE 3.4.2/411 <D5> (thread=Cluster, member=n/a): Member 1 joined Service yogsothothCache with senior member 1
2011-05-14 10:53:09.533/2.134 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:yogsothothCache, member=2): Service yogsothothCache joined the cluster with senior service member 1
2011-05-14 10:53:09.544/2.145 Oracle Coherence GE 3.4.2/411 <D5> (thread=DistributedCache:yogsothothCache, member=2): Service yogsothothCache: received ServiceConfigSync containing 259 entries
Value for status: sleeping
Value for key: 31415

The final two highlighted lines tell it all: we got the correct value for each cluster's cache using the known keys. In addition, other highlighted lines reveal:

• The files in clientImplCthulhu/configChthulu are referenced (lines 6 and 13) in order to join the Cthulhu cluster (line 15).
  
• The files in clientImplYogSothtoth/configYogSothoth are referenced (lines 21 and 28) in order to join the YogSothoth cluster (line 30).
  

Part III: Application Details

The simple stuff

As already noted, the Invocation class does not interact directly with Coherence, instead it works through two interfaces. Let's start with these interfaces:

package cultist;

public interface CthulhuCultist {

  String call(String key);
}

package cultist;

public interface YogSothothCultist {

  Integer summon(String key);
}

Nothing complicated here: each interface has one method designed to retreive a value for a given key. Now for their implementations:

package cultist;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;

import cultist.CthulhuCultist;

public class CthulhuCultistImpl implements CthulhuCultist {

  private final NamedCache cache;

  public CthulhuCultistImpl() {
    cache = CacheFactory.getCache("cthulhuCache");
  }

  public String call(String key) {
    return (String) cache.get(key);
  }
}

package cultist;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;

import cultist.YogSothothCultist;

public class YogSothothCultistImpl implements YogSothothCultist {

  private final NamedCache cache;

  public YogSothothCultistImpl() {
    cache = CacheFactory.getCache("yogsothothCache");
  }

  public Integer summon(String key) {
    return (Integer) cache.get(key);
  }
}

Each of these should look very familiar since the code that is used is based on what was in the original Invocation class. The only changes are:

• The NamedCache object is now a final field, not a local variable, and is set by the no-arg constructor.
  
• The method implementation returns the cache value instead of printing it to the console.
  

So far so good. The question you should be asking now is: how does each of these instances know about the cluster it is going to connect to? The answer lies with the afore mentioned configuration files, so let's look at them next.

First off, both the cthulhu-cache.xml and yogsothoth-cache.xml configuration files are identical to the ones being used to run the two clusters (re-use is good!). However, the tangosol-coherence-override.xml file that appears twice is not the same file repeated twice. Here's what each copy looks like:

<?xml version='1.0'?>
<coherence>
  <cluster-config>
    <member-identity>
      <cluster-name>Cthulhu</cluster-name>
    </member-identity>
    <multicast-listener>
      <address>224.1.1.1</address>
      <port>12345</port>
    </multicast-listener>
  </cluster-config>
  <configurable-cache-factory-config>
    <class-name>com.tangosol.net.DefaultConfigurableCacheFactory
    </class-name>
    <init-params>
      <init-param>
        <param-type>java.lang.String</param-type>
        <param-value>configCthulhu/cthulhu-cache.xml</param-value>
      </init-param>
    </init-params>
  </configurable-cache-factory-config>
</coherence>

<?xml version='1.0'?>
<coherence>
  <cluster-config>
    <member-identity>
      <cluster-name>YogSothoth</cluster-name>
    </member-identity>
    <multicast-listener>
      <address>224.2.2.2</address>
      <port>23456</port>
    </multicast-listener>
  </cluster-config>
  <configurable-cache-factory-config>
    <class-name>com.tangosol.net.DefaultConfigurableCacheFactory
    </class-name>
    <init-params>
      <init-param>
        <param-type>java.lang.String</param-type>
        <param-value>configYogSothoth/yogSothoth-cache.xml</param-value>
      </init-param>
    </init-params>
  </configurable-cache-factory-config>
</coherence>

Looking at the highlighted lines you will see the missing command line values. In addition, there are references to the two cache configuration files. This raises the next question: what needs to be done in order for these files to be read? The answer is: nothing. When Coherence starts up it will scan the classpath for a file named tangosol-coherence-override.xml and, if it finds one, then it will read that file and apply the various settings. Aha, you say: we have two copies of tangosol-coherence-override.xml, so how do we know which one will be read? The answer to that question will take us closer to the whole class loader portion of the solution.

First, recall that we obtained our CthulhuCultist and YogSothothCultist instances via a CultistFactory, so perhaps we should take a look at that class next:

package cultist;
import java.net.URL;
import util.InvertedClassLoader;

public class CultistFactory {

  public static <T> T getCultist(Class<T> cultistInterface, Class<? extends T> cultistImpl, 
      String clientImplDirectory, String overrideDirectory) {
    URL coherenceURL = 
      Thread.currentThread().getContextClassLoader().getResource("coherence.jar");
    URL cultistClientRootURL = 
      Thread.currentThread().getContextClassLoader().getResource(clientImplDirectory + "/");
    URL overrideURL = 
      Thread.currentThread().getContextClassLoader().getResource(overrideDirectory + "/");
    T cultist = null;
    try {
       InvertedClassLoader cl = 
         new InvertedClassLoader(coherenceURL, cultistClientRootURL, overrideURL);
       cultist = cl.selfLoad(cultistImpl).loadClass(cultistImpl).newInstance();
    }
    catch (Exception e) {
      // fail!
      e.printStackTrace();
      System.exit(-1);
    }
    return cultist;
  }
}

The getCultist method takes four arguments:

• The first argument is a generic used to determine the return type of the method.
  
• The second argument provides a concrete implementation for the first arguments type.
  
• The third argument provides the (root) folder name where the compiled class files for the second argument reside.
  
• The fourth argument provides the name of the directory holding the configuration files for the cluster that will be connected to.
  

URLs are obtained for all but the first argument and are then used to construct an instance of an InvertedClassLoader which is then called to obtain an instance of the first argument.

Warning: Class loader ahead!

This now brings up to the InvertedClassLoader class which does the real work. (Disclosure: I did not personally write this class myself. What appears below is excerpted from a larger class (with the same name) that appears as one of the internal framework classes used at Overstock.com. I have received permission to include the code here and by extension anyone who wishes to use this code may freely copy it for their own use.).

package util;

import java.net.URL;
import java.net.URLClassLoader;
import java.util.HashSet;
import java.util.Set;

public class InvertedClassLoader extends URLClassLoader {

  public InvertedClassLoader(URL... urls) {
    super(urls, Thread.currentThread().getContextClassLoader());
  }

  public InvertedClassLoader selfLoad(Class<?> classToNotDelegate) {
    classesToNotDelegate.add(classToNotDelegate.getName());
    return this;
  }

  @Override public Class<?> loadClass(String className, boolean resolve) throws ClassNotFoundException {
    if (shouldNotDelegate(className)) {
      Class<?> clazz = findClass(className);
      if (resolve) {
        resolveClass(clazz);
      }
      return clazz;
    }
    else {
      return super.loadClass(className, resolve);
    }
  }

  public <T> Class<T> loadClass(Class<? extends T> classToLoad) throws ClassNotFoundException {
    final Class<?> clazz = loadClass(classToLoad.getName());
    @SuppressWarnings("unchecked") final Class<T> castedClass = (Class<T>) clazz;
    return castedClass;
  }

  private boolean shouldNotDelegate(String className) {
    if (classesToNotDelegate.contains(className)) {
      return true;
    }
    return false;
  }

  private final Set<String> classesToNotDelegate = new HashSet<String>();
}

This class is an extension of the standard URLClassLoader class, with one method, loadClass, being overridden and one new public method, selfLoad, being added. The original goal of this class as stated in its JavaDocs is:

A class loader which will not follow the usual java class laoder delegation model for certain classes. This is useful when needing to load a class which depends on jars which one does not want in the general class path, while the class itself implements an interface which is in the general class path. For example, if foo.jar relies on a version of xerces.jar which might conflict with other classes in our application, but foo.jar contains an interface com.foo.Service which does not reference anything in xerces.jar, and a implementation com.foo.ServiceImpl which does, the following code can give us an instance of ServiceImpl:

Service service = 
    new InvertedClassLoader(fooJarUrl, oldXercesJarUrl)
          .selfLoad(ServiceImpl.class)
          .loadClass(ServiceImpl.class)
          .newInstance();

Note that foo.jar will need to be in the main application class path, while the old xerces.jar will not be in the main application's class path. Also note that it is important to treat the created instance as a Service instance, rather than ServiceImpl instance. This is because ServiceImpl will be loaded by a different class loader than the application's class loader, and thus will not be cast-compatible with the application's class loader. However, since we allow the Service class to be loaded by the parent class loader (i.e. the application's class loader), the instance will be cast-compatible with Service.

In our case, the issue is not so much incompatibility of a given jar file (in our case coherence.jar) with our main application code, so much as it is that we want to load the contents of that jar more than one time (namely, once per cluster being joined).

The selfLoad method is crucial, here is its JavaDoc entry:

Instruct the loader to load the specified class itself, rather than loading it from the parent class loader. Any classes which will rely on classes found in the urls passed to the constructor should be indicated here.

In order to get this to work an InvertedClassLoader instance keeps track of all the classes that it is supposed to self load (via calls to selfLoad) in an internal collection; it then checks this collection when loadClass is called to decide if it should first delegate to the parent class loader (via super.loadClass) or not.

How it all ties together

Using the InvertedClassLoader we are able to override the default class loader delegation model in a programatic fashion. Namely, even though the application class loader has already loaded an instance of both CthlhuCultistImpl and YogSothothCultistImpl (since they are referenced in the Invocation code), we override the delegation model (via the call to selfLoad) to allow the InvertedClassLoader instance local to the CultistFactory.getCultist method to load the requested interface implementation. Of course, in order to do this the InvertedClassLoader needs to know where the relevant files are, hence our use of the URLs created from the arguments passed to the getCultist method.

When it comes time to load the Coherence related code referenced in the CthulhuClientImpl and YogSothothImpl classes the default class loader delegation works in our favor, as the JVM will first delegate to the parent class loader (i.e. the class loader for the class currently being loaded) which in this case is the InvertedClassLoader instance. Since we have different InvertedClassLoader instances for each call to getCultist we end up loading the Coherence classes more than once. Each time the classes in coherence.jar are loaded, Coherence ends up searching for a tangosol-coherence-override.xml file; since we control what files are looked at via the URLs constructed from the parameters passed to the getCultist method, we are able to completely determine which version of tangosol-coherence-override.xml is read.

On a final note, this solution is in some ways a work in progress. One problem in particular is that we do nothing to keep from creating more than one instance of a given cluster (as it stands right now, the CultistFactory will create one instance for every call to getCultist). However, I wasn't asked to come up with the best solution, just an answer as to whether it could be done or not.