Java: Binary Compatibility - more than meets the eye

You were a fool, Joseph Curwen, to fancy that a mere visual identity would be enough. Why didn't you think of the speech and the voice and the handwriting? It hasn't worked, you see, after all.

- "The Case of Charles Dexter Ward", H.P. Lovecraft

Just a quick entry before Thanksgiving, this time concerning binary compatibility for Java.

Most of the time when a method is altered it's easy to spot when a change is not backward compatible with the prior version. After all, if the prior method took a String as its argument and now takes List<String> instead, clearly any client code calling the method will need to be updated. But what happens when you loosen the restrictions, for instance you originally required a HashMap and you want to change the method to require just a Map interface? If you've not encountered this situation before, you may find the answer suprising: your client's code will not run unless they first recompile with your updated code.

To illustrate things, we'll begin with a Cultist interface:

Cultist

public interface Cultist {

  public String chant();
}

Next up is a concrete instance of the interface:

CthulhuCultist

public class CthulhuCultist implements Cultist{

  public String chant() {
    return "Ia! Cthulhu R'lyeh";
  }

  @Override
  public String toString() {
    return "Cthulhu cultist";
  }
}

Now we introduce a poorly designed utility class who's method doesn't take advantage of the interface:

CultistService

public class CultistService {

  static public String getChant(CthulhuCultist cultist) {
    return cultist.chant();
  }
}

Finally, a class that uses the service:

CultistConfusion

public class CultistConfusion {

  public static void main(String[] args) {
    CthulhuCultist cultist = new CthulhuCultist();
    System.out.println("Cultist is: " + cultist);
    
    CultistService cs = new CultistService();
    String chant = cs.getChant(cultist);
    System.out.println("Chant is: " + chant);
  }
}

Compiling everything and running CultistConfusion yields the following:

codefhtagn: javac *.java
codefhtagn: java CultistConfusion
Cultist is: Cthulhu cultist
Chant is: Ia! Cthulhu R'lyeh

So far so good. Now let's update CultistService so that the method signature uses the Cultist interface (as it should have all along):

public class CultistService {

  static public String getChant(Cultist cultist) {
    return cultist.chant();
  }
}

If we recompile just CultistService and then try to run the main class again we see the following:

codefhtagn: javac CultistService.java
codefhtagn: java CultistConfusion
Cultist is: Cthulhu cultist
Exception in thread "main" java.lang.NoSuchMethodError: CultistService.getChant(LCthulhuCultist;)Ljava/lang/String;
 at CultistConfusion.main(CultistConfusion.java:10)

NoSuchMethodError: clearly things are no longer working for the client. However, if we just recompile the client (with no other code changes) then everything works again:

codefhtagn: javac CultistConfusion.java
codefhtagn: java CultistConfusion
Cultist is: Cthulhu cultist
Chant is: Ia! Cthulhu R'lyeh

So the change we made to CultistService is fully compatible with our client, provided we recompile the client before we attempt to run it.

To gain a better understanding of what's going on here we need to take a look at the bytecode that was generated the first time we compiled CultistConfusion.java:

codefhtagn: javap -c CultistConfusion
... lines omitted...
   42: aload_1
   43: invokevirtual #14; //Method CultistService.getChant:(LCthulhuCultist;)Ljava/lang/String;
   46: astore_3
... lines omitted...

Take note of the fact that the argument for the getChant method includes the
class for the argument. This is not a coincidence or mistake; it is the behavior that is required as per the Java Language Specification. Therefore, it should then come as no suprise that things fail when we run the code with an updated CultistService that lacks a getChant(CthulhuCultist) method. A look at the bytecode for the recompiled CultistConfusion.java shows the correct expected call and hence why things work begin to work again:

codefhtagn: javap -c CultistConfusion
... lines omitted...
   42: aload_1
   43: invokevirtual #14; //Method CultistService.getChant:(LCultist;)Ljava/lang/String;
   46: astore_3
... lines omitted...

An important lesson to learn from this is that you should never just drop a newer version of a jar file (or compiled class file) into a runtime environment and expect existing code to "just work." At a minimum, you should recompile your source code any time a dependency has changed as well; in addition take care to ensure that the classes and jars on the runtime classpath are the same classes and jars that were used to compile the code. If you are maintaining a class or library that others use and you need to change a method signature, be aware that changing the type of an argument for a method from a concrete class to an interface for the class will break binary compatibility for your clients. Instead, consider adding the new method and deprecating the old (had we done that with CultistService then CustistConfusion would have run without needing to recompile).

On a final note: I personally first encountered this issue when figuring out why some TestNG tests for a project would compile fine in Eclipse and yet threw NoSuchMethodError exceptions when I went to run them in Eclipse. Adding to the confusion was the fact that the tests compiled and ran just fine from the command line (via Maven). After some digging it turns out that the version of Eclipse I was running (in conjuction with the M2E Eclipse plugin) was compiling the tests with one version of TestNG and running them with another version and there had been method changes similar to what we did with CultistService between the two versions of TestNG.