Backwards compatibility with Java bytecode editing
Table of Contents
1 Description
This page describes how to edit java bytecode to build a class with two methods of the same name and arguments, but with different return values.
It shows that we can change the return type of a method in a class and still be backwards source and binary compatible.
It also shows that we can change the return type of a method in an interface and still be binary compatible, and why that is not enough.
2 News
- 26 Jun 2015: initial
3 Backwards compatibility
A library is backwards compatible when you can use a newer version of the library without having to change code that used a previous version.
Adding a new public method to a final class is backwards compatible. Adding a new non-default method to an interface is not backwards compatible.
A library is source compatible when you can compile with a new version of the library without having to change old code.
A library is binary compatible when you can run with a new version of the library without having to recompile old code.
A library changes a method from public void foo(Integer number)
to
public void foo(Number number)
. This change is source compatible
because Number
is a superclass of Integer
. But this change is not
binary compatible because the compiled java looks for a method that
takes exactly an Integer
, not a superclass of an Integer
.
A library removes a constant public static final int ANSWER = 42;
.
This change is binary compatible because compiled java references 42
directly and not through ANSWER
. But this change is not source
compatible because the compiler can no longer find the ANSWER
field.
4 Changing the return type of a method in a class
We want to change IntegerIterator
such that it implements Iterator<Integer>
.
4.1 The change
import java.util.NoSuchElementException; // version 1 public final class IntegerIterator { private final int[] ints; private int i; public IntegerIterator(int[] ints) { this.ints = ints; this.i = 0; } public int next() { if (!hasNext()) { throw new NoSuchElementException(); } return ints[i++]; } public boolean hasNext() { return i < ints.length; } }
import java.util.NoSuchElementException; import java.lang.UnsupportedOperationException; import java.util.Iterator; // version 2 public final class IntegerIterator implements Iterator<Integer>{ private final int[] ints; private int i; public IntegerIterator(int[] ints) { this.ints = ints; this.i = 0; } @Override public Integer next() { if (!hasNext()) { throw new NoSuchElementException(); } return ints[i++]; } @Override public boolean hasNext() { return i < ints.length; } @Override public void remove() { throw new UnsupportedOperationException(); } }
4.2 The problem
The problem with version 2 is that we remove the method public int
next()
which breaks binary compatibility. It does not break source
compatibility because the java compiler does not care about return
type when calling methods, and it will automatically unbox Integer
to int
.
public final class Usage1 { public static final void main(String[] args) { for (IntegerIterator it = new IntegerIterator(new int[]{1, 2, 3}); it.hasNext();) { System.out.println(it.next()); } } }
Let us review what happens when we compile and use the program
Usage1
with version 1 and version 2 of IntegerIterator
.
Case | Compile | Run | Outcome | Reason |
---|---|---|---|---|
Working | version 1 | version 1 | Works | |
Binary compatible | version 1 | version 2 | Runtime error | Cannot find int next() |
Forward compatible | version 2 | version 1 | Runtime error | Cannot find Integer next() |
Source compatible | version 2 | version 2 | Works | The compiler does not care about return type when calling methods, it calls the version 2 method Integer next() |
Forward compatibility means having some code compiled with a new version, and being able to run that code using an older version. This is the opposite of backwards binary compatibility. We do not care about forward compatibility.
We already have that source compatibility, but we also want binary compatibility to work.
4.3 The solution
To fix the problem we need to create a version of IntegerIterator
that has public Integer next()
(to satisfy Iterator<Integer>
), but
also has public int next()
from version 1.
// version 2.1 public final class IntegerIterator implements Iterator<Integer>{ private final int[] ints; private int i; public IntegerIterator(int[] ints) { this.ints = ints; this.i = 0; } public int next() { return next().intValue(); // call to "public Integer next()" } @Override public Integer next() { if (!hasNext()) { throw new NoSuchElementException(); } return ints[i++]; } @Override public boolean hasNext() { return i < ints.length; } @Override public void remove() { throw new UnsupportedOperationException(); } }
Trying to compile version 2.1 give us several errors. The main one is that java does not allow two methods with the same name and arguments in the same class, even when their return types are different.
v2.1/IntegerIterator.java:19: error: method next() is already defined in class IntegerIterator public Integer next() { ^ v2.1/IntegerIterator.java:5: error: IntegerIterator is not abstract and does not override abstract method next() in Iterator public final class IntegerIterator implements Iterator<Integer>{ ^ v2.1/IntegerIterator.java:14: error: next() in IntegerIterator cannot implement next() in Iterator public int next() { ^ return type int is not compatible with Integer where E is a type-variable: E extends Object declared in interface Iterator v2.1/IntegerIterator.java:15: error: int cannot be dereferenced return next().intValue(); ^ 4 errors
However, in java bytecode it is allowed to have two methods with the same name and arguments but different return values.
Rather than rely on a java compiler to create our version 2.1
IntegerIterator.class
, we can build it ourselves with the BCEL
library:
import org.apache.bcel.*; import org.apache.bcel.classfile.*; import org.apache.bcel.generic.*; public final class IntegerIteratorCreator { public static final void main(String[] args) throws Exception { JavaClass master = Repository.lookupClass("IntegerIterator"); // version 2 ClassGen cg = new ClassGen(master); InstructionList il = new InstructionList(); // public int next() MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, Type.INT, Type.NO_ARGS, new String[0], "next", "IntegerIterator", il, cg.getConstantPool()); InstructionFactory factory = new InstructionFactory(cg); // The bytecode instructions for "return next().intValue();" // where next() is the next method that returns an Integer. il.append(factory.ALOAD_0); il.append(factory.createInvoke("IntegerIterator", "next", Type.getReturnType("Ljava/lang/Integer;"), Type.NO_ARGS, Constants.INVOKEVIRTUAL)); il.append(factory.createInvoke("java.lang.Integer", "intValue", Type.INT,Type.NO_ARGS, Constants.INVOKEVIRTUAL)); il.append(factory.createReturn(Type.INT)); mg.setMaxStack(); cg.addMethod(mg.getMethod()); // write out IntegerIterator.class String output = master.getClassName() + ".class"; cg.getJavaClass().dump(output); } }
We now have a version 2.1 IntegerIterator.class
file with our two
next()
methods. Note that we also have public java.lang.Object
next();
, this method is added by the regular java compiler.
danamlund$ javap IntegerIterator.class Compiled from "IntegerIterator.java" public final class IntegerIterator implements java.util.Iterator<java.lang.Integer> { public IntegerIterator(int[]); public java.lang.Integer next(); public boolean hasNext(); public void remove(); public java.lang.Object next(); public int next(); }
Let us again review what happens when we compile and use the program
Usage1
with version 1, version 2, and version 2.1 of IntegerIterator
.
Case | Compile | Run | Outcome | Reason |
---|---|---|---|---|
Working | version 1 | version 1 | Works | |
Binary compatible | version 1 | version 2 | Runtime error | Cannot find int next() |
Binary compatible 2 | version 1 | version 2.1 | Works | Finds the int next() we added to bytecode |
Forward compatible | version 2 | version 1 | Runtime error | Cannot find Integer next() |
Source compatible | version 2 | version 2 | Works | The compiler does not care about return type when calling methods, it calls the version 2 method Integer next() |
Binary compatible 3 | version 2 | version 2.1 | Works | Finds Integer next() |
Forward compatible 2 | version 2.1 | version 1 | Runtime error | Cannot find Integer next() |
Source compatible 2 | version 2.1 | version 2 | Works | The compiler chooses Integer next() and ignores int next() without complaint |
Source compatible 3 | version 2.1 | version 2.1 | Works | The compiler chooses Integer next() and ignores int next() without complaint |
Case Binary compatible 2 and 3 shows that our change is now binary compatible. We also see that our change is source compatible from Case Source compatible 3.
A downside to handcrafting class files is that developing and
publishing the library becomes more difficult. It is a good idea to
create script that generates IntegerIterator.class
and bundles it by
itself in a jar file. The library source can then depend on
IntegerIterator.jar
without having a version of
IntegerIterator.java
lying around.
Not having an IntegerIterator.java
prevents it being edited by
accident and prevents a non-backwards compatible version of
IntegerIterator.class
from being included in a future release.
5 Changing the return type of a method in an interface
We now attempt the more difficult problem of expanding an interface by
having it implement Iterator<Integer>
. We can use a default method
to add void remove()
and still be source and binary compatible. But we have
to remove int next()
before we can add Integer next()
which
Iterator<Integer>
requires.
5.1 The change
// version 1 public interface IntegerIterator { int next(); boolean hasNext(); }
import java.lang.UnsupportedOperationException; import java.util.Iterator; // version 2 public interface IntegerIterator extends Iterator<Integer> { @Override Integer next(); @Override boolean hasNext(); @Override default void remove() { throw new UnsupportedOperationException(); } }
5.2 The problem
The problem with version 2 is that we removed the method public int
next()
which breaks binary and source compatibility.
Version 2 from the previous section was still source compatible. This difference is because the java compiler does not care about return type when calling methods, but it does care about them when verifying that a declared method overrides a method.
import java.util.NoSuchElementException; public final class Usage1 { public static final void main(String[] args) { for (IntegerIterator it = new IntegerIteratorImpl(new int[]{1, 2, 3}); it.hasNext();) { System.out.println(it.next()); } } private static final class IntegerIteratorImpl implements IntegerIterator { private final int[] ints; private int i; public IntegerIteratorImpl(int[] ints) { this.ints = ints; this.i = 0; } @Override public int next() { if (!hasNext()) { throw new NoSuchElementException(); } return ints[i++]; } @Override public boolean hasNext() { return i < ints.length; } } }
import java.util.NoSuchElementException; import java.util.Iterator; import java.lang.Iterable; import java.lang.UnsupportedOperationException; public final class Usage2 { public static final void main(String[] args) { for (Integer i : foreach(new IntegerIteratorImpl(new int[]{1, 2, 3}))) { System.out.println(i); } } private static <E> Iterable<E> foreach(final Iterator<E> iterator) { return new Iterable<E>() { @Override public Iterator<E> iterator() { return iterator; } }; } private static final class IntegerIteratorImpl implements IntegerIterator { private final int[] ints; private int i; public IntegerIteratorImpl(int[] ints) { this.ints = ints; this.i = 0; } @Override public Integer next() { if (!hasNext()) { throw new NoSuchElementException(); } return ints[i++]; } @Override public boolean hasNext() { return i < ints.length; } @Override public void remove() { throw new UnsupportedOperationException(); } } }
Let us review what happens when we compile and use the programs
Usage1
and Usage2
with version 1 and version 2 of IntegerIterator
.
Case | Compile | Run | Outcome | Reason |
---|---|---|---|---|
Usage1 |
||||
Working | version 1 | version 1 | Works | |
Binary compatible | version 1 | version 2 | Runtime error | Cannot find int next() |
Forward compatible | version 2 | version 1 | Compile error | IntegerIteratorImpl does not override Integer next() |
Source compatible | version 2 | version 2 | Compile error | IntegerIteratorImpl does not override Integer next() |
Usage2 |
||||
Forward compatible 1 | version 1 | version 1 | Compile error | Usage2 uses version 2 |
Forward compatible 2 | version 1 | version 2 | Compile error | Usage2 uses version 2 |
Forward compatible 3 | version 2 | version 1 | Runtime error | IntegerIterator does not implement Iterator |
Working | version 2 | version 2 | Works |
The Forward compatible 1 and 2 cases for Usage2
are forward source
compatibility. Forward source compatibility means code written for a
new version of a library will compile with an old version.
We do not care about forward compatibility. The goal is to get binary and source compatibility. But source compatibility will be a problem.
5.3 The solution to binary compatibility
The problem with binary compatibility is that int next()
is
missing. We need to create a new version of IntegerIterator
containing both Integer next()
and int next()
methods.
import java.lang.UnsupportedOperationException; import java.util.Iterator; // version 2.1 public interface IntegerIterator extends Iterator<Integer> { default int next() { return next().intValue(); // call to "default Integer next()" } @Override default Integer next() { return Integer.valueOf(next()); // call to "default int next()" } @Override boolean hasNext(); @Override default void remove() { throw new UnsupportedOperationException(); } }
Trying to compile version 2.1 give us several errors. The main one is that java does not allow two methods with the same name and arguments, even when their return types are different.
Another problem is that the two next()
methods form an infinite loop
each calling the other when an implementation does not override either
of the methods. This is not a problem in practice because we will
either compile against version 1 where int next()
is mandatory, or
we will compile against version 2 where Integer next()
is
mandatory. We never compile against version 2.1 because the java
compiler complains about two methods with the same name and arguments
when declaring methods (this was not a problem in the previous
section because there we were just calling methods).
Like before, we build version 2.1 IntegerIterator.class
ourselves
using the BCEL library:
import java.util.*; import org.apache.bcel.*; import org.apache.bcel.classfile.*; import org.apache.bcel.generic.*; public final class IntegerIteratorCreator { public static final void main(String[] args) throws Exception { JavaClass master = Repository.lookupClass("IntegerIterator"); // version 2 ClassGen cg = new ClassGen(master); // Remove existing non-default next method for (Method method : cg.getMethods()) { if ("next".equals(method.getName())) { cg.removeMethod(method); } } Type integerType = Type.getReturnType("Ljava/lang/Integer;"); // default int next() { return next().intValue(); } (where next() returns Integer) { InstructionList il = new InstructionList(); MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, Type.INT, Type.NO_ARGS, new String[0], "next", "IntegerIterator", il, cg.getConstantPool()); InstructionFactory factory = new InstructionFactory(cg); il.append(factory.ALOAD_0); il.append(factory.createInvoke("IntegerIterator", "next", integerType, Type.NO_ARGS, Constants.INVOKEINTERFACE)); il.append(factory.createInvoke("java.lang.Integer", "intValue", Type.INT, Type.NO_ARGS, Constants.INVOKEVIRTUAL)); il.append(factory.createReturn(Type.INT)); mg.setMaxStack(); cg.addMethod(mg.getMethod()); } // default Integer next() { return Integer.valueOf(next()); } (where next() returns int) { InstructionList il = new InstructionList(); MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, integerType, Type.NO_ARGS, new String[0], "next", "IntegerIterator", il, cg.getConstantPool()); InstructionFactory factory = new InstructionFactory(cg); il.append(factory.ALOAD_0); il.append(factory.createInvoke("IntegerIterator", "next", Type.INT, Type.NO_ARGS, Constants.INVOKEINTERFACE)); il.append(factory.createInvoke("java.lang.Integer", "valueOf", integerType, new Type[]{ Type.INT }, Constants.INVOKESTATIC)); il.append(factory.createReturn(integerType)); mg.setMaxStack(); cg.addMethod(mg.getMethod()); } // write out IntegerIterator.class String output = master.getClassName() + ".class"; cg.getJavaClass().dump(output); } }
We now have a version 2.1 IntegerIterator.class
file with our two
default next()
methods. Note that we no longer have public
java.lang.Object next();
, because we now have an interface instead of
a class.
danamlund$ javap IntegerIterator.class Compiled from "IntegerIterator.java" public interface IntegerIterator extends java.util.Iterator<java.lang.Integer> { public abstract boolean hasNext(); public void remove(); public int next(); public java.lang.Integer next(); }
Let us again review what happens when we compile and use the programs
Usage1
and Usage2
with version 1, version 2, and version 2.1 of IntegerIterator
.
Case | Compile | Run | Outcome | Reason |
---|---|---|---|---|
Usage1 |
||||
Working | version 1 | version 1 | Works | |
Binary compatible 1 | version 1 | version 2 | Runtime error | Cannot find int next() |
Binary compatible 2 | version 1 | version 2.1 | Works | Finds the int next() we added to bytecode |
Forward compatible 1 | version 2 | version 1 | Compile error | IntegerIteratorImpl does not override Integer next() |
Source compatible 1 | version 2 | version 2 | Compile error | IntegerIteratorImpl does not override Integer next() |
Source compatible 2 | version 2 | version 2.1 | Compile error | IntegerIteratorImpl does not override Integer next() |
Forward compatible 2 | version 2.1 | version 1 | Compile error | IntegerIteratorImpl does not override Integer next() |
Source compatible 3 | version 2.1 | version 2 | Compile error | IntegerIteratorImpl does not override Integer next() |
Source compatible 4 | version 2.1 | version 2.1 | Compile error | IntegerIteratorImpl does not override Integer next() |
Usage2 |
||||
Old library version | version 1 | version 1 | Compile error | Usage2 uses version 2 |
Forward compatible 1 | version 1 | version 2 | Compile error | Usage2 uses version 2 |
Forward compatible 2 | version 1 | version 2.1 | Compile error | Usage2 uses version 2 |
Forward compatible 3 | version 2 | version 1 | Runtime error | IntegerIterator does not implement Iterator |
Working | version 2 | version 2 | Works | |
Binary compatible | version 2 | version 2.1 | Works | Runtime does not care about the "extra" method int next() |
Forward compatible 4 | version 2.1 | version 1 | Compile error | Compiler tries to have Integer next() override int next() |
Source compatible 1 | version 2.1 | version 2 | Compile error | Compiler tries to have Integer next() override int next() |
Source compatible 2 | version 2.1 | version 2.1 | Compile error | Compiler tries to have Integer next() override int next() |
Case Usage1
Binary compatible 2 and case Usage2
Binary compatible
shows that version 2.1 is binary compatible.
However, Case Usage2
Source compatible 1 and 2 shows that version
2.1 is not source compatible. In fact, it not only lack backwards
source compatibility (case Usage2
Source compatible 1), it also
lacks same-version source compatibility (case Usage2
Source
compatible 2).
In the next section we will show that lacking source compatibility is a serious problem.
5.4 The bad solution to source compatibility
We are not source compatible because version 2.1 cannot be used when
compiling. The compiler thinks that the Integer next()
method in
IntegerIteratorImpl
is trying to override int next()
in
IntegerIterator
, and it complains that the types do not match. This
complaint sounds like it comes from a verification phase, I do not
know what would happen if the compiler got past the verification phase
and tried to compile the code.
Not being able to compile with version 2.1 is a big problem for
library developers. First of all it means they have to release two
versions of the library: a library-runtime.jar
and a
library-compile.jar
.
library-runtime.jar
will have version 2.1, and work fine because we
are binary compatible.
library-compile.jar
is a problem. We cannot add version 2.1 because
that cannot compile client code that uses IntegerIterator
. We could
add version 2 to library-compile.jar
but then all client code that
uses version 1 will have to be rewritten to fix compile
errors. Instead of rewriting code, clients could split their code-base
into code that uses version 1, and code that uses version 2. The split
code base can be merged together again when the code is built.
These are the best solutions I can think of, and they are both
terrible. A far better (better meaning backwards compatible) approach
is to keep version 1 as it is, and introduce version 2 as a new class
IntegerIterator2
.