Backwards compatibility with Java bytecode editing

Table of Contents

1 Description

This page describes how to edit java bytecode to build a class with two methods of the same name and arguments, but with different return values.

It shows that we can change the return type of a method in a class and still be backwards source and binary compatible.

It also shows that we can change the return type of a method in an interface and still be binary compatible, and why that is not enough.

2 News

  • 26 Jun 2015: initial

3 Backwards compatibility

A library is backwards compatible when you can use a newer version of the library without having to change code that used a previous version.

Adding a new public method to a final class is backwards compatible. Adding a new non-default method to an interface is not backwards compatible.

A library is source compatible when you can compile with a new version of the library without having to change old code.

A library is binary compatible when you can run with a new version of the library without having to recompile old code.

A library changes a method from public void foo(Integer number) to public void foo(Number number). This change is source compatible because Number is a superclass of Integer. But this change is not binary compatible because the compiled java looks for a method that takes exactly an Integer, not a superclass of an Integer.

A library removes a constant public static final int ANSWER = 42;. This change is binary compatible because compiled java references 42 directly and not through ANSWER. But this change is not source compatible because the compiler can no longer find the ANSWER field.

4 Changing the return type of a method in a class

We want to change IntegerIterator such that it implements Iterator<Integer>.

4.1 The change

import java.util.NoSuchElementException;
// version 1
public final class IntegerIterator {
    private final int[] ints;
    private int i;

    public IntegerIterator(int[] ints) {
        this.ints = ints;
        this.i = 0;
    }

    public int next() {
        if (!hasNext()) {
            throw new NoSuchElementException();
        }
        return ints[i++];
    }

    public boolean hasNext() {
        return i < ints.length;
    }
}
import java.util.NoSuchElementException;
import java.lang.UnsupportedOperationException;
import java.util.Iterator;
// version 2
public final class IntegerIterator implements Iterator<Integer>{ 
    private final int[] ints;
    private int i;

    public IntegerIterator(int[] ints) {
        this.ints = ints;
        this.i = 0;
    }

    @Override
    public Integer next() {
        if (!hasNext()) {
            throw new NoSuchElementException();
        }
        return ints[i++];
    }

    @Override
    public boolean hasNext() {
        return i < ints.length;
    }

    @Override
    public void remove() {
        throw new UnsupportedOperationException();
    }
}

4.2 The problem

The problem with version 2 is that we remove the method public int next() which breaks binary compatibility. It does not break source compatibility because the java compiler does not care about return type when calling methods, and it will automatically unbox Integer to int.

public final class Usage1 {
    public static final void main(String[] args) {
        for (IntegerIterator it = new IntegerIterator(new int[]{1, 2, 3}); it.hasNext();) {
            System.out.println(it.next());
        }
    }
}

Let us review what happens when we compile and use the program Usage1 with version 1 and version 2 of IntegerIterator.

Case Compile Run Outcome Reason
Working version 1 version 1 Works  
Binary compatible version 1 version 2 Runtime error Cannot find int next()
Forward compatible version 2 version 1 Runtime error Cannot find Integer next()
Source compatible version 2 version 2 Works The compiler does not care about return type when calling methods, it calls the version 2 method Integer next()

Forward compatibility means having some code compiled with a new version, and being able to run that code using an older version. This is the opposite of backwards binary compatibility. We do not care about forward compatibility.

We already have that source compatibility, but we also want binary compatibility to work.

4.3 The solution

To fix the problem we need to create a version of IntegerIterator that has public Integer next() (to satisfy Iterator<Integer>), but also has public int next() from version 1.

// version 2.1
public final class IntegerIterator implements Iterator<Integer>{ 
    private final int[] ints;
    private int i;

    public IntegerIterator(int[] ints) {
        this.ints = ints;
        this.i = 0;
    }

    public int next() {
        return next().intValue(); // call to "public Integer next()"
    }

    @Override
    public Integer next() {
        if (!hasNext()) {
            throw new NoSuchElementException();
        }
        return ints[i++];
    }

    @Override
    public boolean hasNext() {
        return i < ints.length;
    }

    @Override
    public void remove() {
        throw new UnsupportedOperationException();
    }
}

Trying to compile version 2.1 give us several errors. The main one is that java does not allow two methods with the same name and arguments in the same class, even when their return types are different.

v2.1/IntegerIterator.java:19: error: method next() is already defined in class IntegerIterator
    public Integer next() {
                   ^
v2.1/IntegerIterator.java:5: error: IntegerIterator is not abstract and does not override abstract method next() in Iterator
public final class IntegerIterator implements Iterator<Integer>{ 
             ^
v2.1/IntegerIterator.java:14: error: next() in IntegerIterator cannot implement next() in Iterator
    public int next() {
               ^
  return type int is not compatible with Integer
  where E is a type-variable:
    E extends Object declared in interface Iterator
v2.1/IntegerIterator.java:15: error: int cannot be dereferenced
        return next().intValue();
                     ^
4 errors

However, in java bytecode it is allowed to have two methods with the same name and arguments but different return values.

Rather than rely on a java compiler to create our version 2.1 IntegerIterator.class, we can build it ourselves with the BCEL library:

import org.apache.bcel.*;
import org.apache.bcel.classfile.*;
import org.apache.bcel.generic.*;

public final class IntegerIteratorCreator {
    public static final void main(String[] args) throws Exception {  
        JavaClass master = Repository.lookupClass("IntegerIterator"); // version 2

        ClassGen cg = new ClassGen(master);
        InstructionList il = new InstructionList();
        // public int next()
        MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, Type.INT, Type.NO_ARGS, new String[0],
                                     "next", "IntegerIterator", il, cg.getConstantPool());
        InstructionFactory factory = new InstructionFactory(cg);

        // The bytecode instructions for "return next().intValue();"
        // where next() is the next method that returns an Integer.
        il.append(factory.ALOAD_0);
        il.append(factory.createInvoke("IntegerIterator", "next", 
                                       Type.getReturnType("Ljava/lang/Integer;"),
                                       Type.NO_ARGS, Constants.INVOKEVIRTUAL));
        il.append(factory.createInvoke("java.lang.Integer", "intValue",
                                       Type.INT,Type.NO_ARGS, Constants.INVOKEVIRTUAL));
        il.append(factory.createReturn(Type.INT));
        mg.setMaxStack();
        cg.addMethod(mg.getMethod());

        // write out IntegerIterator.class
        String output = master.getClassName() + ".class";
        cg.getJavaClass().dump(output);
    }
}

We now have a version 2.1 IntegerIterator.class file with our two next() methods. Note that we also have public java.lang.Object next();, this method is added by the regular java compiler.

danamlund$ javap IntegerIterator.class 
Compiled from "IntegerIterator.java"
public final class IntegerIterator implements java.util.Iterator<java.lang.Integer> {
  public IntegerIterator(int[]);
  public java.lang.Integer next();
  public boolean hasNext();
  public void remove();
  public java.lang.Object next();
  public int next();
}

Let us again review what happens when we compile and use the program Usage1 with version 1, version 2, and version 2.1 of IntegerIterator.

Case Compile Run Outcome Reason
Working version 1 version 1 Works  
Binary compatible version 1 version 2 Runtime error Cannot find int next()
Binary compatible 2 version 1 version 2.1 Works Finds the int next() we added to bytecode
Forward compatible version 2 version 1 Runtime error Cannot find Integer next()
Source compatible version 2 version 2 Works The compiler does not care about return type when calling methods, it calls the version 2 method Integer next()
Binary compatible 3 version 2 version 2.1 Works Finds Integer next()
Forward compatible 2 version 2.1 version 1 Runtime error Cannot find Integer next()
Source compatible 2 version 2.1 version 2 Works The compiler chooses Integer next() and ignores int next() without complaint
Source compatible 3 version 2.1 version 2.1 Works The compiler chooses Integer next() and ignores int next() without complaint

Case Binary compatible 2 and 3 shows that our change is now binary compatible. We also see that our change is source compatible from Case Source compatible 3.

A downside to handcrafting class files is that developing and publishing the library becomes more difficult. It is a good idea to create script that generates IntegerIterator.class and bundles it by itself in a jar file. The library source can then depend on IntegerIterator.jar without having a version of IntegerIterator.java lying around.

Not having an IntegerIterator.java prevents it being edited by accident and prevents a non-backwards compatible version of IntegerIterator.class from being included in a future release.

5 Changing the return type of a method in an interface

We now attempt the more difficult problem of expanding an interface by having it implement Iterator<Integer>. We can use a default method to add void remove() and still be source and binary compatible. But we have to remove int next() before we can add Integer next() which Iterator<Integer> requires.

5.1 The change

// version 1
public interface IntegerIterator {
    int next();
    boolean hasNext();
}
import java.lang.UnsupportedOperationException;
import java.util.Iterator;
// version 2
public interface IntegerIterator extends Iterator<Integer> { 
    @Override
    Integer next();

    @Override
    boolean hasNext();

    @Override
    default void remove() {
        throw new UnsupportedOperationException();
    }
}

5.2 The problem

The problem with version 2 is that we removed the method public int next() which breaks binary and source compatibility.

Version 2 from the previous section was still source compatible. This difference is because the java compiler does not care about return type when calling methods, but it does care about them when verifying that a declared method overrides a method.

import java.util.NoSuchElementException;

public final class Usage1 {

    public static final void main(String[] args) {
        for (IntegerIterator it = new IntegerIteratorImpl(new int[]{1, 2, 3}); it.hasNext();) {
            System.out.println(it.next());
        }
    }

    private static final class IntegerIteratorImpl implements IntegerIterator {
        private final int[] ints;
        private int i;

        public IntegerIteratorImpl(int[] ints) {
            this.ints = ints;
            this.i = 0;
        }

        @Override
        public int next() {
            if (!hasNext()) {
                throw new NoSuchElementException();
            }
            return ints[i++];
        }

        @Override
        public boolean hasNext() {
            return i < ints.length;
        }
    }
}
import java.util.NoSuchElementException;
import java.util.Iterator;
import java.lang.Iterable;
import java.lang.UnsupportedOperationException;

public final class Usage2 {

    public static final void main(String[] args) {
        for (Integer i : foreach(new IntegerIteratorImpl(new int[]{1, 2, 3}))) {
            System.out.println(i);
        }
    }

    private static <E> Iterable<E> foreach(final Iterator<E> iterator) {
        return new Iterable<E>() {
            @Override
            public Iterator<E> iterator() {
                return iterator;
            }
        };
    }

    private static  final class IntegerIteratorImpl implements IntegerIterator {
        private final int[] ints;
        private int i;

        public IntegerIteratorImpl(int[] ints) {
            this.ints = ints;
            this.i = 0;
        }

        @Override
        public Integer next() {
            if (!hasNext()) {
                throw new NoSuchElementException();
            }
            return ints[i++];
        }

        @Override
        public boolean hasNext() {
            return i < ints.length;
        }

        @Override
        public void remove() {
            throw new UnsupportedOperationException();
        }
    }
}

Let us review what happens when we compile and use the programs Usage1 and Usage2 with version 1 and version 2 of IntegerIterator.

Case Compile Run Outcome Reason
Usage1        
Working version 1 version 1 Works  
Binary compatible version 1 version 2 Runtime error Cannot find int next()
Forward compatible version 2 version 1 Compile error IntegerIteratorImpl does not override Integer next()
Source compatible version 2 version 2 Compile error IntegerIteratorImpl does not override Integer next()
Usage2        
Forward compatible 1 version 1 version 1 Compile error Usage2 uses version 2
Forward compatible 2 version 1 version 2 Compile error Usage2 uses version 2
Forward compatible 3 version 2 version 1 Runtime error IntegerIterator does not implement Iterator
Working version 2 version 2 Works  

The Forward compatible 1 and 2 cases for Usage2 are forward source compatibility. Forward source compatibility means code written for a new version of a library will compile with an old version.

We do not care about forward compatibility. The goal is to get binary and source compatibility. But source compatibility will be a problem.

5.3 The solution to binary compatibility

The problem with binary compatibility is that int next() is missing. We need to create a new version of IntegerIterator containing both Integer next() and int next() methods.

import java.lang.UnsupportedOperationException;
import java.util.Iterator;
// version 2.1
public interface IntegerIterator extends Iterator<Integer> { 

    default int next() {
        return next().intValue(); // call to "default Integer next()"
    }

    @Override
    default Integer next() {
        return Integer.valueOf(next()); // call to "default int next()"
    }

    @Override
    boolean hasNext();

    @Override
    default void remove() {
        throw new UnsupportedOperationException();
    }
}

Trying to compile version 2.1 give us several errors. The main one is that java does not allow two methods with the same name and arguments, even when their return types are different.

Another problem is that the two next() methods form an infinite loop each calling the other when an implementation does not override either of the methods. This is not a problem in practice because we will either compile against version 1 where int next() is mandatory, or we will compile against version 2 where Integer next() is mandatory. We never compile against version 2.1 because the java compiler complains about two methods with the same name and arguments when declaring methods (this was not a problem in the previous section because there we were just calling methods).

Like before, we build version 2.1 IntegerIterator.class ourselves using the BCEL library:

import java.util.*;
import org.apache.bcel.*;
import org.apache.bcel.classfile.*;
import org.apache.bcel.generic.*;

public final class IntegerIteratorCreator {

    public static final void main(String[] args) throws Exception {  
        JavaClass master = Repository.lookupClass("IntegerIterator"); // version 2

        ClassGen cg = new ClassGen(master);

        // Remove existing non-default next method
        for (Method method : cg.getMethods()) {
            if ("next".equals(method.getName())) {
                cg.removeMethod(method);
            }
        }

        Type integerType = Type.getReturnType("Ljava/lang/Integer;");

        // default int next() { return next().intValue(); } (where next() returns Integer)
        {
            InstructionList il = new InstructionList();
            MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, Type.INT, Type.NO_ARGS, new String[0],
                                         "next", "IntegerIterator", il, cg.getConstantPool());
            InstructionFactory factory = new InstructionFactory(cg);
            il.append(factory.ALOAD_0);
            il.append(factory.createInvoke("IntegerIterator", 
                                           "next", 
                                           integerType,
                                           Type.NO_ARGS, 
                                           Constants.INVOKEINTERFACE));
            il.append(factory.createInvoke("java.lang.Integer", 
                                           "intValue",
                                           Type.INT,
                                           Type.NO_ARGS, 
                                           Constants.INVOKEVIRTUAL));
            il.append(factory.createReturn(Type.INT));
            mg.setMaxStack();
            cg.addMethod(mg.getMethod());
        }

        // default Integer next() { return Integer.valueOf(next()); } (where next() returns int)
        {
            InstructionList il = new InstructionList();
            MethodGen mg = new MethodGen(Constants.ACC_PUBLIC, 
                                         integerType,
                                         Type.NO_ARGS, new String[0],
                                         "next", "IntegerIterator", il, cg.getConstantPool());
            InstructionFactory factory = new InstructionFactory(cg);
            il.append(factory.ALOAD_0);
            il.append(factory.createInvoke("IntegerIterator", "next", 
                                           Type.INT,
                                           Type.NO_ARGS, 
                                           Constants.INVOKEINTERFACE));
            il.append(factory.createInvoke("java.lang.Integer", "valueOf",
                                           integerType,
                                           new Type[]{ Type.INT }, 
                                           Constants.INVOKESTATIC));
            il.append(factory.createReturn(integerType));
            mg.setMaxStack();
            cg.addMethod(mg.getMethod());
        }


        // write out IntegerIterator.class
        String output = master.getClassName() + ".class";
        cg.getJavaClass().dump(output);
    }
}

We now have a version 2.1 IntegerIterator.class file with our two default next() methods. Note that we no longer have public java.lang.Object next();, because we now have an interface instead of a class.

danamlund$ javap IntegerIterator.class                                     
Compiled from "IntegerIterator.java"
public interface IntegerIterator extends java.util.Iterator<java.lang.Integer> {
  public abstract boolean hasNext();
  public void remove();
  public int next();
  public java.lang.Integer next();
}

Let us again review what happens when we compile and use the programs Usage1 and Usage2 with version 1, version 2, and version 2.1 of IntegerIterator.

Case Compile Run Outcome Reason
Usage1        
Working version 1 version 1 Works  
Binary compatible 1 version 1 version 2 Runtime error Cannot find int next()
Binary compatible 2 version 1 version 2.1 Works Finds the int next() we added to bytecode
Forward compatible 1 version 2 version 1 Compile error IntegerIteratorImpl does not override Integer next()
Source compatible 1 version 2 version 2 Compile error IntegerIteratorImpl does not override Integer next()
Source compatible 2 version 2 version 2.1 Compile error IntegerIteratorImpl does not override Integer next()
Forward compatible 2 version 2.1 version 1 Compile error IntegerIteratorImpl does not override Integer next()
Source compatible 3 version 2.1 version 2 Compile error IntegerIteratorImpl does not override Integer next()
Source compatible 4 version 2.1 version 2.1 Compile error IntegerIteratorImpl does not override Integer next()
Usage2        
Old library version version 1 version 1 Compile error Usage2 uses version 2
Forward compatible 1 version 1 version 2 Compile error Usage2 uses version 2
Forward compatible 2 version 1 version 2.1 Compile error Usage2 uses version 2
Forward compatible 3 version 2 version 1 Runtime error IntegerIterator does not implement Iterator
Working version 2 version 2 Works  
Binary compatible version 2 version 2.1 Works Runtime does not care about the "extra" method int next()
Forward compatible 4 version 2.1 version 1 Compile error Compiler tries to have Integer next() override int next()
Source compatible 1 version 2.1 version 2 Compile error Compiler tries to have Integer next() override int next()
Source compatible 2 version 2.1 version 2.1 Compile error Compiler tries to have Integer next() override int next()

Case Usage1 Binary compatible 2 and case Usage2 Binary compatible shows that version 2.1 is binary compatible.

However, Case Usage2 Source compatible 1 and 2 shows that version 2.1 is not source compatible. In fact, it not only lack backwards source compatibility (case Usage2 Source compatible 1), it also lacks same-version source compatibility (case Usage2 Source compatible 2).

In the next section we will show that lacking source compatibility is a serious problem.

5.4 The bad solution to source compatibility

We are not source compatible because version 2.1 cannot be used when compiling. The compiler thinks that the Integer next() method in IntegerIteratorImpl is trying to override int next() in IntegerIterator, and it complains that the types do not match. This complaint sounds like it comes from a verification phase, I do not know what would happen if the compiler got past the verification phase and tried to compile the code.

Not being able to compile with version 2.1 is a big problem for library developers. First of all it means they have to release two versions of the library: a library-runtime.jar and a library-compile.jar.

library-runtime.jar will have version 2.1, and work fine because we are binary compatible.

library-compile.jar is a problem. We cannot add version 2.1 because that cannot compile client code that uses IntegerIterator. We could add version 2 to library-compile.jar but then all client code that uses version 1 will have to be rewritten to fix compile errors. Instead of rewriting code, clients could split their code-base into code that uses version 1, and code that uses version 2. The split code base can be merged together again when the code is built.

These are the best solutions I can think of, and they are both terrible. A far better (better meaning backwards compatible) approach is to keep version 1 as it is, and introduce version 2 as a new class IntegerIterator2.

Author: Dan Amlund Thomsen

Created: 2019-05-09 Thu 19:53

Validate