Version française disponible

This post summarizes the work we have achieved within my team to migrate our micro-services from Java 8 to Java 11 for the website Hotels.com.

In summary, for each of the services we own, we have made the following steps:

• Make the code compile with Java 11
• Run the Java 11 compatible service on Java 8
• Run the service on Java 11

In reality, we had some extra steps because when we began the migration, Java 11 was not released yet, we could only use Java 10.
The assumption was if the code compiles on Java 10, there will not be so much work to migrate to Java 11 as the biggest change about modularity was introduced with Java 9 and the Jigsaw project. Thankfully, it was the case!

Star Wars lightsaber upgrade...

1. Make the code compile with Java 11

This was the longest part. Indeed, we had to bump the version up of most of the frameworks and tools we are using. Especially, we had to handle the migration from Spring Boot 1 to 2 and Spring 4 to 5. As these are major versions, we had to fix a couple of breaking changes.

Spring Boot

For Spring Boot 2, the official migration guides for Spring Boot 2.0 and Spring Boot 2.1 are well written and detailed.

• The profile loading has evolved
• The relaxed binding of properties is a bit less relaxed
• Some properties were renamed and others made unavailable (e.g. security.basic.enabled property that has to be replaced with a WebSecurityConfigurerAdapter)
• Some endpoints were renamed (e.g. the actuator healthchecks)
• The bean overriding is now disabled by default, which is something we were using in our integration tests, we had to re-enable it with the new property spring.main.allow-bean-definition-overriding.

Spring

The migration to Spring 5 was quite straightforward from a code point of view with few minor changes. The hard part was related to the fact the project is legacy and we had to deal with complex Spring XML configuration and the migration to Dropwizard Metrics 4.

Misc

Few frameworks are still not compatible with Java 9+ as they are not actively maintained by the community. In our case, we had to find a workaround for Cassandra Unit. We did not want to invest time to change of testing framework as we are planning to move to DynamoDB.

We also had to deal with the Maven dependency hell because some required dependencies were bringing old dependencies not compatible with Java 9+. In most of the cases, adding some exclusions in the POM solved it.

Local environment

Locally, we added a simple set of aliases to our bash profile file to switch between the Java versions.

alias setjava8="export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/"
alias setjava10="export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home/"
alias setjava11="export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home/"

On IntelliJ, changing of Java version can be done from the project settings (“Project SDK” dropdown).

CI environment

For the Continuous Integration side (we are using Bamboo), we updated the agent to use Java 11.
We noticed that it’s not possible to have different versions of the agent for the branch plans and the master plan as the plan configuration is global. It means updating the agent to Java 11 will break master if someone else was pushing a change to master (e.g. a new feature or a bug fix totally independent from the Java 11 migration).
To mitigate this issue and avoid a red build, it was important for us to make sure the project was compiling with Java 11 and that all tests were passing locally before updating the agent in order to merge the Java migration pull request quickly. Another option would be to set back the agent to Java 8 temporarily once the branch plan was green on Java 11 without forgetting to set it back to Java 11 just before the merge.

2. Run the Java 11 compatible service on Java 8

Once everything was fixed, merged and the master build was green, we had to ensure the Java 11 compatible version was running fine in our test environments. Basically make sure nothing was broken… We had unit, integration and end-to-end tests so our level of confidence was quite high. Just to be safe, we did some extra exploratory and manual testing on the API with some exotic and edge case requests to make sure it was behaving correctly. We also ensure the logs and Grafana dashboards were fine.

The next step was to push the new version in production. The service was still running with Java 8 even if the code was Java 11 compatible (and compiled), we did not want to introduce too many changes at the same time, we don’t like risky releases after all. We handle this release with extra care because of the multiple refactoring and versions bump up. After looking at the Grafana dashboards for a few days, comparing metrics before and after the migration, it shows that all went well.

3. Run the service on Java 11

The ultimate goal was to run the service on Java 11. In theory, it will be as simple as updating the Docker file to use the Java 11 image and push the artifact in production. However, in practice it was not that simple…

First of all, we had to update the Java JVM arguments (the -d64 parameter is deprecated and will prevent the service to start, we also had to update the GC logs argument).

Then, we quickly realized that the service logs had disappeared from Splunk in our on-premises production environment, the logs were actually showing up in the future while it was working fine on AWS. We had to update the logback config to fix this “temporal distortion” ;) by updating the date pattern from %d{ISO8601} to %d{yyyy-MM-dd'T'HH:mm:ss.SSSZ,UTC}.

We had another weird error VerifyError: Bad type on operand stack that arises during the deployment to production, AppDynamics was preventing some instances from starting due to some exotic bytecode manipulation. For some reason, it was fine on prod-canary, then started to fail after a successful deployment on a couple of instances! We had to disable AppDynamics, which was fine as we are not using this tool in our team.

As we were moving to Java 11, we also had to update some of our Grafana dashboards to reflect the use of a new Garbage Collector – G1.

Conclusion

Today, 3 services providing the user notifications using Spring Boot 2, and 1 service providing the website header & footer using Spring 5 are running smoothly on Java 11, it has been several weeks now. They are using the default G1 Garbage Collector and we did not encounter any weird behavior related to memory footprint or any other performance issue. On the other hand, we did not see any improvement in our response time. But now we are using a Java LTS (Long Term Support) release, the migration was a success.

What’s next? Java 12 is going to be released in March 2019. At this time, we still don’t know if we will use this version or wait for the next Java LTS. It will probably depend on which features are included.

 Version française disponible

Simon Ritter gave a presentation about JDK 9 at QCon London.

QCon London 2016 - Project Jigsaw in JDK 9 - Modularity comes to Java

Java Compatibility Policy

He started with an interesting fact: Java has a general and longstanding compatibility policy.
If an application uses only supported APIs on version N of Java, it should work on version N+1, even without recompilation.
That’s why, to date there are:

• 23 classes, 18 interfaces and 379 methods that have been deprecated
• And none have been removed.

JDK 9 brings breaking changes

The JDK has been heavier and heavier over the years. With the version 9, things are going to change. For the first time, JDK 9 will bring incompatibilities.

• A small number of supported API will be removed
• The binary structure of the JRE and JDK will change (see details below)
• It is not allowed to use a single underscore _ as a variable name. It is now a reserved keyword.
• There will be a new version scheme so that it is easier to distinguish major, minor, and security-update releases, e.g. Java 9.1.2.1. The old versioning format is difficult to understand, e.g. 8h51 and 8u60 are not very meaningful.

The JDK structure is changing. The JRE folder is removed, tools.jar and rt.jar are no longer present, as there was some duplication between them.

Pre-JDK 9 structure

To have a complete understanding of the pre-JDK 9 structure, you can read JDK 8 and JRE File Structure.

JDK 9 structure

The JDK 9 structure is much cleaner.

JEP 260 proposal is at the heart of JDK 9 and Jigsaw.

The idea is to make most of the JDK’s internal APIs inaccessible by default but leave a few critical, widely-used internal APIs accessible, until supported replacements exist for all or most of their functionality.

The goal is to encapsulate all deprecated API in a module, then eventually getting rid of it in the future (JDK 10?).

Since JDK 8, a command line tool exists to analyze the dependencies of your JAR or class: jdeps. Give it a try! It can be useful to discover if you are using JDK-internal APIs, you should avoid using these dependencies as they may be removed in the future.

Jigsaw and modules

We talk about module just before, but what is it?
Jigsaw brings modularity to the JDK. It makes Java more scalable and flexible, it also improves security, maintainability and performance.
The JDK is split up into modules, and then we take only what we need.

Module

A module is a grouping of code (collection of packages), it can also contain:

• Native code
• Resources
• Configuration data

My first module

To define a module, you need to create a module-info.java file, including (for example):

module com.company.mymodule { 
  requires com.company.myothermodule; 
  requires java.sql; 
}

You can have transitive dependencies (a bit like Maven) thanks to the keyword public.

module java.sql { 
  requires public java.logging; 
}

This means if I have a dependency on the module java.sql, I will have access to all classes included in java.logging (as long as the types are public).

We now have a module dependency graph.

Package visibility

Use of the keyword exports.

module com.company.myothermodule { 
  exports com.company.myothermodule.alpha;  
  exports com.company.myothermodule.beta; 
}

What is exported is visible, by default it’s not!

In Java 9, we have to redefine the keyword public, it comes in different flavours:

• Public to everyone
• Public, but only to specific modules
• Public, only within a module

Public does not mean necessary accessible anymore, that’s a fundamental change.
You can read the Jigsaw specification to know more.

Compilation and execution with module path

Forget about the classpath parameter, there is a new modulepath parameter (or -mp) in the javac / java command line. Its value is one or more directories that contain modules that are required to compile / execute your application.

Also, note there will be default module for the JAR files that are not module-compatible yet (automatic module).