With the recent addition of the cf create-buildpack and cf update-buildpack commands, we’ve been looking into changes to how the Java Buildpack is deployed on Cloud Foundry. There is a move afoot to remove the buildpacks from the DEAs in favor of using those commands (with an automated initial install to preserve user-experience) and an outcome of this was the need to more easily create ‘packaged’ buildpacks. In the process of delivering these packages we also realized that we could also deliver “offline” buildpacks. To understand why “offline” buildpacks are interesting, it helps to understand why the buildpack is designed to look for dependencies on the Internet in the first place.
The buildpack uses a number of dependencies (32 of them by last count). Since checking binaries into source control is a sure way to balloon a Git repository (everyone who clones the repository will have to download those binaries for eternity and updates lead to multiple copies in each clone), we had to find another way to get these dependencies into a user’s droplet. For that, we built a repository abstraction. The repository is a network connected collection of URIs with an index that points to versions of each dependency.
By default the repository that the buildpack tries to connect to is hosted on S3, in a region near https://run.pivotal.io (for performance reasons). However, that is not required. The repository abstraction was designed so that with a single, simple change a user could host a repository internally on their own network (like your average enterprise would do). We made it incredibly easy to replicate our repository onto a local network to encourage this behavior.
“But”, you might say, “32 dependencies can’t be that large”. In truth, a single version of them is not (about 180MB), but your average enterprise Java shop uses a collection of versions that is likely to be different than every other enterprise Java shop out there. On top of that, the versions of dependencies that they’re using isn’t likely to change very rapidly (despite the security concerns implied). So, if you total up all of the versions across our entire dependency collection, it weighs in at 6.5GB!! You can see the incredible number of versions in our indexes:
- App Dynamics
- Auto Reconfiguration
- Groovy
- MariaDB JDBC
- OpenJDK
centos6 - OpenJDK
lucid - OpenJDK
mountainlion - OpenJDK
precise - New Relic
- Play JPA Plugin
- PostgreSQL JDBC
- Spring Boot CLI
- Tomcat
- Tomcat Lifecycle Support
- Tomcat Logging Support
At any given time, any customer we have may want to use any version in those lists. More importantly, they may want to use the version that they’ve certified, until the end of time. Enterprise Java shops regularly do this, which is why we use the secure-by-default line talking about aggressive updates.
This poses some issues through because clearly we can’t have 6.5GB buildpacks out there. To solve this problem, we now have a build that will create an “offline” buildpack with exactly what you might ever need, based on the repositories as they look when packaged. The build actually uses a bunch of the buildpack’s code to iterate over all of the components, access their repositories, resolve the intended versions, and download them into a cache to be used during staging.
The downside of this is that security fixes become the customer’s responsibility. As new versions of dependencies (or new dependencies for that matter) are added, a new buildpack has to be packaged and deployed (cf update-buildpack), making this much more of a burden and less likely to be kept up to date.
The end result of this work is that the Java Buildpack now has three execution modes:
- Easy Mode: Uses the repository at http://download.run.pivotal.io. This is the default, and what we would recommend to anyone who asks.
- Expert Mode: Uses a full replica of the repository at http://download.run.pivotal.io but hosted on an internal network. This is what we would recommend to any customer that didn’t want to access the Internet. It’s easy to keep applications secure and up-to-date, but requires the expertise to run a web-server and keep it in sync with http://download.run.pivotal.io.
- Offline Mode: Uses only the packaged internal cache. This is what we’d recommend if you wanted a single, self-contained artifact although the downsides around keeping up to date are there.
After the change to using cf create-buildpack instead of a DEA submodules, the Java Buildpack that is used by default will stay in Easy Mode. We’ve endeavored to make the use of both Expert Mode and Offline Mode easy, but if you find that you have trouble with them, we’re always willing to listen. If you’re looking to try this functionality out for yourself today, you can find the repository and instructions at https://github.com/cloudfoundry/java-buildpack.
