GrootFS is a container image management tool. It was developed by a Cloud Foundry team in order to replace built-in functionality that currently exists in Garden-runC, as well as to enable Garden-runC to move closer to OCI standards and rootless containers.
GrootFS deals with container images and Cloud Foundry stacks. It is able to pull container images from Docker registries (or import local Cloud Foundry stacks), cache them locally in the Diego Cells, and manage the container root filesystems lifecycle. Container image management so far has been the job of garden-shed, an internal component that lives inside Garden-runC.
Why GrootFS?
The Garden team has experienced significant pain with filesystems from the early days of Garden. Claudia and Tiago (both members of the CF GrootFS team) recently presented the history of filesystems for Cloud Foundry containers. As Dr. Julz (CF Garden Project Lead) likes to say: “Filesystems are hard.”
Claudia and Tiago are talking production filesystems for containers – Photo by @pbouwer.
The last incarnation of pre-GrootFS filesystems for Cloud Foundry containers was a combination of loop devices, AUFS and ext4. It certainly did the job and served the platform well for almost two years. There were, however, significant drawbacks to it. Firstly, AUFS is not a filesystem maintained by the Linux Kernel Community. It is rather a Kernel module, mostly maintained by a handful of individuals. It is used by many in the container community, but there have been (and will be) long-standing issues when it comes to newer Kernels and distributions other than Ubuntu.
Secondly, as AUFS does not natively support disk quotas, the current solution has to implement these under the covers in some quite unusual ways involving dynamically created loop devices. Although it works for our use case, this solution has various limitations and we’re keen to move away from it.
Finally, garden-shed was built primarily for the Cloud Foundry buildpack use case — that is, single layer root filesystems with a container-specific read/write layer. Even though it supports multi-layered Docker images, the code around this was mostly based on Docker’s internal graph driver code and was not easily extensible. As Cloud Foundry starts to adopt the new OCI image standard and to more heavily use and support multi-layered images, we realized we needed a new maintainable code-base for interacting with more complex use cases around working with layered images. Dr. Julz gave a talk on OCI Images for Droplets and Buildpacks at the recent Cloud Foundry Summit Silicon Valley.
What about cflinuxfs2?
GrootFS is not a replacement for cflinuxfs2. It is a container image and filesystem management tool and has nothing to do with the contents of actual container images (Docker images) or Cloud Foundry stacks (such as cflinuxfs2). The default stack GrootFS lays down for Cloud Foundry Buildpack apps is cflinuxfs2. Custom stacks are not supported by default. Custom stacks authors can follow the instructions in the Cloud Foundry Documentation in order to make their stack usable by GrootFS.
How is GrootFS different?
GrootFS makes use of the new image plugin interface in Garden-runC. Garden-runC calls GrootFS as a separate process, before creating or deleting a container and while it is gathering usage metrics from running containers. GrootFS itself is simply a CLI tool that implements the container root filesystem lifecycle. One of the main objectives of the CF GrootFS team was to design the GrootFS CLI so that it can even be used outside of Cloud Foundry.
The new Garden-runC architecture – Diagram by @karampok.
GrootFS is creating root filesystems on top of OverlayFS and XFS. We use XFS for disk quotas since it has one of the most mature directory-based disk quota implementations within Linux Kernel filesystems. OverlayFS, similarly to AUFS, enables copy-on-write layering. It is increasingly becoming the go-to choice for container filesystems by many vendors.
OCI Images and GrootFS
The OCI Image Spec recently shipped its first stable version. Cloud Foundry was early in adopting the OCI Runtime Spec and runC and we were looking at the Image Spec from the early days of GrootFS.
GrootFS comes with support for OCI Images. We treat OCI Images similarly to Cloud Foundry stacks and Docker images. Essentially, the OCI Image spec gives us the opportunity to convert different image types to the same format internally.
The OCI Image support is not exposed to Cloud Foundry users directly. Instead, it is mainly consumed and driven by the OCI Buildpacks work which CF Garden is doing.
Building images from a Dockerfile and the read-write layer.
Rootless
Rootless containers have been a hot topic of discussion in the container ecosystem. Since runC added support for rootless containers, Garden-runC and GrootFS followed so that today we have an (experimental) way to run Garden-runC as a non-root user.
GrootFS can run as a non-root user as most of the operations it performs don’t really require root user privileges. Pulling a Docker image, unpacking an image for a rootless container, caching images, etc., are operations that the Linux Kernel allows us to execute as an unprivileged user. There are, however, some operations (such as setting disk quotas) which are only available to root.
Validation and Testing
GrootFS is a new component and since it’s replacing existing functionality, we need to make sure that we are not introducing any regressions. We have been particularly concerned with performance regressions but functionality or security regressions have also been considered.
VMware has been running GrootFS since early July. The VMware Cloud Operations team authored an article in the VMware Engineering Blog which describes the journey we went through to make sure that GrootFS works sufficiently well in a production environment.
SUSE has also been using GrootFS in its containerized version of Cloud Foundry. SUSE prefers GrootFS because it doesn’t want to have the AUFS Kernel module installed in its Kernels and the SUSE stemcell.
The Future of garden-shed
With GrootFS’ arrival, the CF Garden team will soon be looking to deprecate garden-shed. In the meantime, we would like to gain some more confidence that GrootFS works in a variety of production environments and make it the default option in cf-deployment.
Conclusion
GrootFS is the new container image management component for Cloud Foundry. It was built to replace garden-shed, the existing filesystem management component built-in to Garden-runC. The replacement is necessary because of garden-shed’s reliance on AUFS, and the fact that it would be very hard to refactor garden-shed in order to support OCI images. GrootFS is designed and built to support Cloud Foundry’s needs but also operates as an independent CLI tool outside of Cloud Foundry. It enables rootless operations and OCI Images which will be very useful as Cloud Foundry moves further down the path of rootless containers and OCI Buildpacks respectively.
