Introducing CF-Extensions GitHub Bot and Hub Project

TL;DR

CF-Extensions is the Project Management Council (PMC) for Cloud Foundry that manages projects that extend the platform. Currently hovering around a dozen — but rapidly increasing — the CF-Extensions PMC leads vet, curate, and help manage all extensions projects in Cloud Foundry.

Since all CF-Extensions projects are GitHub projects, I created a GitHub bot that automates much of the interactions and management tasks between the different proposed extensions and the PMC leader. Best of all, it’s done via GitHub APIs and uses GitHub as its database. A project simply needs to add a topic of cf-extensions to its project’s settings and optionally add a .cf-extensions JSON file in its repo to override various metadata entries about its content — for example, description, contact info, etc.

Using the collected data, the new cloudfoundry-incubator/cf-extensions project [1] provides a central repository for the collected projects, including pointers to releases, status and contact information. Essentially, it acts as a hub for all CF-Extensions projects that are automatically up to date. Go to ext.cloudfoundry.org (*) to get started.

Introduction

Cloud Foundry is governed by three PMCs: Runtime, BOSH, and Extensions. As the lead for the CF-Extensions (**) PMC, my job is to help vet, curate, and bring structure into the many extensions to Cloud Foundry that exist currently and that are being proposed. The primary goal is to make extensions discoverable and readily usable to the community.

We are currently talking about dozens of projects distributed across organizations and from various geographies around the world. It is already a logistic challenge that I foresee will only get worse. This is especially true when dealing with disparate projects run by different organizations in different timezones.

Background

My slice of the Cloud Foundry codebase has to do with the extensions. These are all projects that extend the platform, are not considered core, or simply need to be incubated. Example categories of such projects include service brokers, BOSH CPIs, buildpacks and various others considered “experimental.”

In the past three months, we have accepted two projects as official incubating extensions. These were Kubo (Kubernetes + BOSH) and BBR (BOSH Backup & Restore). There are many more coming down the pipeline.

Problem

As the CF-Extensions PMC expands, the primary problem becomes the need for a centralized catalog of all accepted extensions and upcoming extensions, especially their statuses (Proposed, Incubating, Rejected and so on). Additionally, a centralized location should allow for access to salient information about each extension — for example, Git repo URL, contact point and email, leading organizations, and various related information.

Solutions

This blog post gives a broad overview of the previous and current (better) solution to the problem above. Later on, we also give a primer useful to CF-Extensions instigators (proposers), and conclude with some ideas on future directions and next steps.

Obvious Solution

The obvious solution to the problem is to manually keep track of everything using online tools. Google Docs, for instance, makes for a good easy way to keep track of the list of proposed projects and their contacts’ information [2]. By sharing links to this doc, we can get many to contribute to it and keep it up to date.

This approach has many drawbacks. Some of the important ones are:

1. Difficult to find. Google Docs links are mangled. Even if shared in a shortcut, it’s hard to search and find it.
2. While doc updates are distributed, I still need to remind folks to update or do it myself.
3. As the list gets longer, it’s harder to find correct information and to keep everything up to date.
4. It’s difficult to give an easy overview of the entire pipeline of projects and where they are in the CF-Extensions process [4].
5. Adding new information in the “database” of a project is hard to do systematically and to communicate due to lack of structure.

Better Solution: Using GitHub to manage GitHub repos

What if we used GitHub to keep track of which projects are in CF-Extensions and their statuses? By having each extensions project tag themselves as being a “cf-extensions” and adding some small amount of metadata, we should be able to create a bot that uses the GitHub APIs to find these projects, determine their statuses, compute relevant information and compile a list of all CF-Extensions projects. The best part is this list can be updated often, have many forms and ordering variants.

The only two things to do are:

1. Keep track of the status of projects. This should be done in the open but with the CF-Extensions lead having the ability to change status or accept PRs for changing a project’s status.
2. Have each proposed potential extension tag their repos and add the necessary metadata. The metadata could also be submitted as a PR.

Introducing: cloudfoundry-incubator/cf-extensions

The cloudfoundry-incubator/cf-extensions exactly implements such a bot. It will run periodically as a CloudFoundry Golang app to update the /data and /docs directory of its repository. The /data folder contains the JSON databases of all discovered extensions (tracked and untracked) as well as the official list of extensions that are accepted and have statuses. The /docs folder contains generated docs for the catalog of projects. The following diagram summarizes the work done by the CF-Extensions bot each time it runs.

Figure 1: cf-extensions bot run process

The resulting summary list of extensions created in the docs folder is accessible from either pointing your browser to cloudfoundry-incubator/cf-extensions/docs/index.md or eventually from ext.cloudfoundry.org.

Getting Started: A Primer

In order to be listed as a CF-Extensions project, instigators need to follow the following steps:

1. Make their repos public.
2. If the repo is not part of one of the the Foundation’s sponsoring organizations, then move it to github.com/cloudfoundry-community.
3. Modify their GitHub topics list to include “cf-extensions“.
4. Next time the cf-extensions bot runs, it will submit an issue to your repo with instructions to create a .cf-extensions JSON metadata file and what values to include.

Once you are ready to submit your extension’s proposal to the CF-Extensions PMC, then follow these additional steps:

1. Read through the two-pages CF-Extensions process [4].
2. Create a CF-Extensions proposal doc following the template [3].
3. Submit proposal to [email protected] and follow discussions.
4. Take part in the monthly CF-Extensions PMC calls [5].

During all these steps, you are welcome to contact [email protected] if you have any questions. Once a proposal is submitted and accepted, its status will move from “Unknown” to “Proposed” and the project will automatically move from the untracked database to the tracked database and be added to the generated catalog of managed CF-Extensions projects.

It’s important to note that the CF-Extensions process is self-driven. That is, each proposer/instigator can choose to move projects through the process as quickly (within some limits) or as slowly as they choose. A project can also be retracted at any time.

Conclusion: How to Contribute and Next Steps

While the CF-Extensions PMC was created to allow anyone to add to the rich ecosystem that is Cloud Foundry, this openness created a bit of a logistical nightmare keeping track of all extensions. In the past three months we have officially added two extensions to CloudFoundry but many more have indicated interests and are working to submit their proposals.

As CF-Extensions keeps growing, there is increasingly a challenge in keeping track of all extensions, their status, their URL links and their statistics. Additionally, it’s becoming harder to create a brief accurate summary of all extensions with status that can be communicated to the entire Cloud Foundry community in order for folks to discover what is available and could potentially be useful to their current CF workflows. In essence, there is a need to create a kind of CF-Extensions hub.

The cloudfoundry-incubator/cf-extensions GitHub bot was created out of the need to reduce this logistical burden in keeping lists of GitHub projects and in syncing various metadata with the community. By requiring extensions instigators to tag (add a topic in GitHub) their projects and provide a little bit of metadata, we are able to dynamically construct an accurate database of all extensions to Cloud Foundry and organize various views of this database.

Some future features may include but are not limited to:

1. Providing different views of the tracked and untracked projects databases.
2. Expanding cf-extensions bot search to all organizations part of the CFF.
3. Generating emails to cf-dev mailing list to periodically communicate new proposed extensions to the community along with when their statuses have changed.

If you would like to participate in maintaining and improving the cloudfoundry-incubator/cf-extensions GitHub bot project [1], which itself is added as an extension (tool category with distributed commit), we welcome you to clone and peruse the code and read current project TODOs. All pull requests as well as issues and suggestions and bug reports are encouraged.

References

[1] https://github.com/cloudfoundry-incubator/cf-extensions

[2] CF-Extensions Projects Lists Docs

[3] CF-Extensions Template

[4] CF-Extensions Process

[5] CF-Extensions Notes

[6] https://developer.github.com/v3/

Notes

(*) The ext.cloudfoudry.org sub-domain is being setup to forward to the project’s GitHub main page.

(**) In this post, I refer to the PMC using “CF-Extensions” spelling with camel case, however, I use the “cf-extensions” spelling (all lower case) to refer to the GitHub project and bot.