I am lucky to have the opportunity to work at Stark & Wayne while earning my PhD in Computer Science and Engineering at the University at Buffalo. Stark & Wayne has lots of customers using Cloud Foundry, including GE, Intel, Swisscom, Zipcar, SAP and others. I myself have been working on various VMware Cloud Foundry projects for GE Digital for the GE Predix platform, leveraging services such as RDPG, RabbitMQ, Redis, PostgreSQL and SHIELD. All of these services are deployed using continuous integration (CI) tool Concourse pipelines and are provided to GE Predix platform users.
Xiujiao Gao, Cloud Engineer, Stark & Wayne LLC & PhD Student at the State University of New York at Buffalo
BOSH & Concourse
At Stark & Wayne we help GE deploy VMware Cloud Foundry and various services using BOSH. BOSH is an open source tool that can deploy not only Cloud Foundry, but can also deploy and manage the lifecycle of almost any other software/services, especially large distributed systems, while supporting multiple infrastructures like VMware vSphere, vCloud Director, Amazon Web Services EC2 and OpenStack. We also take advantage of several important features of BOSH such as performance monitoring, failure recovery, and software updates.
We use Concourse together with BOSH for release engineering and deployment lifecycle management. For release engineering, each new commit will automatically trigger a BOSH deployment with code changes and will run tests afterward.
The GE Predix platform has several VPCs/environments. To solve the pain of the deployment lifecycle from sandbox to production in different datacenters, we use Concourse pipelines to automate the whole deployment and tests process. We set up Concourse pipelines for all of the BOSH deployments in the GE Predix platform, which includes VMware Cloud Foundry, RDPG, RabbitMQ, SHIELD and Redis.
Reliable Distributed Provisioning Grid (RDPG)
The first project I worked on with the team at Stark & Wayne for the GE Predix platform was RDPG, a “Reliable Distributed Provisioning Grid” providing PostgreSQL as its first service for Cloud Foundry applications. RDPG has a management cluster as well as service clusters. The management cluster stores all the metadata, manages the service clusters and provides the service broker for talking with Cloud Foundry; the service clusters provide PostgreSQL databases for Cloud Foundry applications. RDPG itself is a BOSH release and can scale up with the ability to grow the service VMs CPU/Disk/RAM resources in order to allow more service instances per VM. It can also scale out with the ability to increase capacity by adding new service clusters instead of simply assigning more resources to the existing VMs. This is accomplished by modifying the BOSH manifest and redeploying using BOSH.
You can find Stark & Wayne’s open source fork here.
Highly Available RabbitMQ Service
Another project I have been working on is a RabbitMQ service for Cloud Foundry. RabbitMQ provides highly available features by deploying a RabbitMQ cluster across multiple availability zones using BOSH. The features inside the RabbitMQ cluster are configured through the BOSH manifest so that all the queues/messages are mirrored on all the RabbitMQ server nodes.
RabbitMQ itself does not deal well with network partitions, so we used the Rabbitmqctl tool, as well as the failure recovery features of BOSH and SHIELD in order to ensure recovery from a network partition or other disasters.
SHIELD
SHIELD is an open source modularized backup and restore solution which utilizes a plugin based architecture to backup and restore BOSH deployments. SHIELD can backup and restore core Cloud Foundry internal services as well as non-core Cloud Foundry services such as Redis, PostgreSQL, MySQL, RabbitMQ and others. GE uses SHIELD to backup and restore their Cloud Foundry core and services.
I find the SHIELD framework fascinating. SHIELD is distributed as a BOSH release and is designed so it does not require a SHIELD daemon running to restore BOSH. It can backup/restore BOSH as well. SHIELD consists of a core daemon, backup target plugins, storage plugins, a CLI (command line interface) and a web user interface. It is very easy to write a target plugin to support backup and restore for new services based on SHIELD.
Conclusion
It is very exciting to be involved with the GE Predix project and its implementation of VMware Cloud Foundry. The implementation is becoming more and more sophisticated as GE deploys additional services and applications over time. The work that GE is doing with Cloud Foundry is fundamentally changing their business and their customers’ businesses in a revolutionary way.
I can’t wait to continue my Cloud Foundry technological adventures at Stark & Wayne so I can help more clients succeed.
