Introduction

This article gives an overview of Service Fabrik, its supported features and how you can implement them for your backing services.

Service Fabrik is a Service Broker, which implements the Open Service Broker API and is designed to work with Cloud Foundry. It has been developed to solve the challenges in provisioning and management of enterprise-scale backing services for enterprise-scale Cloud Foundry applications. While Service Fabrik conforms to Open Service Broker API specification, it has taken extra steps to define the operational management aspects of services such as backup, restore, instance availability and monitoring. There are well defined APIs provided by Service Fabrik for backup and restore which can be implemented by Service Owners to provide data backup and restore capabilities for their services.

Service Fabrik also provides a backup and restore library which abstracts various functionalities by implementing provider specific clients for AWS, OpenStack and Azure platforms. While it is not necessary to use this library, it will help the Service Owners to save precious time by avoiding the need to implement the same functionality again. For monitoring, Service Fabrik generates metrics for Cloud Foundry lifecycle operations as well as broker specific operations which can be consumed by plugging Riemann.

Features

Here is a list of features currently provided by Service Fabrik:

Service provisioning using BOSH and Docker Swarm
Well-defined extension APIs for Backup and Restore of Cloud Foundry service instances and normal BOSH deployments
Generic Backup and restore API library supporting AWS, OpenStack and Azure
Backup scheduler for periodic backups
Logical Volume Driver for Docker Swarm
Monitoring event generation for Riemann

High Level Architecture Overview

The below diagram provides a high-level architectural overview of Service Fabrik. Let us describe each of these blocks:

CF CLI – This is the Cloud Foundry-provided command line interface to interact with Cloud Controller for operations like bind, unbind.
1. Plugin – This is a custom plugin developed by the Service Fabrik team which provides additional capabilities such as backup and restore. The plugin will soon be made public.
SAP Cloud Cockpit – This is a custom home-grown GUI console which provides the same capabilities as provided by Cloud Foundry CLI. This is not a publicly available component.
Cloud Controller – This is a Cloud Foundry component which provides REST endpoints for clients to access the Cloud Foundry platform.
Service Fabrik Broker – Service Fabrik orchestrates the provisioning and management of backing services. It provides two variants of service provisioning – one using BOSH for enterprise-scale services and other using Docker Swarm for developer-only services. There are additional capabilities (over and above the Open Service Broker API) provided by the Service Fabrik platform which are explained below:
1. Backup and Restore – These are extension APIs defined by the framework over and above the ones defined in the Open Service Broker API Specification.
2. Monitoring – Service Fabrik generates monitoring events for Cloud Foundry lifecycle operations such as create, delete, bind, unbind, etc. It also generates events for broker-specific operations such as backup and restore. Currently the framework-generated events can be consumed by plugging Riemann.
3. Scheduler – This component helps to schedule the backups in a timely periodic manner. The framework provides an API which accepts CRON expressions or human readable time interval to schedule backups.
4. Updates – While Open Service Broker API defines API for instance update, Service Fabrik provides additional capability to automatically update service instances in case of a stemcell or service update.
BOSH – BOSH is a platform component and a tool for release engineering of distributed systems, which Service Fabrik uses to provision a BOSH-based service instance.
Swarm – Swarm Manager is invoked by Service Fabrik to provision service instances as Docker containers.
IAAS Blob Store – This is used to store service instance backups and backup metadata
PostgreSQL/MongoDB Deployment – These are the running service instances provisioned by Service Fabrik using BOSH based releases of PostgreSQL and MongoDB.
1. Agent – This is the component implemented by Service Owners using the Agent API defined by Service Fabrik. The agent acts as an intermediate between the broker and service instance. Once implemented, the services can provide capabilities such as create users, delete users, start backup, abort backup, start restore, abort restore.
Docker – Swarm Manager interacts with Docker nodes to provision service instances running as Docker containers using Docker images provided by Service Owner.

Service Lifecycle

Service Fabrik provides two flavors of service provisioning. One is BOSH-based for enterprise scale deployments and other is Docker-based for developers. In the upcoming section, we will discuss the lifecycle of a service from both the perspectives. As depicted below, lifecycle of a service can be divided into five steps:

Service Development

Service development typically involves writing agent implementation and creating a BOSH release for the agent. However, for Docker-based services, agent implementation is not required.

BOSH Service Development

Service Owners implement service agents which are supposed to perform service-specific operations. Agents must be part of a BOSH service instance and will be contacted by the broker to delegate operations the broker cannot handle by itself (e.g. creating/deleting credentials while binding/unbinding, running a backup or restore operation, etc.). Consequently, the agent acts as an intermediate between the broker and the service. This is to decouple the two components, to enable autonomous releases, and to avoid dependencies between the Service Fabrik and individual services. A service development typically comprises of the following steps:

Developing a Service Agent – Here Service Owners will implement an agent for the underlying service. The specification of Agent API can be found here. While implementing backup and restore functionality as defined in the agent API, Service Owners can utilize the backup and restore library provided by Service Fabrik. This is a generic library which abstracts various functionalities by implementing provider specific client for AWS, OpenStack and Azure platforms. An example agent implementation can be found in our sample Blueprint Service.
Create BOSH release for the Agent – Post-development of service agent, Service Owners have to create the BOSH release for the agent. This agent will be co-deployed along with the service. An example BOSH release for the agent can be found here. There is a job defined for the agent in this BOSH release.

Docker service development

As the Docker services are only meant for developers, they are not supported for additional capabilities such as backup and restore — therefore an agent implementation is not required.

Service BOSH Release or Docker Image

Depending on the type of deployment, Service Owners will have to either create a service BOSH release or a Docker image. An example BOSH release can be found in our sample Blueprint Service. A sample Docker image for Blueprint service can be found in Docker hub.

Service Onboarding

For onboarding a BOSH based service on the Service Fabrik platform, Service Owners are required to provide a minimal manifest and a service catalog. This minimal manifest should only provide the job details while other fields of the deployment manifest such as network, resource pools, etc., are automatically injected by the Service Fabrik platform. A sample minimal manifest can be found here.

The description on fields other than jobs can be found here.

The second prerequisite is to provide a service catalog which defines the various plans offered by the service. A sample service catalog can be found here.

Lastly you need to add a reference of the service catalog file to the Service Fabrik platform as mentioned here. This yml file contains the Service Definitions for all the services which can be provisioned by the Service Fabrik platform.

Now follow the steps mentioned here to deploy Service Fabrik.

For Docker-based services, you need to only provide service catalog. A sample can be found here.

Service Provisioning and Binding

On completion of the Service Fabrik deployment, the first step is to register the Service Fabrik broker with the Cloud Foundry platform which can be performed as described here. Next you can provision a service and bind it to an application as described here.

Service Management

Once your service is up and bound to an application, Service Fabrik can be used to manage the deployed service using the following capabilities. While there is full support available on all the functionalities for BOSH-based services, monitoring is the only supported capability for Docker based services.

Instance Availability – Service Fabrik provides a state API which can be used to retrieve the current availability status of an instance.
Backup and Restore – Service Fabrik has well-defined APIs for the management of mission critical data. Functionalities exposed by the API includes start backup, abort backup, state of the last backup operation and similarly for restore operations.
Backup Management – Service Fabrik has well-defined APIs for the management of data backups. APIs with functionalities to list the currently available backups, details on specific backup, delete backup and status of backup/restore operations for all the instances in a particular space are currently provided.
Scheduler – In a production deployment, you need the capability to automatically schedule backups this is where the Service Fabrik scheduler API helps. This API provides capabilities to schedule an automated backup, cancel an automated backup and retrieve backup schedules.
Monitoring – Service Fabrik generates metrics for service lifecycle events such as create, delete, bind, unbind. It also generates metrics on broker specific events such as backup and restore. These monitoring events can be consumed by adding Riemann to your Service Fabrik deployment.

Conclusion

And that’s a wrap! I hope this article has given you an understanding of various moving parts of Service Fabrik and where to find information to tackle each one of them. Do let me know whether you found this article useful. You can write directly to me or raise an issue here.