This case study has been contributed by Armakuni.
Over the past 30 years, UK-based Comic Relief has raised over £1 billion for good causes around the world. Through its two campaigns, Red Nose Day and Sport Relief, Comic Relief uses the money donated by the British public to make its vision a reality: a just world, free from poverty.
Comic Relief was founded in 1985 by comedy scriptwriter Richard Curtis and comedian Lenny Henry in response to famine in Ethiopia. Each year, Comic Relief has grown in size and influence. In its first telethon in 1998, it raised £76,610. By 2015, it raised £99,418,831.
Every two years, Comic Relief’s Red Nose Day takes over primetime Friday night programming on BBC TV for seven hours. This telethon is responsible for the majority of Comic Relief’s annual fundraising. Viewers donate money via the website or a network of 14,000 call center operators (who donate their time for free) across 120 call centers.
“With an increasing number of donations to Comic Relief being taken online, it is critical we have 100% confidence in our donations platform. Armakuni provides us with that confidence.” — Derek Gannon, COO, Comic Relief
The Old Donations Platform
The directors realized the existing platform was nearing its end of life and they needed a new platform. When such a large percentage of your annual money is made in only seven hours, you want to guarantee the platform is not going to fail!
“Armakuni’s involvement gives us 100% confidence our donations platform can handle the considerable volume of traffic we experience during event night. Not only that — Armakuni has significantly reduced the cost and time associated with the build and operation of the platform.” — Zenon Hannick, CTO, Comic Relief
Our Pitch
In 2012, Armakuni was approached by Comic Relief and asked to pitch for the ground-up platform rewrite. Armakuni was known to Comic Relief— the founders had worked on the team at Comic Relief before leaving to start Armakuni.
Armakuni pitched an incremental, agile approach. At the time, Armakuni was a small team, and was competing with a number of larger, more established tech firms, several of which had been involved in standing up the donations platform in previous years.
Comic Relief liked that Armakuni’s concept could be tried and tested before any critical long term decisions had to be made. Armakuni proposed to work on a time and materials basis, enabling the directors to disengage if things weren’t going the way they wanted. The Armakuni Way is the guarantee that the board would see fortnightly demonstrations of the project. This allowed all involved to thrash through the issues, risks, unknowns and problems, miles in advance of the live night. Despite Armakuni being a bit of a small fish in a big pond, its innovative approach and risk-averse strategy won over the board at Comic Relief.
The Brief
The brief wasn’t for the faint-hearted. The solution needed to:
- Scale, with 100% performance reliability
- Support 14,000 call center operators and up to 100,000 concurrent web sessions
- Cater to up to 400 donations/second, and an estimated 800,000 donations in total
- Be PCI compliant
- Keep build and run costs to a minimum
Armakuni’s Approach
Armakuni used a polyglot approach. This included multi-IaaS, multi-PaaS, and multi-language, with multiple payment-service providers. Infrastructure-wise, Armakuni used PaaS to allow a single contract to the application development across multiple IaaS platforms. After exploring several options and performing proof of concepts with OpenShift and Cloud Foundry, the team settled on Cloud Foundry, deploying on both Amazon Web Services (AWS) and vSphere in a London data center.
“From scratch, we can deploy the entire platform, infrastructure and applications, fully tested, on to three continents in about 90 minutes—and handle the live TV show!” — Tim Savage, Director, Armakuni
Armakuni now uses AWS and Google Compute Engine (GCE). The importance of being able to switch between clouds avoids the dangers and frustrations of being “locked-in.”
Armakuni used an agile approach, to both disconnect applications from infrastructure and to enable continuous deployment of both infrastructure and applications.
Armakuni used continuous deployment through to production, enabled by a rigorous Test Driven Development (TDD) approach, and integrated security and load tests into its pipelines.
Architecture-wise, Armakuni employed a microservices environment to enable decoupling responsibility horizontally where required. Armakuni used stateless applications, and chose availability over consistency, adopting an “eventually consistent” model, which meant that in the worst case scenario, they could lose reporting in favor of continuing to take donations.
And Armakuni did all this in nine months with a team of five.
Overcoming Problems
Of course, Armakuni faced some problems along the way. For example, the team initially considered OpenShift, but found that, at the time, Cloud Foundry was the more mature of the two PaaS offerings. The team has also spent many hours load testing with payment providers, finding out whether they could actually handle the level of traffic that they were claiming.
“Armakuni’s use of Cloud Foundry PaaS enables us to deploy our donations platform on-demand to multiple infrastructure providers around the globe. And the cloud native approach and use of test-driven development enables us to deploy a change into production within 30 minutes. Even on event night when we’re processing 300 transactions per second!” — Zenon Hannick, CTO, Comic Relief
The Result of Armakuni + Comic Relief
The result: an overriding success.
- Performant to the tune of up to 400 donations per second
- Highly scalable and resilient, and is deployed and scaled programmatically
- Resilient across multiple points of failure; infrastructure, connectivity and payment provider
- From scratch, the entire platform, infrastructure and applications, fully tested, can be deployed to the separate IaaS providers across three continents in about 90 minutes
- Unlimited horizontally scalability “on demand”
- Significant reduction in team size; a team of five builds, deploys and runs the donations platform—even on event night
- No need for proprietary, high-spec infrastructure and licenses
- Potential for human error removed; everything-as-code automates applications, services, environment provisioning and deployment
- Fully tested deployment providing maximum confidence in the platform.
“Our work has been so fundamental that Gartner highlighted us as a case study of multi-cloud usage.” — Tim Savage, Director, Armakuni
Continued support
In March 2017, Armakuni will be supporting Comic Relief’s event night again. The team is proud to work with such a prestigious and high-profile organization, and of the Armakuni Way of working to benefit such an important charity. For details of how you can donate to Comic Relief, please visit Red Nose Day.
