Anna Shipman : JFDI

Building tools to provision our machines

07 May 2014

Over the last few months, I’ve been leading a small team building tools to automate creation and configuration of our servers.

Tools to automate provisioning

Currently, our production environment is hosted with Skyscape, and we manage it using VMware vCloud Director. Skyscape, and other providers using vCloud Director, expose a UI for managing VMs and networks.

However, a UI isn't suitable when you need to bring a lot of machines up in a repeatable, reliable way, for example when migrating the site to a new platform. In addition, if you can automate provisioning, the configuration files represent your infrastructure as code, providing documentation about your system.

VMware vCloud Director supports the vCloud API, but there was no widely-available tooling to orchestrate this kind of operation using the API. So we built vCloud Tools.

Our suppliers currently all use vCloud Director

The UK Government’s policy is cloud first and we procure services through CloudStore (which uses the G-Cloud framework). As government, it’s vital that we provide secure, resilient services that citizens can trust, and we need various assurances from our suppliers to support that.

There are a number of Infrastructure as a Service (IaaS) platforms that are able to provide these assurances, but when we procured our current suppliers last year, the vendors that met our requirements (both technical and non-functional, eg self-service provisioning, a pay-as-you go cost model) used vCloud Director.

We very much want to encourage potential new suppliers, but given the availability in the market, we can be reasonably certain we’ll be using at least one VMware-based solution for at least the next 12-18 months, and it’s likely that many other transformation projects will also be drawing on the same pool of hosting providers. So it’s definitely worth investing time to make provisioning easy.

Preventing vendor lock-in

Having an easy way to provision environments will allow us to move more easily between vendors that use vCloud Director, which will help prevent supplier lock-in. But we also don’t want to lock ourselves in to VMware, so we are taking several steps to guard against that:

Previous iterations of provisioning tooling

Automation of provisioning is something we’ve been iterating on since we launched GOV.UK 18 months ago. Prior to vCloud Tools, we used a tool we had built called vCloud Provisioner. This was really useful when moving onto our first platform, but because it was built quickly it has a lot of hard-coded detail about the environment, so it doesn’t help us be flexible when moving between suppliers. In addition these hard-coded details include sensitive information, so we can’t share it, meaning it cannot be useful to anyone outside of the GOV.UK team.

Several members of the infrastructure team worked on subsequent iterations of the provisioning tooling. These iterations included vCloud Box Spinner and vcloudtools. However, the people working on these were building these tools alongside their other business-as-usual work keeping GOV.UK up and running, so when they ran into issues, it was difficult to find time to address them. With the migration to the new platform looming, we needed to prioritise this piece of work in order to be able to bring up a new environment quickly, reliably and easily.

We think vCloud Tools will be more robust

There are several things we have done to improve the chances of producing robust, more widely useful tools in this iteration.

We have committed the time and resources to this, forming a small team who focus entirely on vCloud Tools rather than firefighting or other operations work, and we are coding in the open, so we won’t fall into the trap of including sensitive information that it is later too hard to factor out.

Not only are the GOV.UK team “eating our own dogfood” by using vCloud Tools to provision our new hosting environments, there are two other GDS teams also using the tools, the Identity Assurance Programme and the Performance Platform. Other exemplar projects have started using the tools, and we have already accepted pull requests from people we do not know, so there is the beginnings of a community around the tools. This keeps us from making it too GOV.UK specific and means that we get a lot of extremely useful user feedback.

And in the meantime we are contributing back to Open Source - in one recent release of fog, a huge number of the contributions were from the GOV.UK vCloud Tools team (who were, at that time, me (Anna Shipman), Sneha Somwanshi, Mike Pountney and Dan Abel).

These factors mean that we feel confident that we are going to produce tools that will continue to be very useful to us and other teams for some time.

More will follow

There will be future posts about particular aspects of what we’ve done and how, including one soon on how we used vCloud Tools in our platform migration. If there’s anything you’d particularly like more detail on, please let us know in the comments below.

This post originally appeared on the GDS Technology Blog.