The emergence of cloud technologies has created new opportunities for application development. Environments can now be version-controlled, and multiple instances can be spun up to run several tasks at once. An excellent tool for cloud development is Vagrant, the most powerful benefit of which is the consistent creation and re-creation of identical servers, fully provisioned to run or test code. Not only does this eliminate the all-too-common “works on my machine” roadblock, but it also opens the potential for remote development in circumstances where expensive, in-house hardware is normally required (e.g., GPUs or large disk arrays).
Why Use Vagrant
There are several benefits to developing in the cloud with Vagrant. First, there is no need to configure package installs or virtual environments. All of this is handled through Vagrant and the configuration management tool of your choice. Additionally, instances are disposable, so if something drastic happens – configurations are mangled or libraries are removed – the instance can be safely destroyed and a new one created in its place. Furthermore, you can save time by utilizing multiple instances to run different versions of code or for distributed computing.
Vagrant can also yield benefits when used with continuous integration (CI). Rather than running several code tests consecutively (often a lengthy proposition), dedicated instances can be created and provisioned, and tests executed concurrently via Vagrant. This also means that compute resources can be created on-demand, thus eliminating the potential of idle resources driving up the cloud services bill.
Getting Started with Vagrant
The following has been verified using Vagrant v1.7.4 and above. Most often, installing Vagrant from your standard repository is sufficient, but if you want to ensure you have the latest version, follow the installation instructions on the Vagrant downloads page. Vagrant has several providers, which allow it to integrate with a number of different virtualization technologies. For AWS, you must install a custom provider, vagrant-sdk, which is maintained by the same developer as Vagrant itself. Following the installation for vagrant-aws, be sure to install the “dummy” AWS box, which provides a blank slate from which to start your instance.
A Ruby application, Vagrant gives you almost the entirety of the Ruby language at your disposal. It should be noted that Vagrant runs on its own dedicated Ruby library, thus normal gem install procedures do not apply. However, any Ruby gem can be installed into the Vagrant library by installing plugins. One particularly helpful plugin is aws-sdk, allows you to programmatically interact with AWS resources within your Vagrantfile.
All-in-all, the following should install everything you need (assuming RHEL-based Linux):
Note that, based on the above commands, the value of the `config.vm.box` will be “aws_blank”.
Preparing the AWS Infrastructure for Vagrant
Before continuing into some of the finer details of provisioning and the Vagrantfile, you need a suitable AWS infrastructure for creating instances. It’s highly recommended that you set up a standard VPC, complete with public and private subnets. In most cases, it’s best for instances to be within the private subnet, but this means that you must be connected directly to the VPC via a VPN. In this case, be sure to modify the ‘aws.ssh_host_attribute’ value in the Vagrantfile. Otherwise, create your instances in the public subnet with a public IP. This will also dictate your Security Group configuration, which will need to allow SSH access from wherever you are.
The most important piece of your infrastructure is, quite possibly, an IAM Role, which the Vagrant instance will assume when created. Though Vagrant itself will need to use your AWS access key to create the instance, you don’t want to transfer your credentials or store them in version-control, as this is a potential security risk. Instead, a dedicated IAM Role will enable the instance to access allowed resources and services, such as an S3 bucket or an RDS database.
Lastly, of course, you’ll need to generate an SSH key (either manually or thru the AWS Web Console). SSH key management can be difficult, which is why having a single, dedicated key for Vagrant is ideal. Alternatively, however, you can generate an SSH key for each staff member who will be spinning up instances, especially if access to these instances must be strictly regulated. This also applies to AWS access keys. It is considered best practice for each user or machine using Vagrant to have its own access keys, so that access to the AWS account can be properly tracked and later audited (with CloudTrail enabled).
Templating the Vagrantfile
The goal of the Vagrantfile should be ease of use and flexibility. Often times, the same Vagrantfile is used both as a part of the continuous integration (CI) process and by developers individually; thus, it’s necessary to template the Vagrantfile. Values should be passed at “vagrant up” either dynamically or explicitly on the command line.
To create a new instance, Vagrant uses a set of AWS access keys. These access keys can be dynamically loaded when using the AWS CLI or SDKs. As Vagrant is a Ruby application, using the Ruby AWS SDK seems only natural. With the aws-sdk plugin installed, the Vagrantfile can be configured to automatically load the AWS access keys by looking in the standard locations: local environment variables or the access credentials ini file. This is accomplished simply:
Note that :profile_name is “default” by default (naturally), but if dealing with multiple profiles (i.e., more than one set of access keys) this can be manually set.
In many situations, it’s useful to be able to pass values directly on the command line. Such is the case when designating the location of the SSH key to be used. Though the default name and location can be set in the Vagrantfile, each machine or developer may have multiple AWS keys to manage, so names and locations can be highly variable. To solve this, use the ‘getoptlong’ Ruby library and input your arguments at the top of the Vagrantfile. For example:
While the default location in this example is set to “~/.ssh/my_key.pem”, using ‘getoptlong’ allows for passing the SSH key location directly on the command line, as follows:
Note that the argument flags must come before the “up” command. It should be noted that, as option flags are added to your Vagrantfile, you will need to be sure to add the standard Vagrant options along with the custom options (see here).
Expanding upon these two Ruby libraries can greatly increase the usefulness and flexibility of your Vagrantfile. That said, you’re not limited to what’s presented here – indeed, you have much of the Ruby language at your disposal. Once you have a templated Vagrantfile that meets your needs, however, there are still two steps to keep in mind.
Provisioning and Burning an AMI
Vagrant comes with a host of built-in provisioners, including Ansible, Chef and Salt, so you should be able to use the configuration management tool your organization normally uses. With Vagrant integration, config values can be passed thru Vagrant to the provisioner, allowing for dynamic, one-size-fits-all configurations. Additionally, having robust config management is important when burning AMIs.
As with any cloud deployment, immutable infrastructure plays an increasingly large role, and this is true for cloud development as well. Burning an AMI (or a set of AMIs) to be used with Vagrant is important to keep the development moving. In general, a new instance should be created, provisioned and ready to be used in the time it takes to get a cup of coffee.
The manual AMI creation process can be somewhat tedious and time consuming. To automate the process, use a utility like Packer which creates virtual images (in this case, an AMI) from a given configuration. This configuration can then be included in version-control, allowing for consistent development and deployment.
As the development of cloud-ready applications becomes more and more of a necessity, it’s important to leverage cloud technologies to enable and empower developers to take full advantage of cloud resources. I hope that this article has been helpful, though it is merely a small glimpse at the potential made available in AWS. I encourage you to leave a comment or question below, and even continue the conversation within your own organization about where dev meets cloud.