Posted on by admin

Tutorial: How to Extend AWS CloudFormation with Lambda

AWS CloudFormation is an amazing infrastructure management tool that allows you to define requests for resources in a domain-specific templating language. CloudFormation has built-in binary logic within the template in the form of Condition Functions. Nearly every resource in AWS can be defined in CloudFormation, but there is an occasional resource or pattern of building that isn’t natively supported. Fortunately, for these instances there is a workaround. We can extend CloudFormation with custom logic. In this blog post I’ll show you how to extend AWS CloudFormation with the CloudFormation Custom Resource in conjunction with AWS Lambda functions.

Why would you want to extend CloudFormation? I can think of three scenarios:

  • Sometimes there’s a lag between the release of a new AWS resource, service or feature, and when it’s reflected in the CloudFormation templates. For example, AWS NAT Gateways were not available in CloudFormation for nearly a month after their release. And it was also more than a month before there was any way to create API Gateway resources with CloudFormation, and to this day there’s still not a way to use the Swagger importer through CloudFormation without extending with custom logic. With the pattern I’m about to demonstrate, you’ll gain the ability to quickly test the latest AWS features while still utilizing CloudFormation for the entire infrastructure management.
  • You may have lookups or computations that you’d like to perform on the fly in the CloudFormation stack, such as dividing out your virtual private cloud (VPC) CIDR range into equal subnet CIDRs. When peering one VPC with another you may need to add a network ACL on the peered VPC to allow the new VPC CIDR range access. While network ACLs can be created with CloudFormation, you need to know an open rule number, especially if you’re dynamically peering multiple VPCs with the hub VPC.
  • There may be patterns of building things that you would like to use rather than conforming to the CloudFormation way. An example of building in a pattern that’s not supported by CloudFormation, and the example I demonstrate below, is attaching a VPC to an existing internal hosted zone.

Problem

Currently in CloudFormation the only way to associate a VPC with a Route 53 internal hosted zone is to create the internal hosted zone within CloudFormation itself. In order to build a new VPC through CloudFormation and have it associated to a pre-existing internal hosted zone, you must extend CloudFormation with a custom resource.

Preparation and Demonstration

Before you create a custom resource in CloudFormation, you must first prepare a CloudFormation template that creates a Lambda function. I’ll demonstrate a nested template: a child template will create our Lambda function, another child template will create the VPC, and after both are complete the parent template will call the custom resource to attach the VPC to a Route 53 internal hosted zone. The internal hosted zone ID will be passed to the parent template by use of a parameter. Let’s get started.

Assumptions

  1. You have an AWS account.
  2. You’re using Bash.
  3. You have pip installed.
  4. You have the AWS CLI installed, preferably version 1.10.x or greater.
  5. You have configured the CLI and set up AWS Identity and Access Management (IAM) access credentials.

Step 1: Create an internal hosted zone.

When creating a internal hosted zone you must specify a VPC. We will use one of the default VPCs created with your AWS account. If you don’t have a default VPC, you can create a dummy VPC to use. If you feel more comfortable using the console, feel free. If you already have an internal hosted zone you can skip this step.

Throughout this tutorial you are responsible for replacing {yourName} with a unique string.

VPCID=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[].VpcId' --output text --region us-east-1)
aws route53 create-hosted-zone --name example.internal --caller-reference 2016-08-06 --vpc VPCRegion=us-east-1,VPCId=$VPCID --hosted-zone-config Comment="command-line version",PrivateZone=true

You now have an internal hosted zone. It may take a moment to produce as the create-hosted-zone call is asynchronous.

Get the internal hosted zone Id. This command will list all hosted zones. Query for the one we just created, get its Id, and cut just the id (Note: When returned by this command the Id starts with /hostedzone/{Id}). The following command queries, and cuts, to display only the information we need.
aws route53 list-hosted-zones --query "HostedZones[?Name == 'example.internal.'].Id" --output text | cut -d/ -f3

Save the string output by the command above, you will use it in Step 5, to replace {YourInternalHostedZoneId}

Step 2: Create an S3 Bucket.

You’ll need an S3 bucket from which to work. We’ll use this bucket to upload our CloudFormation templates and our Lambda code zip. Create the bucket with the following CLI command or through the console. Keep in mind that S3 bucket names are globally unique, and you’ll have to come up with a bucket name for yourself.
aws s3 mb s3://extend_cfn_example_{yourName}

Step 3: Clone the example Github project.

I prepared a Github project with all of the example CloudFormation and code to get you off the ground. Clone this Github project to your local machine.
https://github.com/dbrainnet/extend_cfn_example

Step 4: Run the scripts.

You must run two scripts from within the Github project. Both of these scripts are to be ran from the base of the repository.

Script 1: build_lambdas.sh — This script will utilize pip to install the required packages for the Lambda function to the local directory. Zip up the Lambda function with all of the dependencies and place it in a new directory ./builds/.
./scripts/build_lambdas.sh

Script 2: s3_sync.sh — This script will sync all the necessary files (the builds and CloudFormation directories) to your S3 bucket.
./scripts/s3_sync.sh -b extend_cfn_example_{yourName}

Step 5: Create the CloudFormation Stack.

This is the final step in the demonstration. The following command sets the stack name to extend-cfn-example. The template URL is specified. You will need to modify this to point at your own bucket. The parameters come next. The CloudToolsBucket parameter is the name of your bucket. The PrivateDomain parameter is the name of your internal hosted zone, and the InternalHostedZone parameter is the Id of the internal hosted zone you’ve created. After this you must provide IAM capabilities to this CloudFormation stack, because we must create an IAM role for the Lambda function to run.
aws cloudformation create-stack --stack-name extend-cfn-example --template-url https://s3.amazonaws.com/extend_cfn_example_{yourName}/cloudformation/network/top.json --parameters ParameterKey=CloudToolsBucket,ParameterValue=extend_cfn_example_{yourName} ParameterKey=PrivateDomain,ParameterValue=example.internal ParameterKey=InternalHostedZone,ParameterValue={YourInternalHostedZoneId} --capabilities CAPABILITY_IAM

Wait for the CloudFormation stack to complete, and then check in on the internal hosted zone. It should be associated with the new VPC. This concludes the demonstration. Next we will talk about what went into the code and how the code interacts with CloudFormation, and vice versa.

Explanation and how it’s done

In this section I explain how the Lambda function is set up, how it was built, libraries you’ll need to install, and the interaction between CloudFormation and the function.

The Lambda Function

Lambda functions can be overwhelming at first. There are a few things you need to know before you start:

  1. Runtime: The runtime is the Lambda environment in which you want to run. In this demonstration we used the python2.7 runtime.
  2. Handler: The handler is the path to the function. I typically use the default, which is to name the file ‘lambda_function.py’ and the function defined as lambda_handler. I do this to keep things simple. This way we always know the handler name and can switch code out without having to rename the handler. The handler in this case would be ‘lambda_function.lambda_handler’
  3. Role: The role is the IAM role that the Lambda handler will inherit when running. In our demonstration, we needed to be able to make the call to associate the VPC to the internal hosted zone as well as log to CloudWatch Logs. Note that this role must have a principal of service: lambda.amazonaws.com.
  4. Memory: The amount of memory needed for the function to run must be specified during its creation. For the demonstration we used the default, 128MB.
  5. Timeout: The timeout indicates how long the Lambda function is allowed to run. The maximum time currently allowed is five minutes. This demonstration utilized a one minute timeout.
  6. Code: There are two ways to provide code to Lamba: inline or through a zip package. Because we need to import a couple libraries we must build and upload a zip. Through CloudFormation if you provide a zip, you must point to an S3 location.

The AWS CloudFormation

First you must create a role for you Lambda function to use (example). Next, you must create the Lambda function (example).  Lastly, you must call the custom resource (example). You create a custom resource just like any other resource in CloudFormation, however the Type will start with “Custom::” and can end in anything you like. For example, in our demonstration we used “Custom::AttachVpcToHostedZone”. There is only one necessary property to be specified to a custom resource and that is the “ServiceToken”. The ServiceToken value should be the ARN of your Lambda function. From there, the rest of the properties are up to you. These properties will be passed to the Lambda function in the form of a Lambda event.

The Code and the Libraries

The Event

The code that goes into the Lambda function itself is fairly straightforward. Each Lamba handler takes two parameters: event and context. The event is the data passed to the Lambda function, and the context is a special AWS object providing runtime context. The event will carry all the information we care about to perform our task. The event is a JSON object (already translated to a dictionary in python runtime) with the following properties:

  1. RequestType: A string detailing if CloudFormation is running a Create, an Update or a Delete of the custom resource.
  2. ResponseURL: A pre-signed S3 URL to which your code must upload a response.
  3. StackId: The Id of the stack calling the custom resource.
  4. ResourceType: The name you’ve chosen to provide the custom resource in the Type field.
  5. LogicalResourceId: Logical Id you’ve provided to the custom resource.
  6. PhysicalResourceId: Sent only with Updates or Deletes, the physical Id of the resource, as provided by your Lambda function on the create of the custom resource.
  7. ResourceProperties: These are the properties you provide to the CloudFormation custom resource in the properties section in the form of a JSON object (already translated to a dictionary in Python runtime). This attribute is the bulk of the information. This is where you pass information from CloudFormation into the custom resource. In our demonstration we passed the internal hosted zone Id, the region and the VPC Id.
  8. OldResourceProperties: Provided only on Update, this object has all the ResourceProperties from before this update. This object provides the custom resource the ability to detect the difference between the new properties and the old properties. Remember, in CloudFormation, a resource only updates if one of the properties changes.

For more information about the event object, see the AWS Documentation.

The Work

Now that we know how our resource properties are passed to the function, we can write code to utilize those properties to make our custom resource do what we want. I organize my custom resources in a particular way: I build a class with create, update and delete methods. In the lambda_handler function which gets called when the Lambda function is invoked, I initiate the class, check if the RequestType is Create, Update or Delete, and call the appropriate method of the class.

In this case, the ‘Create’ RequestType will create the VPC, Route 53 internal hosted zone association. The ‘Delete’ RequestType will disassociate the VPC from the Route 53 internal hosted zone. The ‘Update’ RequestType actually calls the create method, then the delete method, because a change to any of the ResourceProperties requires a replacement.

The Response

It’s super important that you understand how to return information to the CloudFormation service. If you don’t make a return request to CloudFormation your stack will hang until timeout. Thanks to Jorge Bastida and his cfn-response (Github link) project you can install a library through pip called cfn-response (Pypi link). You’ll find this requirement in the requirements.txt file within the demonstration. The cfn-response send function, takes a number of parameters described below and crafts an HTTPS response to the S3 signed URL that CloudFormation provides in the event. CloudFormation checks for the response in S3 periodically. It seems to check for an update every 30 seconds. You can tell it’s not a call back because sometimes there’s a period of time between when Lambda finishes and when CloudFormation updates. The arguments to the send function are as follows:

  • Event: First positional argument, it wants the event from your lambda_handler. The function uses the event for it’s StackId, RequestId, LogicalResourceId and the ResponseURL.
  • Context: Second positional argument, the context from your lambda_handler. The function uses the context for the name of the log_stream_name.
  • Response_status: Third positional argument, deems if the create, update or delete was a success for failure. The only two values that CloudFormation accepts is ‘SUCCESS’ and ‘FAILED’. Jorge created two variables named just that, that you can use. You’ll see them imported in the demonstration code.
  • reason=None: Keyword argument, defaults to None. This is what shows up in the CloudFormation Events under Status Reason. This is helpful for debugging. Anytime there’s an exception caught in the demonstration, the class sets a status reason with the exception and returns FAILED.
  • response_data=None: Keyword argument, defaults to None. The response data argument expects a dictionary. This dictionary of key value pairs is the data that is available to CloudFormation via the “Fn::GetAtt” intrinsic function. With the GetAtt function you can use the LogicalId of this custom resource, and the attribute name is any key that can be found in the response_data dictionary you return.
  • physical_resource_id=None: Keyword argument, defaults to None, however returns the log stream Id if left as None. The physical resource id argument expects a string. This is the value that will be returned within CloudFormation when you reference this custom resource by logical Id. Because the association we built does not have an Id, the demonstration does not set the physical_resource_id.

Now you understand how to run custom logic and return values dynamically back into your CloudFormation template.

Logging

Logging in Lambda can be done by using the Python standard logging library or even print. You’ll see the logging library setup in the demonstration and how it’s used. The Lambda function must run under a role with the IAM capability to CloudWatch Logs. Anything printed or logged through the logging library will be published to CloudWatch Logs.

Testing Locally

Testing during development is necessary and can be painful within Lambda and CloudFormation. If you have a bug, your CloudFormation stack can hang for hours. I suggest setting a timeout on the Update or Create when you do a test in CloudFormation. Also, until you’re ready to test in CloudFormation, set up a way to test locally.

In the demonstration I set up the Python option parser, took all of the information I needed to mimic the event object and ran the Lambda handler. By setting up a main we can call the function locally with command line arguments and use our IAM user credentials in the form of a profile rather than an IAM role. You will see some things in the demonstration code take a slightly different pattern if the context object is equal to None. Feel free to install the requirements locally and run a few tests with the command line. If you specify an Update you must also specify the “old_vpc_id” and “old_hosted_zone_id” options to build the “OldResourceProperties” object of the event.

Conclusion

There you have it! We’ve set up an example project that creates a custom resource. The resource we created was the association between a VPC and a Route 53 internal hosted zone. The CloudFormation demonstrated was in a nested pattern, creating an IAM role, a Lambda function, a Virtual Private Cloud, and invoking the Lambda function with a CloudFormation custom resource. The custom resource code demonstrated creates, updates and deletes. We also saw how to test Lambda code locally and learned a little about Lambda logging. You now know how CloudFormation custom resources pass properties to Lambda through the event object, and how to send information back to CloudFormation using the cfn-response library.

Now that you know how to extend AWS CloudFormation with Lambda, CloudFormation is essentially limitless. How will you use this tutorial to extend CloudFormation? Share your use cases below.

Editor’s Note:

Would you like to learn more? Join RightBrain Networks on Thurs., August 25 for our monthly AWS Michigan MeetUp. Derek will present on this same topic at the RightBrain Networks office in Ann Arbor. And, if you can’t be there in person, you can register for the live webcast.