How to use Cloudformation in AWS to create a self terminating instance

One of the nice features of AWS is the ability to bring boxes up and down in an automated fashion using autoscaling groups, no matter how many times I see it happen it still puts a smile across my geeky face, it's magic I tell yah!

Leveraging the power of AWS autoscaling groups it's also possible to bring instances up and down to a schedule, I've found this useful for bringing up instances to perform batch tasks that terminate themselves once the task is finished, saving you some cash in the process and also allowing you to use a instance to get the job quickly.

The autoscaling group

The key to the whole thing is the autoscale group, defining a group with a min and max of zero creates an instanceless stack, here's a quick Cloudformation snippet that demonstrates this:

"SelfTerminatingAutoScalingGroup":{
   "Type":"AWS::AutoScaling::AutoScalingGroup",
   "Properties":{
      "AvailabilityZones":{
         "Fn::GetAZs":{
            "Ref":"AWS::Region"
         }
      },
      "LaunchConfigurationName":{
         "Ref":"SelfTerminatingLaunchConfiguration"
      },
      "Tags":[
         {
            "Key":"Name",
            "Value":"SelfTerminating",
            "PropagateAtLaunch":"true"
         }
      ],
      "MinSize":"0",
      "MaxSize":"0",
      "DesiredCapacity":"0"
   }
}

Scheduled Tasks

Scheduled tasks are a feature of AWS autoscaling that enables you to scale a group up or down at a given time. Normally this would be used to add or subtract instances during predictable quiet or busy periods. In our case we simply want to bring up a single box that will that perform our batch task, here's a CF snippet that will bring up an instance everyday at midnight:

"SelfTerminatingScaleUpScheduledAction":{
   "Type":"AWS::AutoScaling::ScheduledAction",
   "Properties":{
      "AutoScalingGroupName":{
         "Ref":"SelfTerminatingAutoScalingGroup"
      },
      "MaxSize":"1",
      "MinSize":"1",
      "DesiredCapacity":"1",
      "Recurrence":"0 0 * * *"
   }
}

Launch Configuration

This where the magic happens! The launch configuration describes what each instance in our autoscale group should look like, when the scheduled action scales up the group this configuration will be used to bring up an instance. The interesting bit here is the UserData field, here we describe a bootstrap process for each instance, and this is where we can stick our batch task, followed by a calll to the AWS API that will scale the autoscale group back to zero instances, effectively terminating the instance.

"SelfTerminatingLaunchConfiguration":{
   "Type":"AWS::AutoScaling::LaunchConfiguration",
   "Metadata":{
      "AWS::CloudFormation::Init":{
         "config":{
            "packages":{
               "yum":{
                  "aws-cli":[

                  ]
               }
            }
         }
      }
   },
   "Properties":{
      "ImageId":{
         "Fn::FindInMap":[
            "AWSRegionArch2AMI",
            {
               "Ref":"AWS::Region"
            },
            {
               "Fn::FindInMap":[
                  "AWSInstanceType2Arch",
                  {
                     "Ref":"InstanceType"
                  },
                  "Arch"
               ]
            }
         ]
      },
      "InstanceType":{
         "Ref":"InstanceType"
      },
      "SecurityGroups":[
         {
            "Ref":"SelfTerminatingSecurityGroup"
         }
      ],
      "KeyName":{
         "Ref":"KeyName"
      },
      "IamInstanceProfile":{
         "Ref":"SelfTerminatingInstanceProfile"
      },
      "UserData":{
         "Fn::Base64":{
            "Fn::Join":[
               "",
               [
                  "#!/bin/bash\n",
                  "yum update -y aws-cfn-bootstrap\n",
                  "/opt/aws/bin/cfn-init -s ",
                  {
                     "Ref":"AWS::StackId"
                  },
                  " -r SelfTerminatingLaunchConfiguration",
                  " --region ",
                  {
                     "Ref":"AWS::Region"
                  },
                  "\n",
                  "echo 'Do something useful'\n",
                  "instance_id=`curl http://169.254.169.254/latest/meta-data/instance-id`\n",
                  "autoscale_group=`aws ec2 describe-tags --filters \"Name=resource-id,Values=$instance_id\"",
                  " --region ",
                  {
                     "Ref":"AWS::Region"
                  },
                  " \"Name=key,Values=aws:autoscaling:groupName\"",
                  " | sed -ne 's\/[ ]*\"Value\":\\s\"\\(.*\\)\",\/\\1\/p'`\n",
                  "aws autoscaling update-auto-scaling-group --auto-scaling-group-name $autoscale_group",
                  " --region ",
                  {
                     "Ref":"AWS::Region"
                  },
                  " --min-size 0 --max-size 0 --desired-capacity 0\n"
               ]
            ]
         }
      }
   }
}

In the example above the batch task simply consists of echoing 'Do something useful' but of course you could put any batch task here. Let's take a look at the last couple of lines of the UserData bootstrap, these are the AWS CLI commands that scale down the instances autoscale group:

"instance_id=`curl http://169.254.169.254/latest/meta-data/instance-id`\n",

"autoscale_group=`aws ec2 describe-tags --filters "Name=resource-id,Values=$instance_id"", " --region ", {"Ref":"AWS::Region"}, " "Name=key,Values=aws:autoscaling:groupName"", " | sed -ne 's/[ ]*"Value":\s"\(.*\)",/\1/p'`\n",

"aws autoscaling update-auto-scaling-group --auto-scaling-group-name $autoscale_group", " --region ", { "Ref":"AWS::Region" }, " --min-size 0 --max-size 0 --desired-capacity 0\n"

The first command uses the AWS metadata service to get the instances id. Next we need to get the autoscaling group that the instance belongs to, we can find by looking at the tags that have been assigned to the instance, one of which will be the autoscale group assigned to the instance. When creating resources via CF they will be named dynamically (eg. SelfTerminating-SelfTerminatingAutoScalingGroup-5Y4GF2JAJMV7) so you can't rely on a predefined value. So use the describe-tags command to grab the tag that contains the autoscale group name and then use sed to extract just the name, discarding the rest of the response from the API. Now that we have the group name, we can simply use the API to scale the group back down to zero as shown in the last command.

Tying it all together

Here's a full example of a self terminating instance in a Cloudformation template, this example is parametised so you can select your scale up time as needed. The IAM persmissions are required to allow an instance to update it's autoscale group.

For your actual batch task you could hard code it into the CF template, what I've done in the past is though create a shell script that contains all the commands that I want to run, I put this into S3 (and into a VCS as well) and add an IAM role to the CF template that gives instances access to this script. The instance can then simply download the script via the AWS API and then execute it. You can capture the output of the script and save it to a log file which can then be uploaded to S3 (o emailed) so you can see how the script ran, because the instance kills itself after it has finished there will not be any server logs to look at should anything fail.