CloudFormation and AWS Deployment Automation

CloudFormation, OpsWorks, and Elastic Beanstalk: why you can (almost) never apply too much automation to your deployment management.

If you are like me (and I know that many of you are), then I am sure this has happened to you. You get your hands on a new toy (like CloudFormation) and just want to get going. Rather than read through some of the excellent online help documentation provided by an army of technical writers, you just ‘borrow’ some Google-friendly scripts (or templates) and tweak them for your project.

Until you hit a dead end.

Something doesn’t work the way you thought it would, and you’ve got to go back and debug everything you’ve done borrowed.

Sound familiar? Are there times you wished you had a better understanding of a tool’s inner workings?

AWS CloudFormation iconI’ve been using CloudFormation for quite some time now. Looking back at my path of enlightenment, I do remember things I wish I had understood better or at least given more attention at the time.

So let me share a few nuggets with you in a way that will hopefully flatten your learning curve just a bit. From there I hope to discuss some of the key opportunities and challenges that will likely follow your adventures into coded infrastructure deployments.

Open your mind to AWS automation

Newcomers to AWS cloud services may be tempted to dismiss CloudFormation templates as time-wasting “paperwork.” Why automate the provisioning of a vanilla EC2 instance with CloudFormation (or a similar tool) if you can just as easily order up a brand new clean instance from the AWS Console?

Careful. Don’t forget that launching a new server is never just about the operating system. Back in the ‘old’ days of hot, noisy server rooms, did you ever have to provision a bare metal server running nothing more than a pristine, untouched operating system? I doubt it! What about your application stack, database environment, and network configuration? Do you remember how many days and weeks it took to get a new server running the way you like (even if we ignore the ordering and delivery process)?

And when was the last time that you needed only one of each type? Once upon a time, “one process” meant one server. Ha! I also remember those good old days, when development, test, and production were all based on a single code base running on a server hidden somewhere in the attic. Today you tend to require at least four environments to facilitate the software development life cycle. For good measure, you’ll probably also want to add one more for a BlueGreen Deployment.

So even virtual servers take time to build, and even (especially) virtual servers need to be spun up multiple times. This sounds like an argument for managing your infrastructure as code. Sounds like automation just became king of the jungle.

AWS provides great helper scripts that come pre-installed with all Amazon provided machine images or as executables for installation on your own images. In combination with the instructions you provide within the stack templates, those automation scripts enable you to deploy an entire infrastructure stack with just a few clicks (unless, of course, you choose to automate even this through the CloudFormation API).

Understanding the interdependencies and different roles played by the various sections of the template and automation scripts will help you successfully develop your stack.

CloudFormation init (cfn-init)

The most commonly used CloudFormation script (besides cloud-init – more on that in a later post) would unarguably be cfn-init (“CloudFormation init”). cfn-init reads and processes the instructions provided within template metadata. To run cfn-init you need to call it from within the user data instructions or as part of any of your image’s start-up processes.

You might like to know that the user data instructions are ‘magically’ executed by cloud-init. Cloud-init is an open source package that is widely used for bootstrapping cloud instances. More details on this, too, will have to wait for another post.

Cfn-init accepts a number of command line options. At a minimum, you will need to provide the name of the CloudFormation stack and the name of the element that contains the instance metadata instructions.

/opt/aws/bin/cfn-init -v --stack YourStackName --resource YourResourceName

This could either be the launch configuration or an EC2 instance definition inside the CloudFormation template.

It is important for you to understand that the instance itself isn’t ‘seeded’ with the template instructions as part of the launch. In fact, the instance itself has no knowledge of the fact that its launch was initiated by CloudFormation. Instead, the cfn-init script reaches out to the public CloudFormation API endpoint to retrieve the template instructions. This is important to remember if you’re going to launch your instance from inside a VPC that has no Internet connectivity or that gets connectivity via a proxy server that needs special configuration.

CloudFormation and Configuration sets

CloudFormation init instructions can be grouped into multiple configuration sets. I strongly suggest you take advantage of this to allow greater resource isolation and more modular configurations (i.e., template fragments that can more easily be reused). With its procedural template instructions, CloudFormation doesn’t necessarily support DRY coding practices – and nor should it.

However, if your setup requires you to install a common set of applications or configurations on each instance (think: anti-virus, regulatory compliance, or log forwarding agents), you will be well served by separating each element into its own configuration settings. Doing this alongside a centralized source control management system or an advanced text editor like Sublime or Notepad++, you can then easily maintain and re-use common stack elements.

Note: this isn’t the only way to ensure common components are always rolled into the stack. In a previous post on architecting on AWS, I wrote about the advantages and trade-offs for scripted launches vs. the use of pre-baked, customised machine images.

However, configuration sets don’t necessarily scale well for larger environments. If you want to automate your infrastructure across tens or hundreds of templates, you will soon hit limits. As your environment requires patching, and you start refactoring your code fragments, you need to manually ensure that every stack in your environment is kept up-to-date.

Once you reach that point, you should start to explore the use of Continuous Deployment solutions that can hook into AWS for a more automated management of stacks across multiple environments.

Keep an open mind about alternatives

Which leads nicely to my closing words. I am sure you’ve all heard the popular saying:

If all you have is a hammer, everything looks like a nail‘.

Rest assured that your infrastructure and deployment solution is subject to the same paradigm. When I started using scripted deployments on AWS, I made good use of the user data scripts. I split everything up into individual bash or PowerShell scripts that I deployed to the instances, and called them from within the user data or cascaded them amongst each other. And I felt very clever!

At least, until my fleet of instances started to grow. Then I discovered that a lot of that effort could be avoided by using CloudFormation. So my instance definitions moved to CloudFormation Init metadata, which gave me additional flexibility. CloudFormation Init then allowed me to define in a declarative way what actions I wanted to perform on an instance, and in which order – much like a YAM-based cloud-init configuration, but on a full-stack scale rather than a single instance. No longer did I have to navigate to a specific directory, download an RPM package using wget or curl, install it using the package manager, ensure the application is started at boot time, and so on. Instead, I can just provide declarative instructions inside one or more of the seven supported configuration keys.

Once again, I began to feel very smart. I started to organize my individual declarative instructions in configuration sets, managed them in a central repository for re-use, etc. Until – well you can probably already guess it by now – until I discovered that it was time to consider the use of AWS Opsworks and Elastic Beanstalk resources inside my CloudFormation stack.

AWS Opsworks abstracts your configuration instructions further away from the declarative configuration in the init metadata. Using a managed Chef service you have access to a large variety of pre-defined recipes for the installation and configuration of additional system components. Since those recipes are continuously maintained and updated by the wider community, you don’t need to re-invent the wheel over and over again.

Not that re-inventing the wheel has no benefits; imagine if we still used stone. But it’s obvious that the wisdom and throughput of a whole community can be much higher than the capability of an individual.

The same can be said for Elastic Beanstalk. Where OpsWorks helps you to accelerate the deployment of common components, Elastic Beanstalk lets you automate the resilient and scalable deployment of your application into the stack without you even having to describe or configure the details for load balancing and scaling.

CloudFormation: in summary

The point I would like to make is that in a world where “the slow eats the fast”, we can never settle permanently for any given solution. The whole technology community, including AWS, is constantly evolving to allow organizations to innovate, develop and ship features at an ever-increasing rate. This is achieved partly through the continuous abstraction away from the core underlying infrastructure and services and through a combination of traditional features with new functionality and innovation.

To stay on top of the game as an IT professional, you will need to constantly challenge the status quo and, where applicable, make the leap of faith to investigate and learn new ways of doing our business.

We at Cloud Academy are here to support you in providing individual courses and learning paths that allow you to accelerate your learning in new areas. Not only on CloudFormation.

Avatar

Written by

Christian Petters

As a Solutions Architect, Christian is helping organisations to find the most appropriate solution to address their unique business problems. He is passionate about the opportunities that are provided by modern cloud services and covers topics in AWS and MS Azure with a particular focus on Microsoft technologies.


Related Posts

Alisha Reyes
Alisha Reyes
— April 8, 2020

New on Cloud Academy: AWS Solutions Architect – Associate Exam Prep, Azure Courses, Google Associate Cloud Engineer Exam Prep, Programming Labs, and Much More

Free content on Cloud Academy More and more customers are relying on our technology and content to keep upskilling their people in these months, and we are doing our best to keep supporting them. While the world fights the COVID-19 pandemic, we wanted to make a small contribution to he...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • programming
Joe Nemer
Joe Nemer
— April 3, 2020

Breaking News: All AWS Certification Exams Now Available Online

Remote proctoring for all AWS certifications Cloud Academy is an Advanced AWS Technology Partner, and we are happy to announce all AWS certification exams are available online!  What does this mean for you? You can stay focused on your certification goal. Or you can start a certifica...

Read more
  • AWS
  • AWS certification
  • AWS Certifications
Connie Benton
Connie Benton
— April 1, 2020

How To Build a Career with AWS Certifications

From Iaas and PaaS solutions to digital marketing, cloud computing reshapes the world of technology. As the influence of this technology grows, so does investment. Tens of billions of dollars are being spent on cloud computing-related services each year. This influx is continuing to inc...

Read more
  • AWS
  • Certifications
Vijayakumar Athithan
Vijayakumar Athithan
— March 27, 2020

What is Cognito in AWS?

Web applications usually allow a valid username and password combination for successful sign in to the application. Modern authentication flows incorporate more approaches to ensure user authentication. When using AWS, this is no exception, thanks to the abilities and features offered b...

Read more
  • AWS
  • AWS Cognito
  • Solutions Architect
Avatar
Andrew Larkin
— March 20, 2020

The 12 AWS Certifications: Which is Right for You and Your Team?

As companies increasingly shift workloads to the public cloud, cloud computing has moved from a nice-to-have to a core competency in the enterprise. This shift requires a new set of skills to design, deploy, and manage applications in cloud computing. As the market leader and most ma...

Read more
  • AWS
  • AWS Certifications
Alisha Reyes
Alisha Reyes
— March 17, 2020

Cloud Academy’s Blog Digest: How Do AWS Certifications Increase Your Employability, How to Become a Microsoft Certified Azure Data Engineer, and more

With everything going on right now, it's likely that the only thing you've been reading lately is related to the coronavirus pandemic. It's important to stay informed during these times, but it's also good to jump into something that can take your mind off of the current situation for j...

Read more
  • AWS
  • Azure
  • blog digest
  • Certifications
  • Cloud Academy
  • programming
  • Security
Avatar
Cloud Academy Team
— March 13, 2020

Which Certifications Should I Get?

As we mentioned in an earlier post, the old AWS slogan, “Cloud is the new normal” is indeed a reality today. Really, cloud has been the new normal for a while now and getting credentials has become an increasingly effective way to quickly showcase your abilities to recruiters and compan...

Read more
  • AWS
  • Azure
  • Certifications
  • Cloud Computing
  • Google Cloud Platform
Alisha Reyes
Alisha Reyes
— March 7, 2020

New on Cloud Academy: Intro to GitOps; AWS Courses; Java, Python, Amazon Linux 2, Ubuntu, & Docker Playgrounds; and much more

New Lab Playgrounds This month, our Content Team released six new "playground labs." Our playground labs provide a safe and secure sandbox environment for you to explore your own ideas, follow along with Cloud Academy courses, or answer your own questions — all without having to instal...

Read more
  • AWS
  • Azure
  • gitops
  • Google Cloud Platform
  • lab playground
  • programming
Alisha Reyes
Alisha Reyes
— March 6, 2020

New on Cloud Academy: Intro to GitOps; AWS Courses; Java, Python, Amazon Linux 2, Ubuntu, & Docker Playgrounds; and much more

New Lab Playgrounds This month, our Content Team released six new "playground labs." Our playground labs provide a safe and secure sandbox environment for you to explore your own ideas, follow along with Cloud Academy courses, or answer your own questions — all without having to instal...

Read more
  • AWS
  • Azure
  • gitops
  • Google Cloud Platform
  • lab playground
  • programming
Patrick Navarro
Patrick Navarro
— March 4, 2020

AWS Certifications: How Do They Increase Your Employability and Progress Your Career?

AWS certifications are no walk in the park. They’re designed to validate in-depth, specialist knowledge and comprehensive experience, often requiring months of dedicated studying to earn even for those already working with the cloud platform. But the rewards that AWS professionals ca...

Read more
  • AWS
  • AWS certification
  • certification
Avatar
Chandan Patra
— February 21, 2020

Elasticsearch vs. CloudSearch: AWS Cloud Search Choices

Elasticsearch vs. CloudSearch: What's the main difference? Let's compare AWS-based cloud tools: Elasticsearch vs. CloudSearch. While both services use proven technologies, Elasticsearch is more popular, open source, and has a flexible API to use for customization; in comparison, CloudS...

Read more
  • AWS
  • Azure
  • cloudsearch
  • elasticsearch
Avatar
Andrew Larkin
— February 13, 2020

Cloud Academy Content Roadmap Updates

Welcome to our Q1 2020 roadmap. This is the content we plan to build over the next three months, between February 1 - and April 30, 2020. Let's look at some of our roadmap highlights. Atlassian Bamboo for CI/CD We had a lot of requests for practical guides on how to apply DevOps tool...

Read more
  • Artificial Intelligence
  • AWS
  • Azure
  • Docker
  • Google Cloud Platform
  • Kubernetes
  • Machine Learning