CloudFormation, OpsWorks, and Elastic Beanstalk: why you can (almost) never apply too much automation to your deployment management.
If you are like me (and I know that many of you are), then I am sure this has happened to you. You get your hands on a new toy (like CloudFormation) and just want to get going. Rather than read through some of the excellent online help documentation provided by an army of technical writers, you just ‘borrow’ some Google-friendly scripts (or templates) and tweak them for your project.
Until you hit a dead end.
Something doesn’t work the way you thought it would, and you’ve got to go back and debug everything you’ve
Sound familiar? Are there times you wished you had a better understanding of a tool’s inner workings?
I’ve been using CloudFormation for quite some time now. Looking back at my path of enlightenment, I do remember things I wish I had understood better or at least given more attention at the time.
So let me share a few nuggets with you in a way that will hopefully flatten your learning curve just a bit. From there I hope to discuss some of the key opportunities and challenges that will likely follow your adventures into coded infrastructure deployments.
Open your mind to AWS automation
Newcomers to AWS cloud services may be tempted to dismiss CloudFormation templates as time-wasting “paperwork.” Why automate the provisioning of a vanilla EC2 instance with CloudFormation (or a similar tool) if you can just as easily order up a brand new clean instance from the AWS Console?
Careful. Don’t forget that launching a new server is never just about the operating system. Back in the ‘old’ days of hot, noisy server rooms, did you ever have to provision a bare metal server running nothing more than a pristine, untouched operating system? I doubt it! What about your application stack, database environment, and network configuration? Do you remember how many days and weeks it took to get a new server running the way you like (even if we ignore the ordering and delivery process)?
And when was the last time that you needed only one of each type? Once upon a time, “one process” meant one server. Ha! I also remember those good old days, when development, test, and production were all based on a single code base running on a server hidden somewhere in the attic. Today you tend to require at least four environments to facilitate the software development life cycle. For good measure, you’ll probably also want to add one more for a BlueGreen Deployment.
So even virtual servers take time to build, and even (especially) virtual servers need to be spun up multiple times. This sounds like an argument for managing your infrastructure as code. Sounds like automation just became king of the jungle.
AWS provides great helper scripts that come pre-installed with all Amazon provided machine images or as executables for installation on your own images. In combination with the instructions you provide within the stack templates, those automation scripts enable you to deploy an entire infrastructure stack with just a few clicks (unless, of course, you choose to automate even this through the CloudFormation API).
Understanding the interdependencies and different roles played by the various sections of the template and automation scripts will help you successfully develop your stack.
CloudFormation init (cfn-init)
The most commonly used CloudFormation script (besides cloud-init – more on that in a later post) would unarguably be cfn-init (“CloudFormation init”). cfn-init reads and processes the instructions provided within template metadata. To run cfn-init you need to call it from within the user data instructions or as part of any of your image’s start-up processes.
You might like to know that the user data instructions are ‘magically’ executed by cloud-init. Cloud-init is an open source package that is widely used for bootstrapping cloud instances. More details on this, too, will have to wait for another post.
Cfn-init accepts a number of command line options. At a minimum, you will need to provide the name of the CloudFormation stack and the name of the element that contains the instance metadata instructions.
/opt/aws/bin/cfn-init -v --stack YourStackName --resource YourResourceName
This could either be the launch configuration or an EC2 instance definition inside the CloudFormation template.
It is important for you to understand that the instance itself isn’t ‘seeded’ with the template instructions as part of the launch. In fact, the instance itself has no knowledge of the fact that its launch was initiated by CloudFormation. Instead, the cfn-init script reaches out to the public CloudFormation API endpoint to retrieve the template instructions. This is important to remember if you’re going to launch your instance from inside a VPC that has no Internet connectivity or that gets connectivity via a proxy server that needs special configuration.
CloudFormation and Configuration sets
CloudFormation init instructions can be grouped into multiple configuration sets. I strongly suggest you take advantage of this to allow greater resource isolation and more modular configurations (i.e., template fragments that can more easily be reused). With its procedural template instructions, CloudFormation doesn’t necessarily support DRY coding practices – and nor should it.
However, if your setup requires you to install a common set of applications or configurations on each instance (think: anti-virus, regulatory compliance, or log forwarding agents), you will be well served by separating each element into its own configuration settings. Doing this alongside a centralized source control management system or an advanced text editor like Sublime or Notepad++, you can then easily maintain and re-use common stack elements.
Note: this isn’t the only way to ensure common components are always rolled into the stack. In a previous post on architecting on AWS, I wrote about the advantages and trade-offs for scripted launches vs. the use of pre-baked, customised machine images.
However, configuration sets don’t necessarily scale well for larger environments. If you want to automate your infrastructure across tens or hundreds of templates, you will soon hit limits. As your environment requires patching, and you start refactoring your code fragments, you need to manually ensure that every stack in your environment is kept up-to-date.
Once you reach that point, you should start to explore the use of Continuous Deployment solutions that can hook into AWS for a more automated management of stacks across multiple environments.
Keep an open mind about alternatives
Which leads nicely to my closing words. I am sure you’ve all heard the popular saying:
‘If all you have is a hammer, everything looks like a nail‘.
Rest assured that your infrastructure and deployment solution is subject to the same paradigm. When I started using scripted deployments on AWS, I made good use of the user data scripts. I split everything up into individual bash or PowerShell scripts that I deployed to the instances, and called them from within the user data or cascaded them amongst each other. And I felt very clever!
At least, until my fleet of instances started to grow. Then I discovered that a lot of that effort could be avoided by using CloudFormation. So my instance definitions moved to CloudFormation Init metadata, which gave me additional flexibility. CloudFormation Init then allowed me to define in a declarative way what actions I wanted to perform on an instance, and in which order – much like a YAM-based cloud-init configuration, but on a full-stack scale rather than a single instance. No longer did I have to navigate to a specific directory, download an RPM package using wget or curl, install it using the package manager, ensure the application is started at boot time, and so on. Instead, I can just provide declarative instructions inside one or more of the seven supported configuration keys.
Once again, I began to feel very smart. I started to organize my individual declarative instructions in configuration sets, managed them in a central repository for re-use, etc. Until – well you can probably already guess it by now – until I discovered that it was time to consider the use of AWS Opsworks and Elastic Beanstalk resources inside my CloudFormation stack.
AWS Opsworks abstracts your configuration instructions further away from the declarative configuration in the init metadata. Using a managed Chef service you have access to a large variety of pre-defined recipes for the installation and configuration of additional system components. Since those recipes are continuously maintained and updated by the wider community, you don’t need to re-invent the wheel over and over again.
Not that re-inventing the wheel has no benefits; imagine if we still used stone. But it’s obvious that the wisdom and throughput of a whole community can be much higher than the capability of an individual.
The same can be said for Elastic Beanstalk. Where OpsWorks helps you to accelerate the deployment of common components, Elastic Beanstalk lets you automate the resilient and scalable deployment of your application into the stack without you even having to describe or configure the details for load balancing and scaling.
CloudFormation: in summary
The point I would like to make is that in a world where “the slow eats the fast”, we can never settle permanently for any given solution. The whole technology community, including AWS, is constantly evolving to allow organizations to innovate, develop and ship features at an ever-increasing rate. This is achieved partly through the continuous abstraction away from the core underlying infrastructure and services and through a combination of traditional features with new functionality and innovation.
To stay on top of the game as an IT professional, you will need to constantly challenge the status quo and, where applicable, make the leap of faith to investigate and learn new ways of doing our business.
New on Cloud Academy: Git Labs, CKA and CKAD Lab Challenges, AWS and Azure Learning Paths, AGILE, and Much More
We just kicked off our first Free Weekend of 2020. This means we've unlocked our Training Library for just 72 hours. Until Sunday at 11:59 pm (PST), you can get unlimited access to our industry-leading learning paths, courses, certification prep exams, and our most popular hands-on labs...
How to Encrypt an EBS Volume
Keeping data and applications safe in the cloud is one of the most visible challenges facing cloud teams in 2020. Cloud storage services where data resides are frequently a target for hackers, not because the services are inherently weak but because they are often improperly configured....
Heroku vs. AWS: Which Cloud Solution Works Best in 2020
Heroku vs. AWS: Introduction Сloud-based platforms get more and more recognition. According to Statista, just in the third quarter of 2019, cloud market revenues reached $27.5 billion. By moving to the cloud, businesses can focus on their strategy and other processes instead of dealing...
How to Unlock Complimentary Access to Cloud Academy
Are you looking to get trained or certified on AWS, Azure, Google Cloud Platform, DevOps, Cybersecurity, Information Security, Python, Java, or another technical skill? Then you'll want to mark your calendars. Starting Friday, February 7 at 12:00 a.m. PST (3:00 a.m. EST), Cloud Acade...
Cloud Academy’s Blog Digest: Top 5 AWS Salary Report Findings, How To Become a Cybersecurity Professional, 8 Financial Benefits of Cloud Migration, and more
Now that it's 2020, how many times have you caught yourself dating a paper 2019? Don't lie. It's happened at least once or twice — or a handful of times — I'm sure. And if you're a member of the "perfect club" that hasn't made any 2020 mistakes, then we're still happy to have you in our...
Top 5 AWS Salary Report Findings
At the speed the cloud tech space is developing, it can be hard to keep track of everything that’s happening within the AWS ecosystem. Advances in technology prompt smarter functionality and innovative new products, which in turn give rise to new job roles that have a ripple effect on t...
New on Cloud Academy: Red Hat, Agile, OWASP Labs, Amazon SageMaker Lab, Linux Command Line Lab, SQL, Git Labs, Scrum Master, Azure Architects Lab, and Much More
Happy New Year! We hope you're ready to kick your training in overdrive in 2020 because we have a ton of new content for you. Not only do we have a bunch of new courses, hands-on labs, and lab challenges on AWS, Azure, and Google Cloud, but we also have three new courses on Red Hat, th...
Cloud Academy’s Blog Digest: Azure Best Practices, 6 Reasons You Should Get AWS Certified, Google Cloud Certification Prep, and more
Happy Holidays from Cloud Academy We hope you have a wonderful holiday season filled with family, friends, and plenty of food. Here at Cloud Academy, we are thankful for our amazing customer like you. Since this time of year can be stressful, we’re sharing a few of our latest article...
Google Cloud Platform Certification: Preparation and Prerequisites
Google Cloud Platform (GCP) has evolved from being a niche player to a serious competitor to Amazon Web Services and Microsoft Azure. In 2019, research firm Gartner placed Google in the Leaders quadrant in its Magic Quadrant for Cloud Infrastructure as a Service for the second consecuti...
New Lab Challenges: Push Your Skills to the Next Level
Build hands-on experience using real accounts on AWS, Azure, Google Cloud Platform, and more Meaningful cloud skills require more than book knowledge. Hands-on experience is required to translate knowledge into real-world results. We see this time and time again in studies about how pe...
New on Cloud Academy: AWS Solution Architect Lab Challenge, Azure Hands-on Labs, Foundation Certificate in Cyber Security, and Much More
Now that Thanksgiving is over and the craziness of Black Friday has died down, it's now time for the busiest season of the year. Whether you're a last-minute shopper or you already have your shopping done, the holidays bring so much more excitement than any other time of year. Since our...
Understanding Enterprise Cloud Migration
What is enterprise cloud migration? Cloud migration is about moving your data, applications, and even infrastructure from your on-premises computers or infrastructure to a virtual pool of on-demand, shared resources that offer compute, storage, and network services at scale. Why d...