Cloud Foundry: Understanding the Core Components
Explore the key components you'll need to build an entire Cloud Foundry architecture.In a recent post, I spoke about some of Cloud Foundry's main...Learn More
Pivotal Cloud Foundry (PCF) is an open source platform based on Cloud Foundry and offered as a collaboration between Pivotal, EMC, and GE. Pivotal Cloud Foundry runs on almost all popular cloud infrastructures, including VMWare, AWS, and OpenStack. PCF as a platform is dynamic, developer friendly, and features full-lifecycle support.
Organizations implementing Pivotal Cloud Foundry as their cloud platform free themselves from managing application infrastructure. When integrating freely available third-party tools and services, they can also achieve high-availability, auto-scaling, dynamic routing, multi-lingual support, and log analysis.
Pivotal Cloud Foundry performs exceedingly well when intelligently designed and maintained, but there are still some time-consuming tasks that demand an admin’s attention. One such operational task is ensuring that installation settings and essential internal databases are regularly backed up. Pivotal recommends that you back up your installation settings by exporting them at regular intervals (weekly, bi-weekly, monthly, etc). We’re going to discuss designing an effective and reliable back up process…and how to apply an archive when you need to restore your installation.
Note: According to Pivotal Cloud Foundry documentation, exporting your installation only backs up your installation settings. It does not back up your VMs or any external MySQL databases that you might have configured on the Ops Manager Director Config page.
Before jumping in, it’s a good idea to make sure that you’ve covered all the prerequisites you’ll need to make Pivotal Cloud Foundry happy. You’ll need:
Backing up a Pivotal installation is critical for the operation and availability of your Pivotal Cloud Foundry data center. Backing up Pivotal Cloud Foundry data centers is like creating restore points on a Windows machine. In the event of a crash or the failure of an upgrade process, you can restore your back up settings to fall back to an earlier, functional image. Here’s what you’ll need to do:
To make sure that your system is ready for the process, there are some important details that will need taking care of in the pre-backup stage:
Pivotal Cloud Foundry’s Cloud Controller Database maintains a database with records of orgs, spaces, apps, services, service instances, user roles, etc. Backing up this database is critical if you want to protect your existing settings (and you DO want to protect your existing settings).
$bosh target <IP_OF_YOUR_OPS_MANAGER_DIRECTOR> $bosh login
Your username: director Enter password: Logged in as `director'
$bosh deployments >> /pcf-backup/deployments_09_20_2015.txt
$bosh download manifest DEPLOYMENT-NAME LOCAL-SAVE-NAME $bosh download manifest cf-1234xyzabcd1234 cf-backup-09_20_2015.yml
$bosh deployment cf-backup-09_20_2015.yml
$bosh vms cf-1234xyzabcd1234
$bosh -d cf-backup-09_20_2015.yml stop cloud_controller-partition-cdabcd1234b253f40 $bosh -d cf-backup-09_20_2015.yml stop cloud_controller_worker-partition- cdabcd1234b253f40
ccdb: address: 18.104.22.168 port: 2544 db_scheme: postgres
vm Credentials vcap / xyz1234567989pqr
$find /var/vcap | grep 'bin/psql'
Your output should look something like this:
$/var/vcap/data/packages/postgres/<random-string>/bin/pg_dump -h 22.214.171.124 -U admin -p 2544 ccdb > ccdb_09_20_2015.sql
#scp email@example.com:/home/vcap/ccdb_09_20_2015.sql /pcf-backup
This will complete the CCDB backup process.
vm Credentials vcap / xxxxxxxxxxxx Credentials root / xxxxxxxxxxxxxxxxxx
$find /var/vcap | grep 'bin/psql'
#/var/vcap/data/packages/postgres/<random-string>/bin/pg_dump -h 126.96.36.199 -U root -p 2544 uaa > uaa_09_20_2015.sql
# scp firstname.lastname@example.org:/home/vcap/uaa_09_20_2015.sql /pcf-backup
This completes the UAADB backup process.
The Console Database is referred to as the Apps Manager Database in Elastic Runtime 1.5.
Vm Credentials vcap / xxxxxxxxxxxxx Credentials root / xxxxxxxxxxxxxxxxxxx
$find /var/vcap | grep 'bin/psql' $/var/vcap/data/packages/postgres/<random-string>/bin/pg_dump -h 188.8.131.52 -U root -p 2544 console > console_09_20_2015.sql
$scp email@example.com:/home/vcap/console_09_20_2015.sql /pcf-backup
This completes the Console Database backup process.
$tar cz shared > nfs_09_20_2015.tar.gz
$scp firstname.lastname@example.org:/var/vcap/store/nfs_09_20_2015.tar.gz /pcf-backup
This completes the NFS Server backup process.
Backup your MySQL Database:
$bosh download manifest p-mysql-abcd1234f2ad3752 mysql_09_20_2015.yml
$mysqldump -u root -p -h 184.108.40.206 --all-databases > user_databases_09_20_2015.sql
This completes the MySQL DB backup process.
$bosh -d cf-backup-09_20_2015.yml start cloud_controller-partition-cdabcd1234b253f40 $bosh -d cf-backup-09_20_2015.yml start cloud_controller_worker-partition- cdabcd1234b253f40
Restoring a Pivotal Cloud Foundry deployment requires that you reinstall your installation settings restoration and key system databases. Or, in other words, everything we backed up in the previous operations. You’ll need to follow these steps:
We’ll use the UAADB as an example. The rest will follow the same process.
$ bosh stop <uaa job>
$ssh vcap@[uaadb vm ip]
$/var/vcap/data/packages/ /postgres/<random-string> /bin/psql -U vcap -p 2544 uaa
drop schema public cascade; create schema public;
$scp uaa.sql vcap@[uaadb vm IP]: #UAADB server $/var/vcap/data/packages/postgres/<random-string>/bin/psql -U vcap -p 2544 uaa < uaa.sql
$bosh start <uaa job>
A Pivotal Cloud Foundry backup process can be scheduled (and scripted) to create restore points for your installation. You could also use the settings file backups to launch a new installation in a different availability zone or even on a different platform. PCF has provided excellent documentation on both the backup and restore process.
Thoughts? Add your comments below.
AWS's WaitCondition can be used with CloudFormation templates to ensure required resources are running.As you may already be aware, AWS CloudFormation is used for infrastructure automation by allowing you to write JSON templates to automatically install, configure, and bootstrap your ...
As companies increasingly shift workloads to the public cloud, cloud computing has moved from a nice-to-have to a core competency in the enterprise. This shift requires a new set of skills to design, deploy, and manage applications in the cloud.As the market leader and most mature p...
The announcements at re:Invent just keep on coming! Let’s look at what benefits these two new EC2 instance types offer and how these two new instances could be of benefit to you. If you're not too familiar with Amazon EC2, you might want to familiarize yourself by creating your first Am...
Google Cloud Platform (GCP) has evolved from being a niche player to a serious competitor to Amazon Web Services and Microsoft Azure. In 2018, research firm Gartner placed Google in the Leaders quadrant in its Magic Quadrant for Cloud Infrastructure as a Service for the first time. In t...
In order to understand AWS VPC egress filtering methods, you first need to understand that security on AWS is governed by a shared responsibility model where both vendor and subscriber have various operational responsibilities. AWS assumes responsibility for the underlying infrastructur...
Is it possible to create an S3 FTP file backup/transfer solution, minimizing associated file storage and capacity planning administration headache?FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have conf...
Microservices are a way of breaking large software projects into loosely coupled modules, which communicate with each other through simple Application Programming Interfaces (APIs).Microservices have become increasingly popular over the past few years. The modular architectural style,...
There are many use cases for tags, but what are the best practices for tagging AWS resources? In order for your organization to effectively manage resources (and your monthly AWS bill), you need to implement and adopt a thoughtful tagging strategy that makes sense for your business. The...
Amazon S3 is the most common storage options for many organizations, being object storage it is used for a wide variety of data types, from the smallest objects to huge datasets. All in all, Amazon S3 is a great service to store a wide scope of data types in a highly available and resil...
One of the main promises of cloud computing is access to nearly endless capacity. However, it doesn’t come cheap. With the introduction of Spot Instances for Amazon Web Services’ Elastic Compute Cloud (AWS EC2) in 2009, spot instances have been a way for major cloud providers to sell sp...
A Comparison of Machine Learning Services on AWS, Azure, and Google CloudArtificial intelligence and machine learning are steadily making their way into enterprise applications in areas such as customer support, fraud detection, and business intelligence. There is every reason to beli...
The AWS Command Line Interface (CLI) is for managing your AWS services from a terminal session on your own client, allowing you to control and configure multiple AWS services.So you’ve been using AWS for awhile and finally feel comfortable clicking your way through all the services....