Skip to main content

SaltStack Deployments: best practices for automation

You must perform tasks on multiple servers. You log-in to each server carefully and work. What if you have thousands of servers? SaltStack is the way to go.

You must perform certain tasks on multiple servers. You log-in to each one and complete these tasks carefully. Perhaps you might want to undertake more complicated operations, like installing software and configuring it based on your specific criteria. What if you have hundreds or thousands of servers? Imagine logging onto them one by one and carrying out these tasks? THERE IS A BETTER WAY! SaltStack deployments is it.

SaltStack is a distributed remote execution and configuration management system used to run commands on targeted nodes. It’s used for massive scale deployments following client-server model (master-minion ) over a secure and encrypted protocol. It’s a command line tool written in Python, and is lightweight as far as resources and requirements are concerned. SaltStack uses the Python ZeroMQ library to accomplish these tasks at high speed. It is open sourced under Apache-2 license, and boasts a productive and vibrant community.

Goals

Set up SaltStack master and minion. Provision a virtual machine and install a sample package on it.

Installation and Configuration

Installation of SaltStack can be done by following this Documentation. In this tutorial, I am going to show a set-up of a salt-master and two salt-minion nodes with Oracle-Linux version 6.5 installed on them.

Environment

Operating System: Oracle Linux 6.5
Salt Master: salt-master
Salt Minion: minion1.com, minion2.com

[root@salt-master ~]# yum install salt-master

Salt configuration files are stored in /etc/salt, /srv/salt. It’s better to keep verbose logging for troubleshooting purposes.

[root@salt-master ~]# vim /etc/salt/master

The default setting is ‘warning’ change to the following.

log_level: debug
log_level_logfile: debug

Restart salt master for changes to take effect:

[root@salt-master ~]# salt-master -d

Install salt-minion on both the target nodes and configure it’s file to talk with salt-master.

[root@minion1  ~]# yum install salt-minion
[root@minion2  ~]# yum install salt-minion

Install salt-minion on both the target nodes and configure it’s file to talk with salt-master.
Edit the file /etc/salt/minion. Locate and un-comment the line “#master:salt” and replace ‘salt’ with FQDN or IP of salt-master server.

[root@minion1  ~] vim /etc/salt/minion

Restart the salt-minion service:

service salt-minion restart

Do similarly on minion node 2 as well.

[root@minion2  ~] vim /etc/salt/minion

Restart the salt-minion service:

service salt-minion restart

Add minion’s private key to salt-master. -L is used to list keys and -A is for accepting.

[root@salt-master ~]# salt-key -F master
Local Keys:
master.pem:  44:55:66:77:cc:aa:bb:ff:ee:32:7:83:82:81:63:99
master.pub:  22:76:54:ee:83:99:97:98:ed:cd:bb:ff:85:82:66:77
Accepted Keys:
minion1.com:  aa:bb:cc:dd:1d:75:81:ee:ff:92:22:22:11:xx:yy:zz
Unaccepted Keys:
minion2.com:  zz:yy:xx:dd:1d:75:81:ee:ff:92:22:22:11:aa:bb:cc
[root@salt-master ~]# salt-key --finger minion2.com
Unaccepted Keys:
minion2.com:  zz:yy:xx:dd:1d:75:81:ee:ff:92:22:22:11:aa:bb:cc
[root@salt-master ~]# salt-key -L
Accepted Keys:
minion1.com
Denied Keys:
Unaccepted Keys:
minion2.com
Rejected Keys:

Accept keys by running the following command.

[root@salt-master ~]# salt-key -A

The following keys are going to be accepted:

Unaccepted Keys:
minion2.com
Proceed? [n/Y] Y
Key for minion minion2.com accepted.

You can get json output format as well using the following command:

[root@salt-master ~]# salt-key -L --out=json
{
   "minions_rejected": [],
   "minions_denied": [],
   "minions_pre": [],
   "minions": [
       "minion2.com",
       "minion1.com"
   ]
}
[root@salt-master ~]# salt --versions-report

Salt Version:

 Salt: 2015.8.1

Dependency Versions:

Dependency Versions:
        Jinja2: 2.2.1
      M2Crypto: 0.20.2
          Mako: Not Installed
        PyYAML: 3.11
         PyZMQ: 14.5.0
        Python: 2.6.6 (r266:84292, Jul 23 2015, 05:13:40)
          RAET: Not Installed
       Tornado: 4.2.1
           ZMQ: 4.0.5
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 1.4.1
         gitdb: Not Installed
     gitpython: Not Installed
         ioflo: Not Installed
       libnacl: Not Installed
  msgpack-pure: Not Installed
msgpack-python: 0.4.6
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
        pygit2: Not Installed
  python-gnupg: Not Installed
         smmap: Not Installed
       timelib: Not Installed
System Versions:
          dist: oracle 6.5
‘salt’        machine: x86_64
       release: 2.6.32-431.29.2.el6.x86_64
        system: Oracle Linux Server 6.5

Now that you have salt-master and salt-minion,  and they trust each other, you can check  the connections by issuing test.ping command. This will return “True” if they communicate between each other.

[root@salt-master ~]# salt minion2.com test.ping
minion2.com:
   True

Salt command syntax involves the salt command, target, action.

[root@salt-master ~]# salt '*' cmd.run "service httpd restart"

You can run any available command on any connected and authenticated minion.
Now let’s look at the Configuration Management section:
Salt’s configuration files and directives are kept within /srv/salt by default. This is the place where all configuration files you want to copy to target minions reside. Salt will not touch your master’s configurations files, so don’t worry. Salt follows YAML syntax for template files. To enable configuration management functionality,  you need to edit the salt-master file once again. Open /etc/salt/master file and locate the line that refers to file_roots and uncomment it to look like this:

file_roots:
 base:
   - /srv/salt

Depending upon how you installed salt, you may have to create this directory /srv/salt/.
The base configuration lies within /srv/salt directory with filename top.sls. This file provides mappings for other files and can be used be to set base configuration for all servers.
Let us write some simple checks to see whether user name ‘bob’ is present on a target node or not. If it’s not, then create it.
user_bob.sls file resides under /srv/salt/base directory

[root@salt-master base]# salt minion1.com state.sls user_bob  
minion1.com:      
----------  
         ID: user_bob
   Function: user.present       
       Name: bob      
     Result: True  
    Comment: User bob is present and up to date  
    Started: 07:42:31.339705
   Duration: 25.963 ms
    Changes:      
Summary for minion1.com
------------  
Succeeded: 1  
Failed:    0
------------  
Total states run:     1
Total run time:  25.963 ms

Now we will install PCRE (Perl Compatible Regular Expression) package on target node-2 i.e minion2.com. The general method of installing it is via configure-compile-install. Develop a shell script that will compile and install PCRE. The “base” directory contains shell script and a top.sls file with relevant information. With all the .sls files in place and configuration files ready to go, the last step is to tell salt to configure your nodes remotely. state.highstate triggers this synchronization. I always use state.show_highstate before running state.highstate because it tells what will happen on the target nodes.

 [root@salt-master base]# salt 'minion2.com' state.show_highstate
minion2.com:
    ----------
    /root/pcre_binary:
        ----------
        __env__:
            base
        __sls__:
            pcre-8-37
        file:
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              mode:
                  755
            |_
              ----------
              makedirs:
                  True
            - directory
            |_
              ----------
              order:
                  10000
    configure-pcre:
        ----------
        __env__:
            base
        __sls__:
            pcre-8-37
        cmd:
            - cwd:/root/pcre_binary/pcre-8.37
            |_
              ----------
              names:
                  - ./configure
                  - make
                  - make install
            - run
            |_
              ----------
              require:
                  |_
                    ----------
                    cmd:
                        extract-pcre
            |_
              ----------
              order:
                  10003
    extract-pcre:
        ----------
        __env__:
            base
        __sls__:
            pcre-8-37
        cmd:
            |_
              ----------
              cwd:
                  /root/pcre_binary
            |_
              ----------
              names:
                  - tar zxvf pcre-8.37.tar.gz
            - run
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        pcre-8.37
            |_
              ----------
              order:
                  10002
    pcre-8-37:
        ----------
        __env__:
            base
        __sls__:
            pcre-8-37
        file:
            - name:/opt/pcre-8.37.tar.gz
            |_
              ----------
              source:
                  - salt://opt/pcre-8.37.tar.gz
            - managed
            |_
              ----------
              order:
                  10001

Now execute with state.highstate.

[root@salt-master base]# salt 'minion2.com' state.highstate

 

minion2.com:
----------.
        ID: pcre
   Function: file.managed
       Name: /pcre-8.37.tar.gz
     Result: True
    Comment: File /pcre-8.37.tar.gz updated
    Started: 01:02:54.117150
   Duration: 116.626 ms
    Changes:
             ----------
             diff:
                 New file
             mode:
                 0644
----------
         ID: extract-pcre
   Function: cmd.run
       Name: cd /
/bin/tar xvf pcre-8.37.tar.gz
cd pcre-8.37
./configure
make
make install
     Result: True
    Comment: Command "cd /
             /bin/tar xvf pcre-8.37.tar.gz
             cd pcre-8.37
             ./configure
             make
             make install
             " run
    Started: 01:02:54.245660
   Duration: 24757.312 ms
    Changes:
             ----------
            pid:
                 4155
             retcode:
                 0
             stderr:
                libtool: warning: relinking 'libpcreposix.la'
                 libtool: warning: relinking 'libpcrecpp.la'
             stdout:
                 pcre-8.37/
                 pcre-8.37/pcre_scanner.h
< … output truncated ... >
Summary for minion1.com
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  24.874 s

This way we see that PCRE package is installed under /root directory on the target node.

Summary

We just walked through the power of storing configurations as text files, and executing them remotely on target nodes.  Salt does not execute these states sequentially, which means that if failure occurs in one node, it will continue to install on another node, and so on. One can thereby set up a deployment procedure. The whole system is suitable for any network and is capable of automated software distribution and network provisioning.
There are other IT automation tools like Chef/Puppet/Ansible/Fabric which execute similar tasks. Chef and Puppet are a bit complicated with their initial setup and running is what I have already experienced and also heard from a person I respect. It’s up to you to decide which language works best for you and your team. You are going to write initial configuration details until you turn your Infrastructure into a code. Visit the Salt project page for more information, and yes you can post your doubts / queries on mailing-lists or jump over IRC for a quick friendly response.
                 

Written by

Am an Open Source enthusiast, a Linux Sysadmin with close to 8 years of IT experience. Being with Open Source community for more than a decade I decided to become a member as an Individual Supporter of LINUX FOUNDATION. Have been working on Cloud Computing, remote execution and config management IT automation tools for over 3 years. I have contributed to Open Source community projects - Belenix, OpenStack. I do participate in User community meetups happening in and around the city so as to keep up with latest technology and learn from folks.

Related Posts

Albert Qian
— August 28, 2018

Introducing Assessment Cycles

Today, cloud technology platforms and best practices around them move faster than ever, resulting in a paradigm shift for how organizations onboard and train their employees. While assessing employee skills on an annual basis might have sufficed a decade ago, the reality is that organiz...

Read more
  • Cloud Computing
  • Product Feature
  • Skill Profiles
— July 31, 2018

Cloud Skills: Transforming Your Teams with Technology and Data

How building Cloud Academy helped us understand the challenges of transforming large teams, and how data and planning can help with your cloud transformation.When we started Cloud Academy a few years ago, our founding team knew that cloud was going to be a revolution for the IT indu...

Read more
  • Cloud Computing
  • Skill Profiles
— June 26, 2018

Disadvantages of Cloud Computing

If you want to deliver digital services of any kind, you’ll need to compute resources including CPU, memory, storage, and network connectivity. Which resources you choose for your delivery, cloud-based or local, is up to you. But you’ll definitely want to do your homework first.Cloud ...

Read more
  • AWS
  • Azure
  • Cloud Computing
  • Google Cloud
Albert Qian
— May 23, 2018

Announcing Skill Profiles Beta

Now that you’ve decided to invest in the cloud, one of your chief concerns might be maximizing your investment. With little time to align resources with your vision, how do you objectively know the capabilities of your teams?By partnering with hundreds of enterprise organizations, we’...

Read more
  • Cloud Computing
  • Product Feature
  • Skill Profiles
— April 5, 2018

A New Paradigm for Cloud Training is Needed (and Other Insights We Can Democratize)

It’s no secret that cloud, its supporting technologies, and the capabilities it unlocks is disrupting IT. Whether you’re cloud-first, multi-cloud, or migrating workload by workload, every step up the ever-changing cloud capability curve depends on your people, your technology, and your ...

Read more
  • Cloud Computing
— March 29, 2018

What is Chaos Engineering? Failure Becomes Reliability

In the IT world, failure is inevitable. A server might go down, an app may fail, etc. Does your team know what to do during a major outage? Do you know what instances may cause a larger systems failure? Chaos engineering, or chaos as a service, will help you fail responsibly.It almost...

Read more
  • Cloud Computing
  • DevOps
— November 22, 2017

AWS re:Invent 2017: Themes and Tools Shaping Cloud Computing in 2018

As the sixth annual re:Invent approaches, it’s a good time to look back at how the industry has progressed over the past year. How have last year’s trends held up, and what new trends are on the horizon? Where is AWS investing with its products and services? How are enterprises respondi...

Read more
  • AWS
  • Cloud Adoption
  • Cloud Computing
  • reInvent17
— October 27, 2017

Cloud Academy at Cloud Expo Santa Clara, Oct 31 – Nov 2

71% of IT decision-makers believe that a lack of cloud expertise in their organizations has resulted in lost revenue.1  That’s why building a culture of cloud—and the common language and skills to support cloud-first—is so important for companies who want to stay ahead of the transfor...

Read more
  • Cloud Computing
  • Events
— October 24, 2017

Product News: Announcing Cloud Academy Exams, Improved Filtering & Navigation, and More

At Cloud Academy, we’re obsessed with creating value for the organizations who trust us as the single source for the learning, practice, and collaboration that enables a culture of cloud.Today, we’re excited to announce the general availability of several new features in our Content L...

Read more
  • Cloud Computing
— August 29, 2017

On 'the public understanding of encryption' Tweet by Paul Johnston

Some of the questions by journalists about encryption prove they don't get it. Politicians don't seem to get it either (most of them). In fact, outside technology, there are some ridiculous notions of what encryption means. Over and over again, the same rubbish around encrypti...

Read more
  • Cloud Computing
— July 13, 2017

Our Hands-on Labs have a new look

Building new hands-on labs and improving our existing labs is a major focus of Cloud Academy for 2017 and beyond. If you search "types of adult learning," you will get approximately 16.9 gazillion hits. Many will boast about how they meet the needs of a certain type of learner (up to 70...

Read more
  • Cloud Computing
  • hands-on labs
— July 11, 2017

New infographic: Cloud computing in 2017

With 83% of businesses ranking cloud skills as critical for digital transformation in 2017, it’s great news for anyone with cloud architecting experience, and for those considering a career in cloud computing. In our new infographic, we compiled some of the latest industry research to l...

Read more
  • Cloud Computing