The course is part of this learning path
This course introduces you to Jenkins, a popular open-source tool used to perform Continuous Integration and Continuous Delivery.
We spend time early on reviewing the key Jenkins features and associated terminology. We then take you through a deep dive in configuring Jenkins to perform automated builds using the Jenkins web administration console in hands-on demonstrations, ensuring that you become familiarised with Jenkins and how to administer it. We’ll demonstrate features such as:
- Creating and configuring pipelines using a Jenkinsfile
- Configuring Jenkins pipelines using the Blue Ocean interface
- Defining build execution environments using docker containers
- Setting up and scaling out Jenkins with multiple build agents and executors using SSH
- Setting up build pipelines for Java-based projects using Gradle
What you'll learn:
- How to scale out Jenkins using Master and Build Agent setups using SSH
- The benefits of codifying pipeline build instructions using a Jenkinsfile
- How to leverage Docker containers within a Jenkins pipeline to provide additional build isolation and flexibility
- How to install and use the newer more modern pipeline centric BlueOcean user interface
- How to integrate and leverage 3rd party build tools like Gradle, Maven, Yarn, Webpack, and many more within a Jenkins pipeline
This training course provides many hands-on demonstrations where you will observe first hand how to use Jenkins to build and release different types of software projects, for example:
- Building a back end application developed using Java, Gradle, and Docker, requiring Jenkins to compile the source code, packaging it into a WebArchive file, and then finally releasing it into a Tomcat based Docker image complete with Splunk based instrumentation for logging and monitoring
- A basic understanding of CICD, or Continuous Integration and Continuous Delivery
- A basic understanding of software development and the software development life cycle
- A basic understanding of version control and associated workflows
- Software Build and Release Engineers
- Software Developers
- DevOps Practitioners
The following GitHub repo contains sample Jenkins configurations used within the provided demonstrations:
The following supporting Jenkins documentation is available online:
- [Instructor] Welcome back! In this lecture, we'll provide you with an understanding of how to build, scale-out, and setup a multi node or master agent Jenkins infrastructure environment. We'll discuss the various motives as to why you would consider deploying Jenkins in a multi server setup. Okay, let's begin. For starters, Jenkins can quite easily be installed and operated on a single server setup, assuming that the server in question is adequately spec'd in terms of both RAM and raw disk space. However, as your CICD requirements start to grow, potentially in terms of build job numbers, frequency of builds, complexity of builds, software frameworks and/or operating system build requirements, you will likely find that you will end up running Jenkins in a multi-node configuration.
In a multi-node configuration, a Jenkins master can be configured to distribute build jobs amongst multiple Jenkins build agents. Motivating reason to scale-out and introduce multiple build agents vary per environment. Some of the more common reasons may include Jenkins master server security. This requirement is needed when you don't want to compromise the security of a master server. Building on the master server can be dangerous if unvetted build jobs perform unintentional or intentional manipulation of the master server setup.
Build job security. This requirement is sometimes needed when a build job requires some form of access to sensitive material that you want to protect. Building on a dedicated access-controlled build server can help isolate and maintain security over the sensitive material. Tooling and required software frameworks. For example, you maybe building an iPhone application that has a compilation dependency on Apple's Xcode. Therefore, this build job would be farmed out to a Jenkins agent installed on a macOS server. Or, alternatively, you maybe building a Microsoft .NET Windows application that must be built on a Windows server. Hence, this build job would be farmed out to a Windows-based build agent. A floating master server utilization. Building on the master server can impact overall use if server utilization peaks and performance bottlenecks. Offloading builds onto dedicated agents can ensure that the master server has full utilization for performing administration only tasks. Faster build job times. By having dedicated build agents, build jobs can be optimized to complete as quickly as possible. The count of build agents and/or the build agent hardware specs can be increased and scaled out when more build performance is required.
And finally, geography of builds. You may have a requirement to perform build jobs in different geographic locations, maybe in different office locations or different time zones. In this case, build agents can be created and installed in the required location, with build jobs distributed amongst them. Communication between the master and each of its build agents is performed using TCP/IP-based network connections. The connectivity can be initiated in either direction, but is typically done as best practice from the master. Initiating connections from the master to the build agents has several advantages over the opposing direction, mainly the master maintains scheduling and dispatching control over where and when build jobs take place. A typical setup involves installing an SSH service on the build agent side. The master then initiates an SSH connection to the build agent using a pre-generated public/private keypair for authentication. Having initiated the SSH connection, the master will transfer a JAR file, namely the remoting.jar file, to the build agent server. The remoting.jar file contains a compacted version of Jenkins, containing just the required components to perform a distributed build.
Let's walk through the Jenkins administration requirements to setup a distributed build agent. In this scenario, we'll create a Linux-based Jenkins agent running Ubuntu 18.04. Before configuring the setup within the Jenkins administration console, we would run the following bash script on the Ubuntu server. This script performs the following actions in order. One, updates the available package lists. Two, installs the OpenSSH package to provide an SSH daemon which the Jenkins master server SSH client will connect to. Three, installs a JRE which is required to run the agent JAR remoting file, which is pushed down from the master server when an SSH connection is successfully established. Four, creates a new user named jenkins and sets the home directory for the new user as /var/lib/jenkins. Inside this directory a .ssh directory is created. Five, ssh-keygen is then used to generate the SSH public private keypair used to establish SSH authentication between the master and agent server. Six, the public key is then stored in the authorized_keys file. Seven, and finally, we reset the folder permissions recursively on the /var/lib/jenkins directory, giving exclusive access to the jenkins user. This completes the agent side of the server setup.
Next, we return to Jenkins administration console and set a new SSH username with private key credential, copying in the private key portion copied over from the agent server. We set the username to jenkins. We now have all the credential management in place and we are ready to configure a new build agent node. To do so, we navigate to the Manage Jenkins section and click on the Manage Nodes option. You are then prompted to supply a node name and whether to make it a permanent agent or not. Permanency dictates whether the agent is going to stick around or not. A non-permanent agent may, for example, be spawn as part of an elastic cloud hosted architecture involving auto-scaling, for example. For most requirements, we can simply go with a permanent agent.
Next, we need to set the specifics regarding the new node. In particular, we will specify the number of executors. In this case, we'll raise the number of executors to four. The remote root directory. Here, we set it to be the home directory we allocated to the jenkins user, namely /var/lib/jenkins. A label is specified, which we can use later within our build job and/or build pipeline configuration to dictate where that particular build job or pipeline execution will take place. Launch method. This one is important and here, we've chosen to select the Launch agents via SSH option. With this option selected, we then need to specify the following; the agent host address, the credentials that we previously configured within Jenkins, and we need to pick an appropriate strategy for the host key verification. In this case, we're going with the known hosts file strategy.
This now completes the configuration phase for both the master and build agent nodes. With all the configuration in place, all that is left is to do is activate and launch the agent by clicking the Launch agent button. The Jenkins master will then attempt to form an SSH connection to the new Jenkins agent, presenting and authenticating with the SSH private key that it has been configured with. A successful connection concludes with the message, Agent successfully connected and online. With the new agent online and connected, we are now presented with a view of the stats associated with the agent, including the number of build executors available. In this case, we can see that the new build agent supports four build executors as expected. When troubleshooting connecting and launching new build agents, review the launch console log carefully. Typical problems often encountered when launching agents are inbound SSH connections on port 22 are blocked on the agent side, the SSH daemon has not been started on the agent side, and/or SSH key mismanagement.
Okay, that completes this lecture on how to build, scale-out, and setup a multi-node or master agent Jenkins infrastructure environment. Go ahead and close this lecture and we'll see you shortly in the next one.
Jeremy is the DevOps Content Lead at Cloud Academy where he specializes in developing technical training documentation for DevOps.
He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 20+ years. In recent times, Jeremy has been focused on DevOps, Cloud, Security, and Machine Learning.
Jeremy holds professional certifications for both the AWS and GCP cloud platforms.