Build Agent - Scaling Out the Build Infrastructure
Build Agent - Scaling Out the Build Infrastructure
1h 12m

This course introduces you to Jenkins, a popular open-source tool used to perform Continuous Integration and Continuous Delivery.

We spend time reviewing the key Jenkins features and associated terminology. We then explore configuring Jenkins to perform automated builds using the Jenkins web administration console in hands-on demonstrations, familiarising you with Jenkins and how to administer it. We’ll demonstrate features such as:

  • Creating and configuring pipelines using a Jenkinsfile
  • Configuring Jenkins pipelines using the Blue Ocean interface
  • Defining build execution environments using docker containers
  • Setting up and scaling out Jenkins with multiple build agents and executors using SSH
  • Setting up build pipelines for Java-based projects using Gradle

Learning Objectives

What you'll learn:

  • How to scale out Jenkins using Master and Build Agent setups using SSH
  • The benefits of codifying pipeline build instructions using a Jenkinsfile
  • How to leverage Docker containers within a Jenkins pipeline to provide additional build isolation and flexibility
  • How to install and use more modern pipeline centric BlueOcean user interface
  • How to integrate and leverage third-party build tools like Gradle, Maven, Yarn, Webpack, and many more within a Jenkins pipeline


This training course provides many hands-on demonstrations where you will observe first-hand how to use Jenkins to build and release different types of software projects, for example:

  • Building a front-end application which has been developed using the React Javascript framework, using technologies such as Webpack and Yarn
  • Building a back-end application developed using Java, Gradle, and Docker, requiring Jenkins to compile the source code, packaging it into a WebArchive file, and then finally releasing it into a Tomcat-based Docker image complete with Splunk-based instrumentation for logging and monitoring


  • A basic understanding of CICD, or Continuous Integration and Continuous Delivery
  • A basic understanding of software development and the software development life cycle
  • A basic understanding of version control and associated workflows

Intended Audience

  • Software Build and Release Engineers
  • Software Developers
  • DevOps Practitioners


The following GitHub repo contains sample Jenkins configurations used within the provided demonstrations:

Supporting Documentation

The following supporting Jenkins documentation is available online:


- [Instructor] Okay, welcome back! In this demonstration, we're going to do something different. 

What we're going to do now is we're going to scale out our build infrastructure. So at the moment, we've got a single Jenkins master server and all of our builds have been occurring on that particular instance. What we want to do now is add a build agent into the environment. Before we do the configuration, let's consider some of the reasons why you'd want to add in a new build agent. Firstly, it may be a matter of scaling out the overall build environment. So you might have a large number of build jobs all occurring concurrently on the same master instance. 

By adding in additional build agents, you can scale out and spread the build jobs across them. Secondly, you might want to specialize your build agent setup so that, for instance, you have a build agent that runs Windows and therefore it can build .net jobs. Or you might have a build agent that needs to run Linux and can do Linux compilation for kernel builds. And thirdly, it's just good practice to actually move the build jobs off the master instance. This allows you to maintain the integrity of the master instance and not change it by adding in build tools. It also prevents it from being compromised if your users who are using Jenkins can actually add in tools. They could either intentionally or unintentionally break the security of the master instance. Okay, we'll now swap over into the AWS console. Here we can see the Jenkins demo EC2 instance. So this is the Jenkins master node. So we'll launch a new instance. Again, we'll select Ubuntu 18.04 for the operating system. 

We'll configure it as a t3.small instance. And we'll deploy it into the same network as the Jenkins master node. We'll auto-assign a public IP address to allow us to SSH in and set up the tools for the build agent. In a production environment, you wouldn't do this, you would have the build agent reside just on a private network without the IP address but for demonstration purposes this will allow us to move along quickly. Okay, we'll leave the rest of the default settings as-is. Under storage, we'll increase the default root EBS volume to 20 gig. Click on Tags, we'll give it a name. We'll call it JenkinsDemoAgent1. Click on Security Group. We'll create a new security group. We'll call it the JenkinsDemoAgent security group. For now, we'll remove the current SSH rule. We'll add another rule in. It'll be SSH, port 22, it'll be from the private IP address of the Jenkins master node. We'll explain it, this is the Jenkins master node private IP address. We'll click Review and Launch. 

Okay, so we can see our JenkinsDemoAgent1 server is now up and running, as indicated by the running state. We'll take a copy of the public IP address and we'll jump over into our terminal. Here, we'll now SSH into this instance. And we'll begin the setup and install the required packages to turn this EC2 instance into a Jenkins build agent. So the first thing we'll do is we'll do sudo apt-get update to update the packages. Next, we need to install a JDK. In this case, we'll install OpenJDK version 8. So we run sudo apt-get install openjdk-8-jdk, Enter. Yes. Okay. We then need to install the OpenSSH server. So communication is performed between the Jenkins master node and any build agent using SSH. So we'll install this. 

We then need to set up a Jenkins user, and the path for the Jenkins user is typically /var/lib/jenkins. Next we'll create a .ssh directory within the Jenkins user home path. We'll now cd into this directory. And we'll use the SSH keygen to generate our public and private key pair. Enter for no passphrase. Okay that's been generated. If we do a directory listing, it's the public portion and the private portion. We'll now cat out the contents of the public portion into the authorized keys file. If we do a directory listing and we now look at authorized keys. That contains our public key. And finally we'll update the owner of the Jenkins home directory to be the Jenkins user that we previously created. So now if we navigate back and we do an ls-la, we can see that the Jenkins user now owns the .ssh directory. And if we navigate back into it and again, do an ls-la, all of the files belong to the Jenkins user. Okay. So we will now cat out the contents of the private key. To do so, we'll upgrade to root. And again, echo out the contents. So we'll need this when we jump back into the Jenkins web administration console. So we'll copy this. And now we'll jump back into the browser, and we're back within Jenkins. We'll click on Credentials. 

Hover over Global, and click Add Credentials. Under Kind, we'll change this to SSH Username with private key. For User, we specify the Jenkins user. We'll select Private Key, we'll paste in the private key that we just copied. We'll leave the Passphrase empty. The ID will be Jenkins. And we can give it a description, Jenkins SSH for Build Agent1. We'll click OK. So we've configured our credential. We'll now go back to Jenkins and we'll click on Manage Jenkins. Scroll down to Manage Nodes. 

We'll click on New Node. We'll give it a node name, this case we'll call it agent1. We'll specify that it's a permanent agent and we'll click OK. Give it a description, Build Agent1. Number of executors, we'll increase this to four. The remote directory needs to be the same directory that we set up on our build agent. Paste it here. Labels, we'll call this agent1. The launch method will be via SSH. The host will be the private IP address. So we copy the private IP address of our build agent, paste it there. 

Under Credentials, we select the Jenkins credential that we just previously set up. And then under the Host Key Verification Strategy, we'll select Manually Trusted Key Verification Strategy. And then we enable the manual verification on the initial connection. Okay, with all of those settings in place, we'll save. And now we have our new agent node configured, but it hasn't yet done a connection. So we'll select it, and then under the agent1 page, we'll click the Launch Agent button. So the first time we attempt a connection, you'll see that we get an SSH warning because we haven't yet trusted the SSH host key. So we go over to the Trust SSH Host Key option, we'll select it. And here, we've got the fingerprint of our build agent host. 

So we'll click the Yes button to trust it and then we launch the agent again. And this time, the authentication was successful and the agent has successfully connected. If we go back to the list, we can now see that our agent is indeed online. And we can also see under the Build Executor Status that there are four idle executors on agent1. So this is a excellent result. We've configured a brand new build agent, we've successfully authenticated to it from the master node, using SSH. And we have four idle executors ready and waiting for build jobs. 

At the moment, we have zero bytes for the swap space but we'll come back and we'll fix this in the next demonstration. Let's jump back to Jenkins and we'll create a new build job. It'll be a pipeline job, we'll call it BuildJob8. Click OK. We'll select the Pipeline option to jump down into pipeline section. So what we'll paste in here is a pre-configured scripted pipeline. And the key point here is that we're instructing the build job to run on agent1, which is the new build agent that we've just configured. 

So we'll click Apply. We'll click Save. And we'll click Build Now. Now if everything goes to plan, we'll schedule our build job and we'll run it. If we navigate into it, okay the build job has just completed as per our standard "Finished, success" message. If we scroll to the top, the key thing that I want to highlight here is that this build job, BuildJob8, has run on agent1. If we go back to the Project view, we can see that our artifact has been captured, our three stages completed successfully, as per the green boxes. 

And if we jump back into our build agent terminal, if we navigate back one directory, and this time we navigate into the workspace directory, and we do a directory listing, indeed we can see that BuildJob8 definitely occurred on this new build agent. So this is a great result and it demonstrates how to add in and scale out your build infrastructure by adding new build agents. 

Finally, let's go back to our Jenkins setup. And on BuildJob8, we'll just summarize the configuration. Heading down into Pipeline. So again, we instructed this build job to be a scripted pipeline and the key take-away here is that we've instructed the build job to run on agent1. If you go back to Jenkins, go back to Manage Jenkins, we can see that we have our master agent and our build agent1. And it has four available executors on it.

About the Author
Learning Paths

Jeremy is a Content Lead Architect and DevOps SME here at Cloud Academy where he specializes in developing DevOps technical training documentation.

He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 25+ years. In recent times, Jeremy has been focused on DevOps, Cloud (AWS, Azure, GCP), Security, Kubernetes, and Machine Learning.

Jeremy holds professional certifications for AWS, Azure, GCP, Terraform, Kubernetes (CKA, CKAD, CKS).