Create a Jenkins CICD Pipeline to Build a Docker Image with Splunk Integration

Integrating Jenkins with Docker provides you with a robust CICD method for building and packaging runnable containers. Building and deploying your docker containers is only half the challenge. Once they are in production, you will want to monitor and assess their operational performance. Splunk can provide you with this capability, through its support for enterprise-grade monitoring, logging, and diagnostics collection.

In this lab, you will launch a Jenkins and Splunk CICD and monitoring environment using Docker containers on a provided EC2 instance. You will then configure a Jenkins build pipeline to build, compile, and package a sample Java servlet web application into a runnable Docker image. The Jenkins build process will create the Docker image using a Dockerfile. The build process integrates Splunk logging into the Docker image, by installing the Splunk Forwarding Agent at build time. At runtime, the docker container will then have the ability to publish runtime logging information into the Splunk service. 

This lab is aimed at DevOps and CICD practitioners, and, in particular, build and release engineers interested in managing and configuring Jenkins together with both Docker and Splunk for enterprise-grade monitoring capabilities.

Lab Objectives

Upon completion of this lab, you will be able to:

• Install and configure a Jenkins and Splunk CICD and monitoring environment using Docker containers
• Configure Jenkins with the Gradle plugin to perform the core build and packaging for a sample Java servlet web application
• Configure Jenkins with the Docker plugin for automated docker container build artifact management
• Create and set up a Jenkins build pipeline using a Jenkinsfile stored within a GitHub repo
• Launch a custom-built Tomcat Java servlet web application docker container, complete with Splunk logging integrated using the Splunk Forwarding Agent
• Use the Splunk administration web console to search and report on collected and aggregated runtime data points emitted from custom Tomcat Docker container

Lab Prerequisites

You should:

• Be comfortable with SSH to remotely administer a Linux-based server
• Be comfortable with basic Linux administration

Lab Environment

This lab will start with the following AWS resources being provisioned automatically for you:

• A single EC2 instance, named cicd.platform.instance, which will have a public IP address attached

To achieve the Lab end state, you will be walked through the process of:

• SSHing into the EC2 instance, named cicd.platform.instance
  • Use Docker Compose to launch the following Docker containers:
    
    • Jenkins
    • Splunk
    • Socat
• Using a browser, administer and configure Jenkins - installing the required plugins. Connectivity to Jenkins will be done via the cicd.platform.instance Public IP address 
• Using a browser, administer and configure Splunk. Connectivity to Splunk will be done via the cicd.platform.instance Public IP address 
• Create a Jenkins build pipeline and configure it to build a sample Java servlet web application hosted on GitHub, with the build artifact being a Docker image hosted on the cicd.platform.instance
• Execute the Jenkins build pipeline and confirm that it has completed successfully, registering the resulting Docker image hosted on the cicd.platform.instance
• Spin up the newly-created Docker image containing the Tomcat-hosted sample Java servlet web application
• Generate sample user activity within the sample Java servlet web application - this will result in usage traffic being sent to the Splunk service
• Use the Splunk administration console to search and report on the collected usage patterns

Updates

August 31st, 2023 - Updated instructions and screenshots to reflect latest Jenkins UI

February 9th, 2023 - Updated lab to output the URL needed to access Jenkins and Splunk

October 3rd, 2022 - Updated the instructions and screenshots to reflect the latest UI and updated lab to use EC2 Instance Connect

January 3rd, 2022 - Updated screenshots and instructions to reflect the latest Jenkins UI