SystemTap: Getting Started with Dynamic EC2 Instance Diagnostics

Just because you can’t see or hear your Amazon EC2 instances the way you can in your local data center, doesn’t mean you shouldn’t be concerned about them. Think of SystemTap as a fabulous window into your VM.

Have you ever wondered what’s going on deep inside your Linux instance? Why some processes run so slowly? Why your disk is thrashing around so wildly? Have you tried to write your own system’s tools and wish that you were able to export or access certain systems’ internal variables or information? Have you tried to write your own Linux kernel module but felt that the learning curve was way too steep?

Let me introduce you to SystemTap.

(The beautiful SystemTap logo was designed by Andy Fitzsimon – who sat next to me when we were both working at Red Hat in Singapore.)
SystemTap is an open source software package to help you analyze or troubleshoot a running Linux instance. It provides a dynamic instrumentation framework that lets you probe a system to gather information and a scripting language translator that interprets scripts written in a C or Perl-like language into C code, that could be compiled into Linux kernel modules. These modules will then be loaded into the system to generate outcome specified by your script.

But enough introduction. Let’s get started already!

Installing SystemTap on your AWS EC2 instance

This article assumes that you know how to create and launch an EC2 instance. If not, I recommend taking Cloud Academy’s Compute Fundamentals for AWS Course as part of the AWS Fundamentals Learning Path.

Connect to the instance using the OpenSSH SSH client.

$ ssh -i "cloudacademy-keypair.pem" ec2-user@[redacted]

Note the Linux kernel version we are using. In this case, it’s version 3.10.0, and the release number is 229.el7.

$ uname -a
Linux ip-[redacted].ap-southeast-1.compute.internal 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

To prepare the Linux instance for kernel-space probing with SystemTap, we will need to install some RPM packages, including kernel-debuginfo and systemtap. We need to ensure that all the kernel packages we will install on the instance have the same version and of the same architecture.

To install a debuginfo package, we’ll enable the “Red Hat Enterprise Linux Server 7 Debug (Debug RPMs)” yum repository that, by default, is disabled.

$ sudo su -
# yum repolist disabled
Loaded plugins: amazon-id, rhui-lb
repo id                                                                repo name
rhui-REGION-rhel-server-debug-extras/7Server/x86_64                    Red Hat Enterprise Linux Server 7 Extra Debug (Debug RPMs)
rhui-REGION-rhel-server-debug-optional/7Server/x86_64                  Red Hat Enterprise Linux Server 7 Optional Debug (Debug RPMs)
rhui-REGION-rhel-server-debug-rh-common/7Server/x86_64                 Red Hat Enterprise Linux Server 7 RH Common Debug (Debug RPMs)
rhui-REGION-rhel-server-debug-rhscl/7Server/x86_64                     Red Hat Enterprise Linux Server 7 RHSCL Debug (Debug RPMs)
rhui-REGION-rhel-server-debug-supplementary/7Server/x86_64             Red Hat Enterprise Linux Server 7 Supplementary Debug (Debug RPMs)
rhui-REGION-rhel-server-extras/7Server/x86_64                          Red Hat Enterprise Linux Server 7 Extra(RPMs)
rhui-REGION-rhel-server-optional/7Server/x86_64                        Red Hat Enterprise Linux Server 7 Optional (RPMs)
rhui-REGION-rhel-server-releases-debug/7Server/x86_64                  Red Hat Enterprise Linux Server 7 Debug (Debug RPMs)
rhui-REGION-rhel-server-releases-source/7Server/x86_64                 Red Hat Enterprise Linux Server 7 (SRPMs)
rhui-REGION-rhel-server-rhscl/7Server/x86_64                           Red Hat Enterprise Linux Server 7 RHSCL (RPMs)
rhui-REGION-rhel-server-source-extras/7Server/x86_64                   Red Hat Enterprise Linux Server 7 Extra (SRPMs)
rhui-REGION-rhel-server-source-optional/7Server/x86_64                 Red Hat Enterprise Linux Server 7 Optional (SRPMs)
rhui-REGION-rhel-server-source-rh-common/7Server/x86_64                Red Hat Enterprise Linux Server 7 RH Common (SRPMs)
rhui-REGION-rhel-server-source-rhscl/7Server/x86_64                    Red Hat Enterprise Linux Server 7 RHSCL (SRPMs)
rhui-REGION-rhel-server-source-supplementary/7Server/x86_64            Red Hat Enterprise Linux Server 7 Supplementary (SRPMs)
rhui-REGION-rhel-server-supplementary/7Server/x86_64                   Red Hat Enterprise Linux Server 7 Supplementary (RPMs)
repolist: 0

Now we’ll use yum-config-manager to enable the repo.

# yum-config-manager --enable "Red Hat Enterprise Linux Server 7 Debug (Debug RPMs)"

Now you’ll be able to search for the debuginfo packages.

# yum search kernel | grep ^kernel-debuginfo
kernel-debuginfo.x86_64 : Debug information for package kernel
kernel-debuginfo-common-x86_64.x86_64 : Kernel source files used by

To ensure that we have the same Linux kernel version, we need to install the kernel and kernel-devel packages along with the kernel-debuginfo packages. If yum returns a different kernel version from the kernel-debuginfo packages, you may need to specify the version manually.

# yum -y install kernel-debuginfo kernel kernel-devel

Depending on your Internet connectivity, it may take some time to download these packages. To give you an idea how large some of these packages are, the kernel-debuginfo-common package is 41MB and the kernel-debuginfo package is 259MB. Fortunately, unless the kernel version and release number change, you only need to install these packages once.

You will need to reboot your machine to apply the new kernel.

# init 6

Once we’re running again, let’s reconnect to the instance.

$ ssh -i "cloudacademy-keypair.pem" ec2-user@[redacted]

Verify that we are using the right kernel version. Notice that the release number is now 229.11.1.el7. Each time Red Hat spins a new kernel, it will update the release number of the package.

$ uname -a
Linux ip-[redacted].ap-southeast-1.compute.internal 3.10.0-229.11.1.el7.x86_64 #1 SMP Wed Jul 22 12:06:11 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

We can use the rpm command to make sure that the version we booted is the same as the debuginfo packages we have installed. This is important.

$ rpm -qa | grep ^kernel
kernel-tools-3.10.0-229.el7.x86_64
kernel-3.10.0-229.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-229.11.1.el7.x86_64
kernel-3.10.0-229.11.1.el7.x86_64
kernel-devel-3.10.0-229.11.1.el7.x86_64
kernel-tools-libs-3.10.0-229.el7.x86_64
kernel-debuginfo-3.10.0-229.11.1.el7.x86_64

Next, use yum to list all the systemtap-related packages.

# yum list systemtap*
Loaded plugins: amazon-id, rhui-lb
Available Packages
systemtap.x86_64                                              2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-client.x86_64                                       2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-debuginfo.i686                                      2.6-10.el7_1                                    rhui-REGION-rhel-server-releases-debug
systemtap-debuginfo.x86_64                                    2.6-10.el7_1                                    rhui-REGION-rhel-server-releases-debug
systemtap-devel.x86_64                                        2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-initscript.x86_64                                   2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-runtime.x86_64                                      2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-sdt-devel.i686                                      2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-sdt-devel.x86_64                                    2.6-10.el7_1                                    rhui-REGION-rhel-server-releases
systemtap-server.x86_64                                       2.6-10.el7_1                                    rhui-REGION-rhel-server-releases

Then install the main systemtap package, and yum will take care of its dependencies.

# yum -y install systemtap

Once all the systemtap-related packages are installed, exit from the root shell to return to the normal user shell.

# logout

Running our first SystemTap script

We have installed all the necessary RPM packages and we are almost ready to run our very first “Hello World” script!

SystemTap uses a simple permission model to grant users’ permissions to use the tool to prevent people from misusing or abusing the tool. Unless you’re a member of the stapusr and stapdev groups, you will not be allowed to build and run SystemTap scripts. Since we are going to write and run our own script, and we hopefully know exactly what we are doing, we will add ourselves to the stapusr and stapdev groups.

$ sudo usermod -G stapusr,stapdev ec2-user
$ grep ^stap /etc/group
stapusr:x:156:ec2-user
stapsys:x:157:
stapdev:x:158:ec2-user

You will need to logout and re-login to the instance to reflect that you are now members of these two groups.

$ logout
$ ssh -i "cloudacademy-keypair.pem" ec2-user@[redacted]

Type id to see if you are in stapusr and stapdev groups.

$ id
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user),156(stapusr),158(stapdev) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

If you are, let’s party on! stap is the SystemTap tool you’ll use to run the script.
There are two ways to write a script. You can either write it in a .stp extension file, or simply specify the script on the command-line with the -e option.

We will opt for the latter since we will use a one-liner script to test if the installation works as expected.

$ stap -e 'probe begin { log("hello world") exit () }'
hello world

Voilà! We have successfully run our first SystemTap script!

In our next article, we will explore some interesting scripts that come with SystemTap.

Avatar

Written by

Eugene Teo

Eugene Teo is a director of security at a US-based technology company. He is interested in applying machine learning techniques to solve problems in the security domain.


Related Posts

Valery Calderón Briz
Valery Calderón Briz
— October 22, 2019

How to Go Serverless Like a Pro

So, no servers? Yeah, I checked and there are definitely no servers. Well...the cloud service providers do need servers to host and run the code, but we don’t have to worry about it. Which operating system to use, how and when to run the instances, the scalability, and all the arch...

Read more
  • AWS
  • Lambda
  • Serverless
Avatar
Stuart Scott
— October 16, 2019

AWS Security: Bastion Host, NAT instances and VPC Peering

Effective security requires close control over your data and resources. Bastion hosts, NAT instances, and VPC peering can help you secure your AWS infrastructure. Welcome to part four of my AWS Security overview. In part three, we looked at network security at the subnet level. This ti...

Read more
  • AWS
Avatar
Sudhi Seshachala
— October 9, 2019

Top 13 Amazon Virtual Private Cloud (VPC) Best Practices

Amazon Virtual Private Cloud (VPC) brings a host of advantages to the table, including static private IP addresses, Elastic Network Interfaces, secure bastion host setup, DHCP options, Advanced Network Access Control, predictable internal IP ranges, VPN connectivity, movement of interna...

Read more
  • AWS
  • best practices
  • VPC
Avatar
Stuart Scott
— October 2, 2019

Big Changes to the AWS Certification Exams

With AWS re:Invent 2019 just around the corner, we can expect some early announcements to trickle through with upcoming features and services. However, AWS has just announced some big changes to their certification exams. So what’s changing and what’s new? There is a brand NEW ...

Read more
  • AWS
  • Certifications
Alisha Reyes
Alisha Reyes
— October 1, 2019

New on Cloud Academy: ITIL® 4, Microsoft 365 Tenant, Jenkins, TOGAF® 9.1, and more

At Cloud Academy, we're always striving to make improvements to our training platform. Based on your feedback, we released some new features to help make it easier for you to continue studying. These new features allow you to: Remove content from “Continue Studying” section Disc...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • ITIL® 4
  • Jenkins
  • Microsoft 365 Tenant
  • New content
  • Product Feature
  • Python programming
  • TOGAF® 9.1
Avatar
Stuart Scott
— September 27, 2019

AWS Security Groups: Instance Level Security

Instance security requires that you fully understand AWS security groups, along with patching responsibility, key pairs, and various tenancy options. As a precursor to this post, you should have a thorough understanding of the AWS Shared Responsibility Model before moving onto discussi...

Read more
  • AWS
  • instance security
  • Security
  • security groups
Avatar
Jeremy Cook
— September 17, 2019

Cloud Migration Risks & Benefits

If you’re like most businesses, you already have at least one workload running in the cloud. However, that doesn’t mean that cloud migration is right for everyone. While cloud environments are generally scalable, reliable, and highly available, those won’t be the only considerations dri...

Read more
  • AWS
  • Azure
  • Cloud Migration
Joe Nemer
Joe Nemer
— September 12, 2019

Real-Time Application Monitoring with Amazon Kinesis

Amazon Kinesis is a real-time data streaming service that makes it easy to collect, process, and analyze data so you can get quick insights and react as fast as possible to new information.  With Amazon Kinesis you can ingest real-time data such as application logs, website clickstre...

Read more
  • amazon kinesis
  • AWS
  • Stream Analytics
  • Streaming data
Joe Nemer
Joe Nemer
— September 6, 2019

Google Cloud Functions vs. AWS Lambda: The Fight for Serverless Cloud Domination

Serverless computing: What is it and why is it important? A quick background The general concept of serverless computing was introduced to the market by Amazon Web Services (AWS) around 2014 with the release of AWS Lambda. As we know, cloud computing has made it possible for users to ...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
Joe Nemer
Joe Nemer
— September 3, 2019

Google Vision vs. Amazon Rekognition: A Vendor-Neutral Comparison

Google Cloud Vision and Amazon Rekognition offer a broad spectrum of solutions, some of which are comparable in terms of functional details, quality, performance, and costs. This post is a fact-based comparative analysis on Google Vision vs. Amazon Rekognition and will focus on the tech...

Read more
  • Amazon Rekognition
  • AWS
  • Google Cloud Platform
  • Google Vision
Alisha Reyes
Alisha Reyes
— August 30, 2019

New on Cloud Academy: CISSP, AWS, Azure, & DevOps Labs, Python for Beginners, and more…

As Hurricane Dorian intensifies, it looks like Floridians across the entire state might have to hunker down for another big one. If you've gone through a hurricane, you know that preparing for one is no joke. You'll need a survival kit with plenty of water, flashlights, batteries, and n...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • New content
  • Product Feature
  • Python programming
Joe Nemer
Joe Nemer
— August 27, 2019

Amazon Route 53: Why You Should Consider DNS Migration

What Amazon Route 53 brings to the DNS table Amazon Route 53 is a highly available and scalable Domain Name System (DNS) service offered by AWS. It is named by the TCP or UDP port 53, which is where DNS server requests are addressed. Like any DNS service, Route 53 handles domain regist...

Read more
  • Amazon
  • AWS
  • Cloud Migration
  • DNS
  • Route 53