Applying the Stealth Intrusion Detection system to server security on AWS
I hope it won’t come as a surprise to anyone when I say that maintaining effective server security is an ongoing and uphill struggle. Whether your production servers live in a local data center or in the cloud, no matter how many layers of protection you build, you can never be sure you’ve closed all the doors.
I’m going to describe configuring and deploying an intrusion detection system (IDS) to enhance production server security on AWS infrastructure. But we’ll first need to understand what an IDS is and what it can bring to the table, why AWS’s own formidable security tools might not be enough, and how the Stealth system in particular actually works.
What is an IDS?
One of a sysadmin’s greatest nightmares involves unauthorized intruders somehow gaining access to a sensitive system. Perhaps the passwords weren’t strong enough, or maybe authentication keys were sent over insecure channels. But, either way, the genie’s out of the bottle and you’re stuck pulling another all-nighter trying to stuff it back in. At this point, all the firewalls and password vaults in the world won’t do a thing for you. The bad guys are inside.
An IDS can regularly monitor your system looking for tell-tale signs of unauthorized activity and, when it does, alert pre-defined administrators. Frank B. Brokken’s Linux-based Stealth package works by creating a record of the healthy state of your key system files, and then regularly running checks to compare the current states of those files against their original, healthy condition. Stealth, as its name suggests, is a much more effective server security tool because it does its work in a way that’s virtually invisible to the client system, and is launched from a special server that itself, is going to be nearly impossible to access.
What’s wrong with AWS server security?
Nothing. AWS does everything you could ask of them. They protect access to the physical servers hosting your applications and they offer you flexible and reliable Security Groups and ACLs so you can completely manage all network traffic heading to or from your resources. And access can be further defined by carefully managing users, groups, and roles using AWS’s Identity and Access Management (IAM).
But they can’t be expected to prevent certain kinds of intrusion attack. So that aspect of server security is going to be your headache. And that’s where Stealth comes in.
Building server security through Stealth
The Stealth package wasn’t built with AWS in mind and I’m not sure it’s ever been used that way. (Of course, if it was, the very last thing people would want to do is advertise the fact.) But besides the fact that it would be nice to have internal integration with AWS’s SNS service to simplify alerts, I can’t think of anything that could be improved.
The first step is to create a Stealth server.
For clarity’s sake, I’ll refer to the VM running Stealth to monitor our production server as the “Stealth server,” and the production server itself as the “Stealth client” – even if describing a “server” as a “client” is a bit counter-intuitive.
Since this VM will require very little hardware, using a t2.micro will give us more than enough power. Selecting an Ubuntu AMI is probably safest. While we’re setting things up, we’ll obviously need SSH access to the machine, but once things are going, we’ll shut that down by deleting our security group’s SSH Rule. I’d also suggest that you launch your Stealth server into its own VPC – and certainly one with no connection to the clients it will be monitoring.
Once you’ve logged in to your server, install the Stealth, Postfix, and Mailutils packages. The latter two packages will handle outgoing email alerts.
sudo apt-get update sudo apt-get install stealth postfix mailutils
During the Postfix part of the installation, you’ll be asked to select a mail server configuration type. You can go with the default “Internet Site” value:
You should also be able to accept the default value for “system mail name.”
You should create a new system user called, say, stealth-user.
sudo adduser stealth-user
Switch users to open a shell session as stealth-user and generate a keypair.
su stealth-user ssh-keygen -t rsa
You’re going to copy this keypair and save it to the root of your production server (Stealth client). Move to the .ssh directory and display the public keyfile using
cd ~/ssh cat id_rsa.pub
…and carefully copy the contents of id_rsa.pub – and nothing but those contents – to your clipboard or to a file on your local computer. Now, let’s open an SSH session on the Stealth client and start a shell as root. It has to be root specifically because Stealth will need to read files and file attributes that require administrator permissions. Since you won’t be there to enter an admin password, it will have to be automated. This does carry some risk, but server security always requires a bit of a tricky balancing act.
NOTE: NEVER trust command line suggestions you read on the Internet – and especially not suggestions using root – unless you are absolutely sure you understand what they will do. You have been warned.
sudo su
Move to the root directory and list all its contents (even hidden files and directories):
cd /root && ls -a
If you don’t see a directory called .ssh, then create one:
mkdir .ssh
Now, use your favorite text editor to create (or add to) a public key file in your Stealth client’s .ssh directory:
nano /root/.ssh/authorized_keys
…paste the key contents you copied from your Stealth server into this file and save it. You will probably see a warning from our good friends at Amazon that you should not use root to login. This is normally good advice, but not for this particular scenario.
Now, you should test the key file by trying to open an SSH session on the Stealth client’s root account from the Stealth server:
ssh root@client_server_IP_address
You should get access without having to enter a password.
You can now close any open Stealth client sessions, as we shouldn’t need to visit again. Don’t forget to edit the server client’s Security Group to open up SSH access to the IP address of your Stealth server.
With the connection set up and working, we’ll turn our attention to the Stealth policy script that will actually do all the work. We will go through the policy one section at a time, but here’s the full file the way it might end up appearing (using a fictitious Stealth client IP address):
DEFINE SSHCMD /usr/bin/ssh root@55.3.212.44 -T -q /bin/bash --noprofile
DEFINE EXECSHA1 -xdev -perm /111 -type f -exec /usr/bin/sha1sum {} \;
USE BASE /home/stealth-user/stealth/55.3.212.44
USE EMAIL some_name@gmail.com
USE MAILER /usr/bin/mail
USE MAILARGS -s "55.3.212.44 STEALTH report"
USE SSH ${SSHCMD}
USE DD /bin/dd
USE DIFF /usr/bin/diff
USE PIDFILE /var/run/stealth-
USE REPORT report
USE SH /bin/sh
GET /usr/bin/sha1sum /home/stealth-user/tmp
LABEL \nCheck the client sha1sum program
LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /home/stealth-user/tmp/sha1sum
LABEL \ncheck the client /usr/bin/find program
CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
LABEL \nsuid/sgid/executable files uid or gid root on the / partition
CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}
LABEL \nconfiguration files under /etc
CHECK LOG = remote/etcfiles /usr/bin/find /etc -type f -not -perm /6111 -not -regex "/etc/\(adjtime\|mtab\)" -exec /usr/bin/sha1sum {} \;
So let’s digest this in bite-sized chunks:
DEFINE SSHCMD /usr/bin/ssh root@55.3.212.44 -T -q /bin/bash --noprofile
DEFINE EXECSHA1 -xdev -perm /111 -type f -exec /usr/bin/sha1sum {} \;
The variables SSHCMD (which defines how Stealth will open an SSH connection on the Stealth client) and EXECSHA1 (which identifies the contents of checksums for suid and gid executable files) are set for future use. The SSHCMD value will be called by the USE line:
USE SSH ${SSHCMD}
…to launch an SSH session. And the EXECSHA1 value will be used as part of this CHECK LOG command:
CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}
…which will write the resulting list to the /…/remote directory on the Stealth server machine.
USE BASE /home/stealth-user/stealth/55.3.212.44
The BASE line establishes the location where Stealth will save data files on the local Stealth server. In this case, they’ll be saved to a directory named after our Stealth client’s IP address. By the way: you might have to create the /…/stealth/ directory manually.
mkdir /home/stealth-user/stealth
These three lines:
USE EMAIL some_name@gmail.com USE MAILER /usr/bin/mail USE MAILARGS -s "55.3.212.44 STEALTH report"
…define where email alerts will be sent – in this case, to a fictitious address. Stealth will use the mail service we installed earlier and will attach a subject line that incorporates any message you feel will make the email’s contents clear enough at a glance. As I mentioned before, it would be nice if we could trigger AWS’s SNS alerts directly, but that’s a small price to pay for an improved server security!
The next two USE lines define more system parameters. Copying files will be done using the venerable Unix program dd, and file contents will be compared using diff.
USE DD /bin/dd USE DIFF /usr/bin/diff
The remaining lines in our sample policy file are actual commands.
LABEL \nconfiguration files under /etc
CHECK LOG = remote/etcfiles /usr/bin/find /etc -type f -not -perm /6111 -not -regex "/etc/\(adjtime\|mtab\)" -exec /usr/bin/sha1sum {} \;
This CHECK LOG command, for instance, will use the find command to compare all the files in the /etc directory – that meet specific conditions – to the previously logged versions. Should any key file attributes have changed, an email alert will be issued, accompanied by the label: “configuration files under /etc”.
If you’ve updated the:
DEFINE SSHCMD /usr/bin/ssh root@55.3.212.44 -T -q /bin/bash --noprofile
…line to reflect the actual IP address of your Stealth client, then you should be ready to go. From the directory where you’ve saved your policy file (and assuming that you called it “policy”), type:
stealth policy
If everything went as planned, you should receive an email report describing the specific files that have been added.
Create a cron job to automate server security monitoring
Well, as much fun as that was, it’s not yet all that helpful. Do you really want to manually launch Stealth every thirty minutes or so all through the long, cold nights? This sounds like a good use-case for a cron job.
Create a file with an appropriate variation of these contents:
4,34 * * * * stealth-user test -x /usr/bin/stealth && /usr/bin/stealth -s /home/stealth-user/skipfiles -q /home/stealth-user/policy
…and save it to the /etc/cron.d/ directory. Here’s what it will do:
At four and thirty-four minutes past each hour of every day of the week, cron will, as the user “stealth-user”, check for the existence of the Stealth program, and then run Stealth using our policy. But it will also tell it to read a plain-text file you could create (I called mine skipfiles) that can contain the names and locations of any files on our Stealth client that you don’t want Stealth to read. This can be very useful to help reduce false positives – i.e., files that are regularly updated through the normal functioning of your system.
If you’re using your Stealth server to monitor multiple client servers, you can create individual policy files (with unique names) and, if you like, save them all to the same directory. Stealth will take care of the subdirectories for you.
Once you’re comfortable that you’ve got everything up and running the way you like, you should isolate your Stealth server by deleting the SSH rule on the server’s AWS Security Group. Even if it’s likely that you’ll have to go in again to edit your configuration or add a policy for a new server, you’re still better off keeping the VM as closed to the world as possible.
As I said back at the top, when it comes to server security, paranoid just isn’t good enough. Just think about this: if someone were to hack your Stealth server, he’d have access to the root accounts on all your servers!
You can take comfort from the fact that Amazon makes opening up access very easy: all you’ve got to do is edit the Security Group rules from the console. Or even better, using the AWS command line interface. When I deployed Stealth using an old, decommissioned Compaq PC some time ago, updating the configuration meant searching through the office for an unused keyboard and monitor – or just steal one from one of the DevOps guys when he wasn’t looking.
Stealth is a really great piece of software that works beautifully as one part of your larger server security profile. This is, however, server security, which means that you absolutely have to get it right the first time. Make liberal use of the package’s excellent documentation.
I’d love to hear back from you about your successes and challenges…and about creative ways you manage to integrate Stealth into your AWS infrastructure.
