Whether you are trying to figure out what keeps making your system crash, whether hackers have found their way inside, or why your PHP code isn't working, logs can contain the answers you're looking for. In this video, we'll learn about some of the more common Linux system logs and how to configure them so that their precious data will be available when you need it. Most Linux logs will live in or below the /var/log directory.

Configuring and managing Linux logs

As you can see, there are log files specific to many system functions like boot, and on this Fedora system, Yum. How log messages are handled is on systems using systemd controlled by the journald service. You can query the contents of the systemd journal using journalctl. If you want to go right to the end of the file, which can be very long, add -e. You can also use journalctl to manage the journal files.

Journalctl --diskusage will tell you how much space the files are currently using. You can edit journal behavior through the /etc/systemd/journald.conf file. System max use and system max file size will, for instance, control how large your system files will be allowed to grow. The logger tool is something that on first glance may seem a bit a silly. Using logger on the command line, you can manually send the text screen to a log file you specified, but since you are the one writing it, why would you need a record? Like anything else on Linux, whatever you do from the command line can also be done as part of a script. So imagine that you've written a script that executes some process. Since you may not be around when a script is ran to find out whether, say, the process exceeded or not, inserting a logger line into the script can create a record that will include not only your text string, but the process ID, user, and a timestamp.

Using -p , you can specify a facility name. Meaning, the name of the target log file, and level name, like alert, crit, and warn. This command will send this message to the printer log as an alert.

These are the valid facility names along with their numeric equivalents. And these are the valid level names.

Since just about everything that happens on a Linux system will generate log entries, you can easily imagine how the logs themselves can grow. I've seen log files approaching terabytes in size. You don't want to lose any important data, but you also don't want your drives to be choked by older logs. The accepted solution is to rotate your log files at set times, with older files being permanently deleted or moved into archive storage. Log rotation is controlled by a file in the etc directory called, appropriately enough, logrotate.conf. In this case, most logs will be rotated every week with expired files being kept for three weeks after that. As we can see, once a log is rotated out, meaning it is renamed with a suffix based on the current date, a new entry file will be created to receive new messages in its place. Files can also be compressed to save space.

The truth is that not all logs are controlled from logrotate.conf. Note the reference to the /etc/logrotate.d directory. That's where the RPM package manager drops configuration files for the logs associated with its packages. The include statement pointing to this directory in logrotate.conf will ensure that these config files are read.

Log files on systemd

Distributions not yet using systemd will often use syslog or rsyslog to control logging. Older Ubuntu systems, for instance, will have a file named etc/rsys slash log.conf to control many logging parameters. That file also points to configuration files in the etc/rsyslog.d directory that control log behavior. Syslog -ng is another logging control system that adds functionality like content-based filtering and TCP network transfers. On some systems, klogd will control kernel message prioritizing. The LPIC expects you to be aware of these alternate non-systemd systems.

Let's review. Most log files are kept in the /var/log directory. Journalctl can be used to read entries in the systemd journal. Journal behavior can be controlled from the /etc/systemd/journald.conf file. Logger will send a text message to a log file. Logger -p can specify facility and level names. You can control log rotation through /etc/logrotate.conf. Finally, you should be aware of alternate logging systems like syslog, rsyslog, syslog -ng, and klogd.

Managing logs on AWS with CloudWatch and CloudTrail

Now we'll talk just a bit about logging on AWS. The Amazon CloudWatch service allows you to monitor system activity from multiple resources. You can configure CloudWatch to track and generate alarms based on your usage metrics from your AWS account resources, or even monitor and manage the traditional log data produced by Linux instances running on EC2. Amazon's CloudTrail tracks and records all API calls received by your AWS resources. CloudTrail logs are saved to Amazon's Object Storage Service, S3. As these logs grow in size, you can rotate and archive them in much the same way we do with logrotate, using S3 bucket versioning accompanied by life cycle rules that archive older logs to Glacier, Amazon's cheaper long-term storage service.