Course Introduction & Intro to Logging
Google Cloud Logging
Log Collection
Metrics Explorer
Summary
The course is part of these learning paths
See 2 moreLogging is very important today given the volume and variety of data we deal with across different customer use cases. This course will enable you to take a more proactive approach towards identifying faults and crashes in your applications through the effective use of Google Cloud Logging. As a result, you will learn how to delegate the operational overhead to GCP through automated logging tools, resulting in a more productive operational pipeline for your team and organization.
Learning Objectives
Through this course, you will equip yourself with the required skills for streaming log data to Google Cloud Logging service and use metrics to understand your system's behavior. The course will start with an introduction to the Cloud Logging Service and then demonstrate how to stream logs using Cloud Logging Agent and the Python client library.
Prerequisites
To get the most out of this course, you should already have an understanding of application logging.
Intended Audience
This course is suited for anyone interested in logging using Google Cloud Platform (GCP) Cloud Logging.
Resources
- Source code for this course: https://github.com/cloudacademy/Managing-Application-Logs-and-Metrics-on-GCP
- Google Cloud fluentd source code: https://github.com/GoogleCloudPlatform/google-fluentd
- Google Cloud fluentd additional configurations: https://github.com/GoogleCloudPlatform/fluentd-catch-all-config
- Google Cloud fluentd output plugin configuration: https://cloud.google.com/logging/docs/agent/logging/configuration#cloud-fluentd-config
- Package release history: https://pypi.org/project/google-cloud-logging/#history
- Metrics Explorer pricing: https://cloud.google.com/stackdriver/pricing#metrics-chargeable
Welcome to the Cloud Logging Agent lecture. In this lecture, we will learn about Cloud Logging Agent, how to install Cloud Logging agent and its configuration to send log data to Google Cloud Logging service.
First, let's understand what a Cloud Logging Agent is and how it works. Cloud Logging agent is an application based on an Open Source tool, fluentd, and is often termed as google-fluentd. The source code of google-fluentd can be found at this GItHub repository. You can also get some additional configurations at this GitHub repository.
Cloud Logging enables us to collect log data from multiple sources and transfer it to the Google Cloud Logging service reliably. With the Cloud Logging agent, we can filter the log entries on the fly and buffer before transferring to the Google Cloud Logging service. Cloud logging agent or google-fluentd comes with its default configuration that can stream most common logs such as Linux Syslog, Apache access logs, Chef logs, Cassandra logs, MongoDB logs, MySQL logs, Redis logs, and many more to Google Cloud Logging service. For the full list of default logs configuration, please refer to the Google Cloud Documentation linked with this lecture.
Besides the default logs, a user can provide the custom configuration to stream other software logs to the Google Cloud Logging service. The logs from the Cloud Logging agent are named as; project/<PROJECT-ID>/logs/<LOG-ID>. Here, the PROJECT-ID is your Google Cloud Platform Project ID, and the Log ID is the name of the log, for example, syslog or mysql. Cloud Logging agent stores its root configuration at /etc/google-fluentd/google-fluentd.conf, if you are on Linux, and imports all the configuration files from /etc/google-fluentd/config.d directory. If you are on Windows, root configuration is present at C:\Program Files \Stackdriver\LoggingAgent\fluent.conf. We will go through some configuration files during the upcoming demos.
Now, we may have a question; what operating systems are supported to run a Cloud Logging agent, or where can I run a Cloud Logging agent? The answer to the above question is; we can have a Cloud Logging agent running on Google Compute Engine VM Instances, or Amazon EC2 instances. And it is compatible with many operating systems such as CentOS, Debian, RedHat, Amazon Linux.
Wait, did I mention the Amazon EC2 instance? Yes, you heard it right. We can stream logs from an Amazon EC2 instance to the Google Cloud Logging service, provided the logging agent configured on the EC2 instance can communicate with the Google Cloud Logging API. Streaming the logs from an EC2 instance is beyond the scope of this course, but you can check out the Google Cloud documentation for further reading.
Pradeep Bhadani is an IT Consultant with over nine years of experience and holds various certifications related to AWS, GCP, and HashiCorp. He is recognized as HashiCorp Ambassador and GDE (Google Developers Expert) in Cloud for his knowledge and contribution to the community.
He has extensive experience in building data platforms on the cloud and as well as on-premises through the use of DevOps strategies & automation. Pradeep is skilled at delivering technical concepts helping teams and individuals to upskill on the latest technologies.