Logging is very important today given the volume and variety of data we deal with across different customer use cases. This course will enable you to take a more proactive approach towards identifying faults and crashes in your applications through the effective use of Google Cloud Logging. As a result, you will learn how to delegate the operational overhead to GCP through automated logging tools, resulting in a more productive operational pipeline for your team and organization.
Learning Objectives
Through this course, you will equip yourself with the required skills for streaming log data to Google Cloud Logging service and use metrics to understand your system's behavior. The course will start with an introduction to the Cloud Logging Service and then demonstrate how to stream logs using Cloud Logging Agent and the Python client library.
Prerequisites
To get the most out of this course, you should already have an understanding of application logging.
Intended Audience
This course is suited for anyone interested in logging using Google Cloud Platform (GCP) Cloud Logging.
Resources
- Source code for this course: https://github.com/cloudacademy/Managing-Application-Logs-and-Metrics-on-GCP
- Google Cloud fluentd source code: https://github.com/GoogleCloudPlatform/google-fluentd
- Google Cloud fluentd additional configurations: https://github.com/GoogleCloudPlatform/fluentd-catch-all-config
- Google Cloud fluentd output plugin configuration: https://cloud.google.com/logging/docs/agent/logging/configuration#cloud-fluentd-config
- Package release history: https://pypi.org/project/google-cloud-logging/#history
- Metrics Explorer pricing: https://cloud.google.com/stackdriver/pricing#metrics-chargeable
Before we can use the Cloud Logging service on Google Cloud Platform, we need to make sure that the Cloud Logging API is enabled in the GCP Project. Let's quickly go to Google Cloud Console and enable the API. To enable the Cloud Logging API, let's click on the navigation menu and click on APIs and Services. Now here in the middle top page, we click on Enable APIs and Services.
In the search box, search for Cloud Logging and choose Cloud Logging API. Here, let's click on Enable to enable this API. It might take a few seconds for this API to come in Enable mode. If the API is already enabled in your GCP Project then you might see the Manage button instead of Enable with API Enabled and a green tick on the side.
So far, we have learned about the features of Cloud Logging and how it works under the hood. Now it is time to get in action and get a feel of Cloud Logging service using GCP Console. This is the Google Cloud Console homepage. We first make sure that we are in the correct GCP Project.
For this demo, I am using the GCP Project with ID gcp-demo-8888. Now, we navigate to the Cloud Logging service by clicking on the hamburger menu and scrolling down to Cloud Operations, and selecting Logging. This opens up the Log Explorer UI. The Log Explorer lets us view, search and analyze the logs. This UI offers many features and let's see them one by one.
The Log Explorer UI has a four key components: Query builder, Log fields, Histogram, and Query results. Query builder lets us use the Logging Query Language or drop-down menu to build the queries. It also has the tabs for recently executed queries, saved queries, and query suggestions. Query builder also has a Save button to save the current query, Stream Logs to view the logs as they arise, and Run Queries to run the provided queries. Log fields provide an efficient way to refine the queries based on Log type, Log severity level, et cetera. The histogram helps to visualize the frequency of data. This can be super useful in many cases, for example, if we want to know when the particular error started or peak of the particular type of log, et cetera. Query results show the log entries returned by the query.
We can expand a log entry and view it in JSON format. There are three useful options for each Log entry. Expand Nested fields to expand the Log entry JSON. Copy to clipboard to copy the Log entry and Copy link if we need to share the particular Log entry with a team member.
Let's look at the timestamp field, these are in GMT and can be changed to another time-zone using the drop-down menu. This Jump to now option helps to refresh the query results and displays the current results. Action drop-down offers options to set up Log-based metrics, download the log results and create a sync for the logs. If we want to hide the Log fields or Histogram, we can do so by clicking on Page Layout and then unchecking Log fields and Histogram. Time range lets us define the time range for the query results. The default selection is one hour but you can select the presented option or define the custom range.
By clicking on Share Link, we can generate the shortened URL for the current query. Logs we have seen so far are generated by different GCP services. In the coming lectures, we will learn how we can send the custom application logs or server logs to the Cloud Logging API.
Pradeep Bhadani is an IT Consultant with over nine years of experience and holds various certifications related to AWS, GCP, and HashiCorp. He is recognized as HashiCorp Ambassador and GDE (Google Developers Expert) in Cloud for his knowledge and contribution to the community.
He has extensive experience in building data platforms on the cloud and as well as on-premises through the use of DevOps strategies & automation. Pradeep is skilled at delivering technical concepts helping teams and individuals to upskill on the latest technologies.