Logging is very important today given the volume and variety of data we deal with across different customer use cases. This course will enable you to take a more proactive approach towards identifying faults and crashes in your applications through the effective use of Google Cloud Logging. As a result, you will learn how to delegate the operational overhead to GCP through automated logging tools, resulting in a more productive operational pipeline for your team and organization.
Through this course, you will equip yourself with the required skills for streaming log data to Google Cloud Logging service and use metrics to understand your system's behavior. The course will start with an introduction to the Cloud Logging Service and then demonstrate how to stream logs using Cloud Logging Agent and the Python client library.
To get the most out of this course, you should already have an understanding of application logging.
This course is suited for anyone interested in logging using Google Cloud Platform (GCP) Cloud Logging.
- Source code for this course: https://github.com/cloudacademy/Managing-Application-Logs-and-Metrics-on-GCP
- Google Cloud fluentd source code: https://github.com/GoogleCloudPlatform/google-fluentd
- Google Cloud fluentd additional configurations: https://github.com/GoogleCloudPlatform/fluentd-catch-all-config
- Google Cloud fluentd output plugin configuration: https://cloud.google.com/logging/docs/agent/logging/configuration#cloud-fluentd-config
- Package release history: https://pypi.org/project/google-cloud-logging/#history
- Metrics Explorer pricing: https://cloud.google.com/stackdriver/pricing#metrics-chargeable
Welcome. In this lecture, we will learn about Google Cloud Logging. Let's first understand what is Cloud Logging in the Google Cloud Platform world. In GCP, Cloud Logging is an exabyte-scale, fully managed service that can store all of our logs securely. It also allows us to analyze the log data and set up alerts. This means we have a logging solution that we do not have to worry about building, managing or scaling infrastructure.
Now let's understand some key features of Cloud Logging. First of all, it can ingest huge amounts of data and it scales smoothly with the size of data. Because it is a fully managed service, we do not have to worry about managing and patching the software stack, provisioning servers, or fixing disk failures.
Cloud Logging provides a modern user interface that allows users to search, analyze, and sort the logs using only a few clicks or by writing queries. This UI is called Log Explorer. Log Explorer offers many functionalities, for example, histogram, query editor, saving a query, query builder, et cetera. We will get a feel of Log Explorer later in the course during the demo.
Cloud Logging is nicely integrated with other Google Cloud services, and hence we can see the logs from different GCP services. In addition to logging the activities of GCP services, Cloud Logging allows us to write custom logs using its APIs. This means we can write logs from a system or an application running on-premises, or on another Cloud Platform. We can also set alerting based on the log event or log-based metric.
For example, getting an alert when there is an error of type critical. Cloud Logging allows us to export data to BigQuery in one click. This enables advanced analytics on the log data using BigQuery. Using this, one can get deep insights about the data. It also has a feature called Error Reporting that can automatically analyze the logs and aggregate them in the error groups.
In an organization, we get logs from different systems, and not all logs are equally important, and do not need to be stored for a long time. Cloud Logging solves this problem with log retention, where we can define how long the log data should live. And if there is a need to keep the data for a longer time for archival purposes, then it can be easily exported to Cloud Storage. This reduces our log storage cost.
In a nutshell, Cloud Logging is a central log management system that can help different sets of users like system administrators, application developers, security, compliance, and other teams, in many ways.
Pradeep Bhadani is an IT Consultant with over nine years of experience and holds various certifications related to AWS, GCP, and HashiCorp. He is recognized as HashiCorp Ambassador and GDE (Google Developers Expert) in Cloud for his knowledge and contribution to the community.
He has extensive experience in building data platforms on the cloud and as well as on-premises through the use of DevOps strategies & automation. Pradeep is skilled at delivering technical concepts helping teams and individuals to upskill on the latest technologies.