Setting up topics and subscriptions is pretty easy in a development environment.  However, once you start generating lots of messages and have multiple subscriptions to multiple topics, things can get a lot more complicated.  Eventually, identifying and diagnosing issues in such a complex system is going to become a challenge.  That is why in this lesson, I want to talk about proper monitoring.

Cloud Pub/Sub automatically exports all it’s metrics to Cloud Monitoring.  So you will already have some default dashboards set up and ready to go.  But you can also add custom dashboards, as well as alerts.  So let’s take a look at Cloud Monitoring.

Now you can find dashboards by clicking on the menu button to the left here.  If you have created some topics and subscriptions, you should already have a link for Cloud Pub/Sub here.  Let’s take a look at the default dashboards that are available.

So here you can see general information about all your topics.  There is a chart for general operations and the average message size.  You can click on one of your topics to get more specific information about it.  You also can find information about the associated subscriptions if you click on the “Subscriptions” tab here.

Now these are the default dashboard you have available.  But you can also create your own custom dashboards as well.  If there are specific metrics you are interested in, you can create new graphs based on those.  Typically you aren’t going to want to spend a lot of time staring at dashboards.  Usually you will want to get an alert if something goes wrong.  So next I’ll show you how to setup an Pub/Sub alert.

You can access the alerts page from the menu on the left.  Just click on the option named “Alerting”.  From here you can create a new policy for your alert.  Start out by clicking on the “Create Policy” button.  Next you have to pick a metric to monitor.  You can find the three main types of Pub/Sub metrics by using the filter.  We can see there are snapshots, subscriptions, topics.  There are also options for Pub/Sub Lite as well.

Creating an alert is pretty straightforward at this point.  First pick the metric you want to monitor.  I am going to choose “unacked messages”.  Then add any filters you need here.  I want to keep this example as simple as possible, so I’ll skip that and move forward.  Now you need to specify a threshold.  This will specify the number of unacknowledged messages that I want to be alerted about.  It looks like I’ve been up to 350 in the past, so I’ll set this to 500.  And then hit “Next”.

Now I need to choose the type of alert.  You can get alerted by email, text, Slack message and there’s a bunch of other options.  I’m just going to keep this simple and use an email alert that I set up previously.  You can create a more robust alerting solution if you wish.  Now there are a bunch of other options as well, but they are optional.  I do need to name the alert to distinguish it from any others.

Once I hit “Create Policy”, I will start getting alerts when my total unacknowledged message count hits 500 or higher.  And that is how you set up an alert.  You will probably want to set up a lot more for any production system.

Let me go back and show you some more common metrics you will probably want to monitor.  Under Pub/Sub snapshot, you might want to look at “oldest snapshot message age” or “snapshot updates”.  Now these will be useful for making sure you are successfully creating snapshots.  If you are using snapshots, you want to be notified if they suddenly aren’t being generated.

Under Pub/Sub subscription, you probably should monitor the total number of unacknowledged messages to make sure your subscribers can keep up with the current load.  You might also look at “oldest unacked message age” as well.  You can use these two numbers to identify if you need more subscribers, or if there are a few stuck messages, or even if you are in danger of losing some messages completely.

You might also consider looking at “acknowledged message operations” for pull subscriptions.  If you create a filter where response code != success, this will show you how many pull subscribers are missing the ACK deadline.  You can also check push subscriptions by looking at “push requests”.  Again, you will want to add a filter to remove the response codes that equal success.  These metrics will let you know if your subscribers are under-provisioned or if some messages are crashing your clients.

For Pub/Sub topics, you probably want to monitor publish message operations and group by the response code.  Now you can use this to determine if Pub/Sub is healthy and accepting requests.  You can also look at publish requests to make sure that your publishers are still working and that messages are flowing into your topics.

Now these are the most common metrics you should be looking at monitoring.  You will probably also need others depending upon your specific needs and setup.

The last thing I want to show you is how to monitor your Pub/Sub quotas.  For this we will search for IAM & Admin.  Then scroll down and click on “Quotas”.  This page will show you all your quotas for all products.  Now we want to just see the quotas for Pub/Sub and we can do that by using the filter.  First specify Cloud Pub/Sub.  And you might want to specify your region as well.  Now this is going to show you your Pub/Sub API quota limits as well as your usage.  You can see that I am not anywhere near going over my limit.  But when you start using Pub/Sub in a production environment, you want to make sure you do not get throttled.  

You can also find the metrics used to generate this page.  If you want to configure a custom alert, you can do so.

Well, that just about covers monitoring for Cloud Pub/Sub.  Remember, proper monitoring will allow you to detect and fix issues before they affect your customers.  So it’s important that you understand how this all works.