How to Use Pub/Sub
The course is part of this learning path
Organizing and responding to events in a decentralized system (such as with microservices or IoT) can be a challenge. In this course, you will learn how to use Google Cloud Pub/Sub to create a reliable, asynchronous, messaging service for any scale.
- Understand what Cloud Pub/Sub is
- How to send and receive messages
- What the typical use cases are
- How to get started
- GCP Developers
- GCP Data Engineers
- Anyone preparing for a Google Cloud certification (such as the Professional Data Engineer exam)
- Access to a Google Cloud Platform account is recommended
In this lesson, I'm gonna give you a practical demonstration of how to use Cloud Pub/Sub. First, I'm going to create a topic and a subscription. Then I'm gonna send some messages and you can see how the acknowledgement feature works. Afterwards, you should be capable of setting up Pub/Sub and using it yourself.
To start out, you're gonna need to log into the Google Cloud Console. After that you want to navigate to the Cloud Pub/Sub page. Now you can either scroll through the navigation menu on the left here, or you can search for Pub/Sub in the search bar. In the side menu, you're gonna see Topics and Subscriptions. These are options for Pub/Sub. You're also gonna see Lite Topics and Lite Subscriptions. These options are for Pub/Sub Lite. Now you don't want to get these two confused. Since most people won't ever need Pub/Sub Lite and since it takes extra work to set up, for this demonstration, I'm just gonna stick to using Pub/Sub.
To start out, we got to create a topic. Make sure you pick topics from the side menu and then click on the Create Topic button. Next, you need to pick a name for this topic. You want your name to be descriptive, but remember they can't contain any spaces. So I'm gonna name this one demo_topic_1. So we see here, it's gonna offer to create a default subscription. This is because if you push a message to this topic before creating at least one subscription, your message is going to be discarded. If no one's listening, then there's no point in sending, right? Since this is just a demo I'm not worried about losing anything important. So I'm just gonna disable that. If I were building a real production system and messages were already being generated, I might want to leave that checkbox ticked. Let's confirm by clicking on the Create Topic button. And we see that since I disabled the default subscription, I also get a warning.
Next, I'm gonna show you how to create a subscription. Choose Subscriptions from the side menu, and then click on the Create Subscription button. I need to enter a name. I'm going to enter demo_subscription_1. And then I need to pick the topic to subscribe to. So we'll choose demo_topic_1. And now I have a bunch of different options. I have to specify if this is going to be a Pull or a Push subscription. A Push subscription requires providing an end point that's gonna listen for incoming messages. Now this is gonna make things a little harder to demonstrate so I'm just going to stick with Pull. I can also modify how long unacknowledged messages will be retained in the topic. Now the maximum is seven days, but you can make it shorter if you wish. I can also tell the topic to retain messages even after they've been acknowledged.
Now there's certain cases where this might be desirable, but usually you're gonna leave this unchecked. You can set an expiration period for the subscription. Now, basically, if a subscriber goes silent for a while, the subscription will be automatically deleted. This is because sometimes your subscribers won't clean up after themselves properly. A subscriber might switch topics or might just be deleted entirely. And then the old subscription is left hanging around. This feature is gonna clean up any unused subscriptions. There are a bunch of other options as well, but these are for more advanced use. I'm just gonna skip the rest and we'll accept the defaults. So I'll scroll down to the bottom and click on Create.
All right, now I have a topic and a subscription. I can start sending messages. I'm going to open up a second tab. And then in the first we'll have our topic and the second we'll have our subscription. Now on the Topics page, if I scroll down and click on the Messages tab, you'll see that I can manually publish new messages. Let me go ahead and create a few.
First, I'll click on Publish Message. Now we can see here at the top that I'm allowed to create a series of messages for testing. So let's say I wanted to create a total of 100 messages and send them out one minute apart. This allows me to do just that. Now this is helpful if I need to test or debug a subscription, but in this case, I just want to send a few messages to verify that everything works. For my first message, I'm just going to send a simple string. I'll enter, This is the first message. Then I just have to click on the Publish button to send it.
Now I want to send another message. For this one, I want to do something a little bit more complicated. So I'm gonna send a JSON object. This should give you a good idea of the different kinds of messages that you can send. If I go to the Subscription tab, I can do a Pull and I can get my messages. So if I scroll down, click on the Messages tab and click Pull, I can see my two messages that were sent. Now, if I click on Pull again, I get the same two messages. Now that's because I have not acknowledged them yet.
Now on this screen, to be able to send an acknowledgement, I need to enable ACK messages. Click on this check mark and then try pulling again. Now in the ACK column, I can click and acknowledge each message. All right, and now when I Pull, that same message no longer shows up. I'll go ahead and acknowledge the other message. Now, neither one is showing up. So that's about the simplest demonstration that I can do of sending and receiving messages. So this works, but it's not a very realistic scenario. You're not gonna be publishing and reading messages using the console.
So now I want to show you how to publish messages using the Google Cloud SDK and how to read messages using a Cloud Function. I'm gonna keep using the same topic so I don't need to create a new one. Typically when you're publishing messages, you're gonna use the Cloud SDK to download and install the appropriate library. If you're coding in Python, you're gonna get the Python library and then you'll call the appropriate functions. In this example, I'm gonna use the gcloud command line tool. Since I can call this from a bash script, I'm gonna say this technically counts as scripting. Everything is already set up and configured for me in Cloud Shell, so let's use that.
The command used for publishing a message, looks like this. I just need to supply the appropriate topic ID and the message. If I check my subscription again, I can see the new message. All right, so now you know how to programmatically publish a message to a topic.
Next, let's see how to programmatically receive a message. For this next part, I'm going to create a Cloud Function. I'll navigate to Cloud Functions and then click on Create Function. I need to give a name and specify a region. And for the trigger type, I need to select Cloud Pub/Sub, and then select a topic. By default, it will generate a simple function that will read all new messages automatically. I don't have to write anything myself. So we'll just click on Save, Next and Deploy.
Now we have to wait while the function is deployed. So we'll see that every time I add a new message, the function is going to read it and it's gonna print it out. We can view the logs to see the results. So now the function has been deployed. Let's take a look at the logs. Notice we don't see any messages yet. If I send a new message, we should see it in the logs. We got the new message. So this was a much more realistic demonstration of how you're going to be sending and receiving messages. Now typically your code is gonna do a lot more than just print them out, but this should be close enough that you can now start using Pub/Sub in your own projects.
Daniel began his career as a Software Engineer, focusing mostly on web and mobile development. After twenty years of dealing with insufficient training and fragmented documentation, he decided to use his extensive experience to help the next generation of engineers.
Daniel has spent his most recent years designing and running technical classes for both Amazon and Microsoft. Today at Cloud Academy, he is working on building out an extensive Google Cloud training library.
When he isn’t working or tinkering in his home lab, Daniel enjoys BBQing, target shooting, and watching classic movies.