When Cloud Pub/Sub attempts to deliver a message, the subscriber is supposed to acknowledge it in a certain amount of time.  If the message is not acknowledged, then Pub/Sub will retry sending the message.  By default, it is going to send the message immediately.  But you can also enable an exponential backoff policy for your subscription as well.

Sometimes a subscriber will refuse to acknowledge a message, and eventually that message will be considered undeliverable.  This can happen for any number of reasons.  The subscriber might be offline for an extended period of time.  It’s also possible that the message is causing the subscriber to throw an exception.  Perhaps the data is corrupt or maybe it is just formatted in an unexpected way.

To help deal with this issue, you can set up a “dead letter topic”.   Dead letter topics are topics that store any undeliverable messages from another topic.  In this way, you can be notified when there is a problem, and you can ensure no messages are ever lost.  Dead letter topics also can be useful for testing and debugging. 

So, when you set up a subscription, you can set the maximum number of retries as well as the dead letter topic to use.  Any undeliverable messages will be moved to the specified topic and have a field added, called “delivery_attempt”.  This field will contain the number of delivery attempts made to the subscriber.

Now, I want to demonstrate how to set up a Dead Letter Topic.

I first need to create a topic that will hold the original messages.  I’ll call this one “topic-3”.

Before I create my subscription, I also need to create my Dead Letter Topic.  Now a Dead Letter Topic is created in exactly the same way as a regular topic.  There is no real difference, I just need to give it an appropriate name.  So I’ll call it “topic-3-lost”.  This topic will end up holding any lost or undeliverable messages from “topic-3”.

Now that I have both topics created, I can create the subscription.  I’ll call it “subscription-3”.  And I need to select “topic-3” as the topic I will subscribe to.  Now make sure you pick the main topic here, not the dead letter topic.

Then I will scroll down and enable “Dead lettering”.   And now I can select the topic that I wish to use as a Dead Letter Topic.  And also can choose the number of delivery attempts that will be made before moving a message over to the Dead Letter Topic.  In this case, let me set it to 10.

And click “Create” to confirm.

Now I am almost done, but not quite yet.  In order to actually move messages from our main topic to our dead letter topic, we need to grant the proper permissions.  Luckily, it’s as simple as clicking on the provided buttons.

I have already selected the dead letter topic, so I don’t have to do anything here.  I do need to grant permission for publishing messages to the dead letter topic, so I’ll click here.  And I also need to grant permission for reading messages from the main topic, so I’ll click here.

Now the last thing I have to do is to create a subscription to the dead letter topic.

Remember, if a topic does not have a subscription, then any message published to it will be discarded.  So right now, if any messages are moved to the dead letter topic, we are going to lose them.  So I will create “subscription-3-lost”.

I ended up with two topics.  Topic-3 is the main topic and topic-3-lost will hold any messages that can’t be delivered after 10 attempts.  

I am also ending up with two subscriptions.  Subscription-3 will receive messages published to the main topic.  And if any message is undeliverable, subscription-3-lost will end up getting those.  

So that’s it.  That is how to set up a dead letter topic.