Typically, Cloud Pub/Sub will deliver each message once.  However, sometimes a message will be delivered multiple times.  This can create problems.  So in this section I am going to discuss how to handle duplicate messages.

First, you can help minimize duplicates by extending the message deadline.  Duplicates are often caused by a subscriber taking too long to acknowledge the message.  Setting a longer deadline should help reduce the duplication rate.  It is also possible that your subscribers are simply getting overwhelmed.  You might consider adding more subscribers to keep up with the load.   Or you can upgrade your subscribers so that they can each handle more.  Finally, enforcing message ordering can cause an increase in duplicated messages.  So if ordering is not strictly required, you could try disabling that as well.  Now while these techniques can help reduce the number of duplicates, they will most likely not eliminate them 100%.

Usually filtering out duplicates on the subscriber side isn’t too difficult.  You can often add a little extra logic to check if you have already processed a message.  If you were, say, encoding videos, you would first check to see if the current video has already been encoded.  Also, you should be aware that all messages in a given topic are assigned a unique message ID.  You can use this field to keep track of messages that have already been processed.  For example, you can store the processed IDs in a database or cache.  You can also use the IDs to compare two messages that you suspect might be duplicates.  If your subscribers are built to be as robust and fault tolerant as possible, you shouldn’t have to worry about the occasional duplicate.

Sometimes, you won’t be able to tolerate any duplicates at all.  If you need to guarantee exactly-once-processing of your messages, then consider using the Apache Beam programming model.  Apache Beam lets you interact with Cloud Dataflow, and you can use the Beam PubSubIO connector to read from Cloud Pub/Sub.  Essentially, you will define a pipeline and then let Apache Beam filter out all the duplicates for you.

Remember, Cloud Pub/Sub by itself only offers at-least-once delivery.  If you need exactly-once-delivery, you are going to need some additional logic.