With Pub/Sub Google offers a message broker as a service with all the benefits a customer receives when using the Google Cloud Platform (GCP). This includes high availability, autoscaling and easy handling. Pub/Sub, which stands for Publisher/Subscriber, is fully integrated in the GCP ecosystem but may also be used as a standalone service. With this message broker it is possible to publish data to a specific topic from different endpoints and on the other hand subscribe to a topic and receive the data via push or pull asynchronously. This article gives an introduction to Pub/Sub, its modules, functionalities and the basic setup as well as some hands-on demonstrations of the core functionalities using the gcloud tool.

Core Concepts​

The Pub/Sub Service consists of four key elements: the topic, the publisher, the subscription/subscriber and the message itself.¹

For each purpose it is necessary to create a topic, which is basically a named resource to which messages are sent by publishers. A publisher is an application which creates and sends messages to a topic.²

The subscriber and the subscription go hand in hand. In order to receive messages from a topic, it is necessary to create a subscription. A topic can have multiple subscriptions, but a given subscription always belongs to one topic. If there are multiple subscriptions on a given topic, all subscriptions receive a copy of the message.³

A subscriber is an application with a subscription to a topic from which it receives messages. Although it is possible to connect multiple subscribers to one subscription, it is not advisable as the data will be scattered across the subscribers nondeterministically. However, this scenario can make sense if multiple applications are used as subscribers in order to achieve load balancing.

Pub/Sub offers push- and pull-subscriber models. The pull-subscriber sends requests to an HTTP-API to "pull" new messages that have arrived. The push-subscriber instead waits and receives a webhook each time a new message arrives. In general, the pull model is more suitable for high volumes of messages where throughput is critical whereas the push model is more often used for event-based processing/triggering mechanisms. More details on subscription models can be found at: https://cloud.google.com/pubsub/docs/subscriber

A message can be any data of up to 10MB in size for the payload combined with optional message attributes as key-value pairs. Messages are buffered in the Pub/Sub-Queue until the subscriber confirms the reception with an "acknowledge". The message will stay in the subscription until it is acknowledged or its expired due to the retention policy which is configurable and can be anything between 10 minutes to 7 days.⁴

As shown in the picture, the relation between subscriber and publisher can be many-to-one, called fan in, one-to-many, called fan out, or many-to-many.

Getting Started with Pub/Sub​

The goal for this hands-on introduction is to configure a topic and a subscription to demonstrate the message buffer in which messages remain until they are acknowledged by the subscriber.

To create topics the Pub/Sub-administrator role permissions are needed, for publishers, there is a specific Pub/Sub-publisher role and for subscribers the Pub/Sub-subscriber role can be used. More details on Pub/Sub specific role & access management can be found at: https://cloud.google.com/pubsub/docs/access-control

For the following example commands, we use this configuration:

project-id: example-project
topic-id: hello-world-topic
subscription-id: hello-world-subscription

Creating topic and subscription:

To create a topic, use the following command:

 To view all existing topics, including the newly created topic, use:

To receive incoming messages on a topic, a subscription needs to be created.
In this example the message retention duration on this subscription is set to 7 days. For more information see the available options: https://cloud.google.com/pubsub/docs/admin#using_subscription_properties

To view all existing subscriptions, including the newly created, use: 

If you want to list all subscriptions belonging to one topic use:

Publishing and pulling messages:

Now that a topic and a subscription have been created, messages may be published to the topic with the following command:

In order to receive messages from to the topic, pull from the subscription. This can be done via gcloud using the following command:

While this command pulls the message from the subscription, it does not acknowledge it. Therefore, it remains in the subscription and this way the message may be pulled multiple times. Use the --auto-ack option to automatically acknowledge the message instead:

A second execution of the command shows that the message is no longer present in the subscription.

If you are finished and would like to clean up, remove the previously created resources with the following commands

Test locally with the emulator

The emulator allows to host a Pub/Sub message queue locally. It can be handled like the Pub/Sub service in GCP with topics, publisher and subscriber. The emulator does not support Cloud Console or gcloud pubsub commands. Nevertheless, it represents a great possibility to try out Pub/Sub or test applications locally without the need to authenticate with a GCP account.

A quick start guide can be found in the Google Documentation: https://cloud.google.com/pubsub/docs/emulator

Short Summary

With the Pub/Sub Cloud Service Google provides an easy solution to setup and configure a managed message broker. Topics are useful to organize different data streams and target data to different consumers or subscribers as they are called in Pub/Sub. The different delivery methods of push and pull plus the multiple possible setups of publisher and subscriber relations cover a wide range of use-cases. We notice the Pub/Sub Service gaining popularity among our customers and see great synergies when using other parts of the well-integrated Google Cloud Products such as Cloud Dataflow for large scale data processing or Cloud Functions for event-based actions.

This blog article is a joint work of Paul Hannes and Johannes Julius Schmitz, consultants of ORDIX AG.