How to Stream Events from PubNub to RabbitMQ

PubNub Events & Actions is a PubNub service that allows you to capture specific PubNub events, such as publishing a message and sending these events to third-party ecosystems. We support sending PubNub Events to these ecosystems, which we define as a PubNub Action, such as connecting a webhook or directly connecting to your cloud services, such as AWS, with many more coming soon. Everything is configured now in the admin portal with zero coding involved. 

Please refer to our documentation for more information on getting started with PubNub Events & Actions. You can also check out our how-to on setting up events and actions, covering the basics of creating Event Listeners and Actions. Also, check out the how-to on filtering messages using JSONPath for more details on filtering events.

Advanced Message Queuing Protocol (AMQP) is a messaging protocol that allows client applications to communicate with messaging middleware brokers through a scalable and secure API. Originated in 2003 by John O’Hara at JPMorgan Chase in London, AMQP facilitates complex messaging solutions, including IoT data streaming and backend services integration. By mid-2005, efforts to form an AMQP user group saw participation from IT giants like Cisco, Red Hat, and IONA Technologies, emphasizing the importance of publish/subscribe messaging patterns.

Messaging brokers receive messages from publishers (applications that publish them, also known as producers) and route them to consumers (applications that process them).

Since it is a network protocol, the publishers, consumers and the broker can all reside on different machines.

RabbitMQ is an open-source messaging-queuing software known as a message broker or queue manager. RabbitMQ implements AMQP 0.9.1 as the basis for a set of standards controlling the entire message queue process. AMQP 0.9.1 was published in November 2008, which is also the version used in this tutorial. Simply put, it is software where queues are defined to which applications connect to transfer a message or messages. 

A message can include any kind of information. It could, for example, have information about a process or task that should start on another application (which could even be on another server), or it could just be a simple text message. The queue-manager software stores the messages until a receiving application connects and takes a message off the queue. The receiving application will then process the message. 

A message broker acts as a middleman for various services (e.g. a web application). They can reduce workloads and delivery times of web application servers by delegating tasks that would normally take up a lot of time or resources.

Connecting PubNub Events to RabbitMQ combines real-time messaging with efficient message queuing. This connection enables scalable message processing and creating proprietary data, ensuring full ownership and control over your information flow.

Ensure you are signed up for a PubNub Account, and check out Events & Actions Pricing page for cost details. You can also navigate to Events & Actions within the Pubnub Admin Portal and select “Upgrade” to view the pricing plans for Events & Actions. For this how-to, we will use the PubNub Debug console to publish messages and check the processing of those messages inside a RabbitMQ instance.

We will set up a RabbitMQ instance inside AWS using Amazon MQ, allowing you to set up Brokers. There are many different ways to get a message broker or, more specifically, a RabbitMQ broker hosted, such as running a RabbitMQ Image inside a Docker Container and hosting it using an EC2 instance. However, for this how-to, we will be using Amazon MQ. 

To create a RabbitMQ instance inside Amazon MQ, follow the steps below:

1. Log in to Amazon Web Services (AWS)

2. In the search bar, search “Amazon MQ” and click on the Amazon MQ service

3. Click on the “Create brokers” button to configure your broker

4. Under “Broker engine types,” select “Rabbit MQ.”

5. Select your deployment mode (Depending on the use case of your application) 

6. Select “Single-instance broker” as you don’t need multiple availability zones.

7. Enter a broker name such as “rabbit-1.”

8. Under RabbitMQ access, enter a username and password for the RabbitMQ Management console. You will use this later to sign into your RabbitMQ instance.

9. Remember your username and password, as you won’t be able to view it again.

10. Select “Create Broker”. The broker will take approximately 20 minutes to create.

After following these steps, your Amazon Web Portal should look like the following:

Your Message broker is now set up to start processing PubNub Events. We now just have to connect your RabbitMQ instance to PubNub Events & Actions.

We will set up a PubNub Event Listener to listen for specific PubNub events and even filter out PubNub Events if needed. Event listeners wait for a particular set of conditions to trigger a pre-defined action, such as sending PubNub messages to your RabbitMQ instance. For more information on filtering PubNub events, check out our documentation. 

To create an Event Listener, follow the steps below:

1. Log in to the Admin Portal.

2. Click the Events & Actions tab on the left-hand side of the portal.

3. Follow the popup dialog and select one of your applications

4. Follow the popup dialog and choose a keyset

5. Click the “ + Event Listener” button 

6. Choose your Event Source. For this how-to, we will be choosing “Messages.”

7. Under “Event Producer,” select “Pub/Sub.”

8. Under “Event Type,” select “Message sent.”

9. Select your Filter Type for this how-to, “No filter.”

10. For example, click the Pencil Icon to name your event listener: “Rabbit-1 Listener.”

11. Click “Save Changes” in the bottom right of the screen.

After following these steps, the event on the PubNub portal should look like the following:

Now, we will set up a Rabbit MQ action on the PubNub portal:

1. Back on the “Events & Actions” tab select “+ Add Action”

2. Under Action Type, select “AMQP.”

3. Select the drop-down “Add event listener” and select the event listener created above. In this case, select “Rabbit-1 Listener.”  

We will need to get the AMQP URL to our RabbitMQ instance. First, AMQP URLs are formatted in the following format:

amqps://username:password@host.name:port

The username and password you assigned to the RabbitMQ instance will be used here so PubNub can send messages to your instance. In my case, the URL would be defined as such:

amqps://username:password@b-d7d2f067-c7be-45a9-8c20-de79a3c309d0.mq.us-west-2.amazonaws.com:5671

You can find the hostname by navigating to the Rabbit MQ instance inside Amazon MQ.

1. Select your instance; if you have followed along, select the “rabbit-1” instance.

2. Scroll down. Under “Connections” and “Endpoints,” copy the hostname and port.

In this case, my hostname was:

b-d7d2f067-c7be-45a9-8c20-de79a3c309d0.mq.us-west-2.amazonaws.com:5671

So far, our action should look like this. However, we need to create and exchange and routing key. To do this, follow the steps below:

1. Navigate back to the Amazon MQ “rabbit-1” instance.

2. Scroll down to Connections and click on the RabbitMQ web console link. This will direct you to the RabbitMQ management console.

3. You should see a web page that looks like this:

Enter your username and password defined earlier when creating the RabbitMQ instance. After you have logged into your RabbitMQ Management console, we will create a queue. Follow the steps below to get your routing key:

1. After logging into the management console, navigate to the tab “Queues.”

2. Select the drow down “Add a new queue.”

3. Give the queue a name; this will be your routing key

4. Select “Add queue.” 

Your Management console should now look like this:

Now, we will create an exchange for your queue. An exchange is where messages are sent. Exchanges take messages and route them into 0 or more queues. The routing algorithm depends on the exchange type and rules called bindings. Let’s make an exchange.

1. Navigate to the tab “Exchanges”.

2. Select the exchange “amqp.direct”.  

3. Select the drop-down “Bindings”. Here is where we are going to bind the exchange to your queue.

4. In the tab “To queue,” we want to write “Publisher” as that is the name of our queue.

5. The routing key can be whatever you prefer. I used “amqp.publisher” for mine.

Your RabbitMQ console setup should now reflect the exchange and routing key configuration, tailored for handling real-time data and notifications in a microservices architecture.

Copy the name of your Routing Key and Exchange landfill out the required fields in the PubNub action:

We can use the PubNub Debug Console to test out our integration. Navigate to the PubNub Admin Portal, and on the left side, select the “Debug Console.” Since we have no filters in our Event Listener, any time we Publish a Message to any channel, our action should fire, and the PubNub message should be sent to be processed by our RabbitMQ instance. Simply type in any message and hit “Send” on either client. We will now check if our RabbitMQ instance has received this message.

Once we have published a message, we should be able to see the message show up in the “Publisher” queue inside the RabbitMQ Management Console, demonstrating efficient message throughput and the robustness of our real-time messaging system. You should see your message inside the RabbitMQ console like this.

In this case, I have published 7 messages, which are all in the queue ready to be processed.

This how-to described the steps to set up PubNub Events & Actions and a hosted RabbitMQ instance for developing responsive messaging systems ensuring high message delivery throughput. There are many use cases for why someone would want to process their own PubNub messages, such as defining some operation with the real-time interactivity PubNub provides with your data connected to your web servers. For example, utilizing the PubNub Unity SDK, real-time processing of user actions in multiplayer games can be used for matchmaking, live game status updates, leaderboards, and in-game chat, providing a more interactive gaming experience. There are many other use cases for web apps, so check out our Javascript or Java SDK. 

While this article has discussed one of the PubNub Actions you can set up with PubNub E&A, there are many more, which include:

• Make a Webhook call (i.e. call your server for each filtered event)

• Save to SQS (i.e. save each Event to Amazon’s SQS queuing service)

• Send to Kinesis (i.e. send to Amazon’s Event Ingestion Service)

• Send to S3 (i.e. store the data in an Amazon S3 bucket)

• Send to Kafka (i.e. send the data to Apache Kafka)

• Create IFTTT Action (i.e. send to If This Then That)

Feel free to contact the developer relations team at devrel@pubnub.com with any questions or concerns.