Events & Actions: How to Filter Messages Using JSONPath

Events & Actions (E&A) enables you to filter events in your application's ecosystem and send real-time data to third-party systems for storage, analytics, or ML/AI without having to write code. PubNub Events are generated during the app's lifecycle, such as when a device publishes a message via an API call and can trigger these events with a particular event listener. When the event listener is activated, Actions are the configurable way to send data using a webhook, Amazon SQS, and Amazon Kinesis. You can learn more about the difference between E&A and Functions in our blog post, and the prerequisites on how to configure E&A in the Admin Portal. Set up the different actions associated with these events in our how-to guide.

To specify the Events’ Conditions that trigger Actions, you can use filters to identify which part of a message you need to parse to trigger a listener. While selecting No Filter doesn’t filter messages and triggers actions for all events, you can filter specific data with one of two filters: Basic and Advanced JSONPath. Although the Basic Filters allow you to specify a channel or sender ID to filter, the Advanced JSONPath Filter allows you to filter for specific information contained in your message payload using a JSON inspection language called JSONPath. Since all events generated by PubNub follow a schema used to write JSONPath expressions, you can set up complex expressions to filter for multiple conditions at a time, including custom metadata, deploying only the relevant data. 

The following Advanced JSONPath expressions are examples of how you can use these expressions to filter for relevant Events in your own ecosystem:

1. $.[?(@.uuid == ‘user_id’ && @.channel == 'channel_name’)] - Match on specific User and Channel

2. $.[?(@.uuid =~ /.*user_id.*/i) && @.channel.*)] - Match on User with regex in every channel.

3. $.message[?(@.text contains 'temperature')] - Filter on message text that contains specific text.

4. $.meta.sensorsList.sensor[?(@.sensor_price < 10)] - Matches on messages where the sensor_price field’s value in a nested property in the meta Condition that is less than 10.

5. $.meta.sensorsList.sensor[?(@.sensor_price < 10 && @.status == 'online')] - Match when the sensor_price field is less than 10 and the status is ‘online’.

Continue reading this guide if you would like to learn about the other ways of formatting complex Advanced JSONPath Expressions, why you would want to use the Advanced JSONPath Filter in E&A, and understand the structure of JSONPath used to format the filter expressions.

To learn how to use the Advanced JSONPath Filter, it’s best to understand the differences between the different filter options available. In the Events & Actions section of the Admin Portal, create a new Event Listener that is listening for Messages for your app and keyset. Please review our how-to guide on how to set up Events & Actions in the Admin Portal if you would like more details. There are three types of filters to choose from in the Filter Type dropdown list: No Filter, Basic Filters, and Advanced JSONPath.

No Filter doesn’t filter messages and triggers actions for all messages. This is used when you don’t want to filter for any Events but would like to still trigger an Action for all messages.

Basic Filters can be used to trigger Actions for predetermined conditions for the Event and act as templates to handle the JSONPath filtering for you. There exist two basic filters for the Messaging Event to trigger an Action: Channel and Sender ID.

Selecting the Channel filter will trigger an action after a message is published on a specific channel. The Sender ID filter will trigger an action after a message is published by a specific sender. You then would select a boolean condition that will match your specific string Value input (think of this like the condition in a WHERE clause). Users can select the Exact Match condition, which triggers an action only if there is an exact match. The Contains condition triggers an action if the specified value is found within the filter value. The Value field requires a value that must match the condition the user is trying to filter. 

To see this in action, select the Basic Filters filter type, and choose the Sender ID condition that has an exact match of admin-user.

1. Create an Action by scrolling down in the Event Listener and giving it an appropriate name.

2. Make a Webhook Action.

3. Provide the Webhook URL. You can obtain a free Webhook URL from webhook.site. Stay on this site, as you will also test your webhook after publishing a message on the PubNub Network.

4. Save the Action and Event. 

5. Open the Debug console, a troubleshooting tool that allows you to test your PubNub connectivity and app integration. Add your publish and subscribe keys from your keyset. Add the UUID “admin-user” and the channel “temperature-capture-channel”.

6. Send a message payload from Device 1, with the following JSON format in the Message section:

7. In the webhook.site where you obtained your webhook URL, there should be a JSON response that contains relevant the message, as it came from the client with the ID admin-user.

8. Change the UUID to something else and publish the message once more. In the webhook.site, there should be no response, as the IDs do not match.

While the Basic Filter can quickly filter on predetermined filters such as Channel and Sender ID, the Advanced JSONPath Filter allows for much more flexibility as it can look into different data points in the Event payload and allow for a more robust setup of event handlers than Basic Filters. All events generated by PubNub follow a schema that’s used to write JSONPath expressions, so you can translate this usage across different functionality with PubNub. You can even combine multiple conditions that will trigger an action in the same path expression, where normally you would need to create multiple Events of different predetermined filters to achieve the same result.

JSONPath can be used to define the following Event Conditions to trigger an action:

As an example, navigate back to your Event in the Admin Portal and change the Filter Type to Advanced JSONPath. Send the same JSON data as before.

Suppose you want to filter for messages not only from the user admin-user but also on the channel temperature-capture-channel. You can copy the following JSONPath Filter Expression to filter for this information:

Send the JSON message with the payload above from the admin-user UUID in the Debug Console and view the webhook to see your message get caught. Now send the same message after changing the UUID to see that the message is not caught by webhook.site.

Although the JSONPath expression used in this example is utilizing multiple conditions to filter for at a time, you can create more complex JSONPath expressions to filter for detailed information, such as for custom metadata you are sending across the PubNub Network.

As previously mentioned, JSONPath is an inspection language for JSON that allows you to obtain specific information from a JSON structure or JSON document. Similar to how XPath is used to navigate to specific parts of an XML document and SQL for database queries, JSONPath expressions are used to traverse the path in a JSON structure to find the value of nodes that matches the entered expression. Each component of the structure can be represented as nodes and their connections to adjacent nodes (think of traversing through a tree or graph). The expressions are case-sensitive and contain specific syntax used to format the expression with a combination of logical operators and functions to evaluate the different data types. You can even use strings or even Regular Expressions to match multiple values that match the RegEx pattern.

If you are unfamiliar with JSONPath, it’s recommended that you look at the JSONPath GitHub documentation. PubNub’s E&A system uses this exact implementation of JSONPath (based on a Java Library) and provides documentation on three key aspects of JSONPath: syntax, operators, and functions. The documentation also provides examples of formatting JSONPath expressions to filter for specific information with a given JSON structure.

The Advanced JSONPath Filter message publish event schema implements JSONPath by providing the specific conditions for users to construct in their payload:

These Conditions are used to construct the JSONPath expression, beginning with the $ symbol, which represents the root member object or node. The $ symbol is then followed by the Conditions that you wish to filter in the JSON structure and is formatted in a special way known as a notation. Two notations can be used: dot-notation, such as $.meta.tempList.reading, or bracket-notation, such as $['meta']['tempList']['reading'].

How you define the notation is dependent on what information you have included in the JSON payload that is sent across the PubNub network. All message publish events generated by PubNub follow the schema above that’s used to write JSONPath expressions, so you can translate this usage across different functionality for PubNub. The message and meta Conditions are of the content type JSON objects themselves, meaning you can filter on complex JSON objects within the payload. You will need to format your JSON payloads that match this schema (at least in the Debug Console as the SDKs handle this for you) to ensure you are evaluating the correct child element when formatting your expression. 

Here are some common and specific use cases of Advanced JSONPath Filter Expressions you can define in E&A to filter for messages that satisfy the specified Condition(s). You can use these JSONPath Filter Expressions in your ecosystem, but please be sure to replace any of the values with your own:

1. $.[?(@.uuid == ‘user_id’)] - Match on specific User

2. $.[?(@.uuid =~ /.*user_id.*/i)] - match users with regex, case insensitive.

3. $.[?(@.channel =~ /.*channel_name.*/i)] - match channels with regex, case insensitive 

4. $.[?(@.uuid == ‘user_id’ && @.channel == 'channel_name’)] - Match on specific User and Channel

5. $.[?(@.uuid =~ /.*user_id.*/i && @.channel == 'channel_name')]

6. $.[?(@.uuid =~ /.*user_id.*/i) && @.channel.*)] - Match on User with regex in every channel.

7. $.[?(@.message.text == 'New temperature readings.')] - Filter on message payload for exact text.

8. $.message[?(@.text == 'New temperature readings.')] - Another way to filter on message payload for an exact test.

9. $.message[?(@.text contains 'temperature')] - Filter on message text that contains specific text.

10. $.[?(@.message['some_property'] == 'some_value')] - Match on arbitrary fields in the message payload, such as timetoken or timestamp.

11. $.meta.sensorsList.sensor[?(@.sensor_price < 10)] - Matches on messages where the sensor_price field’s value in a nested property in the meta Condition that is less than 10.

12. $.[?(@.meta.sensorsList.sensor in ['warn', 'alert'])] - Match on arbitrary array elements in message metadata that is in the set 'warn' or 'alert'.

13. $.[?(@.meta.property contains ‘new reading:’)] - Match when a specific property’s value in the meta condition contains specific text.

14. $.meta.sensorsList.sensor[?(@.sensor_price < 10 && @.status == 'online')] - Match when the sensor_price field is less than 10 and the status is ‘online’.

15. $.[?(@.channel =~ /.*channel_id*/i && @.meta.property < 10 && @.message.status != 'error')] - Match on complex combinations: match on channels (case insensitive) with channel_id, as well as a specific property’s value in the meta Condition is less than 10 and the status in the message Condition is not an error. 

16. $.[?(@.channel == ‘chat’ && @.meta.length() > 3] - Match on publishes that have a channel that is chat exactly, and metadata object set as an array with at least 4 elements (greater than 3).

17. $.[?(@.message.sensor_reading.status nin [‘OK’, ‘ERROR’])] - Match on publishes that have a message field with a custom JSON object with a sub-object sensor_reading with a field status that has a value, not in the set OK or ERROR.

18. $.meta.sensor_reading: - Match all messages that contain a sensor_reading top-level field in the meta-object. Note explicit JSON null values such as sensor_reading: null will evaluate as true since the field exists.

To see the Advanced JSONPath Filter expressions in action, let’s take a look at a more complex payload that can be sent across the PubNub Network. Add the following JSON payload to the message field in the Debug Console.

In the Metadata section (within the braces), add the following:

Change the JSONPath Filter Expression value in E&A to match those in the examples provided above, replacing it with the specified conditions you want to match for. For the meta Condition to filter the Metadata section, you’ll need to enable the Objects feature in the keyset to enable permissions. Back in the Debug Console, use the same publish, subscribe, channel, and UUID values, but format the message field with the example JSON payload used above. Send the message after changing to use the different JSONPath expressions above and view the caught Event at webhook.site.

These are just a few different Advanced JSONPath Filter Expressions you can use to filter on and extract from messages sent across the PubNub network. Remember, keep in mind that you can combine different conditions and perform functions to filter for the messages that suit your needs. To ensure that you are formatting and optimizing your Advanced JSONPath Filter Expressions correctly, use a JSONPath evaluator website to see if the messages will be filtered correctly before attempting to run it in E&A.

The Advanced JSONPath Filter is a powerful filter that enables users in Events & Actions to filter Events for specific information contained in your message payload using a JSON inspection language called JSONPath. Since all Events generated by PubNub follow a schema used to write JSONPath expressions, you can set up complex expressions to filter for multiple conditions at a time to export only the relevant messages. You can include custom fields in the message and meta conditions, allowing you the flexibility to filter for the specific conditions that meet your requirements, all without having to touch your application’s code.

To learn more about PubNub’s Events & Actions, we recommend you review the following resources to understand how to add Events & Actions to your existing system to help manage all Events happening in your application’s workflow and trigger additional business logic using a listener.

• Events & Actions Documentation

• A How-to guide detailing Events & Actions, and how to set them up for your own system

• A blog post on the difference between E&A and Functions

• JSONPath Documentation & Online evaluator

Feel free to reach out to the Developer Relations Team at devrel@pubnub.com for any questions or concerns.