Build

The Right Way to Log JSON Messages to a Private Database

2 min read • Stephen Blum on Feb 8, 2018

At time of writing, PubNub does not index your JSON messages using FTS Indexing. Additionally you will want to log your JSON messages for future use cases. Your data is valuable. You may need full-text search indexing. Your database can do this. Common databases include MongoDB, MySQL, Oracle DB, PostgreSQL, SQLite and others. Also you may use an API provider like Algolia for full-text searching.

AI and ML allow you to create insight from your data using Tensorflow. You may want to run data analysis for the message content using EMR / Hadoop or other big data analysis software. This guide will help you save your JSON Messages into your data store for input vectorizing into an AI Model.

On StackOverflow, we received this question:

What is the right way to log every published message and save it to my server db.

There are two options which I can think of:

  1. Use Function After Publish event and forward the message to a dedicated logger channel. the server will subscribe to the channel and save the arrived messages to db. Here rise another question: when I forward the message to another channel in the Function will it also trigger the Function?
  2. use PubNub XHR request/response function and call to a rest API on my server with the published message and save it to db

What is the best practice concerning performance and cost?

Log JSON Messages to Your Private Database

While many approaches exist, one is the best. Using Functions, you can asynchronously save JSON messages reliably to your database using an OnAfter Publish Event.

Note: your database needs to be accessible via a secured HTTPS interface.

Assuming you already have a stream of JSON messages being published to a PubNub data channel (chat messages, Bitcoin prices), follow these easy steps. You will create a real-time function that is triggered with every Publish event.

Here’s how you do it:

  1. Collect a list of channel(s). You may be interested in ALL channels *.
  2. In your Admin Dashboard find your app, then click Functions.
  3. Create a new Module for your specified API key set.
  4. Create a new OnAfter event handler on * channel.
  5. Use the following example code to save, asynchronously, your JSON messages to your database.
  6. Make sure to modify the URL/Parameters to fit your needs.

Asynchronous HTTPS Save Message Function

export default request => { 
    const xhr  = require('xhr');
    const post = { method : "POST", body : request.message };
    const url  = "https://my.company.com/save";
    // save message asynchronously
    return xhr.fetch( url, post ).then( serverResponse => {
        // Save Success! 
        return request.ok();
    }).catch( err => {
        // Save Failed! handle err
        return request.abort();
    });
}
0