Working with the Key-Value Store

Working with Key-Value Store Advanced Functions Techniques.jpg

This is the second post of our 4-part series on Advanced Functions programming techniques. We’ve also talked about Encryption, triggering 3rd party web services, and Event Handler Types.

What is Key-Value Store?

Functions is incredibly powerful in that it allows you to call a web service in real time as your data is flowing through the network. But what about storing data? That’s where the Key-Value Store comes into play.

KV Store is short for key-value store, also known as key-value database. The Functions KV Store is designed around low-latency, so it doesn’t have any locks. It’s eventually consistent, meaning you can store something, and that storage will eventually propagate to the network.

Click here for documentation on KV Store. We’ve also got a great overview in our Intro to Functions Training (skip to 20:37 for the KV Store section).

Key-Value Store Overview

In this article, we’ll showcase the different capabilities of KV store, and dive into a simple example of how to handle data stream message events to implement a messaging “Stats Bot” using custom serverless event handling in a real-time Angular 2 web application.

The “Stats Bot” is a demonstration of the Functions KV Store, our highly scalable data store that integrates directly with Functions handlers. Our “Stats Bot” handler processes message events and persist message statistics to the Functions KV Store. Although the example is small (using fixed commands and limited data), it could easily be extended to call remote web services for message persistence, natural language detection, sentiment analysis, bridging to external message systems and more.

Here’s how our sample app looks. The “Stats Bot” feature displays recent message statistics so that the web client UI code can stay simple and just display the data.

Functions KV Store

How do we implement a “Stats Bot” in PubNub? First, let’s take a moment to review the relevant concepts and features.

PubNub makes implementing real-time systems much easier by integrating enterprise-grade security, presence, storage and other advanced features into a huge (and growing!) number of client toolkits. The SDKs we’ve personally used with these features include JavaScript, Java, Android, iOS, Ruby and Python. If you’re using another language, check out the PubNub SDK page – chances are, it’s already supported!

The core of PubNub is Data Streams – real-time message channels that allow an unbounded numbers of devices to publish and subscribe to messages. The Functions feature allows developers to attach custom code to those message events in one of two ways, either as a “before publish or fire” event which allows modification of message attributes, or as an “after publish or fire” event which takes place after the message has been published to the message channel. (There is also an “after presence” event for implementing Presence-related functionality).

In this case, we create a BLOCK that intercepts the “before publish or fire” events in a message channel to implement the “Stats Bot” functionality. The big question is, how do we make those stats persistent in the context of data streams? This is a prime use case for the Functions KV Store, which offers scalable data storage capabilities for real-time applications. Here’s how our BLOCK works:

If the text of the message is the special command “/my_stats”, then we retrieve the “latest_msg” and “count” values for the user from the KV store (meaning the last message sent and total message count respectively).
Otherwise, we update the “latest_msg” and “count” values for the user in the KV store accordingly.

The KV store is scoped to the PubNub subscribe key we’re using, so it’s easy to create functionality that spans all the data stream channels in our application. Given the available operations (get, set, and get/increment counters), the KV store could be used for caching results of remote APIs, latest activity, or statistics such as voting, presence information, or IP blacklisting and rate limiting.

Obtaining Your PubNub Developer Keys

The first things you’ll need before you can create a real-time application with PubNub are publish and subscribe keys. Just in case you haven’t already, you can create an account, get your keys and be ready to use the PubNub network in less than 60 seconds using the handy signup form.

Once you do that, the publish and subscribe keys look like UUIDs and start with “pub-c-” and “sub-c-” prefixes respectively. Keep those handy – you’ll need to plug them in when initializing the PubNub object in your HTML5 app below.

Setting up the BLOCK

With Functions, it’s really easy to create code to run in the network. Here’s how to make it happen:

Go to the application instance on the PubNub Admin Dashboard.

Create a new block.

Paste in the block code from the next section.

Start the block, and test it using the “publish message” button and payload on the left-hand side of the screen.

That’s all it takes to create your serverless code running in the cloud!

Diving into the Code – the BLOCK

You’ll want to grab the 32 lines of BLOCK JavaScript and save them to a file, say, pubnub_kvstore_block.js. It’s available as a Gist on GitHub for your convenience.

First up, we create a function to handle incoming messages. We include a dependency on the ‘kvstore’ module so that we can query and update values in the data store.

export default (request) => {
    const db = require('kvstore');

Next, we determine the “keys” to use for the data store (based on the user ID).

var by = request.message.by || request.params.uuid;
var last_key = by + ":last_msg";
var count_key = by + ":count";

If the message text is the special value “/my_stats”, we query the data store and decorate the message with the relevant values. The key thing to observe here is that the KV store uses promises, so we use then() to pass the handler to the promise. There are 2 operations required for our stats bot: retrieving the latest message (using the get() operation), and incrementing the message count (using the getCounter() operation). This means our JavaScript function has 2 levels of nesting corresponding to the sequence of promises.

if (request.message && (request.message.text == '/my_stats')) {
    //
    // special "my_stats" command handling
    //
    return db.get(last_key).then((last_msg) => {
        return db.getCounter(count_key).then((count) => {
            request.message.last_msg = last_msg || "<not found>";
            request.message.count = count || 0;
            return request.ok();
        });
    });
}

Otherwise, it’s just a normal message – we determine the message timestamp (based on the Functions environment) and decorate the message accordingly. Finally, we perform the 2 required KV store operations to set() the value of the last message and increment the message count using incrCounter(), then return the message itself.

    //
    // normal case, store the message details
    //
    request.message.at = new Date().toISOString();
    return db.set(last_key, request.message).then((last_result) => {
        return db.incrCounter(count_key).then((count_result) => {
            return request.ok();
        });
    });
}

All in all, it doesn’t take a lot of code to add a custom “Stats Bot” to our application. We like that!

OK, let’s move on to the UI!

Diving into the Code – the User Interface

You’ll want to grab these 95 lines of HTML & JavaScript and save them to a file, say, pubnub_kvstore_ui.html.

The first thing you should do after saving the code is to replace two values in the JavaScript:

YOUR_PUB_KEY: with the PubNub publish key mentioned above.
YOUR_SUB_KEY: with the PubNub subscribe key mentioned above.

If you don’t, the UI will not be able to communicate with anything and probably clutter your console log with entirely too many errors.

For your convenience, this code is also available as a Gist on GitHub, and a Codepen as well. Enjoy!

Dependencies

First up, we have the JavaScript code & CSS dependencies of our application.

<!DOCTYPE html>
<html>
  <head>
    <title>Angular 2</title>
    <script src="https://unpkg.com/core-js@2.4.1/client/shim.min.js"></script>
    <script src="https://unpkg.com/zone.js@0.7.2/dist/zone.js"></script>
    <script src="https://unpkg.com/reflect-metadata@0.1.9/Reflect.js"></script>
    <script src="https://unpkg.com/rxjs@5.0.1/bundles/Rx.js"></script>
    <script src="https://unpkg.com/@angular/core/bundles/core.umd.js"></script>
    <script src="https://unpkg.com/@angular/common/bundles/common.umd.js"></script>
    <script src="https://unpkg.com/@angular/compiler/bundles/compiler.umd.js"></script>
    <script src="https://unpkg.com/@angular/platform-browser/bundles/platform-browser.umd.js"></script>
    <script src="https://unpkg.com/@angular/forms/bundles/forms.umd.js"></script>
    <script src="https://unpkg.com/@angular/platform-browser-dynamic/bundles/platform-browser-dynamic.umd.js"></script>
    <script src="https://unpkg.com/pubnub@4.3.3/dist/web/pubnub.js"></script>
    <script src="https://unpkg.com/pubnub-angular2@1.0.0-beta.8/dist/pubnub-angular2.js"></script>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" />
  </head>

For folks who have done front-end implementation with Angular2 before, these should be the usual suspects:

CoreJS ES6 Shim, Zone.JS, Metadata Reflection, and RxJS : Dependencies of Angular2.
Angular2 : core, common, compiler, platform-browser, forms, and dynamic platform browser modules.
PubNub JavaScript client: to connect to our data stream integration channel.
PubNub Angular2 JavaScript client: provides PubNub services in Angular2 quite nicely indeed.

In addition, we bring in the CSS features:

Bootstrap: in this app, we use it just for vanilla UI presentation.

Overall, we were pretty pleased that we could build a nifty UI with so few dependencies. And with that… on to the UI!

The User Interface

Here’s what we intend the UI to look like:

The UI is pretty straightforward – everything is inside a main-component tag that is managed by a single component that we’ll set up in the Angular2 code.

<body>
  <main-component>
    Loading...
  </main-component>

Let’s skip forward and show that Angular2 component template. The h3 heading should be pretty self-explanatory. We provide a text box for the message and a simple button to perform the publish() action to send a request to be processed by the BLOCK.

<div class="container">
    <pre>
    NOTE: make sure to update the PubNub keys below with your keys,
    and ensure that the BLOCK settings are configured properly!
    </pre>
    <h3>MyApp Functions KV Store Integration</h3>
    <br />
    Message:
    <br />
    <input type="text" [(ngModel)]="toSend" placeholder="message" />
    <input type="button" (click)="publish()" value="Send!" />
    <br />
    <small>(type '/my_stats' to see your message stats)</small>
    <hr/>
    <br/>
    <br/>
    <ul>
      <li *ngFor="let item of messages.slice()">
        <div>{{JSON.stringify(item.message)}}</div>
      </li>
    </ul>
</div>

The component UI consists of a simple list of events (in our case, the message objects). We iterate over the messages in the component scope using a trusty ngFor. Each message includes the data from the network. In our case, the payloads include a text message attribute as well as additional attributes decorated by the BLOCK.

And that’s it – a functioning real-time UI in just a handful of code (thanks, Angular2)!

The Angular2 Code

Right on! Now we’re ready to dive into the Angular2 code. It’s not a ton of JavaScript, so this should hopefully be pretty straightforward.

The first lines we encounter set up our application (with a necessary dependency on the PubNub AngularJS service) and a single component (which we dub main-component).

<script>
var app = window.app = {};
app.main_component = ng.core.Component({
    selector: 'main-component',
    template: `...see previous...`

The component has a constructor that takes care of initializing the PubNub service and configuring the channel name, and initial values. NOTE: make sure the channel matches the channel specified by your BLOCK configuration and the BLOCK itself!

}).Class({
    constructor: [PubNubAngular, function(pubnubService){
        var self = this;
        self.pubnubService = pubnubService;
        self.channelName = 'kv-channel';
        self.JSON = JSON;
        self.messages = [];
        self.toSend = "";

Early on, we initialize the pubnubService with our credentials.

pubnubService.init({
    publishKey: 'YOUR_PUB_KEY',
    subscribeKey: 'YOUR_SUB_KEY',
    ssl:true
});

We subscribe to the relevant channel, create a dynamic attribute for the messages collection, and configure a blank event handler since the messages are presented unchanged from the incoming channel.

    pubnubService.subscribe({channels: [self.channelName], triggerEvents: true});
    self.messages = pubnubService.getMessage(this.channelName,function(){
      // no handler necessary, dynamic collection of msg objects
    });
}],

We create a publish() event handler that performs the action of publishing the new message to the PubNub channel.

    publish: function(){
        this.pubnubService.publish({ channel: this.channelName, message: {text:this.toSend} });
        this.toSend = "";
    }
});

Now that we have a new component, we can create a main module for the Angular2 app that uses it. This is pretty standard boilerplate that configures dependencies on the Browser and Forms modules and the PubNubAngular service.

app.main_module = ng.core.NgModule({
    imports: [ng.platformBrowser.BrowserModule, ng.forms.FormsModule],
    declarations: [app.main_component],
    providers: [PubNubAngular],
    bootstrap: [app.main_component]
}).Class({
    constructor: function(){}
});

Finally, we bind the application bootstrap initialization to the browser DOM content loaded event.

document.addEventListener('DOMContentLoaded', function(){
    ng.platformBrowserDynamic.platformBrowserDynamic().bootstrapModule(app.main_module);
});

We mustn’t forget close out the HTML tags accordingly.

});
</script>
</body>
</html>

Not too shabby for about 95 lines of HTML & JavaScript!

Functions References & Diving Deeper

There are a couple other things worth mentioning with respect to the Functions KV store and other Data Stream features.

You can find detailed documentation on the KV Store here and about Functions here.

In this article, we touched on a couple operations, but there are a few more KV Store operations we’d like to mention:

set(key, value, optional TTL) : to set a JSON object value in the KV store with optional TTL in minutes.
get(key) : to retrieve a JSON object from the KV store.
setItem(key, value, optional TTL) : to set a simple string value in the KV store with optional TTL in minutes (optimized).
getItem(key) : to retrieve a simple string object (optimized).
removeItem(key) : to remove a stored value from the KV store.
getCounter(key) : to retrieve the value of a counter from the KV store (or 0 if not present).
incrCounter(key, optional val) : to increment a counter by the specified amount (or 1 if not specified).

Some additional Data Stream features you might find useful for your application include:

Server-Side Filtering : using meta attributes and the Stream Controller API
History and Replay : using the Storage & Playback APIs
Presence : real-time status and custom state using the Presence API
And more!

All in all, we found it pretty easy to use advanced KV store features on PubNub data streams using the API, and we look forward to using some of the more advanced capabilities in our future messaging applications!

Conclusion

Thank you so much for joining us in the KV store post of our Advanced Functions Techniques series! Hopefully it’s been a useful experience learning about working with different KV store operations for various use cases. In future articles, we’ll dive deeper into additional techniques for crafting Functions and integrating other nifty services in real time web applications.

Stay tuned, and please reach out anytime if you feel especially inspired or need any help!

Working with Key-Value Store Advanced Functions Techniques