Our customers are pushing the envelope in how they create and utilize real-time technology Each application developer comes to PubNub with a set of demanding expectations that are driven by a satisfying user experience for their customers. At PubNub, we deliver hundreds of millions of real-time data streams to an enormous global audience in a quarter second or less, all day, every day.
The concept of real-time communication has changed through the course of history. Today, not everyone is familiar with exactly what a real-time data stream is but we definitely know real-time when we see it. Few of us realize how often real-time data streams are used to satisfy our never-ending thirst for knowing what is happening “right now” and even fewer are familiar with what is required to deliver real-time experiences across the Internet at scale.
Near-Time vs. Real-time Data
Not all applications require real-time data. They really only require that the data delivered be relevant to the intended task. The challenge in today’s world is that rate at which data becomes irrelevant has accelerated beyond what traditional real-time communication delivery solutions can handle.
Hourly or Daily Data
It used to be that information was fairly static and needed to be updated infrequently. Once a day, or less for example. While this is occasionally still true of applications today, the value of such information appears to be in decline.
Take the daily outcome of the stock-market results as an example. This data is either static data (yesterday’s result) or dynamic (today’s result) and it only changes every 24 hours or so. If we only needed to know a summary of the day’s events but didn’t need to know what happened in any one instant in the day, then our application could provide a summary of the day’s stock market performance and our users could check it out at their leisure.
Before this, as far back as 1797, we just read the newspaper or waited for a Reuter’s carrier pigeon to bring the latest financial news of the day to us. This was outstanding up-to-date information for its time. Today, you are at a frightening disadvantage as a market trader if you are using this type of data stream network.
Minutes or Seconds Data Stream
Today’s dynamic web applications provide updated information every so often. This type of data is dynamic with an update frequency of a few minutes or seconds.
Sticking with the financial example, for those looking to track the performance of the stock market throughout the day, using an update that occurs every few seconds might suffice. This provides all the information needed at a rate that lets us feel reasonably sure we understand what is happening during the course of the day. Updates that happen every few seconds might be considered near time, but certainly are not real time.
Back in 1870, it was possible to get the same frequency update by watching a ticker-tape print out 20 minute delayed results at a rate of one character per second. This was a cutting-edge data stream network and probably the first time you could subscribe to a networked data stream. Today you can go to the financial section of your online news source to get these type of dynamic updates.
Delivering this type of data over the internet in the 21st century is still fairly challenging. Building an application that keeps information up to date and reliably available with a reasonable level of performance for a global audience requires an infrastructure that scales to demands and delivers a quality of service expected by consumers. This is what a Content Distribution Network is all about and for the last decade, CDNs have been a requirement for doing business at scale on the internet.
Continual & Instantaneous Data Stream with Millisecond Timescale (or less)
Today, applications are publishing data about events at the moment of their creation. Data is published to subscribers and processed extremely fast making it possible to react to changing conditions in real time. This type of data is required for:
It’s simply impossible to realize these use cases with traditional “Near-Time” architecture. These use cases are about as real time as real-time can get; they enable swift reactions to a near-instantaneous stream of fast data before an opportunity is lost.
Next generation taxi companies such as Uber, Lyft and Get-Taxi have been using real-time data to their advantage and collectively are now starting to dwarf more traditional transportation models that have existed for literally hundreds of years.
In the financial sector, firms now make money by observing market fluctuations on timescales as short as a few milliseconds. Trading houses now require a continuous source of instantaneous market data with as close to zero latency as possible. This is a stream of data that represents the pulse of market conditions at any given moment. Trading houses typically position their offices so that this data stream travels the quickest path from the exchange to the quant’s financial algorithms. Every millisecond of latency saved is a competitive advantage and equates to an opportunity to react to earn more revenue.
Finally, auction houses require instantaneous data updates to ensure results are fair. Having the right end-to-end platform for delivering real-time capabilities like real-time notifications, in-app chat, alerts, and updates is key to online auction success.
Data Stream Analytics
Delivering this type of data at scale requires a special type of infrastructure. Although real time data stream delivery has a similar set of requirements to that of a traditional CDN: global scale and data centers, 99.999% reliability, and quality of experience it also comes with a set of unique requirements that include:
always-on data socket
tight coupling between the application logic and the network functions
support for today’s popular development languages, operating systems, and all types devices
End-to-end real-time communication platforms have been designed to address exactly this set of challenges for today’s real-time apps. As the world quickly embraces the benefits of real time, scalable, extensible platform solutions are becoming the de-facto choice to make these types of applications possible. It simply doesn’t make sense for developers to spend time and resources re-inventing the wheel when it is much faster, easier, and cheaper to outsource delivery to a proven solution.
Asynchronous Updates: milliseconds timescale and unpredictable in nature.
There is another type of dynamic real-time data that doesn’t necessarily produce a continual or periodic stream of data but also has the potential to be highly variable. In this case, the information may be updated at some point in time that is not predetermined but the response to the update needs to happen as quickly as possible.
Sensors that test whether a certain condition is true or not, for example, send a trigger the very moment an event registers. The world of the Internet Of Things is filled with all types of devices that require real-time signaling of this sort. Real-time communication platforms are the perfect fit for delivering asynchronous updates and notifications to IoT applications on millisecond timescales including:
Wearables
Embedded Devices
This video demonstrates a serious example: an airbag for motorcyclists worn within the rider’s jacket that is triggered remotely by a sensor on the motorcycle. The rider’s airbag takes 25ms to deploy once triggered, so the sensor needs to send the trigger event within 60ms of the motorcycle impacting an object for the rider to be sufficiently protected. There is very little room for error and the rider is relying on a real time data stream to prevent serious injury or death.
Real-TimeData Stream Examples
As you can see, the idea of what real-time is has changed over time. Looking back over centuries, we can see that our appreciation of up-to-date information has been colored by the age in which we live. Today, by necessity, a real-time data stream amounts to more information delivered faster than either you or I can physiologically grasp.
Real-time Data Streams can be used in all sorts of creative ways. Below are a few examples:
