Events to Measures – Scalable Analytics Calculations using PatternBuilders in the Cloud

By Terence Craig

One part of the secret sauce that enables PatternBuilders to provide more accessible and performant user experiences for both creators and consumers of streaming analytics models is its infrastructure. Our infrastructure makes it easy to combine rich search capabilities for a diverse set of standard analytics that can be used to create more complex streaming analytics models. This post will describe how we create those standard analytics that we call Measures.

In my last post about our architecture, we delved into how we used custom SignalReaders as the point of entry for data into Analytics PBI.  We’ve tightened up our nomenclature a bit since our last post, so it’s worth reviewing some of our definitions:

Nomenclature	Description
Feed	An external source of data to be analyzed.  These can include truly real-time feeds such as stock-tickers, the Twitter firehose, or batch feeds, such as CSV files converted to data streams.
Event	An external event within a Feed that analysis will be performed on. For example, a stock tick, RFID read, PBI performance event, tweet, etc.  AnalyticsPBI can support analysis on any type of event as long as it has one or more named numeric fields and a date.  An Event can have multiple Signals.
Signal	A single numeric data element within an Event, tagged with the metadata that accompanied the Event, plus any additional metadata (to use NSA parlance) applied by the FeedReader. For example, a stock tick would have Signals of Price and Volume among others.
Tag	A string representing a piece of metadata about an Event.  Tags are combined to form Indexes for both Events and Measures.
FeedReader (formerly SignalReader)	A service written by PatternBuilders, customers, or third parties to read particular Feed(s), convert the metadata to Tags, and potentially add metadata from other sources to create Events.  Simple examples include a CSV reader and a stock tick reader. An example of a more complex reader is the reader we have created for the University of Sydney project that filters the Twitter firehose for mentions of specific stock symbols and hyperlinks to major media articles and then creates an Event that includes a Signal derived from the sentiment scores of those linked articles.  That reader was discussed here.A FeedReader’s primary responsibility is to create and index an object that converts “raw data” received from one or more Feeds to an Event. To accomplish this it does the following: Captures an Event from a feed – stock ticker, RFID channel, the Twitter firehose, etc. Uses the Event itself and any appropriate external data to attach or enrich metadata and numeric data to the Event. Creates a MasterIndex from all of the metadata attached to the Event. This MasterIndex and the Date associated with this Event is used to create Measures and Models later on in the process.  It can also attach geo data if appropriate. Extracts the numeric Signals for that Event. Pushes the Event object onto a named queue – the “EventToBeCalculatedQueue”–for processing. This queue, like all PatternBuilders queues, has a pluggable implementation. It can be in memory (cheaper, and faster) or persistent (more costly and slightly slower). One of the great advantages of the various cloud services, including our reference platform Azure, is the availability of scalable, fast, reliable, persistent queues.
Measure	A basic calculation that is generated automatically by the PatternBuilders calculation service and persisted. Measures are useful in and of themselves but they are also used to dynamically  generate results for more complex streaming Analytic Models.

As the topic of this post is Events to Measures, let’s create a simple Measure and follow it thru the process. For this purpose, we’ll be working with a simplified StockFeedReader that will create a tick Event from a tick feed that includes two Signals – Volume and Price – for stock symbols on a minute-by-minute basis. The reader will enrich the Feed’s raw tick data with metadata about the company’s industries and locations. After enrichment, the JSON version of the event would look like this:

{
     "Feed": "SampleStockTicker",
     "FeedGranularity": "Minute",
     "EventDate": "Fri, 23 Aug 2013 09:13:32 GMT",
     "MasterIndex": "AcmeSoftware:FTSE:Services:Technology",
     "Locations":  [
          {
              "Americas Sales Office": {
                  "Lat": "40.65",
                  "Long": "73.94"
               }
          }
          {
               "Europe Sales Office": {
                  "Lat": "51.51",
                  "Long": "0.12"
               }
          }
      ],
      "Tags":  [
          {
              "Tag1": "AcmeSoftware",
              "Tag2": "Technology",
              "Tag3": "FTSE"
          }
       ],
       "Signals":  [
          {
               "Price": "20.00",
               "Volume": "10000"
          }
       ]
}

Note that there is a MasterIndex field that is a concatenation of all the Tags about the tick. When the MasterIndex is persisted, it is actually stored in a more space efficient format but we will use the canonical form of the index as shown above throughout this post for clarity.

A MasterIndex has two purposes in life:

1. To allow the user to easily find a Signal by searching for particular Tags.
2. To act as the seed for creating indexes for Measures and Models. These indexes, along with a date range, are all that is required to find any analytic calculations in the system.

Once an Event has been created by a FeedReader, the FeedReader uses an API call to place the Event on the EventToBeCalculatedQueue. Based on beta feedback, we’ve adopted a pluggable queuing strategy. So before we go any further, let’s take a quick detour and talk briefly about what that means.  Currently, PatternBuilders supports three types of queues for Events:

• A pure in-memory queue. This is ideal for customers that want the highest performance and the lowest cost and who are willing to redo calculations in the unlikely event of machine failure. To keep failure risk as low as possible, we actually replicate the queues on different machines and optionally, place those machines in different datacenters.
• Cloud-based queues. Currently, we use Azure ServiceBus Queues but there is no reason that we couldn’t also support other PaSS vendor’s queues as well. The nice thing about ServiceBus queues is that the latest update from Microsoft for Windows 2012 allows them to be used on-premise against Windows Server with the same code as for the cloud—giving our customers maximum deployment flexibility.
• AMPQ protocol. This allows our customers to host FeedReaders and Event queues completely on-premise while using our calculation engine.  When combined with encrypted Tags, this allows our customers to keep their secrets “secret” and still enjoy the benefits of a real-time cloud analytics infrastructure.

Once the Event is placed on the IndexRequestQueue, it will be picked up by the first available Indexing server which monitors that queue for new Events (all queues and Indexing servers can be scaled up or down dynamically). The indexing service is responsible for creating measure indexes from the Tags associated with the Event.  This is the most performance critical part of loading data so forgive our skimpiness on implementation details but we are going to let our competition design this one for themselves :-).  Let’s just say that conceptually the index service creates a text search searchable index for all non-alias tags and any associated geo data. Some tags are simply aliases for other Tags and do not need measures created for them. For example, the symbol AAPL is simply and alternative for Apple Computer, so creating an average volume metric for both APPL and Apple is pointless since they will always be the same. Being able to find that value by searching on APPL or Apple on the other hand is amazingly useful and is fully supported by the system.

More formally:

<Geek warning on>

The indexes produced by an Event will be:

where n equals the number of non-alias tags and the upper limit for k is equal to n.

</Geek warning off>

From our simple example above, we have the following Tags: AcmeSoftware, FTSE, Services, and Technology.  This trivial example will produce the following Indexes:

AcmeSoftware
FTSE
Services
Technology
AcmeSoftware:FTSE
AcmeSoftware:Services
AcmeSoftware:Technology
FTSE:Services
FTSE:Technology
Services:Technology
AcmeSoftware:FTSE:Services
AcmeSoftware:FTSE:Technology
AcmeSoftware:Services:Technology
FTSE:Services:Technology
AcmeSoftware:FTSE:Services:Technology

The indexing service can perform parallel index creation across multiples cores and/or machines if needed. As Indexes are created, they, and each Signal in the Event, are combined into a calculation request object and placed in the MeasureCalculationRequestQueue queue that is monitored by the Measure Calculation Service.

The analytics service will take each index and use it to create/update all of the standard measures (Sum, Count, Avg, Standard Deviation, Last, etc.) for each unique combination of index and the Measure’s native granularity for each Signal (Granularity management is complex and will be discussed in my next post).

Specifically, the Calculation Service will remove a calculation request object from the queue and perform the following steps for all Measures appropriate to the Signal:

1. Attempt to retrieve the Measure from either cache or persistent storage.
2. If not found, create the Measure for the appropriate Date and Signal.
3. Perform the associated calculation and update the Measure.

Graphically the whole process looks something like this:

The advantages of this approach are manifold.  First, it allows for very sophisticated search capabilities across Measures and Models.  Second, it allows deep parallelization for Measure calculation. This parallelization allows us to scale the system by creating more Indexing Services and Calculation Services with no risk of contention and it is this scalability which allows us to provide near real-time, streaming updates for all Measures and most Models.  Each Index, time, and measure combination is unique and can be calculated by separate threads or even separate machines. A measure can be aggregated up from its native granularity using a pyramid scheme if the user requests it (say by querying for an annual number from a measure whose Signal has a native granularity of a minute). A proprietary algorithm prevents double counting for the edge cases where Measures with different Indexes are calculated from the same Events.

So now you’ve seen how we get from a raw stream to a Measure.  And how, along the way, we’re able to enrich meta and numeric data to enable both richer search capabilities and easier computation of more complex analytics models.  Later on, we explore how searches are performed and models are developed—you will see how this enrichment process makes exploring and creating complex analytics models much easier than the first generation of  big data, business intelligence, or desktop analytics systems.

However, before we get there we need to talk about how PatternBuilders handles dates and Granularity in more detail.  At our core, we are optimized for time-series analytics and how we deal with time is a critical part of our infrastructure. This is why in my next post we will be doing a deep (ok medium deep) dive into how we handle pyramidal aggregation and the always slippery concepts of time and streaming data. Thanks for reading and as always comments are free and welcomed!

Enterprise Software in the Cloud: Why We Chose Azure as our First PaaS Platform

By Terence Craig

I’ve been absent from the blog too long, but if you’ve been following my colleagues (Mary and Marilyn) postings, you’ll see it’s been a very busy and fruitful time at PatternBuilders.  While I’m still overdue for the next segment of the architecture blog series, I thought I would take a break and talk a bit about some of the things we learned as we moved our product and business model to Microsoft Azure.

As someone who has worked with Microsoft technology and partnered with them off and on over the last two decades (even flirting with going to work for them a couple of times), the most surprising discovery was how serious Microsoft has become about the cloud, open source, and being an active and supportive partner for startups.  As many of you who have been around as long as I have will no doubt remember, this is a very different, some would say revolutionary, move for the world’s most powerful proprietary software company.  We had some concerns when we became members of Microsoft’s Azure Startup program BizSpark Plus and subsequently the more exclusive BizSpark One, but it has turned out to be a great experience for us on both the business and technical level.

AnalyticsPBI for Azure: Turning Real-Time Signals into Real-Time Analytics

By Terence Craig

For the second post on AnalyticsPBI for Azure (first one here), I thought I would give you some insight on what is required for a modern real-time analytics application and talk about the architecture and process that is used to bring data into AnalyticsPBI and create analytics from them. Then we will do a series of posts on retrieving data. This is a fairly technical post so if your eyes start to glaze over, you have been warned.

In a world that is quickly moving towards the Internet of Things, the need for real-time analysis of high velocity and high volume data has never been more pronounced. Real-time analytics (aka streaming analytics) is all about performing analytic calculations on signals extracted from a data stream as they arrive—for example, a stock tick, RFID read, location ping, blood pressure measurement, clickstream data from a game, etc. The one guaranteed component of any signal is time (the time it was measured and/or the time it was delivered).  So any real-time analytics package must make time and time aggregations first class citizens in their architecture. This time-centric approach provides a huge number of opportunities for performance optimizations. It amazes me that people still try to build real-time analytics products without taking advantage of them.

Until AnalyticsPBI, real-time analytics were only available if you built a huge infrastructure yourself (for example, Wal-Mart) or purchased a very expensive solution from a hardware-centric vendor (whose primary focus was serving the needs of the financial services industry). The reason that the current poster children for big data (in terms of marketing spend at least), the Hadoop vendors, are “just” starting their first forays into adding support for streaming data (see CloudEra’s Impala, for example) is that calculating analytics in real-time is very difficult to do. Period.

Big Data: Either Embrace It or Be Among the Walking Dead (Thoughts on Strata West 2012)

By Mary Ludloff

I’ve been meaning to blog about Strata West for the last week or so but felt the need to take a step back and look at the conference objectively. Of course, we’ve also been very busy at PatternBuilders working on our latest release (where correlation is the king and financial services is the queen or vice versa), engaging with potential partners and customers, and all the other activities that make up a startup’s life. In other words, during and after the conference we’ve barely been able to catch our collective breath (as well as get some much needed rest)!

So before I talk about the conference as a whole as well as some of the sessions and folks that caught my eye and of course, our book signing event (yes, Terence and I signed many books for conference attendees), I wanted to give a final shout out to our stellar Big Data and SCM panelists: Lora Cecere, Pervinder Johar, and Marilyn Craig. Thank you all for participating and for taking on this very broad topic! Much ground was covered, including the need for more rigorous cold chain management to ensure the efficacy of drugs, the amount of food that is spoiled and thrown away (one out of every three fruits and vegetables and two out of every five chickens) due to poor logistics management, and how big data can be used to transform the auto repair industry. What I loved about this panel (and yes, I am admittedly biased) was that it focused on real world problems that companies, industries, and societies are dealing with today. By the way, our panel was part of Strata Jumpstart—billed as the missing MBA for Big Data and it certainly lived up to its billing!

McKinsey Study: Location, Location, Location, Part 2

By Mary Ludloff

Greetings one and all and happy new year! As promised, part 2 of my post on McKinsey’s drill-down into the tremendous benefits location data offers to new businesses (and business models) as well as to all of us. If you need to refresh your memory (since the author was a wee bit late in meeting her stated publishing date), part 1 is available here.  Certainly, the report, “Big data: The next frontier for innovation, competition, and productivity,” is chock full of illuminating ways that big data can be leveraged within specific industries, but personal location data is a somewhat different beast as it cuts across  industries. For example, telecom, retail, and media (through location-based advertising) all stand to reap tremendous rewards.

Now, as I said in part 1 and will state again in part 2: I have a bit of angst around the collection and use of personal location data (see my many posts on privacy or our book on “Privacy and Big Data”). But that does not negate what can be gained if it is properly collected and used and with the appropriate regulations and guidance in place (my gosh—I am beginning to sound like one of the privacy policies I hate to read!). Put simply: all company’s data collection and usage policies should be clearly stated and always offered on an opt-in basis. Okay, privacy issues have been dealt with so let’s move on!

No, Hadoop Doesn’t Own Big Data Analytics!

By Terence Craig

A number of folks have asked me if I was concerned about Microsoft’s  recent announcement that they would be partnering with HortonWorks and abandoning their own distributed processing technology for Hadoop.  While I thought this was an unfortunate choice on Microsoft’s part (the Dryad project’s implementation of multi-server Linq was pretty compelling), since HPC is a small part of Microsoft’s business, it probably made sense from a business standpoint.   In any case, we (as in all of us at PatternBuilders) are not concerned and just to be clear: we don’t believe that this announcement (or any other) means that the many Hadoop ecosystem players own the still forming big data analytics market.

That is not to say that the announcement isn’t proof of the strength of the Hadoop ecosystem. Hadoop is a nifty technology that offers one of the best distributed batch processing frameworks available, although there are other very good ones that don’t get nearly as much press, including Condor and Globus.  All of these systems fit broadly into the High Performance, Parallel, or Grid computing categories and all have been or are currently used to perform analytics on large data sets (as well as other types of problems that can benefit from bringing the power of multiple computers to bear on a problem). The SETI project is probably the most well know (and IMHO, the coolest) application of these technologies outside of that little company in Mountain View indexing the Internet.

McKinsey Study: Location, Location, Location, Part 1

By Mary Ludloff

Yes, it’s that time again: a deep drill-down into a specific big data area, courtesy of McKinsey’s voluminous report on “Big data: The next frontier for innovation, competition, and productivity.” You may be wondering about the five month delay since my last foray into this particular study but, well, we have been just a bit busy with our book (Privacy and Big Data) while working on some very cool features for our Analytics Platform as well as handling all our other PatternBuilders responsibilities!

I also must confess to a bit of angst regarding location data, especially when it pertains to where we are located as opposed to where things (like shipping boxes) are located. From a privacy standpoint, this is a rather large (okay, huge) area of concern but it’s not the data itself that we should be worried about. As in most things surrounding the privacy debate, it is how the myriad of companies, organizations, and government agencies collect and use our personal location information without our knowledge or consent that we should be worried about.

Video: Big Data Made Easy. Sticky – see below for latest posts.

Privacy and Big Data: Conferences, Interviews, Webcasts, and a New Resource Page!

By Mary Ludloff

Greetings one and all! Terence and I have been very busy with our day, night, and weekend jobs! Naturally, in the midst of QA’ing the latest release of PatternBuilders Analytics Platform, our book commitments have also been on the rise with interviews, a podcast, and some upcoming conferences that we will be speaking at. The past few weeks have been a wee bit crazy for us, but also a lot of fun. Our book certainly has sparked some lively conversations:

• The rise of data collection by some known, and many unknown parties, as well the selling of that data to third parties for unknown (there’s that word again) uses.
• The role of government, regulation, and policies in the digital world we all inhabit.
• The various constituencies and their privacy agendas.
• How privacy could become a competitive advantage for those companies who are transparent about the collection and various uses of our data.

Roundup: About 4 Tech Giants, All Things Private, Social Media Stats, Maps, and Big Data!

By Mary Ludloff

Greetings one and all! It’s been a while since I posted about the more interesting articles, blogs, videos, etc., that I have come across and I thought that now is as good a time as ever to cover some interesting items you may have missed in the past few weeks. The topics are far ranging, thoughtful, illuminating, and at times, contentious, but that’s why they are interesting. So without further ado, let’s get to it!

Four Tech Giants Battle It Out

If you haven’t already, set aside some time to read Fast Company’s take on the (coming soon) great tech war of 2012. The combatants? Apple, Facebook, Google and, Amazon. The prize? Us—I think! This thoughtful piece by Farhad Manjoo looks at how these four goliaths will battle it out on the technology innovation field to, essentially, win the hearts, minds, and wallets of all of us:

“Think of this: You have a family desktop computer, but you probably don’t have a family Kindle. E-books are tied to a single Amazon account and can be read by one person at a time. The same for phones and apps. For the Fab Four, this is a beautiful thing because it means that everything done on your phone, tablet, or e-reader can be associated with you. Your likes, dislikes, and preferences feed new products and creative ways to market them to you. Collectively, the Fab Four have all registered credit-card info on a vast cross-section of Americans. They collect payments (Apple through iTunes, Google with Checkout, Amazon with Amazon Payments, Facebook with in-house credits). Both Google and Amazon recently launched Groupon-like daily-deals services, and Facebook is pursuing deals through its check-in service (after publicly retreating from its own offers product).”