Why isn’t BI a Solution for Big Data and Real-Time Analytics?

By Terence Craig

It’s a legitimate question that we get asked a lot: why can’t I use my multi-billion dollar BI system to manage my big data/real time analytics problem(s)?  I have found that my tongue in cheek answer “we need you buy our software and services because we have families to feed” though true, is not as compelling to customers as I would like which leads to today’s post.

To put the question another way: what can PatternBuilders and the rest of the new approaches to data and analytics like Hadoop or Mongo (which, by the way, our platform uses and is a great technology) offer you over and above what large BI company X offers?

Here’s the short answer. (Trust me, I could write a PhD thesis on the long answer.)

The above technologies (including ours) where created to fill a huge void in software solutions that focused on Big Data and the real-time analytics built on top of that data.  These are problems that traditional desktop analytics systems (like SAS/Excel) or their big brothers, Business Intelligence and Data Warehouse solutions (too many out there to name and far too confusing due to all the renaming going on because of acquisitions), do not solve very well, if at all.

Organizations of all types are drowning in data. The automated transactional systems that run the typical large enterprise, whether it’s manufacturing operations, POS Systems, ERP, Clinical Operations systems, etc., produce a tremendous amount of data. And as more processes become automated and Internet collaboration between organizations grows, this data will continue to grow exponentially, outstripping the traditional database, BI, and analytic architectures. That’s because all of these systems were designed for much smaller problems in a very different era.  Back when:

• Jimmy Carter was President.
• Sun was an independent company.
• A gigabyte of data was a lot.
• A gigabyte of memory was impossible.
• A low-end personal computer cost $6,000.
• Multi-threading was a research project.
• SSD was better known as an entitlement program rather than a paradigm destroying storage technology.

At that time, technologies and methodologies we now take for granted today did not even exist, such as:

• The Web
• XML
• Java
• .NET
• Domain-Driven Design (A concept where software development projects are driven by the domain and domain logic—for example, interfaces should be designed using the language and processes of the business user.)

To put it more succinctly, the original Data Warehouse designers had a lot more constraints to deal with in terms of computing power and a lot less data and users.  In fact, the idea of designing a system that would provide real-time analysis of data that comes at rates measured in Terabytes/Day would have seemed like science fiction, as would the idea that they would have to create a front-end that could be distributed securely world-wide to thousands of users across the globe simultaneously.

Fulfilling these requirements cannot be done by tinkering with 30 year old technologies.  It requires starting from scratch.  This is what we are doing at PatternBuilders along with a lot of other talented teams and companies out there.  We are on the cusp of understanding what can be accomplished by taming large data, but it is going to be an exciting journey that has already created some great technologies and will lead to the creation of some great companies as well.