It’s All About the Data

By Terence Craig

Welcome.

A few years ago, we founded PatternBuilders after noticing that those of us in the software business had won the war and had convinced the entire world that capturing every possible transaction in a digital format and automating all their processes was not only a good idea, but a necessary survival skill.  At the same time, however, we almost universally failed in providing ways to do any real analysis on the tons of digital data that were now being captured and available to be used.

Just a bit of technology-based historical perspective for you. Sure there were BI systems available to perform analysis, but while sales where great, actual usage was tiny. Worse yet, the analysis delivered was pretty much limited to sums and averages at an aggregation level so high as to be virtually useless.  Real analysis was, and remained, the province of Wall Street firms. In other organizations, companies, and industries, analytics typically meant harassed individual contributors trying to shoe horn huge amounts of data into Excel.  (If you want some fun, try creating a forecast for a multi-billion dollar business with only 65K rows, or 1MM if you’re working with Office 2010.) Or if you had lots of budget to churn through, analysis seemed like black magic performed by SAS in a back room filled with PhD’s.  Those PhDs, by the way, had as difficult a time talking to most business folks as the IT guys had making those same folks happy by taking weeks to change and reload sales cubes. And if the business users needed something sooner, well, they could always “learn” SQL to figure out the numbers they needed for the operations review tomorrow.   And this was back when a gigabyte was considered to be a lot of data.

Today, we have moved faster than anybody could have imagined, from a gigabyte neighborhood to a petabyte world. In a couple of years, exabyte will be as common a term as gigabytes are today.

What does all of this mean? Well, it provides amazing new problems for software vendors to solve and boundless opportunities for organizations of all types to improve their processes and react faster to an increasingly real-time world.  For non-profit organizations, real-time analysis of Big Data will enable new ways to better serve their constituencies for things as diverse as the disruption of terrorist networks and the prediction of manhole explosions.  On the commercial side, the variety of applications is stunning. The ability to affordably and quickly crunch huge datasets and provide real time notifications opens a whole new world of applications and new business models. It also brings up a host of legal, moral, and privacy issues.

It is these applications, the technology required to deliver them, and the legal and moral issues that surround them, that we will be discussing.  We look forward to the conversation and your feedback.

We will also make occasional forays into joys and frustrations of building a company but the blog’s main focus will reflect ours—it’s all about the data baby!