Posts tagged ‘real-time analysis’

AnalyticsPBI for Azure: Turning Real-Time Signals into Real-Time Analytics

By Terence Craig

PBI 3 0 archslide 3For the second post on AnalyticsPBI for Azure (first one here), I thought I would give you some insight on what is required for a modern real-time analytics application and talk about the architecture and process that is used to bring data into AnalyticsPBI and create analytics from them. Then we will do a series of posts on retrieving data. This is a fairly technical post so if your eyes start to glaze over, you have been warned.

In a world that is quickly moving towards the Internet of Things, the need for real-time analysis of high velocity and high volume data has never been more pronounced. Real-time analytics (aka streaming analytics) is all about performing analytic calculations on signals extracted from a data stream as they arrive—for example, a stock tick, RFID read, location ping, blood pressure measurement, clickstream data from a game, etc. The one guaranteed component of any signal is time (the time it was measured and/or the time it was delivered).  So any real-time analytics package must make time and time aggregations first class citizens in their architecture. This time-centric approach provides a huge number of opportunities for performance optimizations. It amazes me that people still try to build real-time analytics products without taking advantage of them.

Until AnalyticsPBI, real-time analytics were only available if you built a huge infrastructure yourself (for example, Wal-Mart) or purchased a very expensive solution from a hardware-centric vendor (whose primary focus was serving the needs of the financial services industry). The reason that the current poster children for big data (in terms of marketing spend at least), the Hadoop vendors, are “just” starting their first forays into adding support for streaming data (see CloudEra’s Impala, for example) is that calculating analytics in real-time is very difficult to do. Period.

(more…)

December 12, 2012 at 5:22 pm 8 comments

Speaking on Inman Connect Panel on Real Estate and Big Data

By Terence Craig

I apologize for falling behind on blogging, but between several new hires,  major partnerships, and the industry finally starting to understand the need for product-driven (instead of project-driven) big data, things have been very hectic. Good, but hectic.

I did want to pull my head off my keyboard for a minute to tell you about participating in the big data & real estate panel this Thursday at Connect San Francisco.  Our panel will be moderated by industry luminary Brad Inman @bradInman.

Real estate has always been a data-driven business and is relying more and more on the insights and operational nimbleness provided by big data.  For those of you who are scratching your heads and going, “Huh, Real Estate and big data?” – think about it for a minute.  The real estate industry is “using” big data to do all kinds of things and drive all kinds of business models, such as:

  • Commercial landlords using smart thermostats and smart windows adjusted in real-time to save energy.
  • Capturing real-time parking meter data to make real-time decisions about how long to leave a retail location open.
  • Using real-time video analysis to stop vandalism before it happens.
  • Offering sophisticated analytics – see consumer facing sites like Truila and Zillow.
  • Risk Modeling – check out RMS. Like most of the PatternBuilders team, they were “doing” Big Data before the term was invented.

If you are attending the show, stop by and say hi. If you are interested in Big Data & Real Estate, look for our post-Connect blog next week. In it, we will talk about some great insights about the New York real estate market derived from a ton of data we grabbed from the NYC public data market which was then spun up in the PatternBuilders framework on our brand spanking new Microsoft Azure cloud beta release.

August 1, 2012 at 9:37 pm Leave a comment

Big Data and Cloud not a fit? Comments on Infoworld Article

By Terence Craig

Since Disqus seems to have completely eaten (bleh) my comment on @davidlinthicum’s very interesting InfoWorld post – Big data and the cloud: A far from perfect fit, I decided to just expand my comments and make a short blog post out of it. IMHO the problems that David is describing are more a reflection of problems with batch oriented technologies like Hadoop (more on my take on Hadoop here) in the cloud than a general problem for cloud based big data solutions.

Computing always has, and probably always will have, a bias towards creating batch focused technologies at the beginning of any large paradigm shift.   But as new technologies are absorbed, understood, and move from early adopter to more mainstream use, the batch paradigm will inevitably start to shift to streaming and real-time. We have seen this again and again (from punch cards to touch sensitive tablets, downloaded media to streaming media, DOM to SAX parsers, HTML to Ajax, paper maps to real-time GPS). The reason this evolution almost always occurs is simple: humans live and think in real-time and when our tools do as well we are more productive and happier.  So why do we have this bias for batch processing in our first generation computational technologies? Simply put, because batch processing is a lot easier.

(more…)

February 23, 2012 at 3:01 pm Leave a comment

FinancePBI Begins its Shakeout Flight in the Cloud

By Terence Craig

I have been a little quiet on the blogging front recently as I and the rest of the PatternBuilders team have been focused on getting ready to launch our new financial services application: FinancePBI. It is the first cloud-based analytical platform for the Financial Services market.  While this is our first public announcement of our entry into the market, behind the scenes we have been gearing the company up for a big splash for several months:

  • Partnered with ActiveFinancial one of the premier real-time stock ticker vendors in the world.  Look for more data partnerships shortly.
  • We have added Doug Jeffrey to our board of advisors and board of directors.  Doug is an executive with deep Wall Street and startup expertise who has already done outstanding things in the short time he has been with us.
  • We have also partnered with the University of Sydney to use our technology to examine the influence of primary sources (NY Times, etc.) and secondary social media (Twitter, etc.) content on a company’s stock price over a 12 month period. This project will be done exclusively in the cloud and it’s our hope is that we will be able to convince our commercial partners to allow this PatternBuilders instance to be available to the general public. Of course, this would happen after the research is published. (more…)

February 7, 2012 at 8:04 pm 1 comment

No, Hadoop Doesn’t Own Big Data Analytics!

By Terence Craig

A number of folks have asked me if I was concerned about Microsoft’s  recent announcement that they would be partnering with HortonWorks and abandoning their own distributed processing technology for Hadoop.  While I thought this was an unfortunate choice on Microsoft’s part (the Dryad project’s implementation of multi-server Linq was pretty compelling), since HPC is a small part of Microsoft’s business, it probably made sense from a business standpoint.   In any case, we (as in all of us at PatternBuilders) are not concerned and just to be clear: we don’t believe that this announcement (or any other) means that the many Hadoop ecosystem players own the still forming big data analytics market.

That is not to say that the announcement isn’t proof of the strength of the Hadoop ecosystem. Hadoop is a nifty technology that offers one of the best distributed batch processing frameworks available, although there are other very good ones that don’t get nearly as much press, including Condor and Globus.  All of these systems fit broadly into the High Performance, Parallel, or Grid computing categories and all have been or are currently used to perform analytics on large data sets (as well as other types of problems that can benefit from bringing the power of multiple computers to bear on a problem). The SETI project is probably the most well know (and IMHO, the coolest) application of these technologies outside of that little company in Mountain View indexing the Internet. (more…)

December 12, 2011 at 1:41 pm 3 comments

Real-time Analytics: It’s Always Decision Time!

By Mary Ludloff

Greetings all! I just came across a great video from eWEEK talking about the growing need for real-time (aka streaming) analytics:

“For years, business intelligence has provided valuable information to help executives and managers make decisions to increase sales, improve operations, and seize new business opportunities. With the quickening pace of business today and the need to make faster decisions based on more timely data, companies are complementing this data using information mined from social networks, mobile sensors, and even location-based information from smartphones. To get the best value from this wealth of new data sources, the data analysis must be done in real time. This allows decisions to be made based on the true conditions at that particular time.”

(more…)

September 23, 2011 at 12:04 pm 4 comments

Thoughts on Identity Theft, Government IT, Facebook and Google Plus

By Mary Ludloff

Before I begin, I must admit to being in a very grumpy mood which may (ever so slightly) color the topics in the post title. As you know, Terence and I are not only “work-working” but in our copious amounts of spare time, working on our book (plug fully, absolutely intended). While the book is going well, I must say that it has affected my Zen-like ability to remain calm (I can hear Terence chortling) when others may “lose it.” This leads me (you’ll understand in a minute) to the issue of identity theft. (more…)

July 20, 2011 at 6:58 am Leave a comment

Social Media Initiatives: Marketing Fundamentals are the Same, Vehicles are Different

By Mary Ludloff

Whenever you throw a bunch of marketers in a room and say, “let’s talk about doing some social media outreach,” I can guarantee you that there will be an energetic (yelling and gesticulating) conversation with some eye rolling involved. Whenever you throw a bunch of social media experts in a room and say, “Company A does not see the value of their social media programs,” I can guarantee you that there will be an energetic (yelling and gesticulating) conversation with some eye rolling involved. Whenever you throw a bunch of software engineers in a room who have come specifically to talk with you about what you would like to see in a social media analytics tool, I can guarantee you that there will be an energetic (me yelling and gesticulating) conversation with some eye rolling (engineering and product management team) involved.

Why all the storm and dirge? Well, social media has moved from an “interesting fad” to an entrenched channel but the jury is still out on how we marketers are going to “measure success.” A recent Harvard Business Review social media study found the following:

“… nearly two thirds of the participating organizations said that they are either currently using social media channels, or have plans in the works. However, only 12 percent of the companies surveyed said they felt they were effective users of social media. These were the companies most likely to use multiple channels; use metrics for reporting; have a strategy for social media usage; and integrate social media into their overall marketing strategy.” (more…)

May 3, 2011 at 7:15 pm 1 comment

Riding the Data Waterfall

Our new streaming analytics engine.

By Terence Craig

As promised, I am going to spend the next few posts discussing some of the new features in our analytics framework, otherwise known as PAF.  This is our largest and most complex release so far.  We are very proud of it—both in how far the framework has come and how closely it matches our vision of what a world class analytics system would look like when we started the company a few years ago.

One of my favorite features, and certainly the biggest change in this release, is that our analytics engine is now completely streaming based.  I think that this, along with our improved ad-hoc analysis support, is going to improve our customers’ day-to-day to experience with both calculating and using analytics in their businesses.    (more…)

January 26, 2011 at 6:01 pm 11 comments

Speaking at Strata’s “Making Data Work” Conference

By Terence Craig

I am excited to announce that I will be speaking at Strata’s big data conference this year. As I mentioned in a previous post, Big Data and Big Analytics has really gained momentum over the last year.  Of course, having a major conference sponsored by the venerable O’Reilly organization is a great sign that the techniques, services, and technologies that make up the Big Data ecosystem are becoming mainstream.

My co-presenter is retail analytics guru extraordinaire Marilyn Craig, who currently runs the World Wide Analytics group for peripheral manufacturer Logitech.  Marilyn was the chief analytics officer at PatternBuilders for several years and made sure that we kept focused on the analytical needs of business users as opposed to techno-fetishism.  It is a philosophy we maintain to this day and our product is better for it. (more…)

January 5, 2011 at 5:57 pm Leave a comment

Older Posts


Video: Big Data Made Easy

PatternBuilders Corporate

Special privacy section!

Previous Posts


%d bloggers like this: