All Together Now: All You Need is a Text Box!

By Terence Craig

All you need is text, Text is all you need (sing to the tune of The Beatles’ All you need is love).   If you are one of our regular readers you will remember that several months ago I wrote a manifesto on what the perfect analytics system would look like.  One of the last points was:

It must be as accessible as Excel (still the number one analytics tool in the world).

I was wrong – Excel is the number one non-specialized analytics tool in the world but in terms of usage, it is dwarfed in comparison to a very well know specialized analytics toolkit. The creators of this tool are a little company that you may have heard of:  it does no evil and analyzes the Internet to bring you back everything on the web based on a simple text query.  But behind that simple text box, Google has one of the most sophisticated analytics infrastructures in the world:

• It can deduce your interests.
• Give you the most relevant results.
• And show you appropriate information based on them, as well as bring back highly personalized ads.

Google is not only the largest big data analytics company in the world, but it also has the easiest to use tools—proof that text is all you really need!

“Cool,” you say, “But what does that have to do with PatternBuilders?  You guys are not a specialized Internet search engine—you are Big Data Analytics system that serves specialized domains so you can’t be as simple as Google.” We would have agreed with this a couple of months ago but like most crossing the chasm moments, technology took us down a different and better path.

We realized that there were huge performance benefits in going from a hybrid relational/Mongo approach to a pure Mongo approach (described here). So we took a step back and started thinking about all the cool things we could do.  We had our unique indexing structure, linearly scalable performance from both our analytics and query engine, and a blazing fast document database. What could we do with all of that power?   Why not try and do something different?  Like say, bringing the Google level ease of use to Big Data Analytics?  It wasn’t easy but we pulled it off.   Oh, and in case you were wondering here is how we did on the rest of the manifesto requirements:

• Does its UI allow non-statisticians to easily query/explore statistics created by others?
• How much hardware do you need for your required performance? Does it do fanout?   We are very hardware efficient – Currently processing real time stock ticker data with a few commodity servers and our performance increases linearly as you add servers.
• Does the system come with pre-built common metrics for the customers particular industries – for example, GEMROI in retail or the Joint Commission statistics for hospitals? Yes, and it is very easy to create new ones in any .NET language.
• How easy is it to create new analyses and how long does it take get results from this new analysis?  Creation is easy (see above) and since our entire back end is streaming, once a new metric is added results become available instantly (see this post for details).
• Is it secure? Highly – but this is something we constantly work on.
• Does it have a flexible deployment model (cloud or on premise)? We get an A+ on this one – not only do we support on-premise and cloud deployments (EC2, Rackspace), but we also support a hybrid model as well. Our hybrid model allows key data like patient info or click-thru statistics to be kept on premise by the customer and encrypted analytic results to be kept in the cloud.
• Does its user interface provide useable performance on the web? We use Silverlight for our UI layer, except in our new mobile app (coming soon) and despite Microsoft’s ham handed marketing treatment of SL in their drive to make Windows 8 an iPad killer, we still believe in it as a platform and have used it to create a UI that is performant, great looking, and cross browser portable!
• Can its scripting language be configured to become an analytics DSL in the user’s problem space? Yes – but we are working on improving the scripting language to make it more type safe.  We want to move from constructs like KPI.Name.GetResult(“Avg Sales Price”) to constructs like AvgSalesPrice.GetResult().
• Can it easily absorb data from different sources and different formats?   Yes we been having a lot of fun in the beta mashing things up left and right.   Our latest demo is a mash up of a live stock market feed, Twitter Streams, and RSS daily Weather data.
• Is it easy to keep up and running?  Yes but we are working on making it even easier. Our new server admin GUI is a great start.

In all modesty, we knocked the ball out of the park here.  The release is in final test and should go GA in early November.  Blog at you next week!