Digesting Big Data

Big Data holds a big promise. But has that promise paid out already? Or are you heading for Big Dollar Disaster? Many take inventory of their data and find out they have terabytes of data lying around. Surely something should be done with that, so here’s how we see a lot of companies going about implementing ‘something’ for their Big Data.

• Buy hardware, set up a cluster and install some flavor of Hadoop.

• Break their brains to figure out how to run map-reduce jobs and compute some analytics.

Until they find out that the size of meaningful data within those terabytes is considerably smaller and there are other, cheaper alternatives available.

Don’t get me wrong: it is great to work on exciting new technologies and get your hands dirty on innovative concepts. But is it delivering business value?

Digestive system

We like to compare working with (Big) Data to a digestive system. There are several stages the data flows through and some of these are getting too much attention while others need more.

Phase One: Ingestion.
The art of collecting data and storing it.

Phase Two: Digestion.
Digestion is processing your raw data into something that you can extract value from.

Phase Three: Absorption.
This stage is all about extracting insights from your data.

Phase Four: Assimilation.
In the fourth stage you want to put the insights to action.

Phase Five: Egestion.
This fifth phase runs parallel to all others and is about getting rid of the unwanted, unclean, unnecessary parts of your data, invalid insights and predictions at every step of the process.

Right focus

A lot of ‘Big Data’ attention is given to the first two stages: Ingestion and Digestion. This is where all the Hadoop, Elastic Cloud computing, Map Reduce talk is concentrated. A lot of VC money (round about $1 Billion!) is competing for a piece of that cake. But what you really need is a quick way to check if you can assimilate new insights and create the value that your business needs. That is the exciting part! So you really want to focus on Absorption and Assimilation, to taste, test and prove. Focus on the business value and work your way back to technology. How? In one of our next posts we’ll guide you through some sensible steps to get started with your (big) data.

At BigML we take pride in building a smart, simple-to-use platform so you can create new and actionable insights in a few clicks. And it doesn’t require your company’s capital to invest in cutting edge technology. We’ve done that for you. We will enable you to find new insights and put them to action as soon as possible, maybe in a small corner of your business, to make sure you are doing the right things. From there you can expand, discover, make mistakes and find where the value is.

Hyped

Let’s say the word: we think capturing big data and big data processing is hyped. It’s great technology, sure. But it needs to serve a purpose: creating new insights out of big data and putting them to action. Let’s focus on that and the rest will follow as and when you actually need it.