Big data is a rapidly growing trend that cannot be ignored. From IBM, we know that we create 2.5 quintillion bytes of data every day, which is equivalent to 57.5 billion 32 GB iPads! To top it off, 90 per cent of the data in the world today has been created in the last two years alone(1).

With the mass adoption of social media and the rise of tracking technologies, we are no longer able to ignore the untapped potential of how brands can leverage these large amounts of data. There are obviously opportunities as we can see from the mashups(2) of different applications and Internet services to make life more effective and efficient for all of us.

Uncovering multiple data points can be painfully messy

Big data and new technologies pave the way forward for us as marketers, but before we jump onto the bandwagon of the big data trend, I would like to propose that we take a step back and look at our existing data. At XM, the analytics team has analysed and seen the common mistake that many organisations repeat when preparing report. This stems from trying to combine multiple data points from individual silos to make the best sense of the data. A common example in our field would be for teams to look at email marketing data, search engine marketing campaigns data, CRM data and web, app and mobile data. Looking at the plethora of data points, someone (a team hopefully) is then tasked to combine all the data together (oftentimes in excel) and try to make sense of what’s happening to their bottom line. Multiple data points have become a huge pain point for many because it is difficult and time consuming.

This is however, necessary and important because it allows us to form the big picture and see how each data point affect and influence another, and what impact they bring collectively to impacting the bottom-line, as well as other milestones we set and measure. We will be able to better segment our target audience and formulate a more focused marketing message based on the data. We can even go into one to one, relationship marketing if we can string all the data together to form a one-customer view; this is the holy grail of direct online marketing. In this day and age, it is possible to do all of the above. After all, if Target can understand that a girl is pregnant before her Dad does, we should all strive to be this laser focused on our data.

How do we automate and simplify data collation from multiple disparate sources so that we can focus on adding value by deriving insights from unified data, rather than spending precious time combining data on a weekly or monthly basis? At XM, we do this through coding via libraries of APIs for our clients and some of these are made readily available by our vendors. If you are on Adobe SiteCatalyst, an easy way to do this will be through the use of Adobe Genesis where you will be provided fuss-free integration without any technical coding required.

Whichever way you choose, integrating data and forming the big picture is no longer a choice, but a necessity. Collating and combining data effectively and efficiently forms the basis of quick data-driven decision-making.

Let’s look within before looking out.

1. Bringing big data to enterprises
2. Mashups 

Wouldn’t it be great if we can one day predict the brand of car our customers will buy when they are at their peak, what brand of cornflakes they would eat every Sunday morning, when they will watch another 2012/Independence Day-type movie, right down to what gender of baby they are likely to have when they marry? “Predicting the unpredictable, quantifying the unquantifiable”, says Dogbert the guru*.  But before we are able to fully achieve that in the twilight years, lets see how we can predict the future with the pre-historical method of predictive analytics, summarized in eight easy steps (inspired by Dogbert).

1.) Use a previous campaign or a test as your base.

Say for example you need a conversion of 1000 respondents, and the industry/benchmark average is 2%, then we probably need a base of 50,000 to achieve your target or do an adequate test.

2.) Add geographic, demographic, psychographic & behavioural data to your base

Both the responders and the non-responders. Who wants to be known as Segment A when we can be described more adequately and be given a nice personality.

3.) Know what are all the possible ways (and the most effective way) of reaching your base

We do have mobile phones and a life in the social space.

4.) Test and Control

Divide your data into at least 2 groups everytime, equally. Test group carries the hypothesis, control group validates.

5.) Trash the anomalies

Don’t you hate it when you thought one of your content pillar is doing extremely well in terms of average time spent and when you deep dive into the data you realise that it’s some idiot who left the computer on with your website in the browser running throughout the lonely night? Discard the outliers, be it purchase patterns or web patterns.

6.) Design your modeling framework

We will usually start out with a typical multiple regression model before we move on the the more sophisticated models like CHAID etc. “A regression is an equation that describes the relationship between a dependent variable and more than one independant variable”.  Statistical definition of dependant variable connotes the action/consequence that will be influenced by how you set up the test environment, and indenpendant variable as the components that will not be influenced (e.g. demographic, geographic, behavioural data).

7.) Grade and weigh each variable and develop an algorithm

This step will help you in shortlisting the most important variables that will influence your results (e.g. income, age, family size, etc). One will probably need a PHD in statistics and a SAS or SPSS software to run a regression model and weigh the different variables, and to finally develop an algorithm ranking the deciles.

8.) Score the validation group

If the predicted results derived from the algorithm is a close match to your previous campaign/test, then  the algorithm developed will be useful to predict future campaigns. It should also help you identify the target segment more likely to respond to your campaign through the scoring exercise.

If all else fails, hire a consultant to do the work while you make yourself another cuppa! (*highly recommended)

Last but not least, meet Dogbert, the data guru.