Tuesday, March 4, 2014

Big Data and all the hype behind it

Big Data – the biggest big deal created after internet and mobile, and the year 2014 – the year when many companies will be reaping the benefits of high investment they did in this data analytics in 2013. Big data has become like a basic necessity for most of the companies like LinkedIn, Google and Facebook these days but it is not only these big brand companies that are utilizing it. Recently I read an article about how even a used car auction company, Manheim is using big data to decide on its selling price and the target customers. It really amazes me how big data has become a mandatory investment in some of the industries just like setting up a telephone connection or creating a website.
What are the reasons that led to such massive data generation? The simple reasons – technology. With the easy access to internet people have readily available facts about almost everything. There are people who use this knowledge and hence there are people who are producing and making this knowledge available. Everything can be now used to generate data with the onset of Internet of Things, cloud computing and mobile phones. Past 2 years has been like tsunami of data generation. In fact, the world generated 90% of the total world’s data in last 2 years and is estimated to produce 300 times more data by 2020 than we had in 2005
What is the impact of so much data lying around? Data scientists around the world have already started using this data in lot of money generating and useful applications. However, there is lot more to come. If we go by the trend right now, we can say that big data will become the inseparable part of most industry operations. As more and more companies will see benefits in utilizing the data, more technologies will be developed to harness the data. However, the important point in future about big data will not be just to utilize it, it will be more about making sense out of it. The data is not just about the volume, variety and velocity with which it is produced, but about the value that it can create.
How big data can be used? It can be used almost anywhere and everywhere provided we know how.
 Manufacturing industries can use it to determine the best suppliers, improve production efficiencies and understand the customer demand
 Advertisement agencies can use it to design a more effective ad campaign, select the right time slots on television or right magazines and targeting right customers
 Pharmaceutical companies can use it to determine their key market, improve efficiency of R & D and testing processes and selecting right customers (Of course!)
 Nike can use it to stock in right clothes/shoes in the shop and while deciding on the discount offers to give customers
 Telecom industry can use data to decide on the data packs, call rates and deals, and to extract all the buyer surplus while giving out the offers to customers
These are only few examples of big industries but big data can also be used by small industries like a bakery, grocery store or a salon. Big data is definitely a key to increasing competitiveness and productivity. However, we also know that ‘If it was that easy, everybody would have done it’. It’s not that easy obviously. The data is massive (as I said earlier) but we need to know firstly, the exact question that we want to figure out  the right subset of the data set that need to be analyzed to get that
answer  the value that will be generated from that answer  the money that we need to invest to conduct the analysis  Is all that investment > the value generated from the data.
But excess of everything is bad! So what are the drawbacks of this massive datafication? The answer is “Many Social and Privacy threats”. Big data means large data and when this data is collected people often do not know what means are used and how this information will be used. For example, a collected at a super market can be further used by store manager to know how much time you spend in a super market. This information can be further used by the online stores or by the thieves. You can never know. Big data can also increase social stratification in society as it gives more means of knowing the information to the richer section as opposed to poor section who do not have access to all the information. As we say in economics – In case of asymmetric information, the one with the higher information is always benefited. Further the datafication is leading to increased cybercrime. Recently a smart refrigerator that could automatically connect to internet was hacked and used to send lot of spam messages. This is just an example but if we keep on connecting everything to the internet, just to satisfy our urge to create more and more data every second, we could lead ourselves more harm than good in future.
Apart from social and privacy impacts, the data also have some technical drawbacks. Firstly, the data is massive so lot has to be invested in assimilating and storing it. Secondly, the current analysis techniques like simulation and optimization can often produce misleading results with the big data. For example, there are lot of parameters in the data and sometimes normal simulation techniques can lead to false correlation between parameters. Hence, the companies can be at high risk of taking the wrong decision based on the data. Before utilizing the data the need is to understand the analysis methods and to take the decision only after weighing it with the actual business scenarios and not just on the numbers.
In conclusion big data is a source of all hidden and non-hidden information. Information leads to good and better decision making in any given constraints, better decisions lead to best results, be it related to world, business, politics and social life, poverty eradication, selling or buying.