Big Data


Have you ever wondered about human history and our existence on this earth? Sounds like a heavy discussion, as it’s not just about a few years, but many known and unknown million years and now 2021 also becomes a part of the same. While the earliest traces of human existence go back to six million years, we evolved to become humans only about 2,00,000 years ago. More interestingly, the civilized form of our living began only 6,000 years ago. The post-industrialization era is just 200 years old; the democratic world order has been established only within the last 100 years, and the modern digital world after the rise of the internet is merely into its 37th year. In the context of human history, our current living, system and culture are insignificant when compared to the several million years before this day, however, from the perspective of human progress and development, it is undisputed that the past couple of centuries weighs a lot more than the thousand centuries before. Do you know what made the difference? – the modern art of ‘recording’.

Think of taking up an incomplete task – if you don’t know what has happened so far, you will have no option but to restart from scratch. However, if you have notes of the person previously working on the same, you can do much a faster job, as you already have a platform, to begin with. The art of recording our history has evolved exponentially over the years – from writings on the stones and walls once upon a time, we developed papyrus sheets and later, its modern form – the paper. However, the same is now being replaced by computing devices, in this new era that we refer to as the digital age. Data is being recorded everywhere – from social media posts to the income tax returns, the equity markets, the land records, the gaming and sports scores, the store purchases or the online orders, a database is being maintained by everyone. And it is not just words, as technology has enabled us to record even pictures, voices, videos, maps, charts, biometrics and even the genome sequencing of the DNA strands inside the cells of organisms – everything in ones and zeros!

Recording data is like placing a bookmark in a book that you are reading – it helps in understanding what has happened so far, and what needs to be done ahead of. However, the future has much superior plans for data than merely recording it, and if you are a business person, you ought to be not just aware of this development, but also accommodate the change into your future course of action. Business is all about opportunities!


Record-keeping has always been a tedious clerical job. However, with digitalisation things became simpler as data could not only be recorded, but also stored in a compact form, organised in a researchable manner, and convertible into inputs for analysis. And thus, the world began switching to electronic records. With the spread of the internet, data recording received a further impetus, as the inputs could be directly collected from the end people through front-end representatives inputting the details, direct customer emails, web forms filled by people, software logs, and inputs through websites and mobile apps. As a result, businesses generated large databases of orders, purchases, sales, products, audiences, regional preferences, feedbacks, complaints, etc. Those at the helm of the corporates quickly realised that this data could help them predict the future and spot the trends. Thus, data recording became more aggressive which has resulted in today, in all our clicks, taps, scrolls, and screentime being recorded. Businesses are almost snooping us, except that we don’t realise it and thus, no discomfort, no complaints.

Today food and fashion preferences, electoral vote swing, demand-supply dynamics, security market movements, sports game results, pollution and global warming, the spread of diseases, space activities, etc. everything is analysed and predicted based on past data. However, in the fast-paced world that we live in today, a human analysing the data and then acting on it within minutes is still not fast enough. Computing devices have added unparalleled speeds to our tasks and automated many things. We now aim to implement the same in our decision-making as well. However, decision-making is a human thing and involves multiple considerations that cannot be combined even in a complex mathematical formula. So, is it possible to automate decision-making? Let’s explore that idea in this article.

What is Big Data?

Since the 1990s, Big Data has been mentioned by people on several occasions. The world was entrenched in big data even before the term was coined. Big data is a term, as straightforward, as it sounds. Technological breakthroughs have reduced the cost of data storage and computation. It is easier and less expensive to store data than ever before. Thus, companies have been recording data at a very high rate, and the same has become so voluminous, or complex that the traditional methods of analysing the data don’t work anymore. Industry analyst Doug Laney had articulated big data in the early 2000s, in three V’s –

Volume, the data under consideration is humongous

Velocity, the data streams at an unprecedented speed and need to be handled timely

Variety, the data can be structured and numeric, or unstructured text documents, emails, videos, audios, ticker data and transactions, etc.

Over years, two more Vs have become important –

Value, data has intrinsic value but is of no use until that value is discovered

Veracity, the entire data recording process is a waste, if the data is not truthful, the reliability quotient of the data needs to be high.

The Stock Exchanges such as BSE and NSE are prime examples of big data as they generate about one terabyte of new trade data per day. Meanwhile, social media websites also generate about 500+ terabytes of new data every day which consists of a photo and video uploads, messages and comments. Jet engines also generate 10+ terabytes of data in a 30 minutes flight and with thousands of flights per day, data generated may be in Petabytes.

Why is Big Data important?

Big data has become a capital asset for corporates. The type or amount of data being recorded isn’t the important point anymore, as it is cheap and easy, anyone can do it. However, what an organisation does with such data and how it analyses the same for insights to improve decisions and strategic business moves is what ‘big data’ targets. Today, several data analytics companies provide solutions to handle such data and generate useful business insights from the same. Think of Google, Facebook, Microsoft, Apple and Amazon, a large part of the value they offer comes from their data, as they are constantly analyzing to produce more efficient products.

Big Data serves several purposes to several companies –

Developing and offering products

Companies use big data to anticipate customer demand. For example, FMCG companies can establish which products are in high demand or reasons why a product failed, or what new product needs to be brought in, based on demand and feedback. Similarly, the Over-the-top (OTT) platforms can add more content of a particular genre based on user watch history and feedback, across different regions. Companies build predictive models for new products and services by classifying key qualities of products or services and modelling the relationship between those qualities and commercial success. Companies also use data and analytics from focus groups, social media, test markets, and early store rollouts to plan, produce, and launch new products.

Customer experience and efficiency

The race for customers is everywhere. However, the company with a clearer view of customer experience wins the race, most often. Big data enables companies to gather data from social media, web visits, call logs, and other sources to improve the interaction experience and maximize the value delivered. With big data, companies can analyze and assess production, customer feedback and returns, and other factors to reduce outages and anticipate future demands. It also helps in improving decision-making in line with current market demand. As a result of the data, companies can offer personalized products, implement dynamic pricing, reduce customer churn out, and handle issues proactively. 

Predictive maintenance and security

Manufacturing companies predict mechanical failures which are deeply buried in structured data such as the year, make, and model of equipment, as well as in unstructured data that covers log entries, sensor data, error messages and temperature records. Analysis of these data indicates potential issues before the problems happen. This helps the company in deploying maintenance more cost-effectively and also maximizes the parts and equipment uptime. Meanwhile, big data also helps in identifying security breaches and patterns in data that indicate fraud. It also helps in aggregating information to make regulatory reporting much faster. Companies also use big data for innovation by studying the interdependencies among humans, institutions, entities, and processes and accordingly determining new ways to use those insights.

Big Data and Artificial Intelligence

Artificial Intelligence (AI) is the branch of computer science that is concerned with building smart machines that can mimic human intelligence. When we are interacting with Siri, Alexa and Google, chatting with a bot instead of a customer service executive, or simply automatically filtering the spam emails, we are using artificial intelligence. Even the recommendations received on Netflix and other websites are based on artificial intelligence. These are machines that have been upgraded to smartly perform the tasks such as talking,  chatting, selectively sorting out, or recommending which usually a human brain is capable of performing. Although these are relatively new and often buggy, it is only a small depiction of the capabilities and opportunities ahead. Self-driving cars have been a breakthrough in this field which extensively use artificial intelligence along with sensors and radars, to simulate human decision-making. There are six branches of artificial intelligence –

1) Machine Learning (ML) which allows machines to learn on a cumulative basis, instead of being programmed. E.g. image recognition, speech recognition etc.

2) Artificial Neural Network (ANN) which tries to replicate the nerves and nerves system of the human body, using a set of mathematical algorithms to define relationships between data and accordingly make decisions. E.g. Instagram’s feed recommendation, Amazon’s product recommendations etc.

3) Robotics which develops robots to perform functions that humans body can perform E.g. Forklift Robots, Vision Systems

4) Expert Systems (ES) which deals with extracting knowledge from databases by implementing reasoning and insights rules based on user queries. E.g. Algorithm based stock trading

5) Fuzzy Logic (FL) tries to replicate human decision making in situations where multiple solutions are possible, assigning degree values from 0.0 to 1.0 to the answers, instead of giving true or false answers. E.g. Air conditioning according to room temperature, Refrigerators and microwave ovens, etc.

6) Natural Language Processing (NLP) deals with developing communication between computers and humans by using natural human languages. E.g. Smart digital assistants (Alexa, Google Assistant), Predictive text, auto-complete, speller checkers and Grammar checkers while typing, etc.

Artificial Intelligence requires a massive scale of data to learn and improve decision-making processes. Machines are capable of processing large data if the process is defined, by analysing the current situations against their database of such situations. However, without big data, developing this database of different situations wouldn’t be possible, as it requires analysis of a large number of situations. Therefore, Big Data and Artificial Intelligence have a synergistic relationship. With their convergence, advanced analytics capabilities like augmented or predictive analytics can be leveraged. Automation is already a part of our lives and there is a growing emphasis on automating more human activities with help of artificial intelligence. Thus, big data has a larger role to play in the future, as the future technologies depend on the same.

Business opportunities

“Biryani is the most ordered dish in 2021. Paneer Butter Masala and Butter Naan accounted for 1.1 million in revenue. Gulab Jamun is the most favoured dessert followed by Ras Malai” as per the statistics revealed by Zomato and Swiggy. The online food delivery platforms have huge databases of food preferences of the people – not just the popular foods, but also the food preferences according to the region, according to the time of day (or night), according to age group, gender and also the weather. Big Data is not an opportunity merely for the data analytics business but also for all other businesses. Zomato and Swiggy can easily sell this data to other food processing companies or restaurants who can then accordingly make adjustments in their products and services. Similarly, Uber and Ola have databases that contain details of taxis, city traffic, and customer preference for the type of cabs. Automobile companies can buy their data to develop cars that fit for serving as taxis. Facebook and other social media platforms have huge databases of likes and dislikes, hashtags, images and videos according to age, gender, location, time posted, etc. which is usually sold to other companies for advertisements and product developments. Amazon has a huge database of customers, their location, products that they prefer, etc which can be sold to other brands who manufacture such products which enables it to develop new and innovative products.

Every business generates certain data based on its industry. Small businesses usually do not record such data. However, if recorded and maintained well, it can serve value not just to the business, but also be sold for money. Conglomerates are looking to improve their products and services. Advertisers and marketing agencies are looking to improve their target customer base. Apps and other service companies are looking to reach their target audiences. Several websites and apps have adopted this model as their business where all they do is collect data in exchange for certain products and services, at times, even for free. For example, the content-based apps that provide content for free, however, collect, store and sell the data collected from their app through taps, clicks, voices, images and other means.

The bottom line

Data is an undisputed epicentre of future commercial activities and has massive importance in the future ahead. There’s no doubt the position that fuel has enjoyed during industrialisation, the same position would be the case for data, in the digitalisation era. That’s the reason why corporate houses like Reliance which has been in the energy, petrochemicals and telecom business is leveraging their brand to establish businesses to capture data and e-commerce sector such as Reliance Jio Phone, Jio Apps, JioMart, etc. However, this has not been the case with smaller businesses. Several business sectors can generate valuable data, however, being unaware of the opportunity that the future provides, there are no means implemented. By implementing data recording, collection and analysis, businesses generate data that is valuable not just for their businesses but also for others looking for it. After all, business is all about solving problems, the bigger the problem, the better the payoff.