Big Data’s Role in Predicting the Outcomes of Big Events

The popularity of big data and its usefulness in business has led to the adoption of data analytics across many industries. A 2016 survey of Fortune 1,000 companies by New Vantage Partners, a business strategy consulting firm, found that more than 60% of companies are using big data in some capacity. Those companies are expected to spend more than a cumulative $50 million on data analytics in 2017.

The rise of data analytics has led to increased media attention toward event forecasting websites such as FiveThirtyEight and PredictWise, which use advanced statistical models and opinion poll analysis to predict political elections, economic trends and sporting events.

Recently, these statistics-heavy platforms were used to help predict the outcomes of national events such as Brexit and the 2016 presidential election.

Many opinion polls and data analysis surveys indicated voters would choose to stay in the European Union (EU). Likewise, polls in the U.S. predicted Hillary Clinton would defeat Donald Trump last November.

In the aftermath of those nationwide events, the reliability of big data was questioned because statistical models predicted the wrong outcomes. Or did they?

The Misinterpretation of Big Data

Collecting large sets of data has become a mainstay for many industries, but analyzing and interpreting it has been a challenge for many organizations.

In a 2015 survey conducted by PricewaterhouseCoopers (PwC), 1,800 senior executives from mid- to large-sized companies were surveyed, and results revealed that only a small percentage of companies reported effective data management practices. Of the executives surveyed, 43% said they had received few tangible benefits from big data, while 23% reported zero benefits from data analytics.

The fault, in many cases, is not with the data itself, but with the way it’s gathered, examined and translated. In terms of the predictions regarding Brexit and the 2016 presidential election, there was nothing fundamentally wrong with the data. The problem was the people in charge of collecting, analyzing and interpreting it. 

Data sets and data-driven forecasting models can often reflect their own creator’s biases. This, and the subjective interpretation of data, are examples of how data can be misunderstood.  

According to The New York Times, data experts say the danger with data analysis is trusting it without grasping its limitations and the potentially flawed assumptions of the people behind the creation of predictive models.

Big Data Provides Probabilities, Not Answers

Predictions are merely a probability something might happen, and no predictions are guarantees. The Brexit vote and the 2016 presidential election are examples of the potential shortcomings of polling, analysis and interpretation, both in how the numbers were presented and perceived by the public.

Many of the opinion polls and predictions leading up to the European Union referendum were based on telephone surveys, which indicated a majority of voters wanted to remain in the EU. The same type of opinion polls projected Trump losing the presidential election in almost every forecast model. When looking at these two cases, the collection and interpretation of data may have been flawed. Opinion polls are typically done in person and in some cases, people may not express their true beliefs and risk starting an argument or offending someone. 

In both cases, online polls provided a better indication of the eventual outcome, despite their unscientific nature. Predicting political, economic or sports outcomes is challenging to do weeks or even months ahead of time. The smallest variables can cause changes that can affect the outcome of the event.

Forecasting Big Data

Collecting data is one thing. Interpreting it and understanding the potential risks of making flawed assumptions is another.

Data analysts create methods to collect data and analyze small but important patterns that emerge from large data sets. Data analysts, also known as data scientists, can help business leaders take a more practical approach to data. This starts with knowing exactly what information is needed, rather than collecting large quantities of statistics that serve no purpose.

Bridging the gap between data scientists and those reporting the data is the logical next step to help improve event forecasting. Data scientists help provide a sound strategy for interpreting data and using it in the business. According to the PwC survey, of the 66% of respondents who reported little to no benefits of big data, a majority of them did not have a data analyst on staff. Others reported having data analysts but not utilizing them in the right context.

It’s important to understand what types of data help in making the right business decisions. Looking back at the data gathered during Brexit and the 2016 presidential election, some of that data proved to be more accurate because it came from a more reliable source.

Despite big data’s perceived failure, its usefulness in business is growing. However, employing professionals who understand how to collect the right kinds of data and interpret it correctly will be key to produce more reliable forecasting.