Big Data journey is still a bumpy ride

As Big Data definitively shifts from its hype cycle to adoption cycle, there is significant investment, lots of excitement, and heaps of expectation. It is time to take stock. And that is what a lot of analysts—large MR firms, specialized research firms, big consulting companies and vendors—have been doing of late.

As CIO&Leader analyzes about a dozen such reports, one thing is becoming clear. The dominant sentiment is that of disappointment, if not despair. While no one is questioning the promise of Big Data—at least not yet—the experience has taught many a great deal of lessons where things have gone wrong.

And there is one simple answer from all these reports—implementation. Details vary, though.

The ‘magnitude’

Organizations are losing USD 600 million each year to ‘bad’ Big Data initiatives, says the research and education firm TDWI (The Data Warehouse Institute). Consulting firm, Capgemini, reports 73% organizations do not profit from Big Data analytics. Only 2% of organizations are generating value from analytics, according to a report by ZS Asociates, a research firm.

Yet, there is no slowdown on spending yet.

One in eight organizations are still ready to spend more in Big Data analytics, reveals the research by Capgemini. More than half (52%) of the organizations have admitted in a ZS Associates survey that they are spending heavily on Big Data analytics and the spending is going to continue. Text analytics firm, ATTIV/O, in one of its reports, says that 81% of organizations will increase their investment in Big Data in the next five years.

This suggests that the companies are still hopeful and expect to learn from their mistakes, which are mostly to do with implementation.

So, what are the issues? Where do the gaps lie? Our analysis of research reports reveal there are six major areas of concern.

They are:

• Sheer size of data
• Inadequate data quality management practices
• Lack of alignment between IT and business managers
• Failure to derive actionable insights from data
• Lack of visibility of available data across the organization
• Skills shortage

Big Data is too big to handle. Here, we are talking of just one of the three Vs—volume.

According to a research carried out by 451 Research for data accuracy engine provider, Blazent, two-third of the companies use less than 60% data that they generate. Only 9% use 80-100% of data. By 2020, predicts Znet, the world will be producing a minimum of 44 zettabytes data annually, making the management an even bigger issue. Companies are blinded by the sheer size of Big Data. As per Blazent, a majority of 81.5% of companies believe their organizations’ quality of data to be better than it really is.

Data quality management has a long way to go. As per research firm, StreamSet, 68% organizations cited managing data as their biggest challenge. Yet, almost half of the organizations (44.5%), according to Blazent, use rudimentary approach for managing data errors by using reports and then taking subsequent (after the fact) corrective action. Manual data cleansing is used by 37.5% organizations and 8.5% companies avoided data quality management completely.

Almost half of organizations using Data Quality Management (DQM) techniques were unhappy with their techniques, as per Blazent. The state of data quality despite the latest technologies is so bad that 60% of IT leaders and C-suit employees have no faith in their organization's data quality management practices.

Organizations not only face bad data flowing towards their data streams but also fail to cleanse it despite DQM technologies. Nine out of ten companies, that is, 87% of organizations suffer from data pollutants and 74% of organizations have pollutants in their data stores currently, despite data quality management technologies, as per StreamSet. This ruins the whole efforts for analysis and Big Data management.

Companies lack in basic infrastructure to have a good quality, secure and efficient data. From data management to security to pipeline efficiency – basic infrastructure calls for active and efficient data governance.

There’s little alignment between IT and top managers. More than half of the organizations (57%) have senior managements and IT employees working separately on Big Data, as per ATTIV/O report. Out of which, in 22% of organizations, there’s no C-suite employees who is responsible for data analytics; whereas in the rest, only one C-suite employee works independently for data derivation.

The Blazent report on data quality management says that those who are ‘responsible’ for data quality (i.e., IT) and those who are held accountable (the business managers) are poorly aligned. This leads to failed data management.

Businesses fail to derive actionable insights. Many organizations limit themselves to technical analysis of data, without deriving any actionable insight. Data leaders spend more time collecting data, cleaning and organizing it than analyzing it with context; this is a common finding in most of the reports. As per IDC by 2020, organizations which are able to analyze all relevant data and deliver actionable information, will achieve an extra USD 65 Billion in productivity benefits over their peers.

The organization is often opaque when it comes to data. Data collected by one department of company is not visible to another. Only 39% of companies have data visibility across the organization, as per the Big Data Decision Making survey by ATTIVI/O. Visibility lacks at multiple stages. Companies that had a proper framework to access and share structural data across divisions also find it hard to deal with unstructured data. Many companies can not utilize data even if accessed and processed properly because it fails to reach the right employee.

Last but not the least, shortage of skilled staff is a big big issue. A majority of 68% of data leaders, as per StreamSet research, believe that lack of skilled employees to analyze data is the biggest challenge in leveraging data. Lack of proper talent was marked as the biggest problem in big data in ATTIV/O survey by 22% companies.

For example, in APEJ, according to an IDC Futurescape report, the shortage of skilled staff will persist and extend from data scientists to architects and experts in data management even as the market grows by 29% CAG through 2020.

Some of the challenges, such as data quality management are basic. Others, such as C-suite involvement require fundamental changes at organizations. For the Big Data journey to continue smoothly, these challenges have to be met. The good part is, excitement and hopes are intact. Actions must match them.

Sneaker