Why Is Data Quality Still So Hard to Achieve?

Back to News & Press
Dataversity
October 31, 2023
by Vasu Sattenapalli
Why Is Data Quality Still So Hard to Achieve?

We exist in a diversified era of data tools up and down the stack – from storage to algorithm testing to stunning business insights. In fact, it's been more than three decades of innovation in this market, resulting in the development of thousands of data tools and a global data preparation tools market size that's set to expand at a compound annual growth rate (CAGR) of 18.6% from 2021 to 2028. This begs the question: With all these tools at our disposal, as well as ample awareness of the importance of data, why is Data Quality still so hard to achieve?

For starters, even with all our focus on data, there are still several factors creating cracks in our modern data pipelines. What are they, and how can we address them to help organizations align and ultimately achieve higher Data Quality?

Exploding Data Volume

One major factor putting pressure on our data pipelines is the sheer volume and variety of data at our disposal. The amount of data that exists has exploded, growing exponentially in recent years. In fact, it is estimated that 90% of the world's data was generated in the last two years alone. And by 2025, global data creation is projected to grow to more than 180 zettabytes. This indicates that organizations are collecting an ever-increasing amount of data.

Even if at one point an organization felt that it had achieved high Data Quality, it's incredibly unlikely that they've been able to maintain that level of assurance as the volume of data has grown. And with the advent of big data and the proliferation of data sources, such as social media, IoT devices, and sensors, organizations are grappling with enormous datasets that also have varying structures and formats. This diversity makes it challenging to maintain consistent Data Quality standards across the board – especially as new structures and formats continue to be created.

Persistent Silos Separating Data

Second, it's a well-known issue that data is often siloed within organizations. Different departments and teams collect and manage their data independently, leading to fragmentation and a lack of standardized practices for Data Quality. As individual departments continue to add on more and more tools, it becomes even more challenging to get those tools to integrate at the department level and the organization level, resulting in inconsistencies and errors that are difficult to detect and rectify. Furthermore, when data is siloed, each tool introduces new transition points across each stage of the data pipeline, from data ingestion and transformation to analysis and reporting. This can compromise Data Quality – creating new opportunities for errors to creep in – and identifying the source of issues can be like searching for a needle in a haystack.

Difficulty Creating a True Data Culture

Finally, Data Quality is not just a technical issue; it's also a cultural and organizational challenge. Currently, most data tools are designed for use by data professionals working with data, not for the average user. Because they are often used by data analysts or data scientists to manipulate data in order to gain insights or create reports, they may require coding knowledge or prior experience using other data tools for data cleansing, data transformation, data visualization, and data analysis. This may have worked well when data professionals were the only ones using data tools, but that is no longer the case. Modern Data Quality requires collaboration across departments and teams – not just IT teams or data scientists. Without a real culture of Data Governance and accountability, Data Quality issues will persist.

A Three-Pronged Fix

The good news is that there are some ways to address the challenges outlined above. A three-pronged approach that addresses all three – improving the data itself, facilitating cross-silo data use, and fostering a true data culture – is the best method.

Improve the Data Itself

Automation and AI are changing the game by helping organizations accelerate manual data-checking tasks. By automating Data Quality checks and validation processes, organizations can reduce human errors and ensure more consistent, reliable data. This is beneficial for keeping up with the vast amounts of data being generated, but there is a catch. Organizations must be careful not to rely too heavily on automation – implementing these technologies without a plan in place to monitor for problems can cause more issues down the road.

Facilitate Cross-Silo Data Use

Data catalogs and similar tools can make it easier for team members across the organization to discover and access data across silos. Additionally, data catalogs can help establish clear data ownership and stewardship roles across the organization as they also help establish clear data ownership and stewardship roles across the organization.

Foster a True Data Culture

In terms of addressing culture within an organization, many data tools are becoming more user-friendly and accessible to a broader range of users – especially those who want self-service analytics capabilities. This makes them useful for data-savvy business users, as well as for data professionals. Organizations that provide user-friendly self-service data analytics tools can empower more team members to contribute toward achieving better Data Quality. Building a culture of Data Governance and accountability, establishing clear standards, and investing in education and training programs to upskill employees is important.

Conclusion

By leveraging the combination of automation, collaboration tools, and self-service analytics, organizations can take a comprehensive approach to addressing the challenges that contribute to poor Data Quality. In doing so, they can improve the quality of their data, facilitate cross-silo data use, and foster a true data culture – all of which are necessary for achieving and maintaining high Data Quality.

Originally published in

Dataversity

Published in

Dataversity

Author

Vasu Sattenapalli

Date

October 31, 2023

Share this Article