Data quality, observability and hidden factors at play
Gone are the days when data quality was a back-office function. Today, data is the lifeblood of businesses.
As the variety and volume of data has exploded due to increased usage, businesses now rely on data freshness. According to Kirk Haslbeck (pictured), vice president of data quality at Collibra, Collibra NV equips the individuals who load and consume a company’s data with observability tools for tasks such as root cause analysis , which is necessary to capture the big picture and ensure improved quality.
“We’ve always covered data quality and we think people want to know more. They need more information and they want to see the breaking records and breaking trends together so they can correlate the root cause,” Haslbeck said. “So we really focus on root cause analysis, business impact linking it to lineage, catalog and metadata. And as this grows, you can truly achieve total data governance.
Haslbeck spoke with theCUBE Industry Analyst Dave Vellante during “Data Citizen’s 2022”, during an exclusive show on theCUBE, SiliconANGLE Media’s live streaming studio. They explained why data quality has become a hot topic in the corporate world and how observability fits into the picture. (*Disclosure below.)
Native Database drop-down menu
According to Haslbeck, by enabling functions like compute in databases like Snowflake, BigQuery, Databricks, Delta Lake, and SQL Pushdown, data intelligence and governance are even higher. This is made possible through native database pushback.
“While we’ve always worked with the same databases in the past, we do something called native database pushback, where all compute and data activity happens in the database,” a- he declared. “We now perform all compute and data operations in databases like Snowflake. And that means that without installation or configuration, you can connect to the Collibra Data Quality application and run all your data quality in the database.
For better decision-making, having the correct data is a game-changer, Haslbeck pointed out. Collibra helps achieve this goal through intelligence and data governance.
“If you and I were to create a new healthcare app and monitor people’s heart rates, imagine if we were wrong, what might the ramifications be?” He asked. “With the acquisition of what was a lineage company years ago and then my company OwlDQ – now Collibra Data Quality – Collibra is perhaps best positioned for total data governance and intelligence in the ‘space.”
Data observability paired with the right scale has the ability to generate alerts whenever trends are broken. This explains why root cause analysis comes into play, according to Haslbeck.
“What’s been so exciting is that we have these types of observing techniques; those data monitors that can actually track the past performance of every field at scale,” he said. “So we’re kind of moving away from this world where you have to write a condition and then when that condition breaks, it’s always been known as a break record. But what about break trends and root cause analysis?
According to Haslbeck, the fusion of observability and data quality solves the problem of stale data, thereby making data timeliness and freshness possible.
“This all points to the same idea that what you’re observing may no longer be a condition of data quality,” he noted. “It could be a failure in the data pipeline. And with thousands of data pipelines in play for every business, there are several happening every day. »
Here’s the full video interview, which is part of SiliconANGLE and theCUBE’s coverage of the Data Citizens 2022 event:
(*Disclosure: TheCUBE is a paid media partner for the Data Citizens 2022 event. Neither Collibra NV, the sponsor of theCUBE event coverage, nor other sponsors have editorial control over the content of theCUBE or SiliconANGLE .)