Why Data Quality Sucks

Data Quality Is Painful to Build

Issues are Needles in the Haystack

The principal reason data quality is so painful is that the problem space is enormous. Every column, every row, every relationship between tables is a potential failure point. A dataset with 200 columns and 10 million rows presents an almost infinite number of ways things can go wrong. Even knowing where to start requires expertise that most teams don't have, and the effort to systematically assess that space is prohibitive at any meaningful scale.

This is what makes writing good rules so difficult. The rules that catch real problems are rarely obvious. "Null checks" and "value in set" constraints are table stakes. The issues that actually cause damage are subtle.

And yet stakeholders tend to treat data quality as binary — the data is good or it's bad. And most often, it's bad. In reality, it's a continuous optimization exercise. Teams are trying to maximize coverage of a sprawling attack surface with limited resources. Classic framing of data quality as a pass/fail exercise creates a fundamental mismatch that drives chronic underinvestment and perpetual disappointment. Under these conditions, data quality teams inevitably default to shallow checks that give the illusion of effectiveness while leaving the most damaging issues completely uncovered.

Long Cycles to Implement Basic Checks

Even simple validation rules require a surprising amount of coordination.

In smaller companies, this may seem extreme. But in large enterprises, organizational complexity, compliance requirements, and separation of duties make this level of coordination not just common but mandatory.

Now multiply that by the hundreds or thousands of checks a mature data estate actually needs. The backlog is infinite, the throughput is slow, and every rule carries an ongoing maintenance burden that compounds over time.

Thankless, Under-Resourced Work

There's a fundamental economic problem that prevents organizations from addressing this head on: success in data quality is invisible. When everything works, nothing breaks. There's no dashboard that shows the "problems that didn't happen." The ROI case is preventing a negative, and nobody gets promoted for preventing a fire that never started.

Given that reality, the economics are a tough sell in fast-moving, value-driven organizations. It's high-cost to implement, hard to measure returns, and a single data quality incident can undermine the credibility of the entire exercise. Data quality teams end up trapped in an impossible position — doing difficult, thankless work where success is invisible and failure is catastrophic. That kind of environment burns people out, drives turnover, and produces the mediocre outcomes everyone complains about. The people doing this work know it matters. They just can't prove it in a language the business responds to.

Traditional Data Quality Is Ineffective

Even when organizations invest heavily, the results are often underwhelming. The reasons are structural, not a matter of effort or competence.

The Business-IT Disconnect

Business users know what "good data" looks like.

They carry deep institutional knowledge about what the data means and what patterns signal a problem. But they can't express that knowledge in code.

Engineers, on the other hand, can write the checks. They can build sophisticated validation frameworks and deploy them at scale. What they lack is the domain context to know what actually matters.

The business expert describes the intent. A systems analyst translates it to requirements. The engineer interprets it. Somewhere in that game of telephone, the original meaning is lost. The result is rules that are either too naive (they miss the real problems) or too strict (they cry wolf until everyone stops listening).

Rules Rot

Data quality is not a set-it-and-forget-it exercise, but it almost always ends up being treated like one. Schemas evolve, business logic shifts, and new systems are onboarded. A rule that was perfectly calibrated six months ago may be flagging false positives today — or worse, silently missing real issues because the data moved out from under it.

The reasoning behind a rule (why it exists, what it's protecting against, and what business context made it necessary) lives in someone's head, not in the system. When that person moves on, changes roles, or simply forgets, the rule becomes an orphaned artifact. It still runs. It still alerts. But nobody knows whether it's still relevant, and nobody wants to be the one to turn it off. Alert fatigue sets in, notifications pile up, and teams start ignoring quality signals entirely. This makes the system worse than useless. It creates a false sense of coverage.

Fixing Is Harder Than Finding

Detection without remediation is just noise. And this is where most data quality tooling stops. It tells you something is wrong, but not what to do about it. Today, remediation happens through manual interventions and expensive "war rooms" once issues are escalated.

Finding a problem is cheap. Triage and repair is expensive. Executives don't consider data quality work complete until the fix is delivered and downstream systems are whole again, but today that remediation cycle is highly manual. Delivering the fixes that actually resolve data quality issues is the key to connecting quality work to the value drivers the business requires to justify these activities.

What a Better World Looks Like

For the first time, advances in large language models and agentic AI systems have fundamentally altered the underlying economics of data quality management. What makes this moment different is not incremental improvement to existing approaches, but a structural shift in how data quality artifacts are created, maintained, and enforced.

The Cost Curve Is Inverting

Today, the question every data consuming organization faces is: "Can we afford to design, implement, and maintain enforcement for this?" Every rule has a real cost — engineering time to build, cross-team coordination to scope, ongoing effort to maintain. That cost forces triage. Teams cover the obvious risks and handle everything else reactively.

When the marginal cost of creating a rule drops to near-zero, the question inverts: "Why not check this?"

The ROI equation that has trapped data quality teams for decades finally flips. Data quality becomes justifiable not by proving the absence of failure, but by the sheer breadth of coverage relative to cost. It's the same inflection that transformed software testing. Once the cost of writing tests dropped far enough, coverage became an expectation rather than a luxury. Data quality is approaching that same threshold.

The Ownership Model Changes

When domain experts can express quality expectations in natural language and have the machine generate the implementation, ownership fundamentally shifts. The business-IT translation layer (where quality has gone to die for decades) collapses.

Rules become readable, auditable, and maintainable by the people who actually understand what the data should look like. A business analyst can define an expectation and see it become an executable check without writing a line of code. They can review the generated logic, approve it, and modify the intent later when business conditions change.

This doesn't diminish the role of IT — it elevates it. Engineers stop spending their days translating business intent into boilerplate validation logic and triaging alert escalations. Instead, they focus on the infrastructure, pipeline reliability, and systems architecture work that actually delivers differentiated value to the business. IT moves out of escalation hell and into the work they were hired to do.

Quality Becomes a Living System

Static rulesets decay from the moment they're written. A living quality system does the opposite. It compounds in value over time.

Agentic LLM systems can analyze data and operational artifacts to generate rules proactively, scanning the whole haystack rather than hoping an engineer finds the needle. They can re-evaluate rules against changing schemas and distributions, flagging when a check is going stale before it becomes a blind spot.

Domain knowledge, the institutional understanding of what data should look like and why, finally gets encoded into the system in a durable, evolving way. Rules stop rotting because the context that created them is no longer trapped in someone's head. It's part of the system itself.

The Widening Gap

Organizations that capitalize on this shift stand to gain a decisive and durable advantage.

The technology is not perfect, and the approaches to delivering on this vision are still being refined. But those things are not the bottleneck. The organizations that begin now, even imperfectly, will build an institutional asset that grows more valuable with every rule written, every schema change absorbed, and every remediation cycle completed. Each month of accumulated intelligence makes the next month's coverage broader and the system harder for competitors to replicate. Those that start encoding their domain knowledge now will see this compounding benefit earliest and fastest.

Teams that don't will stay trapped in the old economics: expensive to build, impossible to maintain, and perpetually one schema change away from a production fire. The automation and AI initiatives that sit on top of the data foundation — the very programs leadership is betting on — will never deliver on their promise if the underlying data can't be trusted. The gap between these two trajectories widens fast. And for one side of that gap, data quality continues to suck.