AI model outputs are only as reliable as the data they learn from. Most organizations understand this in theory. But where exactly is that data coming from, and does it reflect how people actually behave?

‍

Almost every leadership team right now is having some version of the AI conversation. Efficiency gains. Headcount assumptions. Security risk.  

But what is the model actually learning from, and how confident are we in that data?

Whether you're fine-tuning a model for internal decision-making, training a recommendation engine, or using AI to shape how you allocate spend, the output is only as good as what went in.

The quality problem didn’t disappear. AI magnified it.

Those of us who spent time in the consumer insights industry know that data quality has always been contested. Survey panels have battled fabricated responses for years. Respondent fatigue, fraud, low engagement — these aren't new concerns. The industry has largely learned to manage around them rather than solve them completely.

AI doesn't fix that underlying problem. In some ways, it amplifies it.

The appeal of synthetic data is obvious: it's cheap, scalable, and can be generated on demand. When organizations struggle to get real respondents cost-effectively, synthetic feels like a solution. But synthetic data doesn't create new signal. It models what has already happened and extrapolates from it. Feed that into a training pipeline and you’re no longer grounding the model in observed reality. You’re grounding it in a version of reality shaped by the limitations and biases already present in the source data.

I've heard people in the market research world talk about synthetic respondents with genuine enthusiasm and I understand the economic logic. I don't think it solves the quality problem; it just makes it less visible.

Garbage in, garbage out — still relevant

People have always asked hard questions about financial models. What are the assumptions? Where do the inputs come from? Does the output feel right given what we know about the business? That scrutiny exists because bad assumptions can produce confident-looking numbers, and confident-looking numbers get acted on.

AI models work the same way. If the data they're trained on is incomplete, self-reinforcing, or disconnected from how people actually behave in the real world, the outputs will be confident and wrong. The model doesn't know what it's missing. It just keeps producing results.

The companies taking this seriously are already asking tougher questions. Can we trace what this model is trained on? If a strategic decision is being shaped or validated by an AI system, the provenance of that system's training data is a legitimate concern for senior decision-makers — just as the assumptions in a DCF model are.

That framing, "what were the inputs?", has existed for a long time and now it just needs to be applied to AI.

Why "real" is getting harder to find

There's a broader dynamic that makes this more urgent. As AI-generated content proliferates, the ratio of synthetic signal to observed human behavior in available data is shifting. Web-scraped training sets increasingly contain AI-generated content. Survey responses increasingly contain AI-assisted answers. The feedback loop we've written about before — where models train on data partially generated by earlier models — is not a future risk. It's already happening.

In that environment, knowing that a dataset reflects real people making real decisions — across apps, across platforms, across competitors — is increasingly the thing that differentiates reliable signal from noise. Data provenance, the ability to trace where data originated and defend how it was collected, is shifting from a compliance checkbox to a strategic asset.

The organizations that treat it that way now will be better positioned when regulators, investors, and partners start asking the same questions routinely. The EU AI Act's requirements around training data transparency are already moving in this direction.

The competitive blind spot hiding in first-party data

There's a specific version of this problem that's easy to overlook. Most AI systems built inside large companies are trained primarily on first-party data — what users do on your platform, not what they do everywhere else. That's understandable. It's the data you have. But it means your model has a structural blind spot: it knows nothing about the competitive context in which your users are making decisions. It can't see that a user's engagement on your platform is declining because they've shifted time to a competitor. It can't see that a category is being disrupted by a new surface — AI search, agentic shopping, a new platform — because none of that shows up in your own logs. Models trained on first-party data will optimize confidently for a view of the world that's only partially true.

The question worth asking now

The question that isn't being asked often enough is: what is our AI actually learning from, and how do we know it reflects reality?

For companies making material decisions on product, on pricing, on market entry, on resource allocation, that question matters. An AI system trained on incomplete, synthetic, or opaque inputs won't fail loudly. It will produce plausible outputs that quietly drift away from what's actually happening in the market.

This is particularly true when market behavior is shifting fast. A model trained on data from 12 or 18 months ago may be optimizing confidently for patterns that have already changed — consumer behavior around AI-native discovery, for instance, is moving faster than most training pipelines can track.

At some point, models still need grounding in observed behavior. Observed behavior — real people, real decisions, real competitive dynamics, captured directly — is what keeps models grounded when the market shifts in ways no historical dataset anticipated.

That's not a technical capability. It's a strategic one.