The Salience of Data

Let me begin with a thought experiment.

Imagine you know nothing about OpenAI’s or Anthropic’s financials. No ARR figures, and no valuation marks from the latest round. Now add one more condition, and it may be less hypothetical than it sounds: assume every frontier lab has access to a similar amount of compute, similar data, and similar talent.

Here is the question: sitting behind this veil of ignorance, how would you tell whether any of these model labs has a sustainable moat in the next 3-5 years? This question was actually much, much more difficult to answer couple of years ago. The fact that Anthropic was valued only $15 Billion in December 2023 may be indicative of how contrarian it was to bet on a model company and how challenging it was to articulate why these model companies would be able to outcompete the incumbent big tech companies when such big tech companies appeared to have almost unbounded access to capital, compute, talent, and data. To be fair, it is still quite challenging to articulate which particular model companies will end up dominating this layer. At least, today OpenAI and Anthropic have far more credible claims why they can have access to similar compute, and talent. It still seems silly to argue that OpenAI or Anthropic would have higher compute than Alphabet or Meta in the next 3-4 years. Talent in AI labs are also pretty mobile, so it’s hard to assume that as a source of any sustainable moat either. You can perhaps mention culture, but while I’m not denying culture as a source of potential moat, I always suspect investors mention about culture as a source of moat when it becomes difficult to pinpoint the source of sustainable moat.

How about data? You can legitimately argue one of the big reasons Anthropic has been such a resounding success is their focused bet on the best use case of these models: coding! And thanks to such a focused bet, they now have access to user data in coding which can beget to further improvement of the model. But even then, Anthropic’s coding model has found its greatest product market fit since December 2025. So, we aren’t even a year into this data flywheel moat and there may still be time for other model companies to respond pretty effectively to this moat. Codex is gaining ground and just yesterday, this tweet by Jukan made the case that xAI still has a pretty compelling shot at coding thanks to their acquisition of Cursor. Some excerpts from his tweet:

“One of the more interesting Grok bull cases I heard at ICML was this:

The core idea is that xAI may actually be better positioned than OpenAI Codex in the coding-agent market.

The reason Claude Code is currently leading in coding agents is not just model quality. Claude Code effectively pioneered the category at scale, which gave it one of the largest user pools in the industry. More users mean more real-world coding data. That data can then be used to improve Claude Code’s quality, which attracts even more users, creating a flywheel of more users, more data, and a better product.

Seen through this lens, xAI’s acquisition of Cursor starts to make a lot more sense.

Cursor likely has a much larger real-world user base and coding dataset than Codex. If xAI can effectively train on and leverage that data, the argument is that overtaking Codex may only be a matter of time.”

Given the success of coding use case, even Meta today entered the arena with their launch of Muse Spark 1.1 model.

Nonetheless, I do suspect data is likely the most reasonable explanation ex-ante why any particular model company may gain an upper hand over others while the rest of the inputs may be closer to commodity since all the relevant players will essentially have access to those. The fact that data is indeed the key source of long-term differentiation isn’t quite a new hypothesis. Back in June 2023, James Betker, a Research Engineer at OpenAI, in a piece titled “The “it” in AI models is the dataset” argued exactly this (emphasis mine):

“…I’ve trained a lot of generative models. More than anyone really has any right to train. As I’ve spent these hours observing the effects of tweaking various model configurations and hyperparameters, one thing that has struck me is the similarities in between all the training runs.

It’s becoming awfully clear to me that these models are truly approximating their datasets to an incredible degree.

…What this manifests as is – trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point.

…This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset.”

In a more recent piece, Will DePue, another OpenAI engineer who just left the company couple of months ago, re-iterated that data remains the key path to model differentiation. He wrote a compelling piece titled “A Stargate for Data”; some key excerpts below:

At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger clusters and datasets, and received incredible new capabilities in return.

But this exponential scaling is bound to hit some limits. Oddly enough, compute has compounded fairly smoothly without limit, with trillions flowing into hypercluster buildout. Instead, we’re starting to hit the limits of an exponential demand for data. Gone are the days of being purely in the compute-limited regime, where we had effectively infinite internet data but never enough GPUs, we’re now entering a data-limited regime.

Only a fraction of useful data in the world is on the public internet, the rest is stored inside private datasets, corporations, personal archives, universities, governments, and otherwise. Labs can and will continue to license these private datasets, or create them from scratch, like Anthropic’s book scanning project. And we’ll increasingly task human experts to manufacture new high-quality data, with a large fraction of hard RL training tasks already being sourced this way.

But collecting this data, unlike before, will be expensive. As the free internet dries up and demand for data rises, we should see labs investing equally in data as compute, likely spending a significant fraction of their compute budgets on data. As we see trillions spent on compute, we should also expect hundreds of billions spent on data (human data & collection budgets), given their equivalent importance. And, notably, data spend is already tracking this way: total data spend across vendors, not counting internal lab efforts, is already roughly $7 billion per year. It’s quite reasonable we’ll see >10x by 2030.

Last year, I highlighted another piece by Jack Morris: “There Are No New Ideas in AI… Only New Datasets” which also corroborated to the primacy of data . From his post:

Our breakthrough is probably not going to come from a completely new idea, rather it’ll be the resurfacing of something we’ve known for a while.

But there’s a missing piece here: each of these four breakthroughs enabled us to learn from a new data source:

1. AlexNet and its follow-ups unlocked ImageNet, a large database of class-labeled images that drove fifteen years of progress in computer vision

2. Transformers unlocked training on “The Internet” and a race to download, categorize, and parse all the text on The Web (which it seems we’ve mostly done by now)

3. RLHF allowed us to learn from human labels indicating what “good text” is (mostly a vibes thing)

4. Reasoning seems to let us learn from “verifiers”, things like calculators and compilers that can evaluate the outputs of language models

…The obvious takeaway is that our next paradigm shift isn’t going to come from an improvement to RL or a fancy new type of neural net. It’s going to come when we unlock a source of data that we haven’t accessed before, or haven’t properly harnessed yet.

One obvious source of information that a lot of people are working towards harnessing is video. According to a random site on the Web, about 500 hours of video footage are uploaded to YouTube *per minute*. This is a ridiculous amount of data, much more than is available as text on the entire internet. It’s potentially a much richer source of information too as videos contain not just words but the inflection behind them as well as rich information about physics and culture that just can’t be gleaned from text.

It’s safe to say that as soon as our models get efficient enough, or our computers grow beefy enough, Google is going to start training models on YouTube. They own the thing, after all; it would be silly not to use the data to their advantage.

Given this context, I am currently much more appreciative of Meta’s 49% stake in Scale AI. I have noticed that these “data labeling” companies are often mocked by investors, but based on my understanding of AI models’ key source of differentiation in the long term, I wonder whether Meta should have bought the entire company. I guess that might still happen and the primary reason they didn’t might be due to regulatory concerns. However, it still doesn’t seem abundantly clear who will have privileged access to differentiated datasets in the long term. And if data is indeed the true elixir to model differentiation, that also implies different models may have distinctly different strengths and weaknesses based on the access to data source they have managed to gain access. If an AI lab cannot secure the licensing deals for proprietary data, all the compute in the world will merely produce a highly efficient, mathematically perfect approximation of a commodity dataset. The structural advantage hence may shift slightly away from those who only own the picks and shovels, and toward those who own the land where the gold is buried. Ultimately, competitive dynamics in the next phase of AI will likely be defined by the unit economics of data acquisition.

Subscribers get the daily journal and five+ years of Deep Dives, i.e. full-length analyses with financial models on 65+ companies. The daily is just how I think out loud between the Deep Dives!

Current Portfolio

Please note that these are NOT my recommendation to buy/sell these securities, but just disclosure from my end so that you can assess potential biases that I may have because of my own personal portfolio holdings. Always consider my write-up my personal investing journal and never forget my objectives, risk tolerance, and constraints may have no resemblance to yours.

My current portfolio is disclosed below: