The Cat Paper
I finished listening to Acquired’s final episode on Alphabet yesterday. One of the interesting stories from this episode was the significance of the “cat paper”. Here’s Acquired highlighting this particular moment in history:
You talk to anyone at Google, you talk to anyone in AI, they’re like, “Oh yeah, the cat paper.”…just to like underscore how seminal this is, we actually talked with Sundar in prep for the episode. And he cited seeing the cat paper come across his desk as one of the key moments that sticks in his brain in Google’s story. A little later on, they would do a TGIF where they would present the results of the CAT paper and you talk to people at Google, they’re like, “That TGIF, oh my god, that’s when it all changed.”
I knew the cat paper was an important milestone, but perhaps I didn’t appreciate the extent of it, especially the role it played in other tech companies as well.
In 2012, the Google Brain team undertook an experiment at an unprecedented scale. They connected 16,000 computer processors to create a massive neural network (a computer system loosely modeled on the human brain) with over a billion connections. They then fed this network 10 million random, unlabeled still images taken from YouTube videos and essentially told the system: “Find the patterns.”
When the researchers looked inside the artificial brain they had built, they discovered something astonishing: the network had taught itself to recognize cats. One specific digital “neuron” activated strongly whenever the system was shown an image containing a cat.
The importance of this result wasn’t that Google could now find cats. It was how the system learned to find them. Before this paper, the dominant approach was Supervised Learning. This is like teaching with labeled flashcards. To train an AI, humans had to painstakingly label vast amounts of data (e.g., tagging thousands of photos as “Cat” or “Not a Cat”). As you can understand, this process was slow, expensive, and impossible to do at the scale of the internet.
The Cat Paper demonstrated the power of Unsupervised Learning at scale. The AI was never told what a cat was. It invented the concept of a cat simply by observing recurring patterns in the raw data.
The internet is composed almost entirely of messy, unstructured, and unlabeled data i.e. billions of hours of video, trillions of images, and endless audio clips. The Cat Paper proved that neural networks, if large enough and given enough data, could begin to make sense of this vast ocean of information without explicit human instruction.
The Acquired guys mentioned not only this was instrumental for Google but it also was perhaps directly responsible for hundreds of billions of revenue in companies such as Meta and ByteDance. How does recognizing a cat translate into the staggering revenues of Google, Meta, and ByteDance? The connection is Recommendation Engines.
User engagement is the key foundation on which companies such as Google (especially YouTube), Meta or ByteDance are built. To maximize engagement, they must continuously show you content you are interested in. This requires the platform to understand two things at an incredible scale: a) Understand the Content: What is this video, image, or post actually about? b) Understand the User: What does this specific user like?
The techniques validated by the Cat Paper unlocked the ability to understand content at scale. Without this paper, the internet today might have looked very different.
The paper also led to the events that made Google realize GPUs would be much better fit for certain workloads. In fact, in 2014, Google decided to put a big order for GPUs from Nvidia. From Acquired:
…they settle on a plan to order 40,000 GPUs from Nvidia…For a cost of $130 million.
That’s a big enough price tag that the request gets elevated to Larry Page who personally approves it even though finance wanted to kill it.
As an aside, let’s look at Nvidia at the time. This is a giant giant order. Their total revenue was $4 billion. This is one order for 130 million. I mean Nvidia is primarily consumer graphics card company at this point and their market cap is $10 billion. It’s almost like Google gave Nvidia a secret that hey, not only does this work in research…but neural networks are valuable enough to us as a business to make a hundred plus million dollar investment in right now.
Back in 2017, Bezos made an interesting point how business results are often lagging indicator of all the decisions the company made few years ago:
“When somebody … congratulates Amazon on a good quarter … I say thank you. But what I’m thinking to myself is … those quarterly results were actually pretty much fully baked about 3 years ago. Today I’m working on a quarter that is going to happen in 2020. Not next quarter. Next quarter for all practical purposes is done already and it has probably been done for a couple of years.”
Indeed, the cat paper likely created so many ripple effects within and outside Google that it perhaps played a key role in shaping our internet today. It also made me think it may be more important for investors today to spend more time going through research papers to stay at least somewhat informed where the future may be heading. “Attention is all you need” was published in 2017 which directly led to ChatGPT moment in 2022. Perhaps someone is going to publish a paper today or in a couple of years that may fundamentally alter the tech landscape 5-10 years from now. This makes analyzing some of these tech companies and investing in them potentially harder than appreciated. A DCF of a tech company can be very fragile if technical breakthrough can reshape the competitive dynamics profoundly. There’s no DCF anyone can ever build that would lead you to think Nvidia may go from $10 Billion market cap to $4.7 Trillion in just 10 years! As someone who builds DCF models, I am not saying we should invest based on YOLO vibes, rather I’m highlighting the inherent limitations in analyzing technology companies.
When Buffett indicated many technology companies don’t fall under his “circle of competence”, he is certainly not saying he doesn’t understand their business models today; I suspect he recognizes his limitations in assessing the wide range of outcomes for tech companies more than most investors do.
In addition to “Daily Dose” (yes, DAILY) like this, MBI Deep Dives publishes one Deep Dive on a publicly listed company every month. You can find all the 63 Deep Dives here.
Current Portfolio:
Please note that these are NOT my recommendation to buy/sell these securities, but just disclosure from my end so that you can assess potential biases that I may have because of my own personal portfolio holdings. Always consider my write-up my personal investing journal and never forget my objectives, risk tolerance, and constraints may have no resemblance to yours.
My current portfolio is disclosed below: