Nvidia Strikes Technology Licensing Deal with Groq — Why Is It Buying the Tech Instead of the Company?

TradingKey - On December 24, local time, Groq, a U.S. AI chip startup, announced on its official website that it had reached a non-exclusive technology licensing agreement with NVIDIA.

According to the terms of the agreement, NVIDIA will pay approximately $20 billion in cash, and Groq will license its core AI inference technology to NVIDIA, with the latter planning to integrate these technologies into its future products. Concurrently, Groq's founder Jonathan Ross, President Sunny Madra, and several key R&D team members will officially join NVIDIA, to drive the implementation and expansion of the relevant technologies.

Although earlier market rumors suggested NVIDIA intended to acquire Groq outright for approximately $20 billion in cash, which would have been its largest M&A deal ever, this claim was subsequently denied by both parties.

NVIDIA explicitly stated that this was not an acquisition but solely an intellectual property licensing collaboration. Groq also emphasized that the company will maintain operational independence, with Chief Financial Officer Simon Edwards taking over as CEO, and its existing cloud service business, GroqCloud, will continue to operate normally.

Addressing Shortcomings

In an internal email to employees, NVIDIA CEO Jensen Huang clearly articulated the strategic intent behind the collaboration with Groq. He noted that through this partnership, the company will integrate Groq's ultra-low latency processors into its NVIDIA AI Factory architecture to support larger and more complex real-time AI inference workloads.

Huang added, "While we are bringing in top talent and licensing Groq's intellectual property, we are not acquiring the company Groq."

As generative AI enters a phase of rapid commercial deployment, market demand for computing power architecture is shifting. Industry projections anticipate that by 2030, "inference tasks" will account for up to 75% of the total AI computing workload, representing a market size of $255 billion. In contrast, the large-scale GPUs predominantly used today are more suited for the training phase, and their high cost and substantial power consumption render them unsuitable for all scenarios.

Groq's LPU (Language Processing Unit) technology possesses this value. In specific AI inference tasks, its processing speed and latency performance surpass traditional GPUs. Through this licensing partnership, NVIDIA has not only acquired critical technology but also brought core talent into its fold, positioning itself to rapidly address its shortcomings in the inference domain.

This strategic strengthening initiative is underpinned by NVIDIA's growing financial prowess. As of the end of last October, the company's cash and short-term investments had reached $60 billion, more than tripling from $13.3 billion at the beginning of 2023.

Unlike the large-scale acquisition of Mellanox for $6.9 billion in 2019, this Groq transaction unfolded as a non-exclusive technology licensing plus talent integration model, a deal structure that mirrors models adopted by Microsoft, Amazon, and Google over the past two years, and represents a common strategy for tech giants to circumvent regulatory scrutiny recently.

Even without a major acquisition, NVIDIA's defensive positioning is unfolding. Recently, the company has heavily invested in the AI ecosystem, including capital injections into AI infrastructure company Crusoe and model developer Cohere, increasing its stake in AI cloud service provider CoreWeave, and planning to invest $100 billion in OpenAI and $5 billion in Intel.

Why Groq?

Founded in 2016, Groq is a prominent Silicon Valley startup specializing in the design of AI model inference acceleration chips (LPUs) , whose founder, Jonathan Ross, was a key member of Google's TPU project and subsequently led part of the original TPU team to establish a new venture.

Reportedly, its core product, the LPU, is designed based on a unique 'Temporal Instruction Set Computer' (TISC) architecture, utilizing SRAM as a storage medium, offering high-bandwidth data throughput and ultra-low latency characteristics, which significantly enhances the operational efficiency of large language models.

According to third-party evaluations, when running large models like Meta Llama and Mistral, the Groq LPU achieved a generation speed exceeding 500 tokens per second, significantly outperforming an H100 GPU under equivalent conditions.

However, this high performance comes with a trade-off: a single card has only 230MB of memory capacity. A full deployment requires stacking numerous LPUs, leading to significantly higher deployment costs and power consumption compared to current mainstream GPU systems.

Consequently, many industry insiders believe that, at this stage, LPUs cannot fully replace GPUs in terms of either cost-effectiveness or ecosystem compatibility. Nevertheless, their strategic value as a crucial piece of the puzzle for improving the collaborative efficiency between training and inference cannot be overlooked.

Dylan Patel, Chief Analyst at SemiAnalysis, pointed out that while the first-generation product may not yet challenge the H100's dominance, Groq's subsequent two generations of LPUs are poised to be highly competitive, which likely prompted NVIDIA's early entry.