Nvidia Licenses Groq AI Inference Technology, Google TPU Architect Joins Team

Nvidia has secured a licensing agreement for AI inference technology from the startup Groq, a move that includes key personnel from Groq joining Nvidia. The deal, which CNBC initially reported as a $20 billion acquisition, was clarified by Groq as a non-exclusive technology licensing arrangement.

Under the agreement, Groq's founder and CEO, Jonathan Ross, along with President Sunny Madra and other engineering team members, have transitioned to Nvidia. Ross is recognized for his role in developing Google's Tensor Processing Unit (TPU). Groq will continue to operate independently, with its cloud business, GroqCloud, remaining unaffected. Simon Edwards, formerly Groq's CFO, has been appointed as the new CEO.

While the specific financial terms of the transaction were not disclosed, initial reports suggested a valuation of approximately $20 billion, which would represent Nvidia's largest investment or asset acquisition to date.

Focus on AI Inference

The licensing deal underscores Nvidia's strategic focus on AI inference, a sector Jensen Huang, Nvidia's CEO, previously stated would see "a billion-fold" increase in demand. Groq specializes in low-latency chips designed for inference tasks, distinguishing its approach from traditional GPU-centric solutions.

Groq, founded in 2016 by Jonathan Ross and other former Google engineers, focuses on creating high-speed, low-latency, and cost-effective AI chips specifically for the inference stage. The company completed a $750 million funding round in September, valuing it at $6.9 billion.

Groq introduced its Language Processing Unit (LPU) in early 2024, which demonstrated the ability to output 500 tokens per second, outperforming existing large language models at the time. A single LPU card, with 230MB of memory and a reported cost of $20,000, is described as being 10 times faster than Nvidia GPUs for inference tasks.

LPU Architecture and Scalability

Groq's LPU utilizes a Temporal Instruction Set Computer architecture, which differs from GPUs by reducing the need for frequent data loading from High Bandwidth Memory (HBM). This design aims to optimize clock cycle utilization, ensuring stable latency and throughput, while also mitigating HBM shortages and reducing costs.

The LPU's design allows for the seamless linking of multiple Tensor Streaming Processors (TSPs), which helps avoid bottlenecks common in GPU clusters and improves scalability. This architecture enables linear performance scaling as more LPUs are added, simplifying hardware requirements for large-scale AI models and facilitating application expansion without extensive system re-architecture.

Groq's technology addresses the need for low-latency, high-efficiency inference, which is a critical area as the AI market shifts from a "training-centric" to an "inference-centric" model. While Nvidia GPUs dominate the training phase, the inference market is competitive, with players like AMD, Google TPUs, and various startups. Nvidia's integration of Groq's low-latency technology is seen as a move to complement its product offerings and strengthen its position in data centers and AI infrastructure.

Key Personnel Transition

Jonathan Ross, the former lead designer of Google TPU, initiated the project that led to the development of the first-generation TPU chip while at Google. He later joined Google X's Rapid Eval Team, where he contributed to several new business initiatives for Alphabet. Ross, a self-taught individual who audited courses before attending New York University's Courant Institute, also studied under deep learning pioneer Yann LeCun.

The inclusion of Ross and his team from Groq into Nvidia suggests an aggressive strategy by Nvidia to address its capabilities in inference computing and expand its leadership in the inference market.