We're here in sunny San Jose California for the return of an event that's been a long-time coming: NVIDIA's in-person GTC. The Spring 2024 event, NVIDIA's marquee event for the year, promises to be a big one for NVIDIA, as the company is due to deliver updates on its all-important datacenter accelerator products – the successor to the GH100 GPU and its Hopper architecture – along with NVIDIA's other professional/enterprise hardware, networking gear, and, of course, a slew of software stack updates.

In the 5 years since NVIDIA was last able to hold a Spring GTC in person, a great deal has changed for the company. They're now the third biggest company in the world, thanks to explosive sales growth (and even further growth expectations) due in large part to the combination of GPT-3/4 and other transformer models, and NVIDIA's transformer-optimized H100 accelerator. As a result, NVIDIA is riding high in Silicon Valley, but to keep doing so they also will need to deliver the next big thing to push the envelope on performance, and keep a number of hungry competitors off their turf.

Headlining today's keynote is, of course, NVIDIA CEO Jensen Huang, whose kick-off address has finally outgrown the San Jose Convention Center. As a result, Huang is filling up the local SAP Center arena instead. Suffice it to say, it's a bigger venue for a bigger audience for a much bigger company.

So come join the AnandTech crew for our live blog coverage of NVIDIA's biggest enterprise keynote in years. The presentation kicks off at 1pm Pacific, 4pm Eastern, 20:00 UTC.

---

04:04PM EDT - This is Ryan. Apologies for the late start here folks, it's been an interesting time getting everyone down to the floor and settled. (I'm told the capacity of SAP Arena for concerts is 18,500 people. With this floor layout, there may be closer to 20,000 here)

04:04PM EDT - So without further ado, here's Jensen!

04:04PM EDT - "This is not a concert. You have arrived at a developers conference"

04:04PM EDT - "I sense a very heavy weight in the room all of a sudden"

04:05PM EDT - Jensen is recapping the many technologies to be seen here. "Even artificial intelligence"

04:06PM EDT - Showing off a iist of exhibitors. It's a very big list of names

04:07PM EDT - "The computer is the single most important instrument in society today"

04:07PM EDT - How did we get here? Jensen drew a comic flow chart

04:09PM EDT - CUDA became a success... eventually. A bit later than Jensen would have liked

04:09PM EDT - 2023: Generative AI emerged, and a new industry begins

04:09PM EDT - "The software never existed before, it is a brand new category"

04:10PM EDT - Comparing generative AI to the industrial revolution and the age of energy

04:11PM EDT - "We're going to talk about how we're doing computing next"

04:11PM EDT - As well as software and applications in this industry. And how to start preparing for what's next

04:11PM EDT - Everything we'll be shown today is a simulation, not an animation

04:12PM EDT - "It's being animated with robotics. It's being animated with artificial intelligence"

04:12PM EDT - "Everything is homemade"

04:13PM EDT - Rolling a demo reel of GPU-accelerated applications

04:13PM EDT - Warp, PhysX Flow, Photoshop, and more

04:15PM EDT - NVIDIA's GPUs are only worth using because of the software written for them. So NVIDIA has always made it a point to showcase what the software development community has been up to

04:15PM EDT - "General purpose computing has run out of steam"

04:15PM EDT - Need new ways to keep growing computing. Keep consuming computing

04:16PM EDT - It's not about driving down the cost, it's about driving up the scale

04:16PM EDT - Jensen would like to simulate everything they do in digital twin virtual environments

04:17PM EDT - NVIDIA will have partners joining them today

04:17PM EDT - Jensen will be announcing several important partnerships

04:17PM EDT - Ansys

04:18PM EDT - Sysopsys. NVIDIA's very first software partner. Literally

04:18PM EDT - Accelerating computational lithography

04:18PM EDT - TSMC is announcing today that they're going into production with cuLitho

04:19PM EDT - Cadence, the EDA tool maker, is joining Club GPU as well

04:20PM EDT - Going to connect Cadence's digital twin platform to Omniverse

04:20PM EDT - Omniverse will be the fundamental operating system for digital twins

04:20PM EDT - Now recapping the history of large language models, and the hardware driving them

04:21PM EDT - Doubling the parameter count requires increasing the token count

04:21PM EDT - 1.8T parameters is the current largest model. This requires several billion tokens to train

04:22PM EDT - Even a PetaFLOP GPU would take 30 billion seconds to train that model. That's 1000 years

04:22PM EDT - "What we need are bigger GPUs" "Much much bigger GPUs"

04:22PM EDT - The answer is to put a whole bunch of GPUs together

04:23PM EDT - As well as developing technologies like NVLink and tensor cores

04:23PM EDT - To help the world build these bigger systems, NVIDIA has to build them first

04:24PM EDT - "We need even larger models"

04:26PM EDT - "We're going to have to build even bigger GPUs"

04:26PM EDT - And here's Blackwell. "A very, very big GPU"

04:27PM EDT - Named after David Backwell, the mathematician and game theorist

04:27PM EDT - Now rolling a video

04:27PM EDT - Two dies on one package, full cache coherency

04:27PM EDT - 208B transistors

04:27PM EDT - 192GB HBM3E@8Gbps

04:27PM EDT - 1.8TB/sec NVLink bandwidth per chip

04:28PM EDT - Available as an accelerator and as a Grace Blackwell Superchip

04:28PM EDT - NVLink 5. Which comes with a new switch chip

04:29PM EDT - And NVDIA is building a rack-scale offering using GB200 and the new NVLink opertions, GB200 NVL72

04:29PM EDT - NVLink 5 scales up to 576 GPUs

04:29PM EDT - "Blackwell is not a chip. It's the name of a platform"

04:30PM EDT - Holding up Blackwell next to a GH100 GPU

04:30PM EDT - "It's okay, Hopper"

04:30PM EDT - "You're very good. Good girl"

04:31PM EDT - 10TBps link between the dies

04:31PM EDT - No memory locality issues or cache issues. CUDA sees it as a single GPU

04:32PM EDT - The chip goes into two types of systems. B100, which is designed to be drop-in compatible with H100/H200 HGX systems

04:33PM EDT - Jensen showing off Blackwell boards... and trying not to drop them

04:33PM EDT - GB200. 1 Grace CPU + 2 Blackwell GPUs (4 GPU dies)

04:33PM EDT - GB200 is memory coherent

04:34PM EDT - "This is s miracle"

04:34PM EDT - NVLink on top, PCIe Gen 6 on the bottom

04:34PM EDT - And that's Grace Blackwell

04:34PM EDT - "But there's more!"

04:35PM EDT - "We need a whole lot of new features"

04:35PM EDT - Second generation transformer engine

04:35PM EDT - Automaticaly and dynamically recast numerical formats to a lower precision when possible

04:36PM EDT - AI is about probability. Need to figure out when you can use lower precisions and when you can't

04:36PM EDT - Support down to FP4

04:36PM EDT - 5th generation NVLink

04:38PM EDT - Jensen is also pushing the importance of reliability. No massive cluster is going to stay up 100% of the time, especially running weeks on end

04:38PM EDT - So NVIDIA has added a RAS engine that can do a full, in-system self-test of a node

04:39PM EDT - Security remains a popular subject as well. Protecting the results of a model training, protecting the model itself. Which means supporting full speed encryption throughout

04:40PM EDT - Blackwell also adds a line decompression engine that can sustain 800GB/sec of transfers

04:40PM EDT - New: FP4 support. FP6 as well

04:41PM EDT - Meanwhile FP8 performance is 2.5x that of Hopper

04:41PM EDT - FP4 gains are even greater, since Hopper can't process FP4 natively

04:42PM EDT - "The future is generative"

04:43PM EDT - 5x the inference/token generation ability of Hopper

04:44PM EDT - But why stop there?

04:44PM EDT - Meanwhile there's also the new NVLink chip. 50B transistors, built on TSMC 4NP

04:45PM EDT - Connecting GPUs to behave as one giant GPU

04:46PM EDT - Which helps NVIDIA build the DGX GB200 NVL72

04:46PM EDT - 18 rack units of GB200 nodes, each node with 2 GB200s

04:46PM EDT - 9 rack units of NVSwitches

04:47PM EDT - This gives the GB200 NVL 720 PFLOPS (sparse) of FP8 throughput

04:47PM EDT - 1 EFLOPS of FP4 in one rack

04:48PM EDT - 5000 NVLink cables. 2 miles of cables

04:48PM EDT - And those are all copper cables. No optical transceivers needed

04:48PM EDT - That saved 20kW to be spent on computation

04:50PM EDT - Weight: 3000 pounds (prime, many more pounds)

04:51PM EDT - Traijning GPT-MoE-18.T would take 90 days on a 8000 GPU GH100 system consuming 15W

04:51PM EDT - GB200 NVL72 can do it on 2000 GPUs with 4MW of power

04:52PM EDT - Inference of LLMs is a challenge due to their size. They don't fit on one GPU

04:54PM EDT - Throughput is everything.

04:57PM EDT - Now looking at Blackwell vs Hopper

04:58PM EDT - And this is where FP4 and NVLink switch come in

04:59PM EDT - "This ability is super, super important"

05:00PM EDT - NVIIDA started with two customers and has many now:

05:02PM EDT - AWS, Google/GCP, Microsoft/Azure, Oracle are all on board

05:04PM EDT - Now for a demo of Wistron, one of NV's partners, who is building a digital twin of their factory using NVIDIA's technology

05:05PM EDT - The digital twin allows them to test operations and look for ways to improve

05:05PM EDT - And that's Wistron

05:06PM EDT - "That's what happens when you don't reherse"

05:07PM EDT - Recapping Alexnet in 2012, identifying cats

05:08PM EDT - AI is changing how software is written. Don't write code to teach a computer to identify a cat, create an AI model and train that to ID cats

05:09PM EDT - "Anything you can digitize, so long as there are patterns, you can learn from it"

05:10PM EDT - (We've now segued to software)

05:11PM EDT - Now on to Earth-2, NVIDIA's simulation for AI-powered weather forecasting and climatology

05:12PM EDT - CoreDiff to improve the resolution of weather models from 25km to 2km

05:12PM EDT - Generative supersampling for weather prediction

05:13PM EDT - For Taiwan right now, coming to the rest of the world later. Working with The Weather Company

05:14PM EDT - Now on to healthcare and NVIIDA

05:15PM EDT - Now rolling another video for computational drug testing

05:16PM EDT - BioNEMO

05:16PM EDT - NVIDIA has acess to a massive number of models across all fields

05:16PM EDT - But they think they're still too hard for companies to use

05:17PM EDT - "Inference is an extraordinary computation problem"

05:17PM EDT - "We're going to invent a new way to receive and operate software"

05:17PM EDT - NVIDIA Inference Microservice. A NIM

05:18PM EDT - A pre-trained model packaged and optimized to run across NVIDIA's install base

05:18PM EDT - Packaged with all the software needed to run it. CUDA libraries, APIs, etc

05:19PM EDT - Basically containerized AI software packages that are optimized for NV GPUs and come with a simple API to access them

05:19PM EDT - NIMs are getting their own landing page: https://ai.nvidia.com

05:19PM EDT - The future will be assembling a team of AIs to handle a task

05:21PM EDT - NVIDIA already has NIMs running across the company

05:21PM EDT - Including a chatbot for building chips. Co-designers with NVIDIA's engineers

05:22PM EDT - NVIDIA ChipNeMo, based on Llama 2 70B

05:22PM EDT - Asking it what a CTL is - Compute Trace Library

05:24PM EDT - "We are effectively an AI foundry"

05:24PM EDT - Comparing NVIDIA to TSMC

05:24PM EDT - Three pillars: NIMs, microservices, and the DGX cloud

05:27PM EDT - "The enterprise IT industry is sitting on a goldmine"

05:28PM EDT - They're sitting on a lot of data that could be turned into co-pilots

05:30PM EDT - "When you're ready to run these AI chatbots, you'll need an AI factory"

05:30PM EDT - Plugging partner Dell

05:31PM EDT - Now how to get AI to understand the physical world?

05:32PM EDT - It will take 3 computers. A traditional AI system, an OVX Omniverse system, and an AGX system for robitcs

05:35PM EDT - Discussing how all of these systems will interact. Ultimately robotics need a simulated environmnet to be trained in

05:35PM EDT - Video real time. All of this is running in real-time

05:36PM EDT - Demoing robots fed with sensor data, being controled by a higher AI

05:37PM EDT - "We've made Omniverse a lot easier to access"

05:38PM EDT - NVIDIA really wants to sell industrial customers on Omniverse Cloud

05:39PM EDT - NVIDIA has been partnering with Siemens on this

05:40PM EDT - Siemens is connecting their crown jewel accelerator to Omniverse

05:44PM EDT - Announcing today: Omniverse Cloud streams to Apple's Vision Pro headset

05:45PM EDT - Now on to automotive

05:45PM EDT - At the beginning of next year, NV will be shipping in Mercedes and other brands

05:46PM EDT - Drive Thor: World's only ASIL-D system that can run AI

05:46PM EDT - BYD will be adopting Drive Thor

05:47PM EDT - NVIDIA's Jetson platform for robotics is getting an update as well

05:47PM EDT - Jetson Orin

05:47PM EDT - New SDK to go with it: Isaac Perceptor

05:48PM EDT - Vision reconstruction and depth perception

05:49PM EDT - Plugging use of NV technologies in robotic arms, as well. And all the software they provide to help set this up

05:50PM EDT - And Jetson Thor? For humanoid robotics

05:50PM EDT - NVIDIA is making a new set of models and a software stack to teach humanoid robots

05:51PM EDT - Groot, which runs on Thor

05:51PM EDT - Now rolling a video on humanoid robotics/Groot

05:53PM EDT - Project Groot: A general purpose foundational model

05:54PM EDT - Learn human movement by observing them

05:54PM EDT - Powered by Jetson Thor platform

05:55PM EDT - And Jensen is lining up with robots

05:55PM EDT - General Robotics 0 0 Three (GROOT)

05:56PM EDT - And some other Jetson-based robots have wandered on stage

05:56PM EDT - Learned to walk in Isaac Sim

05:56PM EDT - Disney robots, it seems

05:57PM EDT - Pro tip: don't use "stop" with robots

05:57PM EDT - Now for the recap of today's keynote

05:58PM EDT - A new industrial revolution, powered by generative AI, with Blackwell at its core

05:58PM EDT - New computer creates new kind of software, which means new means of distributing that software. NIMs

05:59PM EDT - And in the world of robotics, all of these systems need a platform, and that is Omniverse

06:00PM EDT - This is what we announced to you today

06:00PM EDT - HGX B100, NVLink Switch, GB200 Superchip Compute Node, Quantum X800 Switch, Spectrum X800 Switch

06:01PM EDT - "One more thing to show you"

06:01PM EDT - Rolling a final video

06:03PM EDT - "Powering the new era of computing: NVIDIA"

06:03PM EDT - And that's a wrap for us. Our initial Blackwell story is up now, with more details to come once I can get more bandwidth