Understanding Nvidia Omniverse, which wowed everyone at SIGGRAPH - AI4kids

Game Changer - Nvidia Omniverse

Preface

Nvidia has recently appeared frequently in various media and news, sparking heated discussions around the world. The reason is that its own open source software Omniverse won the Best Presentation Award at the most prestigious annual computer graphics conference SIGGRAPH 2021 with its jaw-dropping results.

This article will take readers to look at Nvidia’s three award-winning case studies. Omniverse integrates “virtual environment simulation” and “AI” to cross over into different fields and drive process innovation.

Metaverse

The Metaverse is “a virtual environment where you can be present with people in digital spaces”

Zuckerberg Tweet

Do you remember the virtual world where the protagonist roamed in the movie "Ready Player One"? Humans have built a completely virtual world. In this world, in addition to being able to reproduce everything in the real world, things that do not exist in reality can also be realized; the creator has all the control over this world like the creator.

"Metaverse", as one of the key areas of future technological development, will significantly change the current social model. This article will briefly introduce Omniverse, the open source software of Nvidia, the world's first simulation platform built on the concept of Metaverse, and how it allows users to use AI tools to shape and simulate the world they design and influence the development of various fields.

About Omniverse

NVIDIA Omniverse is a digital creation platform based on CUDA and RTX ray tracing technology. In addition to being able to easily build lifelike digital 3D models and simulate real-world physical phenomena, all presentations are almost instant thanks to CUDA's powerful parallel computing capabilities.

In addition to rendering and simulation, Omniverse also integrates many practical "AI tools" to accelerate the digital creation process; this initiative is undoubtedly the reason why it won the Best of Show Award. We will learn about these AI tools in this article.

Demo-1 Lifelike digital avatar

What made Nvidia so popular at SIGGRAPH 2021 was the release of behind-the-scenes footage of CEO Huang Renxun's speech at GTC 2021; one of the clips was a speech given by an extremely realistic digital avatar instead of a real person.

At the GTC 2021 conference, almost no one noticed that Huang Renxun in front of them was actually a virtual 3D character. Therefore, once the behind-the-scenes footage was released, it immediately dominated the Internet and media coverage, and also won the Best Presentation Award of the conference.

Project technical process analysis

This project involved the development of 34 CG artists and 12 researchers, and was a massive undertaking. Most of the projects adopt the standard process of 3D animation industry, including 3D modeling, skeleton binding, animation production, rendering, etc., combined with Omniverse's RTX ray tracing technology, the rendering effect is amazing.

Digital avatar model building

In order to create a digital double of CEO Huang Renxun, the technology used was "Photogrammetry". Through a large number of cameras, Huang Renxun was photographed from different angles, and his 3D shape was reconstructed using computer vision algorithms.

After the model is created, according to the traditional process of the 3D industry, control items must be established for the 3D character to drive the shape changes between the character's movements and body appearance. Control items are usually divided into body and face, and the process requires experienced 3D artists to spend a lot of time on manual settings.

AI Flip Image Creation Process

Audio2face — Voice-driven digital human expression generation

In the process of setting up and animating the facial controls of digital avatars, Omniverse has overturned the traditional manual work method and replaced it with a built-in AI tool called "audio2face". By inputting a user-entered voice, it can immediately generate facial expressions and facial muscle movements for the 3D digital character.

Face-vid2vid — Generate photorealistic face animations

The facial expressions generated by audio2face are very detailed, but lack realism, so the artist collaborated with the Nvidia AI team to use the deep learning generative model "face-vid2vid" to transform the originally unrealistic facial animations into a lifelike Huang Renxun himself.

face-vid2vid face style transfer

Use the results generated by face-vid2vid for face synthesis and replacement

At this point, Omniverse's AI has proven to be powerful enough to replace the tedious process of traditional animation. The digital character body animation follows the traditional method of recording the character's movements through motion capture and having Huang Renxun's digital stand-in perform the same movements.

Capturing the actors’ movements

Display the dynamic capture results in Omniverse

Allow digital stand-in actors to perform the same movements as the captured person

Finally, through the RTX ray tracing technology in Omniverse, the performance of the digital stuntman presents extremely realistic results. Huang Renxun's facial animation generated by Face-vid2vid and the character's body animation produced by Omniverse were finally combined in the synthesis software to produce a fake CEO speech clip at the GTC 2021 conference.

Conclusion

Nvidia's demonstration this time sounded the first bell of change for the image creation page, proving that the innovative approaches of AI and technology can replace the already mature standard processes in the field of image creation; and even be more efficient to a certain extent.

Want to learn more? Click the button for more information!