Generative Modeling of 3D Objects and Scenes with Neural Representations

02/06/2023 to 02/06/2023

Abstract: Generative AI has revolutionized many areas not only of computer science but also our everyday lives, and the amount of generative AI-powered tools we will use in the future continues to grow. In particular large language models (LLMs) and large-scale diffusion models have achieved generation capabilities of unprecedented qualities. However, these models are restricted to text and 2D images, while our everyday lives occur in 3D space and there is growing interest in Generative AI in 3D. This includes generative applications for 3D content in the AR/VR domain, for movies, in the domain of autonomous driving, and many more. Despite recent advances in this domain, generative 3D modeling requires further research to meet existing demands. To this end, we propose a research collaboration with a leading researcher in the field of 3D generative AI with the goal of exploring novel generative 3D modeling approaches. More specifically, we want to investigate the generation and editing of 3D objects and scenes from various data modalities, including 2D as well as 3D data. We first explore how small to mid-sized datasets of 3D shapes in can be combined with the abundance of 2D image data available to train photorealistic 3D (latent) diffusion models of shapes across a wide set of object categories (eg, ImageNet) which is in contrast to most exist models that focus only on few categories. In a second step, we aim to explore how this approach can be extended to the generation of scenes and 3D structure with physically-based materials that enable photo-realistic relighting. Finally, we will investigate how context-based editing and stylization can be achieved with our technology, enabling new creative workflows for 3D artists and designers.