CEDoc - UM6P - Ai Movement : Reconstruction and Generation of Time-Varying 3D Scenes (14323)

Located in the heart of the Kingdom's capital at Technopolis, University Mohammed VI Polytechnic (UM6P), is an internationally-oriented higher education institution, which is committed to an educational system based on the highest standards of teaching and research in fields crucial to the sustainable economic development of Morocco and Africa.

Ai movement, the International centre for Artificial Intelligence in Morocco, attached to UM6P, is a center of excellence in Artificial Intelligence, which aims to foster the emergence of Moroccan and African know-how in Artificial Intelligence and Data Science. It is UNESCO's reference center for artificial intelligence in Africa. Ai movement's goal is to develop a trustworthy, ethical, responsible and inclusive AI on the African continent.

1 Motivations & Problem Statement

Building models of the real world, including their 3D geometry, visual appearance, and motion is fundamental to both computer graphics and computer vision. This ability makes it possible to synthesize photorealistic novel views, which play a vital role in applications like filmmaking, augmented reality, and virtual reality. It also significantly eases content creation for AR/VR and video games by avoiding laborious manual design processes. Moreover, accurate reconstruction is essential for intelligent systems that need to interpret and respond to real-world environments and human actions in a safe and meaningful way. However, since the real world is inherently dynamic, reconstructing scenes involving non-rigid, time-varying objects remains a particularly challenging problem. In this thesis project, we aim to develop learning-based approaches for reconstructing dynamic, non-rigid 3D scenes from diverse input modalities, including sparse point clouds and monocular videos. Our goal is to push the boundaries of dynamic scene reconstruction and infer high-quality 3D geometry and motion over time, enable photorealistic rendering, and support downstream applications such as AR/VR content generation and interactive intelligent systems. Specifically, in this thesis project, we aim to explore the following problems:

• Non-rigid Object Deformation. Deforming non-rigid shapes presents a challenge due to the high degree of freedom involved in controlling the shape deformation. To address this challenge, there have been notable advances in 3D object modeling, particularly in representations through parametric models [1, 2]. However, these parametric approaches often depend on fixed templates, making it challenging to accurately model general non-rigid objects that lack consistent topological structures, particularly when capturing their 4D dynamics. Alternatively, modelfree approaches [3, 4] represent a significant advance by using coordinate-MLP representations for deformable object reconstruction with arbitrary topologies and non-unified structures. However, these state-of-the-art methods still encounter challenges when facing ambiguous inputs of noisy, sparse, or partial point clouds. In addition, they represent dynamics as a sequence of single latent codes and thus struggle to effectively capture underlying shape and motion priors. In this part of the project, we focus on 4D reconstruction of non-rigid objects without relying on shape or motion priors, leveraging implicit representations for their flexibility in modeling objects with arbitrary topologies.

• Dynamic Scene Reconstruction. While impressive advances have been made in reconstructing static 3D scenes [5, 6, 7, 8] from point cloud or multi-view RGB images, recovering the geometry and motion of complex dynamic 3D scenes involving multiple moving objects remains a major challenge, especially when using monocular video inputs, such as recordings from a standard smartphone [9, 10]. A major trend in the recent literature is to use neural implicit fields to model 4D scenes [11, 12]. These models represent the entire time-varying geometry (and often appearance) as a continuous function parameterized by a neural network. However, these approaches require a long training time and are not suitable for real-time rendering. Moreover, they primarily target quasi-static scenes, which do not reflect the complexity of real-world monocular videos. In parallel, there is growing interest in explicit neural point-based models for 4D scenes [10, 13]. these methods show that point-based and hybrid representations (neural and explicit) are now state-of-the-art for dynamic scenes where speed is crucial. Our focus in this part is to explore novel explicit and hybrid spatio-temporal representation for 4D scene reconstruction from a single monocular video.

• Generative Models for 4D Dynamics. Another emerging direction in 4D dynamic reconstruction involves the use of diffusion models and latent generative representations [14, 15]. This approach aims to learn shape and motion diffusion priors that can handle ambiguous observations, such as sparse, noisy, or partial point clouds. However, a significant challenge lies in the tendency of current diffusion models to adapt 3D architectures directly to 4D by processing each frame independently, which can lead to discontinuities in both temporal and spatial consistency. A promising path to address this limitation is to explore more compact latent generative models or 4D implicit representations that better capture spatio-temporal coherence. While previous work has applied diffusion models to specific cases—such as head motion synthesis [15]—our goal is to extend their application to a broader class of deformable surfaces in general non-rigid objects, leveraging latent generative models with implicit representations.

2 Admission Criteria

The PhD position is offered by the International Center for Artificial Intelligence of Morocco (Ai movement) at Mohammed VI Polytechnic University, in collaboration with the 3D Computer Vision team at Inria Rennes in France. Applicants with excellent cursus must be holders of a Master's, an engineering or an equivalent recognized degree in Computer Science or Applied Mathematics. In addition, they should have skills in Programming (Python and C ) and strong communication skills in English. Particular attention will be given to the alignment of the applicant's background with the objectives of this research project.

Ai movement – UM6P

Université Mohammed VI Polytechnique de Rabat

Technopolis, Rocade de Rabat, Maroc