NPMs: Neural Parametric Models

for 3D Deformable Shapes

Technical University of Munich

ICCV 2021

NPMs, composed of learned, latent spaces of shape and pose, enable test-time optimization over the spaces to fit to new observations.

Abstract

Parametric 3D models have enabled a wide variety of tasks in computer graphics and vision, such as modeling human bodies, faces, and hands. However, the construction of these parametric models is often tedious, as it requires heavy manual tweaking, and they struggle to represent additional complexity and details such as wrinkles or clothing.

To this end, we propose Neural Parametric Models (NPMs), a novel, learned alternative to traditional, parametric 3D models, which does not require hand-crafted, object-specific constraints. In particular, we learn to disentangle 4D dynamics into latent-space representations of shape and pose, leveraging the flexibility of recent developments in learned implicit functions. Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit new observations, similar to the fitting of a traditional parametric model, e.g., SMPL. This enables NPMs to achieve a significantly more accurate and detailed representation of observed deformable sequences.

We show that NPMs improve notably over both parametric and non-parametric state of the art in reconstruction and tracking of monocular depth sequences of clothed humans and hands. Latent-space interpolation as well as shape / pose transfer experiments further demonstrate the usefulness of NPMs.

Video

Latent-space Interpolation

Our latent spaces of shape and pose can be traversed to obtain novel shapes and poses. Use the sliders to linearly interpolate between left and right frames.

Shape Space

Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame


Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame


Pose Space

Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame


Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame

Retargeting

NPMs enable reposing a given identity (left) with the motion recovered from an input monocular sequence (right).


Publication

BibTeX

@article{palafox2021npms
    author        = {Palafox, Pablo and Bo{\v{z}}i{\v{c}}, Alja{\v{z}} and Thies, Justus and Nie{\ss}ner, Matthias and Dai, Angela},
    title         = {NPMs: Neural Parametric Models for 3D Deformable Shapes},
    booktitle     = {ICCV 2021},
    year          = {2021},
}