Intro Video


Abstract

With the advent of deep neural networks, learning-based single-view reconstruction has gained popularity. However, in 3D, there is no absolutely dominant representation that is both computationally efficient and accurate yet allows for reconstructing high-resolution geometry of arbitrary topology. After all, the accurate implicit methods are time-consuming due to dense sampling and inference, while volumetric approaches are fast but limited to heavy memory usage and low accuracy. In this paper, we propose VIPNet, an end-to-end hybrid representation learning for fast and accurate single-view reconstruction under sparse implicit point guidance. Given an image, it first generates a volu- metric result. Meanwhile, a corresponding implicit shape representation is learned. To balance the efficiency and accuracy, we adopt PointGenNet to learn some representative points for guiding the voxel refinement with the corresponding sparse implicit inference. A strategy of patch-based synthesis with global-local features under implicit guidance is also applied for reducing memory consumption required to generate high-resolution output. Extensive experiments demonstrate the effectiveness of our method both qualitatively and quantitatively, which indicates that our proposed hybrid learning outperforms separate representation learning. Specifically, our network not only runs 60 times faster than implicit methods but also contributes to accuracy gains. We hope it will inspire a re-thinking of hybrid representation learning.


Poster

Poster

Result

Qualitative Results


Qualitative comparisons with the state-of-the-art. Given the same single-view image as input, the reconstruction are acquired with (a) 3D-R2N2 (in $64^3$); (b) PSG; (c) Pixel2Mesh; (d) AtlasNet; (e) and (f) OccNet in $64^3$ and $256^3$; (g) and (h) Ours in $64^3$ and $256^3$; (i) Ground truth.


Quantitative Results

Quantitative comparisons of our method with the state-of-the-art. The IoU is calculated in 643, while CD and EMD are calculated on 1, 024 uniformly sampled points from output surfaces. The higher is better for IoU, while the lower is better on CD and EMD metrics.
Quantitative comparisons of our method with the state-of-the-art. The IoU is calculated in $64^3$, while CD and EMD are calculated on 1,024 uniformly sampled points from output surfaces. The higher is better for IoU, while the lower is better on CD and EMD metrics.