SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

ACM Transactions on Graphics (Proc. SIGGRAPH Asia), December 2020

Yifei Shi, Junwen Huang, Hongjia Zhang,
Xin Xu, Szymon Rusinkiewicz, Kai Xu

We propose an end-to-end deep neural network learned to predict both reflectional and rotational symmetries from single-view RGB-D images. For each example, we show the input RGB-D images with the object of interest segmented out (see the yellow masks) as well as the detection results over the unprojected 3D point clouds. Reflectional symmetries are depicted with red planes (reflection plane) and rotational symmetries with green lines (rotation axis). Note how our method is able to detect the composition of an arbitrary number of symmetries, possibly of different types, present in the same object.

We study the problem of symmetry detection of 3D shapes from single-view RGB-D images, where severely missing data renders geometric detection approach infeasible. We propose an end-to-end deep neural network which is able to predict both reflectional and rotational symmetries of 3D objects present in the input RGB-D image. Directly training a deep model for symmetry prediction, however, can quickly run into the issue of overfitting. We adopt a multi-task learning approach. Aside from symmetry axis prediction, our network is also trained to predict symmetry correspondences. In particular, given the 3D points present in the RGB-D image, our network outputs for each 3D point its symmetric counterpart corresponding to a specific predicted symmetry. In addition, our network is able to detect for a given shape multiple symmetries of different types. We also contribute a benchmark of 3D symmetry detection based on single-view RGB-D images. Extensive evaluation on the benchmark demonstrates the strong generalization ability of our method, in terms of high accuracy of both symmetry axis prediction and counterpart estimation. In particular, our method is robust in handling unseen object instances with large variation in shape, multi-symmetry composition, as well as novel object categories.

PDF file

PDF file

Yifei Shi, Junwen Huang, Hongjia Zhang, Xin Xu, Szymon Rusinkiewicz, and Kai Xu.
"SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images."
ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 39(6), December 2020.

@article{Shi:2020:SLT,
   author = "Yifei Shi and Junwen Huang and Hongjia Zhang and Xin Xu and Szymon
      Rusinkiewicz and Kai Xu",
   title = "{SymmetryNet}: Learning to Predict Reflectional and Rotational
      Symmetries of {3D} Shapes from Single-View {RGB-D} Images",
   journal = "ACM Transactions on Graphics (Proc. SIGGRAPH Asia)",
   year = "2020",
   month = dec,
   volume = "39",
   number = "6"
}