Jiazhao Zhang*, Chenyang Zhu*, Lintao Zheng and Kai Xu
,"Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation", CVPR 2020. *joint first author
Online semantic 3D segmentation in company with realtime RGB-D reconstruction poses special challenges such as how to perform 3D convolution directly over the progressively fused 3D geometric data, and how to smartly fuse information from frame to frame.
We introduce AdaCoSeg, a deep neural network architecture for adaptive co-segmentation of a set of 3D shapes represented as point clouds. Differently from the familiar single-instance segmentation problem, co-segmentation is intrinsically contextual: how a shape is segmented can vary depending on the set it is in. Hence, our network features an adaptive learning module to produce a consistent shape segmentation which adapts to a set.
We propose a novel approach to robot-operated active understanding of unknown indoor scenes, based on online RGBD reconstruction with semantic segmentation. In our method, the exploratory robot scanning is both driven by and targeting at the recognition and segmentation of semantic objects from the scene. Our algorithm is built on top of the volumetric depth fusion framework (e.g., KinectFusion) and performs real-time voxel-based semantic labeling over the online reconstructed volume. The robot is guided by an online estimated discrete viewing score field (VSF) parameterized over the 3D space of ...
Fenggen Yu, Kun Liu, Yan Zhang, Chenyang Zhu and Kai Xu
, "PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation", CVPR 2019.
Deep learning approaches to 3D shape segmentation are typically formulated as a multi-class labeling problem. Existing models are trained for a fixed set of labels, which greatly limits their flexibility and adaptivity. We opt for topdown recursive decomposition and develop the first deep learning model for hierarchical segmentation of 3D shapes, based on recursive neural networks. Starting from a full shape represented as a point cloud, our model performs recursive binary decomposition, where the decomposition network at all nodes in the hierarchy share weights. At each node, a node classifier is trained to determine the type (adjacency or symmetry) and stopping criteria of its decomposition ...
We introduce SCORES, a recursive neural network for shape composition. Our network takes as input sets of parts from two or more source 3D shapes and a rough initial placement of the parts. It outputs an optimized part structure for the composed shape, leading to high-quality geometry construction....
We present a method for estimating detailed scene illumination using human faces in a single image. In contrast to previous works that estimate lighting in terms of low-order basis functions or distant point lights, our technique estimates illumination at a higher precision in the form of a non-parametric environment map...
Chenyang Zhu, Renjiao Yi,
Wallace Lira, Ibraheem Alhashim, Kai Xuand Hao Zhang, "Deformation-Driven Shape Correspondence via Shape Recognition", ACM Transactions on Graphics (SIGGRAPH 2017), 36(4): 51, 2017.
Many approaches to shape comparison and recognition start by establishing a shape correspondence. We "turn the table" and show that quality shape correspondences can be obtained by performing many shape recognition tasks. What is more, the method we develop computes a fine-grained, topology-varying part correspondence between two 3D shapes where the core evaluation mechanism only recognizes shapes globally...
Ruizhen Hu, Chenyang Zhu, Oliver van Kaick,
Ligang Liu, Ariel Shamir and Hao Zhang, "Interaction Context (ICON): Towards a Geometric Functionality Descriptor", ACM Transactions on Graphics (SIGGRAPH 2015), 33(4): 83, 2015.
We introduce a contextual descriptor which aims to provide a geometric description of the functionality of a 3D object in the context of a given scene. Differently from previous works, we do not regard functionality as an abstract label or represent it implicitly through an agent. Our descriptor, called interaction context or ICON for short, explicitly represents the geometry of object-to-object interactions...
Kai Xu, Rui Ma,
Hao Zhang, Chenyang Zhu, Ariel Shamir,
Daniel Cohen-Or and Hui Huang, "Organizing Heterogeneous Scene Collections through Contextual Focal Points", ACM Transactions on Graphics (SIGGRAPH 2014), 33(4): 35, 2014.