IEEE Geoscience and Remote Sensing Magazine - December 2020 - 48

The general procedure of the individual classification for
PCSS is well described in [95]. As Figure 3 shows, the procedure
entails four stages: neighborhood selection, feature extraction,
feature selection, and semantic segmentation. For each stage,
[95] summarized several crucial methods and tested different
methods on two data sets to compare their performance. According to the authors' experiment, in individual PCSS, the
random forest classifier had a good tradeoff between accuracy and efficiency on two data sets. It should be noted that
the authors of [95] used a so-called deep-learning classifier in
their experiments, but that is an old neural network appearing
in the time of regular machine learning, not the recent deeplearning methods described in the section " Deep Learning. "
Since individual PCSS does not consider the contextual features of points, individual classifiers work efficiently but generate unavoidable noise that causes unsmooth PCSS results. Statistical context models can mitigate this problem. CRF is the
most widely used context model in PCSS. Niemeyer et al. [87]
provided a very clear introduction about how CRF has been
used on PCSS and tested several CRF-based approaches on the
Vaihingen data set. Based on the individual PCSS framework
[95], Landrieu et al. [97] proposed a new PCSS framework that
combines individual classification and context classification.
As shown in Figure 4, a graph-based contextual strategy in this
framework was introduced to overcome the noise problem of
initial labeling, establishing a process called structured regularization or smoothing.
For the regularization process, Li et al. [111] used a multilabel graph-cut algorithm to optimize the initial segmentation
result from the SVM. Landrieu et al. [97], comparing various
postprocessing methods, proved that regularization indeed
improved the accuracy of PCSS.
DEEP LEARNING
Deep learning is the most influential and fastest-growing current technique in pattern recognition, computer vision, and
data analysis [179]. As its name indicates, deep learning uses
more than two hidden layers to obtain high-dimension features
from training data, while traditional handcrafted features are designed with domain-specific knowledge. Before being applied
in 3D data, deep learning appeared as an effective power in a
variety of tasks in 2D computer vision and image processing,
such as image recognition [180], [181], object detection [182],
[183], and semantic segmentation [184], [185]. It has been attracting more interest in 3D analysis since 2015, driven by the

N×d
3D Point
Cloud

multiview-based idea proposed by [186] and the voxel-based
3D convolutional neural network (CNN) proposed by [187].
Standard convolutions originally designed for raster images
cannot easily be directly applied to PCSS, as the point cloud is
disordered and unstructured. (Unstructured is sometimes expressed as irregular or nonraster.) Thus, to solve this problem,
the raw point cloud must be transformed. Depending on the
format of the data ingested into neural networks, deep-learning-based PCSS approaches can be sorted into three categories:
multiview based, voxel based, and point based.
MULTIVIEW BASED
One of the early ways to apply deep learning in 3D was with
dimensionality reduction. With this method, the 3D data are
represented by multiview 2D images, which can be processed
based on 2D CNNs. Subsequently, the classification results
can be restored to 3D. The most influential multiview deep
learning in 3D analysis is multiview CNN (MVCNN) [186].
Although no experiments were performed on PCSS using the
original MVCNN algorithm, it is a good example for learning
about the multiview concept.
The multiview-based methods have solved the structuring
problems of point cloud data well, but there are two serious
shortcomings in these methods. First, they bring about many
limitations and a loss in geometric structures, as 2D multiview
images are just an approximation of 3D scenes. As a result, the
performance of such complex tasks PCSS could be unsatisfactory.
Second, multiview projected images must cover all spaces containing points. For large, complex scenes, it is difficult to choose
enough proper viewpoints for multiview projection. Thus, few
studies have used multiview-based deep-learning architecture for
PCSS. One exception is SnapNet [9], [67], which uses full data set
Semantic-8 of Semantic3D.net as the test data set. Figure 5 depicts
the workflow of SnapNet. In SnapNet, the preprocessing step
aims at decimating the point cloud, computing point features,
and generating a mesh. Snap generation uses various virtual cameras to generate RGB images and depth composite images of the
mesh. Semantic labeling realizes image semantic segmentation
from the two input images by image deep learning. In the last
step, 2D semantic segmentation results are projected back to 3D
space, thereby enabling the acquisition of 3D semantics.
VOXEL BASED
Combining voxels with 3D CNNs is the other approach used
early on in deep-learning-based PCSS. Voxelization solves

N×d
N × d′

Neighborhood
Selection

Feature
Extraction

Feature
Selection

Supervised
Classification

7 Approaches

21 Features

7 Approaches

10 Classifiers

Labeled
3D Point Cloud

FIGURE 3. The PCSS framework described by [95]. The term semantic segmentation in our review is defined as " supervised classification "
in [95]. (Source: [95]; reprinted with permission from Elsevier.)

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

DECEMBER 2020

http://www.Semantic3D.net

IEEE Geoscience and Remote Sensing Magazine - December 2020

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2020