IEEE Geoscience and Remote Sensing Magazine - March 2019 - 19

point cloud-based segment features (e.g., height percentiles and average height) and also spectral segment features
(e.g., mean and median spectra). Using nearest-neighbor
estimation, it was possible to determine the variables of
diameter at breast height, tree height, health status, and
tree species for each crown segment. In the second step,
the biodiversity indicators-structural complexity, extent
of deciduous trees, and number of dead trees-were derived using single tree variables as input.
Considering multiple sensors, hyperspectral and lidar
data were fused in an approach proposed by Man et al.
[123] for urban land use classification (15 classes) with a
combined pixel and feature-level method. Lidar point cloud
information was encapsulated in image layers. Furthermore, the authors aimed at assessing the contribution of
lidar intensity and height information, particularly for the
classification of shadow areas. Their methodology included
pixel-based features, such as the nDSM and intensity image
from lidar, along with the inverse minimum noise fraction
rotation bands, NDVI, and texture features (GLCM) of HSI
data. The derived features were input to a supervised, pixelbased classification (SVM and maximum-likelihood classifiers). Additionally, an edge-based segmentation algorithm
was used to derive segments using lidar nDSM, intensity,
and NDVI images; this was followed by a rule-based classification of the derived objects. The classification outputs
of the pixel- and object-based methods were merged by GIS
raster calculation. The combination of HSI and lidar increased overall accuracy by 6.8% (to 88.5%) compared with
HSI classification alone. The joint pixel- and object-based
method increased overall accuracy by 7.1%, to 94.7%.
Liu et al. [127] used HSI and airborne lidar data as complementary data sources for crown structure and physiological tree information to map 15 different urban tree species.
First, crowns were segmented by watershed segmentation
of the canopy height model. Second, lidar and hyperspectral features were extracted for the crown segments to be
used in the subsequent segment-based random forest (RF)
classification. The 22 lidar-derived crown structural features per segment included, for example, crown shape,
laser return intensity, and laser point distribution. The researchers concluded that the combination of lidar and HSI
increased single-source classification up to 8.9% in terms
of overall accuracy.
A complex fusion strategy for lidar point cloud and HSI
image data in a two-stage neural network classification was
developed by Rand et al. [128]. First, spectral segmentation
of the HSI data was performed by a stochastic expectationmaximization algorithm and spatial segmentation of the
lidar point cloud with a combined mean-shift and dispersion-based approach. Second, the resulting segments from
lidar and HSI data were input to a supervised cascaded neural network to derive the final object class labels. The final
fusion classification map was produced in 3D using the
elevation values from the lidar point cloud. The team's approach resulted in a large increase in overall classification
march 2019

ieee Geoscience and remote sensing magazine

accuracy by multisource fusion (HSI and lidar) to 98.5%,
compared with 74.5% overall accuracy with HSI input only.
CHALLENGES AND TRENDS IN POINT
CLOUD FUSION
Generally, we can see a large gain in the importance of
point clouds. Multisource fusion including point clouds
is already used in a wide variety of application fields [104]
and reveals several trends
◗ There is increasing use of machine-learning methods,
including point clouds or point cloud derivatives.
◗ The majority of current approaches transform and encapsulate 3D point cloud information into 2D images
or voxels and perform fusion and analysis on images or
objects. The derived classification labels are transferred
back to points afterward.
◗ The implementation of fusion (or joint use) for spectral
and 3D point cloud information from single-source
photogrammetry (Structure from Motion and dense image matching) exists now. The link between point clouds
and images is already given via several methodologies.
◗ The fusion of geometric and backscatter point cloud information from lidar exhibits increases in terms of classification accuracy.
Future research on multisource fusion with point clouds
will need to address the combination of point clouds from
different sources and with strongly heterogeneous charac te r istics (e.g., point density and 3D accuracy). So far,
mainly one source of point
ThE UsE OF
clouds is used in the fusion
mULTiTEmpOrAL
process, e.g., the joint use of
inFOrmATiOn is crUciAL
HSI and lidar point clouds.
FOr mAny impOrTAnT
Multispectral [129] and even
AppLicATiOns (FrOm ThE
hyperspectral lidar data [130]
offer new possibilities for
AnALysis OF sLOw AnD
the fusion of point clouds as
smOOThLy EvOLvinG
well as of point clouds with
phEnOmEnA TO sTEEp
multispectral/hyperspectral
AnD AbrUpT chAnGEs).
data. The availability of 3D
point cloud time series [110]
will also enable investigation
of how temporal aspects need to be addressed in fusion and
classification approaches.
The number of contributions on HSI and lidar rasterized data fusion in the remote sensing community is quickly growing because of the complementary nature of such
multisensor data. Therefore, the next section is specifically
dedicated to the fusion of HSI and lidar-derived features
and offers a review of such fusion schemes.
hypErspEcTrAL AnD LiDAr
The efficacy of lidar (characterized as an active remotesensing technique) for the classification of complex areas
(e.g., where many classes are located close to each other)
is limited by the lack of spectral information. On the other
19

IEEE Geoscience and Remote Sensing Magazine - March 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - March 2019