IEEE Geoscience and Remote Sensing Magazine - December 2017 - 25

ranking reveals that, today, the top performing method is
based on CNNs.
Most stereo methods in this ranking proceed along the
following main steps. First, a stereo correspondence search
is performed by computing a similarity measure between
image locations. This is typically carried out exhaustively
for all possible depth values. Next, the optimal depth values are searched by optimization on the cost value. Different optimization schemes-convex optimization,
local-optimization strategies (e.g., SGM), and probabilities
methods (e.g., MRF inference)-are used. Finally, some
heuristic filtering is typically applied to remove gross outliers (e.g., left-right check).
The pioneering work of Zbontar and LeCun [165] utilized a CNN in the first step of the typical stereo pipeline.
In their work, the authors suggested training a CNN to
compute the similarity measure between image patches
(instead of using normalized cross correlation or the census
transform). This change proved to be significant. Compared
to SGM, which is often considered a baseline method, the
proposed method achieved a significantly lower error rate.
For SGM, the error rate was still 18.4%, whereas for the
matching-cost(MC)-CNN method, the error rate was only
8%. After that, other variants of CNN-based stereo methods have been offered, and the best ranking method today
has an error rate of only 5.9%. Table 1 lists the error rates of
the top-ranking CNN-based methods.
In addition to similarity measures, a typical stereoprocessing pipeline contains other engineered decisions
as well. After creating a so-called cost volume from the
similarity measures, most methods use specifically engineered algorithms to find the depths (e.g., based on neighborhood constraints) and heuristics to filter out wrong
matches. New proposals, however, suggest that these other
steps can also be replaced solely by a CNN. Mayer et al.
[169] offered such a paradigm-shifting design for stereo
processing. In their proposal, the stereo-processing problem is modeled solely as a CNN. The proposed CNN takes
two images of a stereo pair as an input and directly outputs
the final disparity map. A single CNN architecture replaces
all the individual algorithm steps utilized so far. The CNN
of Mayer is based on an encoder-decoder architecture with
a total of 26 layers. In addition, it includes crosslinks between contracting and expanding network parts. To train
the CNN architecture, end-to-end training using ground
truth image-depth map pairs is performed. The fascinating
aspect of the proposed method is that the stereo algorithm
itself can be learned from data only. The network architecture does not define the algorithm, but the data and the
end-to-end training define what type of processing the network should perform.
LARGE-SCALE SEMANTIC 3-D CITY RECONSTRUCTION
The availability of semantics (e.g., the knowledge of what
type of object a pixel in the image represents) through
CNN-based classification is also changing the way that 3-D
DECEMBER 2017

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

TABLE 1. THE TOP-RANKED STEREO METHODS FROM THE
MIDDLEBURY STEREO EVALUATION BENCHMARK AS OF
MAY 2017.
METHOD

BAD PIXEL ERROR RATE %

3DMST [166]

5.92

MC-CNN + TDSR [167]

6.35

LW-CNN [168]

7.04

MC-CNN-acrt [165]

8.08

SGM [164]

18.4

LW-CNN: look wider CNN; TDSR: top-down segmented regression.

information is generated from image data. The traditional
3-D generation process neglected object information: the
3-D data were generated from geometric constraints only,
and image data were treated as pure intensity values without
any semantic meaning.
The availability of semantic information from CNNbased classification now makes it possible to utilize this
information in the 3-D generation process. CNN-based
classification allows one to assign class labels to aerial
imagery with unprecedented accuracy [170]. Pixels in
the images are then assigned labels like vegetation, road,
building, and so on. This semantic information can then
be used to steer the 3-D data generation process. Class
label-specific parameters can be chosen for the 3-D data
generation process.
The latest proposal in this area, however, is a joint reconstruction of 3-D and semantic information (Häne et al.
[171]), where 3-D reconstruction is performed with a volumetric method. The area to be
CNN-BASED
reconstructed is partitioned
CLASSIFICATION ALLOWS
into small cells, the size of
ONE TO ASSIGN CLASS
which define the resolution
LABELS TO AERIAL
of the 3-D reconstruction. The
IMAGERY WITH
reconstruction algorithm now
UNPRECEDENTED
finds the optimal partitioning
ACCURACY.
of this voxel grid into occupied and nonoccupied voxels
that fit to the image data. The
result is a 3-D reconstruction of the scene. The work of Häne
et al. also jointly assigns the 3-D reconstruction to a class
label for each voxel, e.g., vegetation, building, road, and sky.
Each generated 3-D data point now also has a class label. The
3-D reconstruction is semantically interpretable. This process is a joint process, with the computation of the occupied
and nonoccupied voxels taking into account the class labels
in the original images. If a voxel corresponds to a building
pixel in the image, it is set to "occupied" with high probability. If a voxel corresponds to a sky pixel in the image, it has
a high probability of being "unoccupied." If a set of voxels is
stacked on top of one another, it is likely that these belong
25



Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2017

IEEE Geoscience and Remote Sensing Magazine - December 2017 - Cover1
IEEE Geoscience and Remote Sensing Magazine - December 2017 - Cover2
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 1
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 2
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 3
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 4
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 5
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 6
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 7
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 8
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 9
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 10
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 11
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 12
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 13
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 14
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 15
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 16
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 17
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 18
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 19
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 20
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 21
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 22
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 23
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 24
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 25
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 26
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 27
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 28
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 29
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 30
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 31
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 32
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 33
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 34
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 35
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 36
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 37
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 38
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 39
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 40
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 41
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 42
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 43
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 44
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 45
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 46
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 47
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 48
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 49
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 50
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 51
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 52
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 53
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 54
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 55
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 56
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 57
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 58
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 59
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 60
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 61
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 62
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 63
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 64
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 65
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 66
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 67
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 68
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 69
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 70
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 71
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 72
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 73
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 74
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 75
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 76
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 77
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 78
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 79
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 80
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 81
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 82
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 83
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 84
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 85
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 86
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 87
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 88
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 89
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 90
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 91
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 92
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 93
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 94
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 95
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 96
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 97
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 98
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 99
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 100
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 101
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 102
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 103
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 104
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 105
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 106
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 107
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 108
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 109
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 110
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 111
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 112
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 113
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 114
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 115
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 116
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 117
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 118
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 119
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 120
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 121
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 122
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 123
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 124
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 125
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 126
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 127
IEEE Geoscience and Remote Sensing Magazine - December 2017 - 128
IEEE Geoscience and Remote Sensing Magazine - December 2017 - Cover3
IEEE Geoscience and Remote Sensing Magazine - December 2017 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2023
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2023
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2023
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2023
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2022
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2022
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2022
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2022
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2021
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2021
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2021
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2021
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2020
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2020
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2020
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2020
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2019
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2019
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2019
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2019
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2018
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2018
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2018
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2018
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2017
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2017
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2017
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2017
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2016
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2016
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2016
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2016
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2015
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2015
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2015
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2015
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2014
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2014
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2014
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2014
https://www.nxtbook.com/nxtbooks/ieee/geoscience_december2013
https://www.nxtbook.com/nxtbooks/ieee/geoscience_september2013
https://www.nxtbook.com/nxtbooks/ieee/geoscience_june2013
https://www.nxtbook.com/nxtbooks/ieee/geoscience_march2013
https://www.nxtbookmedia.com