IEEE Geoscience and Remote Sensing Magazine - December 2022 - 130

specially designed for large-scale data processing. Hadoop's
MapReduce can write data back to physical storage media
after each operation, while most of Spark's operations are
performed in memory. Compared with Hadoop, Spark has
obvious advantages in terms of speed.
Wu et al. [193] designed a parallel algorithm for the
principal component analysis of hyperspectral dimensionality
reduction based on cloud computing by taking Spark
as the computing engine and the HDFS to realize distributed
storage so as to give full play to the high-throughput
access and high-performance distributed computing capability
of the cloud computing environment. Jing et al.
[87] designed a custom flexible distributed dataset using
mosaic remote sensing images and used Spark distributed
memory computing characteristics to avoid intermediate
images being written back to the file system, thus saving
I/O consumption and obtaining better processing performance.
Xu et al. [200] proposed ScienceEarth, a remote
sensing big data processing platform that consists of three
main parts: data, index, and calculation modules, among
which the calculation module is based on the calculation
framework of Spark as the processing and analysis engine.
Tan et al. [169] explained an adaptive cloud remote
sensing data processing method based on Spark. Through
the adaptive MapReduce mechanism, local random-access
memory (RAM) storage of intermediate data, workload
balancing, and other optimization methods, it achieved
better computational efficiency than Hadoop in classifying
massive remote sensing images, and the more tasks, the
more obvious the efficiency improvement. More remote
sensing big data processing continues to be based on Spark,
and its application is very extensive. For example, Boulila
et al. and others proposed a parallel algorithm for remote
sensing image classification based on Spark, and Haut et al.
implemented parallel processing based on Spark for spectral
analysis and compression, image segmentation, image
classification, change detection, and so on [9], [23], [33],
[183], (Haut et al. [241] and Zhou et al. [242]).
Using big data computing engines, some scholars have
carried out special research on the efficient storage, organization,
management, and publishing of remote sensing
data based on a distributed file system to meet the needs of
the HPC of remote sensing big data processing. Kou et al.
[243] introduced a remote sensing data management model
based on the HDFS, which adopted a double storage mechanism
and simulation operation to alleviate the overload
problem of a single NameNode as well as a relational database
management system and the HDFS to store and manage
image files and metadata, respectively, which solved the
problems of the storage space and query efficiency of massive
remote sensing data.
Liu [116] designed a high-resolution image storage model
in the cloud computing environment, divided the data in
advance according to the inherent data blocking characteristics
of the HDFS, and adopted various access algorithms,
such as transparent mode, map service mode, and pyramid
130
mode, to solve the problem of insufficient data storage and
management in the distributed environment and improve
the distributed computing performance. Vinogradov et al.
[244] adopted the OrangeFS file system, instead of the Hadoop
HDFS, and the Hierarchical Data Format 5 as the data
exchange format; optimized the data access mechanism
and I/O dynamic scheduling; and applied a parallel file
system and optimization strategy, which improved the parallel
computing efficiency by 40%.
The wide application of big data promotes the rapid upgrading
of cloud computing technology, and new technologies
are continuously being introduced. Different technical
architectures are suitable for different application scenarios.
Compared with distributed parallel computing in a cluster
environment based on the MPI, the cloud computing
framework has a more complete ecosystem. The cloud computing
framework realizes task scheduling in the MPI process
through a special task scheduling system. In terms of
efficiency, there is no obvious difference between the MPI
and cloud computing; in terms of ease of use, the latter performs
better because of its relatively perfect ecosystem. Yao
et al. [245] proposed two parallel computing frameworks
based on the MPI and Hadoop, both of which integrated
the use of a GPU for processing SAR data with increased
simulated pulses. Within the same hardware environment,
Hadoop had lower efficiency than the MPI, mainly due to
the I/O and start-up time of the basic running environment.
Wang et al. [184] compared the parallel efficiency of the
MPI and MapReduce in a maximum likelihood classification
of remote sensing images within the same software
and hardware environment and concluded that the two
had similar computing performance. For massive remote
sensing data, MapReduce has better overall performance
in terms of programming convenience, data management,
and computing efficiency. When the MPI is used to process
remote sensing big data, it is necessary to consider the running
state of processes, communication among different
processes, monitoring of abnormal situations, and other
issues. The cloud computing framework for remote sensing
big data has a perfectly integrated computing engine, distributed
file system, and powerful task scheduling system.
It can provide a complete set of solutions, and researchers
do not have to spend too much energy on task management,
load balancing, and so on. Therefore, the cloud
computing framework is the preferred choice for remote
sensing researchers to realize parallel computing based on
computer clusters.
REMOTE SENSING GRID COMPUTING
Grid computing was put forward in the mid-1990s as a distributed
computing infrastructure for advanced scientific
and engineering applications. Grids are intrinsically distributed
and heterogeneous but supported in such way that
the user sees a relatively seamless environment [17]. The development
of grid computing has overcome the shortcomings
of the high cost of supercomputers and the need for
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2022
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
