IEEE Geoscience and Remote Sensing Magazine - September 2016 - 16

Data
Staging
Area

Crawler

Data
Extraction

Data
Conversion

Workflow Manager

Data
Validation

File Manager

Metadata
Harvesting

Data
Archiving
Area

Publication
to ESGF

FIGURE 2. The as-implemented architecture of the NASA ESGF cyberinfrastructure comprises a series of workflow stages that combine
Apache OODT software, NetCDF operators, OPeNDAP, Apache Solr, and the ESGF publishing tool kit.

Internet mail extensions type, etc., and it provides data movement capabilities. The crawler provides automated methods
for ingesting, locating, selecting, and interactively extracting
files and metadata managed by the file manager, while simultaneously notifying the workflow manager that pipelines
need to be executed.
The crawler is seeded with an initial data staging area or
a nonlocal OPeNDAP directory of remote sensing data. It
extracts the file and HDF metadata, which it subsequently
presents to the file manager for ingestion. At the same time,
the crawler notifies the workflow manager that the conversion pipeline should be initiated for the variable of interest.
Data extraction is kicked off, and the required five-tuple of
information is extracted. Any necessary conversion is performed in the data conversion step using the NetCDF operators package, which then writes a new NetCDF file based on
the extracted five-tuple. The resulting output is sent to the
data validation step that in turn calls a Python web service
that applies the CMOR checker. If the validation is successful, metadata harvesting collects the NetCDF information
into a Thematic Real-Time Environmental Distributed Data
Services data server, publishes it to Apache Solr, and, ultimately, delivers it to the ESGF in the publishing to ESGF step.
Publishing remote sensing data alongside climate model
output enables better comparisons and understanding that,
in turn, more completely inform those who study the climate and those who make crucial policy decisions affecting
the climate. Our expectation is that using automated workflows to streamline the publication of high-quality data
will significantly improve this crucial activity.
NExT-GENERATION CyBERINFRASTRUCTURE
FOR CLImATE DATA ANALyTICS
Climate model input and output data provide the basis for
intellectual work in climate science. As these data sets grow
in size, new approaches to data analysis are needed. In efforts to address the big data challenges of climate science,
some researchers are moving toward a notion of Climate
Analytics-as-a-Service (CAaaS). CAaaS combines high-performance computing and server-side analytics with scalable
data management, cloud computing, a notion of adaptive
analytics, and domain-specific application programming
16

interfaces (APIs) to improve the accessibility and usability
of large collections of climate data [3], [8]. In this section,
we take a closer look at these concepts and a specific implementation of CAaaS in NASA's Modern-Era Retrospective
Analysis for Research and Applications Analytic Services
(MERRA/AS) project.
HIGH-PERFORMANCE SERVER-SIDE ANALYTICS
At its core, CAaaS must bring together data storage and
high-performance computing to perform analyses over data
where the data reside. MapReduce has been of particular interest because it provides an approach to high-performance
analytics that is proving to be useful in many data intensive
domains [3]. MapReduce enables distributed computing
over large data sets using high-end computer clusters. It is
an analysis paradigm that combines distributed storage and
retrieval with distributed, parallel computation, allocating
to the data repository analytical operations that yield reduced outputs to applications and interfaces that may reside elsewhere. Since MapReduce implements repositories
as storage clusters, data set size and system scalability are
limited only by the number of nodes in the clusters.
MapReduce distributes computations across large data
sets using a large number of computers (i.e., nodes). In a map
operation, a head node takes the input, partitions it into
smaller subproblems, and distributes them to data nodes. A
data node may do this again in turn, leading to a multilevel
tree structure. The data node processes the smaller problem
and passes the answer back to a reducer node to perform the
reduction operation. In a reduce step, the reducer node then
collects the answers to all the subproblems and combines
them in some way to form the output, i.e., an answer to the
problem it was originally trying to solve.
While MapReduce has proven effective for large repositories of textual data, its use in data-intensive science applications has been limited because many scientific data sets
are inherently complex, have high dimensionality, and use
binary formats. Adapting MapReduce to complex, binary
data types have been a major advancement to these efforts.
Due to the importance of MapReduce in large-scale analytics and its widespread use, there has been significant private-sector investments in recent years aimed at improving
ieee Geoscience and remote sensinG maGazine

september 2016

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2016