IEEE Geoscience and Remote Sensing Magazine - December 2015 - 17

User Domain

WAN

Information User
Request
LP DAAC Data Archive and Middleware Services Domain

Graphical
Representation
of Data

OPeNDAP Service
Requests

LAN
N AppEEARS Web Server
A
L

LAN
LP DAAC
Archive

OPeNDAP Clusters
OPeNDAP Access
Read Granules
Local Network
1,000's of Files
10's GB

Middleware Services
Extract Data
Deliver Sub-Sets
1,000's of Files
100's MB

Services Processing
SDK
Services
Array, Grids
10's of MB

AppEEARS
Data Exploration
Statistical Analysis
1's of Files
100's of KB

AppEEARS Services Processing Model
Seconds, Minutes, Hours

Figure 10. AppEEARS Model for In-Transit Processing.

As the size of data archives rapidly increases, so does
the complexity of handling and processing the data by
users. Requiring users to manually download and manipulate each data archive granule necessary to aid their
scientific research is no longer practical. Figures 9 and
10 contrast the traditional model of data processing required to derive scientific information from LP DAAC
archives with the proposed model using AppEEARS. The
new model is able to parse "Analysis Ready Data" and
greatly reduce the work and time required by the user
and the amount of data to be managed by the user. In the
traditional processing model, users access data discovery
tools such as ECHO or EE to request data downloads.
The user must then download data over low-bandwidth
networks to a local system. Manual operations are then
performed to un-package, ingest, process, export and
subset the entire set of granules to extract the relevant
data. This model may require the user to download
thousands of granules before any data sub setting can
occur, because it is performed locally. The user must also
provide interim local storage for tens of GigaBytes (GB)
of data before creating a substantially reduced data set
december 2015

ieee Geoscience and remote sensing magazine

measuring in the hundreds of MegaBytes (MB) suitable
for data analysis. The traditional data processing workflow in this example could consume days, if not weeks or
more, of elapsed time.
In the AppEEARS Services Processing Model, the user
does not download or process LP DAAC archive data. Instead, the user sends requests for higher-level information
products to the AppEEARS web server, located within the
LP DAAC infrastructure. The web application parses the
information request and calls the middleware services
necessary to satisfy the request. All of the data retrieval,
sub setting, and processing functions are performed on
the OPeNDAP cluster systems, which reside next to the
data archive on the same high-bandwidth network infrastructure. The number of files and aggregate size of the
data are reduced from thousands of files and tens of GB
to only a few files and hundreds of KiloBytes (KB) before
transfer to the user over low-bandwidth networks. To accommodate multiple users and requests, a series of service
workers can spin up as requests and workloads warrant.
With this processing model, the elapsed time for requesting information from thousands of LP DAAC granules
17

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2015