IEEE Geoscience and Remote Sensing Magazine - December 2015 - 18

can be reduced by orders of magnitude, compared to traditional methods.
The new paradigm for meaningful use of vast, growing
data archives must transition from delivery of raw data to
delivery of high-level information automatically derived
from the raw data. This paradigm requires interactive,
web-based access to metadata files, metadata embedded in
the data (e.g., MODIS Quality Analysis bits), and the data
itself. By migrating most
data processing away from
the users, and to the servers
and services in close proxaS tHe SIZe of Data
imity to the data archives,
arcHIveS raPIDLY
the perceived effect will be
IncreaSeS, So DoeS
to put the wealth of infortHe comPLeXItY of
mation contained in science
HanDLIng anD ProceSSIng
data archives into the hands
tHe Data BY USerS.
of science data users. To initiate these new data discovery methods and new information delivery paradigms,
typical science use cases of LP DAAC archive data were
used to define, develop, and implement the necessary
infrastructure. These use cases have driven the AppEEARS
functionality requirements, and have resulted in performance considerations to address.
7. Performance ImProvement StrategIeS
One of the key tenets of the LP DAAC performance strategy
is to leverage the proximity of the source data in relation
to middleware services. Because the middleware services
will be running in the same data center as the OPeNDAP
compute clusters and on-line LP DAAC data pool (i.e. USGS
EROS Center), these services can leverage high-speed network technologies and not be subject to high-latency and
low-throughput networks, such as the Internet, which a
typical user would use if downloading raw data directly
from the LP DAAC data pool.
The source data for a sample extraction request can easily
require many GB of source data but only result in MB of data
that are actually extracted, and ultimately only KB of data
are finally downloaded. Therefore, the transfer and processing of many GB of data can be done utilizing high-speed
local networks and the user would only be subject to lower
throughput networks when downloading the resulting extraction, which will be orders of magnitude smaller in size.
This basic performance strategy will result in a significant time savings for users and allow them to focus on analyzing the resulting information derived from data instead
of spending substantial time locating and downloading
large amounts of data over low-bandwidth networks. The
objective is to allow most common, reasonable data access
requests to be completed in a single uninterrupted work
session. Performance improvements will be designed and
implemented when feasible to enable the user experience
to be as interactive as possible.
18

7.1. Asynchronous Processing
When accessing data archives with OPeNDAP HTTP
requests, simple requests respond quickly (i.e. within
seconds). However, some requested processing consists
of many OPeNDAP requests to extract and transform
larger amounts of data. This extract-and-transform cycle
in these cases can possibly take minutes or even hours,
depending on the number of requests required. However, this is still a dramatic performance improvement
when contrasted with the weeks or months required to
manually download and manage large amounts of data,
un-package the data, ingest them into a Geographic
Information System (GIS) or other software tool, and
manually manipulate or post-process the data to produce the desired information. In order to facilitate a
more acceptable user experience for more expansive
processing, the requests are queued to the data server.
Users can view and download the results at a later time
for more demanding use cases. Because Python is used
for some backend services, the Python Celery module
(Celery, 2014) was a logical choice for implementing
a processing queue to enable access to asynchronous
processing. Use of Celery for critical functionality such
as queuing eliminated the need to develop additional
services for those functions. Use of Celery also ensures
these functions have standard interfaces and other necessary attributes.
After the requests to Celery are complete, metadata
about the request such as product name, layer, etc., is
stored in a MongoDB document store (MongoDB, 2014),
which can be used when displaying details about the requests to the users. Some of the artifacts generated by the
processing can be large files (> 60MB) containing tabular
data, which are bundled together with other smaller files.
A download of the bundle as a zip file is offered, but the
user is also presented with the option to preview the data
before downloading. Downloading and viewing 60MB
of data in a web browser client is neither performant nor
feasible. Instead, it is more effective for the appropriate
service calls to save the data directly to disk as a Python
Data Analysis Library data frame (pandas, Python Data
Analysis Library, 2014).
Pandas supports several formats for saving and reading data frames (pandas, IO Tools (Text, CSV, HDF5, ...),
2014). Pandas was chosen because formats such as HDF
can provide efficient and flexible performance compared to other open-source solutions. Again, standard
libraries such as pandas can be an efficient solution for
software development, because of adherence to opensource standards.
Upon a request from the user, the HDF file on disk
is loaded into a pandas data frame, filtered to only the
optional sub set to return to the user, serialized to a JSON
interchange format, and then displayed in the browser
(ecma, 2013). Pandas data frames make any filtering
and sorting of the data efficient. The loading, filtering,
ieee Geoscience and remote sensing magazine

december 2015

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2015