IEEE Geoscience and Remote Sensing Magazine - September 2016 - 20

approaches tend to view big data as an issue of extracting
meaningful patterns from large amounts of unstructured
data for the purpose of finding insights.
As the ESGF community grapples with its scaling challenges, it seeks to find a balance between these competing
views. This is evident in the charge that the ESGF Compute
Working Team (ESGF-CWT)-the international team of collaborators responsible for designing ESGF's next-generation
architecture-has laid out for itself. The team's overarching
goal is to increase the analytical capabilities of the enterprise,
primarily by exposing high-performance computing resources and analysis tools to the community through web services
[30]. Ideally, the ESGF data from the federation's distributed
collections would be united with the web-accessible tools
and compute resources needed to perform advanced analytics at the scale needed for IPCC's increasingly complex work.
However, integrating high-performance computing and
high-performance analytics, i.e., finding an optimal storagecompute balance in ESGF's ecosystem of distributed resources, is not a trivial exercise. ESGF's technical heritage is that of
a large-scale distributed archive. Its nodes basically store and
distribute data. They typically support compute resources
sufficient for streaming data out of storage onto the network
for client consumption, and the behaviors implemented and
exposed by ESGF's web service interface are the basic discovery and download operations of an archive.
Currently, the ESGF is looking to the geospatial community for ideas on how to strike a balance between data analytics and data storage. Improved access to distributed compute
and storage resources has been achieved in the geographic information systems community through a series of standardsmaking activities aimed at enhancing machine-to-machine
interoperabity, one of the most notable being the work of the
Open Geospatial Consortium (OGC). The OGC is an international industry consortium of over 500 companies, government agencies, and universities participating in a consensus
process to develop publicly available interface standards. The
OGC's abstract specifications and implementation standards
are designed to support interoperable solutions that geoenable a wide range of hardware platforms and software applications [31]. To see how improved machine-to-machine
interoperability can lead to increased analytic capabilities
across distributed storage systems, it is helpful to understand
web services and the role that web APIs play in the discussion.
WEb SERVICES AND DOMAIN-SPECIFIC
API ENHANCEMENTS
As described previously, in the world of web services, there
are two types of interfaces. On the service side, a system interface maps the methods, functions, and programs that implement the service's capabilities to Hypertext Transfer Protocol
messages that expose the service's capabilities to the outside
world. Client applications can consume these web service
endpoints to access services. The World Wide Web Consortium views web services as a way to insure machine-tomachine interoperability [32]. The precise messaging format
20

can vary from community to community, often reflecting
the specialized functions or audiences they serve. Significant
standards activities have grown up around the design and
implementation of such web services.
There also are the classic client-side APIs familiar to application developers. Generally, these comprise local libraries
that reside on the developer's host computer and can be statically or dynamically referenced by client applications. They
speed up development, reduce error, and often implement
abstractions that are specialized to the needs of the audiences they serve. They can be used to build applications, workflows, and domain-specific tool kits, workbenches, and integrated development environments. Building on the concepts
underlying CAaaS, the ESGF-CWT is working at both levels.
IMPLEMENTATION APPROACH
The ESGF-CWT is adopting OGC's Web Processing Service
(WPS) interface standard for its next-generation architecture
[33]. The WPS is essentially an xml-based remote procedure
call protocol for invoking processing capabilities as web services. It has been used in the geospatial community for delivering low-level geospatial processing services. However, the
WPS can be generalized to other types of applications and
data because of its simplicity. The WPS uses a single operation (i.e., execute) to invoke remote services; its two other operations (i.e., GetCapabilities and DescribeProcess) are used
for discovery and to query services for information necessary
to build signatures needed by the execute operations.
The ESGF can improve interoperability and accessibility
by defining ESGF community standards at one or more places in its web services architecture. First, the ESGF can define
an ESGF compute node service specification, i.e., an agreedupon capability and naming convention for each conformant
compute node. Regardless of how the services are accessed,
each node would have known capabilities implemented in
known ways. Second, the ESGF can define an ESGF WPS extension specification, i.e., a specialization of the WPS standard wherein the syntax and semantics of the required and
optional fields of WPS response documents are tailored to
the needs of the ESGF. With this approach, regardless of how
services are implemented or named, their means of access
is commonly understood within the federation. Finally, the
ESGF can define an ESGF API, i.e., a client-side API that consumes the web service endpoints exposed by a WPS-compliant ESGF service and presents them to client applications as
a library of easy-to-use function calls tailored to the needs
of the ESGF community. Here, regardless of implementation
and communication details, programmers could access node
capabilities using a familiar programming library.
The ESGF-CWT is developing options two and three: an
ESGF WPS extension specification and an accompanying
client-side ESGF API along the lines of the CDS API (Figure 3).
A reference implementation of an ESGF multimodel averaging service will be released soon. These enhancements will
be of value to the ESGF community because they will improve interoperability at two levels within the ESGF's overall
ieee Geoscience and remote sensinG maGazine

september 2016

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2016