IEEE Geoscience and Remote Sensing Magazine - September 2016 - 11

Environment-Intercomparison Project (CREATE-IP) activities; and the collaborative work of the Intergovernmental Panel on Climate Change (IPCC). This article
provides an overview of some of climate science's big data
problems and the technical solutions being developed to
advance data publication, climate analytics as a service,
and interoperability within the Earth System Grid Federation (ESGF), which is the primary cyberinfrastructure
currently supporting global climate research activities.
TECHNICAL CHALLENGES FOR BIG DATA
The term big data is used to describe data sets that are too
large or complex to be worked with using commonly available tools [1]. Climate science represents a big data domain
that is experiencing unprecedented growth [2]. Some of
the major big data technical challenges facing climate science are easy to understand.
◗ Large repositories mean that the data sets themselves
cannot easily be moved; instead, analytical operations
must migrate to where the data reside.
◗ Complex analyses over large repositories require highperformance computing.
◗ Large amounts of information increase the importance
of metadata, provenance management, and discovery.
◗ Migrating codes and analytic products within a growing network of storage and computational resources
creates a need for fast networks, intermediation, and
resource balancing.
◗ Importantly, the ability to respond quickly to customer demands for new and often unanticipated uses for
climate data requires greater agility in building and deploying applications [3].
In addressing these challenges, it is important to recognize that the work of climate science comprises social as well
as technical practice [4], [5]. There are established human
processes for creating, sharing, and analyzing scientific data
sets, often in a highly collaborative mode. The work is both
valued by society and subject to intense critical scrutiny. It
informs national and international policy decisions. Collectively, these social factors add urgency and complexity to our
efforts to build an effective cyberinfrastructure to support
climate science. This article provides an overview of some of
climate science's big data problems and the technical solutions being developed to improve data publication, analysis,
and accessibility. This material combines the contributions
of those who participated in the 2014 Big Data from Space
Conference session "Big Data Challenges in Climate Science"
[6]-[8]. We use the work being done by the IPCC as the context for our presentation, with particular focus on the global
climate research community's ESGF collaborative infrastructure and the community's CMIP efforts.
BACKGROUND
Our understanding of Earth's processes is based on a combination of observational data records and mathematical
models. The size of our space-based observational data sets
september 2016

ieee Geoscience and remote sensinG maGazine

is growing dramatically as new missions come online. However, a potentially bigger data challenge is posed by the work
of climate scientists, whose models are producing data sets of
hundreds of terabytes or more [9]. There are two major challenges posed by the data-intensive nature of climate science.
First, there is the need to provide effective means for publishing large-scale scientific data collections. This capability is
the foundation upon which a variety of data services can be
provided, from supporting active research to large-scale data
federation, data distribution, and archival storage. The second data-intensive challenge has to do with how these large
data sets are used, i.e., data analytics is the capacity to perform useful scientific analyses over large quantities of data
in reasonable amounts of time. In many respects, this is the
biggest challenge, for without effective means for transforming large scientific data collections into meaningful scientific
knowledge, our climate science mission fails.
To gain perspective on the big data challenges in climate science and the efforts that are underway to address
those challenges, it is helpful to examine four elements operating at the core of global-scale climate research: 1) the
IPCC, which is responsible for integrating scientific results
and presenting them in meaningful ways to policy makers
throughout the world; 2) CMIP experiments that coordinate research on general circulation models (GCMs), which
are arguably the most important tools available to scientists who study the climate; 3) the ESGF, which provides
the distributed infrastructure for publishing climate model
outputs, sharing scientific knowledge, and supporting
global-scale collaboration; and 4) a new wave of data publication activities aimed at integrating observational data
and reanalysis data into the ESGF. In this section, we take a
closer look at each of these elements.
INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE
The IPCC is the leading international body for the assessment of climate change [10]. It was established by the
United Nations (UN) Environment Program and the World
Meteorological Organization (WMO) in 1988 to provide
the world with a clear scientific view on the current state of
scientific knowledge about climate change and its potential
environmental and socioeconomic impacts.
The IPCC is open to all member countries of the UN and
WMO. Currently, 195 countries are members of the IPCC.
Governments participate in the review process and the plenary sessions, where main decisions about the IPCC work
program are made and reports are accepted, adopted, and
approved. Thousands of scientists from all over the world
contribute to the IPCC's work on a voluntary basis, and review is an essential part of the process to ensure an objective and complete assessment of the current information.
The IPCC aims to reflect a range of views and expertise. Its
secretariat coordinates all the IPCC work and liaises with
governments. Because of its scientific and intergovernmental nature, the IPCC embodies a unique opportunity
to provide rigorous and balanced scientific information to
11

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2016