IEEE Geoscience and Remote Sensing Magazine - September 2022 - 333

lifecycle and little to no contextual metadata and data governance
[3]. Consequently, the stored data end up of little
use to the intended users. The lack of data management can
potentially cause problems for users of mirrored data if they
are not vigilant. Yet there is limited information for users to
check whether the data mirrored by a third-party actor on
a cloud platform are the authoritative versions or whether
the mirror has been checked for completeness and data integrity.
Users also need to know whether the original data
have been transformed as parts of the transfer and whether
the transferred data have been checked for scientific correctness.
In addition, making sure that the mirrored data
are regularly checked against the authoritative data for version
updates and completeness is important for data users,
particularly for data streams that are still ongoing.
There are few indications that existing projects, organizations,
and agencies will converge toward a single or a
limited authoritative set of data lakes for EO data on select
cloud platforms. Therefore, it is important for the Earth science
informatics (ESI) community to address some of the
problems caused by data replication and mirroring. Organizations
such as the National Oceanic and Atmospheric
Administration have adopted some form of digital signatures
(https://www.avl.class.noaa.gov/release/system_help/dig_
sig/index.htm) to address this issue.
In this month's " Perspectives " column, we investigate
these problems from a data user's perspective and explore
a range of technical solutions that address this data-traceability
problem. (Note that the user perspectives
stated in the column are derived from the authors' routine
engagement with NASA Earth science data users.)
We realize that this is a complex problem with many
different use cases. The solutions investigated in the
column focus only on addressing a set of specified use
cases. This column hopes to initiate a discussion within
the ESI community, specifically between the EO data
producers/distributors of both authorized and trusted
sources and the third-party agents mirroring data. A
discussion centered around developing common-sense
practices for replicating the data, as well as developing
some guidance for users consuming the mirrored data,
is provided in [4]. The column is structured into sections
that cover use cases, an overview of the traceability
study, a specific selection and definitions of the facets
used to compare different methods, a methodology that
describes the five potential approaches investigated to
address data traceability, and finally, conclusions from
this study.
EXAMPLE USE CASES AND SCENARIOS
The goal of describing these use cases and corresponding
scenarios is to tease out the different challenges that need
to be addressed to allow a data user to easily and confidently
consume the mirrored data.
A user may want subsets of different datasets for their
analysis. Or a user may want to process the full archive of
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
a dataset to create a new data product. An example of the
latter use case is the production of the Harmonized Landsat
Sentinel (HLS) data product that uses both Landsat and Sentinel-2
archives as the input. Products like HLS will become
common as researchers harmonize products from multiple
sensors to produce new products that can be utilized interchangeably
and significantly increase temporal or spatial
coverage.
For both the analysis and processing use cases, the user
can face three scenarios. In the ideal scenario, all the authoritative
data required are available on the same cloud
platform of the user's choice. In this case, the user only requires
access to the cloud storage and some form of uniform
notification mechanism.
In the worst case, the different data products are unavailable
on the cloud platform. The burden now falls on
the user to replicate the archive from different authoritative
sources. The user now requires a uniform scalable data access
mechanism from all the
authoritative sources to minimize
his or her burden.
In the most common case,
some data products may be
available on the selected cloud
platform, but their provenance
is unknown. In this case, the
user now has to not only move
the unavailable data but also
verify the authenticity of
available data. In this scenario,
in addition to a uniform
approach for data access for
the missing data (as mentioned before), the user needs
a uniform approach to verify the authenticity, completeness,
correctness, and consistency of the data available on
the cloud.
DATA TRACEABILITY
We conducted a data-traceability study to explore different
technical approaches to potentially address these use
cases for three different scenarios. Simplifying assumptions
were made to define the scope of this study. First,
the third-party agents replicating the data do not have
any malicious intent in that they are modifying the data
values intentionally. Second, use cases where the original
data have been transformed from an original format to
a cloud optimized data format are not considered within
the scope of this study. Technical details of the study can
be found in [2].
The primary actors in the different approaches are defined
here. Data producers/data distribution centers are
the source of the authoritative or trusted data. They are
referred to as authoritative sources. Data stewards are organizations
within authoritative sources that are charged
with the collection and maintenance of authoritative
data, i.e., the overall data stewardship role. Data users are
333
A DATA LAKE SUPPORTS
MULTIPLE WAYS OF DATA
INGEST AND FLEXIBILITY
TO ACCOMMODATE A
VARIETY OF DATA STORAGE
BASED ON VOLUME AND A
VARIETY WITH DIFFERENT
ACCESS PATTERNS.
https://www.avl.class.noaa.gov/release/system_help/dig_sig/index.htm https://www.avl.class.noaa.gov/release/system_help/dig_sig/index.htm
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
