IEEE Robotics & Automation Magazine - June 2023 - 25

The restrictions of the current FDC systems might be
solved by introducing a Hadoop ecosystem in which some
components specially solve the big data storage, processing,
and data management problems. In the meantime, semiconductor
fabs do not allow the shutdown of production lines
during the FDC system migration. Fortunately, the Hadoop
ecosystem can handle such a concern.
ARCHITECTURE OF A HADOOP ECOSYSTEM
The architecture of a Hadoop ecosystem is illustrated in Figure
1. It is designed with four logical layers to handle the collected
data, including data storage, data processing, data
access, and data management.
At the data storage layer, the Hadoop distributed filing system
(HDFS) is particularly designed to store large data sets
over a variety of physical servers [4]. These servers form a basic
data cluster and can support data storage to a petabyte level.
At the data-processing layer, the main functionalities are
CPU and memory resource allocation within the Hadoop
ecosystem, as well as data processing. Yet Another Resource
Negotiator (YARN) [4] and MapReduce are two components
at this layer applied for CPU and memory resource allocation
and data processing, respectively.
The data access layer is responsible for the data transfer,
access, and query. At this layer, Hive serves for data access
and query, while Sqoop is used for data transfer.
The data management layer is responsible for the data and
Hadoop ecosystem management. This layer mainly applies ZooKeeper
for the Hadoop component coordinator and Metric Monitoring
for the Hadoop ecosystem health condition management.
The Hadoop ecosystem is an open source framework that
provides solutions to big data issues. The main components in
a Hadoop ecosystem are introduced as follows in detail.
Data
Management
Oozie
(Workflow Monitoring)
Metric
Monitoring
(Monitoring)
Data
Access
Hive
(SQL)
Pig
(Dataflow)
HDFS
HDFS [20] is a distributed file system with a high fault tolerance
capability that offers high-throughput access to data. Specifically,
when data are sent to a Hadoop ecosystem, the data
are also replicated to more than one server within the HDFS for
storage. Thus, if one physical failure occurs in the HDFS component,
there always exists a data replication to use. In this way,
large amounts of data can be stored in a reliable manner. It
offers shell commands, Java, and Python application programming
interface (API) methods.
HBASE
In practice, IoT sensors have higher fault probabilities such that
the entire columns of the captured data cannot be sent, resulting
in unstructured data. HBase [21] is especially useful for
dealing with sensor data under IoT scenarios and serves for
storing unstructured data. Also, it has the capability for the
transaction process, including updating, inserting, deletion, etc.
It is a nonrelational standard query language (SQL) component
within a Hadoop ecosystem. HBase allows random and realtime
read/write access for big data and provides a fault-tolerant
storage for unstructured data. Notice that unstructured data are
hard to be processed under the traditional relational database
management (RDBMS) infrastructure, while HBase not only
allows its storage, but also offers random and quick access.
MAPREDUCE
MapReduce [22] is a data-processing engine for large data
sets based on the HDFS component. The core working mechanism
behind MapReduce is to map large data sets into a collection
of small data pieces. Then, such small data pieces are
assigned to a bunch of servers for being processed simultaneously.
Specifically, when a MapReduce data workflow starts,
ZooKeeper
(Management)
Storm
(Data
Streaming)
Flume
(Monitoring)
Others
Avro
(RPC)
Sqoop
(Data Connector)
Data
Process
MapReduce
(Data Process)
YARN
(Resource Management)
Data
HDFS
Storage
(Distributed File System)
HBase
(Column Storage)
FIGURE 1. Hadoop ecosystem top components. HDFS: Hadoop distributed filing system; RPC: remote procedure call; SQL: standard
query language; YARN: Yet Another Resource Negotiator.
JUNE 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
25

IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023