IEEE Technology and Society Magazine - September 2023 - 65

process learning in real-time. Utilizing RL and DRL
algorithms based on POMDP, rather than MDP, is
preferred.
RL and DRL policies and ethics
Since RL and DRL can solve a series of problems
in complex real-world environments that people
sometimes cannot understand accurately, this can
have significant societal, ethical, and governance
implications [40]. Important issues related to RL and
DRL policies and ethics are listed here.
Human oversight: According to the European
Commission's high-level expert group on AI,
human oversight of AI systems is necessary for
supporting human autonomy and decision-making
(Ethics Guidelines for Trustworthy AI, 2019). This
poses a significant obstacle for RL and DRL applications
aimed at enhancing machines' autonomy.
Designed RL and DRL systems constantly make
small decisions that are impossible for humans
to follow, monitor, and review. Humans cannot
detect all errors and intervene in real-time. However,
human oversight may be beneficial for offline
systems. Online RL and DRL systems may learn
and adjust their policy too rapidly for meaningful
human supervision.
While human oversight in RL and DRL systems
can be challenging due to rapid decision-making
and learning speed, a potential solution is combining
human guidance with machine autonomy through a
human-in-the-loop approach. Here, human experts
provide high-level guidance and intervene during
critical situations, while the RL and DRL system
handles routine decisions. This symbiotic relationship
balances human ethical considerations and
machine efficiency, focusing on overarching goals,
incorporating human values into reward function
design and training, and reviewing the offline system
before deployment.
We believe that regular offline human reviews of
the system's performance will, over time, increase
trust in the RL and DRL systems. As with any new
technology, human oversight will naturally decrease
as the technology matures and we become more
familiar with its outcomes and consequences.
Safety and reliability: To enhance RL and DRL
algorithms for CPS applications, a hybrid approach
combining traditional control with RL can improve
safety and reliability. Using established control techniques
for critical aspects mitigates catastrophic
September 2023
outcomes from wrong learning. While RL and DRL
have advantages, caution is needed in safety-critical
systems. Integrating traditional control lets the system
benefit from RL for noncritical parts while maintaining
robustness in safety-critical components.
This approach combines both worlds for a safer CPS
operation.
In designing reward functions for RL and DRL
algorithms for various applications, it is essential
to consider the values of all individuals affected by
these algorithms because optimizing for the incorrect
purpose can have unintended consequences
that are difficult to foresee. In other words, definitions
of inappropriate reward functions have irreversible
effects [40], [41].
A multistakeholder approach in the design phase
involving ethicists, domain experts, and affected parties
can lead to comprehensive and well-balanced
reward functions. Considering diverse perspectives
helps identify biases and unintended consequences,
ensuring a more balanced and responsible RL and
DRL system.
Automation and the future of work: Due to the
advances in RL and DRL algorithms, automationrelated
jobs will increase, particularly in CPS fields,
including robotics and manufacturing. Implementation
of RL and DRL algorithms will require new
programming languages and related software
developers, hardware designers, and systems engineers.
Naturally, as with other new technologies
that have been adopted, some jobs and occupations
will cease to exist. Therefore, these issues
must be considered when analyzing the impact of
AI on the labor market [40].
Proactive measures are crucial to re-skill and
up-skill the workforce. Investing in training programs
and educational initiatives can mitigate
automation's negative effects. While RL and DRL
algorithms may lead to job displacement, they
also create new opportunities. Identifying potential
job transitions and providing tailored training
and support is key for workforce adaptation. Collaborative
efforts among industries, policymakers,
and educational institutions facilitate a smoother
transition into new roles aligned with emerging
technologies.
To address the above issues, Whittlestone et al.
[40] proposed to track progress in DRL and its applications,
considering the implications of DRL progress
for existing AI governance initiatives, including
65
IEEE Technology and Society Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2023
