IEEE Computational Intelligence Magazine - May 2023 - 66

which requires additional training efforts,
or metric learning where comparing highdimensional
embeddings may be timeconsuming.
These negative impacts
must be considered and evaluated under
the perspective of environmental
sustainability.
FIGURE 7. The procedure for active learning [69].
Few-shot Learning: Few-shot learning
is another class of approaches that aims to
learn from less labeled data for training AI
models [75]. To solve few-shot learning
problems, there are two main groups of
methodologies. The first one is based on
meta-learning, also known as learning to
learn [76]. Meta-learning techniques generally
consist ofa teacher model and a student
model, where the teacher learns how
to optimize the student while the student
learns how to perform downstream tasks.
This canbeseenin [77] where the outputs
of the teacher models were used to train
the student model by optimizing and
selecting the parameters from a higher
dimensional space due to the limited training
samples. Generally, initial meta-learners
are biased towards the existing tasks in
parameter optimization, but Jamal and
Qi [78] demonstrated the possibility of
using an agnostic meta-learner to overcome
those limitations. Another promising
work is presented by Hou et al. [79],
where attention modules for meta-learners
were used to achieve the task-specific
initialization ofbasemodels for fast adaptation
in meta-learning.
Another popular approach for fewshot
learning is metric learning. The general
direction is to compare the embeddings
of the support (few-shot training
samples) and query (test samples) sets.
Examples of such work include the Siamese
[80] and Triplet [81] networks,
which compared two embeddings at one
time, and the Matching [82], Prototypical
[83], and Relation [84] networks,
which compared across multiple embeddings
from support and query sets. A
distinguishing feature across metric learning
approaches is the distance measures
for embeddings. The Triplet, Matching,
and Prototypical networks adopt predefined
distance measures, whereas the
Siamese and Relation networks learn the
distance measures by using neural
networks.
Similar to active learning, few-shot
learning can significantly reduce the
amount oflabeled data for model training,
which saves a lot ofresources required for
data collection and annotation, thus
reducing carbon footprint in these tedious
processes. Substantiated examples are
showninTable II, where active learning
and few-shot learning are tested on various
benchmark datasets with varying percentages
of labeled data for model training
[84], [85], [86], [87].Itcan be found
that these data-efficient learning algorithms
can achieve good performance
with a small portion of the labeled data,
which significantly reduces the burden on
data collection and annotation. However,
few-shot learning may not be able to
reduce training effort if learning with
complex algorithms, like meta-learning
2)No Labeled Data
More advanced AI techniques intend to
release the burden ofdata annotation by
learning AI models via data from related
tasks or unlabeled data. Two typical
techniques, i.e., transfer learning and
self-supervised learning, are introduced,
and their contributions to environmental
sustainability are analyzed.
Transfer Learning: Transfer learning
aims to solve a given task (denoted as target
domain) with labeled data from
related ones (denoted as source
domain) [88], as shown in Fig. 8. Note
that generally the target domain only
contains unlabeled data. The main idea in
transfer learning is to minimize the
domain difference between the source
and target domains. There are two types
ofapproaches to minimize domain difference.
One is distance-based methods,
where a distance function, such as
MMD [89], [90],CORAL [91], etc., is
defined to measure the domain difference,
and it will be minimized during
training, such that the gap between the
source and target domains is minimal [92].
Then, the source classifier/regressor
trained using the labeled source domain
data can be used for the classification/
regression task in the target domain. The
other one is adversarial-based methods,
which are inspired by generative adversarial
network (GAN) [93], [94], [95].
TABLE II Results of selected data-efficient learning algorithms in terms of the
percentage of the labeled data for model training and testing accuracy (Acc.) on
various datasets.
Active Learning
MNIST [85]
Percentage
3.4%
Few-shot Learning
CUB [87]
Percentage
4.2%
Acc.
92.9%
Acc.
95.0%
PASCALVOC [86]
Percentage
20%
Omniglot [84]
Percentage
25%
Acc.
99.8%
Acc.
72.3%
66 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2023
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
