IEEE Computational Intelligence Magazine - February 2023 - 71

Furthermore, the above methods often adopt the singletask-based
optimization mechanism; that is, they cannot guarantee
the extracted features are the most suitable for fusion. In
[39], Ma et al. proposed a self-supervised mask-optimization
model for multi-focus image fusion, the auto-encoder is utilized
to reconstruct the fused image in a self-supervised way,
forcing the vital information from source images to be
extracted adaptively. However, the concerning information
for the same region on multi-modal medical images is different,
and region disparity easily leads to the loss of vital information
in the process of feature adaptation. To obtain a better
feature adaption of multi-modal image fusion, a novel selfsupervised
framework is put forward to extract the prior features
ofa pair ofsource images adaptively.
B.CNN and Transformer
Despite the popularity of CNN fusion methods, these algorithms
cannot provide satisfactory fusion performance due to
the difficulty in capturing the local information and the longrange
dependencies effectively. Vision Transformers (Vit)based
solutions have very recently been proposed to address
the issue of long-range dependency in CNNs [40]. Unlike
CNN, Vit-based models can effectively model local and
global feature information. However, the methods based on
transformers need numerous data due to its self-attention
inductive bias ability being weaker than CNN. Zhou et al.
[41] presented an unsupervised MMIF method by combining
a densely connected high-resolution network (DHRNet)
with a hybrid transformer. In this method, the local features
are firstly extracted with the DHRNet, and then input into
the hybrid transformer to produce the global features.
Although the combination of DHRNet and the transformer
solves the problem of self-attention inductive bias ability,
there still exists the problem of a huge amount of computation.
Besides, SwinFusion [42] and SwinBTS [43] have been
proposed to reduce the computational complexity of the
transformer by introducing the pyramid structure. However,
the combined structure of the CNN-transformer for the
above methods is that CNN and the transformer are connected
in series. That is, local features are obtained first and
then global features are obtained, or vice versa. The global
and local features extracted in series are affected by the order,
which will easily lead to incomplete features. We propose an
auto-encoder with Vit and CNN in parallel for extraction of
local and global features of the source image simultaneously.
Furthermore, we construct the positive and negative result
pairs to train the feature extractor by contrastive learning and
design a novel contrastive loss, aiming at the case that the
extracted features are redundant and not unique.
C. Information Exchange
It is common sense in our life that a person with less
knowledge, generally will learn more new knowledge
from the other one with more knowledge, and vice versa
[35]. In other words, people with less knowledge
FIGURE 1 Architecture of SSN-CAEþIE framework. SSN-CAEþIE
consists of CAE and MCIEN.
contribute less to cooperative tasks. Therefore, according
to a reverse proportion of the knowledge learned from
each other, we can estimate their contributions to a certain
task. Based on this point, Nie et al. [35] constructed a
Pulse Coupled Neural network (PCNN)-based information
exchange (PCNN-IE) module and applied it to estimate
the fusion contribution for multi-modal medical images.
However, the modulecannotbeoptimized by thederivative-based
methods, due to the existence of a hard threshold
in PCNN. Also, it cannot be trained on a large-scale
dataset due to the complicated structure of a neuron. Nevertheless,
we can also note in the above DL-based methods
that the features from the network are directly employed
to reconstruct the fused result, only guided by a loss
design. That is their importance or fusion contribution not
be well measured.
Considering the above deficiencies, the proposed method
employs CNN to construct an MCIEN instead ofPCNN and
resorts to it to perform information exchange among the
multi-feature maps and estimate the fusion contribution of
source images effectively.
III. Methodology
Our framework for the MMIF task is a self-supervised framework
named SSN-CAEþIE as illustrated in Fig. 1, which consists
of two parts: CAE and MCIEN. In this section, we
provide an overview of our fusion framework, the network
architecture, and the training process.
A. Overview
Given a pair of medical images, A and B, the MMIF task can
be formulated as a weighted model as follows:
F ¼ wAAþ wB B
(1)
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 71
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
