IEEE Computational Intelligence Magazine - February 2023 - 70

ground-truth images for supervised learning [27]. Therefore,
some semi-supervised deep-learning algorithms [28], [29]
have been proposed for time-series classification (TSC) and
to achieve excellent supervised and unsupervised performances
on a large number of datasets. Furthermore, some
unsupervised methods [30], [31], [32], [33], [34] have also
been proposed and achieved good performance, but the
methods based on generative adversarial network (GAN)
have the problem of unstable training results. The depth
structure information and shallow texture information of a
pair of source images cannot be retained completely at the
same time.
To address the deficiency of the above methods, we propose
a self-supervised network with contrastive auto-encoding
and information exchange, named SSN-CAEþIE, to perform
the MMIF task. To this end, a contrastive auto-encoder
(CAE) based on a combination of a vision transformer (Vit)
and a convolutional neural network (CNN) in parallel is first
constructed to extract the features of source images pairwise.
Furthermore, inspired by the principle of information
exchange [35], a multi-convolutional information exchange
network (MCIEN) is designed to pixel-wisely measure the
fusion contribution between source images. The main contributions
ofthe proposal are as follows.
1) The MMIF task is modeled as a contribution estimation
problem, such that a self-supervised framework consisting
ofCAE and MCIEN is constructed to generate fusion contributions
between source images.
2) This paper proposes a novel CAE combining Vit and
CNN to adaptively extract and integrate the local and
global features from paired source images, in a manner of
contrastive learning. Also, MCIEN is designed to perform
the information exchange among feature maps in multiconvolutional
spaces and finally estimate the fusion contribution
between source images.
3) Effective losses are proposed for auxiliary and downstream
tasks. For the auxiliary task with a CAE, an effective
contrastive loss is designed, based on a group of
positive and negative results constructed elaborately.
Further, a hybrid loss with a weighted fidelity and a regulationtothe
informationexchangeisemployedtotrain
our SSN-CAEþIE.
4) Extensive simulations verify that the proposal is superior to
other typical and state-of-the-art fusion methods.
II. Related Work
A. Deep Learning-Based Fusion Methods
With the rise ofdeep learning in the last few years, a lot ofDLbased
methods have been employed inMMIF. This section covers
the DL-based medical image fusion methods first, and then
briefly introduces a novel improved deep learning method.
Lahoud et al. [25] proposed a real-time image fusion
method named " Zero Learning " (ZL) using a pre-trained
CNN to extract depth feature images as weight images, and
70 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
Zhang et al. [33] proposed a general image fusion framework
based on CNN, named IFCNN. The above two methods
only extract the features of the last layer from the networks,
leading to the loss of partial detailed texture information. In
CNN-based networks, with the increase in network depth, a
degradation problem has been exposed. To address this degradation
problem and improve the information flow between
layers, Li et al. [36] introduced an auto-encoding network
with a Siamese structure into the image fusion task, in which
the encoder is composed of CNN layers and dense blocks.
However, the source image features extracted by the
encoders are independent and lack complementarity in this
method. U2Fusion [34] adopts only one convolutional
encoder to replace two Siamese encoders in the feature
extraction of a pair of source images, in which source images
are concatenated in channel dimensions before being fed into
the network. In principle, this encoder can extract weaker
complementary information from a pair of source images, but
it is difficult to produce better conditional features. Shi et al.
[37] proposed an end-to-end multimodal brain image fusion
framework (MMI-fuse) that applies an autoencoder to extract
the features of source images and uses a decoder model to
restructure the fused medical image. However, it adopts an
information preservation weighted channel spatial attention
model (ICS) to fuse the features which then causes the complexity
of the manual design. Ma et al. [31] proposed a dualdiscriminator
conditional generative adversarial network
(DDcGAN) that establishes an adversarial game between a
generator and two discriminators. It does not require
ground-truth fused images for training. However, the success
of GAN was limited as they were known to be unstable for
training. DDcGAN reduces the structural difference between
the fusion image and the source image, but it still has the
problem of unstable training results. A fast unified image
fusion network based on proportional maintenance of gradient
and intensity (PMGI) [38] has been proposed and is
divided into a gradient path and intensity path for information
extraction to avoid loss of information due to convolution.
PMGI considers the detailed texture information of the
source image but ignores the complementarity of the depth
structure information for the source image. These methods
always use the same representations of medical images from
different modalities, resulting in the distortion of unique
information and limiting the fusion performance. Xu et al.
[30] proposed an unsupervised enhanced medical image
fusion network (EMFusion) that performs both surface-level
and deep-level constraints for enhanced information preservation.
EMFusion concerns the unique features of medical
images from different modalities, but it ignores the fact that
these unique features are related to a pair of source images.
Extracting unique features from a single image will lead to
information redundancy. Therefore, a contrastive loss is proposed
to constrain the extracted features, which ensures that
the unique feature of a single image is maintained and the
same feature of a pair of source images is extracted.
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
