IEEE Computational Intelligence Magazine - February 2023 - 69

an auto-encoder. The paired source images are considered as
positive and negative results used to reconstruct the source
images to avoid the information redundancy problem. This
article proposes preserving both the global and local features
based on prior knowledge by combining transformer and
convolution neural networks in parallel as an auto-encoder. A
contribution estimation model is adopted to fuse multi-modal
medical images. In the contribution estimation stage, an
information exchange block is designed to exchange the
feature maps of source images in multi-kernel convolutions,
and then the multi-convolutional features are utilized to
estimate the best fusion contribution of the paired source
images. Experiments demonstrate that our method gives the
best results than other state-of-the-art fusion approaches.
I. Introduction
T
he multi-modal medical image fusion (MMIF) is
a process of merging multiple images from a singleormultipleimaging
modalities [1]. Different
medical imaging modalities such as computed
tomography (CT), magnetic resonance imaging (MRI), positron
emission tomography (PET), and single-photon emission
tomography (SPET) present dense structures, soft
tissues, metabolism, or other different information for the
human body [2], [3], [4]. For example, CT images can precisely
detect dense structures such as bones and implants,
whereas MRI provides high-resolution anatomical information
for soft tissues, but is less sensitive to the diagnosis of
fractures than CT. In MR, MR-T1 images offer information
on the anatomical structure of tissues, while MR-T2
can depict normal and pathological tissues. In comparison
with these anatomical images, PET and SPECT focus on
functional information such as blood flow and significant
metabolic change. The main interest of the fusion of MRI
and PET/SPECT is to supply anatomical information for
the accurate detection of pathologic areas characterized in
functional imaging by physiological abnormalities. Clinical
implications are diverse and numerous (e.g., a study of
dementia for localizing functional abnormalities [5], [6],
neurotransmission SPECT imaging, and accurate quantification
of monoamine transmitters). The ultimate aim of
MMIF is to improve imaging quality by preserving the
complementary information from multi-modal medical
images for increasing the clinical applicability of images for
medical diagnosis and treatment [7], [8].
At present, various medical image fusion methods have
been proposed and achieved good results. These methods can
be roughly divided into two categories: traditional-based
methods and deep learning (DL)-based methods. Most traditional
medical image fusion algorithms are based on the transform
domain and are at the pixel level [9]. Among them,
sparse representation (SR) and multi-scale transform (MST)
based methods are the most commonly employed traditional
methods in the recent years. The general idea ofMST methods
includes three steps: 1) image transform, 2) coefficients
fusion, and 3) inverse transform. Many researchers introduce
the discrete transform to the image fusion task, such as discrete
cosine transform (DCT) [10], discrete wavelet transform
(DWT) [11], [12], and dual-tree complex wavelet transform
(DT-CWT) [13]. They have obtained good fusion results, but
the change ofresolution during the transformation process will
cause partial distortion and may produce artifacts. Li et al. [14]
proposed a novel laplacian re-decomposition (LRD) framework
tailored to multi-modal medical image fusion to address
color distortion, blurring, and noise in image fusion. A medical
image fusion method with non-subsampled shearlet transform
(NSST) domain and parameter-adaptive pulse-coupled neural
network (NSST-PAPCNN) has been proposed [15], the
NSST [16] can effectively solve the distortion problem due to
its flexible multi-scale image decomposition method with
invariable translation. Zhu et al. [17] presented a Phase Congruency
and Local Laplacian Energy Based Multi-Modality
Medical Image Fusion Method in NSCT Domain (PCLLNSST).
Tan et al. [18] proposed a method based on the application
of a boundary-measured pulse-coupled neural network
fusion strategy and an energy attribute fusion strategy in a
non-subsampled shearlet transform domain (NSST-MSMG).
These MST methods may bring about incorrect artifacts and
are not sensitive to sensor noise, and sparse representation (SR)
methods were proposed to address this issue. SR represents the
salient information of images by establishing the relationship
between features and the sparse coefficients. Some classical
SR-based methods [19], [20], [21] have a low computational
efficiency due to the use of a large number of over-complete
dictionaries and related forward/backward domain transformations.
Liu et al. [22] proposed a sparse representation (SR)
model named convolutional sparsity-based morphological
component analysis (CSMCA). However, the decomposition
method, fusion rule design, and calculation ofthese traditional
methods are laborious and complex, and cannot solve the
defects caused by the manual design ofthe fusion strategy.
Recently, deep learning (DL) has been applied to various
image fusion problems because it can be considered an
automatic feature extractor for more expressive image features.
Compared with traditional image fusion methods,
deep learning has advantages of a stable fusion effect and
fast speed. Some combinations of traditional and DL-based
methods have been proposed. A fusion method based on a
convolutional neural network and contrast pyramid (CNNCP)
[23] that uses the trained Siamese convolutional network
to fuse the pixel activity information of source images
to realize the generation of a weight map and implements a
contrast pyramid to decompose the source image. Besides,
deep learning has also been widely employed for MMIF
tasks in the last few years. Convolutional Neural Networks
(CNN) [24], [25], [26] have been applied in the MMIF and
achieved a good fusion effect that briefly demonstrates the
potential of the learned CNN model in other types of
image fusion problems. Nevertheless, the major obstacle in
utilizing DL-based methods for MMIF lies in the lack of
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 69
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
