IEEE Computational Intelligence Magazine - February 2023 - 75

b) Local contrast feature: Some recent studies in neurobiology
using functional magnetic resonance imaging (fMRI) have
further confirmed that we can extract strong information [46]
features at pixel points with high Phase Congruency (PC)
[47]. Considering that PC is contrast invariant, local contrast
does affect the human visual system's (HVS) perception of
image quality. Therefore, we calculate the image gradient
magnitude (GM) as the secondary feature to encode contrast
information. The proposed method multiplies the PC and
GM of the image as part of the weight of the fidelity term in
the construction ofthe loss function.
V ¼ PCGM
(21)
IV. Experiments
In this section, experimental configures are first presented.
Then, the comparative experiment, parameter discussion, and
ablation experiments are introduced.
A. Experimental Configures
In this section, experimental setup, datasets, compared methods,
and metrics for experimental evaluation are presented.
1) Experimental Setup
We perform the training of our model by Pytorch [48] on a
2080Ti GPU with a memory of 11 GB, and a 3.0 GHz Intel
CPU. The Adam optimizer is employed, and the learning rate
is 0.001, whereas the batch size and the maximum epoch are
set to 1 and 1000, respectively. Moreover, the whole training
phase lasts about 15 hours, whereas the testing phase only takes
about only 0.04 seconds for an image pair ofsize 256256.
2) Training and Test Dataset
To train SSN-CAEþIE, we first select 217 pairs of medical
images with different types, including CT/MR-T1, CT/
MR-T2, MR-T1/MR-T2, MR-T/MR-GAD, MR-T2/
MR-PD, MR-T2/MRI-PET, MR-T2/MRI-SPECT-Tc,
MR-T2/MRI-SPECT-Ti, which are the same size of
256256, with 256 levels. To verify the effectiveness of our
proposed method for the multi-modal image fusion, we select
10 pairs of images from the public datasets1 of CT/MR-T1,
CT/MR-T2, MR-T1/MR-T2, MR-T2/GAD, MR-T2/
MR-PD, MR-T2/MRI-PET, MR-T2/MRI-SPECT-Tc,
and MR-T2/MRI-SPECT-Ti. All image pairs had already
been registered, otherwise, some registration algorithms [49],
[50] need to be employed.
3) Compared Methods
The proposed method is compared with nine images fusion
methods: four traditional methods, including LRD [51],
CSMCA [21], NSST-MSMG [18], and NSST-PAPCNN
[15]; five DL-based methods, composed of ZL [25], PMGI
[52], U2Fusion [34], DDcGAN [31] and IFCNN [33].
1http://www.med.harvard.edu/AANLIB/home.html.
4)Metrics
We select four objective metrics to evaluate the fused results.
Normalized Mutual Information (NMI) [53] is a measure that
overcomes mutual information (MI) instability to measure
how information from the source image is transferred to the
fused image. Feature Similarity Index (FSIM) [47] measures
the structure information and feature information between the
source image and fusion image. The gradient-based fusion
metric QAB/F [54] was used to evaluate the degree of gradient
transferring from the source image to the fused image. Universal
quality index (UIQI) [55] can utilize a universal image quality
index for evaluating the performance ofdifferent gray-level
image fusion schemes. For the four metrics mentioned above,
a large value illustrates the better fusion quality.
B. Comparative Experiments
In this section, to verify the robustness and universality of our
method, we carry out comparative experiments on eight datasets
as shown in Table II. To save space, only a pair ofvisual results of
each dataset is selected for display in Figs. 8, and 9, the quantitative
evaluation of the displayed images is shown in Fig. 10 and
the average evaluation ofall datasets is shown in Fig. 11.
1)Experimental Results on CT/MRI Image Fusion
As shown in Fig. 8, CT/MRI image fusion is tested and the
fused results are compared with the best five methods resulting
from the average ranks in Table I, i.e., EMFusion, MSMG,
ZL, LRD, and CSMCA. For a better comparison, we set up a
red box, enlarged it into a close-up, and then converted it into
a false-color map. In the red boxes for Set-1 and the false-color
maps of Fig. 8, ZL and CSMCA can hardly fully retain the
bone structure information of CT. In addition, as shown in
the marked regions for Set-2 and Set-3, although EMFusion,
LRD, CSMCA, and MSMG well reveal the salient structure
of MR_T2, there is a serious lack of global information (yellow
information in false-color maps) ofMR_T1. Focusing on
the marked regions for Set-4 and Set-5 and their false-color
maps, apart from LRD and our method, the rest of the methods
cannot reveal the structural details and the grassroots effective
information of the MR_PD/GAD. Whereas LRD suffers
from the obscure edge, leading to negative visual effects.
Compared with these methods, the SSN-CAEþIE not only
perfectly highlights the salient structure of the source images,
but also integrates rich texture details and spatial information.
In Table II, the proposed method gives the best scores in terms
ofall metrics for each image set except the UIQI in Set-1, Set3,
and Set-5. However, the UIQI of IFCNN, CSMCA, and
EMFusion are all very weak in other datasets. Our method
presents good fusion performance for various modes of medical
images than other compared methods.
2)Experimental Results on PET/MRI and SPECT/MRI
Image Fusion
To further demonstrate the availability of the proposed
method, PET/MRI and SPECT/MRI fusion are tested in this
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 75
http://www.med.harvard.edu/AANLIB/home.html
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
