IEEE Computational Intelligence Magazine - February 2023 - 73

FIGURE 5 The architecture of MCIEN. The MCIEN is made up by three convolution spaces, an information exchange block (IEB), three CBs, an
inverse proportional operation and the fusion rule. The convolution spaces are the convolution blocks with different sizes of convolution kernel.
" ðA; iÞ and " ðB; iÞ are the multi-kernel features obtained from multi-convolution spaces, and " B
ðA; iÞ and " A
receptive fields to obtain a better estimation of fusion contributions
for source images. As shown in Fig. 5, given inputs eA,
eB from CAE, feeding them into the MCIEN, and then
exchanging their information with three convolution kernels
respectively to obtain the complete exchanged information
" ðA; iÞ; " ðB; iÞ, the sizes of three convolutions become: 33,
55, and 77.
As shown in ALGORITHM 1, the process can be formulated
as:
" A; iðÞ; " B; iðÞ ¼ Conv " A; i1ðÞ; " B; i1ðÞ
; ki
(5)
where " ðA; iÞ and " ðB; iÞ are the multi-convolutional features of
A,and B that are obtained by the convolution ofdifferent kernels
ofsize. Convð; kiÞ denotes the i-th convolution block with
the kernel ofsize ki, i ¼ [1], [2], [3] and " ðA; 0Þ, " ðB; 0Þ¼ eA, eB.
To acquire the full fusion contribution of A and B,we adopt
an information exchange block (IEB) to exchange the multiconvolutional
features ofA and B and formulated as follows:
" B
IEB X; YðÞ¼ split XYðÞ
ðÞA; i ; " A
ðÞB; i ¼ IEBð " A; iðÞ; " B; iðÞÞ
(6Þ
(7)
where denotes the concatenation operation, we split a
concatenated tensor into two pieces as shown in Fig. 6. " B
the exchanged information from B to A in the i-th convolution
space, and vice versa for " A
ðB; iÞ. Then, DEA and DEB are
obtained by reducing the dimension ofexchanged information
through layer-by-layer concatenation and convolution
ðA;iÞ is
ðB; iÞ are the exchanged multi-kernel features.
operation. Finally, the fusion contribution wA, wB of source
images A and B are obtained by inverse proportional operation
(IPO).
IPO DEA; DEBðÞ¼
DEB
DEA þ DEB DEA þ DEB
;
DEA
(8Þ
wA; wB ¼ IPO DEA; DEBðÞ (9)
C. Training
There are two stages in the training of our framework: pretraining
for CAE and self-supervised training for SSNCAEþIE.
Specifically, we train the CAE to make it extracts
the features of source images in pairs contrastively. Then, the
extracted features are utilized to train the MCIEN and finetune
the CAE in a self-supervised way simultaneously.
Algorithm 1. Formulation of the MCIEN
Input: eA; eB;
Initialization: " ðA; 0Þ ¼ eA , " ðB; 0Þ ¼ eB ;
%The multi-kernel convolution:
1 " ðA; iÞ, " ðB; iÞ Convðð " ðA; i1Þ; " ðB; i1ÞÞ, kiÞ, ki¼ [33, 55,
77]
2
%The Information exchange:
IEBðX; YÞ splitðXYÞ
3 " B
ðA; iÞ; " A
ðB; iÞ IEBð " ðA; iÞ; " ðB; iÞÞ
%Dimension reduction:
4 DEA CB3ðCB2ðCB1ð " B
5 DEB CB3ðCB2ðCB1ð " A
%Contribution estimation:
6
IPOðDEA; DEBÞ DEB
8 Output: wA; wB
FIGURE 6 Detailed architecture of IEB. The IEB process is to
concatenate " ðA; iÞ with " ðB; iÞ first, then the concatenated feature map is
ðB; iÞ.
divided into " B
ðA; iÞ and " A
1)Contrastive learning for CAE
The main idea of contrastive learning is to push positive pairs
close and negative pairs apart by a contrastive loss. Inspired by
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 73
ðA; 1Þ " B
ðB; 1Þ " A
;
7 wA; wB IPOðDEA; DEBÞ
end
ðA; 2ÞÞ " B
ðB;2ÞÞ " A
DEA
DEAþDEB DEAþDEB
ðA; 3ÞÞÞ
ðB;3ÞÞÞ
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
