Signal Processing - November 2017 - 107

[2] D. Lahat, T. Adali, and C. Jutten, "Multimodal data fusion: An overview of
methods, challenges, and prospects," Proc. IEEE, vol. 103, no. 9, pp. 1449-1477,
2015.

[27] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic
segmentation," in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015,
pp. 3431-3440.

[3] P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, "Multimodal
fusion for multimedia analysis: A survey," Multimedia Systems, vol. 16, no. 6, pp. 345-
379, 2010.

[28] I. Lenz, H. Lee, and A. Saxena, "Deep learning for detecting robotic grasps," Int.
J. Robotics Res., vol. 34, no. 4-5, pp. 705-724, 2015.

[4] B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi, "Multisensor data
fusion: A review of the state-of-the-art," Inform. Fusion, vol. 14, no. 1, pp. 28-44,
2013.

[29] S. Gu, E. Holly, T. Lillicrap, and S. Levine. (2016). Deep reinforcement learning
for robotic manipulation with asynchronous off-policy updates. arXiv. [Online].
Available: https://arxiv.org/abs/1610.00633

[5] L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms.
Hoboken, NJ: Wiley, 2004.

[30] N. Srivastava and R. Salakhutdinov, "Learning representations for multimodal
data with deep belief nets," presented at Proc. 29th Int. Conf. Machine Learning
(Workshop), 2012.

[6] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep
learning," in Proc. 28th Int. Conf. Machine Learning (ICML-11), 2011, pp. 689-696.

[31] Y. Cao, S. Steffey, J. He, D. Xiao, C. Tao, P. Chen, and H. Müller, "Medical image
retrieval: A multimodal approach," Cancer Informatics, vol. 13, no. Suppl 3, p. 125, 2014.

[7] N. Srivastava and R. R. Salakhutdinov, "Multimodal learning with deep Boltzmann
machines," in Proc. Advances in Neural Inform. Processing Syst., 2012, pp. 2222-
2230.

[32] M. Liang, Z. Li, T. Chen, and J. Zeng, "Integrative data analysis of multi-platform
cancer data with a multimodal deep learning approach," IEEE/ACM Trans. Comput.
Biol. Bioinf., vol. 12, no. 4, pp. 928-937, 2015.

[8] N. Neverova, C. Wolf, G. Taylor, and F. Nebout, "ModDrop: Adaptive multi-modal
gesture recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1692-
1706, 2016.

[33] K. Simonyan and A. Zisserman, "Two-stream convolutional networks for action
recognition in videos," in Proc. Advances in Neural Information Processing Systems,
2014, pp. 568-576.

[9] S. S. Mukherjee and N. M. Robertson, "Deep head pose: Gaze-direction estimation
in multimodal video," IEEE Trans. Multimedia, vol. 17, no. 11, pp. 2094-2107, 2015.

[34] S. Poria, E. Cambria, and A. Gelbukh, "Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis," in Proc. Conf. Empirical Methods on Natural Language Processing, 2015, pp.
2539-2544.

[10] C. Ding and D. Tao, "Robust face recognition via multimodal deep face representation," IEEE Trans. Multimedia, vol. 17, no. 11, pp. 2049-2058, 2015.
[11] S. E. Kahou, X. Bouthillier, P. Lamblin, C. Gulcehre, V. Michalski, K. Konda, S.
Jean, P. Froumenty, et al., "EmoNets: Multimodal deep learning approaches for emotion recognition in video," J. Multimedia User Interfaces, vol. 10, no. 2, pp. 99-111,
2015.
[12] Z. Sitová, J. Šedeˇnka, Q. Yang, G. Peng, G. Zhou, P. Gasti, and K. S. Balagani,
"HMOG: New behavioral biometric features for continuous authentication of smartphone users," IEEE Trans. Inf. Forensics Security, vol. 11, no. 5, pp. 877-892, 2016.
[13] V. Radu, N. D. Lane, S. Bhattacharya, C. Mascolo, M. K. Marina, and F. Kawsar,
"Toward multimodal deep learning for activity recognition on mobile devices," in Proc.
ACM Int. Joint Conf. Pervasive and Ubiquitous Computing: Adjunct, 2016, pp. 185-
188.
[14] A. Jain, J. Tompson, Y. LeCun, and C. Bregler, "Modeep: A deep learning framework using motion features for human pose estimation," in Proc. Asian Conf.
Computer Vision, 2014, pp. 302-315.
[15] A. Valada, G. L. Oliveira, T. Brox, and W. Burgard, "Deep multispectral semantic
scene understanding of forested environments using multimodal fusion," in Proc. Int.
Symp. Experimental Robotics (ISER 2016), 2016, pp. 465-477.
[16] D. Shen, G. Wu, and H.-I. Suk, "Deep learning in medical image analysis," Annu.
Review Biomedical Eng., vol. 19, pp. 221-248, 2017.
[17] R. Kiros, K. Popuri, D. Cobzas, and M. Jagersand, "Stacked multiscale feature
learning for domain independent medical image segmentation," in Proc. Int. Workshop
on Mach. Learning in Medical Imaging, 2014, pp. 25-32.
[18] P. Wu, S. C. Hoi, H. Xia, P. Zhao, D. Wang, and C. Miao, "Online multimodal
deep similarity learning with application to image retrieval," in Proc. 21st ACM Int.
Conf. Multimedia. 2013, pp. 153-162.
[19] M. Simonovsky, B. Gutiérrez-Becker, D. Mateus, N. Navab, and N. Komodakis,
"A deep metric for multimodal registration," in Proc. Int. Conf. Medical Image
Computer and Computer-Assisted Intervention, 2016, pp. 10-18.
[20] S. Liu, S. Liu, W. Cai, H. Che, S. Pujol, R. Kikinis, D. Feng, M. J. Fulham,
et al., "Multimodal neuroimaging neature learning for multiclass diagnosis of
Alzheimer's disease," IEEE Trans. Biomed. Eng., vol. 62, no. 4, pp. 1132-1140,
2015.
[21] P. Mamoshina, A. Vieira, E. Putin, and A. Zhavoronkov, "Applications of deep
learning in biomedicine," Molecular Pharmaceutics, vol. 13, no. 5, pp. 1445-1454, 2016.
[22] Y. Guo, G. Wu, L. A. Commander, S. Szary, V. Jewells, W. Lin, and D. Shen,
"Segmenting hippocampus from infant brains by sparse patch matching with deeplearned features," in Proc. Int. Conf. Medical Image Computer and ComputerAssisted Intervention, 2014, p. 308.

[35] A. Jain, A. Singh, H. S. Koppula, S. Soh, and A. Saxena, "Recurrent neural networks for driver activity anticipation via sensory-fusion architecture," in Proc. 2016
IEEE Int. Conf. Robotics and Automation (ICRA), 2016, pp. 3118-3125.
[36] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh,
"VQA: Visual question answering," in Proc. Int. Conf. Computer Vision (ICCV), 2015,
pp. 2425-2433.
[37] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan,
K. Saenko, and T. Darrell, "Long-term recurrent convolutional networks for visual recognition and description," in Proc. IEEE Conf. Computer Vision and Pattern
Recognition, 2015, pp. 2625-2634.
[38] A. Karpathy, A. Joulin, and F. F. F. Li, "Deep fragment embeddings for bidirectional image sentence mapping," in Proc. Advances in Neural Information Processing
Systems, 2014, pp. 1889-1897.
[39] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image
caption generator," in Proc. IEEE Conf. Computer Vision and Pattern Recognition,
2015, pp. 3156-3164.
[40] M. Ren, R. Kiros, and R. Zemel, "Exploring models and data for image question
answering," in Proc. Advances in Neural Information Processing Systems, 2015, pp.
2953-2961.
[41] J.-H. Kim, S.-W. Lee, D.-H. Kwak, M.-O. Heo, J. Kim, J.-W. Ha, and B.-T. Zhang.
(2016). Multimodal residual learning for visual QA. arXiv. [Online]. Available: https://
arxiv.org/abs/1606.01455
[42] F. J. Ordóñez and D. Roggen, "Deep convolutional and LSTM recurrent neural
networks for multimodal wearable activity recognition," Sensors, vol. 16, no. 1, p. 115,
2016.
[43] D. Wu, L. Pigou, P.-J. Kindermans, N. D.-H. Le, L. Shao, J. Dambre, and J.-M.
Odobez, "Deep dynamic neural networks for multimodal gesture segmentation and recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1583-1597,
2016.
[44] H. Larochelle and Y. Bengio, "Classification using discriminative restricted
Boltzmann machines," in Proc. 25th Int. Conf. Machine Learning, 2008, pp. 536-543.
[45] Y. Huang, W. Wang, and L. Wang, "Unconstrained multimodal multi-label learning," IEEE Trans. Multimedia, vol. 17, no. 11, pp. 1923-1935, 2015.
[46] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A.
Courville, and Y. Bengio, "Generative adversarial nets," in Proc. Advances in Neural
Information Processing Systems, 2014, pp. 2672-2680.
[47] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," in Proc. Int.
Conf. Learning Representations (ICLR), 2014.

[23] N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway,
and J. Liang, "Convolutional neural networks for medical image analysis: Full training
or fine tuning?" IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1299-1312, 2016.

[48] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, "Generative
adversarial text to image synthesis," in Proc. 33rd Int. Conf. Machine Learning
(ICML), 2016, pp. 1060-1069.

[24] I. Sturm, S. Lapuschkin, W. Samek, and K.-R. Müller, "Interpretable deep neural
networks for single-trial eeg classification," J. Neuroscience Methods, vol. 274, pp.
141-145, Dec. 2016.

[49] M. Suzuki, K. Nakayama, and Y. Matsuo. (2016). Joint multimodal learning
with deep generative models. arXiv. [Online]. Available: https://arxiv.org/
abs/1611.01891

[25] S. G. Kim, N. Theera-Ampornpunt, C.-H. Fang, M. Harwani, A. Grama, and S.
Chaterji, "Opening up the blackbox: an interpretable deep neural network-based classifier for cell-type specific enhancer predictions," BMC Syst. Biology, vol. 10, no. 2, p.
54, 2016.

[50] G. Pandey and A. Dukkipati. (2016). Variational methods for conditional multimodal deep learning. arXiv. [Online]. Available: https://arxiv.org/abs/1603.01801
[51] L. Deng, "A tutorial survey of architectures, algorithms, and applications for deep
learning," APSIPA Trans. Signal and Inform. Processing, vol. 3, pp. 1-29, 2014.

[26] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, "Deepdriving: Learning affordance
for direct perception in autonomous driving," in Proc. IEEE Int. Conf. Computer
Vision, 2015, pp. 2722-2730.

[52] M. R. Amer, T. Shields, B. Siddiquie, A. Tamrakar, A. Divakaran, and S. Chai,
"Deep multimodal fusion: A hybrid approach," Int. J. Comput. Vision, pp. 1-17, 2017.
DOI: 10.1007/s11263-017-0997-7.

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|

107
https://www.arxiv.org/abs/1610.00633 http://https:// http://www.arxiv.org/abs/1606.01455 https://www.arxiv.org/ https://www.arxiv.org/abs/1603.01801
Table of Contents for the Digital Edition of Signal Processing - November 2017
