Table 2. Human evaluation results of entries in the 2015 COCO Captioning Challenge. Entry M1 M2 M3 M4 M5 Date Human 0.638 0.675 4.836 3.428 0.352 23 March 2015 Google 0.273 0.317 4.107 2.742 0.233 29 May 2015 MSR 0.268 0.322 4.137 2.662 0.234 8 April 2015 Montreal/Toronto 0.262 0.272 3.932 2.832 0.197 14 May 2015 MSR Captivator 0.25 0.301 4.149 2.565 0.233 28 May 2015 Berkeley LRCN 0.246 0.268 3.924 2.786 0.204 25 April 2015 m-RNN 0.223 0.252 3.897 2.595 0.202 30 May 2015 Nearest Neighbor 0.216 0.255 3.801 2.716 0.196 15 May 2015 PicSOM 0.202 0.25 3.965 2.552 0.182 26 May 2015 Brno University 0.194 0.213 3.079 3.482 0.154 29 May 2015 m-RNN (Baidu/UCLA) 0.19 0.241 3.831 2.548 0.195 26 May 2015 MIL 0.168 0.197 3.349 2.915 0.159 29 May 2015 MLBL 0.167 0.196 3.659 2.42 0.156 10 April 2015 NeuralTalk 0.166 0.192 3.436 2.742 0.147 15 April 2015 ACVT 0.154 0.19 3.516 2.599 0.155 26 May 2015 Tsinghua Bigeye 0.1 0.146 3.51 2.163 0.116 23 April 2015 Random 0.007 0.02 1.084 3.247 0.013 29 May 2015 Table 3. The state-of-the-art image captioning systems in automatic metrics (as of 8 December 2016). 114 Entry CIDEr-D METEOR BLEU-4 SPICE (x10) Date Watson Multimodal 1.123 0.268 0.344 0.204 16 November 2016 DONOT_FAIL_AGAIN 1.01 0.262 0.32 0.199 22 November 2016 Human 0.854 0.252 0.217 0.198 23 March 2015 MSM@MSRA 1.049 0.266 0.343 0.197 25 October 2016 MetaMind/VT_GT 1.042 0.264 0.336 0.197 1 December 2016 ATT-IMG (MSM@MSRA) 1.023 0.262 0.34 0.193 13 June 2016 G-RMI(PG-SPIDEr-TAG) 1.042 0.255 0.331 0.192 11 November 2016 DLTC@MSR 1.003 0.257 0.331 0.19 4 September 2016 Postech_CV 0.987 0.255 0.321 0.19 13 June 2016 G-RMI (PG-BCMR) 1.013 0.257 0.332 0.187 30 October 2016 feng 0.986 0.255 0.323 0.187 6 November 2016 THU_MIG 0.969 0.251 0.323 0.186 3 June 2016 MSR 0.912 0.247 0.291 0.186 8 April 2015 reviewnet 0.965 0.256 0.313 0.185 24 October 2016 Dalab_Master_Thesis 0.96 0.253 0.316 0.183 28 November 2016 ChalLS 0.955 0.252 0.309 0.183 21 May 2016 ATT_VC_REG 0.964 0.254 0.317 0.182 3 December 2016 AugmentCNNwithDe 0.956 0.251 0.315 0.182 29 March 2016 AT 0.943 0.25 0.316 0.182 29 October 2015 Google 0.943 0.254 0.309 0.182 29 May 2015 TsinghuaBigeye 0.939 0.248 0.314 0.181 9 May 2016 IEEE SIGNAL PROCESSING MAGAZINE | November 2017 |