IEEE Geoscience and Remote Sensing Magazine - December 2023 - 84

have directional information have been edited to match
the augmented image.
RSVG
The RSVG dataset [133] was proposed for studying visual
grounding in the RS domain. To construct this dataset, the
authors utilized the DIOR [74] dataset, which contains
23,463 RS images; 192,472 object instances; and 20 object
categories. They developed an automatic image-query
generation method with manual assistance to collect image/expression/box
triplets. The method involves box
sampling, attribute extraction, expression generation, and
worker verification. They preset textual templates and fill
attributes to generate a query expression for each bounding
box. The dataset consists of 38,320 language expressions
across 17,402 RS images and contains 20 object categories.
The image size is 800 800#
pixels, and the spatial resolution
range is from 0.5 to 30 m. The splitting of the dataset
is 26,991; 3,829; and 7,500 images for training, validation,
and testing, respectively.
Table 8 shows the state-of-the-art results of text-image
retrieval models on different text-image datasets in
RS. The results of the UCM-captions dataset indicate that
the fusion-based contrastive learning model (FBCLM) [88]
achieves the highest scores on the R@1 and R@5 metrics
and comes second on the R@10 metric in both text-to-image
and image-to-text retrieval tasks. On the RSICD dataset,
FBCLM produces the highest results in the text-to-image
retrieval task, while the MAFA model [92] shows the best
overall performance with the highest mR score. The results
of the Sydney-captions dataset show that the FBCLM model
obtains the highest text-to-image scores, while the CABIR
[90] model achieves the best image-to-text scores. For the
RSITMD dataset, the CLIP-based model proposed in [87].
The results of the NWPU-captions dataset show that the
highest text-to-image scores are obtained with the SAM [84]
model, while the best image-to-text scores are obtained with
the MAFA [92] model. Finally, the DUM [79] model achieves
the best results on the TextRS dataset.
VISUAL QUESTION ANSWERING DATASETS
In recent years, several datasets for VQA have been constructed
in the RS community. These datasets are annotated
with questions and answers about the content of RS images
and have provided helpful benchmarks for research.
Table 9 presents a comparison of some publicly available
VQA datasets. In the following sections, we describe these
datasets in more detail.
RSVQA
The dataset [99] is the first dataset for the VQA task in RS. RSVQA
consists of two datasets based on image resolution: the
low-resolution dataset (LR) and the high-resolution dataset
(HR). In both datasets, the information required to construct
the questions and answers was obtained from OpenStreetMap.
The RSVQA-HR dataset used U.S. Geological
84
RSIVQA
This VQA dataset [100] is based on two scene classification
and three object detection datasets. The classification
datasets are UCM [144], Sydney [145], and AID
[146]. The object detection datasets include HRRSD
[151] and DOTA [1]. The majority of the questions in
this dataset are automatically generated using sceneand
object-level annotation with only a small fraction
of the dataset annotated by humans. The human-annotated
samples include questions about scenes, objects,
relative locations, color, and shape. The total number of
image/question/answer triplets in the dataset is 111,134.
The questions are of three types: presence, counting, and
other. The dataset is split into three sets: the training
set, which represents 80% of the entire set; the testing
set, which comprises 10%; and the validation set, which
comprises 10%.
RSVQAxBEN
This dataset [102] is the largest freely available dataset for
VQA in the RS domain. It is based on the BigEarthNet classification
dataset [152], which consists of 590,326 Sentinel-2
image patches with 12 spectral bands of varying resolution.
In RSVQAxBEN, only the RGB bands of BigEarthNet were
used, which have a spatial resolution of 10 m. The dataset
includes two types of questions: yes/no questions about the
presence of land cover classes and questions for which the
answer is one or multiple land cover class names.
FloodNET
This dataset [118] is a high-resolution postdisaster damage
assessment dataset containing high-resolution UAV imagery
with pixel-level annotations proposed for classification,
semantic segmentation, and VQA tasks. The data were
acquired following Hurricane Harvey using a tiny UAV
platform, DJI Mavic Pro quadcopters. Hurricane Harvey, a
category 4 hurricane, made landfall near TX and LA, USA,
in August 2017. The Harvey dataset includes video and images
from many flights operated between 30 August and
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2023
Survey high-resolution orthorectified images of New York
and Philadelphia, with image tiles of size 512 # 512 pixels
and a resolution of 15.24 cm/pixel. The RSVQA-LR dataset
was built from seven tiles acquired using the Sentinel-2
satellite over an area of 6.55 km2 in The Netherlands. Each
image in this dataset has a size of 256 # 256 pixels with
RGB spectral channels and a spatial resolution of 10 m.
The dataset contains 772 images, with a default split of 572,
100, and 100 images for training, validation, and testing,
respectively. The total number of questions in the dataset is
77,232, with each image annotated with 100 or 101 questions.
The questions in the dataset include five types: count,
presence, area, comparison, and rural/urban classification.
Questions about object area are only asked in the HR dataset,
while questions about urban/rural region classification
are only asked in the LR dataset.
IEEE Geoscience and Remote Sensing Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2023
