IEEE Geoscience and Remote Sensing Magazine - June 2023 - 86

PERSPECTIVES
NIRAV PATEL
Generative Artificial Intelligence
and Remote Sensing
A perspective on the past and the future
T
he first phase of 2023 has been marked with an explosion
of interest around generative AI systems,
which generate content. This type of machine learning
promises to enable the creation of synthetic data and
outputs in many different modalities. OpenAI's ChatGPT
has certainly taken the world by storm and opened
discourse on how the technology should be used.
Historically, generative models are certainly not new,
OpenAI's ChatGPT HAS
CERTAINLY TAKEN THE
WORLD BY STORM AND
OPENED DISCOURSE ON
HOW THE TECHNOLOGY
SHOULD BE USED.
dating back to the 1950s, with hidden Markov models
and Gaussian mixture models [1], [2], [3]. The recent
development of deep learning
has allowed for generative models'
utility. In the early days of
deep generative models, N-gram
language modeling was utilized
to generate sentences in natural
language processing (NLP) [4].
This modeling did not scale well
to generating long sentences, and
hence, recurrent neural networks
(RNNs) were introduced to deal
with longer dependencies [5]. RNNs were followed by
the development of long short-term memory [6] and gated
recurrent unit methods, which leveraged gating mechanisms
to control memory usage during training [7].
In the computer vision arena (more aligned with remote
sensing), traditional image generation algorithms
utilized techniques such as texture mapping [8] and
texture synthesis [9]. These methods were very limited
The views expressed in this publication reflect those of the author
and do not necessarily reflect the official policy or position of the
U.S. Government or the Department of Defense (DoD).
Digital Object Identifier 10.1109/MGRS.2023.3275984
Date of current version: 30 June 2023
86
and could not generate complex and diverse images. The
introduction of generative adversarial networks (GANs)
[10] and variational autoencoders [11] in the past decade
or so has allowed for more control over the image generation
process to generate high-resolution images.
Generative models in different modalities felt the advancement
of the field in its totality with the introduction
of the transformer architecture [12]. Large language
models, such as the generative pretrained transformer
(GPT), adopt this architecture as the primary building
block, which initially had significant utility in the NLP
world before later modifications to this architecture allowed
for application to image-based streams of information
[13], [14], [15], [16], [17]. Transformers consist
of an encoder and a decoder, where the encoder takes
in an input sequence and generates hidden representations,
while the decoder has a multihead attention and
feedforward NN [1]. See Figure 1 for an NLP example
of a sentence being translated from English to Japanese.
The emergence of these techniques has allowed for
the creation of foundation models, which are the technical
scaffolding behind generative AI capabilities.
Foundation models, such as ChatGPT, learn from unlabeled
datasets, which saves a significant amount of time
and the expense of manual annotation and human attention.
However, there is a reason why the most wellresourced
companies in the world have made an attempt
at generating these models [19].
First, you need the best computer scientists and engineers
to maintain and tweak foundation models, and
second, when these models are training data from the
whole Internet, the computational cost is not insignificant.
OpenAI's GPT-3 was trained on roughly 45 TB of
text data (equivalent to 1 million feet of bookshelf space),
which cost several million dollars (estimated) [19].
With remote sensing applications, anecdotally, I
have witnessed the rise of the use of GANs over the past
few years. This deep learning technique, as mentioned
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2023
https://orcid.org/0000-0002-0314-2579
IEEE Geoscience and Remote Sensing Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2023
