Quarterly Mark December 2023 - 94

Prompt-Based Learning
A successful traditional pre-train and fine-tune method of the pre-training model
such as BERT requires a minimum size of labeled data from downstream tasks, but
in many applications, it is a luxury to find a sufficient amount of labeled data. In
addition, the traditional pre-train and fine-tune paradigm often requires modification
of the language model's architecture when applying to downstream tasks. The
prompt-based learning is inspired by the fact that a task tends to be easier if there is
a task description that helps understand what the task is. By introducing a prompt,
the method transfers a specific task into a masked language prediction problem,
and hence, overcomes two issues: (1) it reduces the need of a large amount of taskspecific
labeled data and (2) it mitigates the gap between pre-training and finetuning.
A prompt usually consists of a template and a verbalizer.
Tem plate formulation. A crucial step of prompt construction is to formulate a
template. A template is a piece of text that wraps the original text with a sequence
including predefined text and mask token [MASK]. Let x be the original input, we
denote a template by F(x). The design of the template depends on the task to be
solved. For example, given an input x in a classification task or a sentiment analysis,
a template can be formulated as
Fx =() [CLS] Itis[MASK]
x
Verbalizer construction. A verbalizer is an injective mapping v: L → V that maps
each label in task-specific label space L to one or more words in the vocabulary V of
a pre-trained language model. For instance, in sentiment analysis, the verbalizer can
be set as { " positive " : " great " } and { " negative " : " terrible " }.
After defining the prompt, we can then use it to solve specific tasks. Specifically,
given an input x and a task specific label space L. We firstly transfer the original
input into the form of a template F(x). Next, we employ a pre-trained language
model with vocabulary V to determine the label y ∈ L such that v(y) ∈ V has the
highest probability to fill the mask token [MASK]. In practice, we can choose a wide
variety of pre-trained language models, ranging from a moderately sized language
model (e.g., BERT, RoBERTa) to a large-sized language model (e.g., GPT-3).
Training strategy. Before applying a prompt to solve downstream tasks, it is
necessary to design the training strategy for a prompt-based method. Because the
downstream task can be incorporated into the prompt, we can directly use the pretrained
language model to predict the desired text of the mask token in the prompt
and obtain the answer for the task of interest. Therefore, in many circumstances the
pre-trained language model can be applied to downstream task without additional
training under prompt-based approach (zero-shot learning). In contrast, we can also
use task-specific data to train the model. The prompt-based paradigm is shown to
be effective when only a small number of task-specific data is available (few-shot
learning).
The parameters of prompt-based learning usually include parameters from the
pre-trained language model and prompt. The fine-tuning strategy varies depending
on the choice of pre-trained model and specific downstream task. We present four
typical fine-tuning methods in prompt-based learning as follows (Liu et al. 2021):
1. Tune-free prompting: No additional prompt parameters are introduced,
and the parameters of pre-trained language models are not updated. The
method directly generates task answers with the aid of the prompt.
2. Fixed prompt language model tuning: No additional prompt parameters are
introduced, and the parameters of pre-trained language models are finetuned
using the traditional pre-train and fine-tune approach. In this method,
94 | ESG Text Classification: An Application of the Prompt-Based Learning Approace

Quarterly Mark December 2023

Table of Contents for the Digital Edition of Quarterly Mark December 2023