IEEE Solid-States Circuits Magazine - Fall 2019 - 25

Input Data

Is the input safe?

Victim Model

Execution
Units

Grid-Search and Heuristic
Algorithms
One way to solve the maximization
problem of (1) is to evaluate all possible
parameter settings and select the one
that renders the best score function.
Naively evaluating all possible parameters can be quite costly. That's why
smart grid-search algorithms are highly
preferred. In many system-design
tasks, the total number of possible configurations can be reduced to a much
smaller set. As an example, we consider
a robustness-assurance task where a
set of defender modules is used to protect a machine-learning model against
integrity attacks. We briefly discuss the
task here (Figure 1); for detailed system specifications, refer to [11]. The
system to be designed in this example
is a domain-specific accelerator implemented on a field-programmable gate
array (FPGA) that can perform the computations of all defender modules as
fast as possible. We assume that N def
defender modules are to be executed in
parallel while each defender itself uses
N PU processing units from the underlying hardware accelerator. The limiting
factors in this example are hardware
resources, such as the available digital
signal processing (DSP) units (#DSP),
memory (#BRAM), flip-flops (#FF), and
logic gates (#LOGIC) on the underlying
FPGA board. The objective here is to
maximize the number of defender
modules by setting N def and N pu,
while abiding by throughput and platform constraints. This problem can be
efficiently solved using a grid-search
approach: for a fixed N def, the total available resources for each defender module are ^#DSP N def h, ^#BRAM N def h,
^#FF N def h, and (#LOGIC N def ). Given
these constraints, the maximum values for N PU and the throughput are
uniquely determined. Therefore, the

PU PU . . .
PU PU

PU PU
PU PU

Distribute Resources
Hardware
Resources
Defender Modules

DSP
Units

Block
RAMs

FlipFlops

Logic
Gates

FPGA Acceleration

FIGURE 1: A chart illustrating domain-specific FPGA acceleration for defending a victim
machine-learning model against integrity attacks. The number of parallel defender modules
(Ndef) and the per-unit parallelism factor (NPU) should be specified according to the FPGA's
resource availability to obtain optimal performance. PU: processing unit; RAMs: random
access memories.

dimensionality of the search space can
be reduced by one. We evaluate the
throughput under different N def s and
the corresponding unique N PU and
select the configuration with the maximum N def that satisfies the throughout
requirement. Domain-specific accelerator design [12] for machine-learning
applications is another example use
case of grid-search methodologies for
system optimization. Unlike the previous examples, many tasks are not
reducible to smaller problems due to
the interdependency among the optimization parameters. One approach for
tackling such tasks is to use a greedysearch algorithm.

Heuristic (Greedy) Algorithms
A greedy algorithm starts from an
initial hyperparameter vector, x, and
modifies this vector toward a better solution in a step-by-step manner. Let us consider the task of deep
neural network (DNN) compression
[13], [14] as an example optimization
problem. DNNs are hierarchical architectures formed by stacking multiple
layers (Figure 2). Each layer extracts a
set of features from an input vector.
In particular, the lth layer performs
a linear operation between the layer
input and a weight tensor W l " l +1, followed by a nonlinearity. The weight
matrix is known as the layer parameter. The goal of DNN compression is
to reduce the model complexity by

applying a compression technique. Figure 2 presents an example compression technique for quantization that
aims to reduce the DNN's memory
footprint by lowering the number of
bits required for model storage and
computation. In this scenario, the optimization hyperparameters are the
number of bits used to represent the
model's parameters/outputs. These
hyperparameters should be specified for each layer of the DNN. The
parameter space to be searched is,
therefore, a vector x ! R L for an Llayer DNN, where the ith element x[i]
is the bitwidth for layer i.
In this example, there are two conflicting design objectives: high DNN
classification accuracy f1 (x) and a
small total memory footprint f2 (x) .
As we reduce the bitwidths, the memory footprint decreases, but the classification accuracy also drops, which
is highly undesirable. Our goal is to

W1→2

W2→3 W3→4
Output Layer

Grid Search

PU PU
PU PU

Input Layer

and evolutionary strategies. We provide several system design examples
for the optimization techniques but
emphasize that these methods are
generic and applicable to a wide variety of problems.

3b

1b

2b

FIGURE 2: An example of a DNN quantization task where the goal is to specify the
per-layer bitwidths.

IEEE SOLID-STATE CIRCUITS MAGAZINE

FA L L 2 0 19

25
IEEE Solid-States Circuits Magazine - Fall 2019

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2019
