Instrumentation & Measurement Magazine 24-2 - 87

Fig. 4. Bias-variance trade-off in machine learning.

Fig. 3. Some variable-output Machine Learning models give different outputs
for the same input, as shown in the figure, by slightly varying the Machine
Learning model at each repeated estimation.

2) Systematic error and random error in I&M are independent, but the same is not true about ML's bias and variance,
which in fact have a trade-off relationship. We can see this in
Fig. 4, which shows that as model complexity increases, variance increases but bias decreases. In fact, optimizing bias
versus variance is one of the biggest challenges in designing an
ML model, and is known as the bias-variance dilemma.
Fig. 4 also shows that estimation error is a combination of
bias, variance, and noise. In fact, if we apply the conventional
ML approach of using sum of squared errors to represent the
cost function, it can be shown [12, eq. 9] that ML's prediction
error, a.k.a. out-of-sample error, can be expressed as:
Error 
Bias2  Variance  Noise (1)

where bias and variance can be expressed as [12, eq. 7-8]:
Bias 

 f (x )  E
S

i1



 fˆ ( x ) (2)
 Dj i 

S

Variance  ED  fˆDj  xi   ED  fˆDj  xi 


i1






  (3)
2

and Noise is the statistical variance of the data in the dataset; i.e., it has zero mean and some variance σ 2. The function
fˆD j is the estimator trained with dataset Dj, and ED is the expected value over all those datasets, which as you recall have
the same size S.
3) Measurement error in I&M is minimum when both systematic and random errors are minimized. However, as we
saw in Fig. 4, error in ML is minimum at a point where the combination of bias, variance, and noise is minimized, not each one
separately. Fig. 4 shows that minimum variance is achieved
by low model complexity, while minimum bias is achieved by
high model complexity; in other words, no model complexity
can achieve minimum bias and minimum variance at the same
time! For example, if we start with a low model complexity,
which from Fig. 4 means that bias is high and variance is low,
we will get a result similar to Fig. 2(ii), part A. As model complexity increases, the result will approach Fig. 2(ii), part B, and
as it increases even further, the result will approach Fig. 2(ii),
part D. The optimum point shown in Fig. 4 will likely occur
somewhere between Fig. 2(ii), part B and part D. This means
that we might actually never be able to reach a result as good as
Fig. 2(ii), part C! For example, ML methods like Decision Trees,
k-Nearest Neighbors, and SVM are known to have low bias
but high variance, whereas methods like Linear Regression

Table 1 - Differences between Fig. 2(ii) and Fig. 3, both of which are ML
Feature

Fig. 2(ii)

Fig. 3

execution time

training time

run time (a.k.a prediction time or estimation time)

input

multiple datasets, each of the same size

same single input vector

ML model

fixed (although the parameters change for each
dataset)

varies

a single black dot

represents the estimation outcome averaged
over all data in one dataset; there are multiple
datasets, so we get multiple black dots.

represents the output of one ML model for the
same input vector; there are multiple ML
models (since the model varies), so we get
multiple black dots.

April 2021

IEEE Instrumentation & Measurement Magazine 87

Instrumentation & Measurement Magazine 24-2

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-2