IEEE Solid-State Circuits Magazine - Spring 2014 - 57

impossible that any interval derived
with this method ever exceeds the
range of measured values, it is
impossible that we imply information
about ranges of x for which we have
no evidence, and the interval limits
will also automatically have the correct number of significant digits.
Note that this method estimates
the median M = P 50.0 rather than the
mean value n. For a symmetric distribution, the median and the mean
are the same, but, in general, they
differ. Calculating the median minimizes the mean absolute distance to
all samples, but the mean value minimizes the mean squared distance,
so the median may anyway be the
more informative measure because
measurement outliers have much
more influence on the mean than on
the median.

A Suggestion for Statistical
Benchmarking
Our suggestion for dealing with
small data sets in future papers is to
agree on a common confidence level
c for comparisons and then, for
comparability with the old ! 1, 2, 3v
thinking, publish ranges for the
median M and for P 15.87 and P 84.13,
P 2.275 and P 97.725, and P 0.135 and
P 99.865 .
A brief survey of recent ESSCIRC
papers shows that having only N = 8
samples is quite common for academic and industry papers, so we set
the confidence level c such that P 15.87
and P 84.13 are determined by the data
extrema in the case N = 8. This means
we calculate c = P " x 1 # P 15.87 , =
1 - 0.251 = 0.749. Therefore, allowing papers with N = 8 samples to
participate in numerical benchmarks
already decides that our common
confidence level shall be 75%; authors
measurements can
N =8
with
then state that their measured data
extrema are the limits of a 75% confidence interval on P 15.87 and P 84.13 .
We can now ask the following
question: what is the minimum N
such that we can make statements
about P 2.28 and P 0.135, which correspond to n - 2v and n - 3v in

normal distributions? Solving (9)
using m = 1 for N, which simply is

This is where we can come back to
the numerical examples right at the
beginning: with N = 8 and c = 0.75,
the P 15.87 and P 84.13 are in the range
x 1 fx 8, and the median is in the range
x 2 fx 7 . For N = 24 and c = 0.75,
the P 15.87 and P 84.13 are in the range
x 3 fx 22, and the median is in the
range x 9 fx 16 . So an interval plot
should be shown as in F
- igure 4(a). To
compare, the same plot is replicated
in Figure 4(b) together with the conventional `n ! v_ limits. The range is
narrower for N = 24 than for N = 8,
so having more results lets us give
a smaller range with the same confidence. Figure 4(b) also shows that
the naively obtained bounds given
by m ! s are so tight that they do
not even contain the 75% confidence
intervals of the respective percentiles.
In our opinion, if research groups
would publish their figures as we
propose here, then a better figure
(e.g., narrower temperature error
range) obtained would be good
enough an indicator to discuss
whether a new circuit merits being
published. However, 75% confidence
may not be enough to make decisions on future research.
To base design decisions on statistical evaluation, it would be better to use 95% confidence limits all
the time and look at the range of the
P 15.87 and P 84.13 of, e.g., temperature
errors. If the new range is narrower,
there is a good chance that the design
change has brought an improvement,
given that it can be expected to have

P {x 1 # Pp} = 1 - (1 - p) N
$ 0.75
log (1 - 0.75)
,
&N$
log (1 - p)
gives minimum sample sizes of
N = 61 and N = 1, 027, respectively.
This means that, for normal MPW
sample sizes (the most we ever got
back was 50), it is never possible
to talk about P 2.28 and P 0.135 with
even a confidence level as low as
75%. With respect to the infamous
6v, observe that there the value
is N . 1.4·10 9, which more or less
tells us: "forget about 6v."
Having
more
measurements
does, however, give an advantage.
Evaluating (9) shows that, at a confidence level of 75%, P 15.87 and P 84.13
are in the range x 1 fx N for N = 8
upward. The confidence increases
for higher N, and at N = 17 we
get to the situation that P 15.87 and
P 84.13 are in the range x 2 fx N -1 with
75% confidence, i.e., we can omit
the two data extrema. From N = 24
upward, it is x 3 fx N -2, so we can
omit the two lowest and highest
values, and so on. This means that
having more measurements makes
it possible to ignore more outliers
in the data set. The same thinking
can be applied to the median and
to 95% confidence, as shown in
Table 1 and as performed by the
companion Web application [3].

P15.87%

P84.13%
x, y

P15.87%

P84.13%

M
(a)

mx − sx

mx + sx
+20
x, y

−20

−10
m y − sy

my
(b)

my + sy

Figure 4: (a) Data sets with median and percentiles estimated according to our method. (b) The
same without M and P labels but drawn together with conventional m ! s limits.

IEEE SOLID-STATE CIRCUITS MAGAZINE

s p r i n g 2 0 14

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Spring 2014