IEEE Spectrum - North American - March 2016 - 35

can then be restarted from the last valid
checkpoint instead of beginning some
immense calculation anew.
This approach won't work indefinitely,
though, because as computers get bigger,
the time needed to create a checkpoint
increases. Eventually, this interval will
become longer than the typical period
before the next fault. A challenge for
exascale computing is what to do about
this grim reality.
Several groups are trying to improve
the speed of writing checkpoints. To the
extent they are successful, these efforts
will forestall the need to do something
totally different. But ultimately, applications will have to be rewritten to withstand a constant barrage of faults and
keep on running.
Unfortunately, today's programming
models and languages don't offer any
mechanism for such dynamic recovery from faults. In June 2012, members
of an international forum composed of
vendors, academics, and researchers
from the United States, Europe, and Asia
met and discussed adding resilience to
message-passing interface, or MPI, the
programming model used in nearly all
supercomputing code. Those present at
that meeting voted that the next version
of MPI would have no resilience capabilities added to it. So for the foreseeable future, programming models will
continue to offer no methods for notification or recovery from faults.

One reason is that there is no standard
that describes the types of faults that
the software will be notified about and
the mechanism for that notification. A
standard fault model would also define
the actions and services available to the
software to assist in recovery. Without
even a de facto fault model to go by, it
was not possible for these forum members to decide how to augment MPI for
greater resilience.
So the first order of business is for the
supercomputer community to agree on
a standard fault model. That's more difficult than it sounds because some faults
might be easy for one manufacturer to
deal with and hard for another. So there
are bound to be fierce squabbles. More
important, nobody really knows what
problems the fault model should address.
What are all the possible errors that affect
today's supercomputers? Which are most
common? Which errors are most concerning? No one yet has the answers.
And while I've talked a lot about faults
causing machines to crash, these are
not, in fact, the most dangerous. More
menacing are the errors that allow the
application to run to the end and give
an answer that looks correct but is actually wrong. You wouldn't want to fly in
an airliner designed using such a calculation. Nor would you want to certify a
new nuclear reactor based on one. These
undetected errors-their types, rates,
and impact-are the scariest aspect of
supercomputing's monster in the closet.
Given all the Gloom and doom
I've shared, you might wonder: How
can an exascale supercomputer ever be
expected to work? The answer may lie
in a handful of recent studies for which
researchers purposely injected different types of errors inside a computer
at random times and locations while it
was running an application. Remarkably enough, 90 percent of those errors
proved to be harmless.
One reason for that happy outcome
is that a significant fraction of the computer's main memory is usually unused.
And even if the memory is being used,
the next action on a memory cell after
the bit it holds is erroneously flipped

may be to write a value to that cell. If
so, the earlier bit flip will be harmless.
If instead the next action is to read
that memory cell, an incorrect value
f lows into the computation. But the
researchers found that even when a
bad value got into a computation, the
final result of a large simulation was
often the same.
Errors don't, however, limit themselves to data values: They can affect the
machine instructions held in memory,
too. The area of memory occupied by
machine instructions is much smaller
than the area taken up by the data, so the
probability of a cosmic ray corrupting an
instruction is smaller. But it can be much
more catastrophic. If a bit is flipped in a
machine instruction that is then executed,
the program will most likely crash. On
the other hand, if the error hits in a part
of the code that has already executed,
or in a path of the code that doesn't get
executed, the error is harmless.
There are also errors that can occur in
silicon logic. As a simple example, imagine that two numbers are being multiplied, but because of a transient error in
the multiplication circuitry, the result is
incorrect. How far off it will be can vary
greatly depending on the location and
timing of the error.
As with memory, flips that occur in silicon logic that is not being used are harmless. And even if this silicon is being used,
any flips that occur outside the narrow
time window when the calculation is
taking place are also harmless. What's
more, a bad multiplication is much like
a bad memory value going into the computation: Many times these have little
or no affect on the final result.
So many of the faults that arise in
future supercomputers will no doubt be
innocuous. But the ones that do matter
are nevertheless increasing at an alarming rate. So the supercomputing community must somehow address the serious
hardware and software challenges they
pose. What to do is not yet clear, but it's
clear we must do something to prevent
this monster from eating us alive. n
poSt Your commeNtS at http://spectrum.
ieee.org/resilience0316

SPectRUM.ieee.oRG

noRth AMeRicAn

MAR 2016

http://spectrum http://www.ieee.org/resilience0316 http://SPectRUM.ieee.oRG

Table of Contents for the Digital Edition of IEEE Spectrum - North American - March 2016