Computational Intelligence - August 2014 - 45

and comparing results. We describe the experimental procedure with the TSP Suite stepby-step in the following section.
A. Implementing the Algorithm

The TSP Suite is a Java 1.7 framework that assists
algorithm developers in implementing and testing
their methods, running experiments and collecting
data, as well as evaluating and comparing results.

Since the TSP Suite is a Java framework, the
optimization algorithm to be investigated must
be implemented as a Java class (program). This class must be an
extended class of a class called TSPAlgorithm and implement
a method solve taking as input an instance of the class
ObjectiveFunction.This instance provides, among others,
1) a method to compute the tour length of a candidate solution (either in path or adjacency representation [27]);
2) a method to compute the distance between two nodes;
3) a function that returns true when the algorithm should
terminate, either because the granted computational budget
has elapsed or the global optimum has been discovered2;
4) the random number generator to be used during the run
of the algorithm; as well as
5) information about the elapsed runtime and best solution
discovered so far.
Additionally, similar to the objective functions used in the
COCO framework [6], it automatically gathers all logging
information (in memory).
To be executable, the algorithm class must have a specific
main method, which is a single line of code that can basically
be copied from the documentation of the TSP Suite. By optionally implementing some methods, an algorithm may be
extended with typed parameters (such as an integer value for
the population size of an EA) that can be passed in via the
command line or in a configuration file.
JUnit tests [28], which can automatically apply a TSP solver
to some of the benchmark instances and check if it produces
invalid results, are provided. In order to unit-test a new algorithm, one additional class with a single one-line method
needs to be provided. Although testing cannot guard against
errors entirely, it may help to reduce them. The TSP Suite
comes with extensive documentation on how to implement
and test TSP solvers.
There are very few requirements on the algorithm type,
structure, and implementation imposed by the TSP Suite. This
makes it easy to both benchmark existing algorithms and to reuse TSP Suite-based algorithms for other purposes.
B. Executing the Experiment

To execute the experiment, the algorithm will be instantiated
and applied to all benchmark instances 30 times each by
default. A folder structure with one folder per benchmark case
and one log file per run will be produced. These log files follow
an easy-to-parse text format and contain, among others,
1) the benchmarking data captured according to Sections
II-A2 and II-A3;
2
The default termination criterion is to exhaust 100n3FEs, 100n4DEs, or 1h of CPU
time, or when the optimum is found. This can be changed via the command line of
the system.

2) the algorithm name and class;
3) all parameter settings (if typed parameters have been
specified);
4) information about the software (Java version, OS) and
hardware (processor, available memory) environment;
5) the seed of the random number generator; and
6) if specified, the name, e-mail address, and website of the
experimenter.
A log file thus serves as a complete, publishable documentation
of a single run, maximizing replicability of experiments.
As mentioned before, there are 110 symmetric TSPLib
instances and the system conducts by default 30 runs per
instance. If each run would use up the maximum time of 1h
even on the smallest instances, the experiment could take
about 138 days to complete. In order to decrease the runtime,
all processors on a machine can be utilized for executing
independent runs. From our experience, a complete experiment with a sub-par algorithm takes about 1 week on an
8-core machine. Additionally, several instances of the same
program may be executed in a cluster. If their output paths
point to the same shared folder, the runs to be performed are
automatically divided among them. This way, the workload of
the experiment is parallelized and distributed, but not the
algorithms themselves. Thus, no additional provisions from
the algorithm implementer (apart from not using globally
shared variables) are required.
C. Evaluating and Documenting the Results

Once an experiment has been conducted using the TSP Suite,
the evaluator program can be applied to the output folder(s)
with the log files. This evaluator generates a comprehensive
report, which includes all the steps detailed in Section II-B.
The report is structured into three parts.
The first part contains a description of the TSP, the experiment, and the evaluation process, i.e., a more extensive version
of the text in this paper up to the start of Section III. This
makes the report a standalone document that can be understood with no further context needed.
The second part of the report is focused on the evaluation
of individual algorithms separately. If the evaluator is fed with
results from multiple experiments, this part contains one section for each of them. In such a section, the parameter settings
of the experiment are listed, several curves are plotted, and an
automatic comparison with results from the literature is performed and presented in tabular form.
If multiple experiments have been conducted, a third section is added to the report, which provides statistical algorithm comparisons. Here, curves of the algorithms can be

august 2014 | IEEE ComputatIonal IntEllIgEnCE magazInE

45



Table of Contents for the Digital Edition of Computational Intelligence - August 2014

Computational Intelligence - August 2014 - Cover1
Computational Intelligence - August 2014 - Cover2
Computational Intelligence - August 2014 - 1
Computational Intelligence - August 2014 - 2
Computational Intelligence - August 2014 - 3
Computational Intelligence - August 2014 - 4
Computational Intelligence - August 2014 - 5
Computational Intelligence - August 2014 - 6
Computational Intelligence - August 2014 - 7
Computational Intelligence - August 2014 - 8
Computational Intelligence - August 2014 - 9
Computational Intelligence - August 2014 - 10
Computational Intelligence - August 2014 - 11
Computational Intelligence - August 2014 - 12
Computational Intelligence - August 2014 - 13
Computational Intelligence - August 2014 - 14
Computational Intelligence - August 2014 - 15
Computational Intelligence - August 2014 - 16
Computational Intelligence - August 2014 - 17
Computational Intelligence - August 2014 - 18
Computational Intelligence - August 2014 - 19
Computational Intelligence - August 2014 - 20
Computational Intelligence - August 2014 - 21
Computational Intelligence - August 2014 - 22
Computational Intelligence - August 2014 - 23
Computational Intelligence - August 2014 - 24
Computational Intelligence - August 2014 - 25
Computational Intelligence - August 2014 - 26
Computational Intelligence - August 2014 - 27
Computational Intelligence - August 2014 - 28
Computational Intelligence - August 2014 - 29
Computational Intelligence - August 2014 - 30
Computational Intelligence - August 2014 - 31
Computational Intelligence - August 2014 - 32
Computational Intelligence - August 2014 - 33
Computational Intelligence - August 2014 - 34
Computational Intelligence - August 2014 - 35
Computational Intelligence - August 2014 - 36
Computational Intelligence - August 2014 - 37
Computational Intelligence - August 2014 - 38
Computational Intelligence - August 2014 - 39
Computational Intelligence - August 2014 - 40
Computational Intelligence - August 2014 - 41
Computational Intelligence - August 2014 - 42
Computational Intelligence - August 2014 - 43
Computational Intelligence - August 2014 - 44
Computational Intelligence - August 2014 - 45
Computational Intelligence - August 2014 - 46
Computational Intelligence - August 2014 - 47
Computational Intelligence - August 2014 - 48
Computational Intelligence - August 2014 - 49
Computational Intelligence - August 2014 - 50
Computational Intelligence - August 2014 - 51
Computational Intelligence - August 2014 - 52
Computational Intelligence - August 2014 - 53
Computational Intelligence - August 2014 - 54
Computational Intelligence - August 2014 - 55
Computational Intelligence - August 2014 - 56
Computational Intelligence - August 2014 - 57
Computational Intelligence - August 2014 - 58
Computational Intelligence - August 2014 - 59
Computational Intelligence - August 2014 - 60
Computational Intelligence - August 2014 - 61
Computational Intelligence - August 2014 - 62
Computational Intelligence - August 2014 - 63
Computational Intelligence - August 2014 - 64
Computational Intelligence - August 2014 - 65
Computational Intelligence - August 2014 - 66
Computational Intelligence - August 2014 - 67
Computational Intelligence - August 2014 - 68
Computational Intelligence - August 2014 - 69
Computational Intelligence - August 2014 - 70
Computational Intelligence - August 2014 - 71
Computational Intelligence - August 2014 - 72
Computational Intelligence - August 2014 - 73
Computational Intelligence - August 2014 - 74
Computational Intelligence - August 2014 - 75
Computational Intelligence - August 2014 - 76
Computational Intelligence - August 2014 - 77
Computational Intelligence - August 2014 - 78
Computational Intelligence - August 2014 - 79
Computational Intelligence - August 2014 - 80
Computational Intelligence - August 2014 - 81
Computational Intelligence - August 2014 - 82
Computational Intelligence - August 2014 - 83
Computational Intelligence - August 2014 - 84
Computational Intelligence - August 2014 - 85
Computational Intelligence - August 2014 - 86
Computational Intelligence - August 2014 - 87
Computational Intelligence - August 2014 - 88
Computational Intelligence - August 2014 - Cover3
Computational Intelligence - August 2014 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com