Computational Intelligence - May 2017 - 29

TAble 1 Description of ten large-scale real projects.
PrOjecT

Full nAMe WITh versIOn

PrOjecT descrIPTIOn

CAmel

ApAChe CAmel Core 2.15

A verSATIle FrAmeWork bASed on enTerprISe InTegrATIon pATTernS

ASSerTJ

ASSerTJ Core †

An ASSerTIon enhAnCed FrAmeWork For JAvA TeSTIng

ConFIgurATIon

ApAChe CommonS ConFIgurATIon 1.10

A generIC ConFIgurATIon InTerFACe oF reAdIng ConFIgurATIon dATA

JgIT

JgIT †

A gIT InTerFACe From JAvA progrAmS

CloSure

google CloSure CompIler †

A FAST CompIler For JAvASCrIpT

ColleCTIonS

ApAChe CommonS ColleCTIonS 4.0

An enhAnCed lIbrAry For JAvA ColleCTIonS

JFreeChArT

JFreeChArT 1.0

A lIbrAry For proFeSSIonAl quAlITy ChArTS

lAng

ApAChe CommonS lAng 3.4

A lAnguAge-SupporT enhAnCed lIbrAry For JAvA

JodATIme

JodA-TIme 2.8.2

An enhAnCed lIbrAry For JAvA TIme And dATe

mATh

ApAChe CommonS mATh 3.5

A mATh And AlgorIThm lIbrAry For JAvA

†

For this project, the source code is not provided with a specific version. hence, we extract the master branch (on January 31th, 2016) in the version control system
for experiments.

A. Test Suite Reduction on Real-World Projects

problem. RAPS is a meta-heuristic via randomized priority
search; RWLS is an efficient search algorithm via weighting
rows in the set covering model. GA [31] is a widely-used population-based algorithm in evolutionary computation. In addition, to show the effectiveness of the multi-level strategy in
MultiWalk, we also show the results by WalkTest with multirestart (WalkTest for short).
Experimental platform. All the experiments run on a
PC with Intel Core 3.6 GHz CPU, 4GB memory, and Ubuntu
12.04. The code instrumentation for the runtime coverage collection [32] is implemented with a Java analysis framework,
Spoon 4.0 [33]. Among the algorithms under evaluation, we
call ILP with an off-the-shelf linear programming tool, lp_
solve 5.5 [34]. We implement all the other algorithms in Java
JDK 1.7.

Our proposed algorithm, MultiWalk, is compared with seven
algorithms in this section. We setup algorithms in use as follows. The timeout of each algorithm is set to two hours
(7,200 seconds). All the heuristics are executed for 30 times to
obtain the average values; the random seed is the timestamp
of the system clock. In WalkTest, the probability p is set to
0.5; 10000 tries are used to obtain local optima. In WalkTest
with multi-restart, WalkTest is restarted for 100 times to
achieve the best solution. In MultiWalk, WalkTest without
restart is used as the embedded local search operator s; the
maximum number of levels is set to a = 20 and the number
of local optima in each level is set to b = 10. Further experiments in Section V-B will show the sensitivity to a and
b in MultiWalk.
Table 3 shows both the test suite size after reduction and
the time cost of running algorithms. In these experimental
results, MultiWalk can obtain the best-known solutions on
nine out of ten projects. One exception is the project Camel,
which is the smallest project under consideration. ILP, as well as
HGS, RAPS, and RWLS, can obtain the optimal solution on

V. Experimental Results

We evaluate our algorithm on ten large real-world Java open
source projects; meanwhile, an exact algorithm, ILP, and six
heuristics are employed in comparing the size of tests after test
suite reduction and the time cost.

TAble 2 Tests and requirements of ten large-scale real projects.
# sATIsFIed requIreMenTs Per TesT
PrOjecT

# TesTs

# requIreMenTs

MIn

MedIAn

MAx

AverAge

sTdev.

CAmel

908

21551

257

7494

1656.9

2206.2

ASSerTJ

1219

3632

722

82.1

66.0

ConFIgurATIon

1333

9062

237

2073

341.7

310.3

JgIT

1448

13678

4203

158.8

342.2

CloSure

1626

14936

1529

2586

1316.6

738.8

ColleCTIonS

4882

12512

591

92.9

87.3

JFreeChArT

2247

29846

108

3945

293.6

557.6

lAng

2767

11857

377

42.1

56.0

JodATIme

4118

10567

150

1845

209.7

179.9

mATh

6196

44947

139

1714

236.3

248.7

may 2017 | IEEE ComputatIonal IntEllIgEnCE magazInE

Table of Contents for the Digital Edition of Computational Intelligence - May 2017