In this work, we introduce an approach, called CIT-daily, which integrates combinatorial interaction testing (CIT) with the daily build processes to systematically test the interactions between the factors/parameters affecting the system’s behaviors, on a daily basis. We also develop a number of CIT-daily strategies and empirically evaluate them on highly-configurable systems. The first strategy tests the same t-way covering array every day throughout the process, achieving a t-way coverage on a daily basis by covering each possible combination of option settings for every combination of t options. The other strategies, on the other hand, while guaranteeing a t-way coverage on a daily basis, aim to cover higher order interactions between the configuration options over time by varying the t-way covering arrays tested. In the experiments, we observed that the proposed approach significantly improved the effectiveness (i.e., fault revealing abilities) of the daily build processes; randomizing the coverage of higher order interactions between the configuration options while guaranteeing a base $t$-way coverage every day, further improved the effectiveness; and the more the higher order interactions covered during the process, the higher the fault revealing abilities tended to be.

Hanefi Mercan

Sabanci University

Turkey

Atakan Aytar

Sabanci University

Giray Coskun

Sabanci University

Turkey

Dilara Müstecep

Sabanci University

Gülsüm Uzer

Sabanci University

Cemal Yilmaz

Sabancı University