Mutation 2020

Mutation analysis is a strong, well studied, faultbased testing criterion that has been shown to lead to strong test suites. Still, when using mutation analysis to compare test suites, previous work has shown that its results can be skewed because of redundant mutants. This problem is avoided by using minimal mutants, a set of representative mutants that is computed from a full matrix of all test results against all mutants. Pitest is a state of the art mutation analysis tool for Java and the JVM that can produce the matrix needed to compute minimal mutants. When a mutant leads to a JVM crash, because of a timeout or a memory error for example, Pitest does not record the full information about a mutant, leading to an incomplete matrix. In this work, we update Pitest in order to obtain the complete matrix, and make our updated version available. We then study the impact of the incomplete matrix on both minimal mutant computation and test selection, using both open source projects and developer-made test suites as well as a set of well tested methods. Finally, using the data collected, we see whether particular characteristics of a mutant or a test can help predict whether a test run against a mutant will lead to a JVM crash. Our experiments show that the missing information in Pitest’s matrix has little impact on the computed set of minimal mutants and the tests it selects. We also find no indication that particular mutation operator or test systematically leads to JVM crashes.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00050

Thomas Laurent

Lero & University College Dublin

Ireland

Fionnuala Wall

Lero and University College Dublin

Ireland

Anthony Ventresque

Lero and University College Dublin

Ireland

Diversity has been proposed as a key criterion to improve testing effectiveness and efficiency. It can be used to optimise large test repositories but also to visualise test maintenance issues and raise practitioners’ awareness about waste in test artefacts and processes. Even though these diversitybased testing techniques aim to exercise diverse behavior in the system under test (SUT), the diversity has mainly been measured on and between artefacts (e.g., inputs, outputs or test scripts). Here, we introduce a family of measures to capture behavioural diversity (b-div) of test cases by comparing their executions and failure outcomes. Using failure information to capture the SUT behaviour has been shown to improve effectiveness of history-based test prioritisation approaches. However, historybased techniques require reliable test execution logs which are often not available or can be difficult to obtain due to flaky tests, scarcity of test executions, etc. To be generally applicable we instead propose to use mutation testing to measure behavioral diversity by running the set of test cases on various mutated versions of the SUT. Concretely, we propose two specific b-div measures (based on accuracy and Matthew’s correlation coefficient, respectively) and compare them with artefact-based diversity (a-div) for prioritising the test suites of 6 different open-source projects. Our results show that our b-div measures outperform a-div and random selection in all of the studied projects. The improvement is substantial with an average increase in average percentage of faults detected (APFD) of between 19% to 31% depending on the size of the subset of prioritised tests.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00051

Francisco Gomes

Chalmers | University of Gothenburg

Sweden

Felix Dobslaw

Chalmers University of Technology

Sweden

Robert Feldt

Chalmers University of Technology

Sweden

Scientists have created many cost reduction techniques for mutation testing, and most of them reduce cost with minor losses of effectiveness. However, many of these techniques are difficult to generalize, difficult to scale, or both. Published results are usually limited to a modest collection of programs. Therefore, an open question is whether the results of a given cost reduction technique on programs studied in the paper will hold true for other programs. This paper introduces a conceptual framework, named SiMut, to support the cost reduction of mutation testing based on historical data and program similarity. Given a new, untested program u, the central idea is applying to u the same cost reduction strategy applied to a group G of programs that are similar to u and have already been tested with mutation, and check for consistency of results in terms of reduced costs and quality of test sets. SiMut includes activities to compute program abstractions and similarity. Based on this information, it supports the application of mutation cost reduction techniques to both G and u. This paper presents the concepts behind SiMut, a proof-of-concept implementation of SiMut, and results from a pilot study. Finally, we discuss some issues related to the use of SiMut, focusing on the composition of a representative dataset to properly explore the potential of our framework.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00052

Alessandro V. Pizzoleto

Federal University of Sao Carlos

Brazil

Fabiano Ferrari

Federal University of São Carlos

Brazil

Lucas D. Dallilo

University of Sao Paulo

Brazil

Jeff Offutt

George Mason University

United States

Mutation Testing (MT) is a state-of-the-art technique for assessing test suite effectiveness. The MT principle is to inject variants, known as mutants, into the System Under Test (SUT). Then, the behaviour of the original SUT is compared to that of the mutated SUT when running the same test suite. If no difference in behaviour is observed, the mutant is said to have survived; otherwise, it is said to have been killed. Despite its strengths, the applicability of MT in practice has been limited by its high computational cost. To mitigate this problem, Predictive Mutation Testing (PMT) has been proposed. PMT uses a classification model based on features related to the mutated code and the test suite to predict the execution results of a mutant without actually executing it. In other words, PMT predicts whether a mutant will be killed or will survive. In previous studies, PMT has been evaluated on several projects in two application scenarios, involving cross-project and crossversion learning. The goal of our research is to investigate how well the proposed PMT method, which has been evaluated on Java, can be extended to other programming languages. For that purpose, we first replicated the previous study and then extended the PMT approach to a single C program. We used random forrest classifiers as our supervised learning approach of choice. Our results indicate that PMT is able to predict the execution results of mutants with high accuracy. On the Java projects, we achieved Area Under Curve (AUC) values above 0.90 with a Prediction Error (PE) below 10%. On the C project, we achieved an AUC value above 0.90 with a PE below 1%. In our analyses we also investigated how sensitive the performance of PMT is to the set of selected features. In particular, we wanted to understand whether adding programming language specific features to a language independent core set of features significantly improve the performance of PMT. Our results are an indicator that, overall, PMT has potential to be applied across programming languages and is robust when dealing with imbalanced data.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00053

Alejandra Duque Torres

Institute of Computer Science, University of Tartu

Estonia

Natia Doliashvili

Institute of Computer Science, University of Tartu

Estonia

Dietmar Pfahl

University of Tartu

Estonia

Rudolf Ramler

Software Competence Center Hagenberg

Context: Testing adaptive systems (ASs) and context-aware systems (CASs) is usually more challenging than testing conventional systems. This is due to their characteristics, such as the high number of possible configurations and runtime adaptations. Therefore, it is important to know the types of faults that may be incurred by these systems so that such faults can be either avoided during implementation, or detected through tests (e.g. in a fault-based testing approach such as mutation). Objective: Characterising and grouping types of faults for ASs and CASs that have been previously described in the literature, but are disperse in various studies. Moreover, identifying which and how the fault types are addressed by the current research on fault-based testing. Method: we used a Systematic Literature Review that addressed testing for ASs and CASs, whose one of the goals was a characterisation of fault types for ASs and CASs. By dealing with such a goal, we analysed 11 primary studies that address fault types. We also used code snippets to illustrate how the fault types may occur. Finally, we analysed the faults that are addressed in other 7 studies that investigated fault-based testing for ASs and CASs. Results: We present lists of specific fault types (26 in total) and fault type categories (6 in total) for AS and CASs, and discuss the relationship between these fault types with current fault-based testing approaches. Conclusion: Our results are original when compared to the state-of-the-art, given that we proposed the first characterisation of fault types for ASs and CASs. Beyond this, we point out room for improvement of fault-based testing of these types of systems, particularly regarding the types of faults addressed by the current research.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00054

Bento Rafael Siqueira

Federal University of São Carlos

Fabiano Ferrari

Federal University of São Carlos

Brazil

Kathiani E. Souza

Federal University of São Carlos

Brazil

Daniel S. M. Santibáñez

Federal University of São Carlos

Brazil

Valter Vieira Camargo

Federal University of Sergipe, São Cristovão

Brazil

Mutation testing can be used to measure the quality of a given test suite. But two flaws prevent wide acceptance. First, there are equivalent mutants - mutants that are semantically equivalent to the unmodified version and therefore unkillable. Manually identifying those mutants is time-consuming and error-prone. Second, initially there are often too few test cases. Mutation testing detects missing cases. But it is too time-consuming to manually write all the required test cases. This paper shows how to use symbolic execution to tackle both problems, i.e., to detect equivalent mutants and exclude them from further analysis, and to automatically generate test cases that kill the remaining mutants. Our evaluation uses a set of 252 publicly available mutants for which it is known that they are hard to classify. Despite the fact that detecting equivalent mutants is an undecidable problem in general, our fully automatic tool MutantDistiller correctly classifies all of them (13 equivalent, 239 non-equivalent). MutantDistiller also generates test cases that kill the nonequivalent mutants.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00055

Michael Baer

Norbert Oster

Michael Philippsen

Friedrich-Alexander University Erlangen-Nürnberg (FAU)

Germany

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00056

Claudinei Brito Junior

Universidade de São Paulo

Brazil

Vinicius Durelli

Universidade Federal de São João del-Rei

Brazil

Rafael S. Durelli

Federal University of Lavras Lavras

Brazil

Simone do Rocio Senger de Souza

University of São Paulo - USP

Brazil

Auri Vincenzi

Federal University of São Carlos

Brazil

Marcio Eduardo Delamaro

Universidade de São Paulo

Brazil