Test-based generate-and-validate automated program repair (APR) systems generate many patches that pass the test suite without fixing the bug. The generated patches must be manually inspected by the developers, a task that tends to be time-consuming, thereby diminishing the role of APR in reducing debugging costs. We present the design and implementation of a novel tool, named Shibboleth, for automatic assessment of the patches generated by test-based generate-and-validate APR systems. Shibboleth leverages lightweight static and dynamic heuristics from both test and production code to rank and classify the patches. Shibboleth is based on the idea that the buggy program is almost correct and the bugs are small mistakes that require small changes to fix and specifically the fix does not remove the code implementing correct functionality of the program. Thus, the tool measures the impact of patches on both production code (via syntactic and semantic similarity) and test code (via code coverage) to separate the patches that result in similar programs and that do not remove desired program elements. We have evaluated Shibboleth on 1,871 patches, generated by 29 Java-based APR systems for Defects4J programs. The technique outperforms state-of-the-art raking and classification techniques. Specifically, in our ranking data set, in 66% of the cases, Shibboleth ranks the correct patch in top-1 or top-2 positions and, in our classification data set, it achieves an accuracy and F1-score of 0.887 and 0.852, respectively, in classification mode. A demo video of the tool is available at https://bit.ly/3NvYJN8.

Ali Ghanbari

Iowa State University

United States

Andrian Marcus

University of Texas at Dallas

United States

Software systems powering OS kernels, basebands, bootloaders, firmware, IoT or automotive build the foundation of infrastructure that billions of people rely on every day. Testing these systems is crucial, especially as their complexity grows and they are often written in unsafe languages such as C/C++.

However, testing such complex systems poses significant challenges, e.g., custom hardware for which there is no emulator, or a non-trivial setup of testing and debugging on the target device. As a result, the commonly used testing techniques and tools are not always easily applicable.

An off-target (OT) testing is a promising technique which addresses these challenges: part of the code is extracted and adapted to run on a different hardware platform with better tool support, easier debugging and higher test throughput. Unfortunately, since the process of creating an OT program has been manual, the technique did not scale well and was mostly used in an ad hoc manner.

In this paper we present a novel complex systems testing approach called Auto Off-target (AoT). Based on the information extracted from the source code and from the build process, AoT can automatically generate OT programs in C. AoT goes beyond the code generation and provides mechanisms that help to recreate and discover the program state in the OT code. The generated OTs are self-contained and independent of the original build environment. As a result, pieces of complex or embedded software can be easily run, analyzed, debugged and tested on a standard x86_64 machine.

We evaluate AoT on tens of thousands of functions selected from OS kernels, a bootloader and a network stack. We demonstrate that majority of the generated OTs can be automatically tested with fuzzing and symbolic execution. We further used AoT in a bug finding campaign and discovered seven bugs in the Android redfin and oriole kernels powering Google Pixel 5 and 6 phones.

Tomasz Kuchta

Samsung Electronics

Poland

Bartosz Zator

Samsung Electronics

The rapid expansion of robotics relies on properly configuring and testing hardware and software. Due to the expense and hazard of real-world testing on hardware, robot system testing increasingly utilizes extensive simulation. Creating robot simulation tests requires specialized skills in robot programming and simulation tools. While there are many platforms and tool-kits to create these simulations, they can be cumbersome when combined with automated testing. We present Maktub: a tool for creating tests using Unity and ROS. Maktub leverages the extensive 3D manipulation capabilities of Unity to lower the barrier in creating system tests for robots. A key idea of Maktub is to make tests without needing robotic software development skills. A video demonstration of Maktub can be found here: https://youtu.be/c0Bacy3DlEE, and the source code can be found here https://github.com/RobotCodeLab/Maktub.

Amr Moussa

North Carolina State University

John-Paul Ore

North Carolina State University

United States

Mutation testing research has indicated that a major part of its application cost is due to the large number of low utility mutants that it introduces. Although previous research has identified this issue, no previous study has proposed any effective solution to the problem. Thus, it remains unclear how to mutate and test a given piece of code in a best effort way, i.e., achieving a good trade-off between invested effort and test effectiveness. To achieve this, we propose Cerebro, a machine learning approach that statically selects subsuming mutants, i.e., the set of mutants that resides on the top of the subsumption hierarchy, based on the mutants’ surrounding code context. We evaluate Cerebro using 48 and 10 programs written in C and Java, respectively, and demonstrate that it preserves the mutation testing benefits while limiting application cost, i.e., reduces all cost application factors such as equivalent mutants, mutant executions, and the mutants requiring analysis. We demonstrate that Cerebro has strong inter-project prediction ability, which is significantly higher than two baseline methods, i.e., supervised learning on features proposed by state-of-the-art, and random mutant selection. More importantly, our results show that Cerebro’s selected mutants lead to strong tests that are respectively capable of killing 2 times higher than the number of subsuming mutants killed by the baselines when selecting the same number of mutants. At the same time, Cerebro reduces the cost-related factors, as it selects, on average, 68% fewer equivalent mutants, while requiring 90% fewer test executions than the baselines.

Aayush Garg

University of Luxembourg

Luxembourg

Milos Ojdanic

University of Luxembourg

Renzo Degiovanni

SnT, University of Luxembourg

Luxembourg

Thierry Titcheu Chekam

SES S.A. & University of Luxembourg (SnT)

Luxembourg

Mike Papadakis

University of Luxembourg, Luxembourg

Luxembourg

Yves Le Traon

University of Luxembourg, Luxembourg

Luxembourg

Question answering (QA) software uses information retrieval and natural language processing techniques to automatically answer questions posed by humans in a natural language. Like other AI- based software, QA software may contain bugs. To automatically test QA software without human labeling, previous work extracts facts from question answer pairs and generates new questions to detect QA software bugs. Nevertheless, the generated questions can be ambiguous, confusing, or with chaotic syntax, which are unanswerable for QA software. As a result, a large proportion of the reported bugs are false positives. In this work, we proposed QATest, a sentence-level mutation based metamorphic testing tool for QA software. To eliminate false positives and achieve precise automatic testing, QATest leverages five Metamorphic Relations (MRs) as well as semantics-guided searching and enhanced test oracles. Our evaluation on three QA datasets demonstrates that QATest outperforms the state-of-the-art in both quantity (8,133 vs. 6,601 bugs) and quality (97.67% vs. 49% true positive rate) of the reported bugs. Moreover, the test inputs generated by QATest successfully reduce MR violation rate from 44.29% to 20.51% when being adopted in fine-tuning the QA software under test.

Qingchao Shen

Tianjin University

China

Junjie Chen

Tianjin University

China

Jie M. Zhang

King's College London

United Kingdom

Haoyu Wang

College of Intelligence and Computing, Tianjin University

Shuang Liu

Tianjin University

China

Menghan Tian

College of Intelligence and Computing, Tianjin University

Fault Localization (FL) is an important first step in software debugging and is mostly manual in the current practice. Many methods have been proposed over the years to automate the FL process, including information retrieval (IR)-based techniques. These methods localize the fault based on the similarity of the reported bug report and the source code. Newer variations of IR-based FL (IRFL) techniques also look into the history of bug reports and leverage them during the localization. However, all existing IRFL techniques limit themselves to the current project’s data (local data). In this study, we introduce, which is an IRFL framework consisting of methods that use models pre-trained on the global data (extracted from open-source benchmark projects). In, we investigate two heuristics: (a) the effect of global data on a state-of-the-art IR-FL technique, namely, and (b) the application of a Word Embedding technique (Doc2Vec) together with global data. Our large-scale experiment on 51 software projects shows that using global data improves on average 6.6% and 4.8% in terms of MRR (Mean Reciprocal Rank) and MAP (Mean Average Precision), with over 14% in a majority (64% and 54% in terms of MRR and MAP, respectively) of the cases. This amount of improvement is significant compared to the improvement rates that five other state-of-the-art IRFL tools provide over. In addition, training the models globally is a one-time offline task with no overhead on ’s run-time fault localization. Our study, however, shows that a Word Embedding-based global solution did not further improve the results.

Nima Miryeganeh

University of Calgary

Sepehr Hashtroudi

University of Calgary

Hadi Hemmati

University of Calgary

Canada

Unit testing is a critical part of the software development process, ensuring the correctness of basic programming units in a program (e.g., a method). Search-based software testing (SBST) is an automated approach to generating test cases. SBST generates test cases with the genetic algorithms by specifying the coverage criterion (e.g., branch coverage). However, a good test suite must have different properties, which cannot be captured by using an individual coverage criterion alone. Therefore, the state-of-the-art approach combines multiple criteria to generate test cases. As combining multiple coverage criteria brings multiple objectives for optimization, it hurts the test suites’ coverage for certain criteria compared with using the single criterion. To cope with this, we propose a novel approach named \textbf{smart selection}. Based on the coverage correlations among criteria and the coverage goals’ subsumption relationships, smart selection selects a subset of coverage goals to reduce the number of optimization objectives and avoid missing any properties of all criteria. We conduct experiments to evaluate smart selection on $400$ Java classes with three state-of-the-art genetic algorithms. On average, smart selection outperforms the original combination (i.e., combining all goals) on $77$ Java classes, accounting for $65.1%$ of the classes having significant differences between the two approaches.

Zhichao Zhou

School of Information Science and Technology, ShanghaiTech University

China

Yuming Zhou

Nanjing University

Chunrong Fang

Nanjing University

China

Zhenyu Chen

Nanjing University

China

Yutian Tang

ShanghaiTech University

China