ACM SIGSOFT Distinguished Paper Award

Compilers, like other software systems, contain bugs, and compiler testing is the most widely-used way to assure compiler quality. A critical task of compiler testing is to generate test programs that could effectively and efficiently discover bugs. Though we can configure test generators such as Csmith to control the features of the generated programs, it is not clear what test configuration is effective. In particular, an effective test configuration needs to generate test programs that are bug-revealing, i.e., likely to trigger bugs, and diverse, i.e., able to discover different types of bugs. It is not easy to satisfy both properties. In this paper, we propose a novel test-program generation approach, called HiCOND, which utilizes historical data for configuration diversification to solve this challenge. HiCOND first infers the range for each option in a test configuration where bug-revealing test programs are more likely to be generated based on historical data. Then, it identifies a set of test configurations that can lead to diverse test programs through a search method (particle swarm optimization). Finally, based on the set of test configurations for compiler testing, HiCOND generates test programs, which are likely to be bug-revealing and diverse. We have conducted experiments on two popular compilers GCC and LLVM, and the results confirm the effectiveness of our approach. For example, HiCOND detects 75.00%, 133.33%, and 145.00% more bugs than the three existing approaches, respectively. Moreover, HiCOND has been successfully applied to actual compiler testing in a global IT company and detected 11 bugs during the practical evaluation.

Junjie Chen

Tianjin University

China

Guancheng Wang

Peking University

Dan Hao

Peking University

China

Yingfei Xiong

Peking University

China

Hongyu Zhang

The University of Newcastle

Australia

Lu Zhang

Peking University

China

We present TEGCER, an automated feedback tool for novice programmers. TEGCER uses supervised classification to match compilation errors in new code submissions with relevant pre-existing errors, submitted by other students before. The dense neural network used to perform this classification task is trained on 15,000+ error-repair code examples. The proposed model yields a test set classification Pred@3 accuracy of 97.7% across 212 error category labels. Using this model as its base, TEGCER presents students with the closest relevant examples of solutions for their specific error on demand. A large scale (N>230) usability study shows that students who use TEGCER are able to resolve errors more than 25% faster on average than students being assisted by human tutors.

Link to Publication: https://github.com/umairzahmed/tegcer

Umair Z. Ahmed

National University of Singapore

Singapore

Renuka Sindhgatta

Queensland University of Technology, Australia

Nisheeth Srivastava

Indian Institute of Technology, Kanpur

Amey Karkare

IIT Kanpur

India

Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, i.e., variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. Indeed, to apply mutation testing in practice a developer needs to (i) generate mutants, (ii) compile the source code, and (iii) run test cases. In cases of systems where test suites count hundrends of test cases, this is unfeasible. As a consequence, the problem of automatically assessing test-case effectiveness in a timely and efficient manner is still far from being solved. In this paper, we present a novel methodology orthogonal to existing approaches. In particular, we investigate the feasibility to estimate test-case effectiveness, as indicated by mutation score, exploiting production and test-code-quality indicators. We firstly select a set of 67 factors, including test and code metrics as well as test- and code-smells, and study their relation with test-case effectiveness. We discover that 41 out of 67 of the investigated factors statistically differs from non-effective and effective tests. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the study reveal that our estimation model only based only on statically computable features has 86% of both F-Measure and AUC-ROC. This means that we can estimate the test-case effectiveness, using source-code-quality indicators, with high accuracy and without executing the tests. In fact, adding line coverage as an additional feature only increases the performance of the model by about the 9%. As a consequence, we can provide a practical approach that is beyond the typical limitations of current mutation testing techniques.

Link to Publication: https://ieeexplore.ieee.org/document/8658120

Giovanni Grano

University of Zurich

Switzerland

Fabio Palomba

Department of Informatics, University of Zurich

Harald Gall

University of Zurich

Switzerland

Analyzing executions of concurrent software is very difficult. Even if a trace is available, such traces are very hard to read and interpret. A textual trace contains a lot of data, most of which is not relevant to the issue at hand. Past visualization attempts either do not show concurrent behavior, or result in a view that is overwhelming for the user.

We provide a visual analytics tool, VA4JVM, for error traces produced by either the Java Virtual Machine, or by Java Pathfinder. Its key features are a layout that spatially associates events with threads, a zoom function, and the ability to filter event data in various ways. We show in examples how filtering and zooming in can highlight a problem without having to read lengthy textual data.

Cyrille Artho

KTH Royal Institute of Technology, Sweden

Monali Pande

KTH Royal Institute of Technology

Qiyi Tang

University of Oxford

United Kingdom

Deep Neural Network(DNN) techniques have been prevalent in software engineering. They are employed to facilitate various software engineering tasks and embedded into many software applications. However, because DNNs are built upon a rich data-driven programming paradigm that employs plenty of labeled data to train a set of neurons to construct the internal system logic, analyzing and understanding their behaviors becomes a difficult task for software engineers. In this paper, we present an instance-based visualization tool for DNN, namely NeuralVis, to support software engineers in visualizing and interpreting deep learning models. NeuralVis is designed for: 1). visualizing the structure of DNN models, i.e., neurons, layers, as well as connections; 2). visualizing the data transformation process; 3). integrating existing adversarial attack algorithms for test input generation; 4). comparing intermediate outputs of different instances. To demonstrate the effectiveness of NeuralVis, we design a task-based user study involving ten participants on two classic DNN models, i.e., LeNet and VGG-12. The result shows NeuralVis can assist engineers in identifying critical features that determine the prediction results. Tool is available at http://118.178.18.181:18080. Video: https://youtu.be/hRxCovrOZFI.

Xufan Zhang

State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, China

Ziyue Yin

State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, China

Yang Feng

University of California, Irvine

United States

Qingkai Shi

Hong Kong University of Science and Technology

China

Jia Liu

State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, China

Zhenyu Chen

Nanjing University

China