The source code must be compiled into machine code, so that machines can understand the intention of software. Compilers are important, because a bug in compilers can hinder or even crash the compilation. As there are often more than one compiler for each programming language, differential testing has been widely used to detect bugs in compilers. Its basic idea is to compile test programs with different compilers, and compare their compilation results to detect bugs. For this research line, test programs are critical, and researchers have proposed various approaches to generate test programs. The state-of-the-art approaches can be roughly divided into random-based and mutation-based approaches: random-based approaches generate random programs and mutation-based approaches mutate programs to generate more test programs. Both lines of approaches mainly generate random code, but it is more beneficial to use real programs, since it is easier to learn the impacts of compiler bugs and it becomes reasonable to use both valid and invalid code. However, most real programs from code repositories are ineffective to trigger compiler bugs, partially because they are compiled before they are submitted. In this experience paper, we apply two techniques such as differential testing and code snippet extraction to the specific research domain of compiler testing. Based on our observations on the practice of testing compilers, we identify bug reports of compilers as a new source for compiler testing. To illustrate the benefits of the new source, we implement a tool, called LeRe, that extracts test programs from bug reports and uses differential testing to detect compiler bugs with extracted programs. After we enriched the test programs, we have found 156 unique bugs in the latest versions of gcc and clang. Among them, 103 bugs are confirmed as valid, and 9 bugs are already fixed. Our found bugs contain 59 accept-invalid bugs and 33 reject-valid bugs. In these bugs, compilers wrongly accept invalid programs or reject valid programs. The new source enables us detecting accept-invalid and reject-valid bugs that were usually missed by the prior approaches. The prior approaches seldom report the two types of bugs. Besides our found bugs, we also present our analysis on our invalid bug reports. The results are useful for programmers, when they are switching from one compiler to another, and can provide insights, when researchers apply differential testing to detect bugs in more types of software

Hao Zhong

Shanghai Jiao Tong University

I received my Ph.D degree from Peking University in 2009. My Ph.D dissertation was nominated for the distinguished Ph.D dissertation award of China Computer Federation. After graduation, I joined Institute of Software, Chinese Academy of Sciences as an assistant professor, and was promoted as an associated professor in 2011. From 2012 to 2014, I was a visiting scholar with University of California, Davis. In 2014, I joined Shanghai Jiao Tong University. I am a recipient of ACM SIGSOFT Distinguished Paper Award, the best paper award of ASE, and the best paper award of APSEC.

With the support of exception handling mechanisms, when an error occurs, its corresponding typed exception can be thrown. The thrown exception can be caught, if the type of the thrown exception matches the type of the expected exceptions. If the thrown exception is caught, the handling code will resolve the error (e.g., closing resources). Although this mechanism is critical for resolving runtime errors, bugs inside this process can have far-reaching impacts. Therefore, researchers have proposed various approaches to assist catching and handling such thrown exceptions and detect corresponding bugs (e.g., resource leaks). If the thrown exceptions themselves are incorrect, their errors will never be correctly caught and handled. In practice, such thrown exceptions have caused real critical bugs. However, to the best of our knowledge, no approach has been proposed to recommend which exceptions shall be thrown. Exceptions are widely adopted in programs, often poorly documented, and sometimes ambiguous, making the rules of throwing correct exceptions rather complicated. A project team can leverage exceptions in a way totally different from other teams. As a result, experienced programmers can have difficulties in determining which exception shall be thrown, even if its surrounding code is already written. In this paper, we propose the first approach, ThEx, to predict which exception(s) shall be thrown under a given programming context. The basic idea is to learn a classification model from existing thrown exceptions in different contexts. Here, the learning features are extracted from various code information surrounding the thrown exceptions, such as the thrown locations and related variable names. Then, given a new context, ThEx can automatically predict its best exception(s). We have evaluated ThEx on 12,012 thrown exceptions that were collected from nine popular open-source projects. Our results show that it can achieve high f-scores and mcc values (both around 0.8). On this benchmark, we also evaluated the impacts of our underlying technical details. Furthermore, we evaluated our approach in the wild, and used ThEx to detect anomalies from the latest versions of the nine projects. In this way, we found 20 anomalies, and reported them as bugs to their issue trackers. Among them, 18 were confirmed, and 13 have already been fixed.

Hao Zhong

Shanghai Jiao Tong University

I received my Ph.D degree from Peking University in 2009. My Ph.D dissertation was nominated for the distinguished Ph.D dissertation award of China Computer Federation. After graduation, I joined Institute of Software, Chinese Academy of Sciences as an assistant professor, and was promoted as an associated professor in 2011. From 2012 to 2014, I was a visiting scholar with University of California, Davis. In 2014, I joined Shanghai Jiao Tong University. I am a recipient of ACM SIGSOFT Distinguished Paper Award, the best paper award of ASE, and the best paper award of APSEC.