Grammar-based testing uses a given grammar to produce syntactically valid inputs. To cover program features, it is necessary to also cover input features—say, all URL variants for a URL parser. Our k-path algorithm for grammar production systematically covers syntactic elements as well as their combinations. In our evaluation, we show that this results in a significantly higher code coverage than state of the art.

Nikolas Havrikov

CISPA Helmholtz Center for Information Security

Germany

Andreas Zeller

CISPA Helmholtz Center for Information Security

Germany

Automated testing techniques are often assessed on coverage based metrics. However, despite giving good coverage, the test cases may miss the gap between functional specification and the code implementation. This gap may be subtle in nature, arising due to the absence of logical checks, either in the implementation or in the specification, resulting in inconsistencies in the input definition. The inconsistencies may be prevalent especially for structured inputs, commonly specified using string-based data types. Our study on defects reported over popular libraries reveals that such gaps may not be limited to input validation checks. We propose a test generation technique for structured string inputs where we infer inconsistencies in input definition to expose semantic gaps in the method under test and the method specification. We assess this technique using our tool SEGATE, Semantic Gap Tester. SEGATE uses static analysis and automaton modeling to infer the gap and generate test cases. On our benchmark dataset, comprising of defects reported in 15 popular open-source libraries, written in Java, SEGATE was able to generate tests to expose 80% of the defects.

Devika Sondhi

IIIT Delhi

Rahul Purandare

IIIT-Delhi

India

Most software systems frequently encounter errors when interacting with their environments. When errors occur, error-handling code must execute flawlessly to facilitate system recovery. Implementing correct error handling is repetitive but non-trivial, and developers often inadvertently introduce bugs into error-handling code. Existing tools require correct error specifications to detect error-handling bugs. Manually generating error specifications is error-prone and tedious, while automatically mining error specifications is hard to achieve a satisfying accuracy. In this paper, we propose EH-Miner, a novel and practical tool that can automatically detect error-handling bugs without the need for error specifications. Given a function, EH-Miner mines its error-handling rules when the function is frequently checked by an equivalent condition, and handled by the same action. We applied EH-Miner to 117 mature applications across 15 software domains. EH-Miner mined error-handling rules with the precision rate of 91.1% and the recall rate of 46.9%. We reported 142 bugs to developers, and 106 bugs had been confirmed and fixed at the time of writing. We further applied EH-Miner to Linux kernel, and reported 68 bugs for kernel-4.17, of which 42 had been confirmed.

Zhouyang Jia

National University of Defense Technology

Shanshan Li

National University of Defense Technology

Tingting Yu

University of Kentucky

United States

Xiangke Liao

National University of Defense Technology, China

Ji Wang

National University of Defense Technology

Xiaodong Liu

National University of Defense Technology

Yunhuai Liu

Peking University

Modern embedded systems are increasingly complex and contain multiple software layers from BSP (Board Support Packages) to OS to middleware to AI (Artificial Intelligence) algorithms like perception and voice recognition. Integrations of inter-layer and intra-layer in embedded systems provide dedicated services such as taking a picture or movie-streaming. Accordingly, it gets more complicated to find out the root cause of a system failure. This industrial proposal describes a difficulty of testing embedded systems, and presents a case study in terms of integration testing.

Ahyoung Sung

Samsung Electronics

Yangsu Kim

Samsung Electronics

Sangjun Kim

Samsung Electronics

Jongin Kim

Samsung Electronics

Neo Jang

Samsung Electronics

As Introductory Programming classes continue to grow, it becomes ever more important to ensure that automatic grading is both efficient yet still maintains high accuracy. Specifically, the use of automated testing for this type of grading has become standard, but it is not without fault. Instructor test suites can often misclassify correct student submissions as incorrect or vice versa. To address this discrepancy, we propose a novel approach which adapts the technique of Test Suite Augmentation to an educational setting in order to improve instructor test suite quality. Our approach first partitions student programs into clusters based on Behavioral Equivalence. We then use these clusters to better guide the Pex test-generator in generating a minimal set of tests which can augment an instructor’s suite and improve grading accuracy. We evaluated our approach on an extensive set of student submissions from a CS1 course and were able to find flaws in existing instructor test suites, and then augment these suites to substantially improve their quality.

Jonathan Osei-Owusu

University of Illinois at Urbana-Champaign

Angello Astorga

University of Illinois at Urbana-Champaign

Liia Butler

University of Illinois at Urbana-Champaign

Tao Xie

Peking University

China

Geoffrey Challen

University of Illinois at Urbana-Champaign

The amount of android applications is having a tremendous increasing trend, exerting pressure over practitioners and researchers around application quality, frequent releases, and quick fixing of bugs. This pressure leads practitioners to make use of automated approaches. However, most of those use source-code as input. Nevertheless, third-party services are not able to use this approaches due to privacy factors. In this paper we present MutAPK, an open source mutation testing tool that enables the usage of APKs as input for this task. MutAPK generates mutants without the need of having access to source code, because the mutations are done in an intermediate representation of the code (i.e., SMALI) that does not require compilation. MutAPK is publicly available at GitHub https://bit.ly/2KYvgP9 VIDEO: https://bit.ly/2WOjiyy

Camilo Escobar-Velásquez

Universidad de los Andes

Colombia

Michael Osorio-Riaño

Universidad de los Andes

Mario Linares-Vásquez

Systems and Computing Engineering Department , Universidad de los Andes , Bogotá, Colombia

Colombia