Industry ShowcaseASE 2020

Concolic execution and fuzzing are two complementary coverage-based testing techniques. How to achieve the best of both remains an open challenge. To address this research problem, we propose and evaluate Legion. Legion uses a variation of the Monte Carlo tree search (MCTS) framework from the AI literature to treat automated test generation as a problem of sequential decision-making under uncertainty. Its best-first search strategy provides a principled way to learn the most promising program states to investigate at each search iteration, based on observed rewards from previous iterations. Legion incorporates a form of directed fuzzing that we call approximate path-preserving fuzzing (APPFuzzing) to investigate program states selected by MCTS. APPFuzzing serves as the Monte Carlo simulation technique and is implemented by extending prior work on constrained sampling. We evaluate Legion against competitors in Test-Comp 2020, as well as measuring its sensitivity to hyperparameters, demonstrating its effectiveness on a wide variety of input programs.

Dongge Liu

The Univeristy of Melbourne

Gidon Ernst

LMU Munich

Germany

Toby Murray

University of Melbourne, Australia

Australia

Benjamin I.P. Rubinstein

University of Melbourne

In the IT project, the traditional testing method is creating test scenarios/cases according to the business flow, manually defining, and testing whether the output value is correct for each input value. Besides, massive test data is also required to perform the test accurately. So far, the traditional IT project has opened the system after manual iterating scenarios/cases tests that are empirically considered sufficient. However, due to time and resources limitation, this method cannot consider all possible cases in the real world. Thus, we cannot test all scenarios and eliminate all potential defects through this traditional testing method. As a result, unexpected errors or exceptional situations often occur even after the system goes live, which can lead to severe failure. This paper demonstrates a real transaction-based automatic testing solution called ‘PerfecTwin’ with real-world examples. PerfecTwin highlights to overcome the limitation of the traditional manual test by automatically verifying the TO-BE system with the AS-IS system’s actual transaction, which can allow eliminating all the defects before the system goes live.

YongSik Kim

SoAh Min

YouKyung Kim

LG CNS

In object-oriented programming, a method is pure if calling the method does not change object states that exist in the pre-states of the method call. Pure methods are widely-used in automatic techniques, including test generation, complier optimization, and program repair. Due to the source code dependency, it is infeasible to completely and accurately identify all pure methods. Instead, existing techniques such as ReImInfer are designed to identify a subset of accurate results of pure method and mark the other methods as unknown ones. In this paper, we designed and implemented MetPurity , a learning-based tool of pure method identification. Given all methods in a project, MetPurity labels a training set via automatic program analysis and builds a binary classifier (implemented with the random forest classifier) based on the training set. This classifier is used to predict the purity for all the other methods (i.e., unknown ones) in the same project. Preliminary evaluation on four open-source Java projects shows that MetPurity can provide a list of identified pure methods with a low error rate. Applying MetPurity to EvoSuite can increase the number of killed mutants in the test generation of EvoSuite. A demo video of this tool can be found at https://youtu.be/ Ac3cmjn4CCs/; the prototype and evaluation data are available at http://cstar.whu.edu.cn/p/metpurity/.

Runze Yu

Wuhan University

Youzhe Zhang

Wuhan University

Jifeng Xuan

Wuhan University

China

The All-SAT (All-SATisfiable) problem focuses on finding all satisfiable assignments of a given propositional formula, whose applications include model checking, automata construction, and logic minimization. A typical ALL-SAT solver is normally based on iteratively computing satisfiable assignments of the given formula. In this work, we introduce BASolver, a backbone-based All-SAT solver for propositional formulas. Compared to the existing approaches, BASolver generates shorter blocking clauses by removing backbone variables from the partial assignments and the blocking clauses. We compare BASolver with 4 existing ALL-SAT solvers, namely MBlocking, BC, BDD, and NBC. Experimental results indicate that although finding all the backbone variables consumes additional computing time, BAsolver is still more efficient than the existing solvers because of the shorter blocking clauses and the backbone variables used in it.

With the 608 formulas, BASolver solves the largest amount of formulas (86), which is 22%, 36%, 68%, 86% more formulas than MBlocking, BC, NBC, and BDD respectively. For the formulas that are both solved by BASolver and the other solvers, BASolver uses 88.4% less computing time on average than the other solvers. For the 215 formulas which first 1000 satisfiable assignments are found by at least one of the solvers, BASolver uses 180% less computing time on average than the other solvers.

Yueling Zhang

Singapore Management University

Geguang Pu

East China Normal University

Jun Sun

Singapore Management University

Singapore

A common problem in MPI programs is deadlock: when two or more processes are blocked indefinitely due to a circular communication dependency. Automatically detecting deadlock is difficult due to its schedule-dependent nature. This paper presents a predictive analysis for single-path MPI programs that observes a single program execution and then determines whether any other feasible schedule of the program can lead to a deadlock. The analysis works by identifying problematic communication patterns in a dependency graph to form a set of deadlock candidates. The deadlock candidates are filtered by an abstract machine and ultimately tested for reachability by an SMT solver with an efficient encoding for deadlock. This approach quickly yields a set of high probability deadlock candidates useful for reasoning about complex codes and yields higher performance overall in many cases compared to other state-of-the-art analyses. The analysis is sound and complete for single-path MPI programs on a given input.

Yu Huang

Southwestern University of Finance and Economics

China

Benjamin Ogles

Brigham Young University

Eric Mercer

Brigham Young University

United States

We propose a novel approach to proving the termination of imperative programs by k-induction. By our approach, the termination proving problem can be formalized as a k-inductive invariant synthesis task. On the one hand, k-induction uses weaker invariants than that required by the standard inductive approach. On the other hand, the base case of k-induction, which unrolls the program, can provide stronger pre-condition for invariant synthesis. As a result, the termination arguments of our approach can be synthesized more efficiently than the standard method. We implement a prototype of our k-inductive approach. The experimental results show the significant effectiveness and efficiency of our approach.

Jianhui Chen

Tsinghua University

Fei He

Tsinghua University, China

Exception handling is an important built-in feature of many modern programming languages such as Java. It allows developers to deal with abnormal or unexpected conditions by try-catch blocks that may occur at runtime in advance. Missing or improper implementation of exception handling can cause catastrophic consequences such as system crash. However, previous studies reveal that developers are unwilling or feel it hard to adopt exception handling mechanism, and tend to ignore it until a system failure forces them to do so. To help developers with exception handling, existing work produces recommendations such as code examples and exception types, which still requires developers to localize the try blocks and modify the catch block code to fit the context. In this paper, we propose a novel neural approach for automated exception handling, which can predict locations of try blocks and automatically generate the complete catch blocks. We collect a large number of Java methods from GitHub and conduct experiments to evaluate our approach. The evaluation results, including quantitative measurement and human evaluation, show that our approach is highly effective and outperforms all baselines. Our work makes one step further towards automated exception handling.

Jian Zhang

Beihang University

Xu Wang

Beihang University

China

Hongyu Zhang

University of Newcastle, Australia

Australia

Hailong Sun

Beihang University

Yanjun Pu

Beihang University

Xudong Liu

Beihang University

Long build times in continuous integration (CI) can greatly increase the cost in human and computing resources, and thus become a common barrier faced by software organizations adopting CI. Build outcome prediction has been proposed as one of the remedies to reduce such cost. However, the state-of-the-art approaches have a poor prediction performance for failed builds, and are not designed for practical usage scenarios. To address the problems, we first conduct an empirical study on 2,590,917 builds to characterize builds times in real-world projects, and a survey with 75 developers to understand their perceptions about build outcome prediction. Then, motivated by our study and survey results, we propose a new history-aware approach, named BuildFast, to predict CI build outcomes cost-efficiently and practically. It can help to obtain fast integration feedback and reduce integration cost. In particular, we introduce multiple failure-specific features from closely related historical builds via analyzing build logs and changed files, and propose an adaptive prediction model to switch between two models based on the build outcome of the previous build. We also investigate a practical online usage scenario of BuildFast, where builds are predicted in chronological order, and measure the benefit from correct predictions and the cost from incorrect predictions. Our experiments on 20 projects have demonstrated that BuildFast can improve the state-of-the-art approach by 47.5% in F1-score for failed builds.

Bihuan Chen

Fudan University, China

China

Linlin Chen

Fudan University

Chen Zhang

Fudan University

Xin Peng

Fudan University, China

China

Using open-source libraries can provide rich functions and reduce development cost. However, some critical issues have also been caused such as license conflicts and vulnerability risks. In this paper, we design and implement an open-source libraries detection tool OSLDetector which uses methods of matching features to detect third-party libraries for multi-platform software in binaries. We took a series of methods such as filtering features and novelty building an internal clone forest to cope with the challenge of feature duplication. The tool can also provide the conflict of licenses and identify possible corresponding vulnerabilities, so these potential risks can be resolved and avoided. To evaluate the efficiency of OSLDetector, we collect 5K libraries containing 9K versions and manage their respective license type and existing vulnerabilities. The experimental results with a precision of 96\% and recall of 92.3\% show that OSLDetector is effective and outperforms similar tools.

Dan Zhang

Tsinghua University

Symbolic execution is a well established technique for software testing and analysis. However, scalability continues to be a challenge, both in terms of constraint solving cost and path explosion. In this work, we present a novel approach for symbolic execution, which can enhance its scalability by aggressively prioritizing execution paths that are already known to be feasible, and deferring all other paths. We evaluate our technique on nine applications, including SQLite3, make and tcpdump and show it can achieve higher coverage for both seeded and non-seeded exploration.

Timotej Kapus

Imperial College London, UK

Frank Busse

Imperial College London

United Kingdom

Cristian Cadar

Imperial College London, UK

United Kingdom

Most programming languages support foreign language interoperation that allows developers to integrate multiple modules implemented in different languages into a single multilingual program. While utilizing various features from multiple languages expands expressivity, differences in language semantics require developers to understand the semantics of multiple languages and their interoperation. Because current compilers do not support compile-time checking for interoperation, they do not help developers avoid interoperation bugs. Similarly, active research on static analysis and bug detection has been focusing on programs written in a single language.

In this paper, we propose a novel approach to analyze multilingual programs statically. Unlike existing approaches that extend a static analyzer for a host language to support analysis of foreign function calls, our approach extracts semantic summaries from programs written in guest languages using a modular analysis technique, and performs a whole-program analysis with the extracted semantic summaries. To show practicality of our approach, we design and implement a static analyzer for multilingual programs, which analyzes JNI interoperation between Java and C. Our empirical evaluation shows that the analyzer is scalable in that it can construct call graphs for large programs that use JNI interoperation, and useful in that it found 74 genuine interoperation bugs in real-world Android JNI applications.

Sungho Lee

Chungnam National University, Korea

South Korea

Hyogun Lee

KAIST

Sukyoung Ryu

KAIST

South Korea

Jupyter notebooks—documents that contain live code, equations, visualizations, and narrative text—now are among the most popular means to compute, present, discuss, and disseminate scientific findings. In principle, Jupyter notebooks should easily allow to reproduce and extend scientific computations and their findings; but in practice, this is not the case. The individual code cells in Jupyter notebooks can be executed in any order, with identifier usages preceding their definitions and results preceding their computations. In a sample of 936 published notebooks that would be executable in principle, we found that 73% of them would not be reproducible with straightforward approaches, requiring humans to infer (and often guess) the order in which the authors created the cells.

In this paper, we present an approach to

1. automatically satisfy dependencies between code cells to reconstruct possible execution orders of the cells; and
2. instrument code cells to mitigate the impact of non-reproducible statements (i.e., random functions) in Jupyter notebooks.

Our Osiris prototype takes a notebook as input and outputs the possible execution schemes that reproduce the exact notebook results. In our sample, Osiris was able to reconstruct such schemes for 82.23% of all executable notebooks, which has more than three times better than the state-of-the-art; the resulting reordered code is valid program code and thus available for further testing and analysis.

Jiawei Wang

Tzu-yang Kuo

The Hong Kong University of Science and Technology

Li Li

Monash University, Australia

Australia

Andreas Zeller

CISPA, Germany

Germany

Library migration is a challenging problem, where most existing approaches rely on prior knowledge. This can be, for example, information derived from change logs or statistical models of API usage.

This paper addresses a different API migration scenario where there is no prior knowledge of the target library. We have no historical changelogs and no access to its internal representation. To tackle this problem, this paper proposes a novel approach (M$^3$), where probabilistic program synthesis is used to semantically model the behavior of library functions. Then, we use an SMT-based code search engine to discover similar code in user applications. These discovered instances provide potential locations for API migrations.

We evaluate our approach against 7 well-known libraries from varied application domains, learning correct implementations for 94 functions. Our approach is integrated with standard compiler tooling, and we used this integration to evaluate migration opportunities in 9 existing C/C++ applications with over 1MLoC. We discover over 7,000 instances of these functions, of which more than 2,000 represent migration opportunities.

Bruce Collie

University of Edinburgh

Philip Ginsbach

GitHub Software UK

Jackson Woodruff

University of Edinburgh

Ajitha Rajan

University of Edinburgh

United Kingdom

Michael F. P. O'Boyle

University of Edinburgh

The increasing adoption of the Linux kernel has been sustained by a large and constant maintenance effort, performed by a wide and heterogeneous base of contributors. One important problem that maintainers face in any code base is the rapid understanding of complex data structures. The Linux kernel is written in the C language, which enables the definition of arbitrarily uninformative datatype, via the use of casts and pointer arithmetic, of which doubly linked lists are a prominent example. In this paper, we explore the advantages and disadvantages of such lists, for expressivity, for code understanding, and for code reliability. Based on our observations, we have developed a toolset that includes inference of descriptive list types and a tool for list visualization. Our tools identify more than 10.000 list fields and variables in recent Linux kernel releases and succeeds in typing more than 90%. We show how these tools could have been used to detect previously fixed bugs and identify 6 new ones.

Nic Volanschi

Inria Bordeaux

Julia Lawall

Inria

Developers continuously invent new practices, usually grounded in hard-won experience, not theory. Game theory studies cooperation and conflict; its use will speed the development of effective processes. A survey of game theory in software engineering finds highly idealised models that are rarely based on process data. This is because software processes are hard to analyse using traditional game theory since they generate huge game models. We are the first to show how to use game abstractions, developed in artificial intelligence, to produce tractable game-theoretic models of software practices. We present Game-Theoretic Process Improvement (GTPI), built on top of empirical game-theoretic analysis. Some teams fall into the habit of preferring “quick-and-dirty” code to slow-to-write, careful code, incurring technical debt. We showcase GTPI’s ability to diagnose and improve such a development process. Using GTPI, we discover a lightweight intervention that incentivises developers to write careful code: add a single code reviewer who needs to catch only 25% of kludges. This 25% accuracy is key; it means that a reviewer does not need to examine each commit in-depth, making this process intervention cost-effective.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0164121219301980

Carlos Gavidia-Calderon

University College London

Federica Sarro

University College London, UK

United Kingdom

Mark Harman

University College London, UK

United Kingdom

Earl T. Barr

University College London, UK

A key challenge in automatic Web testing is the generation of syntactically and semantically valid input values that can exercise the many functionalities that impose constraints on the validity of the inputs. Existing test case generation techniques either rely on manually curated catalogs of values, or extract values from external data sources, such as the Web or publicly available knowledge bases. Unfortunately, relying on manual effort is generally too expensive for most practical applications, while domain-specific and application-specific data can be hardly found either on the Web or in general purpose knowledge bases. This paper proposes DBInputs, a novel approach that reuses the data from the database of the target Web applications, to automatically identify domain-specific and application-specific inputs, and effectively fulfil the validity constraints present in the tested Web pages. DBInputs can properly cope with system testing and maintenance testing efforts, since databases are naturally and inexpensively available in those phases. To extract valid inputs from the application databases, DBInputs exploits the syntactic and semantic similarity between the identifiers of the input fields and the ones in the tables of the database, automatically resolving the mismatch between the user interface and the schema of the database. Our experiments provide initial evidence that DBInputs can outperform both random input selection and LINK, a state-of-the-art approach for searching inputs from knowledge bases.

Diego Clerissi

Giovanni Denaro

University of Milano-Bicocca, Italy

Marco Mobilio

University of Milano Bicocca

Italy

Leonardo Mariani

University of Milano Bicocca

Italy

Compiler bugs can be disastrous since they could affect all the software systems built on the buggy compilers. Meanwhile, diagnosing compiler bugs is extremely challenging since usually limited debugging information is available and a large number of compiler files can be suspicious. More specifically, when compiling a given bug-triggering test program, hundreds of compiler files are usually involved, and can all be treated as suspicious buggy files. To facilitate compiler debugging, in this paper we propose the first reinforcement compiler bug isolation approach via structural mutation, called RecBi. For a given bug-triggering test program, RecBi first augments traditional local mutation operators with structural ones to transforms it into a set of passing test programs. Since not all the passing test programs can help isolate compiler bugs effectively, RecBi further leverages reinforcement learning to intelligently guide the process of passing test program generation. Then, RecBi ranks all the suspicious files by analyzing the compiler execution traces of the generated passing test programs and the given failing test program following the practice of compiler bug isolation. The experimental results on 120 real bugs from two most popular C open-source compilers, i.e., GCC and LLVM, show that RecBi is able to isolate about 23%/58%/78% bugs within Top-1/Top-5/Top-10 compiler files, and significantly outperforms the state-of-the-art compiler bug isolation approach by improving 92.86%/55.56%/25.68% isolation effectiveness in terms of Top-1/Top-5/Top-10 results.

Junjie Chen

Tianjin University, China

China

Haoyang Ma

College of Intelligence and Computing, Tianjin University

Lingming Zhang

University of Illinois at Urbana-Champaign

Software testing is an important and time-consuming task that is often done manually. In the last decades, researchers have come up with techniques to generate input data (e.g., fuzzing) and automate the process of generating test cases (e.g., search-based testing). However, these techniques are known to have their own limitations: search-based testing does not generate highly-structured data; grammar-based fuzzing does not generate test case structures. To address these limitations, we combine these two techniques. By applying grammar-based mutations to the input data gathered by the search-based testing algorithm, it allows us to co-evolve both aspects of test case generation. We evaluate our approach by performing an empirical study on 20 Java classes from the three most popular JSON parsers across multiple search budgets. Our results show that the proposed approach on average improves branch coverage for JSON related classes by 15% (with a maximum increase of 50%) without negatively impacting other classes.

Mitchell Olsthoorn

Delft University of Technology

Netherlands

Arie van Deursen

Delft University of Technology, Netherlands

Netherlands

Annibale Panichella

Delft University of Technology

Netherlands

Methods for randomized testing of compilers to find miscompilation bugs typically require a way to generate programs that are free from undefined behaviour (UB). Tools such as Csmith achieve UB-freedom by heavily restricting the form of generated programs. This leads to highly idiomatic programs, and we hypothesise that this limits the thoroughness with which compilers are tested. Our idea is that researchers should investigate ways to generate less restricted programs that are still UB-free—programs that get closer to the edge of undefined behaviour, but that do not quite cross the edge. We present experiments investigating one instance of idea via a prototype tool, CsmithEdge, that uses a simple dynamic analysis to detect where Csmith has been too conservative in its use of “safe math” wrappers that guarantee UB-freedom for arithmetic operations, eliminating redundant wrappers. By reducing the use of safe math wrappers, CsmithEdge was able to discover two new miscompilation bugs in GCC that could not be found via intensive testing using regular Csmith, as well as achieving substantial differences in code coverage on GCC compared with regular Csmith.

Karine Even-Mendoza

Imperial College London

United Kingdom

Cristian Cadar

Imperial College London, UK

United Kingdom

Alastair Donaldson

Imperial College London

United Kingdom

Block-based programming languages like Scratch support learners by providing high-level constructs that hide details and by preventing syntactically incorrect programs. Questions nevertheless frequently arise: Is this program satisfying the given task? Why is my program not working? To support learners and educators, automated program analysis is needed for answering such questions. While adapting existing analyses to process blocks instead of textual statements is straightforward, the domain of programs controlled by block-based languages like Scratch is very different from traditional programs:

In Scratch multiple actors, represented as highly concurrent programs, interact on a graphical stage, controlled by user inputs, and program statements mainly determine visual aspects and movement.

Analyzing such programs is further hampered by the absence of clearly defined semantics, often resulting from ad-hoc decisions made by the implementers of the programming environment.

To enable program analysis, we define the semantics of Scratch using an intermediate language. Based on this intermediate language, we implement the Bastet program analysis framework for Scratch programs, using concepts from abstract interpretation and software model checking.

Like Scratch, Bastet is based on Web technologies, written in TypeScript, and can be executed using NodeJS or even directly in a browser.

Evaluation on 272 programs written by children suggests that Bastet offers a practical solution for analysis of Scratch programs.

Andreas Stahlbauer

University of Passau

Germany

Christoph Frädrich

University of Passau

Gordon Fraser

University of Passau

Germany

Recent probabilistic model checking techniques can verify reliability and performance properties of software systems affected by parametric uncertainty. This involves modelling the system behaviour using \emph{interval Markov chains}, i.e., Markov models with transition probabilities or rates specified as intervals. These intervals can be updated continually using Bayesian estimators with imprecise priors, enabling the verification of the system properties of interest at runtime. However, Bayesian estimators are slow to react to sudden changes in the actual value of the estimated parameters, yielding inaccurate intervals and leading to poor verification results after such changes. To address this limitation, we introduce an efficient interval change-point detection method, and we integrate it with a state-of-the-art Bayesian estimator with imprecise priors. Our experimental results show that the resulting end-to-end Bayesian approach to change-point detection and estimation of interval Markov chain parameters handles effectively a wide range of sudden changes in parameter values, and supports runtime probabilistic model checking under parametric uncertainty.

Xingyu Zhao

Heriot-Watt University

United Kingdom

Radu Calinescu

University of York, UK

United Kingdom

Simos Gerasimou

University of York, UK

United Kingdom

Valentin Robu

Heriot-Watt University

David Flynn

Heriot-Watt University

Charts are commonly used for data visualization. Generating a chart usually involves performing data transformations, including data pre-processing and aggregation. These tasks can be cumbersome and time-consuming, even for experienced data scientists. Reproducing existing charts can also be a challenging task when information about data transformations is no longer available.

In this paper, we tackle the problem of recovering data transformations from existing charts. Given an input table and a chart, our goal is to automatically recover the data transformation program underlying the chart. We divide our approach into four steps: (1) data extraction, (2) candidate generation, (3) candidate ranking, and (4) candidate disambiguation. We implemented our approach in a tool called UnchartIt and evaluated it on a set of $50$ benchmarks from Kaggle. Experimental results show that UnchartIt successfully ranks the correct data transformation program in the top-10 in $92%$ of the instances. To disambiguate those programs, we use our new interactive disambiguation procedure, which successfully returns the correct program on 98% of the ambiguous instances by asking on average fewer than 2 questions to the user.

Daniel Ramos

INESC-ID/IST, Universidade de Lisboa

Jorge Pereira

INESC-ID/IST, Universidade de Lisboa

Ines Lynce

INESC-ID/IST, Universidade de Lisboa

Vasco Manquinho

INESC-ID/IST, Universidade de Lisboa

Ruben Martins

Carnegie Mellon University

United States

Smartphone vendors are using multiple methods to kill processes of Android apps to reduce the battery consumption. This motivates developers to find ways to extend the liveness time of their apps, hence the name diehard apps in this paper. Although there are blogs and articles illustrating methods to achieve this purpose, there is no systematic research about them. What’s more important, little is known about the prevalence of diehard apps in the wild. In this paper, we take a first step to systematically investigate diehard apps by answering the following research questions. First, why and how can they circumvent the resource-saving mechanisms of Android? Second, how prevalent are they in the wild? In particular, we conduct a semi-automated analysis to illustrate insights why existing methods to kill app processes could be evaded, and then systematically present 12 diehard methods. After that, we develop a system named DiehardDetector to detect diehard apps in a large scale. The experimental result of applying DiehardDetector to more than 80k Android apps downloaded from Google Play showed that around 21% of apps adopt various diehard methods. Moreover, our system can achieve high precision and recall.

Hao Zhou

The Hong Kong Polytechnic University

China

Haoyu Wang

Beijing University of Posts and Telecommunications, China

China

Yajin Zhou

Zhejiang University

Xiapu Luo

The Hong Kong Polytechnic University

Yutian Tang

ShanghaiTech University

China

Lei Xue

The Hong Kong Polytechnic University

Ting Wang

Penn State University

The UI driven nature of Android apps has motivated the development of automated UI analysis for various purposes, such as app analysis, malicious app detection, and app testing. Although existing automated UI analysis methods have demonstrated their capability in dissecting apps’ UI, little is known about their effectiveness in the face of app protection techniques, which have been adopted by more and more apps. In this paper, we take a first step to systematically investigate UI obfuscation for Android apps and its effects on automated UI analysis. In particular, we point out the weaknesses in existing automated UI analysis methods and design 9 UI obfuscation approaches. We implement these approaches in a new tool named UIObfuscator after tackling several technical challenges. Moreover, we feed 3 kinds of tools that rely on automated UI analysis with the apps protected by UIObfuscator, and find that their performances severely drop. This work reveals limitations of automated UI analysis and sheds light on app protection techniques.

Hao Zhou

The Hong Kong Polytechnic University

China

Ting Chen

University of Electronic Science and Technology of China

Haoyu Wang

Beijing University of Posts and Telecommunications, China

China

Le Yu

The Hong Kong Polytechnic University

Xiapu Luo

The Hong Kong Polytechnic University

Ting Wang

Penn State University

Wei Zhang

Nanjing University of Posts and Telecommunications

Mobile operating systems evolve quickly, frequently updating the APIs that app developers use to build their apps. Unfortunately, API updates do not always guarantee backward compatibility, causing apps to not longer work properly or even crash when running with an updated system. This paper presents FILO, a tool that assists Android developers in resolving backward compatibility issues introduced by API upgrades. FILO both suggests the method that needs to be modified in the app in order to adapt the app to an upgraded API, and reports key symptoms observed in the failed execution to facilitate the fixing activity. Results obtained with the analysis of 12 actual upgrade problems and the feedback produced by early tool adopters show that FILO can practically support Android developers. FILO can be downloaded from https://gitlab.com/learnERC/filo, and its video demonstration is available at https://youtu.be/WDvkKj-wnlQ.

Marco Mobilio

University of Milano Bicocca

Italy

Oliviero Riganelli

University of Milano-Bicocca, Italy

Daniela Micucci

University of Milano-Bicocca, Italy

Italy

Leonardo Mariani

University of Milano Bicocca

Italy

Evolutionary intelligence approaches have been successfully applied to assist developers during debugging by generating a test case reproducing reported crashes. These approaches use a single fitness function called Crash Distance to guide the search process toward reproducing a target crash. Despite the reported achievements, these approaches do not always successfully reproduce some crashes due to a lack of test diversity (premature convergence). In this study, we introduce a new approach, called MO-HO, that addresses this issue via multi-objectivization. In particular, we introduce two new Helper-Objectives for crash reproduction, namely test length (to minimize) and method sequence diversity (to maximize), in addition to Crash Distance. We assessed MO-HO using five multi-objective evolutionary algorithms (NSGA-II, SPEA2, PESA-II, MOEA/D, FEMO) on 124 hard-to-reproduce crashes stemming from open-source projects. Our results indicate that SPEA2 is the best-performing multi-objective algorithm for MO-HO. We evaluated this best-performing algorithm for MO-HO against the state-of-the-art: single-objective approach (Single-Objective Search) and decomposition-based multi-objectivization approach (De-MO). Our results show that MO-HO reproduces five crashes that cannot be reproduced by the current state-of-the-art. Besides, MO-HO improves the effectiveness (+10% and +8% in reproduction ratio) and the efficiency in 34.6% and 36% of crashes (i.e., significantly lower running time) compared to Single-Objective Search and De-MO, respectively. For some crashes, the improvements are very large, being up to +93.3% for reproduction ratio and -92% for the required running time.

Pouria Derakhshanfar

Delft University of Technology

Xavier Devroey

Delft University of Technology

Netherlands

Andy Zaidman

Delft University of Technology

Netherlands

Arie van Deursen

Delft University of Technology, Netherlands

Netherlands

Annibale Panichella

Delft University of Technology

Netherlands

Model transformations play an important role in the evolution of systems in various fields such as healthcare, automotive and aerospace industry. Thus, it is important to check the correctness of model transformation programs. Several approaches have been proposed to generate test cases for model transformations based on different coverage criteria (e.g., statements, rules, metamodel elements, etc.). However, the execution of a large number of test cases during the evolution of transformation programs is time-consuming and may include a lot of overlap between the test cases. In this paper, we propose a test case selection approach for model transformations based on multi-objective search. We use the non-dominated sorting genetic algorithm (NSGA-II) to find the best trade-offs between two conflicting objectives: (1) maximize the coverage of rules and (2) minimize the execution time of the selected test cases. We validated our approach on several evolution cases of medium and large ATLAS Transformation Language (ATL) programs.

Link to Publication: https://link.springer.com/article/10.1007%2Fs10515-020-00271-w

Bader Alkhazi

Kuwait University

Kuwait

Chaima Abid

University of Michigan

Marouane Kessentini

University of Michigan

United States

Dorian Leroy

JKU Linz

Manuel Wimmer

Johannes Kepler University Linz

Austria

Approaches for automatic crash reproduction aim to generate test cases that reproduce crashes starting from the crash stack traces. These tests help developers during their debugging practices. One of the most promising techniques in this research field leverages search-based software testing techniques for generating crash reproducing test cases. In this paper, we introduce Botsing, an open-source search-based crash reproduction framework for Java. Botsing implements state-of-the-art and novel approaches for crash reproduction. The well-documented architecture of Botsing makes it an easy-to-extend framework, and can hence be used for implementing new approaches to improve crash reproduction. We have applied Botsing to a wide range of crashes collected from open source systems. Furthermore, we conducted a qualitative assessment of the crash-reproducing test cases with our industrial partners. In both cases, Botsing could reproduce a notable amount of the given stack traces.

Pouria Derakhshanfar

Delft University of Technology

Xavier Devroey

Delft University of Technology

Netherlands

Annibale Panichella

Delft University of Technology

Netherlands

Andy Zaidman

Delft University of Technology

Netherlands

Arie van Deursen

Delft University of Technology, Netherlands

Netherlands

Diversity in the exhibited behavior of a given system is a desirable characteristic in a variety of application contexts. Synthesis of conformant implementations often proceeds by discovering witnessing Skolem functions, which are traditionally deterministic. In this paper, we present a novel Skolem extraction algorithm to enable synthesis of witnesses with random behavior and demonstrate its applicability in the context of reactive systems. The synthesized solutions are guaranteed by design to meet the given specification, while exhibiting a high degree of diversity in their responses to external stimuli. Case studies demonstrate how our proposed framework unveils a novel application of synthesis in model-based fuzz testing to generate fuzzers of competitive performance to general-purpose alternatives, as well as the practical utility of synthesized controllers in robot motion planning problems.

Andreas Katis

University of Minnesota

Grigory Fedyukovich

Florida State University

United States

Jeffrey Chen

University of Minnesota

David Greve

Collins Aerospace

Sanjai Rayadurgam

University of Minnesota

Michael W. Whalen

University of Minnesota

This paper aims to shed light on how loops are used in smart contracts. Towards this goal, we study various syntactic and semantic characteristics of loops used in over 20,000 Solidity contracts deployed on the Ethereum block chain, with the goal of informing future research on program analysis for smart contracts. Based on our findings, we propose a small domain-specific language (DSL) that can be used to summarize common looping patterns in Solidity. To evaluate what percentage of smart contract loops can be expressed in our proposed DSL, we also design and implement a program synthesis toolchain called Solis that can synthesize loop summaries in our DSL. Our evaluation shows that at least 56% of the analyzed loops can be summarized in our DSL, and 81% of these summaries are exactly equivalent to the original loop.

Benjamin Mariano

University of Texas at Austin

Yanju Chen

University of California, Santa Barbara

United States

Yu Feng

University of California, Santa Barbara

China

Shuvendu K. Lahiri

Microsoft Research

Isil Dillig

University of Texas at Austin, USA

Machine Learning models from other fields, like Computational Linguistics, have been transplanted to Software Engineering tasks, often quite successfully. Yet a transplanted model’s initial success at a given task does not necessarily mean it is well-suited for the task. In this work, we examine a common example of this phenomenon: the conceit that ``software patching is like language translation''. We demonstrate empirically that there are subtle, but critical distinctions between sequence-to-sequence models and translation model: while program repair benefits greatly from the former, general modeling architecture, it actually suffers from design decisions built into the latter, both in terms of translation accuracy and diversity. Given these findings, we demonstrate how a more principled approach to model design, based on our empirical findings and general knowledge of software development, can lead to better solutions. We propose several models that leverage the same machine learning tools, but whose architecture, data presentation, and metrics are specialized for the software engineering task. The resulting models perform significantly better than the studied baseline, especially in more program repair appropriate metrics. Overall, our results demonstrate the merit of studying the intricacies of machine learned models in software engineering: not only can this help elucidate potential issues that may be overshadowed by increases in accuracy; it can also help innovate on these models to raise the state-of-the-art further. We will publicly release our replication data and materials at \url{https://github.com/ARiSE-Lab/Patch-as-translation}.

Yangruibo Ding

Columbia University

Baishakhi Ray

Columbia University, USA

United States

Prem Devanbu

University of California

United States

Vincent Hellendoorn

Carnegie Mellon University

United States

Programs are becoming increasingly complex and typically contain an abundance of unneeded features, which could severely harm performance and security. Recently, we have witnessed a surge of debloating techniques that aim to create a reduced version of a program by eliminating the unneeded features therein. To debloat a program, most existing techniques require a usage profile of the program, typically provided as a set of inputs $I$. Unfortunately, these techniques tend to generate a reduced program that is overfitted to $I$ and thus fails to behave correctly for other inputs. To address this limitation of existing techniques, we propose DomGad, which has two main advantages over existing debloating approaches. First, it produces a reduced program that is guaranteed to work for entire subdomains, rather than for specific inputs. Second, it uses stochastic optimization to generate reduced programs that achieve a close-to-optimal trade-off between size reduction and generality (i.e., extent to which the reduced program is able to correctly handle inputs in its whole domain). To assess the effectiveness of DomGad, we applied our approach to a benchmark of ten Unix utility programs. Our results are promising, as they show that DomGad could produce debloated programs that achieve, on average, a 50% code reduction and 95% generality. Our results also show that DomGad performs well when compared with two state-of-the-art debloating approaches.

Qi Xin

Georgia Institute of Technology

United States

Myeongsoo Kim

Georgia Institute of Technology

United States

Qirun Zhang

Georgia Institute of Technology, USA

Alessandro Orso

Georgia Tech

United States

The research community has long recognized a complex interrelationship between test set size, test adequacy criteria, and test effectiveness in terms of fault detection. However, there is substantial confusion about the role and importance of controlling for test set size when assessing and comparing test adequacy criteria. This paper makes the following contributions: (1) A review of contradictory analyses of the relationship between fault detection, test set size, and test adequacy criteria. Specifically, this paper addresses the supposed contradiction of prior work and explains why test set size is neither a confounding variable, as previously suggested, nor an independent variable that should be experimentally manipulated. (2) An explication and discussion of the experimental design and sampling strategies of prior work, together with a discussion of conceptual and statistical problems, and specific guidelines for future work. (3) A methodology for comparing test-adequacy criteria on an equal basis, which accounts for test set size by treating it as a covariate. (4) An empirical evaluation that compares the effectiveness of coverage-based and mutation-based testing to one another and random testing. Additionally, this paper proposes probabilistic coupling, a methodology for approximating the representativeness of a set of test goals for a given set of real faults.

Yiqun Chen

University of Washington

Rahul Gopinath

CISPA Helmholtz Center for Information Security

Germany

Anita Tadakamalla

George Mason University, USA

United States

Michael D. Ernst

University of Washington, USA

Reid Holmes

University of British Columbia

Canada

Gordon Fraser

University of Passau

Germany

Paul Ammann

George Mason University, USA

René Just

University of Washington, USA

WebAssembly is a new programming language built for better performance in web applications. It defines a binary code format and a text representation for the code. At first glance, WebAssembly files are not easily understandable to human readers, regardless of the experience level. As a result, distributed third-party WebAssembly modules need to be implicitly trusted by developers as verifying the functionality requires significant effort. To this end, we develop an automated classification tool WASim for identifying the purpose of WebAssembly programs by analyzing features at the module-level. It assigns purpose labels to a module in order to assist developers in understanding the binary module. The code for WASim is available at https://github.com/WASimilarity/WASim and a video demo is available at https://youtu.be/usfYFIeTy0U.

Alan Romano

University at Buffalo, SUNY

United States

Weihang Wang

University at Buffalo, SUNY

United States

In this paper, we present Sosed, a tool for discovering similar software projects. We use fastText to compute the embeddings of sub-tokens into a dense space for 120,000 GitHub repositories in 200 languages. Then, we cluster embeddings to identify groups of semantically similar sub-tokens that reflect topics in source code. We use a dataset of 9 million GitHub projects as a reference search base. To identify similar projects, we compare the distributions of clusters among their sub-tokens. The tool receives an arbitrary project as input, extracts sub-tokens in 16 most popular programming languages, computes cluster distribution, and finds projects with the closest distribution in the search base. We labeled sub-token clusters with short descriptions to enable Sosed to produce interpretable output.

Sosed is available at https://github.com/JetBrains-Research/sosed/. The tool demo is available at https://www.youtube.com/watch?v=LYLkztCGRt8. The multi-language extractor of sub-tokens is available separately at https://github.com/JetBrains-Research/identifiers-extractor/.

Egor Bogomolov

JetBrains Research

Yaroslav Golubev

JetBrains Research, ITMO University

Russia

Artyom Lobanov

JetBrains Research

Vladimir Kovalenko

JetBrains Research, JetBrains N.V.

Timofey Bryksin

JetBrains Research, Saint Petersburg State University

Russia

Dynamic code, i.e., code that is created or modified at runtime, is ubiquitous in today’s world. The behavior of dynamic code can depend on the logic of the dynamic code generator in subtle and non-obvious ways, e.g., JIT compiler bugs can lead to exploitable vulnerabilities in the resulting JIT-compiled code. Existing approaches to program analysis do not provide adequate support for reasoning about such behavioral relationships. This paper takes a first step in addressing this problem by describing a program representation and a new notion of dependency that allows us to reason about dependency and information flow relationships between the dynamic code generator and the generated dynamic code. Experimental results show that analyses based on these concepts are able to capture properties of dynamic code that cannot be identified using traditional program analyses.

Jesse Bartels

University of Arizona

Jon Stephens

University of Texas at Austin

Saumya Debray

University of Arizona

Android platform provisions a number of sophisticated concurrency mechanisms for the development of apps. The concurrency mechanisms, while powerful, are quite difficult to properly master by mobile developers. In fact, prior studies have shown concurrency issues, such as event-race defects, to be prevalent among real-world Android apps. In this paper, we propose a flow-, context-, and thread-sensitive static analysis framework, called ER Catcher, for detection of event-race defects in Android apps. ER Catcher introduces a new type of summary function aimed at modeling the concurrent behavior of methods in both Android apps and libraries. In addition, it leverages a novel, statically constructed Vector Clock for rapid analysis of happens-before relations. Altogether, these design choices enable ER Catcher to not only detect event-race defects with a substantially higher degree of accuracy, but also in a fraction of time compared to the existing state-of-the-art technique.

Navid Salehnamadi

University of California, Irvine

Abdulaziz Alshayban

University of California, Irvine

Iftekhar Ahmed

University of California at Irvine, USA

United States

Sam Malek

University of California at Irvine, USA

United States

In recent years, IF-This-Then-That (IFTTT) services are becoming more and more popular. Many platforms such as Zapier, IFTTT.com, and Workato provide such services, which allow users to create workflows with “triggers” and “actions” by using Web Application Programming Interfaces (APIs). However, the number of IFTTT recipes in the above platforms increases much slower than the growth of Web APIs. This is because human efforts are still largely required to build and deploy IFTTT recipes in the above platforms. To address this problem, in this paper, we present an automation tool to automatically generate the IFTTT mashup infrastructure. The proposed tool provides 5 REST APIs, which can automatically generate triggers, rules, and actions in AWS, and create a workflow XML to describe an IFTTT mashup by connecting the triggers, rules, and actions. This workflow XML is automatically sent to Fujitsu RunMyProcess (RMP) to set up and execute IFTTT mashup. The proposed tool, together with its associated method and procedure, enables an end-to-end solution for automatically creating, deploying, and executing IFTTT mashups in a few seconds, which can greatly reduce the development cycle and cost for new IFTTT mashups.

Lei Liu

Fujitsu Laboratories of America, Inc.

Mehdi Bahrami

Fujitsu Laboratories of America, Inc.

United States

Wei-Peng Chen

Fujitsu Laboratories of America, Inc.

Automated test case generation tools have been successfully pro- posed to reduce the amount of human and infrastructure resources required to write and run test cases. However, recent studies demonstrate that the readability of generated tests is very limited due to (i) uninformative identifiers and (ii) lack of proper documentation. Prior studies proposed techniques to improve test readability by either generating natural language summaries or meaningful methods names. While these approaches are shown to improve test readability, they are also affected by two limitations: (1) generated summaries are often perceived as too verbose and redundant by developers, and (2) readable tests require both proper method names but also meaningful identifiers (within-method readability). In this work, we combine template based methods and Deep Learning (DL) approaches to automatically generate test case scenarios (elicited from natural language patterns of test case statements) as well as to train DL models on path-based representations of source code to generate meaningful identifier names. Our ap- proach, called DeepTC-Enhancer , recommends documentation and identifier names with the ultimate goal of enhancing readability of automatically generated test cases. An empirical evaluation with 36 external and internal developers shows that (1) DeepTC-Enhancer outperforms significantly the baseline approach for generating summaries and performs equally with the baseline approach for test case renaming, (2) the transformation proposed by DeepTC-Enhancer result in a significant increase in readability of automatically generated test cases, and (3) there is a significant difference in the feature preferences between external and internal developers.

Devjeet Roy

Washington State University

Ziyi Zhang

Washington State University

Maggie Ma

Washington State University

Venera Arnaoudova

Washington State University

United States

Annibale Panichella

Delft University of Technology

Netherlands

Sebastiano Panichella

Zurich University of Applied Sciences

Switzerland

Danielle Gonzalez

Rochester Institute of Technology, USA

United States

Mehdi Mirakhorli

Rochester Institute of Technology

United States

Specification mining, in general, and inferring behavior model of a running system, in particular, are quite useful for several automated software engineering tasks, such as program comprehension, anomaly detection, and testing. Most existing dynamic model inference techniques are white-box, i.e., they require source code to be instrumented to get run-time traces. However, in many systems, access to source code is not possible for parts of the program that use third-party binaries and off-the-shelf-components. One useful scenario for automated black-box behaviour inference is in software control units (such as autopilots), where the software system’s reactions over time changes based on the inputs. Run-time state models of such systems are very powerful means for anomaly detection and debugging. Unfortunately, most black-box techniques that detect state changes over time are either uni-variate (which is limiting the application in real-world systems) or are weak with respect to learning from past behaviour. Therefore, in this paper, we propose a hybrid deep neural network that accepts as input a set of time series, one per input signal of the system, and applies a set of convolution and recurrent layers to both learn the non-linear correlations between signals and the patterns over time. We have applied our approach to a real UAV auto-pilot solution from our industry partner with half a million lines of C code. We ran 888 random recent test cases of the system and inferred states over time. We compared our results with several traditional time series change point detection techniques and showed that our approach can improve their performance 88% to 102%, in terms of finding state change points, measured by F1 score. We also showed that our state classification algorithm provides on average 90.45% F1 score, which improves traditional classification algorithms 7% to 17%.

Mohammad Jafar Mashhadi

University of Calgary

Canada

Hadi Hemmati

University of Calgary

Canada

This paper investigates the problem of classifying Android applications into malicious and benign. We analyze the performance of a popular malware detection tool, Drebin, on malware datasets commonly used in an academic setup and show that the high detection accuracy often stems from learning benign rather than malicious indicators. That, effectively, turns the malware detection tools into benign app detectors. Yet, in practice, malware samples are often larger and can exhibit many behaviors similar to those of benign apps. Under such a challenging setup, looking for benign indicators becomes ineffective and the ability of the tools to detect malware degrades substantially.

In this paper, we propose an approach for identifying malicious portions of an app in the presence of numerous benign features, effectively eliminating “noise” and focusing the detection on truly malicious indicators.We also propose a novel metric estimating the “reasons” for correct malware classification, i.e., whether it is based on the presence of malicious indicators or the absence of benign ones. We show that our proposed approach is effective in both increasing the “standard” classification accuracy and in making more “justifiable” classification decisions.

Michael Cao

The University of British Columbia

Sahar Badihi

University of British Columbia, Canada

Khaled Ahmed

The University of British Columbia

Peiyu Xiong

The University of British Columbia

Julia Rubin

University of British Columbia, Canada

Canada

With the increasing application of deep learning (DL) models in many safety-critical scenarios, effective and efficient DL testing techniques are much in demand to improve the quality of DL models. One of the major challenges is the data gap between the training data to construct the models and the testing data to evaluate them. To bridge the gap, testers aims to collect an effective subset of inputs from the testing contexts, with limited labeling effort, for retraining DL models.

To assist the subset selection, we propose \textbf{M}ultiple-Boundary \textbf{C}lustering and \textbf{P}rioritization (\textbf{MCP}), a technique to cluster test samples into the boundary areas of multiple boundaries for DL models and specify the priority to select samples evenly from all boundary areas, to make sure enough useful samples for each boundary reconstruction.

To evaluate MCP, we conduct an extensive empirical study with three popular DL models and 33 simulated testing contexts. The experiment results show that, compared with state-of-the-art baseline methods, on effectiveness, our approach MCP has a significantly better performance by evaluating the improved quality of retrained DL models; on efficiency, MCP also has the advantages in time costs.

Weijun Shen

Nanjing University

Yanhui Li

Department of Computer Science and Technology, Nanjing University

Lin Chen

Nanjing University

China

YuanLei Han

Nanjing University

Yuming Zhou

Nanjing University

Baowen Xu

State Key Laboratory for Novel Software Technology, Nanjing University