ASE 2019 Journal First Presentations

Increasing interest in securing the Android ecosystem has spawned numerous efforts to assist app developers in building secure apps. These efforts have resulted in tools and techniques capable of detecting vulnerabilities and malicious behaviors in apps. However, there has been no evaluation of the effectiveness of these tools and techniques in detecting known vulnerabilities. The absence of such evaluations puts app developers at a disadvantage when choosing security analysis tools to secure their apps.

In this regard, we evaluated the effectiveness of vulnerability detection tools for Android apps. We reviewed 64 tools and empirically evaluated 14 vulnerability detection tools against 42 known unique vulnerabilities captured by Ghera benchmarks, which are composed of both vulnerable and secure apps. Of the 20 observations from the evaluation, the main observation is existing vulnerability detection tools for Android apps are very limited in their ability to detect known vulnerabilities — all of the evaluated tools together could only detect 30 of the 42 known unique vulnerabilities.

More effort is required if security analysis tools are to help developers build secure apps. We hope the observations from this evaluation will help app developers choose appropriate security analysis tools and persuade tool developers and researchers to identify and address limitations in their tools and techniques. We also hope this evaluation will catalyze or spark a conversation in the software engineering and security communities to require a more rigorous and explicit evaluation of security analysis tools and techniques.

Link to Publication: https://rdcu.be/bM7na

Venkatesh-Prasad Ranganath

Kansas State University

Joydeep Mitra

Kansas State University

United States

This article contributes to defining the design space of program repair. Repair approaches can be loosely characterized according to the main design philosophy, in particular “generate-and-validate” and synthesis-based approaches. Each of those repair approaches is a point in the design space of program repair. Our goal is to facilitate the design, development and evaluation of repair approaches by providing a framework that: a) contains components commonly present in most approaches, b) provides built-in implementations of existing repair approaches. This paper presents a Java framework named Astor that focuses on the design space of generate-and-validate repair approaches. The key novelty of Astor is to provide explicit extension points to explore the design space of program repair. Thanks to those extension points, researchers can both reuse existing program repair components and implement new ones. Astor includes 6 unique implementations of repair approaches in Java, including GenProg for Java called jGenProg. Researchers have already defined new approaches over Astor. The implementations of program repair approaches built already available in Astor are capable of repairing, in total, 98 real bugs from 5 large Java programs. Astor code is publicly available on Github: https://github.com/SpoonLabs/astor.

Matias Martinez

Université Polytechnique Hauts-de-France

Martin Monperrus

KTH Royal Institute of Technology

Sweden

Nowadays software tends to come in many different, yet similar variants, often derived from a common codebase via clone-and-own. Family-based-analysis strategies have recently shown very promising potential for improving efficiency in applying quality-assurance techniques to such variant-rich programs, as compared to variant-by-variant approaches. Unfortunately, these strategies require a single program representation superimposing all program variants in a syntactically well-formed, semantically sound and variant-preserving manner, which is usually not available and manually hard to obtain in practice. In this paper, we present a novel methodology, called SiMPOSE, for automatically generating superimpositions of existing program variants to facilitate family-based analyses of variant-rich software. To this end, we propose a novel N-way model-merging methodology to integrate the control-flow automaton (CFA) representations of N given variants of a C program into one unified CFA representation. CFA constitute a unified program abstraction used by many recent software-analysis tools for automated quality assurance. To cope with the inherent complexity of N-way model-merging, our approach (1) utilizes principles of similarity-propagation to reduce the number of potential N-way matches, and (2) enables us to decompose a set of N variants into arbitrary subsets and to incrementally derive an N-way superimposition from partial superimpositions. We apply our tool implementation of SiMPOSE to a selection of realistic C programs, frequently considered for experimental evaluation of program-analysis techniques. In particular, we investigate applicability and efficiency/effectiveness trade-offs of our approach by applying SiMPOSE in the context of family-based unit-test generation as well as model-checking as sample program-analysis techniques. Our experimental results reveal very impressive efficiency improvements by an average factor of up to 2.6 for test-generation and up to 2.4 for model-checking under stable effectiveness, as compared to variant-by-variant approaches, thus amortizing the additional effort required for merging. In addition, our results show that merging all N variants at once produces, in almost all cases, clearly more precise results than incremental step-wise 2-way merging. Finally, our comparison with major existing N-way merging techniques shows that SiMPOSE constitutes, in most cases, the best efficiency/effectiveness trade-off.

Link to Publication: https://dl.acm.org/citation.cfm?doid=3343019.3313789

Dennis Reuling

Software Engineering Group, University of Siegen

Germany

Udo Kelter

Software Engineering Group, University of Siegen

Johannes Bürdek

TU Darmstadt, Real-time Systems Lab

Malte Lochau

TU Darmstadt

Application programming interfaces (APIs) continually evolve to meet ever-changing user needs, and documentation provides an authoritative reference for their usage. However, API documentation is commonly outdated because nearly all of the associated updates are performed manually. Such outdated documentation, especially with regard to API names, causes major software development issues. In this paper, we propose a method for automatically updating outdated API names in API documentation. Our insight is that API updates in documentation can be derived from API implementation changes between code revisions. To evaluate the proposed method, we applied it to four open source projects. Our evaluation results show that our method, FreshDoc, detects outdated API names in API documentation with 48% higher accuracy than the existing state-of-the-art methods do. Moreover, when we checked the updates suggested by FreshDoc against the developers? manual updates in the revised documentation, FreshDoc addressed 82% of the outdated names. When we reported 40 outdated API names found by FreshDoc via issue tracking systems, developers accepted 75% of the suggestions. These evaluation results indicate that FreshDoc can be used as a practical method for the detection and updating of API names in the associated documentation.

Link to Publication: https://ieeexplore.ieee.org/document/8651318

Seonah Lee

Gyeongsang National University

Rongxin Wu

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology

China

Shing-Chi Cheung

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology

China

Sungwon Kang

Korea Advanced Institute of Science and Technology

To ensure security and privacy, Android employs a permission mechanism which requires developers to explicitly declare the permissions needed by their applications (apps). Users must grant those permissions before they install apps or during runtime. This mechanism protects users’ private data, but also imposes additional requirements on developers. For permission declaration, developers need knowledge about what permissions are necessary to implement various features of their apps, which is difficult to acquire due to the incompleteness of Android documentation. To address this problem, we present a novel permission recommendation system named PerRec for Android apps. PerRec leverages mining-based techniques and data fusion methods to recommend permissions for given apps according to their used APIs and API descriptions. The recommendation scores of potential permissions are calculated by a composition of two techniques which are implemented as two components of PerRec: a collaborative filtering component which measures similarities between apps based on semantic similarities between APIs; and a content-based recommendation component which automatically constructs profiles for potential permissions from existing apps. The two components are combined in PerRec for better performance. We have evaluated PerRec on 730 apps collected from Google Play and F-Droid, a repository of free and open source Android apps. Experimental results show that our approach significantly improves the state-of-the-art approaches APRec_CF_correlation , APRec_TEXT and Axplorer.

Link to Publication: https://xin-xia.github.io/publication/AUSE19.pdf

Zhongxin Liu

Zhejiang University

China

Xin Xia

Monash University

Australia

David Lo

Singapore Management University

Singapore

John Grundy

Monash University

Australia

Precisely modeling complex systems like cyber-physical systems is challenging, which often render model-based system verification techniques like model checking infeasible. To overcome this challenge, we propose a method called LAR to automatically ‘verify’ such complex systems through a combination of learning, abstraction and refinement from a set of system log traces. We assume that log traces and sampling frequency are adequate to capture ‘enough’ behaviour of the system. Given a safety property and the concrete system log traces as input, LAR automatically learns and refines system models, and produces two kinds of outputs. One is a counterexample with a bounded probability of being spurious. The other is a probabilistic model based on which the given property is ‘verified’. The model can be viewed as a proof obligation, i.e., the property is verified if the model is correct. It can also be used for subsequent system analysis activities like runtime monitoring or model-based testing. Our method has been implemented as a self-contained software toolkit. The evaluation on multiple benchmark systems as well as a real-world water treatment system shows promising results.

Link to Publication: https://ieeexplore.ieee.org/abstract/document/8576657

Jingyi Wang

National University of Singapore, Singapore

Singapore

Jun Sun

Singapore Management University, Singapore

Singapore

Shengchao Qin

University of Teesside

United Kingdom

Cyrille Jegourel

ISTD, Singapore University of Technology and Design

Context: Classification techniques of supervised machine learning have been successfully applied to various domains of practice. When building a predictive model, there are two important criteria: predictive accuracy and interpretability, which generally have a trade-off relationship. In particular, interpretability should be accorded greater emphasis in the domains where the incorporation of expert knowledge into a predictive model is required. Objective: The aim of this research is to propose a new classification model, called superposed naive Bayes (SNB), which transforms a naive Bayes ensemble into a simple naive Bayes model by linear approximation. Method: In order to evaluate the predictive accuracy and interpretability of the proposed method, we conducted a comparative study using well-known classification techniques such as rule-based learners, decision trees, regression models, support vector machines, neural networks, Bayesian learners, and ensemble learners, over 13 real-world public datasets. Results: A trade-off analysis between the accuracy and interpretability of different classification techniques was performed with a scatter plot comparing relative ranks of accuracy with those of interpretability. The experiment results show that the proposed method (SNB) can produce a balanced output that satisfies both accuracy and interpretability criteria. Conclusions: SNB offers a comprehensible predictive model based on a simple and transparent model structure, which can provide an effective way for balancing the trade-off between accuracy and interpretability.

Link to Publication: https://link.springer.com/article/10.1007/s10664-018-9638-1

Toshiki Mori

Corporate Software Engineering & Technology Center, Toshiba Corporation

Japan

Naoshi Uchihira

School of Knowledge Science, Japan Advanced Institute of Science and Technology (JAIST)

Japan

Collaboration with other people is a major theme in the information-seeking process. However, most existing works that address the location of features during the maintenance or evolution of software do not support collaboration, or they are focused on code as the main software artifact. Hence, collaborative feature location in models has not enjoyed much attention to date. In this work, we address this concern by proposing an approach, CoFLiM, that enables the collaboration of several domain experts in order to locate the model fragment of a target feature. CoFLiM uses the feature descriptions of the domain experts and their self-rated confidence level to automatically reformulate the relevant feature descriptions in a single query. This query guides the evolutionary algorithm of our approach that finds the model fragment of the feature being located. We evaluate CoFLiM in a real-world case study from our industrial partner. We analyze the impact of CoFLiM in terms of recall, precision, and the F-measure. Moreover, we compare the reformulation of CoFLiM with four baselines. We also perform a statistical analysis to show that the impact of the results is significant. Our results show that collaboration pays off in the location of features in models. The results also show that the self-rated confidence level can be used to locate features in models. Finally, the results show that there are no significant improvements when more than three domain experts are involved, which is relevant in those industrial contexts where the availability of domain experts is scarce.

Link to Publication: https://link.springer.com/article/10.1007%2Fs10515-019-00251-9

Francisca Pérez

SVIT Research GroupUniversidad San Jorge

Spain

Jaime Font

San Jorge University, Spain

Lorena Arcega

San Jorge University

Carlos Cetina

San Jorge University, Spain

In software testing, different testers focus on different aspects of the software such as functionality, performance, design, and other attributes. While many tools and coverage metrics exist to support testers at the code level, not much support is targeted for testers who want to inspect the output of a program such as a dynamic web application. To support this category of testers, we propose a family of output-coverage metrics (similar to statement, branch, and path coverage metrics on code) that measure how much of the possible output has been produced by a test suite and what parts of the output are still uncovered. To do that, we first approximate the output universe using our existing symbolic execution technique. Then, given a set of test cases, we map the produced outputs onto the output universe to identify the covered and uncovered parts and compute output-coverage metrics. In our empirical evaluation on seven real-world PHP web applications, we show that selecting test cases by output coverage is more effective at identifying presentation faults such as HTML validation errors and spelling errors than selecting test cases by traditional code coverage. In addition, to help testers understand output coverage and augment test cases, we also develop a tool called WebTest that displays the output universe in one single web page and allows testers to visually explore covered and uncovered parts of the output.

Link to Publication: https://www.cs.cmu.edu/~ckaestne/pdf/jase18.pdf

Hung Viet Nguyen

Google LLC, USA

Hung Dang Phan

ECpE Department, Iowa State University

Christian Kästner

Carnegie Mellon University

United States

Tien N. Nguyen

University of Texas at Dallas

United States

Execution logs, which are generated by logging code, are widely used in modern software projects for tasks like monitoring, debugging, and remote issue resolution. Ineffective logging would cause confusion, lack of information during problem diagnosis, or even system crash. However, it is challenging to develop and maintain logging code, as it inter-mixes with the feature code. Furthermore, unlike feature code, it is very challenging to verify the correctness of logging code. Currently developers usually rely on their intuition when performing their logging activities. There are no well established logging guidelines in research and practice. In this paper, we intend to derive such guidelines through mining the historical logging code changes. In particular, we have extracted and studied the Logging-Code-Issue-Introducing (LCII) changes in six popular large-scale Java-based open source software systems. Preliminary studies on this dataset show that: (1) both co-changed and independently changed logging code changes can contain fixes to the LCII changes; (2) the complexity of fixes to LCII changes are similar to regular logging code updates; (3) it takes longer for developers to fix logging code issues than regular bugs; and (4) the state-of-the-art logging code issue detection tools can only detect a small fraction (3%) of the LCII changes. This highlights the urgent need for this area of research and the importance of such a dataset.

Link to Publication: https://www.eecs.yorku.ca/~chenfsd/resources/emse2019_chen.pdf

Boyuan Chen

York University

Zhen Ming (Jack) Jiang

York University

This paper explores the structure of research papers in software engineering. Using text mining, we study 35,391 software engineering (SE) papers from 34 leading SE venues over the last 25 years. These venues were divided, nearly evenly, between conferences and journals. An important aspect of this analysis is that it is fully automated and repeatable. To achieve that automation, we used topic modeling (with LDA) to mine 10 topics that represent much of the structure of contemporary SE. The 10 topics presented here should not be “set in stone” as the only topics worthy of study in SE. Rather our goal is to report that (a) text mining methods can detect large scale trends within our community; (b) those topic change with time; so (c) it is important to have automatic agents that can update our understanding of our community whenever new data arrives.

Link to Publication: https://ieeexplore.ieee.org/abstract/document/8465996

George Mathew

Department of Computer Science, North Carolina State University

Amritanshu Agrawal

Wayfair

United States

Tim Menzies

North Carolina State University

United States

Defect prediction models focus on identifying defect-prone code elements, for example, to allow practitioners to allocate testing resources on specific subsystems and to provide assistance during code reviews. While the research community has been highly active in proposing metrics and methods to predict defects on long-term periods (i.e., at release time), a recent trend is represented by the so-called short-term defect prediction (i.e., at commit-level). Indeed, this strategy represents an effective alternative in terms of effort required to inspect files likely affected by defects. Nevertheless, the granularity considered by such models might be still too coarse. Indeed, existing commit-level models highlight an entire commit as defective even in cases where only specific files actually contain defects.

In this paper, we first investigate to what extent commits are partially defective; then, we propose a novel fine-grained just-in-time defect prediction model to predict the specific files, contained in a commit, that are defective. Finally, we evaluate our model in terms of (i) performance and (ii) the extent to which it decreases the effort required to diagnose a defect. Our study highlights that: (1) defective commits are frequently composed of a mixture of defective and non-defective files, (2) our fine-grained model can accurately predict defective files with an AUC-ROC up to 82% and (3) our model would allow practitioners to save inspection efforts with respect to standard just-in-time techniques.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0164121218302656?via%3Dihub

Luca Pascarella

Delft University of Technology

Netherlands

Fabio Palomba

Department of Informatics, University of Zurich

Alberto Bacchelli

University of Zurich

Switzerland

Modern software systems are increasingly dependent on third-party libraries. It is widely recognized that reusing functionality provided by mature and well-tested third-party libraries can improve developers’ productivity, reduce time-to-market, and produce more reliable software. Today’s open-source repositories provide a wide range of libraries that can be freely downloaded and used. However, as software libraries are documented separately but intended to be used together, developers are unlikely to fully take advantage of these reuse opportunities. In this paper, we present a novel approach to automatically identify third-party library usage patterns, i.e., collections of libraries that are commonly used together by developers. Our approach employs a hierarchical clustering technique to group together software libraries based on external client usage. Our approach is based on the analysis of the joint versus separate use of the libraries. The pattern’s libraries are distributed on different usage cohesion levels/layers. Each layer reflects the co-usage frequency between a set of libraries, while the distribution on the different levels demonstrates the graduation in the degree of co-usage frequency. To evaluate our approach, we mined a large set of over 6000 popular libraries from Maven Central Repository and investigated their usage by over 38,000 client systems from the GitHub repository. Our experiments show that our technique is able to detect the majority (77%) of highly consistent and cohesive library usage patterns across a considerable number of client systems.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0164121218301699

Mohamed Aymen Saied

Concordia University

Ali Ouni

ETS Montreal, University of Quebec

Canada

Houari Sahraoui

Université de Montréal

Canada

Raula Gaikovina Kula

NAIST

Japan

Katsuro Inoue

Osaka University

Japan

David Lo

Singapore Management University

Singapore

Context Specification mining techniques are typically used to extract the specification of a software in the absence of (up-to-date) specification documents. This is useful for program comprehension, testing, and anomaly detection. However, specification mining can also potentially be used for debugging, where a faulty behavior is abstracted to give developers a context about the bug and help them locating it.

Objective In this project, we investigate this idea in an industrial setting. We propose a very basic semi-automated specification mining approach for debugging and apply that on real reported issues from an AutoPilot software system from our industry partner, MicroPilot Inc. The objective is to assess the feasibility and usefulness of the approach in a real-world setting.

Method The approach is developed as a prototype tool, working on C code, which accept a set of relevant state fields and functions, per issue, and generates an extended finite state machine that represents the faulty behavior, abstracted with respect to the relevant context (the selected fields and functions).

Results We qualitatively evaluate the approach by a set of interviews (including observational studies) with the company’s developers on their real-world reported bugs. The results show that (a) our approach is feasible, (b) it can be automated to some extent, and (c) brings advantages over only using their code-level debugging tools. We also compared this approach with traditional fully automated state-merging algorithms and reported several issues when applying those techniques on a real-world debugging context.

Conclusion The main conclusion of this study is that the idea of an “interactive” specification mining rather than a fully automated mining tool is NOT impractical and indeed is useful for the debugging use case.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0950584919301004

Mohammad Jafar Mashhadi

University of Calgary

Canada

Taha R. Siddiqui

InfoMagnetics Technologies Corp

Canada

Hadi Hemmati

University of Calgary

Canada

Howard W. Loewen

Department of Electrical & Computer Engineering, University of Calgary

Canada

Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, i.e., variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. Indeed, to apply mutation testing in practice a developer needs to (i) generate mutants, (ii) compile the source code, and (iii) run test cases. In cases of systems where test suites count hundrends of test cases, this is unfeasible. As a consequence, the problem of automatically assessing test-case effectiveness in a timely and efficient manner is still far from being solved. In this paper, we present a novel methodology orthogonal to existing approaches. In particular, we investigate the feasibility to estimate test-case effectiveness, as indicated by mutation score, exploiting production and test-code-quality indicators. We firstly select a set of 67 factors, including test and code metrics as well as test- and code-smells, and study their relation with test-case effectiveness. We discover that 41 out of 67 of the investigated factors statistically differs from non-effective and effective tests. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the study reveal that our estimation model only based only on statically computable features has 86% of both F-Measure and AUC-ROC. This means that we can estimate the test-case effectiveness, using source-code-quality indicators, with high accuracy and without executing the tests. In fact, adding line coverage as an additional feature only increases the performance of the model by about the 9%. As a consequence, we can provide a practical approach that is beyond the typical limitations of current mutation testing techniques.

Link to Publication: https://ieeexplore.ieee.org/document/8658120

Giovanni Grano

University of Zurich

Switzerland

Fabio Palomba

Department of Informatics, University of Zurich

Harald Gall

University of Zurich

Switzerland

Context: The challenge of locating bugs in mostly large-scale software systems has led to the development of bug localization techniques. However, the lexical mismatch between bug reports and source codes degrades the performances of existing information retrieval or machine learning-based approaches. Objective: To bridge the lexical gap and improve the effectiveness of localizing buggy files by leveraging the extracted semantic information from bug reports and source code. Method: We present BugTranslator, a novel deep learning-based machine translation technique composed of an attention-based recurrent neural network (RNN) Encoder-Decoder with long short-term memory cells. One RNN encodes bug reports into several context vectors that are decoded by another RNN into code tokens of buggy files. The technique studies and adopts the relevance between the extracted semantic information from bug reports and source files. Results: The experimental results show that BugTranslator outperforms a current state-of-the-art word embed- ding technique on three open-source projects with higher MAP and MRR. The results show that BugTranslator can rank actual buggy files at the second or third places on average.
Conclusion: BugTranslator distinguishes bug reports and source code into different symbolic classes and then extracts deep semantic similarity and relevance between bug reports and the corresponding buggy files to bridge the lexical gap at its source, thereby further improving the performance of bug localization.

Link to Publication: https://www.sciencedirect.com/science/article/abs/pii/S0950584918300399

Yan Xiao

Department of Computer Science, City University of Hong Kong

Jacky Keung

Department of Computer Science, City University of Hong Kong

Kwabena E. Bennin

Blekinge Institute of Technology, SERL Sweden

Sweden

Qing Mi

Department of Computer Science, City University of Hong Kong

With the advent of social media, developers are increasingly using it in their software development activities. Twitter is also one of the popular social mediums used by developers. A recent study by Singer et al. found that software developers use Twitter to “keep up with the fast-paced development landscape”. Unfortunately, due to the general purpose nature of Twitter, it’s challenging for developers to use Twitter for their development activities. Our survey with 36 developers who use Twitter in their development activities highlights that developers are interested in following specialized software gurus who share relevant technical tweets. To help developers perform this task, in this work we propose a recommendation system to identify specialized software gurus. Our approach first extracts different kinds of features that characterize a Twitter user and then employs a two-stage classification approach to generate a discriminative model, which can differentiate specialized software gurus in a particular domain from other Twitter users that generate domainrelated tweets (aka domain-related Twitter users). We have investigated the effectiveness of our approach in finding specialized software gurus for four different domains (JavaScript, Android, Python, and Linux) on a dataset of 86,824 Twitter users who generate 5,517,878 tweets over one month. Our approach can differentiate specialized software experts from other domain-related Twitter users with an F-measure of up to 0.820. Compared with existing Twitter domain expert recommendation approaches, our proposed approach can outperform their F-measure by at least 7.63%.

Abhishek Sharma

Singapore Management University, Singapore

Yuan Tian

Queens University, Kingston, Canada

Canada

Agus Sulistya

School of Information Systems, Singapore Management University

Dinusha Wijedasa

School of Information Systems, Singapore Management University

David Lo

Singapore Management University

Singapore

Software systems are often released without formal specifications. To deal with the problem of lack of and outdated specifications, rule-based specification mining approaches have been proposed. These approaches analyze execution traces of a system to infer the rules that characterize the protocols, typically of a library, that its clients must obey. Rule-based specification mining approaches work by exploring the search space of all possible rules and use interestingness measures to differentiate specifications from false positives. Previous rule-based specification mining approaches often rely on one or two interestingness measures, while the potential benefit of combining multiple available interestingness measures is not yet investigated. In this work, we propose a learning to rank based approach that automatically learns a good combination of 38 interestingness measures. Our experiments show that the learning to rank based approach outperforms the best performing approach leveraging single interestingness measure by up to 66%.

Link to Publication: https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=4990&context=sis_research

Zherui Cao

Zhejiang University

China

Yuan Tian

Queens University, Kingston, Canada

Canada

Tien-Duy B. Le

School of Information Systems, Singapore Management University

David Lo

Singapore Management University

Singapore

The significant rise in the complexity and configurations of software systems potentially requires a large number of test cases to test configurations of the systems. To deal with this challenge, we present a novel automated approach (termed as SBI) to implant existing test cases of software systems with the functionality to cover untested configurations of the software system; thus significantly reducing the effort to write new test cases to test the untested configurations. Our automated approach consists of two keys steps implemented into software components termed as Test Case Analyzer and Test Case Implanter. Test Case Analyzer automatically analyzes the code of test cases and generates a program dependence graph required by the second component. Test Case Implanter is implemented with a multi-objective search approach that searches for the most suitable test cases for implantation using search operators (i.e., selection, crossover, and mutation operators) followed by implanting the selected test cases by applying mutation operations focusing on adding, modifying, or deleting the test case source code. To guide the search, we defined various objectives: number of configuration variable values covered, pairwise coverage of parameter values of test API commands, number of implanted test cases, number of changed statements, and estimated execution time.
To assess the cost-effectiveness of our approach, we applied it to one industrial case study of video conferencing system and one open source case study. We experimented our approach (i.e., SBI) with three search algorithms: Non-dominated Sorting Genetic Algorithm II (NSGA-II), weight-based genetic algorithm (WGA), and random search (RS). In general, we found that the implanted test cases with SBI using the three algorithms had significantly higher configuration coverage than the original test cases for both case studies. When comparing the three algorithms with our approach, we found that NSGA-II performed the best. Specifically, SBI with NSGA-II managed to achieve, on average 19.3% higher number of configuration variable values and 57.0% higher pairwise coverage of parameter values of test API commands.

Link to Publication: https://www.sciencedirect.com/science/article/abs/pii/S0950584919300540

Dipesh Pradhan

Simula Research Laboratory, Norway

Shuai Wang

Hong Kong University of Science and Technology

Tao Yue

Nanjing University of Aeronautics and Astronautics & Simula Research Laboratory

China

Shaukat Ali

Simula Research Lab

Marius Liaaen

Cisco Systems

Norway

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to >100%). This variability originates from three sources (variability inherent to a benchmark, between trials on the same instance, and between different instances) and different benchmark-environment configurations suffer to very different degrees from any of these sources. The bare-metal instance expectedly produces very stable results, however AWS is typically not substantially less stable. We further study falsely-reported performance changes and minimal-detectable slowdowns along two dimensions in all environments: (1) two deployment strategies, executing test and control group on the same (trial-based sampling) and on different (instance-based sampling) instances, and (2) two state-of-the-art statistical tests, i.e., Wilcoxon rank-sum with Cliff’s Delta effect sizes and bootstrapped overlapping confidence intervals. We show that identical measurements (e.g., same benchmark executed in the same environment without any code changes) suffer from falsely-reported changes when they are taken from a small number of instances and trials. Nonetheless, an increase in samples yields a low number of false positives (i.e., <5%) for all studied benchmarks and environments, both sampling-strategies, and both statistical tests. Regarding minimal-detectable slowdowns, our experiments confirm that testing in a trial-based fashion leads to substantially better results than executing on different instances. With trial-based sampling, slowdowns of 10% or less are detectable with high confidence using both statistical tests. Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns with fewer instances than overlapping confidence intervals: already 5 instances are sufficient to find slowdowns in the range of 5% to 10%.

Link to Publication: https://link.springer.com/article/10.1007/s10664-019-09681-1

Christoph Laaber

University of Zurich

Switzerland

Joel Scheuner

Chalmers | University of Gothenburg

Sweden

Philipp Leitner

Chalmers University of Technology & University of Gothenburg

Sweden

Given the event-driven and framework-based architecture of Android apps, finding the ordering of callbacks executed by the framework remains a problem that affects every tool that requires inter-callback reasoning. Previous work has focused on the ordering of callbacks related to the Android components and GUI events. But the execution of callbacks can also come from direct calls of the framework (API calls). This paper defines a novel program representation, called Callback Control Flow Automata (CCFA), that specifies the control flow of callbacks invoked via a variety of sources. We present an analysis to automatically construct CCFAs by combining two callback control flow representations developed from the previous research, namely, Window Transition Graphs (WTGs) and Predicate Callback Summaries (PCSs). To demonstrate the usefulness of our representation, we integrated CCFAs into two client analyses: a taint analysis using FLOWDROID, and a value-flow analysis that computes source and sink pairs of a program. Our evaluation shows that we can compute CCFAs efficiently and that CCFAs improved the callback coverages over WTGs. As a result of using CCFAs, we obtained 33 more true positive security leaks than FLOWDROID over a total of 55 apps we have run. With a low false positive rate, we found that 22.76% of source-sink pairs we computed are located in different callbacks and that 31 out of 55 apps contain source-sink pairs spreading across components. Thus, callback control flow graphs and inter-callback analysis are indeed important. Although this paper mainly uses Android, we believe that CCFAs can be useful for modeling control flow of callbacks for other event-driven, framework-based systems.

Link to Publication: https://ieeexplore.ieee.org/document/8613913

Danilo Dominguez Perez

Iowa State University

Panama

Wei Le

Iowa State University

Modern software systems, such as Cyber-Physical Systems (CPSs), often require interacting with environment or human. As environment and human are unpredictable or non-determinate, uncertainty is inevitable in such systems. In this paper, we proposed UncerTest to test such systems (i.e., CPSs in our context) with an explicit consideration of uncertainty, through test design, test generation, test optimization and test reporting. UncerTest is a model-based and search-based approach that rely on belief models — a test ready model with explicit subjective uncertainties. In UncerTest, we extended an uncertainty-wise test modeling framework (UncerTum) for designing and enabling of introduction of indeterminacy sources in the environment during test execution. To obtain tests, we first proposed an uncertainty-wise test generation with two strategies, designed for different coverage criteria of models, for enabling an automated test generations with uncertainties. In addition, we developed an uncertainty-wise test minimization to optimize the generated tests based on uncertainty related property using multi-objective search. We conducted an extensive empirical study with two phases to evaluate UncerTest with five use cases of two industrial CPS case studies. In the first phase, eight commonly used multi-objective search algorithms were selected for studying the best algorithm for each of the four minimization strategies. Next, the test cases obtained with UncerTest strategies (i.e., two generation strategies combined with four minimization strategies with the best algorithm) were executed on the two real CPSs for studying the performance of each UncerTest strategies. With the best strategy, we managed to observe 51% more uncertainties due to unknown indeterminate behaviors of the physical environments of the CPSs as compared to the other test strategies. In addition, with the same strategy, we managed to observe 118% more unknown uncertainties that are not specified in the test ready models.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0164121219300561

Man Zhang

Kristiania University

Shaukat Ali

Simula Research Lab

Tao Yue

Nanjing University of Aeronautics and Astronautics & Simula Research Laboratory

China

Context Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from “order effects”, i.e., different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results; specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results.

Objective To provide a method in which distributions generated by LDA are more stable and can be used for further analysis.

Method We use LDADE, a search-based software engineering tool which uses Differential Evolution (DE) to tune the LDA’s parameters. LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands of Software Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark) across Linux platform and for different kinds of LDAs (VEM, Gibbs sampling). Results were scored via topic stability and text mining classification accuracy.

Results In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE’s tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning.

Conclusion Due to topic instability, using standard LDA with its “off-the-shelf” settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.

Link to Publication: https://www.sciencedirect.com/science/article/pii/S0950584917300861

Amritanshu Agrawal

Wayfair

United States

Wei Fu

Department of Computer Science, North Carolina State University

United States

Tim Menzies

North Carolina State University

United States

Many Natural Language Processing (NLP) tasks, such as sentiment analysis or syntactic parsing, have benefited from the development of word embedding models. In particular, regardless of the training algorithms, the learned embeddings have often been shown to be generalizable to different NLP tasks. In contrast, despite recent momentum on word embeddings for source code, the literature lacks evidence of their generalizability beyond the example task they have been trained for.

In this experience paper, we identify 3 potential downstream tasks, namely code comments generation, code authorship identification, and code clones detection, that source code token embedding models can be applied to. We empirically assess a recently proposed code token embedding model, namely code2vec’s token embeddings. Code2vec was trained on the task of predicting method names, and while there is potential for using the vectors it learns on other tasks, it has not been explored in literature. Therefore, we fill this gap by focusing on its generalizability for the tasks we have identified. Eventually, we show that source code token embeddings cannot be readily leveraged for the downstream tasks. Our experiments even show that our attempts to use them do not result in any improvements over less sophisticated methods. We call for more research into effective and general use of code embeddings.

Kang Hong Jin

School of Information Systems, Singapore Management University

Tegawendé F. Bissyandé

SnT, University of Luxembourg

Luxembourg

David Lo

Singapore Management University

Singapore

Code retrieval techniques and tools have been playing a key role in facilitating software developers to retrieve existing code fragments from available open-source repositories given a user query (e.g., a short natural language text describing the functionality for retrieving a particular code snippet). Despite the existing efforts in improving the effectiveness of code retrieval, there are still two main issues hindering them from being used to accurately retrieve satisfiable code fragments from large-scale repositories when answering complicated queries. First, the existing approaches only consider shallow features of source code such as method names and code tokens, but ignoring structured features such as abstract syntax trees (ASTs) and control-flow graphs (CFGs) of source code, which contains rich and well-defined semantics of source code. Second, although the deep-learning-based approach performs well on the representation of source code, it lacks the explainability, making it hard to interpret the retrieval results and almost impossible to understand which features of source code contribute more to the final results. To tackle the two aforementioned issues, this paper proposes MMAN, a novel \underline{M}ulti-\underline{M}odal \underline{A}ttention \underline{N}etwork for semantic source code retrieval. A comprehensive multi-modal representation is developed for representing unstructured and structured features of source code, with one LSTM for the sequential tokens of the code, a Tree-LSTM for the ASTs of the code and a GGNN (Gated Graph Neural Network) for the CFG of the code. Furthermore, a multi-modal attention fusion layer is applied to assign weights to different parts of each modality of source code and then integrate them into a single hybrid representation. Comprehensive experiments and analysis on a large-scale real-world dataset show that our proposed model can accurately retrieve code snippets and outperforms the state-of-the-art methods.

Yao Wan

Zhejiang University

China

Jingdong Shu

Zhejiang University

Yulei Sui

University of Technology Sydney, Australia

Australia

Guandong Xu

University of Technology, Sydney

Australia

Zhou Zhao

Zhejiang University

Jian Wu

Zhejiang University

philip yu

University of Illinois at Chicago

ACM SIGSOFT Distinguished Paper Award

Automated test generation and evaluation in simulation environments is a key technology for verification of automated driving (AD) applications.
Search-based testing (SBT) is an approach for automated test generation that leverages optimization to efficiently generate interesting concrete tests from abstract test descriptions. In this experience paper, we report on our observations after successfully applying SBT to AD control applications in several use cases with different characteristics. Based on our experiences, we derive a number of lessons learned that we consider important for the adoption of \sbt methods and tools in industrial settings. The key lesson is that SBT finds relevant errors and provides valuable feedback to the developers, but requires tool support for writing specifications.

Christoph Gladisch

Corporate Research, Robert Bosch GmbH

Thomas Heinz

Corporate Research, Robert Bosch GmbH

Christian Heinzemann

Corporate Research, Robert Bosch GmbH

Jens Oehlerking

Corporate Research, Robert Bosch GmbH

Anne von Vietinghoff

Corporate Research, Robert Bosch GmbH

Tim Pfitzer

Robert Bosch Automotive Steering GmbH

Despite being adopted in software engineering tasks, deep neural networks are treated mostly as a black box due to the difficulty in interpreting how the networks infer the outputs from the inputs. To address this problem, we propose AutoFocus, an automated approach for rating and visualizing the importance of input elements based on their effects on the outputs of the networks. The approach is built on our hypotheses that (1) attention mechanisms incorporated into neural networks can generate discriminative scores for various input elements and (2) the discriminative scores reflect the effects of input elements on the outputs of the networks. This paper verifies the hypotheses by applying AutoFocus on the task of algorithm classification (i.e., given a program source code as input, determine the algorithm implemented by the program). AutoFocus identifies and perturbs code elements in a program systematically, and quantifies the effects of the perturbed elements on the network’s classification results. Based on evaluation on more than 1000 programs for 10 different sorting algorithms, we observe that the attention scores are highly correlated to the effects of the perturbed code elements. Such a correlation provides a strong basis for the uses of attention scores to interpret the relations between code elements and the algorithm classification results of a neural network, and we believe that visualizing code elements in an input program ranked according to their attention scores can facilitate faster program comprehension with reduced code.

Nghi Duy Quoc Bui

Singapore Management University, Singapore

Singapore

Yijun Yu

The Open University, UK

United Kingdom

Lingxiao Jiang

Singapore Management University

Singapore

Recurrent neural network (RNN) has achieved great success in processing sequential inputs for applications such as automatic speech recognition, natural language processing and machine translation. However, quality and reliability issues of RNNs make them vulnerable to adversarial attacks and hinder their deployment in real-world applications. In this paper, we propose a quantitative analysis framework — DeepStellar — to pave the way for effective security and quality analysis of software systems powered by RNNs. DeepStellar is generic to handle various RNN architectures, including LSTM and GRU, scalable to work on industrial-grade RNN models, and extensible to develop customized analyzers and tools. We demonstrated that, with DeepStellar, users are able to design efficient test generation tools, and develop effective adversarial sample detectors. We evaluated the developed applications on three real RNN models, including speech recognition and image classification.
DeepStellar outperforms existing approaches three hundred times in generating defect-triggering tests and achieves 97% accuracy in detecting adversarial attacks. A video demonstration which shows the main features of DeepStellar is available at https://sites.google.com/view/deepstellar/home/tool-demo.

Xiaoning Du

Nanyang Technological University

Xiaofei Xie

Nanyang Technological University

Yi Li

Nanyang Technological University

Singapore

Lei Ma

Kyushu University

Japan

Yang Liu

Nanyang Technological University, Singapore

Singapore

Jianjun Zhao

Kyushu University

Japan

GUI-based testing has been primarily used to examine the functionality and usability of mobile apps. Despite the numerous GUI-based test input generation techniques proposed in the literature, these techniques are still limited by (1) lack of context-aware text inputs; (2) failing to generate expressive tests; and (3) absence of test oracles. To address these limitations, we propose CraftDroid, a framework that leverages information retrieval, along with static and dynamic analysis techniques, to extract the human knowledge from an existing test suite for one app and transfer the test cases and oracles to be used for testing other apps with the similar functionalities. Evaluation of CraftDroid on real-world commercial Android apps corroborates its effectiveness by achieving 73% precision and 90% recall on average for transferring both the GUI events and oracles. In addition, 75% of the attempted transfers successfully generated valid and feature-based tests for popular features among apps in the same category.

Jun-Wei Lin

University of California, Irvine

United States

Reyhaneh Jabbarvand

University of California, Irvine

United States

Sam Malek

University of California, Irvine

United States

The use of mobile apps is increasingly widespread, and much effort is put into testing these apps to make sure they behave as intended. To reduce this effort, and thus the overall cost of mobile app testing, we propose AppTestMigrator, a technique for migrating test cases between apps in the same category (e.g., banking apps). The intuition behind AppTestMigrator is that many apps share similarities in their functionality, and these similarities often result in conceptually similar user interfaces (through which that functionality is accessed). AppTestMigrator leverages these commonalities between user interfaces to migrate existing tests written for an app to another similar app. Specifically, given (1) a test case for an app (source app) and (2) a second app (target app), AppTestMigrator attempts to automatically transform the sequence of events and oracles in the test for the source app to events and oracles for the target app. We implemented AppTestMigrator for Android mobile apps and evaluated it on a set of randomly selected apps from the Google Play Store in four different categories. Our initial results are promising, support our intuition that test migration is possible, and motivate further research in this direction.

Farnaz Behrang

Georgia Tech

Alessandro Orso

Georgia Tech

United States

Mobile push notification is widely used in mobile platforms to deliver all sorts of information to app users. Although it offers great convenience for both app developers and mobile users, this feature was recurrently reported to serve malicious and aggressive purposes, such as delivering annoying push notification advertisement. However, to the best of our knowledge, our research community has not touched the problem yet, neither providing techniques to detect/prevent them, nor characterizing this issue in the mobile app ecosystem at large-scale. This paper presents the first study to detect aggressive push notifications and further characterize them in large-scale. To this end, we first provide a taxonomy of mobile push notifications and pick out the aggressive ones using a crowdsourcing-based method. Then we propose DaPANDA, a novel hybrid approach, aiming at automatically detecting aggressive push notifications in Android apps. DaPANDA leverages a guided testing approach to systematically trigger and consume push notifications. By instrumenting the Android framework, DaPANDA further collects all the notification-relevant runtime information for flagging aggressive ones. Our experimental results show that DaPANDA is capable of detecting aggressive push notifications across the spectrum of aggressive types. By applying DaPANDA to 20,000 Android apps, it yields over 1,000 aggressive notifications that are further confirmed to be true positives and are shared with our community to promote advanced approaches for detecting aggressive mobile push notifications.

Tianming Liu

Beijing University of Posts and Telecommunications, China

China

Haoyu Wang

Beijing University of Posts and Telecommunications, China

China

Li Li

Monash University, Australia

Australia

Guangdong Bai

Griffith University

Yao Guo

Peking University

China

Guoai Xu

Beijing University of Posts and Telecommunications

The fragmentation issue spreads over multiple mobile platforms such as Android, iOS, mobile web and even WeChat, which hinders test scripts from running across platforms. To reduce the cost of adapting scripts for various platforms, some existing tools apply conventional computer vision techniques to replay the same script on multiple platforms automatically. However, because these solutions can hardly identify dynamic or similar widgets, it becomes difficult for engineers to apply them in practice. In this paper, we present an image-driven tool, namely LIRAT, to record and replay test scripts cross platforms, solving the problem of test script cross-platform replay for the first time. LIRAT records screenshots and layouts of the widgets, and leverages image understanding techniques to locate them in the replay process. Based on accurate widget localization, LIRAT supports replaying test scripts across devices and platforms. We employed LIRAT to replay 25 scripts from 5 application across 8 Android devices and 2 iOS devices. The results show that LIRAT can replay 88% scripts on Android platforms and 60% on iOS platforms.

Shengcheng Yu

Nanjing University, China

China

Chunrong Fang

Nanjing University

Yang Feng

University of California, Irvine

United States

Wenyuan Zhao

Nanjing University

Zhenyu Chen

Nanjing University

China

Automated input generators must constantly choose which UI element to interact with and how to interact with it, in order to achieve high coverage with a limited time budget. Currently, most black-box input generators adopt pseudo-random or brute-force searching strategies, which may take very long to find the correct combination of inputs that can drive the app into new and important states. We propose Humanoid, a deep learning-based approach to automated black-box Android app testing, which can explore the app more efficiently. The key technique behind Humanoid is a deep neural network model that can learn how human users choose actions based on an app’s GUI from human interaction traces . The learned model can be used to guide test input generation to achieve higher coverage. Experiments on both open-source apps and market apps demonstrate that Humanoid is able to reach higher coverage, and faster as well, than the state-of-the-art test input generators. Humanoid is open-sourced at https://github.com/yzygitzh/Humanoid and a demo video can be found at https://youtu.be/PDRxDrkyORs.

Yuanchun Li

Peking University

Ziyue Yang

Peking University

Yao Guo

Peking University

China

Xiangqun Chen

Peking University

Code coverage as the primitive dynamic program behavior information, is widely adopted to facilitate a rich spectrum of software engineering tasks, such as testing, fuzzing, debugging, fault detection, reverse engineering, and program understanding. Thanks to the widespread applications, it is crucial to ensure the reliability of the code coverage profilers.

Unfortunately, due to the lack of research attention and the existence of testing oracle problem, the coverage profilers are far away from being tested sufficiently. Bugs are still regularly seen in the widely deployed profilers, like gcov and llvm-cov, along with gcc and llvm, respectively.

This paper proposes Cod, a fully automated self-validator for effectively uncovering bugs in the coverage profilers. Cod takes a single profiler and a program (either from a compiler’s test suite or generated randomly) as input and uncovers the bugs by identifying the inconsistency of coverage results from the input program and its equivalent mutated variants whose coverage statistics are expected to be identical.

We evaluated Cod over two of the most well-known code coverage profilers, namely gcov and llvm-cov. Within a fourmonth testing period, a total of 196 potential bugs (123 for gcov, 73 for llvm-cov) are found, among which 23 are confirmed by the developers.

Yibiao Yang

Huazhong University of Science and Technology

China

Yanyan Jiang

Nanjing University

China

Zhiqiang Zuo

Nanjing University, China

China

Yang Wang

Nanjing University

China

Hao Sun

Unaffiliated

China

Hongmin Lu

Nanjing University

Yuming Zhou

Nanjing University

Baowen Xu

Nanjing University

Field-exhaustive testing is a testing criterion suitable for object-oriented code over complex, heap-allocated, data structures. It requires test suites to contain enough test inputs to cover all feasible values for object’s fields within a certain scope (input-size bound). While previous work shows that field-exhaustive suites can be automatically generated, the generation technique required a formal specification of the inputs that can be subject to SAT-based analysis. Moreover, the restriction of producing all feasible values for inputs’ fields makes test generation costly.

In this paper, we deal with field coverage as testing criteria that measure the quality of a test suite by examining to what extent the values of inputs’ fields are covered. In particular, we consider field coverage in combination with test generation based on symbolic execution to produce underapproximations of field-exhaustive suites, using the Symbolic Pathfinder tool. To underapproximate these suites we use tranScoping, a technique that estimates characteristics of yet to be run analyses for large scopes, based on data obtained from analyses performed in small scopes. This provides us with a suitable condition to prematurely stop the symbolic execution.

As we show, tranScoping different metrics regarding field coverage allows us to produce significantly smaller suites using a fraction of the generation time. All this while retaining the effectiveness of field exhaustive suites in terms of test suite quality.

Ariel Godio

Dept. of Software Engineering Instituto Tecnológico de Buenos Aires

Valeria Bengolea

Dept. of Computer Science FCEFQyN, University of Rio Cuarto

Pablo Ponzio

Dept. of Computer Science FCEFQyN, University of Rio Cuarto

Nazareno Aguirre

Dept. of Computer Science FCEFQyN, University of Rio Cuarto

Argentina

Marcelo F. Frias

Dept. of Software Engineering Instituto Tecnológico de Buenos Aires

Test code is becoming more essential to the modern software development process. However, practitioners often pay inadequate attention to key aspects of test code quality, such as bug detectability, maintainability and speed. Existing tools also typically report a single test code quality measure, such as code coverage, rather than a diversified set of metrics. To measure and visualize quality of test code in a comprehensive fashion, we developed an integrated test code analysis tool called Phanta. In this show case, we posit that the enhancement of test code quality is key to modernizing software development, and show how Phanta’s techniques measure the quality using mutation analysis, test code clone detection, and so on. Further, we present an industrial case study where Phanta was applied to analyze test code in a real Fujitsu project, and share lessons learned from the case study.

Susumu Tokumoto

Fujitsu Laboratories Ltd.

Japan

Kuniharu Takayama

Fujitsu Laboratories Ltd.

We present TestCov, a tool for robust test-suite execution and test-coverage measurement on C programs. TestCov executes program tests in isolated containers to ensure system integrity and reliable resource control. The tool provides coverage statistics per test and for the whole test suite. TestCov uses the simple, XML-based exchange format for test-suite specifications that was established as standard by Test-Comp. TestCov has been successfully used in Test-Comp ’19 to execute almost 9 million tests on 1720 different programs. The source code of TestCov is released under the open-source license Apache 2.0 and available at https://gitlab.com/sosy-lab/software/test-suite-validator. A full artifact, including a demonstration video, is available at https://doi.org/10.5281/zenodo.3418726.

Dirk Beyer

LMU Munich

Germany

Thomas Lemberger

LMU Munich

Germany

Fuzzing is widely used for vulnerability detection. One of the challenges for an efficient fuzzing is covering code guarded by constraints such as the magic number and nested conditions. Recently, academia has partially addressed the challenge via whitebox methods. However, high-level constraints such as array sorts, virtual function invocations, and tree set queries are yet to be handled.

To meet this end, we present VisFuzz, an interactive tool for better understanding and intervening fuzzing process via real-time visualization. It extracts call graph and control flow graph from source code, maps each function and basic block to the line of source code and tracks real-time execution statistics with detail constraint contexts. With VisFuzz, test engineers first locate blocking constraints and then learn its semantic context, which helps to craft targeted inputs or update test drivers. Preliminary evaluations are conducted on four real-world programs in Google fuzzer-test-suite. Given additional 15 minutes to understand and intervene the state of fuzzing, the intervened fuzzing outperform the original pure AFL fuzzing, and the path coverage improvements range from 10.84% to 150.58%, equally fuzzed by for 12 hours.

Chijin Zhou

Tsinghua University

Mingzhe Wang

Tsinghua University

Jie Liang

Tsinghua University

Zhe Liu

Nanjing University of Aeronautics and Astronautics

Chengnian Sun

Waterloo University

Yu Jiang

Tsinghua University

In the past, researchers have developed a number of popular taint-analysis approaches, particularly in the context of Android applications. Numerous studies have shown that automated code analyses are adopted by developers only if they yield a good “signal to noise ratio”, i.e., high precision. Many previous studies have reported analysis precision quantitatively, but this gives little insight into what can and should be done to increase precision further.

To guide future research on increasing precision, we present a comprehensive study that evaluates static Android taint-analysis results on a qualitative level. To unravel the exact nature of taint flows, we have designed COVA, an analysis tool to compute partial path constraints that inform about the circumstances under which taint flows may actually occur in practice.

We have conducted a qualitative study on the taint flows in 1,022 real-world Android applications. Our results reveal several key findings: Many taint flows occur only under specific conditions, e.g., environment settings, user interaction, I/O. Taint analyses should consider the application context to discern such situations. COVA shows that few taint flows are guarded by multiple different kinds of conditions simultaneously, so tools that seek to confirm true positives dynamically can concentrate on one kind at a time, e.g., only simulating user interactions. Lastly, many false positives arise due to a too liberal source/sink configuration. Taint analyses must be more carefully configured, and their configuration could benefit from better tool assistance.

Linghui Luo

Paderborn University

Eric Bodden

Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM

Johannes Späth

Fraunhofer IEM

Germany

This paper proposes a solution for automated goal-driven exploration of Android applications – a scenario in which a user, e.g., security auditor, needs to dynamically trigger the functionality of interest in an application, e.g., to check whether user-sensitive info is only sent to recognized third-party servers. As the auditor might need to check hundreds or even thousands of apps, manually exploring each app to trigger the desired behavior is too time-consuming to be feasible. Existing automated application exploration and testing techniques are of limited help in this scenario as well, as their goal is mostly to identify faults by systematically exploring different app paths, rather than swiftly navigating to the target functionality.

The goal-driven application exploration approach proposed in this paper, called GoalExplorer, automatically generates an executable test script that directly triggers the functionality of interest. The core idea behind GoalExplorer is to first statically model the application UI screens and transitions between these screens, producing a Screen Transition Graph (STG). Then, GoalExplorer uses the STG to guide the dynamic exploration of the application to the particular target of interest: an Android activity, API call, or a program statement. The results of our empirical evaluation on 93 benchmark applications and 95 the most popular GooglePlay applications show that the STG is substantially more accurate than other Android app UI models and that \tool is able to trigger a target functionality much faster than existing application exploration and testing techniques.

Duling Lai

University of British Columbia

Julia Rubin

University of British Columbia

Canada

The ability to repeat the execution of a program is a fundamental requirement in many areas of computing from computer system evaluation to software engineering. Reproducing executions of mobile apps, in particular, has proven difficult under real-life scenarios due to multiple sources of external inputs and interactive nature of the apps. Previous works that provide record/replay functionality for mobile apps are restricted to particular input sources (e.g., touchscreen events) and present deployment challenges due to intrusive modifications to the underlying software stack. Moreover, due to their reliance on record and replay of device specific events, the recorded executions cannot be reliably reproduced across different platforms.

In this paper, we present a new practical approach, RandR, for record and replay of Android applications. RandR captures and replays multiple sources of input (i.e., UI and network) without requiring source code (OS or app), administrative device privileges, or any special platform support. RandR achieves these qualities by instrumenting a select set of methods at runtime within an application’s own sandbox. In addition, to enable portability of recorded executions across different platforms for replay, RandR contextualizes UI events as interactions with particular UI components (e.g., a button) as opposed to relying on platform specific features (e.g., screen coordinates). We demonstrate RandR’s accurate cross-platform record and replay capabilities using over 30 real-world Android apps across a variety of platforms including emulators as well as commercial off-the-shelf mobile devices deployed in real life.

Onur Sahin

Boston University

Assel Aliyeva

Boston University

Hariharan Mathavan

Boston University

Ayse Coskun

Boston University

Manuel Egele

Boston University, USA

United States

Malware scanning of an app market is expected to be scalable and effective. However, existing approaches use either syntax-based features which can be evaded by transformation attacks or semantic-based features which are usually extracted by performing expensive program analysis. Therefore, in this paper, we propose a lightweight graph-based approach to perform Android malware detection. Instead of traditional heavyweight static analysis, we treat function call graphs of apps as social networks and perform social-network-based centrality analysis to represent the semantic features of the graphs. Our key insight is that centrality provides a succinct and fault- tolerant representation of graph semantics, especially for graphs with certain amount of inaccurate information (e.g., inaccurate call graphs). We implement a prototype system, MalScan, and evaluate it on datasets of 15,285 benign samples and 15,430 malware samples. Experimental results show that MalScan is capable of detecting Android malware with up to 98% accuracy under one second which is more than 100 times faster than two state-of-the-art approaches, namely MaMaDroid and Drebin. We also demonstrate the feasibility of MalScan on market-wide malware scanning by performing a statistical study on over 3 million apps. Finally, in a corpus of dataset collected from Google-Play app market, MalScan is able to identify 18 zero-day malware including malware samples that can evade detection of existing tools.

Yueming Wu

Huazhong University of Science and Technology

China

Xiaodi Li

University of Texas at Dallas

Deqing Zou

Huazhong University of Science and Technology

Wei Yang

University of Texas at Dallas

United States

Xin Zhang

Huazhong University of Science and Technology

China

Hai Jin

Huazhong University of Science and Technology

Developers often look for solutions to programming problems in community Q&A sites like Stack Overflow. Due to the crowdsourcing nature of these Q&A sites, many userprovided answers are wrong, less optimal or out-of-date. Relying on community-curated quality indicators (e.g., accepted answer, answer vote) cannot reliably identify these answer problems. Such problematic answers are often criticized by other users. However, these critiques are not readily discoverable when reading the posts. In this paper, we consider the answers being criticized and their critique posts as controversial discussions in community Q&A sites. To help developers notice such controversial discussions and make more informed choices of appropriate solutions, we design an automatic open information extraction approach for systematically discovering and summarizing the controversies in Stack Overflow and exploiting official API documentation to assist the understanding of the discovered controversies.We apply our approach to millions of java/androidtagged Stack overflow questions and answers and discover a large scale of controversial discussions in Stack Overflow. Our manual evaluation confirms that the extracted controversy information is of high accuracy. A user study with 18 developers demonstrates the usefulness of our generated controversy summaries in helping developers avoid the controversial answers and choose more appropriate solutions to programming questions.

Xiaoxue Ren

Zhejiang University

Zhenchang Xing

Australia National University

Australia

Xin Xia

Monash University

Australia

Guoqiang Li

Shanghai Jiao Tong University

China

Jianling Sun

Zhejiang University

Previous studies showed that replying to a user review usually has a positive effect on the rating that is given by the user to the app. For example, Hassan et al. found that responding to a review increases the chances of a user updating their given rating by up to six times compared to not responding. To alleviate the labor burden in replying to the bulk of user reviews, developers usually adopt a template-based strategy where the templates can express appreciation for using the app or contain the company email for users to react. However, reading large numbers of user reviews every day is not an easy task for developers. The available review-response pairs provide us chances to learn the knowledge relations between reviews and responses.

Although there exists research on studying the popular review patterns (e.g., reviews with longer content and lower rating) that developers tend to respond, approaches to automate the review response process have never been proposed. Inspired by the RNN encoder-decoder model in the natural language processing field, we propose a response generation framework, named RRGen. RRGen explicitly incorporates review attributes, such as user rating and review length, and learns the relations between reviews and corresponding responses in a supervised way from the available training data. Experiments on 58 apps and 309,246 review-response pairs highlight that RRGen outperforms several baselines by 67.4% to 4.5 times in terms of BLEU (an accuracy measure that is widely used to evaluate generation systems). Qualitative analysis also confirms the effectiveness of RRGen in generating the relevant and accurate response.

Cuiyun Gao

Nanyang Technological University, Singapore

Singapore

Jichuan Zeng

The Chinese University of Hong Kong

Xin Xia

Monash University

Australia

David Lo

Singapore Management University

Singapore

Michael Lyu

The Chinese University of Hong Kong

Irwin King

The Chinese University of Hong Kong

ACM SIGSOFT Distinguished Paper Award

Enabled by the pull-based development model, developers can easily contribute to a project through pull requests (PRs). When creating a PR, developers can add a free-form description to describe what changes made in this PR and/or why. Such a description is helpful for reviewers and other developers to gain a quick understanding of the PR without touching the details and may reduce the possibility of the PR being ignored or rejected. However, developers sometimes neglect to write descriptions for PRs. For example, in our collected dataset with over 333K PRs, more than 34% of the PR descriptions are empty. To alleviate this problem, we propose an approach to automatically generate PR descriptions based on the commit messages and the source code comments in the PRs. We regard this problem as a text summarization problem and solve it using a novel sequence-to-sequence model. To cope with out-of-vocabulary words in software artifacts and bridge the gap between the training loss function of the sequence-to-sequence model and the evaluation metric ROUGE, which has been shown to correspond to human evaluation, we integrate the pointer generator and directly optimize for ROUGE using reinforcement learning and a special loss function. We build a dataset with over 41K PRs and evaluate our approach on this dataset through ROUGE and a human evaluation. Our evaluation results show that our approach outperforms two baselines by significant margins.

Zhongxin Liu

Zhejiang University

China

Xin Xia

Monash University

Australia

Christoph Treude

The University of Adelaide

Australia

David Lo

Singapore Management University

Singapore

Shanping Li

Zhejiang University

Evidence shows that developer reputation is extremely important when accepting pull requests or resolving reported issues. It is particularly salient in Free/Libre Open Source Software since the developers are distributed around the world, do not work for the same organization and, in most cases, never meet face to face. The existing solutions to expose developer reputation tend to be forge specific (GitHub), focus on activity instead of impact, do not leverage social or technical networks, and do not correct often misspelled developer identities. We aim to remedy this by amalgamating data from all public Git repositories, measuring the impact of developer work, expose developer’s collaborators, and correct notoriously problematic developer identity data. We leverage World of Code (WoC), a collection of an almost complete (and continuously updated) set of Git repositories by first allowing developers to select which of the 34 million(M) Git commit author IDs belong to them and then generating their profiles by treating the selected collection of IDs as that single developer. As a side-effect, these selections serve as a training set for a supervised learning algorithm that merges multiple identity strings belonging to a single individual. As we evaluate the tool and the proposed impact measure, we expect to build on these findings to develop reputation badges that could be associated with pull requests and commits so developers could easier trust and prioritize them.

Sadika Amreen

University of Tennessee Knoxville

Andrey Karnauch

University of Tennessee Knoxville

Audris Mockus

University of Tennessee - Knoxville

United States

Coding convention plays an important role in guaranteeing software quality. However, coding conventions are usually informally presented and inconvenient for programmers to use. In this paper, we present CocoQa, a system that answers programmer’s questions about coding conventions. CocoQa answers questions by querying a knowledge graph for coding conventions. It employs 1) a subgraph matching algorithm that parses the question into a SPARQL query, and 2) a machine comprehension algorithm that uses an end-to-end neural network to detect answers from searched paragraphs. We have imple- mented CocoQa, and evaluated it on a coding convention QA dataset. The results show that CocoQa can answer questions about coding conventions precisely. In particular, CocoQa can achieve a precision of 82.92% and a recall of 91.10%. Repository: https://github.com/14dtj/CocoQa/ Video: https://youtu.be/VQaXi1WydAU

Tianjiao Du

Shanghai JiaoTong University

Junming Cao

Shanghai JiaoTong University

Qinyue Wu

Shanghai JiaoTong University

Wei Li

Shanghai JiaoTong University

Beijun Shen

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University

Yuting Chen

Shanghai Jiao Tong University

China

Emotion awareness research in SE context has been growing in recent years. Currently, researchers often rely on textual communication records to extract emotion states using natural language processing techniques. However how well these extracted emotion states reflect people’s real emotions has not been thoroughly investigated. In this paper, we report a multi-level, longitudinal empirical study with 82 individual members in 27 project teams. We collected their self-reported retrospective emotion states on a weekly basis during their year-long projects and also extract corresponding emotions from the textual communication records. We then model and compare the dynamics of these two types of emotions using multiple statistical and time series analysis methods. Our analyses yield a rich set of findings. The most important one is that the dynamics of emotions extracted using text-based algorithms often do not well reflect the dynamics of self-reported retrospective emotions. Besides, the extracted emotions match self-reported retrospective emotions better at the team level. Our results also suggest that individual personalities and team’s emotion display norms significantly impact the match/mismatch. Our results should warn the research community about the limitations and challenges of applying text-based emotion recognition tools in SE research.

Yi Wang

Shenzhen University

Developers often reuse code snippets from online forums, such as Stack Overflow, GitHub Gists to learn API usages of software frameworks or libraries. Those code snippets often have ambiguous undeclared external references. This makes it difficult to learn and use those APIs correctly. Reusing those code snippets to solve development tasks also requires resolving external references of those APIs. However, manually resolving fully qualified names (FQN) of API elements is a non-trivial task. In this paper, we propose a novel context-sensitive technique, COSTER, to resolve FQNs of API elements in those code snippets. The technique collects locally specific source code elements as well as globally related tokens as the context of FQNs, calculate association score, and build an occurrence likelihood dictionary. While inferring an API element, it collects the code context and ranks candidate FQNs from the dictionary by considering the association score of the tokens in the context, similarity between the context, and similarity between the API element. Evaluation with code examples collected from GitHub and Stack Overflow posts shows that our proposed technique improves precision and recall by 3-18% compared to existing state-of-the-art techniques. The proposed technique significantly reduces the training time compared to the StatType, a state-of-the-art technique, without sacrificing accuracy. Extensive analyses on results establish the facts of the robustness of the proposed technique.

C M Khaled Saifullah

Department of Computer Science, University of Saskatchewan

Canada

Muhammad Asaduzzaman

Postdoctoral Research Fellow, Software Analysis and Intelligence Lab, Queen's University, Canada

Chanchal K. Roy

University of Saskatchewan

Canada

Inferring program transformations from concrete program changes has many potential uses, such as applying systematic program edits, refactoring, and automated program repair. Existing work for inferring program transformations usually rely on statistical information over a potentially large set of program-change examples. However, in many practical scenarios we do not have such a large set of program-change examples.

In this paper, we address the challenge of inferring a program transformation from one single example. Our core insight is that “big code” can provide effective guide for the generalization of a concrete change into a program transformation, i.e., code elements appearing in many files are general and should not be abstracted away. We first propose a framework for transformation inference, where programs are represented as hypergraphs to enable fine-grained generalization of transformations. We then design a transformation inference approach, GENPAT, that infers a program transformation based on code context and statistics from a big code corpus.

We have evaluated GENPAT under two distinct application scenarios, systematic editing and program repair. The evaluation on systematic editing shows that GENPAT significantly outperforms a state-of-the-art approach, SYDIT, with up to 5.5x correctly transformed cases. The evaluation on program repair suggests that GENPAT has the potential to be integrated in advanced program repair tools—GENPAT successfully repaired 19 real-world bugs in the Defects4J benchmark by simply applying transformations inferred from existing patches, where 4 bugs have never been repaired by any existing technique. Overall, the evaluation results suggest that GENPAT is effective for transformation inference and can potentially be adopted for many different applications.

Link to Publication: https://github.com/xgdsmileboy/GenPat

Jiajun Jiang

Peking University

China

Luyao Ren

Peking University

China

Yingfei Xiong

Peking University

China

Lingming Zhang

The University of Texas at Dallas

ACM SIGSOFT Distinguished Paper Award

The IFDS algorithm can be compute- and memory-intensive for some large programs, often running for a long time(more than expected) or terminating prematurely after some time and/or memory budgets have been exhausted. In the latter case, the corresponding IFDS data-flow analyses may suffer from false negatives and/or false positives. To improve this, we introduce a sparse alternative to the traditional IFDS algorithm. Instead of propagating the data-flow facts across all the program points along the program’s (interprocedural) control flow graph, we propagate every data-flow fact directly to its next possible use points along its own sparse control flow graph constructed on the fly, thus reducing significantly both the time and memory requirements incurred by the traditional IFDS algorithm. In our evaluation, we compare FLOWDROID, a taint analysis performed by using the traditional IFDS algorithm, with our sparse incarnation, SPARSEDROID, on a set of 40 Android apps selected. For the time budget (5 hours) and memory budget(220GB) allocated per app, SPARSEDROID can run every app to completion but FLOWDROID terminates prematurely for 9apps, resulting in an average speedup of 22.0x. This implies that when used as a market-level vetting tool, SPARSEDROID can finish analyzing these 40 apps in 2.13 hours (by issuing 228 leak warnings) while FLOWDROID manages to analyze only 30 apps in the same time period (by issuing only 147 leak warnings).

Dongjie He

University of New South Wales; Institute of Computing Technology, CAS; University of Chinese Academy of Sciences

Haofeng Li

Institute of Computing Technology, CAS; University of Chinese Academy of Sciences

Lei Wang

Institute of Computing Technology, Chinese Academy of Science

Haining Meng

Institute of Computing Technology, CAS; University of Chinese Academy of Sciences

Hengjie Zheng

Institute of Computing Technology, CAS; University of Chinese Academy of Sciences

Jie Liu

University of New South Wales

Shuangwei Hu

vivo AI Lab

Lian Li

Institute of Computing Technology at Chinese Academy of Sciences, China

Jingling Xue

UNSW Sydney

Australia

In the app releasing process, Android requires all apps to be digitally signed with a certificate before distribution. Android uses this certificate to identify the author and ensure the integrity of an app. However, a number of signature issues have been reported recently, threatening the security and privacy of Android apps. In this paper, we present the first large-scale systematic measurement study on issues related to Android app signatures. We first create a taxonomy covering four types of app signing issues (21 anti-patterns in total), including vulnerabilities, potential attacks, release bugs and compatibility issues. Then we developed an automated tool to characterize signature-related issues in over 5 million app items (3 million distinct apks) crawled from Google Play and 24 alternative Android app markets. Our empirical findings suggested that although Google has introduced apk-level signing schemes (V2 and V3) to overcome some of the known security issues, more than 93% of the apps still use only the JAR signing schemes (V1), which poses great security threats. Besides, we also revealed that 7% to 45% of the apps in the 25 studied markets have been found containing at least one signing issue, while a large number of apps have been exposed to security vulnerabilities, attacks and compatibility issues. Among them a considerable number of apps we identified are popular apps with millions of downloads. Finally, our evolution analysis suggested that most of the issues were not mitigated after considerable amounts of time across markets. The results demonstrate the emergency for detecting and repairing app signing issues.

Haoyu Wang

Beijing University of Posts and Telecommunications, China

China

Hongxuan Liu

Peking University

Xusheng Xiao

Case Western Reserve University

United States

Guozhu Meng

Institute of Information Engineering, Chinese Academy of Sciences

China

Yao Guo

Peking University

China

Mobile developers use OAuth APIs to implement Single-Sign-On services. However, the OAuth protocol was originally designed for the authorization for third-party websites not to authenticate users in third-party mobile apps. As a result, it is challenging for developers to correctly implement mobile OAuth securely. These vulnerabilities due to the misunderstanding of OAuth and inexperience of developers could lead to data leakage and account breach. In this paper, we perform an empirical study on the usage of OAuth APIs in Android applications and their security implications. In particular, we develop OAUTHLINT, that incorporates a query-driven static analysis to automatically check programs on the Google Play marketplace. OAUTHLINT takes as input an anti-protocol that encodes a vulnerable pattern extracted from the OAuth specifications and a program P. Our tool then generates a counter-example if the anti-protocol can match a trace of P’s possible executions. To evaluate the effectiveness of our approach, we perform a systematic study on 600+ popular apps which have more than 10 millions of downloads. The evaluation shows that 101 (32%) out of 316 applications that use OAuth APIs make at least one security mistake.

Tamjid Al Rahat

University of Virginia

United States

Yu Feng

University of California, Santa Barbara

China

Yuan Tian

University of Virginia

United States

To detect specific types of bugs and vulnerabilities, static analysis tools must be correctly configured with security-relevant methods (SRM), e.g., sources, sinks, sanitizers and authentication methods—usually a very labour-intensive and error-prone process. This work presents the semi-automated tool SWAN_ASSIST, which aids the configuration with an IntelliJ plugin based on active machine learning. It integrates our novel automated machine-learning approach SWAN, which identifies and classifies Java SRM. SWAN_ASSIST further integrates user feedback through iterative learning. SWAN_ASSIST aids developers by asking them to classify at each point in time exactly those methods whose classification best impact the classification result. Our experiments show that SWAN_ASSIST classifies SRM with a high precision, and requires a relatively low effort from the user. A video demo of SWAN_ASSIST can be found at https://youtu.be/fSyD3V6EQOY. The source code is available at https://github.com/secure-software-engineering/swan.

Goran Piskachev

Fraunhofer IEM

Germany

Lisa Nguyen Quang Do

Google

Switzerland

Oshando Johnson

Fraunhofer IEM

Eric Bodden

Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM

This paper presents Sip4J, a fully automated, scalable and effective tool to automatically generate access permission contracts for a sequential Java program. The access permission contracts, which represent the dependency of code blocks, have been frequently used to enable concurrent execution of sequential programs. Those permission contracts, unfortunately, need to be manually created by programmers, which is known to be time- consuming, laborious and error-prone. To mitigate those manual efforts, Sip4J performs inter-procedural static analysis of Java source code to automatically extract the implicit dependencies in the program and subsequently leverages them to automatically generate access permission contracts, following the Design by Contract principle. The inferred specifications are then used to identify the concurrent (immutable) methods in the program. Experimental results further show that Sip4J is useful and effective towards generating access permission contracts for sequential Java programs.

Ayesha Sadiq

Monash University

Li Li

Monash University, Australia

Australia

Yuan-Fang Li

Monash University

Australia

Ijaz Ahmed

University of Lahore

Sea Ling

Monash University

ACM SIGSOFT Distinguished Paper Award

Compilers, like other software systems, contain bugs, and compiler testing is the most widely-used way to assure compiler quality. A critical task of compiler testing is to generate test programs that could effectively and efficiently discover bugs. Though we can configure test generators such as Csmith to control the features of the generated programs, it is not clear what test configuration is effective. In particular, an effective test configuration needs to generate test programs that are bug-revealing, i.e., likely to trigger bugs, and diverse, i.e., able to discover different types of bugs. It is not easy to satisfy both properties. In this paper, we propose a novel test-program generation approach, called HiCOND, which utilizes historical data for configuration diversification to solve this challenge. HiCOND first infers the range for each option in a test configuration where bug-revealing test programs are more likely to be generated based on historical data. Then, it identifies a set of test configurations that can lead to diverse test programs through a search method (particle swarm optimization). Finally, based on the set of test configurations for compiler testing, HiCOND generates test programs, which are likely to be bug-revealing and diverse. We have conducted experiments on two popular compilers GCC and LLVM, and the results confirm the effectiveness of our approach. For example, HiCOND detects 75.00%, 133.33%, and 145.00% more bugs than the three existing approaches, respectively. Moreover, HiCOND has been successfully applied to actual compiler testing in a global IT company and detected 11 bugs during the practical evaluation.

Junjie Chen

Tianjin University

China

Guancheng Wang

Peking University

Dan Hao

Peking University

China

Yingfei Xiong

Peking University

China

Hongyu Zhang

The University of Newcastle

Australia

Lu Zhang

Peking University

China

We present TEGCER, an automated feedback tool for novice programmers. TEGCER uses supervised classification to match compilation errors in new code submissions with relevant pre-existing errors, submitted by other students before. The dense neural network used to perform this classification task is trained on 15,000+ error-repair code examples. The proposed model yields a test set classification Pred@3 accuracy of 97.7% across 212 error category labels. Using this model as its base, TEGCER presents students with the closest relevant examples of solutions for their specific error on demand. A large scale (N>230) usability study shows that students who use TEGCER are able to resolve errors more than 25% faster on average than students being assisted by human tutors.

Link to Publication: https://github.com/umairzahmed/tegcer

Umair Z. Ahmed

National University of Singapore

Singapore

Renuka Sindhgatta

Queensland University of Technology, Australia

Nisheeth Srivastava

Indian Institute of Technology, Kanpur

Amey Karkare

IIT Kanpur

India

Analyzing executions of concurrent software is very difficult. Even if a trace is available, such traces are very hard to read and interpret. A textual trace contains a lot of data, most of which is not relevant to the issue at hand. Past visualization attempts either do not show concurrent behavior, or result in a view that is overwhelming for the user.

We provide a visual analytics tool, VA4JVM, for error traces produced by either the Java Virtual Machine, or by Java Pathfinder. Its key features are a layout that spatially associates events with threads, a zoom function, and the ability to filter event data in various ways. We show in examples how filtering and zooming in can highlight a problem without having to read lengthy textual data.

Cyrille Artho

KTH Royal Institute of Technology, Sweden

Monali Pande

KTH Royal Institute of Technology

Qiyi Tang

University of Oxford

United Kingdom

Deep Neural Network(DNN) techniques have been prevalent in software engineering. They are employed to facilitate various software engineering tasks and embedded into many software applications. However, because DNNs are built upon a rich data-driven programming paradigm that employs plenty of labeled data to train a set of neurons to construct the internal system logic, analyzing and understanding their behaviors becomes a difficult task for software engineers. In this paper, we present an instance-based visualization tool for DNN, namely NeuralVis, to support software engineers in visualizing and interpreting deep learning models. NeuralVis is designed for: 1). visualizing the structure of DNN models, i.e., neurons, layers, as well as connections; 2). visualizing the data transformation process; 3). integrating existing adversarial attack algorithms for test input generation; 4). comparing intermediate outputs of different instances. To demonstrate the effectiveness of NeuralVis, we design a task-based user study involving ten participants on two classic DNN models, i.e., LeNet and VGG-12. The result shows NeuralVis can assist engineers in identifying critical features that determine the prediction results. Tool is available at http://118.178.18.181:18080. Video: https://youtu.be/hRxCovrOZFI.