The performance of a software system plays a crucial role for user perception. Learning from the history of a software system’s performance behavior does not only help discovering and locating performance bugs, but also identifying evolutionary performance patterns and general trends, such as when technical debt accumulates in a slow but steady performance degradation. Exhaustive regression testing is usually impractical, because rigorous performance benchmarking requires executing a realistic workload per commit, which results in large execution times. In this paper, we propose a novel active revision sampling approach, which aims at tracking and understanding a system’s performance history by approximating the performance behavior of a software system across all of its revisions. In a nutshell, we iteratively sample and measure the performance of specific revisions that help us in building an exact performance- evolution model, and we use Gaussian Process models to assess in which revision ranges our model is most uncertain with the goal to to sample further revisions for measurement. We have conducted an empirical analysis of the evolutionary performance behavior modeled as a time series of the history of 6 real-world software systems. Our evaluation demonstrates that Gaussian Process models are able to accurately estimate the performance- evolution history of real-world software systems with only few measurements and to reveal interesting behaviors and trends.

Stefan Mühlbauer

Bauhaus-University Weimar

Germany

Sven Apel

Saarland University

Germany

Norbert Siegmund

Bauhaus-University Weimar

Germany

Modern web applications rely heavily on databases to query and update information. To ease the development efforts, Object Relational Mapping (ORM) frameworks provide an abstraction for developers to manage databases by writing in the same Object-Oriented programming languages. Prior studies have shown that there are various types of performance issues caused by inefficient accesses to databases via different ORM frameworks (e.g., Hibernate and ActiveRecord). However, it is not clear whether the reported performance anti-patterns (common performance issues) can be generalizable across various frameworks. In particular, there is no study focusing on detecting performance issues for applications written in PHP, which is the choice of programming languages for the majority (79%) of web applications. In this experience paper, we detail our process on conducting performance-aware refactoring of an industrial web application written in Laravel, the most popular web framework in PHP. We have derived a complete catalog of 17 performance anti-patterns based on prior research and our experimentation. We have found that some of the reported anti-patterns and refactoring techniques are framework or programming language specific, whereas others are general. The performance impact of the anti-pattern instances are highly dependent on the actual usage context (workload and database settings). When communicating the performance differences before and after refactoring, the results of the complex statistical analysis may be sometimes confusing. Instead, developers usually prefer more intuitive measures like percentage improvement. Experiments show that our refactoring techniques can achieve up to 1319% and 1417% times speedup for the industrial and the open source application under various scenarios.

Boyuan Chen

York University

Zhen Ming (Jack) Jiang

York University

Paul Matos

Copywell Inc.

Michael Lacaria

Copywell Inc.

Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while an extremely fine-grained load test would simply replay user actions step by step, leading to load tests that are costly to develop, execute and maintain. Workload recovery is core of these load tests. Prior research often captures the workload as the frequency of user actions. However, there exists much valuable information in the context and sequences of user actions. Such richer information would ensure that the load tests that leverage such workloads are more field-representative. In this experience paper, we study the use of different granularities of user behaviour, i.e., basic user actions, basic user actions with contextual information and user action sequences with contextual information, when recovering workloads for use in the load testing of large-scale systems. We propose three approaches that are based on the three granularities of user behaviour and evaluate our approaches on four subject systems, namely Apache James, OpenMRS, Google Borg, and an ultra-large-scale industrial system (SA) from CompanyA. Our results show that our approach that is based on user action sequences with contextual information outperforms the other two approaches and can generate more representative load tests with similar throughput and CPU usage to the original field workload (i.e., mostly statistically insignificant or with small/trivial effect sizes). Such representative load tests are generated only based on a small number of clusters of users, leading to a low cost of conducting/maintaining such tests. Finally, we demonstrate that our approaches can detect injected users in the original field workloads with high precision and recall. Our paper demonstrates the importance of user action sequences with contextual information in the workload recovery of large-scale systems.

Jinfu Chen

Jiangsu University

Weiyi (Ian) Shang

Concordia University, Canada

Ahmed E. Hassan

Queen's University

Canada

Yong Wang

Alibaba Group

Jiangbin Lin

Alibaba Group

As data volume and complexity grow at an unprecedented rate, the performance of data analytics programs is becoming a major concern for developers. We observed that developers sometimes use alternative data analytics APIs to improve program runtime performance while preserving functional equivalence. However, little is known on the characteristics and performance attributes of alternative data analytics APIs. In this paper, we propose a novel approach to extract alternative implementations that invoke different data analytics APIs to solve the same tasks. A key appeal of our approach is that it exploits the comparative structures in Stack Overflow discussions to discover programming alternatives. We show that our approach is promising, as 86% of the extracted code pairs were validated as true alternative implementations. In over 20% of these pairs, the faster implementation was reported to achieve a 10x or more speedup over its slower alternative. We hope that our study offers a new perspective of API recommendation and motivates future research on optimizing data analytics programs.

Yida Tao

Shenzhen University

Shan Tang

Shenzhen University

Yepang Liu

Southern University of Science and Technology

Zhiwu Xu

Shenzhen University

Shengchao Qin

University of Teesside

United Kingdom

The performance issues of apps can influence users’ satisfaction. Therefore, developers exploit application perfor- mance management (APM) tools to locate the potential perfor- mance bottleneck of their apps. Unfortunately, most developers do not understand how APMs monitor their apps during the runtime and whether these APMs have security risks (e.g., confidential data leakage). We demystify APMs by inspecting 25 widely-used APMs that target on Android apps. Currently, there is no systematic analysis of APMs in Android apps. In order to bridge this gap, we build a prototype tool, APMHunter, that can automatically detect the usages of APMs in Apps. We conduct a large-scale empirical study on 500,000 Android apps from Google Play to explore the usage patterns of APMs and discover the potential misuses of APMs. This study reveals our findings from two perspectives: 1) some APMs still employ deprecated permissions and approaches, which makes they cannot work as expected; 2) inappropriate APMs utilization can lead to privacy leakages. Thus, based on our research, we suggest that both APM vendors and developers should design and use APMs scrupulously.

Yutian Tang

The Hong Kong Polytechnic University

Zhan Xian

The Hong Kong Polytechnic University

Hao Zhou

The Hong Kong Polytechnic University

Xiapu Luo

The Hong Kong Polytechnic University

Zhou Xu

Wuhan University

Yajin Zhou

Zhejiang University

Qiben Yan

Michigan State University

We present PeASS (Performance Analysis of Software System versions), a tool for detecting performance changes at source code level that occur between different code versions. By using PeASS, it is possible to identify performance regressions that happened in the past to fix them.

PeASS measures the performance of unit tests in different source code versions. To achieve statistic rigor, measurements are repeated and analyzed using an agnostic t-test. To execute a minimal amount of tests, PeASS uses a regression test selection.

We evaluate PeASS on a selection of Apache Commons projects and show that 81% of all unit test covered performance changes can be found by PeASS. A video presentation is available at https://www.youtube.com/watch?v=RORFEGSCh6Y and PeASS can be downloaded from https://github.com/DaGeRe/peass.

David Georg Reichelt

Universität Leipzig

Germany

Stefan Kühne

Universität Leipzig

Wilhelm Hasselbring

Kiel University

Germany

Bug localization is well-known to be a difficult problem in software engineering, and specifically in compiler development, where it is beneficial to reduce the input program to a minimal reproducing example; this technique is more commonly known as delta debugging. What additionally contributes to the problem is that every new programming language has its own unique quirks and foibles, making it near impossible to reuse existing tools and approaches with full efficiency. In this experience paper we tackle the delta debugging problem w.r.t. Kotlin, a relatively new programming language from JetBrains. Our approach is based on a novel combination of program slicing, hierarchical delta debugging and Kotlin-specific transformations, which are synergistic to each other. We implemented it in a prototype called ReduKtor and did extensive evaluation on both synthetic and real Kotlin programs; we also compared its performance with classic delta debugging techniques. The evaluation results support the practical usability of our approach to Kotlin delta debugging and also shows the importance of using both language-agnostic and language-specific techniques to achieve best reduction efficiency and performance.

Daniil Stepanov

Saint Petersburg Polytechnic University

Marat Akhin

Saint Petersburg Polytechnic University / JetBrains Research

Russia

Mikhail Belyaev

Saint Petersburg Polytechnic University