The purpose of this study is to identify the characteristics of agile development processes that impact user satisfaction. We used user reviews of OSS smartphone apps and various data from version control systems to examine the relationships, especially time-series correlations, between user satisfaction and development metrics that are expected to be related to user satisfaction. Although no metrics conclusively indicate an improved user satisfaction, motivation of the development team, the ability to set appropriate work units, the appropriateness of work rules, and the improvement of code maintainability should be considered as they are correlated with improved user satisfaction. In contrast, changes in the release frequency and workload are not correlated.
Sofia Reis Instituto Superior Técnico, U. Lisboa & INESC-ID, Rui Abreu Faculty of Engineering, University of Porto, Corina S. Pasareanu Carnegie Mellon University Silicon Valley, NASA Ames Research Center
An Empirical Study on Continuous Integration Trends, Topics and Challenges in Stack Overflow
During the last few years, Continuous Integration (CI) has become a common practice in open source and industrial environments in order to reduce the scope for errors and increase the speed to market through automated build and test processes. However, despite this wide adoption throughout the years, little is known about the challenges developers discuss. Analyzing the discussions of developers is required to understand what researchers, educators and practitioners should focus on, and how discussion communities can be helpful to shed the light on CI challenges. In this study, we examine Stack Overflow (SO), the most popular crowd-sourced forum, to understand the challenges developers face under CI context. We collect a corpus of 27,728 CI related developers posts from SO and analyze those posts through a mixed-method with quantitative and qualitative analyzes. To study the trends of CI discussions, we investigated the metadata of CI questions, users and tags. Then, we extract the CI main topics using Latent Dirichlet Allocation (LDA) tuned with Genetic Algorithm (GA). Finally, we investigate the most popular and difficult topics faced by developers and perform a qualitative analysis based on a statistical sample of unanswered questions to get further insights into CI challenges. The LDA clustering reveals that developers face challenges with six main topics namely Build, Testing, Version Control, Configuration, Deployment and CI Culture. Particularly, we found that the build topic is the most popular among the studied topics and that version control and testing topics are the most difficult for SO community. Our study uncovers insights about CI challenges and adds evidence to existing knowledge about CI issues related especially to software build. Based on the results of our study, we conclude several implications for researchers, e.g., need for more effort to investigate the reasons behind the reported issues, educators, e.g., teach CI principals and philosophy, and practitioners, e.g., take the difficult topics into consideration when distributing the tasks.
Are security commit messages informative? Not enough!
Industry Experience Report
The fast distribution and deployment of security patches is important to protect users against cyberattacks. These fixes can be detected automatically by patch management triage systems. However, previous work has shown that automating the task is not easy, in some cases, because of poor documentation or lack of information in security fixes. For many years, standard practices in the security community have steered engineers to provide cryptic commit messages—i.e., patch software vulnerabilities silently—to avoid potential attacks and reputation damages. However, not providing enough documentation on vulnerability fixes is known to damage trust between vendors and users. Current efforts in the security community aim to increase the level of transparency during patch and disclosing times to help build trust in the development community and make patch management processes faster. In this paper, we evaluate how informative security commit messages (i.e., messages attached to security fixes) are and how different levels of information can affect the different tasks in automated patch triage systems. We observed that security engineers provide some levels of detail in security commit messages that can be leveraged to improve or enable one or two of the automated triage tasks but not all of them. In addition, results show that security commit messages need to be more informative—56.6% of the messages analyzed were documented poorly. Best practices to write informative and well-structured security commit messages (such as SECOM) should become a standard practice in the security community.
Analyzing the Resource Usage Overhead of Mobile App Development Frameworks
Mobile app development frameworks lower the effort to write and deploy apps across different execution platforms, e.g., mobile, web, and stand-alone PCs. At the same time, their use may limit native optimizations and impose overhead, increasing resource usage. In mobile devices, higher resource usage results in faster battery depletion, a significant disadvantage. In this paper, we analyze the resource usage of Android benchmarks and apps based on three mobile app development frameworks, Flutter, React Native, and Ionic, comparing them to functionally equivalent, native variants written in Java. These frameworks, besides being in widespread use, represent three different approaches for developing multiplatform apps: Flutter supports deployment of apps that are compiled and run fully natively, React Native runs interpreted JavaScript code combined with native views for different platforms, and Ionic is based on web apps, which means that it does not depend on platform-specific details. We measure the energy consumption, execution time, and memory usage of ten optimized, CPU-intensive benchmarks, to gauge overhead in a controlled manner, and two applications, to measure their impact when running commonly mobile app functionalities. Our results show that cross-platform and hybrid frameworks can be competitive in CPU-intensive applications. In five of the ten benchmarks, at least one framework-based version exhibits lower energy consumption and execution time than its native counterpart, up to a reduction of 81% in energy and 83% in execution time. Furthermore, in three other benchmarks, framework-based and native versions achieved similar results. Overall, Flutter, usually imposes the least overhead in execution time and energy, while React Native imposes the highest in all the benchmarks. However, in an app that continuously animates multiple images on the screen, without interaction, the React Native version uses the least CPU and energy, up to a reduction of 96% in energy compared to the second-best framework-based version. These findings highlight the importance of analyzing expected application behavior before committing to a specific framework.
Analysis of Bug Report Qualities with Fixing Time using a Bayesian Network
Short Paper
Most client software employs a bug-tracking system, which utilizes user-submitted reports (bug reports). that contain information necessary for software developers to fix bugs. The quality of bug reports drastically differs. Bug reports can include severity, priority, and associated issues determined by researching the addressed bug. Herein we investigate the influence of bug report qualities on successfully fixing a bug and estimating the fixing time. We also examine the claim in previous studies that bias and differences in the treatment of bug reports exist due to broad expertness among the reporters. Our approach examines the relationship between the qualities within the bug-fixing cycle and modeling graphical causal dependencies through a Bayesian Network. Bug reports with attachments, dependencies on another bug, and frequent discussions have a higher probability of being fixed. In addition, bug reports with a high severity tend to be fixed faster. Moreover, the difficulty of the bug itself may influence the fixing rate such that a straightforward bug will be fixed easier and faster regardless of the bug report quality.
Outside the Sandbox: A Study of Input/Output Methods in Java
Short Paper
Programming languages often demarcate the internal sandbox, consisting of entities such as objects and variables, from the outside world, e.g., files or network. Although communication with the external world poses fundamental challenges for live programming, reversible debugging, testing, and program analysis in general, studies about this phenomenon are rare. In this paper, we present a preliminary empirical study about the prevalence of input/output (I/O) method usage in Java. We manually categorized 1435 native methods in a Java Standard Edition distribution into non-I/O and I/O-related methods, which were further classified into areas such as desktop or file-related ones. According to the static analysis of a call graph for 798 projects, about 57% of methods potentially call I/O natives. The results of dynamic analysis on 16 benchmarks showed that 21% of the executed methods directly or indirectly called an I/O native. We conclude that neglecting I/O is not a viable option for tool designers and suggest the integration of I/O-related metadata with source code to facilitate their querying.
Automated unit test generation has been extensively studied, with prior research mostly focusing on dynamically compiled or dynamically typed programming languages like Java and Python. However, Go, a popular statically compiled and typed programming language used extensively in server application development, has received limited support from existing tools. To address this gap, we present NxtUnit, an automatic unit test generation tool for Go that uses random testing and is well-suited for microservice architecture. NxtUnit employs a random approach to generate unit tests quickly, making it ideal for smoke testing and providing quick quality feedback. It comes with three types of interfaces: an integrated development environment (IDE) plugin, a command-line interface (CLI) tool, and a browser-based platform. The plugin and CLI tool allow engineers to write unit tests more efficiently, while the platform provides unit test visualization and asynchronous unit test generation. We evaluated NxtUnit by generating unit tests for 13 public open-source repositories and 500 ByteDance in-house repositories, resulting in a code coverage of 20.74% for in-house repositories. We conducted a survey among Bytedance engineers and found that NxtUnit can save them 48% of the time they previously spent on writing unit tests. We have made the tool available at https://github.com/bytedance/nxt_unit.
Identifying Characteristics of the Agile Development Process That Impact User Satisfaction
Short Paper
The purpose of this study is to identify the characteristics of agile development processes that impact user satisfaction. We used user reviews of OSS smartphone apps and various data from version control systems to examine the relationships, especially time-series correlations, between user satisfaction and development metrics that are expected to be related to user satisfaction. Although no metrics conclusively indicate an improved user satisfaction, motivation of the development team, the ability to set appropriate work units, the appropriateness of work rules, and the improvement of code maintainability should be considered as they are correlated with improved user satisfaction. In contrast, changes in the release frequency and workload are not correlated.
