Experience

Experience paper: Testing of mobile apps is time-consuming and requires a great deal of manual effort. For this reason, industry and academic researchers have proposed a number of test-input- generation techniques for automating app testing. Although useful, these techniques have weaknesses and limitations that often pre- vent them from achieving high coverage. We believe that one of the reasons for these limitations is that tool developers tend to focus mainly on improving the strategy the techniques employ to explore app behavior, whereas limited effort has been put into investigating other ways to improve the performance of these tech- niques. To address this problem, and get a better understanding of the limitations of input-generation techniques for mobile apps, we conducted an in-depth study of the limitations of Monkey–arguably the most widely used tool for automated testing of Android apps. Specifically, in our study, we manually analyzed Monkey’s perfor- mance on 68 benchmarks to identify the common limitations that prevent the tool from achieving better coverage results. We then assessed the coverage improvement that Monkey could achieve if these limitations were eliminated. In our analysis of the results, we also discuss whether other existing test-input-generation tools suffer from these common limitations and provide insights on how they could address them.

Farnaz Behrang

Georgia Tech

Alessandro Orso

Georgia Tech

United States

Automated testing of mobile apps has received significant attention in the recent years from researchers and practitioners alike. In this paper, we report on the largest empirical study to date, aimed at understanding the test automation culture prevalent among mobile app developers. We systematically examined more than 3.5 million repositories on GitHub and identified more than 12, 000 non-trivial and real-world Android apps. We then analyzed these non-trivial apps to investigate (1) the trends in adoption of test automation; (2) working habits of mobile app developers in regards to automated testing; and (3) the correlation between the adoption of test automation and the popularity of projects. Among others, we found that (1) only 8% of the mobile app development projects leverage automated testing practices; (2) developers tend to follow the same test automation practices across projects; and (3) popular projects, measured in terms of the number of contributors, stars, and forks on GitHub, are more likely to adopt test automation practices. To understand the rationale behind our observations, we further conducted a survey with 148 professional and experienced developers contributing to the subject apps. Our findings shed light on the current practices and future research directions pertaining to test automation for mobile app development.

Jun-Wei Lin

University of California, Irvine

United States

Navid Salehnamadi

University of California, Irvine

Sam Malek

University of California at Irvine, USA

United States

Most software systems interact with their environment extensively. This is especially true for mobile apps, whose behavior often depends on sensors, external services, and inter-process communications. These interactions can complicate testing activities, as test cases may need a complete environment to be executed. They can also cause issues such as flakiness, for example when the environment behaves in non-deterministic ways. For these reasons, it is common to create test mocks that can eliminate the need for (part of) the environment to be present during testing. Manual mock creation, however, can be extremely time consuming and error-prone. Moreover, the generated mocks can typically only be used in the context of the specific tests for which they were created. To address these issues, we propose MOKA, a general framework for collecting and generating reusable test mocks in an automated way. MOKA leverages the ability to observe a large number of interactions between an application and its environment and uses an iterative approach to generate mocks with different reusability characteristics—advanced mocks generated through program synthesis and basic record-replay-based mocks. In this paper, we describe the new ideas behind MOKA, its main characteristics, a preliminary study, and a set of possible applications that would benefit from our framework.

Mattia Fazzini

University of Minnesota

United States

Alessandra Gorla

IMDEA Software Institute

Spain

Alessandro Orso

Georgia Tech

United States

This paper presents AirMochi, a tool that provides remote access and control of apps by leveraging a mobile platform’s publicly exported \emph{accessibility features}. While AirMochi is designed to be platform-independent, we discuss its iOS implementation. We show that AirMochi places no restrictions on apps, is able to handle a variety of scenarios, and imposes a negligible performance overhead. https://youtu.be/rhPz2Hs4Ius https://github.com/nkllkc/air_mochi

Nikola Lukic

University of Southern California

United States

Saghar Talebipour

University of Southern California

Nenad Medvidović

University of Southern California, USA

United States