Present to Future: Forging the Future of Quantum Software Testing
Currently, gate-based quantum computers are the most popular ones developed by prominent companies such as IBM, Google, and IQM. These quantum computers are programmed using quantum circuits. Ensuring the correctness of these quantum circuits, i.e., they perform their intended computations, is currently the focus of quantum software testing. First, this keynote will focus on the latest developments in quantum software testing, such as coverage criteria, search-based testing, mutation testing, and metamorphic testing. Next, the keynote will discuss some key future research directions in quantum software testing that deserve the community’s attention.
Evaluating the Effectiveness of Neuroevolution for Automated GUI-Based Software Testing
As software systems become increasingly complex, testing has become an essential component of the development process to ensure the quality of the final product. However, manual testing can be costly and time-consuming due to the need for human intervention. This constrains the number of test cases that can be run within a given timeframe and, as a result, limits the ability to detect defects in software in a timely manner. Automated testing, on the other hand, can reduce the cost and time associated with testing, but traditional approaches have limitations. These include the inability to thoroughly explore the entire state space of software or process the high-dimensional input space of graphical user interfaces (GUIs). In this study, we propose a new approach for automated GUI-based software testing utilizing neuroevolution (NE), a branch of machine learning that employs evolutionary algorithms to train artificial neural networks with multiple hidden layers of neurons. NE offers a scalable alternative to established deep reinforcement learning methods and provides higher robustness to parameter influences and improved handling of sparse rewards. The agents are trained to explore software and identify errors while being rewarded for high test coverage. We evaluate our approach using a realistic benchmark software application and compare it to monkey testing, a widely adopted automated software testing method.
Automated Android testing approaches often fail to interact properly with complex UIs consisting of multiple related elements. For instance, to trigger a state transition in a form-based UI, one has to first fill out all input fields and then click on the submit button, but test generators would usually interact with the fields and button in arbitrary order, struggling to trigger the corresponding state transition, and resulting in overall lower code coverage. One way to overcome this problem is to define motif actions, which allow test generators to interact not just with individual UI elements, but with combinations of UI elements related through common patterns of interaction sequences. We designed 12 such motif actions for common scenarios and integrated them into the Android test generation tool MATE. Our experiments demonstrate that these motif actions are applicable to a wide range of apps (86.5% out of a sample of 551 apps). Motif actions are particularly useful on complex apps, where our experiments on 109 such apps demonstrate an average increase of 2.19% activity coverage and 2% line coverage.
Test Case Recommendations with Distributed Representation of Code Syntactic Features
Recorded talk
Frequent modifications of unit test cases are inevitable due to software’s continuous underlying changes in source code, design, and requirements. Since manually maintaining software test suites is tedious, timely, and costly, automating the process of generation and maintenance of test units will significantly impact the effectiveness and efficiency of software testing processes.
To this end, we propose an automated approach which exploits both structural and semantic properties of source code methods and test cases to recommend the most relevant and useful unit tests to the developers. The proposed approach initially trains a neural network to transform method-level source code, as well as unit tests, into distributed representations (embedded vectors) while preserving the importance of the structure in the code. Retrieving the semantic and structural properties of a given method, the approach computes cosine similarity between the method’s embedding and the previously-embedded training instances. Further, according to the similarity scores between the embedding vectors, the model identifies the closest methods of embedding and the associated unit tests as the most similar recommendations.
The results on the Methods2Test dataset showed that, while there is no guarantee to have similar relevant test cases for the group of similar methods, the proposed approach extracts the most similar existing test cases for a given method in the dataset, and evaluations show that recommended test cases decrease the developers’ effort to generating expected test cases.
Continuous Domain Input Abstraction and Fault Detection Capability in Combinatorial Testing
Combinatorial Testing (CT) is a fundamental approach for testing varying applications ranging from mobile applications to autonomous driving. Input modeling is a prerequisite to CT. When the input domain is continuous, abstraction is necessary to generate a CT test suite. In this study, we discuss three novel, work-in-progress research avenues. With these avenues, we aim to maximize the fault detection capability of CT test suites through continuous domain input abstraction optimization. We motivate our research on abstraction by demonstrating how it affects the fault detection capability of CT test suites on three audio plugin implementations. Overall, continuous domain input abstraction optimization remains a promising open challenge.
GUI-Based Software Testing: An Automated Approach Using GPT-4 and Selenium WebDriver
This paper presents a novel method for GUI testing in web applications that largely automates the process by integrating the advanced language model GPT-4 with Selenium, a popular web application testing framework. Unlike traditional deep learning approaches, which require extensive training data, GPT-4 is pre-trained on a large corpus, giving it significant generalisation and inference capabilities. These capabilities allow testing without the need for recorded data from human testers, significantly reducing the time and effort required for the testing process. We also compare the efficiency of our integrated GPT-4 approach with monkey testing, a widely used technique for automated GUI testing where user input is randomly generated. To evaluate our approach, we implemented a web calculator with an integrated code coverage system. The results show that our integrated GPT-4 approach provides significantly better branch coverage compared to monkey testing. These results highlight the significant potential of integrating specific AI models such as GPT-4 and automated testing tools to improve the accuracy and efficiency of GUI testing in web applications.
Chouette: An Automated Cross-Platform UI Crawler for Improving App Quality
The Chouette App Crawler is a cross-platform GUI crawler used at Duolingo to help accelerate the mobile and web app testing cycle. In this talk we will discuss the tool and its results from an industry perspective rather than an academic one as well as tease some future work.
