ITEQS 2020

We present a novel exploratory performance testing algorithm that uses supervised learning to optimize the test suite generation process. The goal of the proposed approach is to generate test suites that contain a large number of positive tests, revealing performance defects or other issues of interest in the system under test. The key idea is to use a deep neural network to predict which test could be positive and to train this network online during the test generation process, designing and executing the test suite simultaneously. The proposed algorithm assumes that the system under test is stateless and the outcome of the tests is deterministic. Also, only integer and floating point inputs are supported. Otherwise, the approach is completely automatic and it does not require any prior knowledge about the internals of the system under test. It can also be used effectively in a continuous integration setting where small variations of a system are tested successively. We evaluate our algorithm using two example problems: searching for bottlenecks in a web service and searching for efficient hardware configurations in a single-board computer. In both examples, the presented algorithm performed several times better than a random test generator and significantly better compared to our previously published algorithm, producing test suites with a large proportion of positive tests.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00030

Ivan Porres

Åbo Akademi University

Finland

Tanwir Ahmad

Åbo Akademi University

Finland

Hergys Rexha

Åbo Akademi University

Finland

Sebastien Lafond

Åbo Akademi University

Finland

Dragos Truscan

Åbo Akademi University

Finland

Performance is one example of the Extra-Functional Properties (EFPs), an essential part of software development with limited resources. In such a context of software development, containers represent a widely used abstraction technology that requires fewer resources, when compared to hypervisor-based virtualization. Containers provide access to environments and speed up development by abstracting the underlying infrastructure layer for different application domains. Volume sharing is a usual abstraction provided by containers for data persistence, which can be accomplished, especially with Network File System (NFS) or Docker’s sharing method. Evaluate the performance of these volumes in containers is critical to software development; however, we did not find available in the literature such evaluation. This paper describes the volume sharing performance for containers using NFS and Docker, which is essential for data persistence. We evaluated the performance of these shared volumes by simulating three application domains: database accesses from a benchmark, direct reads and writes in the file systems and concurrent accesses from an MPI parallel program. Our results show significant differences among the distinct volume sharing methods analyzed for Docker and NFS. While asynchronous NFS has better performances for the set composed of heavy workloads, bursting read and write sequences, the Docker volumes, by the other hand, outperformed for processes with concurrent accesses when using a parallel program.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00031

Naylor Garcia

University of São Paulo - USP

Brazil

Paulo Sergio Lopes de Souza

University of São Paulo - USP

Brazil

Sarita M. Bruschi

University of São Paulo - USP

Brazil

Simone do Rocio Senger de Souza

University of São Paulo - USP

Brazil

Recently, a hazard analysis technique STAMP/STPA has been widely accepted since it is recognized as being suitable for software-intensive systems. Using STAMP/STPA, we can find hazardous scenarios of the target system that cannot be obtained by other traditional hazard analysis methods and those scenarios can be used for validation testing. However, generally the number of obtained scenarios can be huge and the validation testing involves a considerable cost. In this study, we propose a method to prioritize hazardous scenarios identified by STAMP/STPA with the help of a statistical model-checking technique. We give a procedure for systematically transforming the model defined by STAMP/STPA to a formal model for a statistical model-checking tool. We also show the usefulness of the proposed method using an example of train gate control system.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00032

Mitsuaki Tsuji

Nara Institute of Science and Technology

Japan

Toshinori Takai

Nara Institute of Science and Technology

Japan

Kazuki Kakimoto

Nara Institute of Science and Technology

Japan

Naoki Ishihama

Japan Aerospace Exploration Agency

Japan

Masafumi Katahira

Japan Aerospace Exploration Agency

Japan

Hajimu Iida

Nara Institute of Science and Technology

Industrial products, like vehicles and trains, integrate embedded systems implementing diverse and complicated functionalities. Such functionalities are programmable by software and contain a multitude of parameters necessary for their configuration, which have been increasing due to the market diversification and customer demand. In addition, industrial products are often built by aggregating different software parts (components), constituting thus product variants. Product variants with such variability need to be tested adequately, in particular if one is concerned with security vulnerabilities. While efficient automated testing approaches already exist, such as fuzzing, no tool is able to use results from previous testing campaigns to increase the efficiency of security testing the next product variant that shares certain functionalities. This paper presents an approach that can ignore already covered functionalities by previous tests and give more importance to blocks of code that have yet to be checked. The benefit is to avoid repeating unnecessary work, hence increasing the speed and the coverage in the new variant. The approach was implemented in a tool based on the AFL fuzzer and was validated with a set of programs of different versions. The experimental results show that the tool can perform better than AFL in our testing scenario.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00033

Francisco Araújo

LASIGE, Faculdade de Ciências da Universidade de Lisboa

Portugal

Ibéria Medeiros

LaSIGE, Faculdade de Ciências da Universidade de Lisboa

Portugal

Nuno Neves

DI FC UL

Portugal

The ADS-B - Automatic Dependent Surveillance Broadcast - technology requires aircraft to broadcast their position and velocity periodically. The protocol was not specified with cyber security in minds and therefore provides no encryption nor identification. These issues, coupled with the reliance on aircraft to communicate on their status, expose air transport to new cyber security threats, and especially to FDIAs - False Data Injection Attacks - where an attacker modifies, blocks, or emits fake ADS-B messages to dupe controllers and surveillance systems. This paper is part of an ongoing research initiative toward FDIA test generation intended to improve the detection capabilities of surveillance systems. It focuses on the mechanisms used to alter existing legitimate ADS-B recordings as if an attacker had tempered with the communication flow. We propose a set of alteration algorithms covering the taxonomy of FDIA attacks for ADS-B previously defined in the literature. We experiment this approach by generating test data for an AI-based FDIA detection system [8]. Experimental results show that the proposed approach is straightforward to generate attack situations and provides a efficient way to easily generate sophisticated alterations that were not picked up by the detection system.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00034

Aymeric Cretin

Femto-ST Institute

France

Alexandre Vernotte

Femto-ST Institute

France

Antoine Chevrot

Femto-ST Institute

France

Fabien Peureux

Institut FEMTO-ST (UMR CNRS 6174) and Smartesting S&S

France

Bruno Legeard

FEMTO-ST / DISC

France

Testing non-functional properties like security, reliability, or even resilience has gained a lot of attention recently. In my talk, I focus on security and there in particular on providing testing methodologies that can be used to detect known vulnerabilities in order to avoid making mistakes twice. Techniques and methods, I discuss including planning-based and ontology-based testing make use of methods originating from artificial intelligence to create test cases for detecting security issues. Besides providing the foundations behind those methods, I also report on most recent experimental results obtained when using these methods for detecting vulnerabilities regarding injection and cross-side scripting. In addition, we discuss open issues and current research topics in security testing.

Franz Wotawa

Technische Universitaet Graz

Austria

Categorizing existing test specifications can provide insights on coverage of the test suite to extra-functional properties. Manual approaches for test categorization can be time-consuming and prone to error. In this short paper, we propose a semi-automated approach for semantic keywords-based textual test categorization for extra-functional properties. The approach is the first step towards coverage-based test case selection based on extra-functional properties. We report a preliminary evaluation of industrial data for test categorization for safety aspects. Results show that keyword-based approaches can be used to categorize tests for extra-functional properties and can be improved by considering contextual information of keywords.

Link to Publication: https://doi.org/10.1109/ICSTW50294.2020.00035

Muhammad Abbas

RISE Research Institutes of Sweden

Sweden

Abdul Rauf

RISE Research Institutes of Sweden

Sweden

Mehrdad Saadatmand

RISE Research Institutes of Sweden

Sweden

Eduard Paul Enoiu

Mälardalen University

Sweden

Daniel Sundmark

Mälardalen University

Sweden