Hardening is the process of configuring IT systems to ensure the security of the systems’ components and data they process or store. The complexity of contemporary IT infrastructures, however, renders manual security hardening and maintenance a daunting task.

In many organizations, security-configuration guides expressed in the SCAP (Security Content Automation Protocol) are used as a basis for hardening, but these guides by themselves provide no means for automatically implementing the required configurations.

In this paper, we propose an approach to automatically extract the relevant information from publicly available security-configuration guides for Windows operating systems using natural language processing. In a second step, the extracted information is verified using the information of available settings stored in the Windows Administrative Template files, in which the majority of Windows configuration settings is defined.

We show that our implementation of this approach can extract and implement 83% of the rules without any manual effort and 96% with minimal manual effort. Furthermore, we conduct a study with 12 state-of-the-art guides consisting of 2014 rules with automatic checks and show that our tooling can implement at least 97% of them correctly. We have thus significantly reduced the effort of securing systems based on existing security-configuration guides.

Patrick Stöckle

Technical University of Munich

Germany

Bernd Grobauer

Siemens AG

Alexander Pretschner

Technical University of Munich

We address the problem of identifying performance changes in the evolution of configurable software systems. Finding optimal configurations and configuration options that influence performance is already difficult, but in light of software evolution, configuration-dependent performance changes may lurk in a potentially large number of different versions of the system.

In this work, we combine two perspectives – variability and time – and propose a novel approach to identify configuration-dependent performance changes. In a nutshell, we iteratively sample pairs of configurations and versions and measure the respective performance that help us update a model of likelihoods for performance changes. Pursuing a search strategy with the goal of measuring selectively and incrementally further pairs, we increase the accuracy of identified change points related to configuration options and interactions.

We have conducted a number of experiments both on controlled synthetic datasets as well as in real-world scenarios with different software systems. Our evaluation demonstrates that we can pinpoint performance shifts to configuration options and interactions as well as commits introducing change points with high accuracy and at scale. Our experiments on three real-world systems confirm the effectiveness and practicality of our approach.

Stefan Mühlbauer

Leipzig University

Germany

Sven Apel

Saarland University, Germany

Germany

Norbert Siegmund

Leipzig University

Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects users’ expectation on performance changes. If the actual performance behaves differently from the users’ expectation, the related code snippet is likely to be problematic.

In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as test oracles in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-Detector, for detecting real-world configuration-related performance bugs. CP-Detector was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

Haochen He

National University of Defense Technology

Zhouyang Jia

National University of Defense Technology

Shanshan Li

National University of Defense Technology, China

Erci Xu

National University of Defense Technology

Tingting Yu

University of Kentucky

United States

Yue Yu

College of Computer, National University of Defense Technology, Changsha 410073, China

Ji Wang

National University of Defense Technology

Xiangke Liao

National University of Defense Technology, China