SPLConqueror Logo

SPL Conqueror: Measuring and Predicting

Non-Functional Properties of Highly Configurable Software Systems


Description

SPL Conqueror is a library for learning the influences of configuration options on non-functional properties. It produces an influence model, containing terms that represent the influences. These terms are learned in an iterative learning process using linear regression and stepwise feature selection. SPL Conqueror supports several sampling techniques including binary-option heuristics and experimental designs for numeric-option sampling. Given a variability model, SPL Conqueror states, which configurations should be measured. We use a CSP solver as a backend to produce only valid binary configurations. The library is open source under GPL 2: https://github.com/nsiegmun/SPLConqueror

Supplemental Material

ESEC/FSE 2015 paper: Performance-Influence Models for Highly Configurable Systems
ICSE 2012 paper: Predicting Performance via Automated Feature-Interaction Detection
Information and Software Technology paper: Scalable Prediction of Non-functional Properties

Other Measurement Techniques

In the following, we present measurement results. To obtain source code metrics, we use the tool Source Monitor.
Case Study: Complexity

Berkeley DB Source Code Measurements
FAME-DBMS Source Code Measurements
Note, Checkpoints in the xml file represent features of the respective SPL.

Case Study: Footprint (BerkeleyDB)

Feature Footprint
BaseDB 45207
Common 2233
HMAC 5760
DbRegister 5298
FileOps 18295
MemPool 25687
Mutex 2503
DbEnv 18979
Lock 21515
Os 1970
BaseDB.Rename 1565
BaseDB.Remove 2127
BaseDB.Truncate 1010
ConfigTest 1090
Join 3384
Exception 2124
Dump 8559
Diagnostic 278
BTree 57298
RecNo 6134
Qam 22938
Hash 30988
Logging 39077
Statistic 24405
Transaction 25568
XA 2963
ArgCheck 1567
Replication 55742
Crypto 14450
Verify 31105
Recovery 34810
Compact 878
Sequence 6031
Examples 591
Upgrade 1370
ErrorLog 1842
HMAC/HmacOld 225
Mutex/Win32 1195
DbEnv/Join 148
Os/Win32 6692
Join/Statistic 66
Join/ArgCheck 133
Join/Replication 0
BTree/Logging 10687
BTree/Recovery 12662
BTree/Compact 25213
BTree/Upgrade 2718
Qam/Logging 4976
Qam/Statistic 1435
Qam/Verify 2699
Qam/Recovery 7020
Hash/Logging 8151
Hash/Recovery 2787
Hash/Upgrade 816
Statistic/Transaction 2624
Statistic/Replication 3049
Statistic/Sequence 1464
Transaction/Recovery 4619
Case Study: Footprint (FAME-DBMS)

FeatureFootprint
Main 745
OS.Windows 405
Test 900
BufferManager 1258
BufferManager.MemoryAlloc.Static 569
BufferManager.PageFind.Hash 389
BufferManager.PageReplace.LRU 593
BufferManager.PageReplace.LFU 313
DataAndAccessManager 777
DataAndAccessManager.WriteSupport 22
StorageManager.Page.PageC 1333
StorageManager.Page.PageC.WriteC 1938
StorageManager 95
StorageManager.Read.ReadUnIndex 577
StorageManager.Write.WriteUnIndex 847
StorageManager.Read.ReadIndex 54
StorageManager.Write.WriteIndex 77
StorageManager.Index.BPlusTree 3872
Main/OS.Windows 427
OS.Windows/Test 69




Notes on Using SPL Conqueror

SPL Conqueror is written in C# and requires .Net 4.0 or later. To use SPL Conqueror, you need to retrieve the Microsoft Solver Foundation under the academic license. This is free for academic use. SPL Conqueror is open source and developed as a framework such that we develop the implementation of other backend solvers. Please let me know if you are using SPL Conqueror or the measurements presented on this page for your research.

Publications

See full list

Norbert Siegmund, Sergiy S. Kolesnikov, Christian Kästner, Sven Apel, Don Batory, Marko Rosenmüller, and Gunter Saake. Predicting Performance via Automated Feature-Interaction Detection. In Proceedings of International Conference on Software Engineering (ICSE), pages 167–177. IEEE, 2012. Acceptance Rate: 21%

Norbert Siegmund, Marko Rosenmüller, Christian Kästner, Paolo Giarrusso, Sven Apel, and Sergiy Kolesnikov.Scalable Prediction of Non-functional Properties in Software Product Lines
In Proceedings of International Software Product Lines Conference (SPLC), pages 160–169. IEEE, August 2011.Acceptance Rate: 29% Best Paper Award

Norbert Siegmund, Marko Rosenmüller, Martin Kuhlemann, Christian Kästner, Sven Apel, and Gunter Saake. SPL Conqueror: Toward Optimization of Non-functional Properties in Software Product Lines.
Software Quality Journal, Volume to appear, 2011. 10.1007/s11219-011-9152-9.

Norbert Siegmund, Marko Rosenmüller, and Sven Apel. Automating Energy Optimization with Features.
In Proceedings of International Workshop on Feature-oriented Software Development (FOSD), pages 2–9. ACM, October 2010.

Norbert Siegmund, Martin Kuhlemann, Sven Apel, and Mario Pukall. Optimizing Non-functional Properties of Software Product Lines by means of Refactorings.
In Proceedings of Workshop Variability Modelling of Software-intensive Systems (VaMoS), pages 115–122, January 2010.

Norbert Siegmund, Marko Rosenmüller, Martin Kuhlemann, Christian Kästner, and Gunter Saake. Measuring Non-functional Properties in Software Product Lines for Product Derivation.
In Proceedings of the 15th International Asia-Pacific Software Engineering Conference (APSEC), pages 187–194. IEEE Computer Society, December 2008. Acceptance Rate: 29%

Norbert Siegmund, Martin Kuhlemann, Marko Rosenmüller, Christian Kästner, and Gunter Saake. Integrated Product Line Model for Semi-Automated Product Derivation Using Non-Functional Properties.
In International Workshop on Variability Modelling of Software-intensive Systems (VaMoS), pages 25–32, January 2008.

Version History

  • Version 0.6 (March 2015):
    • Complete reimplementation of SPL Conqueror as open-source project
    • Supporting numerical configuration options for the first time
    • Implemented a completely new machine-learning approach
    • Implemented several experimental designs
  • Version 0.5:
    • Integrated optimization of a single non-functional property (minimization or maximization)
    • Definition of non-functional constraints for optimization
    • Integrated automatic detection of feature interactions using the heuristics present in "Predicting Performance via Automated Feature-Interaction Detection." at ICSE'12
  • Version 0.2.1:
    • Refactoring of the measurement screen. Removed depricated functionality.
  • Version 0.2:
    • Improved program feedback (e.g., error messages when compilation failed).
    • Progress information window added.
    • Added setting to turn error messages off.
  • Version 0.1:
    • Initial version.
    • Functionality to approximate feature values
    • Automated detection of feature interacions.
    • Definition of product lines and Features
    • Program generation options for automated variant generation and compilation
    • Definition of a measurement application to measure a program's non-functional properties
    • Definition of different types of non-functional properties (please read the journal article: SPL Conqueror: Toward Optimization of Non-functional Properties in Software Product Lines, for more information)

    • Contact

    The website is still under construction.
    This web site is maintained by Norbert Siegmund at the University of Passau.