We report on some intermediate results from a project investigating the unsoundness of static analysis tools. The project is empirical in nature, trying to capture the status quo of static program analysis, focusing on callgraph construction for Java programs.

Static callgraph construction has to trade off several, often conflicting, design goals: precision, performance and soundness. Soundness (or, using a quantitative term, recall) has attracted more attention recently after the publication of the soundiness manifesto.

Soundness requires that a static analysis models the entire program behaviour for all possible executions. An empirical study on soundness must therefore compare models obtained via static analysis with actual program behaviour. In the past year, we have explored several techniques to collect such program behaviour, including the analysis of exception stack traces from repositories, datasets of real-world programs with synthetic high-coverage drivers, and CVEs. More recently, we have used the insights gained from those studies to construct a micro-benchmark of programs using dynamic Java language features that present barriers for static analysis tools. This talk will focus on the construction of this benchmark. Some aspects to be discussed are:

• The design of the benchmark: programs are minimalistic, use conventions over configurations to facilitate experiments, are executable and designed to have behaviour that is easy to observe.
• What is a Java program, anyway? While there seems to be an obvious answer, we have included some programs that have engineered bytecode to reflect the fact that many static analysis tools use Java byte code as input, and Java byte code is increasingly generated by non-Java compilers and / or manipulated in post compilation steps.
• The selection of features represented, and their categorisation. The benchmark contains 34 programs from the following categories: 1) reflection, 2) serialization, 3) unsafe, 4) dynamic proxies, 5) dynamic classloading, 6) invokedynamic and JNI.
• Soundness is defined with respect to the ground truth of possible program behaviour. But what does this mean? The programs in the benchmark are designed to reveal their behaviour easily by just looking at them, perhaps also consulting the relevant specifications and / or executing the respective program with an included test suite. We will also discuss corner cases where the behaviour of the program is not well-defined (by the language / JVM specification and the core API documentation), and depends on a particular implementation of the JVM.

Li Sui

Massey University, New Zealand

New Zealand

Jens Dietrich

Massey University

New Zealand

Michael Emery

Massey University

Amjed Tahir

Massey University

New Zealand

Shawn Rasheed

Massey University

New Zealand

When analyzing any corpus of programs, care must be taken to ensure that the corpus is truly representative of the entire ecosystem, otherwise the observed features might be far from reality. A naive approach is to increase the size of the dataset, thus diminishing the chance that an interesting feature will be left out. However, such approach may easily lead to overemphasis on features that are mostly present, but not frequently executed.

To tackle this issue, the code duplication patterns in the corpus and the ecosystem must be understood and correlated to the actual frequency of the code in the wild.

In our work we concentrate on the widely used JavaScript language. Originally the language of the web, JavaScript has recently been pushed to server-side and even desktop applications thanks to the node.js framework. Our analysis included all non-forked JavaScript repositories on GitHub and we looked for different levels of file and project similarity. While clone ratios in popular languages can be reasonably high (40% for Java, 73% for Python), we have found that JavaScript contains a staggering amount of 94% files being identical to the remaining 6%. When we looked at whole projects the situation is similar with almost half of JavaScript projects having over 50% of their files found in others.

Deeper analysis identified that vast majority of the duplication found is thanks to very few, but extremely popular frameworks (such as jQuery and express.js) and that in terms of files, our dataset was dominated by projects using node.js (over 70% of total files). Our continuous investigation examines the JavaScript ecosystem in greater depth with special attention paid to the node.js application and to the life-cycle of the copied code. At the end of our work, we want to create and maintain a library of JavaScript sources and their relationships that may be used by others to curate their own datasets.

Petr Maj

Czech Technical University

Celeste Hollenbeck

Northeastern University, USA

Shabbir Hussain

Northeastern University

Jan Vitek

Northeastern University

United States

In this talk, we describe the motivations behind the creation of AndroZoo, a collection of 5+ millions of Android Application, and provide concrete examples of the problems implied by the absence of a widely available and widely used benchmark dataset. Next, we come back on some of the engineering challenges linked with the creation of a large scale dataset (70+TB), and highlight the organisational challenges of maintaining a dataset in the long run in an academic setting. We also quickly present our approach to try to stay as clear as possible of the legal difficulties of distributing non-Free software. Finally, we discuss the adoption of AndroZoo and the fulfillment of our original goals, and we address what we believe are the main limiting factors with regards to AndroZoo’s adoption.

Kevin Allix

University of Luxembourg

The WebKit team designed four major web benchmarks that it uses to measure overall browser performance: JetStream, ARES-6, Speedometer, and MotionMark. JetStream runs a variety of JavaScript workloads focused on measuring the latency and peak performance of the JavaScript VM. ARES-6 measures the performance of JavaScript’s newest language features introduced in the ES2015 specification. Speedometer measures web application responsiveness with a focus on DOM API performance. MotionMark measures graphics performance using tests influenced by graphics techniques that are popular on the web. The WebKit project uses these benchmarks to guide performance optimizations and to find performance regressions. In this talk I will give a detailed explanation of what these four benchmarks are: I will describe what each benchmark was designed to measure, an overview of some of the subtests in each benchmark, and how each benchmark computes an overall score.

Saam Barati

Apple

United States

Node.js brought JavaScript from the browser to the server and NPM, its associated code repository, is now the largest in the world. However, despite its popularity, existing JavaScript benchmarks are not representative of the JavaScript ecosystem. For example, Google recently deprecated its Octane micro-benchmark and now plans to use real-world workloads to optimize its V8 JavaScript engine instead 1. In this talk, we present our initial steps towards building a benchmark that: 1) is representative of the Node.js ecosystem, and 2) lends itself to static and dynamic analysis.

To address our first goal, we start by downloading meta-data for all projects that use NPM on GitHub, collecting metrics such as: # of dependencies, # open and closed issues, # stars, # topics, size, # commits, # contributors, and proportion of project-specific code vs. dependency code. Then, we infer the probability distribution of each metrics (e.g. normal, log-normal, exponential, etc.) and sample projects in such a way that the original probability distributions are preserved. To address our second goal, we further select projects that have working test suites, pre-download all their dependencies, and build a harness that runs tests from all projects. Then, we build a test environment in the form of a container or virtual machine where specific versions of Node.js, web browsers, and databases are pre-installed, in an attempt to minimize variations across users of our benchmark.

During the talk, we will explain how we collected various metrics from GitHub, how we inferred their distributions, and how we sampled projects. We will also share our experience building an initial, executable, benchmark suite of 50 applications, and highlight the challenges we foresee ex-panding it. Finally, we will discuss two potential uses of the benchmark: dynamic analyses for performance, and security, and the additional metrics that we might need to consider.

Petr Maj

Czech Technical University

François Gauthier

Oracle Labs

Australia

Celeste Hollenbeck

Northeastern University, USA

Jan Vitek

Northeastern University

United States

Cristina Cifuentes

Oracle Labs

Australia

In 2017, the Software Development Team at King’s College London performed a benchmarking experiment to compare the warmup time and peak performance of modern programming language Virtual Machines (VMs). The experiment was intended to be the most rigorous to date. Our results were both surprising and disappointing. Not only did few modern VMs achieve a steady state of peak performance when running well known benchmarks, but some even slowed down over time.

This talk focuses not on the results of our experiment, but on our experiences of developing the “Krun” benchmarking system and the statistical analyses we used to process our data. The talk will discuss the difficulties we encountered in eliminating confounding variables and will show you how to present performance results in the absence of steady states.

Whilst Krun enabled us to collect robust and accurate results for our experiment, it tends towards being overkill. Ideally we’d like to build a cut-down version of Krun, but this raises the question of “which of Krun’s features make the most difference to benchmarking quality?”.

Edd Barrett

King's College London

United Kingdom

Sarah Mount

King's College London

United Kingdom

Laurence Tratt

King's College London

United Kingdom

Current research in clone detection suffers from poor ecosystems for evaluating precision of clone detection tools. Corpora of labeled clones are scarce and incomplete, making evaluation labor intensive and idiosyncratic, and limiting intertool comparison.

We are developing InspectorClone, a system for helping measure the precision of clone detection tools. InspectorClone provides automatic mechanisms for clone validation, a user interface to facilitate manual validation of results, and tool reports. In a comparative study, the use of InspectorClone resulted in a reduction of more than 50% of clone pairs that needed human validation effort. The reduction in manual effort will vary for each clone detector and with usage of InspectorClone, but over time we expect the manual effort to go further down. InspectorClone aggregates the individual effort of multiple teams into a single evolving dataset whose quality improves over time, creating an important asset for software clone research.

Crista Lopes

Eclipse OpenJ9 is an open source, industrial-scale JVM built on top of the Eclipse OMR language runtime framework. Benchmarks are core part of monitoring code contributions to the project to avoid performance regressions and to ensure that expect performance gains are realized. In this talk I will discuss the different aspects of performance that the project monitors on a regular basis, how we select benchmarks for monitoring, properties of some of the key benchmarks we use to measure performance, and how we are trying to make performance measurement a first-class citizen in our open code review process. I will also touch on some of the techniques we use when trying to produce minimized micro-benchmarks to recreate performance problems for easier investigation.

Andrew Craik

IBM

V8 is one of the most popular JavaScript VMs. Benchmarks play an essential role in V8’s development cycle. V8 originally focused on peak-performance benchmarks to drive VM development forward. Over time the focus broadened and V8 is fully integrated in a complex benchmarking environment which reach from unittest-like microbenchmarks to full-system benchmarks within the Chrome browser. The results of those benchmarks are made accessible to VM developers by isolating sources of noise and partitioning the time spent per VM component. As an indirect result of these benchmarks we achieved a 20% improvement on the Speedometer benchmark, a good proxy for real-world JavaScript usage.

Toon Verwaest

Google

Germany

Camilo Bruni

Google

Germany

OpenCL is an attractive programming model for high-performance computing systems, with wide support from hardware vendors and significant performance portability. To support efficient scheduling on HPC systems it is necessary to perform accurate performance predictions for OpenCL workloads on varied compute devices, which is challenging due to diverse computation, communication and memory access characteristics which result in varying performance between devices. In this talk I will present a comprehensive benchmark suite for OpenCL in the heterogenous HPC setting: an extended and enhanced version of the Open-Dwarfs OpenCL benchmark suite. Our extensions improve portability and robustness of applications, correctness of results and choice of problem size, and increase diversity through coverage of additional application patterns.

The Architecture Independent Workload Characterization (AIWC) tool can be used to characterize OpenCL kernels according to a set of architecture- independent features. This talk will also discuss the design decisions made to collect AIWC features. AIWC is a useful tool for benchmark developers since it:

1. provides insights around the inclusion of an application via diversity analysis of the feature-space.
2. measures requirements in terms of FLOPs, memory movement and integer ops of any application kernel – which allows the automatic calculation of theoretical peak performance for a given device.
3. can be used to examine the phase-transitional properties of application codes – for instance if the instruction mix changes over time in terms of the balance between floating-point and memory operations.

This work culminates in a methodology where AIWC features are used to form a model capable of predicting accelerator execution times. We used this methodology to predict execution times for a set of 37 computational kernels running on 15 different devices representing a broad range of CPU, GPU and MIC architectures. The predictions are highly accurate, differing from the measured experimental run-times by an average of only 1.2%, and correspond to actual execution time mispredictions of 9 μs to 1 sec according to problem size. A previously unencountered code can be instrumented once and the AIWC metrics embedded in the kernel, to allow performance prediction across the full range of modeled devices. The results suggest that this methodology supports correct selection of the most appropriate device for a previously unencountered code, which is highly relevant to the HPC scheduling setting.

Beau Johnston

Australian National University

Australia

Beau Johnston is completing his PhD at the Australian National University. His work focuses on developing tools to facilitate the efficient scheduling of accelerators in supercomputers. Other research interests include signal processing on HPC and embedded architectures specifically with a focus on computer vision, precision agriculture and brain machine interfaces.

For many experiments in code-centric research, researchers require real-world code to verify theories and conduct evaluations. Given the abundance of available code on platforms such as Maven Central or GitHub, real-world examples can be collected easily in large quantities. However, in order for the experiments to be meaningful, the inspected code needs to be representative for the inspected problem. For example, to properly evaluate a precise call-graph algorithm the code must contain complex virtual call sites where the strength and limitations of the algorithm can be observed. It is labor-intensive to set up these collections of code every time they become necessary. Furthermore, to increase the comparability and repeatability of the experiments, collections of code objects must be well curated so that their construction is traceable and repeatable.

New findings for these collections might invalidate their data (in parts) and related research should be inspected. Static collections (e.g., XCorpus) quickly become outdated and it is hard to annotate them once they are out in the field. During the usage of a collection, interesting data on the collection items might have been created or computed by other researchers. Algorithms (e.g., Averroes) might be available to compute information relevant for an evaluation on the fly thereby extend the original dataset. It is not easy to find this information and relate it to the the items in the collection.

To address these challenges in benchmark creation and maintenance, we introduce Delphi, an online platform to search for representative candidates to construct datasets of real-world code based on various metrics. It consists of an automated data collection, a search engine, and facilities to trace the selection process in order to foster repeatable, tractable, and comparable research. We present the current state of the project as well as our plans to extend the platform with processes for ground truth data uploads, service integration, and curated data invalidation.

Ben Hermann

University of Paderborn

Germany

Lisa Nguyen Quang Do

Paderborn University

Germany

Eric Bodden

Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM