ML4PL

Several papers in the literature claim detection of bugs such as buffer overflow through machine learning techniques. Models get trained to report bugs at the function or file level, and reviewers of the results have to eyeball the code to determine whether there is a bug in that function or file, or not. Contrast this to static code analysers which report bugs at the statement level, showing the path to the bug being reported, allowing reviewers of the reports to easily determine if the statement has a bug or not.

Based on our experience with implementing scalable and precise bug finders in the Parfait tool, we experiment with machine learning to understand how close the techniques can get to a precise static code analyser. In this paper we summarise our finding in using ML techniques to find buffer overflow bugs in programs written in the C language. We treat bug detection as a classification problem. We use feature extraction and train a model to determine whether a buffer overflow has occurred or not at the function level. Training is done over labelled data used for regression testing of the Parfait tool. We evaluate the performance of the different classifiers using the 10-fold cross-validation, and the leave-one-out strategy. To understand the generalisability of the the trained model, we use it on a collection of unlabelled real-world programs and manually check the reported warnings.

Our experiments show that, even though the models give good results over training data, they do not perform that well when faced with larger, unlabelled data. We conclude with some of the open questions that need addressing before machine learning techniques can be used for detecting buffer overflow errors.

Cristina Cifuentes

Oracle Labs

Australia

Xingzhong Du

Oracle Labs

Paddy Krishnan

Yang Zhao

Oracle Labs

Modern object-oriented languages make use of complex subtype relations. Consider Julia programming language: it has user-defined invariant parametric types, existential types, union and covariant product types, as well as some ad hoc rules, like the diagonal rule, facilitating the use of the core mechanism of the language — multiple dispatching. Its subtyping relation had no rigorous description so far, is implemented in 2KLOC of highly-optimized C code touched by only a handful of core developers, and perceived by the community as a black box. Examples sending the algorithm into a loop or crushing the VM keep poping up on the issue tracker. To tame the complexity, some time ago, we aimed for a formal definition of the relation, accompanied with the implementation. But how can we be sure that our implementation indeed corresponds to the cryptic C one? To solve that, we collected a large number of examples of subtype checks from the live Julia sessions and compared the output with that of our implementation’s. It would be more enlightening, however, to learn how to generate pairs of Julia types which are likely to be in subtype relation to feed those into the testing framework. More interestingly, to learn the subtype algorithm itself.

Artem Pelenitsyn

Czech Technical University in Prague

Russia

This paper discusses code anomalies — code fragments that are written in some way that is not typical for the programming language community. Such code fragments are useful to language creators as performance tests or they could provide insights on how to improve the language. With Kotlin as the target language, we discuss how the task of detecting code anomalies for a very large codebase could be solved using well-known anomaly detection techniques. We outline and discuss approaches to obtain code vector representation and to perform anomaly detection on such vectorized data. The paper concludes with our preliminary results.

Timofey Bryksin

Victor Petukhov

ITMO University

Kirill Smirenko

Saint Petersburg State University

Nikita Povarov

JetBrains

Recent availability of large amounts of sensor data from Internet of Things devices opens up the possibility for software systems to dynamically provide fine-grained adaptations to the observed environment conditions, rather than executing only static hard-coded behaviors. However, in current adaptive systems such adaptations still need to be specified beforehand, making the development process cumbersome as well as restricting the system adaptations only to those situations foreseen by the developers. We propose that adaptations should instead be generated by machine learning techniques at run time. Adaptive systems should incorporate an adaptation engine, which, through a mix of supervised and unsupervised learning, learns adaptive behaviors, and packages them into reusable software adaptations. We illustrate this idea with a simple proof-of-concept example using Context-oriented Programming, and focus on the challenges of implementing such an approach in the development of adaptive systems.

Nicolás Cardozo

Universidad de los Andes

Colombia

Ivana Dusparic

Trinity College Dublin, Ireland

Keynote

New combinations of test generation and program analysis make it feasible to automatically infer models for the input language of a program – such as, for instance, extracting a JSON grammar from an appropriate parser. This paves the way towards simple identification of lexical and syntactical entities in program input and communication, which in turn can be used as features for machine learning tasks such as (1) classification (e.g. for identifying abnormal or malicious inputs, or identifying elements related to a failure) or (2) synthesis (e.g. for automatic input repairs). The talk includes live demos of grammar mining and input classification.

Bio: Andreas Zeller is Faculty at the Center for IT-Security, Privacy, and Accountability (CISPA), and professor for Software Engineering at Saarland University, both in Saarbrücken, Germany. His research concerns the analysis of large software systems and their development process. In 2010, Zeller was inducted as Fellow of the ACM for his contributions to automated debugging and mining software archives, for which he also obtained the ACM SIGSOFT Outstanding Research Award, in 2018. His current work combines test generation and specification mining, funded by grants from the European Research Council (ERC).

Andreas Zeller

Saarland University

Germany

Andreas Zeller is Faculty at the Center for IT-Security, Privacy, and Accountability (CISPA), and professor for Software Engineering at Saarland University, both in Saarbrücken, Germany. His research concerns the analysis of large software systems and their development process. In 2010, Zeller was inducted as Fellow of the ACM for his contributions to automated debugging and mining software archives, for which he also was awarded 10-year impact awards from ACM SIGSOFT and ICSE. In 2011, he received an ERC Advanced Grant, Europe’s highest and most prestigious individual research grant, for work on specification mining and test case generation. In 2013, Zeller co-founded Testfabrik AG, a start-up on automatic testing of Web applications, where he chairs the supervisory board. In 2018, he received the highest research award of ACM SIGSOFT, the Outstanding Research Award.

Tools solving the program synthesis problem often take as input partial specification, whether in the form of logical formulas, types or examples. Human users tend to specify just enough to explain the needed solution to themselves, and thus expose the gap between human thought, which is capable of generalization from the partial specificationf, and synthesis tools – and their backend ML models – which are only able to generalize once the form of bias has been selected for them.

Humans are often able to generalize better than algorithms. The language of classifiers of a ML algorithm, which creates its built-in bias, it is still no match for human common sense, which allows for selection from a much larger range of generalization mechanisms. Synthesis tools often address this issue by ranking the possible biases of the tool’s inner learning based on domain-specific information, but this does not scale when more generalizations are added to the tool, nor does it generalize when a synthesis algorithm is transferred to a different problem domain.

We wish to explore methods in which the human user can apply common sense: to select one of several possible biases, to refine specifications so that existing biases are no longer relevant, or to simply intervene manually in the result. In addition, we ask what the limitation of this human intervention is, since common sense can depend on many factors, first and foremost an understanding of the problem being solved and the solution being proposed by the synthesizer.

Hila Peleg

Technion, Israel

Israel

The ability to write ‘closed’ frameworks in terms of abstract supertypes and subsequently extend them via contractually-conforming subtypes is a ubiquitous programming paradigm (e.g. underpinning Object-Orientation). While the motivation for such abstraction is to insulate against requirements change, any change of contract requires extensive (typically manual) refactoring, potentially throughout the entire class hierarchy. As an alternative to defining such abstractions a priori, we describe the broad role that Machine Learning can play in inducing abstractions from a pre-existing codebase. Concrete examples are given in which contacts are enforced by dependent types in the Idris language.

Jerry Swan

University of York

Colin Johnson

University of Kent

Edwin Brady

University of St. Andrews, UK

United Kingdom

Discussion about the current state and future directions of the Machine Learning techniques for Programming Languages.