Background: Automated Intelligent Toolchains are widely used in software engineering to deploy automated program repair techniques, or in software security to identify vulnerabilites. Overall Research Problem: Most studies with automated intelligent tool-chains report uncertainty and evaluations only of the individual components of the chain. How do we calculate the uncertainty and error propagation on the overall automated toolchain? Approach: I plan to replicate research case studies to collect data and design a methodology to reconstruct the overall correctness metrics of the toolchains, or identifying missing variables. Further confirmatory experiments with humans will be performed. Finally, I will implement an artifact to automate the overall assessment of automated toolchains. Current Status: A preliminary validation of published studies showed promising results.

Aurora Papotti

Vrije Universiteit Amsterdam

Netherlands