Strings play many roles in programming because they often contain complex and semantically rich information. For example, programmers use strings to filter inputs via regular expression matching, to express the names of program elements access through some form of reflection, to embed code written in another formal language, and to assemble textual output produced by a program. The omnipresence of strings leads to a wide range of mistakes that developers may make, yet little is currently known about these mistakes. The lack of knowledge about string-related bugs leads to developers repeating the same mistakes again and again, and to poor support for finding and fixing such bugs. This paper presents the first empirical study of the root causes, consequences, and other properties of string-related bugs. We systematically study a diverse set of projects written in JavaScript, a language where strings play a particularly important role. Our findings include (i) that many string-related mistakes are caused by a recurring set of root cause patterns, such as incorrect string literals and regular expressions, (ii) that string-related bugs have a diverse set of consequences, including incorrect output or silent omission of expected behavior, (iii) that string-related bugs occur across all parts of applications, including the core components, and (iv) that almost none of these bugs are detected by existing static analyzers. Our findings not only show the importance and prevalence of string-related bugs, but they help developers to avoid common mistakes and tool builders to tackle the challenge of finding and fixing string-related bugs.

Aryaz Eghbali

University of Stuttgart

Michael Pradel

University of Stuttgart, Germany

Germany

Test-based automated program repair (APR) has attracted huge attention from both industry and academia. Despite the significant progress made in recent studies, the overfitting problem (i.e., the generated patch is plausible but overfitting) is still a major and long-standing challenge. Therefore, plenty of automated techniques have been proposed to assess the correctness of patches either in the patch generation phase or in the evaluation of APR techniques. However, the effectiveness of the existing techniques has not been systematically compared and little is known to their advantages and disadvantages. To fill this gap, we performed a large-scale empirical study in this paper. Specifically, we systematically investigated the effectiveness of existing automated patch correctness assessment techniques, including both static and dynamic ones, based on 902 patches automatically generated by 21 APR tools from 4 different categories (the largest benchmark ever in the literature). Our empirical study revealed the following major findings: (1) static code features with respect to patch syntax and semantics are generally effective in differentiating overfitting patches over correct ones; (2) dynamic techniques can generally achieve high precision while heuristics based on static code features are more effective towards recall; (3) existing techniques are more effective towards certain projects and certain types of APR techniques while less effective to the others; (4) existing techniques are highly complementary to each other. A single technique can only detect at most 53.5% overfitting patches while 93.3% of the overfitting ones can be detected by at least one technique. Based on our findings, we designed an integration strategy to first integrate static code features via learning, and then combine with others by the majorrity voting strategy. Our experiments show that the strategy can enhance the performance of existing patch correctness assessment techniques significantly.

Shangwen Wang

National University of Defense Technology

China

Ming Wen

Huazhong University of Science and Technology, China

China

Bo Lin

National University of Defense Technology

Hongjun Wu

National University of Defense Technology

Yihao Qin

National University of Defense Technology

Deqing Zou

Huazhong University of Science and Technology

Xiaoguang Mao

National University of Defense Technology

Hai Jin

Huazhong University of Science and Technology

A large body of the literature of automated program repair develops approaches where patches are generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state of the art explore research directions that require dynamic information or that rely on manually-crafted heuristics, we study the benefit of learning code representations in order to learn deep features that may encode the properties of patch correctness. Our empirical work mainly investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations. We report on findings based on embeddings produced by pre-trained and re-trained neural networks. Experimental results demonstrate the potential of embeddings to empower learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based embeddings associated with logistic regression yielded an AUC value of about 0.8 in the prediction of patch correctness on a deduplicated dataset of 1000 labeled patches. Our investigations show that learned representations can lead to reasonable performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. These representations may further be complementary to features that were carefully (manually) engineered in the literature.

Haoye Tian

University of Luxembourg

Kui Liu

University of Luxembourg, Luxembourg

China

Abdoul Kader Kaboré

University of Luxembourg

Anil Koyuncu

University of Luxembourg, Luxembourg

Li Li

Monash University, Australia

Australia

Jacques Klein

University of Luxembourg, Luxembourg

Luxembourg

Tegawendé F. Bissyandé

University of Luxembourg, Luxembourg

Luxembourg