Large Language models (LLMs) can be induced to solve non-trivial problems with “few-shot” prompts including illustrative problem-solution examples. Now if the few-shots also include “chain of thought” (CoT) explanations, which are of the form problem-explanation-solution, LLMs will generate a “explained” solution, and perform even better. Recently an exciting, substantially better technique, self-consistency 1 has emerged, based on the intuition that there are many plausible explanations for the right solution; when the LLM is sampled repeatedly to generate a pool of explanation-solution pairs, for a given problem, the most frequently occurring solutions in the pool (ignoring the explanations) tend to be even more likely to be correct! Unfortunately, the use of this highly-performant S-C (or even CoT) approach in software engineering settings is hampered by the lack of explanations; most software datasets lack explanations. In this paper, we describe an application of the S-C approach to program repair, using the commit log on the fix as the explanation, only in the illustrative few-shots. We achieve state-of-the art results, beating previous approaches to prompting-based program repair, on the MODIT dataset; we also find evidence suggesting that the correct commit messages are helping the LLM learn to produce better patches.
Wed 13 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
10:30 - 12:00 | Program Repair 1Tool Demonstrations / NIER Track / Journal-first Papers / Research Papers at Room C Chair(s): Arie van Deursen Delft University of Technology | ||
10:30 12mTalk | Adonis: Practical and Efficient Control Flow Recovery through OS-Level Traces Journal-first Papers Xuanzhe Liu Peking University, Chengxu Yang Peking University, Ding Li Peking University, Yuhan Zhou Peking University, Shaofei Li Peking University, Jiali Chen Peking University, Zhenpeng Chen University College London | ||
10:42 12mTalk | BUGSC++: A Highly Usable Real World Defect Benchmark for C/C++ Tool Demonstrations Gabin An KAIST, Minhyuk Kwon Suresoft Technologies, Kyunghwa Choi Suresoft Technologies, Jooyong Yi UNIST (Ulsan National Institute of Science and Technology), Shin Yoo KAIST Link to publication Pre-print File Attached | ||
10:54 12mTalk | Better patching using LLM prompting, via Self-Consistency NIER Track Pre-print | ||
11:06 12mTalk | The Plastic Surgery Hypothesis in the Era of Large Language Models Research Papers Chunqiu Steven Xia University of Illinois at Urbana-Champaign, Yifeng Ding University of Illinois at Urbana-Champaign, Lingming Zhang University of Illinois at Urbana-Champaign Pre-print | ||
11:18 12mTalk | GAMMA: Revisiting Template-based Automated Program Repair via Mask Prediction Research Papers Quanjun Zhang Nanjing University, Chunrong Fang Nanjing University, Tongke Zhang Nanjing University, Bowen Yu Nanjing University, Weisong Sun Nanjing University, Zhenyu Chen Nanjing University | ||
11:30 12mTalk | ExpressAPR: Efficient Patch Validation for Java Automated Program Repair Systems Tool Demonstrations Yuan-An Xiao Peking University, Chenyang Yang Peking University, Bo Wang Beijing Jiaotong University, Yingfei Xiong Peking University Media Attached File Attached | ||
