EVENT DOSSIER
A new study analyzed the repair processes of 832 Defects4J vulnerabilities in large language models (LLMs) for automated program repair (APR). The results showed that existing evaluation methods lacked a deep understanding of hallucination phenomena during the repair process. The study found that among 812 repaired samples, 72.7% had repair hallucinations, with the main error types being incorrect causal identification (45.9%) and incorrect repair strategies (18.5%). Additionally, models often misjudged which test cases were triggered, predicted line coverage related to branch control flows, or generated additional test cases with missing trigger conditions. Final evaluations indicated that only 21.0% to 55.9% of the generated patches passed the tests conducted by developers.
一项针对大语言模型（LLMs）自动化程序修复（APR）中幻觉现象的多层分析研究指出，现有评估缺乏对修复过程中幻觉的深入洞察。研究人员通过触发测试用例识别、行覆盖率预测和额外测试用例生成三项任务，考察了最终补丁中的修复幻觉及中间产物中的理解幻觉。在对 832 个 Defects4J 漏洞进行的自动评估与人工分析显示，仅 21.0%-55.9% 的生成补丁能通过开发者编写的测试套件；在 812 个样本修复中，72.7% 存在修复幻觉，其中错误因果定位和错误修复策略分别占 45.9% 和 18.5%，且模型常误判触发测试用例、预测涉及分支控制流的行覆盖率或生成缺失触发条件的额外测试用例。