Bilkent University
Department of Computer Engineering
CS 590/690 SEMINAR
Evaluating Review Quality of AI-Assisted Code Review Tools
Umut Cihan
Master Student
(Supervisor: Asst.Prof.Eray Tüzün)
Computer Engineering Department
Bilkent University
Abstract: Code review is crucial for the reliability, maintainability, and security of software projects. With advances in AI, several large language models have emerged capable of commenting on, reviewing, and fixing code. Consequently, numerous tools now exist for performing code reviews most of them being powered by GPT-4(Generative Pre-Trained Transformer) model. This study evaluates GPT-4's performance in code reviewing, specifically its ability to identify issues related to correctness, security, reliability, and maintainability as well as its code suggestions. We aim to conduct a controlled experiment using the HumanEval Dataset and AI-generated code benchmarks. Our methodology involved analyzing reviews of 492 AI-generated and 164 canonical ground truth code blocks to assess the detection of code quality problems and the effectiveness of code suggestions.
DATE: March 25, Monday @ 14:10 Place: EA 502