Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization

Speaker:
Antonio Mastropaolo

Abstract:
Various methods for code summarization have been proposed in literature to automatically document code snippets or functions. To evaluate these approaches, researchers commonly employ automatic metrics such as BLEU, ROUGE, and METEOR. These metrics operate on the premise that higher textual similarity between a generated summary and a developer-authored summary, indicates superior quality. However, there are two reasons for which this assumption falls short: (i) reference summaries, e.g., code comments collected by mining software repositories, may be of low quality or even outdated; (ii) automatically generated summaries, while using a different wording than a reference one, could be semantically equivalent to it, thus still being suitable to document the code snippet. This seminar delves into these shortcomings and explores a potential solution by introducing a new dimension which aims at measuring how well the generated summary aligns with the semantics of the documented code snippet, independently of the reference summary.

Biography:
Antonio Mastropaolo is a Ph.D. student in the Faculty of Informatics at the Università della Svizzera italiana (USI), Switzerland, where he is part of the SEART (SoftwarE Analytics Research Team). He received his MSc. in Software System Security from Università degli Studi del Molise, Italy, in July 2020. His research interests include the study and the application of deep-learning techniques to foster code-related tasks. More information available at: https://antoniomastropaolo.com

Chair:
Matteo Biagiola

*************************

In February 2019, the Software Institute started its SI Seminar Series. Every Thursday afternoon, a researcher of the Institute will publicly give a short talk on a software engineering argument of their choice. Examples include, but are not limited to novel interesting papers, seminal papers, personal research overview, discussion of preliminary research ideas, tutorials, and small experiments.
On our YouTube playlist you can watch some of the past seminars. On the SI website you can find more details on the next seminar, the upcoming seminars, and an archive of the past speakers.