Avaliação da Geração Automatizada de Narrações Esportivas com o Modelo de Linguagem LLaMA

Abstract

Considering the vast amount of media elements, sports data, and football statistics, as well as the issues surrounding the lack of sports coverage in less prominent events, there arises the need to employ experimental studies that focus on automating football match commentary. Additionally, the imperative demand to improve cultural accessibility for individuals with disabilities through the use of emerging technologies reinforces the re levance of this research domain. In this context, Large Language Models (LLMs) prove relevant for yielding remarkable results in content generation. Thus, research is conducted on the use of the Large Language Model Meta AI- LLaMA to understand the perfor mance of this system in crafting soccer commentary from football game events. To do so, it is necessary to conduct experiments, compare and evaluate the system using different parameters, analyze the quality of the commentary in the context of textual readabi lity, and identify patterns of behaviors, flaws, and virtues of the model. An exploratory research methodology was then employed, utilizing granular information obtained play by-play to structure occurrences into event clusters, which were used as input commands for LLaMA. The samples were also subjected to structuring in different sets, aiming to assess the relevance of parameter changes in the model. In this regard, it was obser ved that LLaMA showed favorable results related to reading comprehension, especially in the set with higher sampling temperature, top-k, and top-p parameters. However, they exhibited a low accuracy rate. The model also achieved expected results concerning the goal of possessing narrative genre characteristics that align with the personality of a sports commentator. The increase in information, characteristic of LLMs, proved to be a concerning and deterministic factor for evaluating errors in the output text. These f indings highlight the system’s ability to generate automated sports event commentary, although they reveal the presence of some limitations that require further investigation and improvements from developers.