Avaliação do desempenho do ChatGPT na área de Matemática dos testes do ENEM

Abstract

In recent years there has been a popularization of general-purpose, large language model chatbots that use artificial intelligence to communicate with human beings. These systems have a generally satisfactory range of knowledge in several areas due to their training, which uses massive databases made up of internet data, with ChatGPT being the most popular currently. But one area in which these chatbots often have difficulty presenting correct reasoning and answers is solving mathematical problems, generally presenting lower performance compared to other areas. For this reason, this study was carried out to analyze ChatGPT's performance in mathematics questions from six ENEM tests (2017 to 2023), in order to try and identify the types of difficulties that the program encounters when trying to solve such problems. Categorizations were made depending on the successes and errors made, such as the rate of each one, the thematic units and the types of visual representations in which they fit and the categories of causes of the errors. It was observed that ChatGPT has difficulty answering questions in the area, with an accuracy rate that varies between 22.2% and 55.5%, and that it tends to have more difficulty in questions pertaining to Unit 2 (Using geometric knowledge to perform the reading and representation of reality and act on it) and that have visual representations of “described geometric shapes”. The biggest causes of error found were incorrect interpretations of the proposed problems and a lack of logic and coherence in the answers given, in addition to a high frequency of incorrect calculation of values. In future works, this type of evaluation can be expanded to other chatbots, and can also be carried out by researchers and experts in the field of mathematics in order to improve the optimization of the performance in this topic of this type of tool.