Uma investigação sobre a aplicabilidade de Redes Transformers no contexto de tradução automática para Língua Brasileira de Sinais

Abstract

Significant segments of the world population, including the deaf community, can not fully benefit from Neural Machine Translation (NMT) resources due to various challenges developers faced when building such systems for low-resource languages. Some recent research in Natural Language Processing (NLP) with low resources focuses on creating new linguistic mechanisms and benchmarks. At the same time, another approach aim to customize existing NMT solutions for new languages and domains. Additionally, recent NLP models may apply to low-resource languages and domains without limitations. Some works investigate whether new NMT techniques can also be generalized to different resources regarding data availability and computational resources. In this context, the general objective of this study is to explore Transformer models and analyze their potential applicability in low-resource contexts, which is the case for sign languages. We identified that transformer-based solutions are state-of-the-art for most NLP problems, becoming a new industry standard for various practical problems. For a better evaluation, we adapted and used some promising identified current Transformer models in the machine translation component of the VLibras Suite, and the obtained results were compared with those currently provided by the current LightConv architecture. The first set of experiments evaluated whether such adaptation could also be applied in machine translation from Brazilian Portuguese into Libras. The results indicate that adopting one of the two top-performing architectures (Vanilla Transformer or ByT5) would help increase the accuracy and quality of the translation component of the VLibras Suite, with a maximum percentage increase of up to 12.73% considering the BLEU metric. Through prospecting and evaluation of evident models, considering that the candidate model selection process had a broader search space, a more in-depth study was conducted to try to optimize one of the top-performing models, the Vanilla Transformer. In the second phase of experimentation, a random search and a sweep of relevant hyperparameters were conducted, considering the following hyperparameters in the search space: bpe tokens, warmup updates, relu dropout, attention dropout, dropout, augmentation, and learning rate, which indicated that the first three accounted for approximately 80% of the model’s prediction capacity. The average results obtained for the BLEU metric using the reconfigured Vanilla Transformer model with optimized hyperparameters, the performance was superior by approximately 17,45% compared to the average of the reference model, and 8,95% better than the average achieved by the top-ranked model in the exploration phase, ByT5, which indicates a good potential for the hyperparameter configuration refinement strategy.