Instrument Recognition in Full-Length Polyphonic Songs: An Approach Using Foundational Models

Abstract

Identifying musical instruments in polyphonic music is a crucial task in the field of Music Information Retrieval (MIR), with impacts on music cat aloging, indexing, and analysis. Traditionally, research on instrument identi f ication is typically limited to the classification of short snippets of individual sounds or mixes. However, many practical applications require the detection of instruments in full-length, polyphonic songs. In this paper, we propose a method for identifying musical instruments on entire polyphonic songs, by us ing foundational models. First, we extract the features of the instances in the OpenMICdataset using foundational audio models. Then, we train a multilayer perceptron for instrument identification on snippets. Last, we apply different aggregation methods to obtain song-level instrument predictions. To test the model, we conducted both snippet-level tests, using the test split from the Open MIC dataset, and song-level tests, using a modified version of the MoisesDB dataset. Since the classes were different on both datasets, a mapping was cre ated to convert predictions from the model to nine instrument classes. The model achieved a macro-averaged F1 Score of 0.806 for the 20 classes on the snippets and 0.772 on full-length songs for 9 classes. On the five key instruments (voice, guitar, bass, drums and piano), the model had an F1 Score of 0.91 and 0.914 on the snippet and song level, respectively. By integrating a range of aggre gation techniques, our method offers an advancement over traditional models that are primarily limited to analyzing short audio segments, demonstrating the practical applicability of the model in real-world scenarios.