Multi-Label Sentiment Analysis on 100 Languages with Dynamic Weighting for Label Imbalance

The gist: This work introduces a sentiment analysis framework in a multi-label setting for 100 languages that utilizes dynamic weighting, focal loss, and an optimal class-specific thresholding method to achieve state-of-the-art performance across three languages on the SemEval emotion classification competition.

Problem Overview

The study investigates cross-lingual sentiment analysis in a multi-label setting, which is inherently difficult because it requires making predictions for various languages including even unseen ones, aiming to leverage high-quality English resources for resource-scarce languages The problem is formulated as a multilabel classification task where more than one label can be assigned to a text simultaneously, yet class imbalance remains an inherent issue. Specifically, the tail labels impact performance significantly less compared to common labels when classes are equally weighted in the multi-label setting due to the rarity of relevant examples and result in suboptimal performance.

Methodology Components

The proposed framework incorporates several novel components designed to address the challenges of cross-lingual, multi-label classification and class imbalance. The key elements include:

1. Deep Multilingual Language Modeling: The framework uses XLM-RoBERTa [17] as a feature extractor, tokenizing input sentences into subword units via byte-pair encoding using Sentencepiece Tokenizer [25]. This allows the model to be adapted to other languages since it does not apply any language-dependent preprocessing.

2. Temporal Modeling of Sentence via Recurrent Networks: Bidirectional RNNs or LSTMs are employed to process variable-length sequences of token embeddings, incorporating both forward and backward information. Attention is then applied to the hidden states by weighing each timestep’s hidden state with a single parameter.

3. Multi-Label Focal Loss: The authors adapt focal loss from single-label object recognition literature to enhance the importance of hard examples in the multi-label setting.

4. Novel Dynamic Weighting Method for Label Imbalance: A dynamic weighting method is introduced to dynamically adjust class weights during training to remedy class imbalance, equalizing the contribution of each class to the loss. This method tracks losses using an exponentially smoothed approximation ωt,a and defines αt,a in (4) such that the weights sum up to 1 for any mini-batch iteration.

5. Class Specific Thresholding via Macro-F1 Maximization: A method is derived to choose optimal class-specific thresholds that maximize the macro-f1 score in linear time complexity, reducing the complexity from exponential to linear. This involves finding a threshold vector τ by maximizing MacroF1(δ(cˆa ≥ τa), ca) on the validation set.

Key Contributions

The paper highlights several significant contributions to the field. The primary contributions include:

"First time in the literature to the best of our knowledge, we introduce a multi-label emotion classification method that is capable of producing uniformly high classification performance on 100 different languages using a single model."

The authors also introduced the dynamic weighting method to remedy class imbalance with adaptive loss weights as training progresses, demonstrating significant performance gains compared to previous methods. Furthermore, they derived a method to maximize macro-f1 with class-specific threshold choices that reduces the time complexity from exponential to linear. The system achieved the state-of-the-art performance in 7 out of 9 metrics in three different languages on the SemEval emotion classification competition [1].

Experimental Results and Analysis

Experiments were conducted using datasets from the SemEval emotion classification competition in Arabic, English, and Spanish languages. The evaluation utilized macro averaged F1 (macro-f1), micro averaged F1 (micro-f1), and jaccard index. The results showed that the model achieves the best score on all of the metrics in Arabic and Spanish languages. Cross-lingual experiments demonstrated that training with data from different languages significantly improves the performance of our model. For instance, for English test data, including Arabic to the English training data improves the model more than including Spanish. The influence of dynamic weighting was analyzed, showing significant performance improvement compared to uniform weighting on both only English and combined Arabic, English and Spanish data.

Ablation Study Findings

An ablation study assessed the performance gains obtained by the components in the method. The results indicated that the bidirectional LSTM with focal loss performs significantly better compared to others, improving macro-f1 by 2.3% (absolute) on the validation set. Unidirectional LSTM performed 2.3% worse compared to its bidirectional variant. The XML-CNN, a CNN based model, performed significantly worse compared to the other variants.

Conclusion

The system successfully performs sentiment analysis in 100 different languages by introducing a dynamic weighting method to remedy class imbalance and adapting focal loss from single-label object recognition literature. The framework achieves state-of-the-art performance in 7 out of 9 metrics on the SemEval emotion classification competition [1]. The method has been evaluated in the cross-lingual setting, demonstrating performance gains from training with data from different languages. The dynamic weighting and class-specific thresholding components were shown to yield substantial performance improvements over baselines.

References

[1] S. M. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, “Semeval-2018 task 1: Affect in tweets,” in Proceedings of International Workshop on Semantic Evaluation (Semeval-2018), New Orleans, LA, USA, 2018

[5] R. Plutchik, “A general psychoevolutionary theory of emotion,” in Theories of Emotion. Elsevier, 1980

[6] D. Xu, Y. Shi, I. W. Tsang, Y.-S. Ong, C. Gong, and X. Shen, “Survey on multi-output learning,” IEEE Transactions on Neural Networks and Learning Systems, 2019

[7] T. Wei and Y.-F. Li, “Does tail label help for large-scale multilabel learning?” IEEE Transactions on Neural Networks and Learning Systems, 2019

[8] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, 2015

[9] B. Felbo, A. Mislove, A. Søgaard, I. Rahwan, and S. Lehmann, “Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017

[10] N. Kant, R. Puri, N. Yakovenko, and B. Catanzaro, “Practical text classification with large pre-trained language models,” arXiv preprint arXiv:1812.01207, 2018

[13] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019 <ref:2008.

1. Dynamic weighting method for loss adjustment: The system can dynamically adjust class weights during training to remedy the class imbalance by using a method that equalizes the contribution of each class to the loss. This allows for better performance on rare labels compared to static weighting methods.

2. Macro-f1 maximization for threshold selection: The system can derive a method to choose optimal class-specific thresholds that maximize the macro-f1 score in linear time complexity, which is much faster than exponential grid search for choosing prediction thresholds.

3. Cross-lingual sentiment analysis on 100 languages: The improved system can perform sentiment analysis across 100 different languages using a single model, overcoming the limitations of language-specific models that require separate processing pipelines.

4. Adaptation of focal loss: The framework can adapt the focal loss to our multi-label setting from the single-label object recognition literature, focusing training on hard instances to improve classification accuracy.

5. Feature extraction using XLM-RoBERTa: The model can utilize features extracted from XLM-RoBERTa, a robustly trained BERT on 100 languages, to benefit from large language model capabilities while reducing the amount of required resources.

6. Enhanced temporal modeling with BiLSTM/LSTM and attention: The recurrent network structure can process variable length sequences using bidirectional RNNs or LSTMs, incorporating context through attention scores to capture complex temporal relations in sentences.

7. Improved performance on SemEval competition: The system is expected to achieve the state-of-the-art performance in 7 out of 9 metrics in 3 different languages on the SemEval emotion classification competition using a single model.