Emotional Vietnamese Speech-Based Depression Diagnosis Using Dynamic Attention Mechanism

arXiv:2412.08683 · cs.SD, cs.CV, eess.AS · Submitted 2024-12-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Emotional Vietnamese Speech-Based Depression Diagnosis Using Dynamic Attention Mechanism".

Jane: Here's the gist of the paper. The gist The study proposes a Dynamic Convolutional Block Attention Module (DynamicCBAM) utilized within an Attention-GRU Network to classify emotions from emotional Vietnamese speech signals to diagnose depression tendency.

Tom: Major depressive disorder is a prevalent and serious mental health condition that negatively impacts your emotions, thoughts, actions, and overall perception of the world <ref:2412.08683#pg6>.

Jane: People who are depressed express discomfort, sadness and they may speak slowly, trembly, and lose emotion in their voices <ref:2412.08683#pg8>.

Tom: The research aims to assess the probability of individuals having a tendency to be depressed by analyzing the frequency amplitude of the voice.

Title and authors: Tom: We've talked about the basic setup, and now let's look at exactly what this paper sets out to do, focusing on the title and who wrote it.

Jane: So the title itself, "Emotional Vietnamese Speech-Based Depression Diagnosis Using Dynamic Attention Mechanism," tells us immediately that they are looking at Vietnamese speech for depression signs and using a dynamic attention mechanism in their deep learning model.

Lu: It seems like the focus is really on making the emotion recognition part of the system more adaptive, which is why they introduce this DynamicCBAM block instead of just a standard one.

Meng: Does that mean they're trying to fix some problem where standard attention mechanisms might be too rigid for different kinds of speech data?

Tom: Exactly. The authors are aiming to assess the probability of an individual having a tendency to be depressed by analyzing the frequency amplitude of the voice, which is what they want to do in a more comprehensive and accurate way.

Lalam: They specifically mention that by completing this research, they aim to obtain useful information so they can diagnose depression and detect those signs more accurately and quicker.

Jane: It's about moving beyond just saying someone sounds sad or happy, toward a more specific diagnostic tool based on measurable voice features.

The paper's summary: Tom: Moving into the summary of what they actually did, they are employing this dual-stream approach to handle the audio input.

Jane: They take two paths: one stream goes with raw sound waves, which are five seconds long at a frequency of sixteen kHz, and another stream transforms the raw audio into Mel-Frequency Cepstral Coefficients or MFCCs.

Lu: The raw waveform path is fed through embedding layers to learn temporal dynamics, and then that output feeds into a Bidirectional Gated Recurrent Unit, or Bi-GRU layer to capture those long-term dependencies in the sequence.

Meng: So one stream handles the raw audio's timing and flow using recurrence, while the other stream focuses on spectral features like MFCCs. That seems like a lot of processing power for real-time use.

Tom: It is heavy, but they do that because they believe analyzing these specific characteristics—speed, volume, rhythm, and tone—is key to understanding depression signs in this context.

Jane: The whole point of using the Attention-GRU with DynamicCBAM is to make sure the model can adapt its feature maps along both the channel and spatial dimensions dynamically.

The paper's improvements: Tom: Now let's talk about what they added to improve things, because that’s usually where we find the real meat of the work.

Lu: The main improvement is introducing the DynamicCBAM block itself, which they modified from a standard CBAM by using Omni-Dimensional Dynamic Convolution, or ODConv, instead of standard convolutions in the spatial attention module.

Meng: So they replaced those fixed weight convolutions with something that generates features based on DyConv and CondConv, which they say allows it to capture more complex and varied patterns, especially with MFCC data.

Jane: That adaptation is what they are hoping enhances the representational power of the network by refining those feature maps in a more intelligent way.

Tom: And looking at their experimental results on the Vietnamese Speech Emotion dataset, they report very solid performance metrics. They achieved an UA of zero point eight seven, a WA of zero point eight six, and an F1 score of zero point eight seven for the MFCCs approach alone.

Jane: But then they show that adding this DynamicCBAM block in that dual-stream setup with the MFCCs stream actually boosted the UA to zero point eight six, which they call a significant performance gain over just using MFCCs on its own.

Conclusion: Tom: So to wrap up, this paper proposes a Dynamic Attention GRU Network enhanced by the DynamicCBAM block for classifying emotions from Vietnamese speech to diagnose depression tendency.

Jane: The study concludes that this architecture, particularly when using MFCCs as input, achieves solid results with an F1 score of zero point eight seven and a UA of zero point eight six when the DynamicCBAM is included in the dual-stream configuration.

Lu: From my view, the implication is that incorporating this dynamic attention mechanism directly into the feature extraction process significantly helps in capturing emotion-relevant features from speech data for this specific task.

Meng: For deployment, it means we have a model structure that shows promise, but we still need to see how stable it is when we move it off the training environment and onto live traffic where things get messy.

Lalam: I’m pretty sure the paper's findings are quite reliable given the metrics they reported, suggesting a good way forward for using speech analysis in this area.

Tom: It shows that combining temporal modeling with dynamic attention can yield these specific numbers, and they're looking ahead to further quality and accuracy improvements in future work.

Quang-Anh N.D., Manh-Hung Ha, Thai Kim Dinh, Minh-Duc Pham, Ninh Nguyen Van

International School, Vietnam National University, Hanoi, Vietnam

cs.SD, cs.CV, eess.AS

Submitted: 2024-12-11

Updated: 2024-12-11

Comments: 9 Page, 5 Figures

Code: https://github.com/fiyud/Emotional-Vietnamese-Speech-Based-Depression-Diagnosis-Using-Dynamic-Attention-Mechanism

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 68/100

The gist: The gist The study proposes a Dynamic Convolutional Block Attention Module (DynamicCBAM) utilized within an Attention-GRU Network to classify emotions from emotional Vietnamese speech signals to

Key concepts

Dynamic Convolutional Block Attention Module (DynamicCBAM): This is a modified attention block designed to improve feature maps by adaptively refining them along both channel and spatial dimensions. It replaces standard convolutions with Omni-Dimensional Dynamic Convolution (ODConv), which allows the network to capture more complex patterns in speech features, especially MFCCs.
Dual-Stream Approach: The model processes audio using two separate paths simultaneously. One stream analyzes the raw sound waves directly, while the second stream transforms the audio into Mel-Frequency Cepstral Coefficients (MFCCs). This allows the network to benefit from both temporal dynamics and spectral features of the speech.
Bidirectional Gated Recurrent Unit (Bi-GRU): The Bi-GRU layer is used to capture long-term dependencies in the audio sequence in both forward and backward directions. This capability is crucial for emotion recognition because it helps the model understand how context from both past and future parts of the speech signal influences the current emotional state.
Vietnamese Speech Emotion Dataset (VNEMOS): This is a specific dataset containing both natural and acted Vietnamese speech samples used for training and testing. It provides the real-world data necessary to train a deep learning model to accurately classify emotional characteristics related to depression tendencies in Vietnamese speech.

Terminology

Accuracy — 2× in this paper · explained in 1 episode
Attention Mechanism — 1× in this paper · explained in 3 episodes

Summary

The gist The study proposes a Dynamic Convolutional Block Attention Module (DynamicCBAM) utilized within an Attention-GRU Network to classify emotions from emotional Vietnamese speech signals to diagnose depression tendency.

Introduction and Motivation

Major depressive disorder is a prevalent and serious mental health condition that negatively impacts your emotions, thoughts, actions, and overall perception of the world <ref:2412.08683#pg6>. People who are depressed express discomfort, sadness and they may speak slowly, trembly, and lose emotion in their voices <ref:2412.08683#pg8>. The research aims to assess the probability of individuals having a tendency to be depressed by analyzing the frequency amplitude of the voice. By using deep learning, we are able to assess and analyze the emotional characteristics of a voice in a more comprehensive and accurate manner. An evaluation of specific emotional characteristics in the human voice was performed, including speed, volume, rhythm, and tone.

Methodology

The proposed architecture involves a dual-stream approach to leverage both raw waveform and spectral features. The first stream of the model, the input is sound waves with length of 5 seconds and a frequency of 16 kHz which have been vectorization to go through the embedding layers for feature extraction. This hierarchical structure allows the model to learn increasingly abstract representations of the temporal dynamics in the audio signal. Then, the output of these convolutional layers is then fed into a Bidirectional Gated Recurrent Unit (Bi-GRU) layer. The Bi-GRU is capable of capturing long-term dependencies in both forward and backward directions of the audio sequence which is particularly valuable for emotion recognition tasks where context is crucial. The second stream begins by transforming the raw audio into Mel-Frequency Cepstral Coefficients (MFCCs). The output from embedding layers then passed through the proposed Dynamic-CBAM block, this will enhances the representational power of the network by adaptively refining feature maps along both channel and spatial dimensions.

Dynamic Convolutional Block Attention Module (DynamicCBAM)

The paper proposes a new block named Dynamic-CBAM by modifying the Convolutional Block Attention Module (CBAM) [13] with Omni-Dimensional Dynamic Convolution (ODConv) [14]. The CBAM block containing an hybrid attention mechanism by utilizing channel attention and spatial attention mechanism, given a feature map F Rh×w×c with h, w, c denotes for height, weidth and channel, respectively and f x×x stand for kernal size of the convolution layer. The calculation of the CBAM module is shown in Eq 3 where F′ = Mc(F) ⊗ F and F′′ = Ms (F′) ⊗ F′, FO = I ⊕ CBS (F''). In this paper, we replace the standard convolutions in spatial attention module of CBAM to ODConv. The ODConv is created base on DyConv [16] and CondConv [17]. This adaptive feature enables ODConv to capture more complex and varied patterns in the data as especially in MFCCs data use for this paper.

Experiment and Results

The study uses the Vietnamese Speech Emotion dataset (VNEMOS) [20] that we have compiled in recent work, the dataset contains both natural and acted data. The researchers trained and evaluate the model with hyperparameters setting as epoch = 100, lr = 0.001, batchsize = 32, optimizer = adam using environment as outlined in Table 3. The evaluation metrics used are Precision, Recall, and F1 Score. The Proposed Model MFCCs achieves the highest performance across all metrics with 0.87 UA, 0.86 WA, 0.87 F1-score. This suggests that the proposed model architecture effectively extracts emotion-relevant features from MFCCs alone. The introduction of Dynamic-CBam in the dual-stream configuration with 0.86 UA yields a significant performance boost.

Conclusion

In this study, the application of deep neural networks to identify emotions from speech to recognize depression tendency. A Dynamic Attention GRU Network have been proposed and aslo in this paper, we have introduce a new block named Dynamic-CBAM. All in all the research has an impressive UA, WA, F1 with 0.87, 0.86 and 0.87, respectively. The future will try to improve the quality of the model and also improve the accuracy. This underscores the advancements made in the current study >.

References

[1] American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).

[2] Belmaker, R. H., & Agam, G. (2008). Major depressive disorder. The New England journal of medicine, 358(1), 55–68.

[3] Bromet, E., Andrade, L. H., Hwang, I., Sampson, N. A., Alonso, J., de Girolamo, G., de Graaf, R., Demyttenaere, K., Hu, C., Iwata, N., Karam, A. N., Kaur, J., Kostyuchenko, S., Lépine, J. P., Levinson, D., Matschinger, H., Mora, M. E., Browne, M. O., Posada-Villa, J.)

[4] World Health Organization. (2017). Depression and other common mental disorders: Global health estimates.

[5] Cuijpers, P., Sijbrandij, M., Koole, S. L., Andersson, G., Beekman, A. T., & Reynolds, C. F. (2013). The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: A meta-analysis of direct comparisons.

[6] https://dep.mohw.gov.tw/DOS/mp-113.html

[7] E. Yüncü, H. Hacihabiboglu and C. Bozsahin, Automatic Speech Emotion Recognition Using Auditory Models with Binary Decision Tree and SVM, 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden, 2014, pp. 773-778, doi: 10.1109/ICPR.2014.143.

[8] Ha, Manh-Hung, et al.’ “Emotional Inference from Speech Signals Informed by Multiple Stream DNNs Based Non-Local Attention Mechanism”, EAI Endorsed Transactions on Industrial Networks and Intelligent Systems, vol. 11, no. 4, Aug. 2024. DOI:https://doi.org/10.4108/eetinis.v11i4.4734.

[9] Xingchen Ma, Hongyu Yang, Qiang Chen, Di Huang, and Yunhong Wang. 2016. DepAudioNet: An Efficient Deep Model for Audio based Depression Classification. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge

[10] Haytham M.

Improvements for AI systems

Add noise filtering to raw waveform input before embedding layers to address data corrupted spike-noises in the raw waveform data mentioned in Section 3.2, potentially enhancing accuracy by cleaning input data before processing.
Implement a mechanism for dynamic learning rate adjustment within the Attention-GRU Network during training to ensure faster convergence and better optimization of the model's parameters.
Integrate language barrier mitigation techniques, such as cross-lingual embeddings or transfer learning from multilingual speech datasets, to improve performance on Vietnamese speech data when testing in different scenarios.
Develop an automated system for identifying and isolating audio segments free of noise or distortions based on the model's confidence scores before feeding them into the classification layer to improve robustness.
Enhance the Dynamic-CBAM block by exploring hyperparameter optimization strategies for the reduction ratio gamma in Omni-Dimensional Dynamic Convolution, aiming to capture more nuanced spatial and channel dependencies.

Abstract

Major depressive disorder is a prevalent and serious mental health condition that negatively impacts your emotions, thoughts, actions, and overall perception of the world. It is complicated to determine whether a person is depressed due to the symptoms of depression not apparent. However, their voice can be one of the factor from which we can acknowledge signs of depression. People who are depressed express discomfort, sadness and they may speak slowly, trembly, and lose emotion in their voices. In this study, we proposed the Dynamic Convolutional Block Attention Module (Dynamic-CBAM) to utilized with in an Attention-GRU Network to classify the emotions by analyzing the audio signal of humans. Based on the results, we can diagnose which patients are depressed or prone to depression then so that treatment and prevention can be started as soon as possible. The research delves into the intricate computational steps involved in implementing a Attention-GRU deep learning architecture. Through experimentation, the model has achieved an impressive recognition with Unweighted Accuracy (UA) rate of 0.87 and 0.86 Weighted Accuracy (WA) rate and F1 rate of 0.87 in the VNEMOS dataset. Training code is released in https://github.com/fiyud/Emotional-Vietnamese-Speech-Based-Depression-Diagnosis-Using-Dynamic-Attention-Mechanism