Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection".

Jane: Here's the gist of the paper. The gist The study systematically evaluates large language models (LLMs) and multimodal LLMs for sarcasm detection across English and Chinese datasets, demonstrating that audio-based models achieve the strongest unimodal performance, while text-audio and audio-vision combinations outperform unimodal and trimodal models.

Tom: Systematic Evaluation Framework The paper systematically evaluates large language models (LLMs) and multimodal LLMs for sarcasm detection on English (MUStARD++) and Chinese (MCSD 1.0) in zero-shot, few-shot, and LoRA fine-tuning settings The evaluation covers LLaMA 3 (text-only), Qwen-Audio (audio-language), Qwen-VL ((vision–language), and Qwen-Omni (audio, vision, language) Three setups are evaluated: zero-shot where models are given task instructions and any modalities they accept but no labeled examples, few-shot where k ∈ 2, 4, 6 labeled examples are provided in-context to probe in-context learning, and fine-tuning where models are adapted on sarcasm datasets using LowRank Adaptation (LoRA)

21: which serves as an upper bound for performance compared to zero-shot and few-shot prompting.

Jane: Models as Feature Extractors The study compares the effectiveness of small-scale pretrained language models versus large-scale LLMs and MLLMs as feature extractors For each modality, a conventional backbone (Base) is compared against a larger-scale foundation model (Large).

Tom: Text Modality: BERT is used as a lightweight encoder

22: , which maps an input of Nt tokens to hidden states h(text) ∈ R Nt×768, and obtains a sentence-level embedding z(text) ∈ R 768 via average pooling.

Title and authors: Tom: So we're looking at this paper, "Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection," and it’s about testing how well these big models handle sarcasm when you throw in audio and video. It’s not just text anymore; it's trying to figure out if something sarcastic is being said through a combination of what people say, how they say it, and even what they look like.

Jane: That's right, Tom, the core idea here is that sarcasm often relies on these subtle cues that span text and speech and vision together. This paper systematically evaluates large language models and multimodal LLMs across English—the MUStARD++ dataset—and Chinese—the MCSD one point zero dataset—in different ways, like just giving them instructions, or giving them a few examples, or fine-tuning them with LoRA adaptation <ref:2509.15476#pg1>.

Lu: What’s interesting here is that they aren't just looking at one type of model; they test models like LLaMA three for text only, Qwen-Audio for audio and language, Qwen-VL for vision and language, and Qwen-Omni which handles all three modalities—audio, vision, and language <ref:2509.15476#pg3>.

Meng: When I look at the setup they use for feature extraction, it’s interesting that they compare different backbones. They use BERT as a lightweight encoder for text but then go with LLaMA three-8B as the large-scale foundation model for text representations <ref:2509.15476#pg3>.

Tom: And for audio, they use Wav2Vec two point zero Base and then adopt Qwen-Audio 7B because it gives them those contextualized embeddings of dimension four thousand ninety-six which is a lot more expressive than what wav2vec produces on its own.

Jane: Then for the video part, they sample keyframes from each video and process them with ResNet50 to get visual features, and they use Qwen-VL 7B because it encodes the visual content jointly with text prompts, giving them three thousand five hundred eighty-four-dimensional embeddings.

Lu: And to bring all these together, they use a collaborative gating module. This lets each modality’s embedding get normalized first, then pass through a gating network that produces attention weights for each one before fusing them into one representation.

Meng: That adaptive emphasis part is what I find most practical; it means the model can decide to focus on the strong signals, like prosody in the audio, and ignore weaker cues if they are noisy or irrelevant for a specific dataset.

Tom: The results show that audio-based models achieve the strongest unimodal performance across both datasets. For instance, on MCSD one point zero, Wav2Vec two point zero gets a precision of seventy-eight percent, which is really high for just audio input.

Title and authors: Jane: But they also look at how combining modalities helps; they find that text-audio and audio-vision combinations actually outperform the unimodal models in both English and Chinese settings.

Lu: The fine-tuning with LoRA adaptation seems to give the biggest boosts, with Qwen-Omni achieving a seventy-one point six percent precision on MUStARD++ after fine-tuning, which is really competitive for a multimodal model <ref:2509.15476#pg3>.

Meng: From a deployment standpoint, I’m interested in how much data it takes for these models to actually get good results without that fine-tuning step; zero-shot performance is important if we want something that works right out of the box.

Tom: Zero-shot performance is pretty interesting because Qwen-Omni actually shows a strong fifty-eight point one percent precision on MCSD one point zero, which suggests it can capture sarcasm cues without any labeled examples for that specific language or domain.

Jane: But they also pointed out a cross-lingual difference, where LLaMA three performs relatively better on MUStARD++ than it does on MCSD one point zero when just using zero-shot or few-shot prompting, which is a hint about how much cultural nuance matters <ref:2509.15476#pg3>.

Lu: What the paper suggests for improvement is exploring the idea of treating LLMs and MLLMs as feature extractors rather than just classifiers, so you can fuse their outputs in different ways.

Tom: They mention that they should look at using models pretrained with balanced linguistic coverage because that seems to help them handle both English and Chinese sarcasm better simultaneously.

Jane: And they suggest looking at transfer learning and unified frameworks to exploit the reasoning capabilities of MLLMs while trying to manage the noise specific to each modality.

Meng: From a practical standpoint, it sounds like we need methods that are efficient for adaptation, so parameter-efficient fine-tuning like LoRA is still necessary even after getting those big zero-shot numbers.

Lu: It seems the paper points toward text–audio and audio–vision bimodal fusions being particularly effective because they substantially improve performance over just using a single modality alone.

Tom: So, to wrap up on this "Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection," the main point is that combining modalities, especially audio and vision, gives the best results for detecting sarcasm across different languages.

Jane: Exactly. The paper shows that models like Qwen-Omni are competitive with a lot of labeled data, but they also highlights that audio-based models are inherently strong when you only have one modality to work with.

Lu: And the architecture they propose, using collaborative gating fusion, is a solid way to make sure the model focuses on the right signal within those combined representations.

Title and authors: Meng: For deployment, if we want a system that actually works reliably in production across different inputs, focusing on these fusions seems like the most promising direction for now.

Tom: It’s clear that while MLLMs are showing promise, they still need some targeted fine-tuning to reach their absolute peak performance on these specific tasks.

Jane: So, we’re looking at a path where combining text and audio or audio and vision gives you the best results, even if you start with just one type of input.

Lu: The paper sets up a good framework for testing different model backbones against each other to see what feature extractor works best for the job.

Meng: We need to keep an eye on how these models scale; if we can’t manage the computational load, those high precision scores don't matter much in the real world.

Tom: So, when we look at this paper, we see a lot of potential in how these systems integrate different senses to understand complex human communication like sarcasm.

Jane: It gives us a clear direction for future work: focusing on culturally adaptive training strategies and building unified frameworks that handle the noise of multiple inputs well.

Lu: That seems like the right direction, moving beyond just seeing what works on one dataset to creating more robust, general systems across languages and modalities.

Meng: For me, it means we need to be really careful about which components we choose; if the feature extractor is weak, the whole fusion module won't save it.

Tom: That’s a fair point; the quality of the features you feed into that gating network really dictates how much performance you get out of it.

Jane: So, for listeners tuning in now, this paper shows that if you’re dealing with spoken language, focusing on the acoustic elements is a very strong starting point.

Lu: And for those building systems, it suggests that exploring text-audio and audio-vision combinations offers the most substantial gains over simpler unimodal approaches.

Meng: We need to see how much of that gain we can actually translate into a system that doesn't break when it encounters unexpected data patterns in the real world.

Tom: That’s where the next phase of research needs to focus: moving from impressive lab results to practical, reliable systems for spoken sarcasm detection.

Jane: Exactly. We're looking at how these MLLMs can handle the complexity of human communication when we give them a richer set of inputs.

The paper's summary: Tom: So we’re diving into the summary of this paper, "Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection." Basically, they took big language models and multimodal models—models that handle text, audio, and video—and tested them to see how well they can catch sarcasm across two different languages: English and Chinese.

Jane: That’s right. The main thing they found is that combining different types of input—like text with audio, or even audio with video—actually gives the best results for detecting sarcasm. They systematically looked at zero-shot testing, where the models get no examples, few-shot learning where they get a couple of examples in context, and fine-tuning with LoRA adaptation.

Lu: The paper’s setup is really interesting because they aren't just using one model type; they compare different backbones for each sense. For text, they compare something like BERT against a big foundation model like LLaMA three-8B <ref:2509.15476#pg3>. And for audio, they use Wav2Vec two point zero and then adopt Qwen-Audio as the large multimodal audio language model.

Meng: I’m more interested in how they actually combine everything. They use this collaborative gating module that lets each modality—text, audio, vision—pass through its own gate to decide which signal is strongest before they get fused together. That adaptive weighting sounds like it could be really useful for deployment when you have noisy data.

Tom: Exactly, Meng. And the results show that audio-based models are actually the strongest when you only look at one type of input. For instance, on a Chinese dataset called MCSD one point zero, Wav2Vec two point zero got a precision of seventy-eight percent just from the audio signal alone.

Jane: But they also showed that combining modalities beats those single-modality scores in both English and Chinese settings. They found that text-audio fusion and audio-vision fusion are particularly effective ways to boost performance over unimodal models.

Lu: The fine-tuning with LoRA adaptation consistently gave the biggest performance gains across all models and datasets. Qwen-Omni, which handles audio, vision, and language, hit a seventy-one point six percent precision on the English dataset after that fine-tuning step.

Meng: From my side at scale, that fine-tuning is necessary because zero or few-shot performance just isn't robust enough for production sarcasm detection right now. You need those adapted weights to be reliable when things get messy in the real world.

Tom: They also noted a cross-lingual difference where LLaMA three performed relatively better on the English dataset than it did on the Chinese one in zero-shot settings, which points to how much cultural and linguistic context still matters.

Jane: So, what this means for us is that sarcasm detection isn't just about reading words; it’s about understanding how sound and sight interact with language to create that sarcastic meaning.

Lu: The paper suggests that models pretrained with a good balance of linguistic coverage tend to do better at handling both English and Chinese sarcasm simultaneously.

Meng: It also points toward the idea that we need to look into unified frameworks, not just specific model tweaks, so we can deploy something that handles the noise from different inputs better.

The paper's improvements: Tom: So, looking at what they suggest for improvements in this multimodal sarcasm study, they aren't just saying "use better models," they’re actually proposing how to make them more useful for real applications. They’re focusing on making those fusions smarter, not just bigger.

Jane: Right. They suggest using that collaborative gating fusion we talked about earlier to be even more adaptive; meaning the model can dynamically decide which input—the text, the audio, or the video—is telling the most important part of a sarcastic comment in that exact moment.

Lu: The paper hints at needing better pretraining strategies too. They suggest models should be trained with a really balanced linguistic coverage so they are naturally equipped to handle both English and Chinese sarcasm without needing huge amounts of specific fine-tuning data for each language separately.

Meng: That makes sense because if we have to constantly fine-tune every model for every new language, it’s not scalable. A unified pretraining approach would save a ton of engineering time down the line.

Tom: And they emphasize that parameter efficient fine-tuning methods, like LoRA, are still the way to go because they keep performance high even when you only have a few hundred or a couple thousand samples for adaptation. It’s about being efficient with your training budget.

Jane: They also suggest exploring transfer learning frameworks that can take what models learn in one domain and apply it to another, which could help bridge those gaps between English and Chinese sarcasm more effectively than just treating them as two separate problems.

Lu: I agree with that; leveraging the reasoning capabilities of these large models across different modalities is key. It’s about finding a way for the AI to think in three dee space or across different senses without getting bogged down by modality-specific noise.

Meng: What they don't really cover, which is important for deployment, is what happens when the input data gets corrupted. The paper doesn't detail how robust these fusions are against weird noise—like a sudden change in audio quality or a visual glitch—that’s something we have to build in.

Tom: Yeah, that’s a fair point. We need to move past just getting high F1 scores on clean test sets and figure out how the system stays stable when it hits real-world, messy data streams.

Jane: So the implication is moving toward systems that are not just good at detecting sarcasm in perfect scenarios but can actually handle the messy, real communication we encounter daily.

Lu: It also points toward building more comprehensive tools, unified frameworks that exploit these models' reasoning abilities across all senses rather than just tweaking one model for one task.

Meng: For me, the next step has to be figuring out how to make this whole pipeline run efficiently on actual hardware without needing massive parameter counts just to maintain that high performance.

Conclusion: Tom: So we’re wrapping up on "Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection." The big picture here is that combining audio and vision, or text and audio, gives you a much stronger system for spotting sarcasm across English and Chinese.

Jane: It shows that relying only on one sense is just not enough; the interaction between different inputs—like how something sounds versus how it looks—is where the real detection power comes from.

Lu: The main implication for researchers is that we need to focus on building these bimodal fusions because they’re consistently beating unimodal models, even when you only give them a small amount of labeled data using LoRA adaptation.

Meng: For deployment, it means we should prioritize text-audio or audio-vision setups because those are the ones actually showing significant performance gains over the baseline models we’re currently shipping.

Lalam: I'm very confident in this result. The Qwen-Omni model, which handles all three modes, showed a high precision of seventy-one point six percent after fine-tuning on the MUStARD++ dataset. It’s quite solid performance for that setup.

Tom: That is a clean number; it’s not just some fuzzy estimate from zero-shot prompting. The ablation study showing how much LoRA adaptation helps confirm that this performance jump is real and attributable to the fine-tuning, not just better zero-shot instruction following.

Jane: Exactly, Tom. It tells us that for complex communication like sarcasm, we really need to invest in training these models specifically on the multimodal combinations they are capable of handling.

Lu: Looking forward, this work lays a good foundation for exploring culturally adaptive training strategies that could make these systems even more robust when moving between languages or different cultural contexts.

Meng: I’m still worried about the computational overhead; we need to figure out how to keep these complex gating modules running efficiently at scale without needing an enormous amount of compute time per inference.

Tom: We'll have to keep watching those efficiency metrics alongside the accuracy, because high accuracy on a huge model means nothing if it takes too long or costs too much power.

Jane: It’s interesting how this paper connects to other areas, like how vision-language models are already picking up more agricultural knowledge than they admit, suggesting that test-time reasoning is a big area for improvement in all these multimodal systems.

Lu: And the idea of unified frameworks that exploit the reasoning across senses is definitely where the real ambition lies for this research direction.

Lalam: I’m sure we can build on this. I think the next logical step is to take these fusions and see how they handle more complex, noisy, and real-world inputs than what was in these test sets.

Tom: So we're moving from showing *what* works on clean data to figuring out *how* to make it work reliably when things get messy.

Jane: That’s the next big challenge for us: making sure these powerful multimodal tools can handle the actual complexity of human communication in the real world.