A person can say, 'That's wonderful,' in a voice that makes clear they mean the opposite. Human listeners use words, pitch, pace and context to work out what is happening. A voice AI may have access to both the transcript and the sound, yet still struggle when they disagree. That is the central finding of a new study by Fengji Jing and Guy Laban on speech-emotion recognition in audio-capable language models.[1]
The researchers tested three models across three different corpora: eNTERFACE'05, MELD and CMU-MOSEI. The datasets span acted, conversational and spontaneous speech. Instead of treating every spoken sentence as either perfectly consistent or sarcastic, the team developed a continuous measure of how far the emotional signal in the words differs from the signal in vocal delivery. Greater mismatch predicted higher odds of an incorrect emotion label whenever audio was used, after accounting for other relevant features in the model's analysis.[1]
Why the mismatch is the experiment
Most speech-emotion systems are judged on whether a predicted label matches an annotated emotion: happiness, sadness, anger, or another category. But a global accuracy percentage can hide when the model fails. Words and tone often reinforce each other. When they conflict, the model must weigh different signals rather than rely on a simple cue. That is precisely the kind of situation where a customer-service bot, voice assistant or meeting-analysis system can misunderstand a person.[1]
The study maps text and audio into a shared representation involving valence, arousal and dominance: roughly whether a signal feels positive or negative, how activated it is and the degree of assertiveness it conveys. It then calculates the distance between the two channels. The researchers validated their mismatch measure against human judgment and tested whether it predicted model errors across corpora.[1]
The key result is a pattern, not a claim that one model is always worse at every form of emotion recognition. The authors report increased odds of error with greater incongruence across all three datasets when audio was involved. They controlled for the ground-truth emotion and the distance in feature space. They also repeated the calculation without an emotion lexicon and found that the central result persisted.[1]
Evidence across the three speech datasets
| Evaluation element | What the study reported |
|---|---|
| Audio models | Three audio-capable models testedSource 1 |
| Speech corpora | eNTERFACE’05, MELD and CMU-MOSEISource 1 |
| Main signal | Mismatch between text-derived and audio-derived affectSource 1 |
| Main outcome | Error odds rose with mismatch when audio was presentSource 1 |
| Alternative comparison | Supervised classifiers using similar features showed the opposite patternSource 1 |
What makes the work especially interesting is a counterexample. Supervised classifiers trained on the same type of features behaved in the opposite direction under the paper's tests. That suggests the problem may not be that mismatched speech contains no useful information. It may be the way an audio language model integrates raw audio and text, or how its training and evaluation direct attention to those signals.[1]
A text-only system behaves differently again. The paper says that substituting the model used to read the transcript reversed the sign of one text-only effect on one corpus. That inconsistency is a warning against presenting a universal rule such as 'sarcasm always defeats AI'. The finding concerns tested systems, feature definitions and datasets, with meaningful variation by modality.[1]
The human cost of a confident label
Imagine a service agent marking a call as angry and escalating it, or a conversation analyser describing a tired speaker as uncooperative. Those are illustrative downstream possibilities, not incidents recorded by the authors. The concern is that an emotion label looks objective when it may encode uncertain inference from a fragment of speech. A user may also be speaking in a second language, masking distress or simply using an unusual prosody.[1]
For developers, a safer workflow is to treat emotional interpretation as a tentative signal, retain the original words and offer the listener a chance to correct the inference. It is especially important not to turn a model's guess into a medical or psychological diagnosis. This experiment did not evaluate clinical screening, real-time crisis intervention or the consequences of acting on an incorrect emotion label.[1]
The paper releases code, prompts and data tables, making it possible to examine the definitions and repeat aspects of the work. Further research could vary cultures, dialects, languages, noisy audio, longer conversations and the speaker's own explanation of their intent. There is also a basic design question: would a model improve if it explicitly acknowledged that tone and words conflict rather than forcing them into one category?[1]
The result reaches beyond voice products. Multimodal AI is often marketed as having richer context because it receives more kinds of data. More signals are valuable only if a model combines them appropriately. A transcript and recording can disagree for perfectly understandable human reasons. A system capable of hearing both should be judged partly by whether it knows when not to declare an emotion with confidence.[1]
Sources & further reading
- Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models Jing and Laban, arXiv · 29 Oct 2026 · Accessed 2 Nov 2026