HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

The gist The HALAS dataset establishes the first rigorous non-artificial benchmark for the detection and mitigation of ASR hallucinations by providing human-annotated spans from seven state-of-the-art ASR models on real earnings call recordings.

Dataset Creation Methodology

The researchers introduced HALAS, which is described as the first publicly available dataset of human-labeled hallucinations from seven state-of-the-art ASR models on recordings from the Earnings 22 dataset (Page 1). The dataset was created using the Earnings 22 (E22) dataset [1] which contains earnings calls in English conducted by speakers from 27 countries (Page 5). They utilized 7 state-of-the-art ASR models that achieve low WER results across various datasets in the Open ASR Leaderboard [13] (Page 2). The candidate selection procedure involved using inter-model disagreement, namely inter-model WER, as a proxy for audio difficulty to select segments for annotation (Page 2).

Hallucination Annotation Process

The annotation process was performed by 10 paid professional annotators (English B2+) (Page 2). The instruction for annotators defined hallucinations as A prediction, or a fragment thereof, that has no phonetic correspondence with the content of the audio signal analyzed by the ASR model (Page 2). Annotators were asked to perform several steps, including Audio Validation: The audio was played and carefully analyzed (Page 2) and then Prediction Hallucination Annotation: For each ASR model, the annotators were asked to read the prediction, listen to the audio recording once more, and mark the words or phrases that follow the hallucination definition (Page 2). The initial annotation agreement was high with a Cohen’s kappa of 0.87 (Page 2).

Quantitative and Qualitative Analysis

The analysis of the dataset revealed several key findings regarding model behavior (Page 3). It was found that hallucination rate varies between 21.4% and 43.8%, confirming that all investigated models hallucinate on HALAS audio data (Page 3). Furthermore, Looping occurs only in 1.1% cases on average (Page 3). The distribution of hallucinated phrases showed that for each model, the distribution is heavily skewed toward a small set of the most frequent phrases, with On average, 55% of marked hallucinations correspond to the top 10 phrases of a given model (Page 3).

Benchmarking Hallucination Detection

The paper established HALAS as a benchmark for evaluating detection methods using two main approaches (Page 4). First, detection based on proxy metrics showed that CER and SeMaScore slightly outperforming the others, suggesting that combining those categories might result in improved performance (Page 4). Second, when training an XGBoost classifier on the seven text proxy metrics across different regimes (OWN, OTHER, ALL), the classifier reached a mean ROC-AUC of 0.829 (OWN), 0.828 (OTHER), and 0.835 (ALL) (Page 4). Finally, state-of-the-art methods were tested on Whisper large v3, where the multi-layer DE detector outperformed the single-layer baseline, improving Recall and achieving the highest F1 score of 56.1% (Page 4).

Conclusion

In summary, the work introduced HALAS as a novel dataset for studying ASR hallucinations on real speech recordings (Page 5). The analysis confirmed that all seven evaluated state-of-the-art ASR models are prone to hallucinations, with generated errors exhibiting strong cross-model similarities and distributions heavily skewed toward a small set of phrases (Page 5). The study successfully demonstrated the utility of HALAS by benchmarking detection methods against naturally occurring, human-annotated hallucinations (Page 5).

--- Page 1 ---

HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

Mateusz Baranski ´

ID, Jan Jasinski ´

ID ∗∗, Julitta Bartolewska ID, Marcin Witkowski ID

,

Konrad Kowalczyk ID

Signal Processing Group, Institute of Electronics, AGH University of Krakow, Poland

Abstract

End-to-end Automatic Speech Recognition (ASR) systems hallucinate on natural speech, yet existing mitigation methods are typically evaluated on non-speech or artificially corrupted audio. We introduce HALAS, the first human-annotated dataset of naturally occurring hallucinations from seven state-of-the-art ASR models on real unprocessed earnings call recordings. HALAS provides span-level labels, enabling analysis of hallucination patterns and their severity. Our analysis reveals strong cross-model vocabulary overlap and confirms that hallucinations also occur for almost correctly transcribed speech (characterized by a low Word Error Rate). The proposed benchmark with HALAS shows that the character and semantic-level metrics used as a proxy for hallucination detection reach 81% ROCAUC, while state-of-the-art detection methods achieve an F1 score of only 53.1%. As such, HALAS establishes the first rigorous non-artificial benchmark for the detection and mitigation of ASR hallucinations.

Index Terms: automatic speech recognition, speech-to-text

hallucinations, error detection, human-annotated dataset, ASR error analysis

1. Introduction

Although an impressive generalization of modern ASR models to various datasets and domains is achieved in zero-shot settings, the use of unreliable training data [1], and the overreliance on training patterns [2] occasionally results in hallucinations. Most researchers agree on the definition that ASR hallucinations are erroneous texts in the generated transcription that do not correspond to the words spoken in the audio input [1, 3, 4, 5, 6]. In this research, we follow a similar definition of hallucination: specific errors in ASR predictions, which vary from minor filler words to additions that change or contradict intent, or loopings, i.e. multiple repetitions of a word or phrase. The appearance of hallucinations in the ASR output can indicate a malfunctioning system and lower the user experience, or even lead to misinformation or misinterpretation, especially in a medical context [5, 7].

The very recent methods for the detection of hallucinations might be divided into reference-based methods, which require ground truth text, and non-reference approaches. The first group utilizes reference text to validate whether the ASR output prediction contains hallucinations, e.g., by comparing the output prediction to the reference text using LLMs [6]. Non-reference approaches involve detection based on a lookup-list of the most frequent phrases generated based on non-speech audio input [4] or binary classification of an embedding from a specific decoder layer [8]. Detection based on thresholding of reference-based or non-reference-based proxy metrics such as Word Error Rate (WER) and Perplexity [1] might be considered as a mixed approach. Mitigation of hallucinations from ASR has been explored through pre-processing of audio input using Voice Activity Detection (VAD) [9], fine-tuning of either the weights of the ASR model [10] or its self-attention heads [11] using noisy recordings devoid of speech, for which the model should return empty transcriptions. The assessment of all the abovementioned methods has been performed using audio datasets of non-speech or artificially corrupted speech, limiting their applicability in practice. To our knowledge, the occurrence of ASR hallucinations for unprocessed real speech has not been investigated so far.

In this paper, we aim to fill this gap and introduce HALAS - Hallucination Annotations for Large-scale ASR Systems - the first publicly available dataset of human-labeled hallucinations from seven state-of-the-art ASR models on recordings from the Earnings 22 dataset [12]. First, we show the detailed description of dataset creation, including human annotation. Next, in the qualitative and quantitative analysis, we scrutinize the annotated ASR outputs, and show that all models indeed hallucinate on real data with the phrases of overlapping distributions. The investigation conducted with LLM indicates that a substantial amount of hallucination is severe regardless of the model. Finally, with an established HALAS benchmark that encompasses carefully prepared train and test splits, we evaluate hallucination detection based on multiple proxy metrics and SOTA methods, showing the challenging character of the problem studied using data with natural speech. We also show that a multi-layer extension of a decoder-embedding detector, trained on HALAS, improves upon the baseline for both in-domain and out-of-domain data. All in all, the contributions of the paper are (i) novel HALAS dataset, (ii) the first analysis of hallucinations of SOTA ASR models on real data, and (iii) the proposed benchmark that allows for evaluation of detection methods against naturally occurring, human-annotated hallucinations.

2. Dataset Creation Methodology

The following section presents the entire dataset creation process, depicted in Fig. 1. We assumed that real-world spontaneous speech recordings of varying lengths and qualities would pose challenges to ASR models, thus increasing the chances of hallucinations. We used the Earnings 22 (E22) dataset [12] as it contains earnings calls in English conducted by speakers from 27 countries. The provided audio files with total length 1https://huggingface.co/datasets/distil-whisper/earnings22

--- Page 1 ---

Figure 1: Diagram of audio file pipeline in HALAS creation.

--- Page 2 ---

109 h are divided into 57390 segments with the 5th and 95th percentiles of duration equal to 0.

1. Identify hallucinations in real-time speech by using a multi-layer decoder embedding detector trained on HALAS data to classify predictions with high accuracy (up to 56.1% F1 score).

2. Create robust hallucination detection systems capable of generalizing across different ASR models by training an XGBoost classifier on proxy metrics, achieving mean ROC-AUC scores of 0.829 when testing against unseen models.

3. Develop a system that accurately assesses the semantic severity of hallucinations by using GPT-4o mini to classify errors as minor, moderate, or severe based on majority vote agreement (Fleiss’s kappa >0.79).

End-to-end Automatic Speech Recognition (ASR) systems hallucinate on natural speech, yet existing mitigation methods are typically evaluated on non-speech or artificially corrupted audio. We introduce HALAS, the first human-annotated dataset of naturally occurring hallucinations from seven state-of-the-art ASR models on real unprocessed earnings call recordings. HALAS provides span-level labels, enabling analysis of hallucination patterns and their severity. Our analysis reveals strong cross-model vocabulary overlap and confirms that hallucinations also occur for almost correctly transcribed speech (characterized by a low Word Error Rate). The proposed benchmark with HALAS shows that the character and semantic-level metrics used as a proxy for hallucination detection reach 81% ROC-AUC, while state-of-the-art detection methods achieve an F1 score of only 53.1%. As such, HALAS establishes the first rigorous non-artificial benchmark for the detection and mitigation of ASR hallucinations.