Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese

arXiv:2607.07408 · cs.CL, cs.AI · Submitted 2026-07-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese".

Jane: Here's the gist of the paper. The gist The SAMPA model, a Whisper-based segmenter, achieves competitive boundary-detection performance for terminal prosodic boundaries in Brazilian Portuguese speech by fine-tuning Whisper large-v3 and evaluating various training and test-time filtering configurations Automatic speech segmentation is the task of identifying where boundaries between speech units occur.

Tom: When these units are delimited by prosodic cues, they are commonly described as intonation units and play several roles in spoken communication, including structuring discourse and organizing information flow.

Jane: Earlier automatic approaches to prosodic segmentation relied on more traditional machine-learning pipelines.

Tom: More recently, prosodic segmentation has also benefited from large pretrained speech models.

Title and authors: Tom: So we're looking at "Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese," and the authors are using a Whisper model to do this task. It’s essentially taking a powerful speech recognition system, Whisper large-v3, and adding it the ability to mark where one intonation unit ends and the next begins.

Jane: That makes sense for someone who doesn't read papers; instead of just hearing words, you get an output that has these explicit markers inserted between the words showing those prosodic boundaries >

Lu: The title points to using a transformer-based approach, which is deep learning architecture, to solve this segmentation problem specifically within the context of Brazilian Portuguese speech.

Meng: So it’s not just applying an existing tool; they’re fine-tuning Whisper for this specific task of boundary prediction in Brazilian Portuguese >

Lalam: They are using data from two main sources: the CORAA NURCSP Minimal Corpus and CATNA, which is a publicly maintained corpus from TaRSiLA >

Tom: Which means they are working with manually segmented recordings for training, which is a big deal because traditional methods often struggle with the specific intonation units in Brazilian Portuguese >

Jane: So what they're proposing here is that we can use these large pre-trained models to do this segmentation, which was previously mostly done using older rule-based or traditional machine learning pipelines >

The paper's summary: Tom: To summarize the core idea of "Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese," the authors are proposing SAMPA, which is a Whisper-based segmenter that transcribes BP speech while inserting explicit markers for terminal prosodic boundaries >

Jane: So instead of just getting text, you get text with these special tokens right between words to show where one intonation unit ends and the next begins >

Lu: The architecture itself is adapted from PSST! four, and the main trick is treating this segmentation as an extension of speech transcription, which means adding a delimiter token to Whisper's vocabulary >

Meng: So it’s using an existing, powerful speech model and just teaching it a new way to output text that includes these boundary markers >

Lalam: They are fine-tuning Whisper large-v3 on manually segmented recordings from the NURC-SP dataset and evaluating different training and test-time filtering configurations >

Tom: The implication for us is that we can get this segmentation done using a large pre-trained model like Whisper, which is much more powerful than those older feature-based methods they mentioned in the introduction >

Jane: Which brings up the big question for us: what does this mean for someone who just listens to the show? It means when you’re listening to a BP speaker, an AI could potentially tell you where they are taking a breath or changing their pitch in a way that structures their message better >

Lu: And looking at the analysis section, it suggests that these predictions aren't random; they are informed by morphosyntactic, semantic, and prosodic cues present in the speech >

Meng: I’m curious about those cues. Are we talking about words like discourse markers or conjunctions telling the model where to put a marker >

The paper's improvements: Tom: The paper highlights several ways they improved this approach, specifically showing that applying digital filters can significantly impact their performance metrics on their test sets >

Jane: They found that the LP three thousand two hundred Hertz training configuration resulted in a word error rate of zero point one zero three, which was their best result on the held-out set >

Lu: They also showed that using high-pass filtering, like the HP six hundred Hertz model, can lead to better binary F1 scores on more diverse datasets like MuPe-Diversidades >

Meng: That’s a key piece of information for anyone thinking about deployment; it shows that tuning the pre-processing pipeline isn't just academic; it actually changes how well the model performs in different real-world scenarios >

Lalam: The analysis section also reveals that the model’s predictions are guided by linguistic patterns. For true positives, unigrams immediately before those boundaries included discourse markers, like ne´, and nouns such as cinema or casa >

Tom: So it’s not just listening to raw sound waves; the model is actually latching onto grammatical structure and semantic context when making these segmentation decisions >

Jane: And the acoustic analysis supports this because in a true positive example, they found the F0 contour shows a continuous fall at the end of one terminal unit, which they say is typical of BP declaratives >

Lu: The authors point out that false positives happen because SAMPA often generalizes from plausible acoustic and linguistic cues rather than just inserting boundaries randomly >

Conclusion: Tom: So, to wrap up "Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese," the main finding is that using a Whisper-based segmenter for this task gives competitive results, with best models reaching F1 equals zero point seven three one on the held-out NURC-SP/CATNA test split >

Jane: And they found that by tuning the filtering—using different low and high pass filters—they can get better results across different datasets, with the HP six hundred Hertz model getting a binary F1 of zero point seven nine six on MuPe-Diversidades >

Lu: The analysis confirms that the predictions are informed by morphosyntactic, semantic, and prosodic cues, which is a big step because it proves the model isn't just guessing based on sound alone >

Meng: From a deployment view, the fact that they had to manually filter out noisy recordings shows that getting clean data for training is still a major hurdle when you move this to scale >

Lalam: The authors state their limitation clearly: the system’s error analysis can identify false positives at positions exhibiting smaller boundary-like cues even if they aren't actually terminal intonation-unit boundaries in the reference annotation >

Tom: So, we have a segmenter that works well for BP prosodic boundaries using Whisper, informed by linguistic context, but it still has those tricky false positive cases that need more systematic error analysis >

Jane: It’s a solid piece of work showing how large models can tackle complex segmentation tasks in languages where traditional methods fall short, setting the stage for future research on discourse-level information >

Lu: Yeah, it sets a good foundation by proving that this extension of ASR transcription is viable for prosodic boundary prediction in this context >

Meng: If you’re looking at production pipelines right now, you need to consider how much manual filtering or data cleaning goes into getting the input audio ready for models like this >

Lalam: So, "Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese" is a piece that shows the power of large models when we are dealing with language-specific acoustic challenges >

Rodrigo de Freitas Lima, Julio Cesar Galdino, Marcos Vinicius Treviso

ICMC University of Sao Paulo · Instituto Superior Tecnico University of Lisbon

cs.CL, cs.AI

Submitted: 2026-07-08

Updated: 2026-07-13

Comments: 6 pages, 5 figures, submitted to an IEEE conference

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 73/100

The gist: The gist The SAMPA model, a Whisper-based segmenter, achieves competitive boundary-detection performance for terminal prosodic boundaries in Brazilian Portuguese speech by fine-tuning Whisper

Key concepts

Prosodic Segmentation: The task of identifying where speech units end based on intonation, rhythm, and pauses rather than just words. This helps structure spoken discourse by marking natural breaks in speech flow.
Whisper Large-v3: A large, pre-trained speech recognition model that forms the backbone of SAMPA. It is fine-tuned to recognize text while simultaneously learning to predict and insert special markers indicating where prosodic boundaries occur in Brazilian Portuguese audio.
Terminal Boundary Token: A special delimiter token ('!!!!!') inserted into the transcribed text by SAMPA. This token explicitly marks the predicted end of a speech unit, allowing the model to output both words and boundary markers together.

Terminology

Brazilian Portuguese — 2× in this paper
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The SAMPA model, a Whisper-based segmenter, achieves competitive boundary-detection performance for terminal prosodic boundaries in Brazilian Portuguese speech by fine-tuning Whisper large-v3 and evaluating various training and test-time filtering configurations

Introduction

Automatic speech segmentation is the task of identifying where boundaries between speech units occur. When these units are delimited by prosodic cues, they are commonly described as intonation units and play several roles in spoken communication, including structuring discourse and organizing information flow. Earlier automatic approaches to prosodic segmentation relied on more traditional machine-learning pipelines. More recently, prosodic segmentation has also benefited from large pretrained speech models. For Brazilian Portuguese (BP), [6] adapted heuristic methods originally proposed for English [7], relying mainly on speech rate and silent pauses. These studies show that acoustic cues such as fundamental-frequency movement, tessitura changes, and pauses are useful, but they also suggest that some BP intonation units remain difficult to detect with traditional feature-based methods. This gap motivates the use of large pretrained speech models for BP.

Model Architecture

SAMPA is a deep learning-based segmenter adapted from PSST! [4]. The key idea behind SAMPA is to cast prosodic segmentation as an extension of speech transcription. Instead of producing only lexical text, the model is trained to output a transcription that includes explicit boundary markers. During fine-tuning, terminal boundary positions are represented by a special delimiter token (“!!!!!” in our case) inserted between transcribed words. This formulation requires no architectural changes to Whisper since the delimiter is added to the vocabulary and treated as a regular decoder output token. The resulting architecture involves log-Mel features encoded by the Whisper encoder and decoded as a transcription containing both words and the delimiter token used to mark predicted terminal prosodic boundaries.

Data Preparation

The experiments utilize two BP datasets with manually annotated prosodic segmentation, both representing the Sao Paulo variety of Brazilian Portuguese. The first dataset is the CORAA NURCSP Minimal Corpus (MC) [12], which comprises data from the NURC (Norma Urbana Linguística Culta) project [13]. The second dataset is CATNA (Corpus de Audios e Transcrições Não-Alinhadas) [14], which is publicly maintained by the TaRSiLA project. To generate the final training samples, consecutive segments belonging to the same speaker were concatenated whenever the resulting audio duration remained below 30 seconds. This procedure enabled the creation of a dataset that, when fine-tuning the ASR model, allowed the model to learn to apply the delimiter token in the transcribed text when there is a prosodic boundary in the original audio.

Training and Evaluation

SAMPA is obtained by fine-tuning the Whisper Large-v3 model [5] for prosodic boundary prediction in Brazilian Portuguese, following the methodology proposed by [4]. Five different training configurations were created by applying digital filters to the original audio, including original, unfiltered audio; LP filter with a 3,200 Hz cutoff; HP filter with a 400 Hz cutoff; HP filter with a 600 Hz cutoff; and data augmentation. All models were trained for 4 epochs. In-domain evaluation on the held-out test split showed that the LP 3200 Hz training configuration was the strongest model, with a WER of 0.103, binary and macro F1 of 0.731 and 0.858, respectively. Out-of-domain evaluation on MuPe-Diversidades showed clearer variation across training setups. The HP 600 Hz model obtained the best binary F1 (0.796), while the HP 400 Hz and HP 600 Hz models tied for the best macro F1 (0.890).

Analysis of Cues

An n-gram analysis revealed that the model’s predictions are informed by morphosyntactic, semantic, and prosodic cues. For true positives (TPs), unigrams immediately before predicted boundaries included discourse markers, as ne´, and frequent nouns such as cinema, filme, and casa; unigrams immediately after TPs included conjunctions such as e, mas, and porque. The presence of nouns before the TPs and connectives after them also points to the use of morphosyntactic cues. Acoustic-visual analysis showed that in a true positive example, the F0 contour shows a continuous fall at the end of the first terminal unit (eu tenho ido bastante), a pattern typical of BP declaratives. However, false positive examples suggest that SAMPA often generalizes from plausible acoustic and linguistic cues rather than inserting boundaries arbitrarily.

Conclusion

The paper presented SAMPA, a Whisper-based segmenter for terminal prosodic boundaries in Brazilian Portuguese. The best models reached F1 = 0.731 on the held-out NURC-SP/CATNA test split and F1 = 0.796 on the out-ofdistribution MuPe-Diversidades dataset. The n-gram and acoustic-visual analyses further suggest that the model’s predictions are informed by morphosyntactic, semantic, and prosodic cues. Future work should evaluate these error patterns more systematically and incorporate discourselevel information into automatic segmentation models. The authors used AI tools, specifically ChatGPT 5.5 Pro, during the writing of this paper for English checking and polishing (grammar enhancement). They also used GitHub Copilot to assist with coding. The authors would like to thank Professor Dr. Flaviane Romani Fernandes Svartman for her valuable review of the qualitative prosodic analysis.

--- Page 1 ---

Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese

1st Rodrigo de Freitas Lima

ICMC University of Sao Paulo ˜ Sao Paulo, Brazil ˜ https://orcid.org/0009-0009-4344-1109

2nd Julio Cesar Galdino

ICMC University of Sao Paulo ˜ Sao Paulo, Brazil ˜ https://orcid.org/0000-6378-4648

3rd Marcos Vinicius Treviso

Instituto Superior Tecnico University of Lisbon Lisbon, Portugal https://orcid.org/0002-3286-0609

--- Page 2 ---

Abstract—Automatic prosodic segmentation identifies boundaries between speech units from acoustic and linguistic evidence. Although recent deep learning approaches have produced strong results for English, automatic segmentation for Brazilian Portuguese (BP) still relies mostly on rule-based or traditional machine-learning methods. This paper presents SAMPA, a Whisper-based segmenter that transcribes BP speech while inserting explicit markers for terminal prosodic boundaries

--- Page 3 ---

III. MODEL

SAMPA is obtained by fine-tuning the Whisper Large-v3 model [5] for the task of prosodic boundary prediction in Brazilian Portuguese, following the methodology proposed by [4] for English

--- Page 4 ---

IV. EXPERIMENTAL DETAILS

A. Preprocessing pipeline

Our preprocessing pipeline grouped adjacent segments according to three conditions: (i) preservation of the original file ordering, (ii) speaker continuity, and (iii) maximum segment duration constraint

--- Page 5 ---

TABLE I: Characteristics of our datasets.

Dataset Split Recordings Segments Duration

NURC-SP Train/Validation 30 11,878 28h 4m

NURC-SP Test 4 789 2h58m

--- Page 6 ---

Figure 2: F1 binary results for each filter configuration for the best model (LP 3200Hz) on the test split of NURC-SP (top), and on the MuPe dataset (bottom). The values for the base filter configuration (no filter) and the highest binary F1 achieved are highlighted

--- Page 7 ---

TABLE III: Best results for each training configuration on MuPe-Diversidades.

Training WER Bin.

Improvements for AI systems

Improved AI system can perform automatic prosodic segmentation for Brazilian Portuguese speech by using SAMPA, a Whisper-based segmenter that transcribes BP speech while inserting explicit markers for terminal prosodic boundaries.
The improved system can be fine-tuned on manually segmented recordings from the NURC-SP dataset and evaluate different training and test-time filtering configurations, achieving competitive boundary detection performance with best models reaching F1 = 0.731 on the held-out test split and F1 = 0.796 on MuPe-Diversidades.
The system can be trained using five alternative dataset versions derived from applying digital filters—including low-pass and high-pass filters—to improve performance, as seen when the LP 3200 Hz training configuration achieved a WER of 0.103 and a macro F1 of 0.858 on the held-out test split.
The system can generalize better to more diverse and cleaner datasets like MuPe-Diversidades when trained using high-pass filtering configurations, as the HP 600 Hz model obtained the best binary F1 (0.796) on this dataset despite lower scores on the held-out test split.
The system can be analyzed to understand its decision-making by performing n-gram analysis, which reveals that predictions are informed by morphosyntactic, semantic, and prosodic cues, specifically noting discourse markers immediately before predicted boundaries and conjunctions after them.
The system's error analysis can identify that false positive cases often occur at positions exhibiting smaller boundary-like cues even if they are not terminal intonation-unit boundaries in the reference annotation, suggesting a sensitivity to ambiguous cues like pauses and local F0 movements.