Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training

arXiv:2610.13608 · eess.AS, cs.LG, cs.SD · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training".

Jane: The gist: A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, what does this actually do? They describe it as rendering a recorded utterance at a different vocal intensity while preserving its content and speaker identity. It’s not just scaling the amplitude up or down; it’s about changing the perceived effort of the voice in a way that keeps the speech recognizable.

Lu: The methodology involves transforming encoded frames using this converter Cθ, where x hat equals Dec Cθ Enc x with τ t. This means they are operating in the feature space of a frozen encoder and decoder pair rather than directly manipulating raw audio waveforms.

Meng: That's a specific mathematical formulation to keep things clean. The converter Cθ is designed to be frame-synchronous, meaning it outputs one frame for every input frame, which ensures the sequence alignment doesn't get messed up by changing the duration or speaking rate of the speech.

Lalam: And because Cθ is residual, defined as z plus delta theta, where delta theta predicts the correction from the whole sequence, they’re making sure it’s correcting things globally across the input rather than just frame by frame.

Tom: That residual structure sounds like a solid way to train something that needs to make larger adjustments based on what it sees in the entire sequence. The training objective is pretty detailed, involving adversarial loss, cycle consistency loss, and an identity anchor loss.

Jane: They use three different feature spaces for testing: log-mel spectrograms which is acoustic, WavLM's latent space from a neural codec, and SpeechTokenizer's latent space which relates more to phonetic content. They compare how well the converter performs across these different starting points.

The paper's summary: Tom: The paper suggests several ways to improve things. One big point is that they introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus, which allows for a quantitative assessment of success using linear regression where a slope of one means the target change was fully realized <ref:2610.13608#pg1,introduce an objective protocol that measures the achieved intensity change against the>.

Jane: That objective measurement is important because it moves away from just guessing if you hit the right level. It gives them something concrete to measure their performance against when they are evaluating the system's accuracy on real data.

Lu: They also compared three feature spaces and found that WavLM layer six achieved eighty-three percent of the target intensity change, which is a solid result for that representation space in terms of intensity control <ref:2610.13608#pg3>.

Meng: But they also noted where it falls short. They say that first, the model controls vocal intensity relative to the source recording but is less consistent in reaching an absolute perceived level. That means while it handles changes well, getting the exact final volume might still depend heavily on what you started with.

Lalam: And second, they mentioned that because this method displaces fundamental frequency and spectral measures a little less than original recordings do, there’s room for further improvement in those areas. It doesn't perfectly preserve every acoustic detail from the original recording.

The paper's improvements: Tom: So to wrap up Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training, they found that WavLM layer six achieves eighty-three percent of the target intensity change, and that the perceived level of its conversions increases with the target level <ref:2610.13608#pg3>.

Jane: That confirms their main finding: it handles relative intensity control well across different feature spaces. It's a significant advance because it allows for continuous control in dB SPL, which is much more flexible than the discrete categories we've seen before.

Lu: The limitations they flag are that the model might not be consistent in reaching an absolute perceived level, and they point out that calibrated corpora covering a wide range of vocal intensities are scarce, which limits the training data available.

Meng: For deployment, it means you need to be careful about how much you rely on the absolute output; it’s good for relative adjustments but not perfect for precise volume calibration across all scenarios.

Lalam: So, in summary, they moved away from reference recordings or noise categories to a continuous target level in dB SPL and showed that this approach works when using the right feature space like WavLM L6. It opens up a path for more nuanced vocal manipulation with AI.

Conclusion: Tom: So we've been talking about this paper, "Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training." The main thing is that they built a system that can take an audio recording and change its vocal intensity to any specific decibel level on a continuous scale while keeping the speaker sounding like themselves.

Jane: Right, so the naive question is, can we actually do that without making the voice sound completely robotic or losing what makes it sound human? And they seem to have answered that by using this GAN structure.

Lu: The methodology hinges on a conditional cycle-consistent GAN where they transform encoded frames using a converter C theta such that x hat equals Dec C theta Enc x with tau t. It’s a residual network, which means the correction is predicted across the whole sequence rather than just frame by frame adjustments.

Meng: From my side, I'm thinking about deployment. They compared three different feature spaces: log-mel spectrogram, WavLM layer six and SpeechTokenizer’s latent space. That choice of feature space matters a lot for how well the model works in practice.

Lalam: I looked at it too and it seems that WavLM L6 achieved eighty-three percent of the target intensity change, which is pretty good for a representation that relates closely to phonetic content.

Tom: Eighty-three percent is solid, but they also pointed out some limitations. They say the model might not be consistent in reaching an absolute perceived level on its own, and they noted that because it displaces fundamental frequency and spectral measures a little less than original recordings do, there’s room for improvement there.

Jane: So what does that mean for us? It means while the relative intensity control is strong, getting the exact final volume might still depend a lot on what you started with. That's a caveat we have to keep in mind when using it.

Lu: They also mentioned that calibrated corpora covering a wide range of vocal intensities are scarce, which limits the training data available for the converter C theta. That’s definitely something to think about for scaling this up.

Meng: And from an engineering standpoint, if we want to deploy this reliably across different types of recordings, we need those calibrated datasets. If we don't have them, it's hard to guarantee consistent results in a production environment.

Tom: It’s a trade-off then: you get good control over intensity conversion within the tested spaces, but you still need careful handling of absolute perceived loudness and training data availability.

Jane: So that’s the core idea of "Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training." It shows that we can condition generative models on physical units like decibels, which is a big step for controlling how speech sounds in different environments.

Lu: It’s a useful demonstration of how conditional generation can be used to map one acoustic attribute onto another continuous scale.

Meng: We need more data diversity if we want this to be truly robust for varied production needs.

Lalam: I'm optimistic, though, because the perceived level increases with the target level in their evaluation, which suggests it captures that human perception well on a relative basis.

Tom: Yeah, so while it's not a perfect solution yet—especially with those data constraints—it shows the mechanics of training these conditional GANs for specific acoustic tasks are getting much more powerful when we condition on physical units. Next up, we’re looking at how LLMs handle alignment and cultural knowledge.

Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard

Universite Paris-Saclay · Univ. Grenoble Alpes

eess.AS, cs.LG, cs.SD

Submitted: 2026-10-09

Updated: 2026-10-09

Comments: 5 pages, 2 figures, 1 table. Submitted to IEEE ICASSP 2027. Code: https://github.com/vers-project/vocal-intensity-conversion. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Code: https://github.com/vers-project/vocal-intensity-conversion

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist: A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder

Key concepts

Vocal Intensity Definition: Vocal intensity is defined as sound pressure level (SPL) measured at one meter from the speaker's mouth and is considered the main measure of vocal effort. Existing systems control this through fixed categories or reference recordings, which this paper seeks to improve upon by using a continuous dB SPL target.
Conversion Methodology: The conversion uses a frame-synchronous residual converter Cθ that transforms encoded speech frames. This ensures the output maintains the original sequence alignment and duration, making it possible to alter intensity without changing how fast or long the speech is spoken.
Training Objective: The converter Cθ is trained as a conditional GAN using an objective function (LD) that combines adversarial loss, cycle consistency loss, and identity anchor loss. This training forces the model to generate speech that sounds realistic at the target intensity while staying close to the original content.
Feature Spaces Comparison: The study tested three different feature spaces: log-mel spectrograms (acoustic representation), SpeechTokenizer's latent space (neural codec latent space), and WavLM layer 6 (self-supervised phonetic content). The results indicated that WavLM L6 provided the best performance, achieving 83% of the target intensity change.

Terminology

Adversarial Training — 1× in this paper · explained in 1 episode

Summary

The gist: A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech.

Vocal Intensity Definition and Problem

The paper defines vocal intensity as the sound pressure level (SPL) of a voice measured at one meter from the speaker’s mouth and notes it is the principal measurable correlate of vocal effort. Existing systems control intensity through a reference utterance or discrete categories. The authors propose a conversion model that renders a recorded utterance at a target vocal intensity while preserving its content and speaker identity. They depart from existing literature by conditioning on the equivalent continuous level of the utterance in dB SPL referred to 1 m, rather than a reference recording that already carries the desired effort [9] or a noise category [11] and we measure whether a target level was achieved, which neither system can do since it lacks a target in physical units.

Conversion Methodology

The conversion process is formulated by transforming the encoded frames using the converter Cθ such that xˆ = DecCθ(Enc(x), τt) where Enc and Dec are frozen throughout (Fig. 1) and only Cθ is trained. The converter Cθ is designed to be frame-synchronous, emitting one output frame per input frame, so conversion preserves the alignment of the sequence and cannot alter duration or speaking rate. Furthermore, Cθ is residual, defined as Cθ(z, τt) = z + ∆θ(z, τt), where ∆θ predicts the correction from the whole sequence.

Training Objective

The converter Cθ is trained as the generator of a conditional GAN. The training involves an objective function LD which includes adversarial loss, cycle consistency loss, and an identity anchor loss where the three weights give the positive and the two negatives equal total mass and the generator minimizes LG = E (Dψ(zˆ, τt) − 1)2 (adversarial)> + E Cθ(zˆ, τs) − z squared (cycle)> + E Cθ(z, τs) − z squared (identity)> + E Cθ(zˆ, τs) − z squared (cycle)> + E Cθ(z, τs) − z squared (identity)>>. The discriminator Dψ is conditioned on the target level τ by targeting the joint distribution p(z, τ).

Experimental Setup and Results

The study compares three different feature spaces for the frozen encoder/decoder pair: "the log-mel spectrogram is an acoustic representation, typically used in voice conversion; SpeechTokenizer’s is the latent space of a neural codec; and WavLM layer 6 is a self-supervised representation closely related to phonetic content. The evaluation protocol involves comparing conversions with the result of passing them through the encoder/decoder pair without conversion (resynthesis), and the conversions. The results show that WavLM L6 achieves 83% of the target intensity change. Perceptually, the perceived level increases with the target over the whole range, with a slope of 0.69 (95% interval 0.60–0.78) and the conversions thus retain most of the speaker similarity found between original recordings".

Conclusion and Limitations

The work concludes that WavLM L6 achieves 83% of the target intensity change, and the perceived level of its conversions increases with the target level. The paper identifies three limitations: First, the model controls vocal intensity relative to the source recording but is less consistent in reaching an absolute perceived level. Second, the conversions displace f0 and the spectral measures to a lesser extent than the original recordings do, which leaves room for improvement. Third, calibrated corpora covering a wide range of vocal intensities are scarce, which limits the training data. Future work suggests that a hybrid feature set with a dedicated decoder could combine their strengths and estimating the training source levels with an automatic intensity predictor would address the third limitation <ref:

Improvements for AI systems

Continuous Vocal Intensity Conversion via Conditional Cycle-Consistent GAN: The system can render a recorded utterance at any target sound pressure level (SPL) on a continuous dB scale while preserving its content and speaker identity, moving beyond discrete control methods like LombardTokenizer or PGD-N2L.
Feature Space Comparison for Representation Selection: By training the same converter architecture across three different frozen encoder/decoder pairs (log-mel spectrogram, WavLM L6, SpeechTokenizer), researchers can determine which representation space is best suited for specific acoustic attributes, as the paper notes No feature space therefore leads on every measure.
Objective Intensity Achievement Protocol: The introduction of an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus allows for quantitative assessment of conversion success, using a linear regression approach where A slope of 1 means the target change was fully realized.
Perceptual Intensity Control: Perceptual evaluation demonstrates that perceived vocal intensity increases with the target level on amplitude-normalized stimuli, confirming that the model controls relative vocal intensity well, even though its absolute perceived level depends on the source recording.

Abstract

Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.