FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

[2204.09934] FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

Rongjie Huang 1 1 1 1 Work done during internship at Tencent AI Lab 2 2 2 Equal contribution

  
Max W. Y. Lam 2 2 2 footnotemark: 2

  
Jun Wang 2

  
Dan Su 2

  
Dong Yu 3

  
Yi Ren 1

  
Zhou Zhao 1 3 3 3 Corresponding author
1 Zhejiang University   2 Tencent AI Lab, China   3 Tencent AI Lab, USA {rongjiehuang, rayeren, zhouzhao}@zju.edu.cn, {maxwylam, joinerwang, dansu, dyu}@tencent.com

Abstract

Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthesis. This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. A noise schedule predictor is also adopted to reduce the sampling steps without sacrificing the generation quality. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms without any intermediate feature (e.g., Mel-spectrogram). Our evaluation of FastDiff demonstrates the state-of-the-art results with higher-quality (MOS 4.28) speech samples. Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time. We further show that FastDiff generalized well to the mel-spectrogram inversion of unseen speakers, and FastDiff-TTS outperformed other competing methods in end-to-end text-to-speech synthesis. 4 4 4 Audio samples are available at https://FastDiff.github.io/ .

1

Introduction

With the recent development of deep generative models, speech synthesis has seen an extraordinary progress. Among the conventional speech synthesis methods, WaveNets  Oord et al. ( 2016 ) were demonstrated to generate high-fidelity audio samples in an autoregressive manner yet suffering from prohibitively expensive computational costs. In contrast, non-autoregressive approaches such as flow-based and GAN-based models  Prenger et al. ( 2019 ); Jang et al. ( 2021 ); Kong et al. ( 2020a ); Huang et al. ( 2021 ) were also proposed to generate speech audios with satisfactory speed. However, these models were still criticized for other problems, e.g., the limited sample quality or sample diversity Xiao et al. ( 2021 ) .

In speech synthesis, our goal is mainly two-fold:

•
High-quality: generating high-quality speech is a challenging problem especially when the sampling rate of an audio is high. It is vital to reconstruct details at different timescales for waveforms of highly variable patterns.

•
Fast: high generation speed is essential when considering real-time speech synthesis. This poses a challenge for all high-quality neural synthesizers.

As a blossoming class of generative models, denoising diffusion probabilistic models (DDPMs)  Ho et al. ( 2020 ); Song et al. ( 2020a ); Lam et al. ( 2022 ); Liu et al. ( 2022 ) has emerged to prove its capability to achieve leading performances in both image and audio syntheses  Dhariwal and Nichol ( 2021 ); San-Roman et al. ( 2021 ); Kong et al. ( 2020b ); Chen et al. ( 2020 ); Lam et al. ( 2022 ) . However, current development of DDPMs in speech synthesis was hampered by two major challenges:

•
Different from other existing generative models, diffusion models are not trained to directly minimize the difference between the generated audio and the reference audio, but to de-noise a noisy sample given an optimal gradient. This in practice could lead to overly de-noised speech after a large number of sampling steps, in which natural voice characteristics including breathiness and vocal fold closure are removed.

•
While DDPMs inherently are gradient-based models, a guarantee of high sample quality typically comes at a cost of hundreds to thousands of de-noising steps. When reducing the sampling steps, an apparent degradation in quality due to perceivable background noise is observed.

In this work, we propose FastDiff, a fast conditional diffusion model for high-quality speech synthesis. To improve audio quality, FastDiff adopts a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. To accelerate the inference procedure, FastDiff also includes a noise schedule predictor, which derives a short and effective noise schedule and significantly reduces the de-noising steps. Based on FastDiff, we also introduce an end-to-end phoneme-to-waveform synthesizer FastDiff-TTS, which simplifies the text-to-speech generation pipeline and does not require intermediate features or specialized loss functions to enjoy low inference latency.

Experimental results demonstrated that FastDiff achieved a higher MOS score than the best publicly available models and outperformed the strong WaveNet vocoder (MOS: 4.28 vs. 4.20). FastDiff further enjoys an effective sampling process and only needs 4 iterations to synthesize high-fidelity speech, 58x faster than real-time on a V100 GPU without engineered kernels. To the best of our knowledge, FastDiff is the first diffusion model with a sampling speed comparable to previous for the first time applicable to interactive, real-world speech synthesis applications at a low computational cost. FastDiff-TTS successfully simplify the text-to-speech generation pipeline and outperform competing architectures.

2

Background: Denoising Diffusion Probabilistic Models

Denoising diffusion probabilistic models (DDPMs)  Ho et al. ( 2020 ); Song et al. ( 2020a ); Lam et al. ( 2022 ) are likelihood-based generative models that have recently succeeded to advance the state-of-the-art results in benchmark generative tasks Dhariwal and Nichol ( 2021 ) and have proved its capability to produce high-quality samples. The basic idea of DDPMs is to train a gradient neural network for reversing a diffusion process. Given i.i.d. samples { 𝒙 0 ∈ ℝ D } subscript 𝒙 0 superscript ℝ 𝐷 {{\bm{x}}{0}\in\mathbb{R}^{D}} from an unknown data distribution p d ​ a ​ t ​ a ​ ( 𝒙 0 ) subscript 𝑝 𝑑 𝑎 𝑡 𝑎 subscript 𝒙 0 p{data}({\bm{x}}{0}) , DDPMs try to approximate p d ​ a ​ t ​ a ​ ( 𝒙 0 ) subscript 𝑝 𝑑 𝑎 𝑡 𝑎 subscript 𝒙 0 p{data}({\bm{x}}{0}) by a marginal distribution p θ ​ ( 𝒙 0 ) = ∫ ⋯ ​ ∫ p ​ ( 𝒙 T ) ​ ∏ t = 1 T p θ ​ ( 𝒙 t − 1 | 𝒙 t ) ​ d ​ 𝒙 1 ​ … ​ d ​ 𝒙 T subscript 𝑝 𝜃 subscript 𝒙 0 ⋯ 𝑝 subscript 𝒙 𝑇 superscript subscript product 𝑡 1 𝑇 subscript 𝑝 𝜃 conditional subscript 𝒙 𝑡 1 subscript 𝒙 𝑡 𝑑 subscript 𝒙 1 … 𝑑 subscript 𝒙 𝑇 p{\theta}({\bm{x}}{0})=\int\cdots\int p({\bm{x}}{T})\prod_{t=1}^{T}p_{\theta}({\bm{x}}{t-1}|{\bm{x}}{t})d{\bm{x}}{1}\ldots d{\bm{x}}{T} .

In data distribution as q ​ ( 𝒙 0 ) 𝑞 subscript 𝒙 0 q({\bm{x}}{0}) , the diffusion process is defined by a fixed Markov chain from data 𝒙 0 subscript 𝒙 0 {\bm{x}}{0} to the latent variable 𝒙 T subscript 𝒙 𝑇 {\bm{x}}{T} . For a small positive constant β t subscript 𝛽 𝑡 \beta{t} , a small Gaussian noise is added from 𝒙 t subscript 𝒙 𝑡 {\bm{x}}{t} to the distribution of 𝒙 t − 1 subscript 𝒙 𝑡 1 {\bm{x}}{t-1} under the function of q ​ ( 𝒙 t | 𝒙 t − 1 ) 𝑞 conditional subscript 𝒙 𝑡 subscript 𝒙 𝑡 1 q({\bm{x}}{t}|{\bm{x}}{t-1}) . The whole process gradually converts data 𝒙 0 subscript 𝒙 0 {\bm{x}}{0} to whitened latents 𝒙 T subscript 𝒙 𝑇 {\bm{x}}{T} according to the fixed noise schedule β 1 , ⋯ , β T subscript 𝛽 1 ⋯ subscript 𝛽 𝑇 \beta_{1},\cdots,\beta_{T} . The reverse process is a Markov chain from 𝒙 T subscript 𝒙 𝑇 {\bm{x}}{T} to 𝒙 0 subscript 𝒙 0 {\bm{x}}{0} parameterized by a shared θ 𝜃 \theta , which aims to recover samples from Gaussian noises though eliminating the Gaussian noise added in the diffusion process in each iteration.

It has been demonstrated that diffusion probabilistic models  Dhariwal and Nichol ( 2021 ); Xiao et al. ( 2021 ) can learn diverse data distribution in multiple domains, such as images and time series. While the main issue with the proposed neural diffusion process is that it requires up to thousands of iterative steps to reconstruct the target distribution during reverse sampling. In this work, we offer a fast conditional diffusion model to reduce reverse iterations and improve computational efficiency.

Figure 1: The overall architecture for FastDiff and FastDiff-TTS. The refinement model θ 𝜃 \theta in FastDiff takes noisy audio 𝒙 t subscript 𝒙 𝑡 {\bm{x}}{t} as input and computes ϵ θ ​ ( 𝒙 t | c , t ) subscript bold-italic-ϵ 𝜃 conditional subscript 𝒙 𝑡 𝑐 𝑡 \bm{\epsilon}{\theta}({\bm{x}}_{t}|c,t) conditioned on diffusion time index t 𝑡 t and Mel-spectrogram c 𝑐 c . We use LReLU to denote the leaky rectified linear unit, LVC to denote the location-variable convolution, FC to denote the fully-connected layer, and PE to denote the positional encoding operation.

3

FastDiff

This section presents our proposed FastDiff, a fast conditional diffusion model for high-quality speech synthesis. We first describe the motivation of the design in FastDiff. Secondly, we introduce the iterative refinement model θ 𝜃 \theta for high-quality speech synthesis and the noise predictor ϕ italic-ϕ \phi for accelerated sampling. Furthermore, we describe the training and inference procedures in detail. At last, we extend FastDiff to FastDiff-TTS for fully end-to-end text-to-speech syntheses.

3.1

Motivation

While denoising diffusion probabilistic models have shown high potential in synthesizing high-quality speech samples Chen et al. ( 2020 ); Kong et al. ( 2020b ); Liu et al. ( 2021 ) , several challenges remain for industrial deployment: 1) Different from the traditional generative models, diffusion models catch dynamic dependencies from noisy audio instead of clean ones, which introduce more variation information (i.e, noise levels) in addition to the spectrogram fluctuation. 2) With limited receptive field patterns, a distinct degradation could emerge when reducing the reverse iterations, making diffusion models difficult to get accelerated. As a result, hundred or thousand orders of iterations prevent existing diffusion models from real-world deployment.

In FastDiff, we propose two key components to complement the above issues: 1) FastDiff adopts a time-aware location-variable convolution to catch the details of noisy samples at dynamic dependencies. The convolution operations are conditioned on dynamic variations in speech including diffusion steps and spectrogram fluctuations, equipping the model with diverse receptive field patterns and promoting the robustness of diffusion models during reverse acceleration. 2) To accelerate the inference procedure, FastDiff adopts a noise schedule predictor to reduce the number of reverse iterations, frees diffusion models from hundreds or thousands of refinement iterations. This makes FastDiff for the first time applicable to interactive, real-world applications at a low computational cost.

3.2

Time-Aware Location-Variable Convolution

In comparison with traditional convolution networks, location-variable convolution  Zeng et al. ( 2021 ) shows efficiency in modeling the long-term dependency of audio and gets neural network free from a significant number of dilated convolution layers. Inspired by this, we introduce the Time-Aware Location-Variable Convolution, which is sensitive to time steps in diffusion probabilistic models. At time step t 𝑡 t , we follow  Vaswani et al. ( 2017 ) to embed the step index into an 128-dimensional positional encoding (PE) vector 𝒆 t subscript 𝒆 𝑡 {\bm{e}}_{t} :

  𝒆  t   =          subscript  𝒆  𝑡   absent    \displaystyle{\bm{e}}_{t}=    
 [  sin   (   10    0  ×  4   63    t  )   ,  ⋯  ,  sin   (   10    63  ×  4   63    t  )   ,   \displaystyle\left[\sin\left(10^{\frac{0\times 4}{63}}t\right),\cdots,\sin\left(10^{\frac{63\times 4}{63}}t\right),\right.    





 cos   (   10    0  ×  4   63    t  )   ,  ⋯  ,  cos   (   10    63  ×  4   63    t  )   ]  ,   \displaystyle\,\,\left.\cos\left(10^{\frac{0\times 4}{63}}t\right),\cdots,\cos\left(10^{\frac{63\times 4}{63}}t\right)\right],    

In time-aware location-variable convolution, FastDiff requires multiple predicted variation-sensitive kernels to perform convolutional operations on the associated intervals of input sequence. These kernels should be time-aware and sensitive to variations of noisy audio including diffusion steps and acoustic features (i.e., Mel-spectrogram). Therefore, we propose a time-aware location-variable convolution (LVC) module, which is coupled with a kernel predictor as shown in Figure  1 (b) and Figure  1 (c). We describe the overall calculations below.

For the q 𝑞 q -th time-aware LVC layer, we split the input 𝒙 t ∈ ℝ D subscript 𝒙 𝑡 superscript ℝ 𝐷 {\bm{x}}{t}\in{\mathbb{R}}^{D} using a M 𝑀 M -length window with 3 q superscript 3 𝑞 3^{q} dilations to produce K 𝐾 K segments with each 𝒙 t k ∈ ℝ M superscript subscript 𝒙 𝑡 𝑘 superscript ℝ 𝑀 {\bm{x}}{t}^{k}\in{\mathbb{R}}^{M} :

  {   𝒙  t  1   ,  …  ,   𝒙  t  K   }   =   split  ⁡   (   𝒙  t   ;  M  ,  q  )           superscript   subscript  𝒙  𝑡   1   …   superscript   subscript  𝒙  𝑡   𝐾     split   subscript  𝒙  𝑡   𝑀  𝑞     \{{\bm{x}}_{t}^{1},\ldots,{\bm{x}}_{t}^{K}\}=\operatorname{split}({\bm{x}}_{t};M,q)    

(1)

Next, we perform convolutional operations on the associated intervals of input sequence using the kernels generated by a kernel predictor α 𝛼 \alpha :

 {   𝑭  t   ,   𝑮  t   }      subscript  𝑭  𝑡    subscript  𝑮  𝑡     \displaystyle\{{\bm{F}}_{t},{\bm{G}}_{t}\}    
   =   α  ​   (  t  ,  c  )         absent     𝛼   𝑡  𝑐      \displaystyle=\alpha(t,c)    

(2)

 𝒛  t  k     superscript   subscript  𝒛  𝑡   𝑘    \displaystyle\bm{z}_{t}^{k}    
   =     tanh  ⁡   (    𝑭  t   ∗   𝒙  t  k    )    ⊙  σ   ​   (    𝑮  t   ∗   𝒙  t  k    )         absent      direct-product         subscript  𝑭  𝑡    superscript   subscript  𝒙  𝑡   𝑘     𝜎       subscript  𝑮  𝑡    superscript   subscript  𝒙  𝑡   𝑘       \displaystyle=\tanh(\bm{F}_{t}*\bm{x}_{t}^{k})\odot\sigma(\bm{G}_{t}*\bm{x}_{t}^{k})    

(3)

 𝒛  t     subscript  𝒛  𝑡    \displaystyle\bm{z}_{t}    
    =   concat  ⁡   (   {   𝒛  t  1   ,  …  ,   𝒛  t  K   }   )     ,       absent   concat    superscript   subscript  𝒛  𝑡   1   …   superscript   subscript  𝒛  𝑡   𝐾       \displaystyle=\operatorname{concat}(\{\bm{z}_{t}^{1},\ldots,\bm{z}_{t}^{K}\}),    

(4)

where 𝑭 t , 𝑮 t subscript 𝑭 𝑡 subscript 𝑮 𝑡 \bm{F}{t},\bm{G}{t} denote the filter and the gate kernels for 𝒙 t i superscript subscript 𝒙 𝑡 𝑖 {\bm{x}}_{t}^{i} , respectively, ∗ * denotes the 1d convolution, ⊙ direct-product \odot denotes the element-wise product and concat ⁡ ( ⋅ ) concat ⋅ \operatorname{concat}(\cdot) denotes the concatenation between vectors. Since the time-aware kernels are adaptive to the noise-level and dependent to the acoustic features, FastDiff is capable of precisely estimating de-noising gradient with a superior speed given a noisy signal input.

3.3

Accelerated Sampling

3.3.1

Noise Predictor

To avoid sampling with hundreds to thousands steps, FastDiff adopts the noise scheduling algorithm in the bilateral denoising diffusion models (BDDMs)  Lam et al. ( 2022 ) to predict a sampling noise schedule much shorter than the noise schedule used in training. This scheduling method has been revealed to be superior than other sampling acceleration methods, e.g., the grid search algorithm in WaveGrad  Chen et al. ( 2020 ) and the fast sampling algorithm in DiffWave  Kong et al. ( 2020b ) . The noise predictor iteratively derives a continuous noise schedule 𝜷 ^ ∈ ℝ T m ^ 𝜷 superscript ℝ subscript 𝑇 𝑚 \hat{{\bm{\beta}}}\in{\mathbb{R}}^{T_{m}} . We attach the learning objective and corresponding likelihood in Appendix  A .

3.3.2

Schedule Alignment

In FastDiff, similar to DDPMs, during training we use T = 1000 𝑇 1000 T=1000 discrete time steps. Therefore, when needed to condition on t 𝑡 t during sampling, we also need to approximate T m subscript 𝑇 𝑚 T_{m} discrete time indices by aligning the T m subscript 𝑇 𝑚 T_{m} -step sampling noise schedule 𝜷 ^ ^ 𝜷 \hat{{\bm{\beta}}} to the T 𝑇 T -step training noise schedule 𝜷 𝜷 {\bm{\beta}} , with N

3.4

Training, Noise Scheduling and Sampling

All illustrated in Algorithm  1 , we separately parameterize FastDiff by two modules: 1) a iterative refinement model θ 𝜃 \theta that minimizes a variational bound of the score function, and 2) a noise predictor ϕ italic-ϕ \phi that optimizes the noise schedule for a tighter evidence lower bound. For inference, we first derive the tighter and more efficient noise schedules β ^ ^ 𝛽 \hat{\beta} via an one-shot noise scheduling procedure, which makes FastDiff achieve orders of magnitude faster at sampling. It has been demonstrated  Lam et al. ( 2022 ) that the noise schedule searched for as few as 1 sample could be robust enough to maintain a high-quality generation among all samples in testing set. Secondly, we map the continuous noise schedules to discrete time indexes T m subscript 𝑇 𝑚 T_{m} using schedule alignment. Finally, FastDiff iteratively refines gaussian noise to generate high-quality samples with computational efficiency. The detailed information on training, noise scheduling and inference procedures has been presented in Appendix  C.3 .

3.5

FastDiff-TTS

Existing text-to-speech methods usually adopt a two-stage pipeline: 1) A text-to-spectrogram generation module (a.k.a. acoustic model) aims to generate prosodic attributes according to variance prediction; 2) A conditional waveform generation module (a.k.a. vocoder) adds the phase information and synthesizes a detailed waveform. To further simplify the text-to-speech synthesis pipeline, we propose a fully end-to-end model FastDiff-TTS, which does not require intermediate features or specialized loss functions. FastDiff-TTS is designed to be a fully differentiable and efficient architecture that directly produces waveforms from contexts (e.g. phonemes) without needing to generate acoustic features (e.g., Mel-spectrograms) explicitly.

Algorithm 1 Training refinement network θ 𝜃 \theta

1:    Input : Pre-defined noise schedule β 𝛽 \beta

2:    repeat

3:      Sample 𝒙 0 ∼ q d ​ a ​ t ​ a similar-to subscript 𝒙 0 subscript 𝑞 𝑑 𝑎 𝑡 𝑎 {\bm{x}}{0}\sim q{data} , ϵ ∼ 𝒩 ​ ( 𝟎 , 𝑰 ) similar-to bold-italic-ϵ 𝒩 0 𝑰 {\bm{\epsilon}}\sim{\mathcal{N}}({\bm{0}},{\bm{I}}) , and t ∼ Unif ​ ( { 1 , ⋯ , T } ) similar-to 𝑡 Unif 1 ⋯ 𝑇 t\sim\mathrm{Unif}({1,\cdots,T})

4:       𝒙 t = α t ​ 𝒙 0 + 1 − α t 2 ​ ϵ subscript 𝒙 𝑡 subscript 𝛼 𝑡 subscript 𝒙 0 1 superscript subscript 𝛼 𝑡 2 bold-italic-ϵ {\bm{x}}{t}=\alpha{t}{\bm{x}}{0}+\sqrt{1-\alpha{t}^{2}}{\bm{\epsilon}}

5:      Take gradient descent steps on ∇ θ ∥ ϵ − ϵ θ ( 𝒙 t | c , t ) ∥ 2 2 \nabla_{\theta}\left|{\bm{\epsilon}}-{\bm{\epsilon}}{\theta}\left({\bm{x}}{t}|c,t\right)\right|_{2}^{2}

6:    until  refinement model θ 𝜃 \theta converged

Algorithm 2 Training noise predictor ϕ italic-ϕ \phi

1:    Input : Pre-defined discrete β 𝛽 \beta , trained refinement network θ 𝜃 \theta , hyperparameter τ 𝜏 \tau .

2:    repeat

3:      Sample 𝒙 0 ∼ q d ​ a ​ t ​ a similar-to subscript 𝒙 0 subscript 𝑞 𝑑 𝑎 𝑡 𝑎 {\bm{x}}{0}\sim q{data} , ϵ ∼ 𝒩 ​ ( 𝟎 , 𝑰 ) similar-to bold-italic-ϵ 𝒩 0 𝑰 {\bm{\epsilon}}\sim{\mathcal{N}}({\bm{0}},{\bm{I}}) , and t ∼ Unif ​ ( { τ , ⋯ , T − τ } ) similar-to 𝑡 Unif 𝜏 ⋯ 𝑇 𝜏 t\sim\mathrm{Unif}({\tau,\cdots,T-\tau})

4:       𝒙 t = α t ​ 𝒙 0 + 1 − α t 2 ​ ϵ subscript 𝒙 𝑡 subscript 𝛼 𝑡 subscript 𝒙 0 1 superscript subscript 𝛼 𝑡 2 bold-italic-ϵ {\bm{x}}{t}=\alpha{t}{\bm{x}}{0}+\sqrt{1-\alpha{t}^{2}}{\bm{\epsilon}}

5:       β ^ t = min ⁡ { 1 − α t 2 , 1 − α t + τ 2 α t 2 } ​ ϕ ​ ( 𝒙 t ) subscript ^ 𝛽 𝑡 1 superscript subscript 𝛼 𝑡 2 1 superscript subscript 𝛼 𝑡 𝜏 2 superscript subscript 𝛼 𝑡 2 italic-ϕ subscript 𝒙 𝑡 \hat{\beta}{t}=\min\left{1-\alpha{t}^{2},1-\frac{\alpha_{t+\tau}^{2}}{\alpha_{t}^{2}}\right}\phi({\bm{x}}_{t})

6:      Take gradient descent steps on ∇ ϕ ​ { δ t 2 2 ​ ( δ t 2 − β ^ t ) ‖ ϵ − β ^ t δ t 2 ​ ϵ θ ​ ( 𝒙 t | c , t ) ∥ 2 2 } subscript ∇ italic-ϕ conditional-set superscript subscript 𝛿 𝑡 2 2 superscript subscript 𝛿 𝑡 2 subscript ^ 𝛽 𝑡 italic-ϵ evaluated-at subscript ^ 𝛽 𝑡 superscript subscript 𝛿 𝑡 2 subscript italic-ϵ 𝜃 conditional subscript 𝒙 𝑡 𝑐 𝑡 2 2 \nabla_{\phi}\left{\frac{\delta_{t}^{2}}{2\left(\delta_{t}^{2}-\hat{\beta}{t}\right)}\left|\epsilon-\frac{\hat{\beta}{t}}{\delta_{t}^{2}}\epsilon_{\theta}\left(\bm{x}{t}|c,t\right)\right|{2}^{2}\right}

7:    until  noise predictor ϕ italic-ϕ \phi converged

Algorithm 3 Sampling

1:    Input : Searched β ^ ^ 𝛽 \hat{\beta} in noise scheduling process.

2:   Compute discrete steps T m subscript 𝑇 𝑚 T_{m} sequences via schedule alignment in Section  3.3.2 .

3:   Sample 𝒙 T m ∼ 𝒩 ​ ( 𝟎 , 𝑰 ) similar-to subscript 𝒙 subscript 𝑇 𝑚 𝒩 0 𝑰 {\bm{x}}{T{m}}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})

4:    for   t = T m , ⋯ , 1 𝑡 subscript 𝑇 𝑚 ⋯ 1 t=T_{m},\cdots,1   do

5:      Sample 𝒙 t − 1 ∼ p θ ​ ( 𝒙 t − 1 | 𝒙 t ; β ^ ) similar-to subscript 𝒙 𝑡 1 subscript 𝑝 𝜃 conditional subscript 𝒙 𝑡 1 subscript 𝒙 𝑡 ^ 𝛽 {\bm{x}}{t-1}\sim p{\theta}({\bm{x}}{t-1}|{\bm{x}}{t};\hat{\beta})

6:    end   for

7:    return   𝒙 0 subscript 𝒙 0 {\bm{x}}_{0}

3.5.1

Architecture

The architecture design of FastDiff-TTS refers to a convectional non-autoregressive text-to-speech model – FastSpeech 2  Ren et al. ( 2020 ) as the backbone. The architecture of FastDiff-TTS is illustrated in Figure  1 (d). In FastDiff-TTS, the encoder first converts the phoneme embedding sequence into the phoneme hidden sequence. Then, the duration predictor expands the encoder output to match the length of the desired waveform output. Given the aligned sequence, the variance adaptor adds pitch information into the hidden sequence. Note that it is difficult to use the full audio corresponding to the full text sequence for training due to the typically high sampling rate for high-fidelity waveform (i.e., 24,000 samples per second) and the limited GPU memory. Therefore, we sample a small segment to synthesize the waveform before passing to the FastDiff model. Finally, the FastDiff model decodes the adapted hidden sequence into a speech waveform as in the vocoder task.

3.5.2

Training Loss

FastDiff-TTS does not require specialized loss functions and adversarial training to improve sample quality as suggested by the previous works  Ren et al. ( 2020 ); Donahue et al. ( 2020 ); Kim et al. ( 2021 ) . This, to a large extend, simplifies the text-to-speech generation. The final training loss consists of the following terms: 1) a duration prediction loss L dur subscript 𝐿 dur L_{\text{dur}} : the mean squared error between the predicted and the ground-truth word-level duration in log-scale, 2) a diffusion loss L diff subscript 𝐿 diff L_{\text{diff}} : the mean squared error between the estimated and gaussian noise, and 3) a pitch reconstruction loss L pitch subscript 𝐿 pitch L_{\text{pitch}} : the mean squared error between the predicted and the ground-truth pitch sequences. We empirically found that the pitch reconstruction loss L pitch subscript 𝐿 pitch L_{\text{pitch}} is helpful for handling the one-to-many mapping issue in text-to-speech generation.

Figure 2: Conditional Diffusion Model for Speech Synthesis

4

Related Works

Text-to-speech (TTS) systems aim to synthesize raw speech waveforms from given text. In recent years, Neural network based TTS  Ren et al. ( 2020 ); Kim et al. ( 2020 ); Liu et al. ( 2021 ) has made huge progress and attracted a lot of attention in the machine learning and speech community.

Neural vocoder plays the most important role in the recent success of speech synthesis, which require diverse receptive field patterns to catch audio dependencies: 1) autoregressive model WaveNet  Oord et al. ( 2016 ) requires causal convolutions layers and large filters to increase the receptive field while suffering from slow inference speed. 2) Flow-based generative models  Prenger et al. ( 2019 ) fully utilize modern parallel computing processors to broaden corresponding receptive fields and speed-up sampling, while they usually achieve a limited sample quality. 3) Generative adversarial networks (GANs)  Jang et al. ( 2021 ); Kong et al. ( 2020a ) are one of the most dominant deep generative models in audio generation. UnivNet  Jang et al. ( 2021 ) has demonstrated its success in using local-variable convolution on different waveform intervals, and HIFI-GAN  Kong et al. ( 2020a ) proposes multi-receptive field fusion (MRF) to model the periodic patterns matters. However, GAN-based models are often difficult to train, collapsing  Creswell et al. ( 2018 ) without carefully selected hyperparameters and regularizers, and showing less sample diversity. 4) Recently proposed diffusion models Diffwave  Kong et al. ( 2020b ) and WaveGrad  Chen et al. ( 2020 ) could generate high-quality speech samples, while suffering from a distinct degradation when reducing reverse iterations, making diffusion models difficult to get accelerated. Different from vocoders mentioned above, FastDiff improves the robustness of conditional diffusion model by catching the details of noisy samples at dynamic dependencies, and reduces reverse iterations with predicted noise schedule. The proposed conditional diffuion model allows the high-quality speech synthesis with computational efficiency.

Another important line of work covers directly waveform generation from text: FastSpeech 2s  Ren et al. ( 2020 ) and VITS  Kim et al. ( 2021 ) adopt adversarial training process and spectral losses for improving audio quality, while they do not take full advantage of end-to-end training. Recently proposed WaveGrad 2  Chen et al. ( 2021 ) estimates the gradient of the log conditional density of the waveform given a phoneme sequence, but suffers from a large model footprint and slow inference. Unlike all of the aforementioned methods, as highlighted in section  3.5 , FastDiff-TTS is a fully differentiable and efficient architecture that produces waveforms directly without generating middle features (e.g., spectrograms) explicitly. In additional, our diffuion probabilistic model gets free from hundred or thousands of iterations and enjoy computational efficiency.

5

Experiments

5.1

Setup

5.1.1

Dataset

For a fair and reproducible comparison against other competing methods, we used the benchmark LJSpeech dataset  Ito ( 2017 ) . LJSpeech consists of 13,100 audio clips of 22050 Hz from a Female speaker with about 24 hours in total. To evaluate the generalization ability of our model over unseen speakers in multi-speaker scenarios, we also used the VCTK dataset  Yamagishi et al. ( 2019 ) , which was downsampled to 22050 Hz to match the sampling rate with the LJSpeech datset. VCTK consists of approximately 44,200 audio clips uttered by 109 native English speakers with various accents. For both datasets, we used 80-band Mel-spectrograms as the condition for the vocoding task. The FFT size, window size, and hop size were, respectively, set to 1024, 1024, and 256.

5.1.2

Model Configurations

FastDiff mainly consists of the refinement model θ 𝜃 \theta and noise schedule predictor ϕ italic-ϕ \phi . The refinement model θ 𝜃 \theta comprises three Diffusion-UBlock and DBlock with the upsample or downsample rate of [ 8 , 8 , 4 ] 8 8 4 [8,8,4] , respectively. We adopt a lightweight GALR network effective in separating the added gaussian noise from audio as the noise schedule predictor ϕ italic-ϕ \phi . For end-to-end text-to-speech generation, FastDiff-TTS follows the basic structure in FastSpeech 2  Ren et al. ( 2020 ) , which consists of 4 feed-forward transformer blocks in the phoneme encoder. More details have been attached in the Appendix  B .

5.1.3

Training and Evaluation

The complete training pipeline has been illustrated in Algorithm  1 : FastDiff was trained with constant learning rate l ​ r = 2 × 10 − 4 𝑙 𝑟 2 superscript 10 4 lr=2\times 10^{-4} . The refinement model θ 𝜃 \theta and noise predictor ϕ italic-ϕ \phi were trained for 1M and 10K steps until convergence, respectively. FastDiff-TTS was trained until 500k steps using the AdamW optimizer with β 1 = 0.9 , β 2 = 0.98 , ϵ = 10 − 9 formulae-sequence subscript 𝛽 1 0.9 formulae-sequence subscript 𝛽 2 0.98 italic-ϵ superscript 10 9 \beta_{1}=0.9,\beta_{2}=0.98,\epsilon=10^{-9} . Both models were trained on 4 NVIDIA V100 GPUs using random short audio clips of 16,000 samples from each utterance with a batch size of 16 each GPU. More details have been attached in the Appendix  C .

We crowd-sourced 5-scale MOS tests via Amazon Mechanical Turk to evaluate the audio quality. The MOS scores were recorded with 95% confidence intervals (CI). Raters listened to the test samples randomly, where they were allowed to evaluate each audio sample once. We further include additional objective evaluation metrics including STOI and PESQ to test sample quality. To evaluate the sampling speed, we implemented real-time factor (RTF) accessment on a single NVIDIA V100 GPU. In addition, we employed two metrics NDB and JSD to explore the diversity of generated mel-spectrograms. More information about both objective and subjective evaluation has been attached in Appendix  D .

Model
Quality
Speed
Diversity

MOS ( ↑ ↑ \uparrow )

STOI ( ↑ ↑ \uparrow )

PESQ ( ↑ ↑ \uparrow )

RTF ( ↓ ↓ \downarrow )

NDB ( ↓ ↓ \downarrow )

JSD ( ↓ ↓ \downarrow )

GT

4.52 ± plus-or-minus \pm 0.09

/
/
/
/
/

WaveNet (MOL)

4.20 ± plus-or-minus \pm 0.06

/
/
85.230
33
0.002

WaveGlow

3.89 ± plus-or-minus \pm 0.07

0.961
3.16
0.029
66
0.014

HIFI-GAN

4.08 ± plus-or-minus \pm 0.08

0.956
3.28
0.002
72
0.010

UnivNet

4.13 ± plus-or-minus \pm 0.09

0.971
3.45
0.002
68
0.013

Diffwave (6 steps)

4.18 ± plus-or-minus \pm 0.08

0.966
3.62
0.093
72
0.007

WaveGrad (50 steps)

4.09 ± plus-or-minus \pm 0.07

0.911
2.70
0.390
61
0.008

FastDiff (4 steps)
4.28 ± plus-or-minus \pm 0.07
0.976
3.71
0.017
49
0.006

Table 1: Comparison with other nerual vocoders in terms of quality, synthesis speed and sample diversity. For sampling, we used 50 steps in WaveGrad and 6 steps in DiffWave, respectively

Model
MOS

GT

4.37 ± plus-or-minus \pm 0.06

WaveNet (MOL)

4.01 ± plus-or-minus \pm 0.08

WaveGlow

3.66 ± plus-or-minus \pm 0.08

HIFI-GAN

3.74 ± plus-or-minus \pm 0.06

UnivNet

3.85 ± plus-or-minus \pm 0.07

Diffwave (6 steps)

3.90 ± plus-or-minus \pm 0.07

WaveGrad (50 steps)

3.72 ± plus-or-minus \pm 0.06

FastDiff (4 steps)
4.10 ± plus-or-minus \pm 0.06

Table 2: Comparison with other neural vocoders of synthesized utterances for unseen speakers.

Model

MOS ( ↑ ↑ \uparrow )

STOI( ↑ ↑ \uparrow )

PESQ ( ↑ ↑ \uparrow )

RTF ( ↓ ↓ \downarrow )

GT

4.52 ± plus-or-minus \pm 0.09

/
/
/

w/o Time-aware LVC

4.08 ± plus-or-minus \pm 0.05

0.971
3.45
0.081

w/o Noise Predictor

4.10 ± plus-or-minus \pm 0.06

0.968
3.50
0.033

Continuous, 4 steps

4.09 ± plus-or-minus \pm 0.08

0.970
3.37
0.015

Continuous, 1000 steps

4.14 ± plus-or-minus \pm 0.07

0.980
3.64
3.80

Discrete, 4 steps

4.28 ± plus-or-minus \pm 0.07

0.976
3.71
0.017

Discrete, 1000 steps
4.36 ± plus-or-minus \pm 0.08
0.989
3.86
4.70

Table 3: Ablation study results. Comparison of the effect of each component in terms of quality and synthesis speed.

Model
MOS

GT

4.52 ± plus-or-minus \pm 0.09

GT(voc.)

4.28 ± plus-or-minus \pm 0.07

Cascaded

4.13 ± plus-or-minus \pm 0.07

FastSpeech 2s

3.94 ± plus-or-minus \pm 0.06

WaveGrad 2

3.68 ± plus-or-minus \pm 0.09

FastDiff-TTS
4.03 ± plus-or-minus \pm 0.09

Table 4: Comparison with other text-to-speech models in terms of quality.

5.2

Comparsion with other models

We compared our FastDiff in audio quality, diversity, and sampling speed with competing models, including 1) WaveNet  Oord et al. ( 2016 ) , the autoregressive generative model for raw audio. 2) WaveGlow  Prenger et al. ( 2019 ) , non-autoregressive flow-based model. 3) HIFI-GAN V1  Kong et al. ( 2020a ) and UnivNet  Jang et al. ( 2021 ) , the most dominant and popular GAN-based models. 4) Diffwave  Kong et al. ( 2020b ) and WaveGrad  Chen et al. ( 2020 ) , recently proposed diffusion probabilistic models which achieve state-of-the-art in speech synthesis. For easy comparison, the results are compiled and presented in Table  2 , and we have the following observations:

In terms of audio quality, FastDiff achieved the highest MOS with a gap of 0.24 0.24 0.24 compared to the ground truth audio, and it matched the performance of the autoregressive WaveNet baseline and outperformed the non-autoregressive baselines. For objective evaluation, FastDiff also demonstrated a large improvement in PESQ and STOI. For inference speed, with the efficient noise schedules searched by noise predictor, FastDiff could generate high-quality speech samples within as few as 4 4 4 reverse steps, significantly reducing the inference time compared with competing diffusion architectures. To the best of our knowledge, FastDiff makes diffusion models for the first time applicable to interactive, high-quality real-world speech synthesis at a low computational cost. In terms of sample diversity, we can see that FastDiff still witnessed a gap from autoregressive WaveNet, but it achieve a higher variety for generated speeches than non-autoregressive baselines. More detailed evaluation on sample diversity has been attached in Appendix  F .

5.3

Ablation study

We conducted ablation studies to demonstrate the effectiveness of several designs in FastDiff, including the time-aware location variable convolution and noise predictor in neural vocoding. The results of both subjective and objective evaluations have been presented in Table  4 , and we have the following observations: 1) Replacing time-aware location-variable convolution by traditional convolutional operations causes a distinct degradation in sampling speed and perceptual quality. 2) Using grid search instead of the noise predictor to search schedules had witnessed the decreased audio quality, demonstrating that the noise schedule prediction process provides more efficient reverse sampling without sacrificing quality.

Further, we compare two variants of FastDiff to test the modality differences of diffusion condition (i.e., continuous noise-level or discrete time-step). Note that the former model does not require the schedule alignment process mentioned in Section  3.3.2 . We empirically find that the FastDiff model conditioned on discrete time steps could synthesize samples with higher quality, demonstrating that learning proposed FastDiff with discrete diffusion times could be a better choice. More information on the variant of FastDiff extended to continuous noise schedules has been attached in Appendix  E

5.4

Generalization to unseen speakers

We used 50 50 50 randomly selected utterances of 5 5 5 unseen speakers in the VCTK dataset that were excluded from the training set for the MOS test. Table  2 shows the experimental results for the mel-spectrogram inversion of the unseen speakers: In summary, we noticed that FastDiff achieved state-of-the-art in terms of audio quality for out-of-domain generalization, indicating that FastDiff could universally generate high-fidelity audio from entirely new (unseen) speakers outside the train set.

5.5

End-to-End Text-to-Speech

To demonstrate the robustness of the proposed model in end-to-end text-to-speech synthesis, we compare FastDiff-TTS with other neural TTS systems, including 1) GT, the ground truth audio; 2) GT (voc.), where we first convert the ground truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using FastDiff; 3) PortaSpeech  Ren et al. ( 2021 ) + FastDiff: vocoder cascaded with mel-spectrogram generation using the most popular non-autoregressive TTS models; 4) FastSpeech 2s  Ren et al. ( 2020 ) : the extension of FastSpeech 2 to fully end-to-end text-to-waveform generation with multi-task learning; 5) WaveGrad 2  Chen et al. ( 2021 ) : a diffusion probabilistic model to generate waveforms via gradient estimation. The results are shown in Table  4 : FastDiff-TTS could surpass competing end-to-end speech synthesis models and match the voice quality of the state-of-the-art cascaded TTS systems, demonstrating that FastDiff-TTS is efficient in simplifying the overall text-to-speech synthesis pipeline.

6

Conclusion

In this work, we proposed FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employed a stack of time-aware location-variable convolutions with diverse receptive field patterns to model long-term time dependencies with adaptive conditions. A noise predictor was further adopted to derive tighter schedules for reducing reverse iterations without distinct quality degradation. The extension model FastDiff-TTS discarded intermediate features (e.g., spectrograms) and simplified the end-to-end text-to-waveform syntheses pipeline. Experimental results demonstrated that our proposed model outperformed the best publicly available models in terms of synthesis quality, even comparable to the human level. Moreover, FastDiff showed a significant improvement in synthesis speed, which required as few as 4 4 4 iterations to generate high-quality samples. To the best of our knowledge, FastDiff made diffusion models for the first time applicable to interactive, real-world speech generation with a low computational cost. In addition, FastDiff performed strong robustness and enjoyed high-quality synthesis in out-of-domain generalization to unseen speakers. We will release our code and pre-trained models in the future, and we envisage that our work could serve as a basis for future speech synthesis studies.

References

Chen et al. [2020]

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.

Wavegrad: Estimating gradients for waveform generation.

In Proc. of ICLR , 2020.

Chen et al. [2021]

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan.

Wavegrad 2: Iterative refinement for text-to-speech synthesis.

Creswell et al. [2018]

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath.

Generative adversarial networks: An overview.

Dhariwal and Nichol [2021]

Prafulla Dhariwal and Alex Nichol.

Diffusion models beat gans on image synthesis.

Donahue et al. [2020]

Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan.

End-to-end adversarial text-to-speech.

In Proc. of ICLR , 2020.

Ho et al. [2020]

Jonathan Ho, Ajay Jain, and Pieter Abbeel.

Denoising diffusion probabilistic models.

Huang et al. [2021]

Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao.

Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus.

In Proc. of ACM MM , pages 3945–3954, 2021.

Ito [2017]

Keith Ito.

The lj speech dataset.

Jang et al. [2021]

Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim.

Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation.

Kim et al. [2020]

Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon.

Glow-tts: A generative flow for text-to-speech via monotonic alignment search.

Kim et al. [2021]

Jaehyeon Kim, Jungil Kong, and Juhee Son.

Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.

Kong et al. [2020a]

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.

Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.

Kong et al. [2020b]

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.

Diffwave: A versatile diffusion model for audio synthesis.

In Proc. of ICLR , 2020.

Lam et al. [2022]

Max WY Lam, Jun Wang, Dan Su, and Dong Yu.

Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis.

In Proc. of ICLR , 2022.

Liu et al. [2021]

Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao.

Diffsinger: Singing voice synthesis via shallow diffusion mechanism.

Liu et al. [2022]

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.

Pseudo numerical methods for diffusion models on manifolds.

arXiv preprint arXiv:2202.09778 , 2022.

Oord et al. [2016]

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.

Wavenet: A generative model for raw audio.

Prenger et al. [2019]

Ryan Prenger, Rafael Valle, and Bryan Catanzaro.

Waveglow: A flow-based generative network for speech synthesis.

In Proc. of ICASSP , 2019.

Ren et al. [2020]

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.

Fastspeech 2: Fast and high-quality end-to-end text to speech.

In Proc. of ICLR , 2020.

Ren et al. [2021]

Yi Ren, Jinglin Liu, and Zhou Zhao.

Portaspeech: Portable and high-quality generative text-to-speech.

Rix et al. [2001]

Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra.

Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs.

In Proc. of ICASSP , 2001.

San-Roman et al. [2021]

Robin San-Roman, Eliya Nachmani, and Lior Wolf.

Noise estimation for generative diffusion models.

Song et al. [2020a]

Jiaming Song, Chenlin Meng, and Stefano Ermon.

Denoising diffusion implicit models.

In Proc. of ICLR , 2020.

Song et al. [2020b]

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.

Score-based generative modeling through stochastic differential equations.

In Proc. of ICLR , 2020.

Taal et al. [2010]

Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen.

A short-time objective intelligibility measure for time-frequency weighted noisy speech.

In Proc. of ICASSP , 2010.

Vaswani et al. [2017]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

Attention is all you need.

In Proc. of NeurIPS , 2017.

Xiao et al. [2021]

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.

Tackling the generative learning trilemma with denoising diffusion gans.

Yamagishi et al. [2019]

Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al.

Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).

Zeng et al. [2021]

Zhen Zeng, Jianzong Wang, Ning Cheng, and Jing Xiao.

Lvcnet: Efficient condition-dependent modeling network for waveform generation.

In Proc. of ICASSP , 2021.

Appendix A

Diffusion Probabilistic models

Given i.i.d. samples { 𝐱 0 ∈ ℝ D } subscript 𝐱 0 superscript ℝ 𝐷 {\mathbf{x}{0}\in\mathbb{R}^{D}} from an unknown data distribution p d ​ a ​ t ​ a ​ ( 𝐱 0 ) subscript 𝑝 𝑑 𝑎 𝑡 𝑎 subscript 𝐱 0 p{data}(\mathbf{x}{0}) . In this section, we introduce the theory of diffusion probabilistic model  Ho et al. [ 2020 ]; Lam et al. [ 2022 ]; Song et al. [ 2020a , b ] . First, we present diffusion and reverse process given by denoising diffusion probabilistic models (DDPMs), which could be used to learn a model distribution p θ ​ ( 𝐱 0 ) subscript 𝑝 𝜃 subscript 𝐱 0 p{\theta}(\mathbf{x}{0}) that approximates p d ​ a ​ t ​ a ​ ( 𝐱 0 ) subscript 𝑝 𝑑 𝑎 𝑡 𝑎 subscript 𝐱 0 p{data}(\mathbf{x}_{0}) . Secondly, we introduce the recently proposed bilateral denoising diffusion models (BDDMs) and its tighter evidence lower bound (ELBO) for acceleration.

Diffusion process

Similar as previous work  Ho et al. [ 2020 ]; Lam et al. [ 2022 ]; Song et al. [ 2020a ] , we define the data distribution as q ​ ( 𝐱 0 ) 𝑞 subscript 𝐱 0 q(\mathbf{x}{0}) . The diffusion process is defined by a fixed Markov chain from data x 0 subscript 𝑥 0 x{0} to the latent variable x T subscript 𝑥 𝑇 x_{T} :

   q  ​   (   𝒙  1   ,  ⋯  ,    𝒙  T   |   x  0    )    =     ∏   t  =  1   T     q  ​   (    𝒙  t   |   𝒙   t  −  1     )      ,          𝑞    subscript  𝒙  1   ⋯   conditional   subscript  𝒙  𝑇    subscript  𝑥  0        superscript   subscript  product     𝑡  1    𝑇      𝑞   conditional   subscript  𝒙  𝑡    subscript  𝒙     𝑡  1         \displaystyle q({\bm{x}}_{1},\cdots,{\bm{x}}_{T}|x_{0})=\prod_{t=1}^{T}q({\bm{x}}_{t}|{\bm{x}}_{t-1}),\quad\ \     

(5)

For a small positive constant β t subscript 𝛽 𝑡 \beta_{t} , a small Gaussian noise is added from x t subscript 𝑥 𝑡 x_{t} to the distribution of x t − 1 subscript 𝑥 𝑡 1 x_{t-1} under the function of q ​ ( x t | x t − 1 ) 𝑞 conditional subscript 𝑥 𝑡 subscript 𝑥 𝑡 1 q(x_{t}|x_{t-1}) .

The whole process gradually converts data x 0 subscript 𝑥 0 x_{0} to whitened latents x T subscript 𝑥 𝑇 x_{T} according to the fixed noise schedule β 1 , ⋯ , β T subscript 𝛽 1 ⋯ subscript 𝛽 𝑇 \beta_{1},\cdots,\beta_{T} .

  q  ​   (    𝒙  t   |   𝒙   t  −  1     )    :=   𝒩  ​   (   𝒙  t   ;     1  −   β  t     ​   𝒙   t  −  1     ,    β  t   ​  I   )       assign     𝑞   conditional   subscript  𝒙  𝑡    subscript  𝒙     𝑡  1         𝒩    subscript  𝒙  𝑡            1   subscript  𝛽  𝑡      subscript  𝒙     𝑡  1         subscript  𝛽  𝑡   𝐼       \displaystyle q({\bm{x}}_{t}|{\bm{x}}_{t-1}):={\mathcal{N}}({\bm{x}}_{t};\sqrt{1-\beta_{t}}{\bm{x}}_{t-1},\beta_{t}I)    

(6)

Efficient training is optimizing a random term of t 𝑡 t with stochastic gradient descent:

ℒ θ = ∥ ϵ θ ​ ( α t ​ 𝐱 0 + 1 − α t 2 ​ ϵ ) − ϵ ∥ 2 2 , ϵ ∼ 𝒩 ​ ( 𝟎 , 𝑰 ) formulae-sequence subscript ℒ 𝜃 superscript subscript delimited-∥∥ subscript bold-italic-ϵ 𝜃 subscript 𝛼 𝑡 subscript 𝐱 0 1 superscript subscript 𝛼 𝑡 2 bold-italic-ϵ bold-italic-ϵ 2 2 similar-to bold-italic-ϵ 𝒩 0 𝑰 \displaystyle{\mathcal{L}}{\theta}=\left\lVert\bm{\epsilon}{\theta}\left(\alpha_{t}{\mathbf{x}}{0}+\sqrt{1-\alpha{t}^{2}}\bm{\epsilon}\right)-\bm{\epsilon}\right\rVert_{2}^{2},\bm{\epsilon}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})

(7)

Reverse process

Unlike the diffusion process, reverse process is to recover samples from Gaussian noises. The reverse process is a Markov chain from x T subscript 𝑥 𝑇 x_{T} to x 0 subscript 𝑥 0 x_{0} parameterized by shared θ 𝜃 \theta :

    p  θ   ​   (   𝒙  0   ,  ⋯  ,    𝒙   T  −  1    |   x  T    )    =     ∏   t  =  1   T      p  θ   ​   (    𝒙   t  −  1    |   𝒙  t    )      ,           subscript  𝑝  𝜃     subscript  𝒙  0   ⋯   conditional   subscript  𝒙     𝑇  1     subscript  𝑥  𝑇        superscript   subscript  product     𝑡  1    𝑇       subscript  𝑝  𝜃    conditional   subscript  𝒙     𝑡  1     subscript  𝒙  𝑡        \displaystyle p_{\theta}({\bm{x}}_{0},\cdots,{\bm{x}}_{T-1}|x_{T})=\prod_{t=1}^{T}p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}),    

(8)

where each iteration eliminate the Gaussian noise added in the diffusion process:

  p  ​   (    𝒙  t   |   𝒙   t  −  1     )    :=   𝒩  ​   (   𝒙   t  −  1    ;    μ  θ   ​   (   𝒙  t   ,  t  )    ,    σ  θ   ​    (   𝒙  t   ,  t  )   2   ​  I   )       assign     𝑝   conditional   subscript  𝒙  𝑡    subscript  𝒙     𝑡  1         𝒩    subscript  𝒙     𝑡  1        subscript  𝜇  𝜃     subscript  𝒙  𝑡   𝑡        subscript  𝜎  𝜃    superscript    subscript  𝒙  𝑡   𝑡   2   𝐼       \displaystyle p({\bm{x}}_{t}|{\bm{x}}_{t-1}):=\mathcal{N}({\bm{x}}_{t-1};\mu_{\theta}({\bm{x}}_{t},t),\sigma_{\theta}({\bm{x}}_{t},t)^{2}I)    

(9)

Acceration

Recently, Bilateral denoising diffusion models (BDDMs)  Lam et al. [ 2022 ] demonstrates its tighter evidence lower bound (ELBO) for noise schedule prediction. Given a leaned diffusion network θ 𝜃 \theta , a scheduling network ϕ italic-ϕ \phi could be applied in reducing the gap between the proposed surrogate objective. To be more specific, instead of using the fixed one in diffusion process, a much more efficient N-step noise schedule (i.e., β ^ ^ 𝛽 \hat{\beta} ) could be derived by the well-leaned noise scheduling network ϕ italic-ϕ \phi . The noise schedule could be applied in reverse process, making it possible to explicitly trade-off between inference computation and output quality in one model.

For learning the noise schedule predictor ϕ italic-ϕ \phi , we apply the loss function as a KL divergence term between the forward and the reverse distribution:

ℒ ϕ = 1 2 ​ ( 1 − β t − α t 2 ) ​ ‖ 1 − α t 2 ​ ϵ t − β t 1 − α t 2 ​ ϵ θ ​ ( 𝒙 t , α t ) ‖ 2 2 + C t subscript ℒ italic-ϕ 1 2 1 subscript 𝛽 𝑡 superscript subscript 𝛼 𝑡 2 superscript subscript norm 1 superscript subscript 𝛼 𝑡 2 subscript bold-italic-ϵ 𝑡 subscript 𝛽 𝑡 1 superscript subscript 𝛼 𝑡 2 subscript bold-italic-ϵ 𝜃 subscript 𝒙 𝑡 subscript 𝛼 𝑡 2 2 subscript 𝐶 𝑡 {\mathcal{L}}{\phi}=\frac{1}{2(1-\beta{t}-\alpha_{t}^{2})}|\sqrt{1-\alpha_{t}^{2}}\bm{\epsilon}{t}-\frac{\beta{t}}{\sqrt{1-\alpha_{t}^{2}}}\bm{\epsilon}{\theta}({\bm{x}}{t},\alpha_{t})|{2}^{2}+C{t}

(10)

where C t = 1 4 ​ log ⁡ 1 − α t 2 β t + D 2 ​ ( β t 1 − α t 2 − 1 ) subscript 𝐶 𝑡 1 4 1 superscript subscript 𝛼 𝑡 2 subscript 𝛽 𝑡 𝐷 2 subscript 𝛽 𝑡 1 superscript subscript 𝛼 𝑡 2 1 C_{t}=\frac{1}{4}\log\frac{1-\alpha_{t}^{2}}{\beta_{t}}+\frac{D}{2}(\frac{\beta_{t}}{1-\alpha_{t}^{2}}-1) is a constant that can be ignored during training.

Appendix B

Model Architectures

B.1

FastDiff

As illustrated in Table 5 , we list the hyper-parameters of FastDiff. We further visualize the detailed architectures of the noise predictor and DBlock in the refinement model in Figure  3 .

Hyperparameter
FastDiff

Refinement Model θ 𝜃 \theta

DBlock Hidden Channels 32

DBlock Downsample Ratios [4, 8, 8]

Diffusion UBlock Hidden Channels 32

Diffusion UBlock Upsample Ratios [8, 8, 4]

Time-aware LVC layers Each Block 4

Time-aware LVC layers Kernel Size 256

Diffusion Kernel Predictor Hidden Channels 64

Diffusion Kernel Predictor Kernel Size 3

Diffusion Embedding Input Channels 128

Diffusion Embedding Output Channels 512

Use Weight Norm True

Total Number of Parameters 13M

Noise Predictor ϕ italic-ϕ \phi

Window Length 8 Samples

Segment Size 64

Number of GALR Blocks 2

GALR Blocks Hidden Channels 128

Total Number of Parameters 0.5M

Table 5: Architecture hyperparameters of FastDiff.

Figure 3: The details of network architectures. Left: The GALR-block based noise predictor ϕ italic-ϕ \phi . Right: DBlock in FastDiff θ 𝜃 \theta

B.2

FastDiff-TTS

In this section, we list the model hyper-parameters of FastDiff-TTS in Table 5 .

Hyperparameter
FastDiff-TTS

Phoneme Embedding Dimension 256

Pre-net Layers 3

Pre-net Hidden 256

Encoder Layers 4

Encoder Hidden 256

Encoder Conv1D Kernel 9

Encoder Conv1D Filter Size 1024

Encoder Attention Heads 2

Encoder/Decoder Dropout 0.1

Variance Predictor Conv1D Kernel 3

Variance Predictor Conv1D Filter Size 256

Variance Predictor Dropout 0.5

FastDiff wave decoder Follow Table  5

Total Number of Parameters 40M

Table 6: Architecture hyperparameters of FastDiff-TTS.

Appendix C

Training, Noise scheduling and Inference details

C.1

Diffusion hyperparameters

We list the diffusion hyper-parameters of FastDiff and FastDiff/FastDiff-TTS in Table 7 .

Diffusion Hyperparameter

Noise Scheduling

  τ  =  200   ,      α  t   ^   =  0.54   ,      β  t   ^   =  0.70   ,   N  =  4        formulae-sequence     𝜏  200    formulae-sequence      ^   subscript  𝛼  𝑡    0.54    formulae-sequence      ^   subscript  𝛽  𝑡    0.70      𝑁  4       \tau=200,\hat{\alpha_{t}}=0.54,\hat{\beta_{t}}=0.70,N=4    

Training and Sampling

Pre-defined (T = 1000 1000 1000 ) :

 β  =   Linear  ​   (   1  ×   10   −  4     ,  0.005  ,  1000  )         𝛽      Linear       1   superscript  10     4     0.005  1000      \beta=\text{Linear}(1\times 10^{-4},0.005,1000)    

Grid Search ( T m = 4 subscript 𝑇 𝑚 4 T_{m}=4 ) derived :

  β  ^   =   [   3.6701  ​   e   −  7     ,   1.7032  ​   e   −  5     ,   7.908  ​   e   −  4     ,   7.6146  ​   e   −  1     ]         ^  𝛽       3.6701   superscript  𝑒     7        1.7032   superscript  𝑒     5        7.908   superscript  𝑒     4        7.6146   superscript  𝑒     1        \hat{\beta}=[3.6701e^{-7},1.7032e^{-5},7.908e^{-4},7.6146e^{-1}]    

Noise Predictor ( T m = 4 subscript 𝑇 𝑚 4 T_{m}=4 ) derived :

  β  ^   =   [   3.2176  ​   e   −  4     ,   2.5743  ​   e   −  3     ,   2.5376  ​   e   −  2     ,   7.0414  ​   e   −  1     ]         ^  𝛽       3.2176   superscript  𝑒     4        2.5743   superscript  𝑒     3        2.5376   superscript  𝑒     2        7.0414   superscript  𝑒     1        \hat{\beta}=[3.2176e^{-4},2.5743e^{-3},2.5376e^{-2},7.0414e^{-1}]    

Table 7: Diffusion hyperparameters of FastDiff and FastDiff-TTS.

C.2

Noise Scheduling

Our noise scheduling algorithm mainly follows the bilateral denoising diffusion models  Lam et al. [ 2022 ] :

Algorithm 4 Noise scheduling process

1:    Input : Pre-defined discrete β 𝛽 \beta , trained refinement network θ 𝜃 \theta , hyperparameter N , α t ^ , β t ^ 𝑁 ^ subscript 𝛼 𝑡 ^ subscript 𝛽 𝑡 N,\hat{\alpha_{t}},\hat{\beta_{t}} .

2:    for   t = N , ⋯ , 2 𝑡 𝑁 ⋯ 2 t=N,\cdots,2   do

3:      Sample 𝒙 t − 1 ∼ p θ ​ ( x t − 1 | x t ) similar-to subscript 𝒙 𝑡 1 subscript 𝑝 𝜃 conditional subscript 𝑥 𝑡 1 subscript 𝑥 𝑡 {\bm{x}}{t-1}\sim p{\theta}(x_{t-1}|x_{t})

4:       α ^ t − 1 = α ^ t 1 − β ^ t subscript ^ 𝛼 𝑡 1 subscript ^ 𝛼 𝑡 1 subscript ^ 𝛽 𝑡 \hat{\alpha}{t-1}=\frac{\hat{\alpha}{t}}{\sqrt{1-\hat{\beta}_{t}}}

5:       β ^ t − 1 = min ⁡ { 1 − α ^ t − 1 2 , β ^ t } ​ ϕ ​ ( 𝒙 ^ t − 1 ) subscript ^ 𝛽 𝑡 1 1 superscript subscript ^ 𝛼 𝑡 1 2 subscript ^ 𝛽 𝑡 italic-ϕ subscript ^ 𝒙 𝑡 1 \hat{\beta}{t-1}=\min\left{1-\hat{\alpha}{t-1}^{2},\hat{\beta}{t}\right}\phi\left(\hat{\bm{x}}{t-1}\right)

6:       if   β ^ t − 1 < β 1 subscript ^ 𝛽 𝑡 1 subscript 𝛽 1 \hat{\beta}{t-1}<\beta{1}   then

7:          return   β ^ t , … , β ^ N subscript ^ 𝛽 𝑡 … subscript ^ 𝛽 𝑁 \hat{\beta}{t},\ldots,\hat{\beta}{N}

8:       end   if

9:    end   for

10:    return   β ^ t , … , β ^ N subscript ^ 𝛽 𝑡 … subscript ^ 𝛽 𝑁 \hat{\beta}{t},\ldots,\hat{\beta}{N}

C.3

Schedule Alignment

Here we search and interpolate α s subscript 𝛼 𝑠 \alpha_{s} between two training noise constants l t subscript 𝑙 𝑡 l_{t} and l t + 1 subscript 𝑙 𝑡 1 l_{t+1} , enforcing α s subscript 𝛼 𝑠 \alpha_{s} to get closed to l t subscript 𝑙 𝑡 l_{t} . In the end, we gain the well-mapped diffusion step t m subscript 𝑡 𝑚 t_{m} :

Firstly we compute the corresponding constants respective to diffusion and reverse process:

   l  t   =     ∏   i  =  1   t      1  −   β  i       ,    α  s   =     ∏   i  =  1   s      1  −    β  ^   i          formulae-sequence      subscript  𝑙  𝑡     superscript   subscript  product     𝑖  1    𝑡         1   subscript  𝛽  𝑖           subscript  𝛼  𝑠     superscript   subscript  product     𝑖  1    𝑠         1   subscript   ^  𝛽   𝑖         \displaystyle l_{t}=\prod_{i=1}^{t}\sqrt{1-\beta_{i}},\quad\alpha_{s}=\prod_{i=1}^{s}\sqrt{1-\hat{\beta}_{i}}    

(11)

Here we search and interpolate α s subscript 𝛼 𝑠 \alpha_{s} between two training noise constants l t subscript 𝑙 𝑡 l_{t} and l t + 1 subscript 𝑙 𝑡 1 l_{t+1} , enforcing α s subscript 𝛼 𝑠 \alpha_{s} to get closed to l t subscript 𝑙 𝑡 l_{t} . In the end, we gain the well-mapped diffusion step t m subscript 𝑡 𝑚 t_{m} :

t m = t + l t − α s l t − l t + 1  if  ​ α s ∈ [ l t + 1 , l t ] . formulae-sequence subscript 𝑡 𝑚 𝑡 subscript 𝑙 𝑡 subscript 𝛼 𝑠 subscript 𝑙 𝑡 subscript 𝑙 𝑡 1  if  subscript 𝛼 𝑠 subscript 𝑙 𝑡 1 subscript 𝑙 𝑡 t_{m}=t+\frac{l_{t}-\alpha_{s}}{l_{t}-l_{t+1}}\quad\text{ if }\alpha_{s}\in[\ l_{t+1},l_{t}\ ].

(12)

Where integer t 𝑡 t represents a single pre-defined diffusion step, and s 𝑠 s presents a single step of noise schedule obtained through the scheduling process. Given these two schedules mentioned above, we could conduct schedule alignment and derive the floating-point t m subscript 𝑡 𝑚 t_{m} for much more efficient reverse sampling.

Figure 4: Screenshot of MOS testing.

Appendix D

Evaluation Matrix

D.1

PESQ and STOI

Perceptual evaluation of speech quality (PESQ)  Rix et al. [ 2001 ] and The short-time objective intelligibility (STOI)  Taal et al. [ 2010 ] assesses the denoising quality for speech enhancement.

D.2

NDB and JSD

Number of Statistically-Different Bins (NDB) and Jensen-Shannon divergence (JSD). They measure diversity by 1) clustering the training data into several clusters, and 2) measuring how well the generated samples fit into those clusters.

D.3

Details in MOS Evaluation

All our Mean Opinion Score (MOS) tests are crowd-sourced and conducted by native speakers. The scoring criteria has been included in Table  8 for completeness. The samples are presented and rated one at a time by the testers, each tester is asked to evaluate the subjective naturalness of a sentence on a 1-5 Likert scale. The screenshots of instructions for testers are shown in Figure  4 . We paid $8 to participants hourly and totally spent about $750 on participant compensation.

Rating
Naturalness
Definition

1
Bad
Very annoying and objectionable dist.

2
Poor
Annoying but not objectionable dist.

3
Fair
Perceptible and slightly annoying dist

4
Good
Just perceptible but not annoying dist.

5
Excellent
Imperceptible distortions

Table 8: Ratings that have been used in evaluation of speech naturalness of synthetic and ground truth samples.

Appendix E

Extension to Continuous Condition

Our ablation study extends FastDiff to be conditioned on continuous noise levels and compares it to the basic model with the discrete condition. To be more specific, the FastDiff model conditioned on continuous noise levels does not require an additional schedule alignment process, which has a separated training and sampling procedure:

Algorithm 5 Training refinement network θ 𝜃 \theta (Continuous Condition)

1:    Input : Pre-defined noise schedule l 𝑙 l

2:    repeat

3:      Sample 𝒙 0 ∼ q d ​ a ​ t ​ a similar-to subscript 𝒙 0 subscript 𝑞 𝑑 𝑎 𝑡 𝑎 {\bm{x}}{0}\sim q{data} , ϵ ∼ 𝒩 ​ ( 0 , I ) similar-to italic-ϵ 𝒩 0 𝐼 \epsilon\sim{\mathcal{N}}(0,I) , and   t ∼ Unif ​ ( { 1 , ⋯ , T } ) similar-to 𝑡 Unif 1 ⋯ 𝑇 t\sim\mathrm{Unif}({1,\cdots,T})

4:       α s ∼ Uniform ⁡ ( α t − 1 , α t ) similar-to subscript 𝛼 𝑠 Uniform subscript 𝛼 𝑡 1 subscript 𝛼 𝑡 \alpha_{s}\sim\operatorname{Uniform}\left(\alpha_{t-1},\alpha_{t}\right)

5:       𝒙 s = α s ​ 𝒙 0 + δ s ​ ϵ subscript 𝒙 𝑠 subscript 𝛼 𝑠 subscript 𝒙 0 subscript 𝛿 𝑠 italic-ϵ {\bm{x}}{s}=\alpha{s}{\bm{x}}{0}+\delta{s}\epsilon

6:      Take gradient descent steps on ∇ θ ‖ ϵ − ϵ θ ​ ( 𝒙 s , α s ) ‖ 2 2 subscript ∇ 𝜃 superscript subscript norm italic-ϵ subscript italic-ϵ 𝜃 subscript 𝒙 𝑠 subscript 𝛼 𝑠 2 2 \nabla_{\theta}\left|\epsilon-\epsilon_{\theta}\left({\bm{x}}{s},\alpha{s}\right)\right|_{2}^{2}

7:    until  iterative refinement model θ 𝜃 \theta converged

Algorithm 6 Sampling

1:    Input : Pre-defined β , T 𝛽 𝑇 \beta,T and β ^ ^ 𝛽 \hat{\beta} derived in noise scheduling process.

2:    for   t = T , ⋯ , 1 𝑡 𝑇 ⋯ 1 t=T,\cdots,1   do

3:      Sample 𝒙 t − 1 ∼ p θ ​ ( 𝒙 t − 1 | 𝒙 t ) similar-to subscript 𝒙 𝑡 1 subscript 𝑝 𝜃 conditional subscript 𝒙 𝑡 1 subscript 𝒙 𝑡 {\bm{x}}{t-1}\sim p{\theta}({\bm{x}}{t-1}|{\bm{x}}{t})

4:    end   for

5:    return   𝒙 0 subscript 𝒙 0 {\bm{x}}_{0}

Appendix F

Sample Diversity

Previous works  Dhariwal and Nichol [ 2021 ]; Xiao et al. [ 2021 ] in the image generation task has demonstrated that diffusion probabilistic model outperforms GAN in sample diversity, while the comparison in the speech domain is relatively overlooked. Similarly, we can intuitively infer that diffusion probabilistic models are good at generating high-fidelity diverse speech samples. To verify our hypothesis, we employed two metrics NDB and JSD to explore the diversity of generated mel-spectrograms. As shown in Table  2 , we can see that diffusion probabilistic model achieve a higher JSD and matching NDB score for generated speeches compare to GAN-based model, which is expected for the following reasons:

1. It is well-known that the mode collapse problem  Creswell et al. [ 2018 ] appears in the dominated GAN-based generative models, which leads to very similar output samples from a single or few modes of the distribution, especially in the strongly conditional generation task.
2. In contrast, diffusion probabilistic model is meant to reduce mode collapse compared to one-shot generation. It breaks the generation process into several conditional denoising diffusion steps in which each step is relatively simple to model. Thus, we expect our model to exhibit better training stability and mode coverage.

◄

 Feeling lucky? 
  
 Conversion report 
 Report an issue 
 View original on arXiv  ►