Code-Generated Promo Videos (5): The TTS Landscape — From VODER to Zero-Shot Cloning

In Part 2, we used three lines of edge-tts code to generate voiceovers for our promo video. But edge-tts is just one entry point into the vast world of TTS — where did its neural architecture come from? And beyond calling Microsoft’s API, what else can TTS do today?

This article traces the journey of speech synthesis from VODER in 1939 to Flow Matching in 2025, covering the complete evolution of neural TTS and its frontier capabilities. By the end, you’ll understand: why can’t free edge-tts clone voices? What are the cutting-edge capabilities of modern TTS? And where is this technology heading?

The Evolution of TTS Technology

Text-to-speech technology has undergone five paradigm shifts over nearly a century: from mechanical rule-based synthesis to statistical modeling, from the deep learning breakthrough to generative understanding, and finally toward multimodal fusion.

Formant and Concatenative Synthesis: The Mechanical Era (1939–2000)

The earliest approach was formant synthesis, which generates speech by simulating vocal tract resonance parameters (F1, F2, F3) through electronic circuits or software rules.

Key milestones:

• 1939 VODER: Homer Dudley at Bell Labs demonstrated the first electronic speech synthesizer at the New York World’s Fair. The operator controlled electronic circuits via keyboard and foot pedals to generate speech in real time — humanity’s first attempt at making machines “speak.”
• 1983 DECtalk: Based on MIT Professor Klatt’s formant research, it became the commercial gold standard for TTS. Stephen Hawking’s iconic voice was powered by this technology for decades.
• 1990 PSOLA algorithm: Enabled pitch and duration modification of recorded speech segments without significant quality loss, unlocking practical concatenative synthesis.
• 2000 AT&T Natural Voices: Represented the peak of concatenative TTS quality — natural and fluent, but expanding to new voices required hours of recording.

Statistical Parametric Synthesis: The HMM Era (2005)

Statistical Parametric Synthesis (SPSS) replaced waveform snippets with Hidden Markov Models (HMMs) that learn the statistical distribution of acoustic parameters. During synthesis, it predicts spectral and pitch parameters from text, then reconstructs the waveform via a vocoder.

The 2005 HTS system by Tokuda et al. was the most mature representative of this school. Compared to concatenative synthesis, HMM-based approaches used less storage, could adapt to new voices, and offered flexible emotion and speed control. But the vocoder suffered from a noticeable “buzzy” artifact that limited naturalness — a necessary trade-off between “flexibility” and “quality.”

The Neural TTS Breakthrough (2016–2021)

WaveNet (2016, DeepMind) — The Dawn of Neural TTS

WaveNet was the first model to directly generate raw audio waveforms, using causal and dilated convolutions with an exponentially growing receptive field. For the first time in TTS history, deep learning replaced the DSP-based vocoder.

• Its MOS (Mean Opinion Score, 5-point scale) surpassed 4.0, narrowing the gap between machine and human speech by 50%
• Inaugurated the era of neural audio generation — it became the foundation for all subsequent neural vocoders
• Limitation: inference was extremely slow — generating one second of audio initially took minutes (autoregressive sample-by-sample generation)

Tacotron Series (2017–2018, Google)

Tacotron 1 (2017) reframed text-to-mel-spectrogram conversion as a sequence-to-sequence (Seq2Seq) learning problem, achieving the first end-to-end TTS system — input text, output spectrograms directly.

Tacotron 2 (2017–2018) used a convolutional + BiLSTM encoder with Location-Sensitive Attention, coupled with a WaveNet vocoder. Its MOS was nearly indistinguishable from human recordings. But autoregressive decoding was slow, and the attention mechanism was prone to word skipping or repetition on long sentences — inherent limitations of the autoregressive architecture.

FastSpeech Series (2019–2020, Microsoft) — Speed First

FastSpeech 1 (2019) was the first non-autoregressive TTS architecture. It replaced Tacotron’s attention alignment with an explicit duration predictor, generating the entire spectrogram sequence in a single forward pass for parallel generation.

FastSpeech 2 (2020) added explicit pitch and energy prediction to address Tacotron’s one-to-many mapping problem. It achieved 30–50× speedup over Tacotron and remains the architecture of choice for many production TTS systems.

VITS (2021, KAIST) — The End-to-End Pinnacle

VITS unified the acoustic model and vocoder into a single model, eliminating the mel-spectrogram intermediate representation. Its key innovations included variational inference (VAE) to capture speech randomness, a stochastic duration predictor for more natural rhythm, and adversarial training (GAN) to directly optimize perceptual quality.

VITS is the most widely used foundational architecture in the open-source TTS ecosystem — popular projects like Bert-VITS2 and GPT-SoVITS evolved from it. Its success proved that end-to-end joint training (without staged fine-tuning) is viable for speech synthesis.

Cutting-Edge Models: Generative and LLM Paradigms (2023–2025)

The year 2023 marked another watershed moment for TTS, with two new paradigms emerging almost simultaneously:

The language model paradigm reframes TTS as a language modeling problem. VALL-E (2023, Microsoft) requires only 3 seconds of reference audio for zero-shot voice cloning. Bark (2023, Suno) was among the earliest open-source implementations, supporting speech, sound effects, and music generation. XTTS v2 (2023, Coqui) achieved zero-shot multilingual cloning and gained wide community adoption.

The Flow Matching paradigm replaces the multi-step denoising process of diffusion models with flow matching, achieving comparable quality in fewer sampling steps. F5-TTS (2024, Shanghai Jiao Tong University) completely eliminates phoneme alignment requirements, earning 14,500+ GitHub stars. CosyVoice 2 (2024, Alibaba) strikes an excellent balance of speed and quality within this framework. Kokoro (2024) reaches near-SOTA quality with only 82 million parameters, enabling on-device inference.

Meanwhile, NaturalSpeech 3 (2024, Microsoft) uses a factored diffusion model to decouple speech into four subspaces: content, prosody, timbre, and acoustic details — a systematic pursuit of “human-level” quality.

And GPT-4o (2024) with Gemini 2.0 (2024–2025) represent an even larger trend: TTS is being absorbed as a native capability of multimodal LLMs, no longer a standalone inference pipeline.

flowchart TD
    N1["🎛️ Formant/Concat<br/>1939–2000"]:::start
    N2["📊 HMM Statistical<br/>2005"]:::process
    N3["🧠 WaveNet<br/>Neural Vocoder<br/>2016"]:::special
    N4["⚡ End-to-End<br/>Tacotron/VITS<br/>2017–2021"]:::process
    N5["🎯 Zero-Shot Clone<br/>VALL-E/F5-TTS<br/>2023–2024"]:::output
    N6["🤖 Multimodal LLM<br/>GPT-4o/Gemini<br/>2025"]:::output

    N1 --> N2 --> N3 --> N4 --> N5 --> N6

    classDef start fill:#FF9800,color:#fff
    classDef process fill:#2196F3,color:#fff
    classDef special fill:#9C27B0,color:#fff
    classDef output fill:#4CAF50,color:#fff

This timeline illustrates five paradigm shifts in TTS: from mechanical formant/concatenative synthesis (orange), through HMM statistical modeling (blue), to WaveNet’s deep learning revolution (purple), the maturation of end-to-end architectures (blue), and finally zero-shot cloning with multimodal fusion (green). The overarching trend is data-driven approaches replacing rule-based ones, and generative understanding replacing statistical fitting.

Generational Comparison

The following table covers the key dimensional differences across TTS generations from 1939 to the present:

The Four Current Technology Paradigms

Today’s active TTS technologies fall into four paths:

1. Autoregressive Seq2Seq (Tacotron route): Good quality but slow inference; gradually being phased out by faster architectures.
2. Non-autoregressive Parallel (FastSpeech route): Fast and stable inference; the mainstream choice for industrial deployment.
3. End-to-end VAE+GAN (VITS route): Good balance of quality and speed; the richest open-source ecosystem. If you search for TTS projects on GitHub, you’ll likely encounter this route.
4. Diffusion / Flow Matching (NaturalSpeech / F5-TTS route): Best quality with strong zero-shot cloning; the most active research direction today. F5-TTS and CosyVoice 2 are the flagship models.

Frontier Capabilities

Zero-Shot Voice Cloning

Zero-shot voice cloning is the most significant capability breakthrough in modern TTS. Users provide just seconds to tens of seconds of reference audio, and the system clones the speaker’s timbre for synthesis.

This is the fundamental reason why edge-tts cannot perform voice cloning: its architecture uses a fixed speaker embedding dictionary with no interface for injecting arbitrary speaker vectors. Models like VALL-E and CosyVoice, by contrast, build a generative “voice identity” space during training — inference requires just a few seconds of reference audio to embed any speaker within that space.

Emotion and Style Control

Modern TTS systems have moved beyond neutral reading, supporting fine-grained emotional control:

• CosyVoice: Supports 14 fine-grained control tags ([laughter], [breath], etc.)
• ChatTTS: Optimized for conversational scenarios, supporting laughter, pauses, and prosodic control
• Azure Speech: Supports emotion and role-play through SSML <mstts> tags
• OpenAI gpt-4o-mini-tts: Supports natural language instruction for voice style (e.g., “say this excitedly”)
• Bark: Can produce non-speech sounds (laughter, breathing) and even simple melodies — a representative of “speech + sound effects” fusion

Real-Time Streaming Synthesis

Low-latency real-time synthesis is critical for voice conversation scenarios. First-packet latency is a key metric:

Multimodal LLM Integration

The most important trend of 2024–2025 is TTS being absorbed as a native capability of multimodal LLMs:

• GPT-4o: Natively supports voice input and output for end-to-end voice conversations. No more “ASR → LLM → TTS” pipeline — a single model processes both text and audio token sequences, delivering latency and naturalness far beyond traditional cascaded approaches.
• Gemini 2.0: Multimodal LLM with integrated voice capabilities.
• Sesame CSM (2025): Designed for multi-turn conversations, adaptively adjusting prosody and intonation across the entire dialogue, making AI sound more like a “person” than a “machine.”
• Doubao Real-Time Voice Model: End-to-end ultra-human-like conversation, focusing on emotional expression and natural interaction.

This means TTS is evolving from a “standalone service” into a “native capability of large models.” Future AI conversation systems may no longer need to explicitly invoke a TTS engine — they will simply speak.

Series: ① Overview & Remotion ② edge-tts Voiceover ③ numpy BGM ④ ffmpeg Muxing ⑤ TTS Landscape ⑥ TTS Selection Guide