Code-Generated Promo Videos (2): edge-tts Voiceover & Multilingual Batch Rendering

edge-tts in Practice

This is Part 2 of the series, focusing on Text-to-Speech (TTS) — using edge-tts (Microsoft Azure’s free neural TTS interface) to batch-generate multilingual, multi-voice voiceover files. All code comes from a real project (MiBee NVR 45-second promo) and is ready to reuse.

Installation

edge-tts is a Python async library. Install it inside a virtual environment:

1
2

python -m venv .venv
.venv\Scripts\pip install edge-tts numpy

numpy is not a dependency of edge-tts, but it will be needed for BGM synthesis (Part 3 of this series), so installing it here saves a step.

List all available voices:

1

edge-tts --list-voices

The output includes each voice’s Name, Gender, Locale, ContentCategories, and VoicePersonality for easy filtering.

Voice List

The 6 voices used in this project (3 languages × male/female):

All are Microsoft Azure neural TTS voices with excellent quality. Run edge-tts --list-voices to see the full list (including Japanese, Korean, French, and hundreds more).

Batch Generation Script (Retry + Rate)

gen_all_voices.py is the core script for voiceover generation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import asyncio, os, edge_tts

# 7 Chinese narration texts
ZH = [
    "MiBee NVR，轻量级自托管录像系统……",
    "只需 60 秒，就能把一台树莓派，变成专业级网络录像机。",
    # ...7 clips total
]

# (subdir, texts, voice, rate)
VERSIONS = [
    ("voice_zh_f",  ZH, "zh-CN-XiaoxiaoNeural", "+6%"),
    ("voice_zh_m",  ZH, "zh-CN-YunxiNeural",    "+6%"),
    ("voice_en_f",  EN, "en-US-AriaNeural",     "+13%"),
    ("voice_en_m",  EN, "en-US-GuyNeural",      "+13%"),
    ("voice_yue_m", ZH, "zh-HK-WanLungNeural",  "+6%"),
    ("voice_yue_f", ZH, "zh-HK-HiuMaanNeural",  "+6%"),
]

async def gen_one(text, voice, rate, out, retries=5):
    last_err = None
    for attempt in range(1, retries + 1):
        try:
            c = edge_tts.Communicate(text, voice, rate=rate)
            await c.save(out)
            if os.path.getsize(out) > 1000:   # verify non-empty
                return
            last_err = RuntimeError("empty output")
        except Exception as e:
            last_err = e
        await asyncio.sleep(1.5 * attempt)     # exponential backoff
        print(f"    retry {attempt} for {out} ({last_err})")
    raise RuntimeError(f"failed after {retries} retries: {out}")

async def main():
    for subdir, texts, voice, rate in VERSIONS:
        os.makedirs(subdir, exist_ok=True)
        for i, text in enumerate(texts, start=1):
            await gen_one(text, voice, rate, f"{subdir}/voice_{i}.mp3")
            print(f"[ok] {subdir} clip{i}")

asyncio.run(main())

The retry strategy is critical. edge-tts depends on Microsoft’s online service, which occasionally throws No audio received errors — a transient server glitch. Without retries, batch generation will fail midway. gen_one implements 5 retries with exponential backoff (1.5 × attempt seconds), handling nearly all intermittent failures. Key points:

• Wait time increases with each retry (1.5s, 3s, 4.5s, 6s, 7.5s), giving the server time to recover.
• File size check (> 1000 bytes) after output prevents silent-passing empty/silent audio.
• Only raises an exception after all 5 retries fail — never silently swallows errors.

Rate Adjustment and SSML

The rate parameter in edge-tts maps directly to the SSML <prosody rate="..."> attribute — it’s a synthesis-time prosody adjustment, not a post-hoc speed change. Microsoft’s neural TTS synthesizes speech at the specified rate, so audio quality is preserved. This is fundamentally different from ffmpeg atempo (post-hoc speed change that can introduce artifacts).

Practical experience:

• Chinese / Cantonese: around +6% — sounds natural without dragging.
• English: text is noticeably longer than Chinese, requiring +13% to fit the same time window (English typically has more syllables than the equivalent Chinese text).
• Rate only affects the voiceover itself — it does not affect subtitle or scene timing — so feel free to adjust independently.

SSML Deep Dive

edge-tts uses SSML (Speech Synthesis Markup Language) under the hood to control synthesis details. Beyond rate, SSML supports rich control parameters:

1
2
3
4
5
6
7
8
9

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
    <voice name="en-US-GuyNeural">
        <prosody rate="+13%" pitch="+1st" volume="+10%">
            MiBee NVR, a lightweight self-hosted recording system.
            <break time="500ms"/>
            Deploy it in just 60 seconds.
        </prosody>
    </voice>
</speak>

Key elements explained:

• <voice name="...">: Explicitly selects the voice. edge_tts.Communicate(text, voice=...) internally wraps it into this element.
• <prosody rate="...">: Speech rate adjustment, as a percentage (e.g., +50% for 50% faster, -20% for 20% slower).
• <prosody pitch="...">: Pitch adjustment, using relative values (+2st for two semitones up) or absolute values (220Hz).
• <prosody volume="...">: Volume adjustment, e.g., +20%, silent, x-loud.
• <break time="..."/>: Inserts a silence pause to control reading rhythm.

For fine-grained control, you can pass SSML text directly to edge_tts.Communicate() instead of plain text. This is especially useful when different sentences need different voices or rates.

edge-tts Internals

The TTS Pipeline: How Text Becomes Speech

TTS (Text-to-Speech) is essentially a data processing pipeline that transforms text into playable audio waveforms:

flowchart TD
    Text["📝 Text Input"]:::process
    Text --> Linguistic["🔤 Linguistic Features<br/>Phonemes / Stress"]:::process
    Linguistic --> Acoustic["🎵 Acoustic Features<br/>Mel-spectrogram"]:::special
    Acoustic --> Waveform["🔊 Waveform Generation"]:::special
    Waveform --> Audio["🎧 mp3 Output"]:::output

    classDef process fill:#2196F3,color:#fff
    classDef special fill:#9C27B0,color:#fff
    classDef output fill:#4CAF50,color:#fff

Pipeline steps explained:

1. Text → Linguistic Features: The input text goes through tokenization, part-of-speech tagging, and prosody prediction, producing a phoneme sequence with stress markers. Chinese additionally requires G2P (Grapheme-to-Phoneme) to determine each character’s pronunciation.
2. Linguistic Features → Acoustic Features: A neural network predicts the mel-spectrogram (energy distribution across frequency bands) from the phoneme sequence. This is the most critical TTS step.
3. Acoustic Features → Waveform: A vocoder (e.g., HiFi-GAN) reconstructs the raw PCM waveform from the mel-spectrogram. Microsoft uses high-quality neural vocoders.
4. Encoding Output: The raw PCM waveform is encoded into mp3 format for output.

Neural TTS and Speaker Embedding

Microsoft Azure TTS is built on an end-to-end deep neural network (similar to FastSpeech / VITS architecture), taking text directly to high-quality acoustic representations. A key technology is speaker embedding:

• Each voice corresponds to a fixed-dimensional vector (e.g., 256 dimensions).
• The model itself is a single shared network — it does not switch between voices. The male Chinese voice “Yunxi” and the female English voice “Aria” use the same model.
• During inference, the target speaker embedding is injected into specific network layers, causing the output to “learn” that voice’s pitch, intonation, resonance, and other characteristics.
• This is why one service outputs dozens of voices: the model is shared; only the speaker embeddings differ.

Online Service and Streaming

edge-tts does not synthesize speech locally — it is a client for Azure’s online TTS endpoint:

• Connects to Microsoft’s Azure TTS endpoint (same source as Edge browser’s “Read Aloud” feature).
• Free, no API Key required — which is why it is so popular.
• However, it has server-side rate limiting and is not suitable for production high-concurrency scenarios. The retry mechanism in batch generation is designed specifically for this.
• Uses WebSocket long-connection streaming:

sequenceDiagram
    participant Client as Client
    participant Azure as Azure TTS Endpoint

    Client->>Azure: WebSocket connection established
    Client->>Azure: Send SSML request<br/>text + voice + rate
    loop Streaming synthesis
        Azure-->>Client: mp3 chunk data
    end
    Azure-->>Client: Transfer complete

How streaming works: After establishing a WebSocket connection, the client sends a request containing SSML parameters. Azure begins synthesizing sentence by sentence, pushing each completed segment back to the client via WebSocket immediately. edge_tts.Communicate().save() internally receives these chunks progressively and writes them to a local file. The benefit of this streaming architecture is low time-to-first-audio (no need to wait for full synthesis), but network jitter can break the connection — which is why retries are mandatory.

⚠️ edge-tts is a free tool suitable for personal projects and prototyping. For production, use the official Azure Speech Service SDK (paid, with SLA).

How Cantonese Reads Simplified Chinese (G2P)

A common question: why can the Cantonese voice zh-HK-WanLungNeural read simplified Chinese text directly?

The answer lies in the G2P (Grapheme-to-Phoneme) front-end module:

1. When the model receives simplified Chinese text, it does not simply apply Mandarin pronunciation.
2. The text front-end detects the target voice as zh-HK and automatically uses the Hong Kong Cantonese G2P mapping — mapping “录像” to Cantonese pronunciation (e.g., luk6 zoeng6, not Mandarin lù xiàng).
3. So you can feed simplified Chinese text directly — no manual Jyutping conversion needed. The model handles pronunciation mapping automatically.

Conversely, if you use an English voice like en-US-GuyNeural to read Chinese text, the G2P module will attempt to process Chinese characters with English pronunciation rules — producing gibberish. Always choose a voice ID that matches the target language.

Neural TTS Landscape: From WaveNet to Zero-Shot Cloning

Text-to-speech technology has undergone several paradigm shifts over the past three decades — from mechanical rule-based synthesis to today’s zero-shot voice cloning. Each breakthrough has dramatically narrowed the gap between synthetic and human speech.

Evolution Timeline

Early formant (rule-based) synthesis and concatenative synthesis relied on handcrafted acoustic rules or pre-recorded voice libraries — robotic quality with high expansion costs.

Around 2005, HMM-based parametric synthesis introduced hidden Markov models to generate acoustic parameters statistically, but the “buzzy” vocoder quality remained a fundamental limitation.

In 2016, DeepMind released WaveNet, directly modeling raw audio waveforms. It achieved a Mean Opinion Score (MOS) above 4.0 for the first time, pushing synthetic speech naturalness close to human level — marking the dawn of the neural TTS era.

In 2016, DeepMind released WaveNet, directly modeling raw audio waveforms. It achieved a Mean Opinion Score (MOS) above 4.0 for the first time, pushing synthetic speech naturalness close to human level — marking the dawn of the neural TTS era.

Progress then accelerated rapidly. Tacotron (2017) turned text-to-mel-spectrogram into an end-to-end sequence learning problem, greatly simplifying the traditional pipeline.

FastSpeech (2019) introduced non-autoregressive parallel generation, achieving 30–50× speedup over Tacotron and solving its slow inference problem.

VITS (2021) merged the acoustic model and vocoder into a single model using VAE + GAN for end-to-end joint training, becoming the foundational architecture for many open-source projects.

Since 2023, generative paradigms and LLM-based paradigms have become the new frontier.

Flow Matching models like F5-TTS and CosyVoice 2 eliminate phoneme alignment entirely, balancing quality and speed at a new level.

Meanwhile, VALL-E (2023) reframed TTS as a language modeling problem — cloning any speaker’s voice from just 3 seconds of reference audio with zero-shot capability. This is no longer simple “vocalization” but a generative understanding of voice identity.

flowchart TD
    N1["🎛️ Rules/Concat<br/>Synthesis"]:::start
    N2["📊 HMM<br/>Parametric"]:::process
    N3["🧠 Neural Vocoder<br/>WaveNet 2016"]:::special
    N4["⚡ End-to-End<br/>Tacotron/VITS"]:::process
    N5["🎯 Generative/LLM<br/>Zero-Shot Clone"]:::output

    N1 --> N2 --> N3 --> N4 --> N5

    classDef start fill:#FF9800,color:#fff
    classDef process fill:#2196F3,color:#fff
    classDef special fill:#9C27B0,color:#fff
    classDef output fill:#4CAF50,color:#fff

Diagram explanation: The orange start node (Rules/Concat) represents the mechanical TTS era. Blue nodes (HMM Parametric, End-to-End) trace the main evolution path. The purple node (Neural Vocoder WaveNet) marks the first quantum leap into deep learning. The green terminal node (Generative/LLM + Zero-Shot Clone) represents the current state of the art. The overarching trend: data-driven approaches replace rule-driven ones, end-to-end replaces modular pipelines, and generative understanding replaces statistical fitting.

Where edge-tts Fits in the TTS Landscape

Returning to this series’ practical scenario: the edge-tts you run locally is actually a free client for Microsoft Azure’s cloud neural TTS — built on a FastSpeech-like architecture with speaker embeddings that host hundreds of voices in a single shared model. Its strengths: completely free, no API key required, 400+ voices across 100+ languages, making it ideal for batch promo voiceover.

But edge-tts has clear limitations:

• No zero-shot cloning: It only uses Microsoft’s preset voices (e.g., en-US-GuyNeural). Unlike VALL-E or CosyVoice, it cannot clone an arbitrary speaker from 3 seconds of reference audio. This is an architectural constraint — the cloud model’s speaker embedding dictionary is fixed, with no interface for injecting arbitrary vectors.
• Server rate limiting: The free endpoint has request limits. Batch generation requires retry logic (like the 5-retry exponential backoff in gen_one above).
• No SLA: As an unofficial API, it is unsuitable for production high-concurrency scenarios. For commercial use, switch to Azure Speech Service SDK (paid) or services like ElevenLabs.

If your project needs zero-shot voice cloning (e.g., generating unique voices for specific characters), consider local deployment of CosyVoice (Apache-2.0, commercial-friendly, 3-second reference) or F5-TTS (MIT code, research-grade Flow Matching quality). These run on local GPUs without external API dependencies, but require managing your own GPU environment and model weights.

Free vs Paid: Quick Comparison

⚠️ Prices and figures are point-in-time (2026-07); verify against official pages before relying on them.

This landscape provides a reference frame for your project: edge-tts is fully sufficient for batch promo voiceover; when you need voice cloning or production SLA, upgrade to local models or paid APIs.

Series: ① Overview & Remotion ② edge-tts Voiceover ③ numpy BGM ④ ffmpeg Muxing