Code-Generated Promo Videos (4): ffmpeg Muxing, End-to-End Workflow & Pitfall Cookbook

Overview

This is the final installment of the series. The previous three parts covered generating footage with Remotion, batch voiceover with edge-tts, and offline BGM synthesis with numpy. This part brings everything together: using ffmpeg filter_complex to mux the silent video, 7 voice clips, and one BGM into the final export — along with the end-to-end workflow, a pitfall cookbook, and the underlying principles.

After reading, you will understand:

• Why -c:v copy is much faster and lossless compared to re-encoding
• How to arrange multiple voice clips on a sequential timeline to avoid overlap
• The mathematical meaning of each filter in filter_complex
• How to batch-produce 6 language variants with one command

ffmpeg Muxing

Overall Approach

The muxing stage has a simple job: pack three things into one MP4.

• Video track: Use -c:v copy to directly reuse the silent footage rendered by Remotion — no decoding, no re-encoding, fast and lossless.
• Audio track: Use -filter_complex to mix 7 voice clips and 1 BGM into a single stereo track in real-time.

This works under one premise: the footage is finalized in the Remotion stage. As long as you don’t change the visuals, every muxing run finishes in seconds — the biggest payoff of a layered pipeline.

Timeline: Sequential Placement to Avoid Voice Overlap

This is the most easily overlooked issue in the muxing stage, worth discussing first.

❌ Wrong approach: Hard-align each voice clip’s start to its corresponding scene start at [0.5, 5, 11, 19, 26, 33, 40] seconds. Since each clip is actually 5–8.7 seconds long and the scene intervals are only 4.5–8 seconds, they inevitably overlap and fight each other. The English version, with its longer text, can even exceed the video’s 45-second limit.

The diagram shows the difference between the two approaches clearly. The top row (wrong) hard-aligns each voice clip to its scene start, so clips 5–8.7 seconds long overlap within 4.5–8 second intervals. The bottom row (correct) places them sequentially with a 0.35s GAP between clips, eliminating overlap entirely. The purple dashed playhead sweeps the 45-second timeline, simulating playback progress.

✅ Correct approach: Use ffprobe to read each clip’s actual duration, then arrange them on a sequential timeline — each clip ends with a 0.35s GAP before the next begins. If a version (typically English female) exceeds the video length after placement, apply a very subtle atempo compression (<1.05x, practically imperceptible).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def build_timeline(durs):
    START, GAP, LIMIT = 0.5, 0.35, 44.6   # start offset, inter-clip gap, max usable time
    total = sum(durs)
    span = LIMIT - START - GAP * (len(durs) - 1)   # net time available for speech
    tempo = total / span if total > span else 1.0   # speed up only if over-length
    eff = [d / tempo for d in durs]
    starts, t = [], START
    for e in eff:
        starts.append(t)
        t = t + e + GAP
    return starts, tempo

Why the GAP makes sense: 0.35 seconds is roughly a natural pause in speech — listeners won’t notice it. But it guarantees zero sample overlap between a 5–8 second clip and the next one. Since amix with normalize=0 (see next section) relies on non-overlapping clips to preserve original volume, the GAP serves both perceptual and mix-correctness purposes.

filter_complex Walkthrough

Here is the core of the audio muxing — a filter_complex string (pseudo-code form; see the mixer script below for the actual concatenation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# 1) Each voice: unify format → (atempo only if over-length) → delay to start → boost volume
[i:a]aformat=channel_layouts=stereo:sample_rates=44100,adelay=500|500,volume=1.35[a{i}];

# 2) Merge 7 voices into one stream
[a1][a2]...[a7]amix=inputs=7:normalize=0:dropout_transition=0[voiceraw];

# 3) Split voice into two: one for side-chain, one for final mix
[voiceraw]asplit=2[voice_sc][voice_mix];

# 4) Lower BGM to a very quiet base level
[1:a]aformat=channel_layouts=stereo:sample_rates=44100,volume=0.06[bgmlow];

# 5) Side-chain compress: BGM ducks automatically when voice is active
[bgmlow][voice_sc]sidechaincompress=threshold=0.03:ratio=8:attack=5:release=350:makeup=1[bgmduck];

# 6) Final mix + limiter to prevent clipping
[bgmduck][voice_mix]amix=inputs=2:normalize=0,alimiter=limit=0.95[aout]

Walkthrough:

• aformat=channel_layouts=stereo:sample_rates=44100: Voice clips may be mono MP3s and BGM a stereo WAV — amix requires all inputs to have the same channel layout and sample rate. aformat unifies them. Without it, ffmpeg errors out immediately.
• adelay=500|500: Shifts the entire clip forward by 500ms (| separates left and right channels, in milliseconds). This is the physical implementation of the “timeline” — each clip is delayed to its own start point, laid out sequentially on the time axis.
• volume=1.35: Slightly boosts the voice so it stands out in the final mix. Combined with the ducking from side-chain compression, the voice is present without being jarring.
• amix=inputs=7:normalize=0: Sums multiple streams sample-by-sample. normalize=0 preserves the original amplitude — since clips don’t overlap, at most one stream has signal at any moment, and the sum equals the original single-track volume. If clips overlapped, the sum would exceed 1.0 and cause clipping.
• asplit=2: Duplicates the voice stream into two. One feeds the side-chain compressor (voice_sc), the other joins the final mix (voice_mix).
• sidechaincompress=threshold=0.03:ratio=8:attack=5:release=350:makeup=1: A compressor whose trigger signal comes from a different track (the voice). When voice exceeds the threshold, the compressor instantly reduces BGM volume (ratio=8 = down to 1/8), then recovers with release=350ms when the voice stops. This is the standard broadcast/podcast technique for “voice-over-music” — far more natural than simply lowering BGM volume.
• alimiter=limit=0.95: A final safety limiter. Any sample exceeding 0.95 is clamped, preventing clipping distortion (the “crackling” sound of digital overload).

Below is the complete filter graph expressed as a Mermaid flowchart:

flowchart TD
    V["Video Track<br/>-c:v copy"] --> OUT["Output MP4"]
    BGM["BGM"] --> VOL["volume=0.06"]
    VOICE["7 Voice Clips<br/>→ aformat → adelay → volume=1.35<br/>→ amix(inputs=7,normalize=0)"] --> ASPLIT["asplit=2"]
    ASPLIT --> SC["voice_sc<br/>(side-chain signal)"]
    ASPLIT --> VM["voice_mix<br/>(mix signal)"]
    VOL --> SCS["sidechaincompress<br/>threshold=0.03 ratio=8<br/>attack=5 release=350"]
    SC --> SCS
    SCS --> AMIX2["amix(inputs=2,normalize=0)"]
    VM --> AMIX2
    AMIX2 --> LIMIT["alimiter limit=0.95"]
    LIMIT --> OUT

    classDef default fill:#1a1a2e,stroke:#e0e0e0,stroke-width:1.5,color:#e0e0e0;
    classDef process fill:#2196F3,stroke:#1976D2,color:#fff;
    classDef audio fill:#9C27B0,stroke:#7B1FA2,color:#fff;
    classDef output fill:#4CAF50,stroke:#388E3C,color:#fff;
    classDef trigger fill:#FF9800,stroke:#F57C00,color:#fff;

    class V,OUT output;
    class BGM,VOL,AMIX2,LIMIT process;
    class VOICE,ASPLIT,SC,VM audio;
    class SCS trigger;

Figure 1: ffmpeg filter_complex audio processing graph. Three inputs enter from the left: the video track (green) passes straight through via -c:v copy; the BGM (blue) is first attenuated then enters side-chain compression; the 7 voice clips (purple) are unified in format, delayed to their respective start points, boosted in volume, merged into one stream, then split into a side-chain signal and a mix signal. The side-chain compressor (orange) uses the voice to dynamically duck the BGM, after which both streams merge through a limiter and join the video in the final MP4.

Full Python Mixer Script

The logic above is encapsulated in mix_all.py for batch-processing all language variants:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

import subprocess, os

BASE = "."  # project root
VIDEO = f"{BASE}/MiBeeNVR_Promo_Full.mp4"     # silent footage
BGM   = f"{BASE}/voiceover/bgm.wav"

VERSIONS = [
    ("voice_zh_m",  "MiBeeNVR_Promo_ZH_Male.mp4"),
    ("voice_zh_f",  "MiBeeNVR_Promo_ZH_Female.mp4"),
    ("voice_en_m",  "MiBeeNVR_Promo_EN_Male.mp4"),
    ("voice_en_f",  "MiBeeNVR_Promo_EN_Female.mp4"),
    ("voice_yue_m", "MiBeeNVR_Promo_YUE_Male.mp4"),
    ("voice_yue_f", "MiBeeNVR_Promo_YUE_Female.mp4"),
]

def dur(f):
    """Get audio file duration (seconds) via ffprobe"""
    return float(subprocess.check_output(
        ["ffprobe", "-v", "error", "-show_entries", "format=duration",
         "-of", "csv=p=0", f]).strip())

def mix_one(subdir, out):
    clips = [f"{BASE}/voiceover/{subdir}/voice_{i}.mp3" for i in range(1, 8)]
    durs = [dur(c) for c in clips]
    starts, tempo = build_timeline(durs)          # see 5.2

    filt, mixin = "", ""
    for i in range(1, 8):
        d_ms = int(round(starts[i-1] * 1000))
        tempo_f = f"atempo={tempo:.5f}," if tempo > 1.001 else ""
        filt += (f"[{i+1}:a]aformat=channel_layouts=stereo:sample_rates=44100,"
                 f"{tempo_f}adelay={d_ms}|{d_ms},volume=1.35[a{i}];")
        mixin += f"[a{i}]"
    filt += f"{mixin}amix=inputs=7:normalize=0:dropout_transition=0[voiceraw];"
    filt += "[voiceraw]asplit=2[voice_sc][voice_mix];"
    filt += "[1:a]aformat=channel_layouts=stereo:sample_rates=44100,volume=0.06[bgmlow];"
    filt += ("[bgmlow][voice_sc]sidechaincompress="
             "threshold=0.03:ratio=8:attack=5:release=350:makeup=1[bgmduck];")
    filt += ("[bgmduck][voice_mix]amix=inputs=2:normalize=0,"
             "alimiter=limit=0.95[aout]")

    cmd = ["ffmpeg", "-y", "-i", VIDEO, "-i", BGM] + \
          [x for c in clips for x in ["-i", c]]
    cmd += ["-filter_complex", filt,
            "-map", "0:v", "-map", "[aout]",
            "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
            "-shortest", f"{BASE}/{out}"]
    subprocess.run(cmd, check=True,
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    print("OK ->", out)

for subdir, out in VERSIONS:
    mix_one(subdir, out)

Input order: -i VIDEO -i BGM -i voice_1 ... -i voice_7. In the filter graph, [0:v] is the video, [1:a] is the BGM, and [2:a]~[8:a] are the 7 voice clips. Each clip’s filter string is dynamically generated: read actual duration → build_timeline computes the start offset → adelay places it at that offset.

End-to-End Workflow

Complete Pipeline Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# ① Render footage (required after any scene/icon/font-size change)
cd remotion-videos
npx remotion render promo "../MiBeeNVR_Promo_Full.mp4" --log=error

# ② Generate voiceovers (needed when adding a new language/voice; skip for pure visual tweaks)
cd voiceover
python gen_all_voices.py        # batch-produce all language variants
python gen_bgm.py               # only if changing BGM style

# ③ Mux all variants
python mix_all.py

Three commands, one pipeline. Each time you change the script or tweak the visuals, just repeat step ① (re-render footage) → ③ (re-mux). If you’re only changing BGM style, skip ① and ② and just run ③.

Checklist for Adding a New Language / Voice

1. Add a line to gen_all_voices.py’s VERSIONS: (subdir, texts, voice_id, rate).
2. Add a line to mix_all.py’s VERSIONS: (subdir, output_filename).
3. Run gen_all_voices.py (or run just that single version to save time).
4. Run mix_all.py (it automatically uses the latest MiBeeNVR_Promo_Full.mp4).

⚠️ If you change the visuals, you must re-render AND re-mux. Scripts only modify source code — they don’t automatically produce output. This series has fallen into this trap: enthusiastically editing code, then checking the output and wondering why nothing changed. Remember: edit source → re-render → re-mux, all three required.

Pitfall Cookbook

Further Learning

• Remotion official docs: remotion.dev/docs — Sequence, interpolate, spring, AbsoluteFill
• edge-tts: edge-tts --list-voices for all voices; GitHub rany2/edge-tts
• ffmpeg filter docs: Focus on amix, adelay, sidechaincompress, alimiter, atempo, aformat
• Acoustic fundamentals: Beating, comb filtering, convolution reverb — understanding these three will prevent your BGM from ever causing headaches again

Principles Deep Dive

ffmpeg Principles: How a Filter Graph Muxes Multiple AV Streams

Container vs. Codec: MP4 is a “box.”

MP4 is a container format that holds independent encoded streams. The video stream uses H.264 (inter-frame compression, relying on differences between frames), and the audio stream uses AAC (perceptual audio coding). The silent MP4 rendered by Remotion already contains an H.264 video stream — our job is simply to add audio to it, without touching the video encoding.

Why is -c:v copy both fast and lossless?

-c:v copy tells ffmpeg to skip decoding and re-encoding entirely, copying the video stream byte-for-byte from the input file to the output file. This is only valid when “the footage is finalized and needs no modifications.” In this project, the visuals are set during the Remotion stage; the muxing stage only touches audio.

Compared to re-encoding:

• Speed: copy is O(n) container-level copying (~5 seconds for a 45-second video); re-encoding is O(n) frame-by-frame decode-then-encode (potentially 2–5 minutes)
• Quality: copy is perfectly lossless; even high-bitrate re-encoding introduces minor generational loss

-c:v copy tells ffmpeg to skip decoding and re-encoding entirely, copying the video stream byte-for-byte from the input file to the output file. This is only valid when “the footage is finalized and needs no modifications.” In this project, the visuals are set during the Remotion stage; the muxing stage only touches audio.

Compared to re-encoding:

• Speed: copy is O(n) container-level copying (~5 seconds for a 45-second video); re-encoding is O(n) frame-by-frame decode-then-encode (potentially 2–5 minutes)
• Quality: copy is perfectly lossless; even high-bitrate re-encoding introduces minor generational loss

-filter_complex is a directed graph.

It is far more powerful than a simple -af (audio filter) — -filter_complex lets you organize multiple inputs, multiple outputs, and multiple processing paths into a directed acyclic graph (DAG):

• Each input has a fixed label: [0:v] (video of input 1), [1:a] (audio of input 2), and so on
• Each filter node reads one or more labeled inputs and outputs a new label
• The -map option selects which streams go into the output file

This project’s filter graph has 10 inputs (1 video + 1 BGM + 7 voice clips), with 9 filter nodes (8× aformat+adelay+volume + 1× amix + 1× asplit + 1× sidechaincompress + 1× amix+alimiter). See the Mermaid diagram in 5.3 for a clear visual.

It is far more powerful than a simple -af (audio filter) — -filter_complex lets you organize multiple inputs, multiple outputs, and multiple processing paths into a directed acyclic graph (DAG):

• Each input has a fixed label: [0:v] (video of input 1), [1:a] (audio of input 2), and so on
• Each filter node reads one or more labeled inputs and outputs a new label
• The -map option selects which streams go into the output file

This project’s filter graph has 10 inputs (1 video + 1 BGM + 7 voice clips), with 9 filter nodes (8× aformat+adelay+volume + 1× amix + 1× asplit + 1× sidechaincompress + 1× amix+alimiter). See the Mermaid diagram in 5.3 for a clear visual.

The audio timeline model.

Audio is a continuous time axis, and adelay shifts a waveform along that axis. adelay=500|500 pushes the entire clip forward by 500ms (| separates left and right channels, in milliseconds). The 7 clips are each delayed to their own start point — laid out sequentially on the timeline with zero overlap. This is the physical implementation of the “sequential timeline avoids overlap” principle from 5.2.

Audio is a continuous time axis, and adelay shifts a waveform along that axis. adelay=500|500 pushes the entire clip forward by 500ms (| separates left and right channels, in milliseconds). The 7 clips are each delayed to their own start point — laid out sequentially on the timeline with zero overlap. This is the physical implementation of the “sequential timeline avoids overlap” principle from 5.2.

The math behind amix.

amix performs sample-by-sample addition (sum). With normalize=0, it does not divide by the number of inputs, preserving the original amplitude:

• When streams do not overlap (our case), at most one stream has signal at any given moment, so the sum ≈ the single-track original volume. The voice retains its natural loudness.
• normalize=1 (the default) divides the sum by the number of inputs, which is appropriate for concurrent microphone mixing but unnecessarily attenuates sequential, non-overlapping voice clips.

amix performs sample-by-sample addition (sum). With normalize=0, it does not divide by the number of inputs, preserving the original amplitude:

• When streams do not overlap (our case), at most one stream has signal at any given moment, so the sum ≈ the single-track original volume. The voice retains its natural loudness.
• normalize=1 (the default) divides the sum by the number of inputs, which is appropriate for concurrent microphone mixing but unnecessarily attenuates sequential, non-overlapping voice clips.

sidechaincompress (side-chain compression / ducking).

A conventional compressor attenuates its own input based on that same input’s level. A side-chain compressor reads the trigger signal from a different track.

In this project: [bgmlow][voice_sc]sidechaincompress — the BGM is the signal being processed, and the voice is the trigger. When the voice exceeds threshold=0.03, BGM volume is instantly reduced to 1/8 (ratio=8); when the voice stops, the BGM recovers gradually with release=350ms. The result: BGM automatically “makes way” at the start of every sentence and fades gently back afterward — as natural as a professional radio mix.

A conventional compressor attenuates its own input based on that same input’s level. A side-chain compressor reads the trigger signal from a different track.

In this project: [bgmlow][voice_sc]sidechaincompress — the BGM is the signal being processed, and the voice is the trigger. When the voice exceeds threshold=0.03, BGM volume is instantly reduced to 1/8 (ratio=8); when the voice stops, the BGM recovers gradually with release=350ms. The result: BGM automatically “makes way” at the start of every sentence and fades gently back afterward — as natural as a professional radio mix.

alimiter.

After mixing, the digital signal might exceed 0 dBFS (digital full scale). Clipping occurs when these peaks are flattened, producing a harsh “crackling” distortion. alimiter=limit=0.95 clamps any sample exceeding 0.95, guaranteeing the final output never clips.

After mixing, the digital signal might exceed 0 dBFS (digital full scale). Clipping occurs when these peaks are flattened, producing a harsh “crackling” distortion. alimiter=limit=0.95 clamps any sample exceeding 0.95, guaranteeing the final output never clips.

atempo (time-stretch without pitch change).

atempo changes playback speed without altering pitch. The underlying technique is WSOLA (Waveform Similarity Overlap-Add) or phase vocoding — skipping or repeating waveform segments in the time domain while preserving pitch through overlap and phase adjustment. In this project, it is only applied very gently (<1.05x) to versions whose sequential layout exceeds the 45-second limit (typically the longest English female voice). The effect is practically imperceptible.

atempo changes playback speed without altering pitch. The underlying technique is WSOLA (Waveform Similarity Overlap-Add) or phase vocoding — skipping or repeating waveform segments in the time domain while preserving pitch through overlap and phase adjustment. In this project, it is only applied very gently (<1.05x) to versions whose sequential layout exceeds the 45-second limit (typically the longest English female voice). The effect is practically imperceptible.

Why aformat is mandatory.

Voice MP3s may be mono, while the BGM WAV is stereo. Sample rates may also differ (22050 / 44100 / 48000, etc.). amix requires strictly identical channel layouts and sample rates across all inputs — or it errors out immediately. aformat=channel_layouts=stereo:sample_rates=44100 unifies all audio to 44.1kHz stereo, allowing all downstream filters to operate safely.

Voice MP3s may be mono, while the BGM WAV is stereo. Sample rates may also differ (22050 / 44100 / 48000, etc.). amix requires strictly identical channel layouts and sample rates across all inputs — or it errors out immediately. aformat=channel_layouts=stereo:sample_rates=44100 unifies all audio to 44.1kHz stereo, allowing all downstream filters to operate safely.

The Big Picture: Turning “Creation” into “Parameters + Code”

flowchart TD
    COPY["Script<br/>String"] --> PIPELINE
    VISUAL["Visuals<br/>React Components"] --> PIPELINE
    VOICE["Voiceover<br/>Voice ID"] --> PIPELINE
    BGM["BGM<br/>Synth Parameters"] --> PIPELINE
    MIX["Muxing<br/>Filter Graph"] --> PIPELINE
    PIPELINE["Versionable<br/>Reproducible<br/>Low Marginal Cost<br/>Debuggable"]

    classDef default fill:#1a1a2e,stroke:#e0e0e0,stroke-width:1.5,color:#e0e0e0;
    classDef str fill:#FF9800,stroke:#F57C00,color:#fff;
    classDef react fill:#2196F3,stroke:#1976D2,color:#fff;
    classDef tts fill:#9C27B0,stroke:#7B1FA2,color:#fff;
    classDef bgm fill:#4CAF50,stroke:#388E3C,color:#fff;
    classDef mix fill:#2196F3,stroke:#1976D2,color:#fff;
    classDef result fill:#4CAF50,stroke:#388E3C,color:#fff;

    class COPY str;
    class VISUAL react;
    class VOICE tts;
    class BGM bgm;
    class MIX mix;
    class PIPELINE result;

Figure 2: The overall creative principle. All five creative elements exist as code or parameters: the script is a string (orange), the visuals are React components (blue), the voiceover is a voice ID (purple), the BGM is defined by numpy synthesis parameters (green), and the final mux is a filter graph (blue). Every piece is versionable text — goes into Git, can be diffed and rolled back. The result is a reproducible media production pipeline with low marginal cost (green).

The core value of this methodology lies in turning all creative elements into code:

• Versionable: Script = string, visuals = React components, voice = voice ID, BGM = synthesis parameters, muxing = filter graph — all text, all in Git, all diffable and revertible
• Reproducible: The same parameters always produce the same result. Remotion’s pure-function frame model combined with deterministic muxing scripts guarantee: the Nth run is identical to the first
• Low marginal cost: Footage is rendered once (silent); 6 languages are just 6 different “audio overlay recipes” reusing the same video. Adding a new language = two lines of config + one muxing run — zero cost for visuals
• Debuggable: Each stage is independent (footage / voiceover / BGM / muxing). Problems are isolated by layer: visual bugs → check Remotion; voice artifacts → check edge-tts; headache-inducing BGM → check the synth; timeline overlap → check the mixer script

Once you internalize this principle, you are no longer just “using tools” — you are commanding a media production pipeline with code. Changing a sentence, swapping a voice, adjusting a reverb — these become parameter edits, not creative rework.

Series: ① Overview & Remotion ② edge-tts Voiceover ③ numpy BGM ④ ffmpeg Muxing