@jhustle44 / jcr-guide-pro
Code Commits Issues Pulls Cloud Agent CI/CD
copilot/create-zip-file-again
copilot/add-firebase copilot/change-app-name-to-jcrguru copilot/create-zip-file copilot/create-zip-file-again copilot/merge-any-updates main master
md 788 lines 12.9 KB
Rendered Raw
1 # Voiceover with ElevenLabs
2
3 Complete implementation guide for adding AI-generated voiceovers to videos using ElevenLabs TTS and FFmpeg audio mixing.
4
5 ## Table of Contents
6
7 1. [Setup](#setup)
8 2. [Generating Voiceover Audio](#generating-voiceover-audio)
9
10 3. [Voice Selection](#voice-selection)
11 4. [Mixing Audio into Video](#mixing-audio-into-video)
12
13 5. [Timed Voiceover Segments](#timed-voiceover-segments)
14 6. [Complete Pipeline Example](#complete-pipeline-example)
15
16 ---
17
18 ## Setup
19
20 ### 1. Connect ElevenLabs via Replit Integration
21
22 ElevenLabs is available as a Replit connector. Set it up through the integration system:
23
24 ```javascript
25
26 // In the code execution sandbox:
27
28 const results = await searchIntegrations("elevenlabs");
29
30 console.log(results);
31
32 // Propose the connector to the user (they'll complete OAuth)
33
34 await proposeIntegration("connector:ccfg_elevenlabs_...");
35
36 ```
37
38 After the user authorizes, on the next loop:
39
40 ```javascript
41
42 // Add the connection to the project
43
44 const addResult = await addIntegration("connection:conn_elevenlabs_...");
45
46 console.log(addResult);
47
48 ```
49
50 ### 2. Access Credentials
51
52 Use `listConnections` in the code execution sandbox to get the API key:
53
54 ```javascript
55
56 const conns = await listConnections('elevenlabs');
57
58 if (conns.length > 0) {
59
60 const apiKey = conns[0].settings.api_key;
61
62 // Use this key for ElevenLabs API calls
63
64 }
65
66 ```
67
68 ### 3. Install Dependencies
69
70 No special SDK is needed — ElevenLabs has a straightforward REST API. Use `fetch` or install the official SDK:
71
72 ```bash
73
74 pnpm --filter @workspace/scripts add elevenlabs
75
76 # or for the API server:
77
78 pnpm --filter @workspace/api-server add elevenlabs
79
80 ```
81
82 ---
83
84 ## Generating Voiceover Audio
85
86 ### Using the ElevenLabs REST API directly
87
88 ```typescript
89
90 import * as fs from 'fs';
91
92 async function generateVoiceover(
93
94 text: string,
95
96 outputPath: string,
97
98 apiKey: string,
99
100 voiceId = 'JBFqnCBsd6RMkjVDRZzb', // Default: "George" - warm, narrative voice
101
102 options: {
103
104 stability?: number;
105
106 similarityBoost?: number;
107
108 style?: number;
109
110 modelId?: string;
111
112 } = {}
113
114 ): Promise<string> {
115
116 const {
117
118 stability = 0.5,
119
120 similarityBoost = 0.75,
121
122 style = 0.0,
123
124 modelId = 'eleven_multilingual_v2'
125
126 } = options;
127
128 const response = await fetch(
129
130 `https://api.elevenlabs.io/v1/text-to-speech/${voiceId}`,
131
132 {
133
134 method: 'POST',
135
136 headers: {
137
138 'Content-Type': 'application/json',
139
140 'xi-api-key': apiKey
141
142 },
143
144 body: JSON.stringify({
145
146 text,
147
148 model_id: modelId,
149
150 voice_settings: {
151
152 stability,
153
154 similarity_boost: similarityBoost,
155
156 style,
157
158 use_speaker_boost: true
159
160 }
161
162 })
163
164 }
165
166 );
167
168 if (!response.ok) {
169
170 const error = await response.text();
171
172 throw new Error(`ElevenLabs TTS failed: ${response.status} ${error}`);
173
174 }
175
176 const buffer = Buffer.from(await response.arrayBuffer());
177
178 await fs.promises.writeFile(outputPath, buffer);
179
180 return outputPath;
181
182 }
183
184 ```
185
186 ### Using the ElevenLabs SDK
187
188 ```typescript
189
190 import { ElevenLabsClient } from 'elevenlabs';
191
192 import * as fs from 'fs';
193
194 async function generateVoiceoverSDK(
195
196 text: string,
197
198 outputPath: string,
199
200 apiKey: string,
201
202 voiceId = 'JBFqnCBsd6RMkjVDRZzb'
203
204 ): Promise<string> {
205
206 const client = new ElevenLabsClient({ apiKey });
207
208 const audioStream = await client.textToSpeech.convert(voiceId, {
209
210 text,
211
212 model_id: 'eleven_multilingual_v2',
213
214 voice_settings: {
215
216 stability: 0.5,
217
218 similarity_boost: 0.75,
219
220 style: 0.0
221
222 }
223
224 });
225
226 const chunks: Buffer[] = [];
227
228 for await (const chunk of audioStream) {
229
230 chunks.push(Buffer.from(chunk));
231
232 }
233
234 await fs.promises.writeFile(outputPath, Buffer.concat(chunks));
235
236 return outputPath;
237
238 }
239
240 ```
241
242 ---
243
244 ## Voice Selection
245
246 ### Listing Available Voices
247
248 ```typescript
249
250 async function listVoices(apiKey: string) {
251
252 const response = await fetch('https://api.elevenlabs.io/v1/voices', {
253
254 headers: { 'xi-api-key': apiKey }
255
256 });
257
258 const data = await response.json();
259
260 return data.voices.map((v: any) => ({
261
262 id: v.voice_id,
263
264 name: v.name,
265
266 category: v.category,
267
268 labels: v.labels,
269
270 previewUrl: v.preview_url
271
272 }));
273
274 }
275
276 ```
277
278 ### Popular Pre-made Voices
279
280 These are commonly available ElevenLabs voices good for voiceovers:
281
282 | Voice | ID | Best For |
283
284 |-------|-----|----------|
285
286 | George | `JBFqnCBsd6RMkjVDRZzb` | Warm narration, documentaries |
287
288 | Rachel | `21m00Tcm4TlvDq8ikWAM` | Professional, clear narration |
289
290 | Adam | `pNInz6obpgDQGcFmaJgB` | Deep, authoritative narration |
291
292 | Bella | `EXAVITQu4vr4xnSDxMaL` | Friendly, conversational |
293
294 | Antoni | `ErXwobaYiN019PkySvjV` | Calm, measured narration |
295
296 | Domi | `AZnzlk1XvdvUeBnXmlld` | Energetic, youthful |
297
298 When the user doesn't specify a voice, ask what tone they want (professional, warm, energetic, etc.) and suggest an appropriate voice.
299
300 ### Voice Settings Explained
301
302 - **Stability** (0.0–1.0): Higher = more consistent/monotone, Lower = more expressive/varied. Default 0.5 is balanced. Use 0.7+ for narration, 0.3 for dramatic reads.
303 - **Similarity Boost** (0.0–1.0): How closely to match the original voice. 0.75 is a good default. Higher values may introduce artifacts.
304
305 - **Style** (0.0–1.0): Exaggeration of the voice's style. 0.0 is neutral. Higher values amplify the voice's character but can sound unnatural. Keep at 0.0 for voiceovers.
306
307 ---
308
309 ## Mixing Audio into Video
310
311 ### Mode 1: Replace Audio Entirely
312
313 Remove existing audio and use only the voiceover:
314
315 ```bash
316
317 ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
318
319 -map 0:v -map 1:a \\
320
321 -c:v copy -c:a aac -b:a 192k \\
322
323 -shortest output.mp4
324
325 ```
326
327 ```typescript
328
329 import { execFile } from 'child_process';
330
331 import { promisify } from 'util';
332
333 const execFileAsync = promisify(execFile);
334
335 async function replaceAudio(videoPath: string, voiceoverPath: string, outputPath: string) {
336
337 await execFileAsync('ffmpeg', [
338
339 '-y',
340
341 '-i', videoPath,
342
343 '-i', voiceoverPath,
344
345 '-map', '0:v',
346
347 '-map', '1:a',
348
349 '-c:v', 'copy',
350
351 '-c:a', 'aac', '-b:a', '192k',
352
353 '-shortest',
354
355 outputPath
356
357 ]);
358
359 }
360
361 ```
362
363 ### Mode 2: Mix Over (duck existing audio)
364
365 Layer voiceover on top of existing audio, reducing the existing audio volume:
366
367 ```bash
368
369 ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
370
371 -filter_complex "[0:a]volume=0.2[bg];[bg][1:a]amix=inputs=2:duration=first[outa]" \\
372
373 -map 0:v -map "[outa]" \\
374
375 -c:v copy -c:a aac -b:a 192k \\
376
377 output.mp4
378
379 ```
380
381 ```typescript
382
383 async function mixOverAudio(
384
385 videoPath: string,
386
387 voiceoverPath: string,
388
389 outputPath: string,
390
391 bgVolume = 0.2 // How much to reduce existing audio (0.0 = silent, 1.0 = full)
392
393 ) {
394
395 await execFileAsync('ffmpeg', [
396
397 '-y',
398
399 '-i', videoPath,
400
401 '-i', voiceoverPath,
402
403 '-filter_complex',
404
405 `[0:a]volume=${bgVolume}[bg];[bg][1:a]amix=inputs=2:duration=first[outa]`,
406
407 '-map', '0:v',
408
409 '-map', '[outa]',
410
411 '-c:v', 'copy',
412
413 '-c:a', 'aac', '-b:a', '192k',
414
415 outputPath
416
417 ]);
418
419 }
420
421 ```
422
423 ### Mode 3: Add as Track (full volume both)
424
425 Keep existing audio at full volume, add voiceover on top:
426
427 ```bash
428
429 ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
430
431 -filter_complex "[0:a][1:a]amix=inputs=2:duration=first[outa]" \\
432
433 -map 0:v -map "[outa]" \\
434
435 -c:v copy -c:a aac -b:a 192k \\
436
437 output.mp4
438
439 ```
440
441 ```typescript
442
443 async function addAudioTrack(videoPath: string, voiceoverPath: string, outputPath: string) {
444
445 await execFileAsync('ffmpeg', [
446
447 '-y',
448
449 '-i', videoPath,
450
451 '-i', voiceoverPath,
452
453 '-filter_complex',
454
455 '[0:a][1:a]amix=inputs=2:duration=first[outa]',
456
457 '-map', '0:v',
458
459 '-map', '[outa]',
460
461 '-c:v', 'copy',
462
463 '-c:a', 'aac', '-b:a', '192k',
464
465 outputPath
466
467 ]);
468
469 }
470
471 ```
472
473 ### Adding Voiceover to a Video with No Audio
474
475 If the input video has no audio stream:
476
477 ```bash
478
479 ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
480
481 -map 0:v -map 1:a \\
482
483 -c:v copy -c:a aac -b:a 192k \\
484
485 -shortest output.mp4
486
487 ```
488
489 ---
490
491 ## Timed Voiceover Segments
492
493 For videos that need voiceover at specific timestamps (not continuous narration), generate separate audio clips and position them on a timeline.
494
495 ### Step 1: Define the Script with Timestamps
496
497 ```typescript
498
499 interface VoiceoverSegment {
500
501 text: string;
502
503 startTime: number; // seconds
504
505 voiceId?: string; // optional per-segment voice
506
507 }
508
509 const script: VoiceoverSegment[] = [
510
511 { text: "Welcome to our product overview.", startTime: 0 },
512
513 { text: "Here's how it works in three simple steps.", startTime: 5.5 },
514
515 { text: "Step one: upload your content.", startTime: 10 },
516
517 { text: "Step two: customize your settings.", startTime: 18 },
518
519 { text: "Step three: share with the world.", startTime: 25 },
520
521 ];
522
523 ```
524
525 ### Step 2: Generate Each Segment
526
527 ```typescript
528
529 async function generateSegments(
530
531 segments: VoiceoverSegment[],
532
533 tmpDir: string,
534
535 apiKey: string,
536
537 defaultVoiceId: string
538
539 ): Promise<string[]> {
540
541 const paths: string[] = [];
542
543 for (let i = 0; i < segments.length; i++) {
544
545 const outputPath = path.join(tmpDir, `segment_${i}.mp3`);
546
547 await generateVoiceover(
548
549 segments[i].text,
550
551 outputPath,
552
553 apiKey,
554
555 segments[i].voiceId || defaultVoiceId
556
557 );
558
559 paths.push(outputPath);
560
561 }
562
563 return paths;
564
565 }
566
567 ```
568
569 ### Step 3: Create a Combined Audio Track with Positioning
570
571 Use FFmpeg to position each segment at its target timestamp and mix them into a single audio track:
572
573 ```typescript
574
575 async function assembleTimedVoiceover(
576
577 segments: VoiceoverSegment[],
578
579 segmentPaths: string[],
580
581 totalDuration: number,
582
583 outputPath: string
584
585 ) {
586
587 const inputs: string[] = [];
588
589 const filterParts: string[] = [];
590
591 for (let i = 0; i < segmentPaths.length; i++) {
592
593 inputs.push('-i', segmentPaths[i]);
594
595 const delay = Math.round(segments[i].startTime * 1000);
596
597 filterParts.push(`[${i}:a]adelay=${delay}|${delay}[a${i}]`);
598
599 }
600
601 const mixInputs = segments.map((_, i) => `[a${i}]`).join('');
602
603 const filterComplex = [
604
605 ...filterParts,
606
607 `${mixInputs}amix=inputs=${segments.length}:normalize=0[mixed]`,
608
609 `[mixed]apad=whole_dur=${totalDuration}[out]`
610
611 ].join(';');
612
613 await execFileAsync('ffmpeg', [
614
615 '-y',
616
617 ...inputs,
618
619 '-filter_complex', filterComplex,
620
621 '-map', '[out]',
622
623 '-c:a', 'aac', '-b:a', '192k',
624
625 outputPath
626
627 ]);
628
629 }
630
631 ```
632
633 ### Step 4: Mix the Assembled Voiceover into the Video
634
635 Use one of the mixing modes above (replace, mix over, or add as track) with the assembled voiceover file.
636
637 ---
638
639 ## Complete Pipeline Example
640
641 ```typescript
642
643 import * as path from 'path';
644
645 import * as os from 'os';
646
647 async function addVoiceoverToVideo(
648
649 videoPath: string,
650
651 script: string | VoiceoverSegment[],
652
653 apiKey: string,
654
655 options: {
656
657 voiceId?: string;
658
659 mode?: 'replace' | 'mix' | 'add';
660
661 bgVolume?: number;
662
663 } = {}
664
665 ) {
666
667 const {
668
669 voiceId = 'JBFqnCBsd6RMkjVDRZzb',
670
671 mode = 'mix',
672
673 bgVolume = 0.2
674
675 } = options;
676
677 const tmpDir = path.join(os.tmpdir(), `voiceover_${Date.now()}`);
678
679 await fs.promises.mkdir(tmpDir, { recursive: true });
680
681 const outputPath = videoPath.replace(/(\\\w+)$/, '_voiceover$1');
682
683 if (typeof script === 'string') {
684
685 // Simple full-video voiceover
686
687 console.log('Generating voiceover audio...');
688
689 const voiceoverPath = path.join(tmpDir, 'voiceover.mp3');
690
691 await generateVoiceover(script, voiceoverPath, apiKey, voiceId);
692
693 console.log(`Mixing audio (mode: ${mode})...`);
694
695 switch (mode) {
696
697 case 'replace':
698
699 await replaceAudio(videoPath, voiceoverPath, outputPath);
700
701 break;
702
703 case 'mix':
704
705 await mixOverAudio(videoPath, voiceoverPath, outputPath, bgVolume);
706
707 break;
708
709 case 'add':
710
711 await addAudioTrack(videoPath, voiceoverPath, outputPath);
712
713 break;
714
715 }
716
717 } else {
718
719 // Timed segments
720
721 console.log(`Generating ${script.length} voiceover segments...`);
722
723 const segmentPaths = await generateSegments(script, tmpDir, apiKey, voiceId);
724
725 // Get video duration
726
727 const { stdout } = await execFileAsync('ffprobe', [
728
729 '-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
730
731 ]);
732
733 const duration = parseFloat(JSON.parse(stdout).format.duration);
734
735 console.log('Assembling timed voiceover...');
736
737 const assembledPath = path.join(tmpDir, 'assembled_voiceover.aac');
738
739 await assembleTimedVoiceover(script, segmentPaths, duration, assembledPath);
740
741 console.log(`Mixing audio (mode: ${mode})...`);
742
743 switch (mode) {
744
745 case 'replace':
746
747 await replaceAudio(videoPath, assembledPath, outputPath);
748
749 break;
750
751 case 'mix':
752
753 await mixOverAudio(videoPath, assembledPath, outputPath, bgVolume);
754
755 break;
756
757 case 'add':
758
759 await addAudioTrack(videoPath, assembledPath, outputPath);
760
761 break;
762
763 }
764
765 }
766
767 // Clean up
768
769 await fs.promises.rm(tmpDir, { recursive: true, force: true });
770
771 console.log(`Voiceover added: ${outputPath}`);
772
773 return outputPath;
774
775 }
776
777 ```
778
779 ## Tips
780
781 - **Character limits:** ElevenLabs has a per-request character limit (typically 5,000 characters). For long narrations, split the text into chunks and concatenate the resulting audio files.
782 - **Audio format:** ElevenLabs returns MP3 by default. You can request other formats via the `output_format` parameter (e.g., `pcm_16000` for raw PCM, `mp3_44100_128` for specific bitrate).
783
784 - **Pacing:** If the voiceover needs to match specific video timing, adjust the text length per segment rather than trying to speed up/slow down the generated audio. Shorter text = shorter audio.
785 - **Silence padding:** Use `apad` or `adelay` in FFmpeg to position audio precisely. ElevenLabs doesn't add leading silence, so timing is exact.
786
787 - **Cost awareness:** ElevenLabs charges per character. Long narrations can add up. Let the user know the approximate character count before generating.
788 - **Multilingual:** The `eleven_multilingual_v2` model supports 29 languages. The text language is auto-detected. No special configuration needed — just write the text in the target language.