Use AI to Clone Voices & Speak OTHER LANGUAGES! - Elevenlabs + ChatGPT 4

ElevenLabs’ newly added multilingual text-to-speech model can render the same cloned voice across multiple languages—English, German, Polish, Spanish, Italian, French, Portuguese, and Hindi—while keeping the speaker’s identity largely intact. In side-by-side tests using a voice cloned from the creator, the multilingual model produced English that sounded close to the older monolingual model, but it also introduced noticeable instability during longer passages, including glitches, shifting tone, and background artifacts.

The first major check was a tongue-twister in English generated with ChatGPT, then spoken using both models. The monolingual and multilingual outputs were broadly similar in intelligibility and overall quality, with differences showing up more as generation variation than as a clear upgrade or downgrade. ElevenLabs’ multilingual system was also flagged as experimental, with guidance that some numbers and symbols may be mispronounced—so the tests included attention to how the model handles pronunciation rather than relying only on casual reading.

German provided a clearer contrast. When the cloned voice—trained on English—attempted German, the monolingual model delivered a heavy, thick American accent, while the multilingual model sounded more authentically German to the listener’s ear. The experiment also compared direct German output versus “phonetic” handling (spelling words out to guide pronunciation). Both approaches looked German on the page, but the multilingual model’s output still differed from the monolingual version in cadence and accent strength, suggesting the multilingual training meaningfully changes how the voice shapes foreign phonemes.

Spanish revealed the model’s current ceiling. A longer Spanish story initially sounded decent but then degraded: the audio fluctuated between quiet and loud, the voice changed character, and the narration developed a “glitchy” quality that felt less human. When the story was translated back into English and read with the same cloned voice, the monolingual model sounded smoother and more consistent, reinforcing the pattern that multilingual performance is strongest on shorter, simpler lines and weaker on extended narration.

Beyond multilingual cloning, the transcript also highlights ElevenLabs’ “voice design” workflow for creating voices from scratch. With a paid Voice Design tier, the creator generated a randomized voice, iterated on accent settings (including attempts at British, Australian, and American), and then used the new synthetic voice to read a scripted message. The generated voices were described as capable of sounding impressive, though accent controls appeared inconsistent—changing accent strength could unexpectedly alter pitch or tone.

Overall, the tests suggest ElevenLabs’ multilingual model is a real step toward identity-preserving voice translation, especially for short-form dialogue and languages like German where accent shaping is more convincing. But longer multilingual narration still needs work to eliminate artifacts and maintain stable, human-like delivery—an issue the creator repeatedly noticed when switching between multilingual and monolingual outputs.