Kyutai STT & TTS - A Perfect Local Voice Solution?

Kyutai’s latest local speech stack—QI TTS for text-to-speech and speech-to-text for ASR—pairs fast, small models with voice conditioning that can sound remarkably natural, including on-the-fly voice cloning from a short sample. The practical catch: Kyutai is not releasing the voice embedding/cloning model itself, limiting users to a curated set of downloadable voice embeddings and preventing straightforward fine-tuning for new languages or custom voices.

The ASR side (speech-to-text) is currently limited to English and French, but it transcribes quickly and appears to run efficiently on capable hardware. The bigger story is the architecture Kyutai has been building toward: low-latency speech in, speech out, with ASR feeding into a larger language model and then returning through TTS. Earlier experiments reportedly showed odd behavior when prompting the integrated model, but the latency-focused design remained the key attraction—making local, interactive voice workflows more feasible.

On the TTS side, Kyutai’s QI TTS uses a 1.6B model and supports English and French, with multiple selectable voices. The quality is framed as competitive with other well-known TTS systems (including comparisons to Chatterbox, Dier, and 11 Labs), and the release includes a strong demonstration of voice cloning: TTS is conditioned on a 10-second voice sample, reproducing intonation and voice characteristics even when the sample is “weird” or stylized. In the example, the system quickly generates speech in the target voice, suggesting the model can capture prosody and identity cues from limited audio.

Where the release frustrates power users is in what’s missing. Kyutai provides voice embeddings and a repository of voices based on datasets such as Espresso and VCTK, but explicitly withholds the voice embedding model to ensure cloning happens only with consent. That means developers can’t directly fine-tune the cloning pipeline for other languages or train their own voice-to-embedding mapping without additional tooling or data. The transcript notes Kyutai is “exploring ideas” for expanding beyond English and French, but no fine-tuning path is available yet.

The practical workaround comes from the released materials on Hugging Face and Kyutai’s GitHub code examples. The voices are distributed as safe tensor embeddings (with a “safe tensors” option for embeddings and a “web” option for listening). By loading an embedding, conditioning the QI TTS model on it, and generating audio from custom text, the user can produce speech in the provided voices. The transcript also shows experimentation with voice embeddings: inspecting their tensor shape, saving them, and blending two voices by averaging embeddings to create an in-between “latent space” voice. The result is speech that sits between the two source voices rather than simply switching.

Overall, the release is positioned as a strong local foundation for ASR and TTS, with additional promise if/when an MLX version arrives for Mac hardware. For now, the system delivers fast, high-quality speech synthesis and usable voice control—just not the full cloning model needed for unrestricted customization.