Cloning my Voice Into an AI Assistant

Cloning a voice locally is possible with open-source tools—if the data is clean and the training pipeline is handled carefully. The core takeaway is that voice quality depends less on “magic” and more on disciplined preprocessing: short, noise-free audio clips; accurate transcriptions; and a training setup that matches the model’s software and hardware requirements.

The workflow starts with building a dataset. For a self-voice, the tutorial uses Piper Recording Studio to capture hundreds of spoken utterances through a microphone, then exports the recordings into Piper’s expected structure: a Wave folder plus a metadata CSV pairing each audio filename with its transcription. The process also requires audio hygiene steps—installing FFmpeg, converting to a consistent format (mono, 22.05 kHz), and removing silence and background artifacts. When pulling voice samples from YouTube instead, the same cleanliness rules apply, but the pipeline adds extra steps: downloading audio with yt-dlp, editing out music and interruptions (with Audacity), stripping silence using FFmpeg, and splitting long recordings into many short segments (no longer than ~15 seconds) so the training data matches Piper’s constraints.

Once the dataset is ready, the pipeline transcribes every clip using Whisper (Local Whisper). A Python script runs Whisper over the Wave directory and outputs a metadata CSV that aligns filenames to transcribed text—an essential requirement for training. After that, the training environment is set up by cloning the Piper repository and installing specific dependency versions. A major pain point appears with GPU compatibility: a CUDA 11.7 requirement caused issues on an NVIDIA 4090 setup, and the fix involved adjusting Piper’s requirements and installing a compatible PyTorch version. The tutorial emphasizes that these version mismatches are a common source of “headaches,” and that getting the dependency stack aligned is as important as the model itself.

Training is done by fine-tuning an existing Piper checkpoint rather than starting from scratch. The script runs a preprocessing step, then trains using the checkpoint downloaded from Hugging Face, with options for GPU acceleration and multi-GPU scaling. The author demonstrates three setups: a laptop with WSL and an NVIDIA 3080, a dual 4090 server for training a second voice, and an AWS EC2 instance with multiple GPUs (a g5.12xlarge) to train another voice in the cloud. Training can be interrupted and resumed by pointing to the latest checkpoint in the lightning logs.

After training completes, the final checkpoint is exported into Onyx format (plus a config JSON). The resulting model can be tested via Piper TTS on the command line, then integrated into a local voice assistant. The tutorial shows deploying the Onyx files to a Home Assistant server using a Samba share, configuring the Piper integration, and refreshing the Wyoming device so the new voice appears. Once wired into the assistant, the cloned voice can be used for conversation and scripted prompts.

Finally, the results highlight a practical reality: even with correct tooling, voice quality varies widely depending on the source clips. The author’s own YouTube-derived dataset produced a poor-sounding clone until the dataset was rebuilt using Piper Recording Studio with a larger number of utterances (hundreds). The “best” clone came from pristine, diverse, well-recorded samples and enough training iterations—turning voice cloning into a data-quality problem rather than a purely technical one.