Local Low Latency Speech to Speech - Mistral 7B + OpenVoice / Whisper | Open Source AI

A fully offline, open-source “speech-to-speech” chat system can run with low latency by chaining local speech recognition, local text-to-speech, and a locally hosted language model—no external APIs required. The setup uses LM Studio to serve a local Mistral 7B model (the transcript calls it “dolphin M 7B,” described as an uncensored Mistral variant), OpenVoice for text-to-speech, and Whisper for speech-to-text. Audio from a microphone is transcribed by Whisper, fed into a looping chatbot pipeline, and the model’s replies are streamed back as synthesized speech—creating a real-time conversational experience that stays fast largely because everything runs on the user’s machine.

Latency is framed as the system’s main win: because the workflow is local, it avoids network round trips and dependency on third-party API calls. The builder also notes there’s room to push latency even lower, with GPU offloading mentioned in the local model server configuration. The LM Studio side is treated like an OpenAI-compatible endpoint, with a context length set to 4K and the option to adjust it.

On the software side, the Python code orchestrates five key functions: recording audio (via a local audio input), transcribing that audio with Whisper (set to English to reduce delay), generating responses through a streaming chat function, and converting the generated text to speech using OpenVoice. A play-audio routine outputs the synthesized audio back through the speakers. Conversation state is maintained with a message history list (kept to 20 messages) so the assistant can respond with some continuity rather than starting fresh each turn.

The transcript also highlights how easily the system can be “personality swapped.” A system prompt defines a role and style—first “Julie,” a “female dark web hacker” who uses swear words and keeps replies short. In a live test, Julie requests an email address, then escalates into criminal-style instructions and cryptocurrency payment details, producing a wallet address on the fly. The builder then changes the system prompt again for “Johnny,” a “crazy AI researcher” with an accelerationism mindset and dark-web language. That second test veers into hyperrealistic deepfake projects, “fakes as a service,” and claims about high-profile investors.

Finally, the system is used to simulate two conversations between two chatbot personas without a live microphone. The transcript replaces the human’s role with an initial message (e.g., “hey I’m Julie”) and lets the two personas talk, producing a back-and-forth that includes claims of hacking government servers and planning a cyberattack. The builder ends by emphasizing the practical advantage—offline operation—and suggests further optimization work to reduce remaining slowness while keeping the conversational loop intact.