Local privacy-first voice assistant

Problem

Cloud voice assistants expose recordings and prompts to remote services and stop working when connectivity is unavailable. This project investigates a private alternative that performs speech recognition, response generation, and voice synthesis entirely on the user’s machine while keeping the components modular enough to replace independently.

Approach

The assistant is organized as a localhost client-server pipeline. A microphone client records mono audio and posts a WAV file to FastAPI. The server transcribes it with faster-whisper, sends the text and recent conversation history to a Gemma model served by Ollama, synthesizes the answer through Piper, and returns a URL for the generated audio. Each stage is isolated in its own Python module.

Technical implementation

Audio is standardized at 16 kHz and written to request-specific temporary files. Session histories are kept in memory and truncated to the twelve most recent messages to bound prompt growth. Configuration centralizes the Whisper model size, Ollama endpoint and model, Piper executable, ONNX voice, and system prompt. The API exposes health, pipeline, and generated-audio endpoints, while the client supports both live recording and existing WAV files.

transcript = transcribe_wav(input_wav)["text"].strip()
answer = ask_ollama(transcript, history=history)
synthesize_to_wav(answer, output_wav)

return {"transcript": transcript, "answer": answer, "audio_url": audio_url}