Add Discord voice channel support with per-speaker recognition #2

Merged

Khyretos merged 9 commits from claude/practical-darwin-vrjmvc into main

2026-09-24 23:54:24 +02:00

Khyretos commented

2026-09-24 23:49:20 +02:00

(Migrated from github.com)

Summary

This PR adds Discord voice channel as a new audio source, enabling real-time caption generation for each speaker separately in a Discord voice channel. A bot joins the channel and receives each person's audio as an independent stream, allowing the app to know exactly who said what without mixing or guessing.

Key Changes

New Discord Audio Source

discord_source.py: New module that spawns and manages a Node.js bridge process, handles frame-based protocol communication (JSON events and audio frames), and provides utilities for parsing Discord user IDs
discord_pipeline.py: Per-speaker recognition pipeline that manages individual VAD and Whisper workers for each Discord user, with automatic speaker lifecycle management and idle timeout handling
discord_bridge/bridge.js: Node.js bridge that logs in as a Discord bot, follows a configured user into voice channels, decrypts Discord's end-to-end voice encryption (DAVE), and streams each speaker's audio separately at 16 kHz mono int16

Supporting Infrastructure

audio_input.py: New streaming resampler for handling devices that don't support 16 kHz natively (common in Docker/ALSA environments), with linear interpolation and anti-aliasing
live_whisper.py: New low-latency Whisper request scheduler that merges queued segments instead of stacking them, prevents results from arriving out of order, and bounds lag when the server can't keep up
DISCORD.md: Complete setup guide for configuring the Discord bot and using the feature

Model Management Improvements

download_vosk_models.py: Rewritten to fetch the live Vosk model catalog dynamically instead of using a hardcoded list, with flexible filtering by language and model type. Fixes bugs where models were saved to wrong directory and catalog quickly became stale.

Quality & Reliability Improvements

recognizers.py: Added comprehensive hallucination detection for subtitle credits and outros that Whisper reproduces from training data (YouTube/TV subtitles)
vad.py: Enhanced with max segment length support to dispatch long utterances in pieces rather than waiting for pauses, configurable minimum speech duration, and improved trailing silence handling
subtitles.py: Added set_translation() method to support instant mode showing recognized text immediately while translation arrives asynchronously
session.py: Fixed session lifecycle issues including proper handling of Gradio's unload event (which fires on network blips, not just tab close), WebSocket reconnection, and popout ID persistence
settings_store.py: Added vertical alignment setting and VAD minimum speech duration configuration

Testing

tests/test_discord.py: Tests for bridge framing protocol, ID parsing, and real bridge process in fake-speaker mode
tests/test_live_whisper.py: Tests for request ordering, coalescing, and bounded lag behavior
tests/test_audio_input.py: Tests for streaming resampler across different sample rates
tests/test_download_vosk_models.py: Tests for dynamic catalog fetching and model selection
Updated existing tests for VAD, subtitles, recognizers, and session management

Dependencies & Deployment

requirements.txt: Added websockets for WebSocket support (required by recent Gradio versions)
Dockerfile: Multi-stage build with Node.js for the Discord bridge
discord_bridge/package.json: Node dependencies (@discordjs/voice, @snazzah/davey, etc.)
docker-compose.yml: Added optional DISCORD_TOKEN environment variable

Notable Implementation Details

Discord sends each speaker's audio as a separate stream, eliminating the need for speaker diarization or mixing
The bridge uses Node.js specifically because @discordjs/voice is the only library that currently supports decrypting Discord's DAVE end-to-end voice encryption for received audio
LiveWhisperWorker prevents out-of-order results and bounded lag by merging pending segments instead of queueing them
Audio resampling happens in the input callback to handle devices that don't support 16 kHz natively
VAD now supports max segment length to provide captions during long utterances rather than waiting for pauses
Session management now properly handles network reconnections and Gradio's lifecycle events

https://claude.ai/code/session_01Hr7di2bGC8qCojX7XXQbGx

## Summary This PR adds Discord voice channel as a new audio source, enabling real-time caption generation for each speaker separately in a Discord voice channel. A bot joins the channel and receives each person's audio as an independent stream, allowing the app to know exactly who said what without mixing or guessing. ## Key Changes ### New Discord Audio Source - **discord_source.py**: New module that spawns and manages a Node.js bridge process, handles frame-based protocol communication (JSON events and audio frames), and provides utilities for parsing Discord user IDs - **discord_pipeline.py**: Per-speaker recognition pipeline that manages individual VAD and Whisper workers for each Discord user, with automatic speaker lifecycle management and idle timeout handling - **discord_bridge/bridge.js**: Node.js bridge that logs in as a Discord bot, follows a configured user into voice channels, decrypts Discord's end-to-end voice encryption (DAVE), and streams each speaker's audio separately at 16 kHz mono int16 ### Supporting Infrastructure - **audio_input.py**: New streaming resampler for handling devices that don't support 16 kHz natively (common in Docker/ALSA environments), with linear interpolation and anti-aliasing - **live_whisper.py**: New low-latency Whisper request scheduler that merges queued segments instead of stacking them, prevents results from arriving out of order, and bounds lag when the server can't keep up - **DISCORD.md**: Complete setup guide for configuring the Discord bot and using the feature ### Model Management Improvements - **download_vosk_models.py**: Rewritten to fetch the live Vosk model catalog dynamically instead of using a hardcoded list, with flexible filtering by language and model type. Fixes bugs where models were saved to wrong directory and catalog quickly became stale. ### Quality & Reliability Improvements - **recognizers.py**: Added comprehensive hallucination detection for subtitle credits and outros that Whisper reproduces from training data (YouTube/TV subtitles) - **vad.py**: Enhanced with max segment length support to dispatch long utterances in pieces rather than waiting for pauses, configurable minimum speech duration, and improved trailing silence handling - **subtitles.py**: Added `set_translation()` method to support instant mode showing recognized text immediately while translation arrives asynchronously - **session.py**: Fixed session lifecycle issues including proper handling of Gradio's `unload` event (which fires on network blips, not just tab close), WebSocket reconnection, and popout ID persistence - **settings_store.py**: Added vertical alignment setting and VAD minimum speech duration configuration ### Testing - **tests/test_discord.py**: Tests for bridge framing protocol, ID parsing, and real bridge process in fake-speaker mode - **tests/test_live_whisper.py**: Tests for request ordering, coalescing, and bounded lag behavior - **tests/test_audio_input.py**: Tests for streaming resampler across different sample rates - **tests/test_download_vosk_models.py**: Tests for dynamic catalog fetching and model selection - Updated existing tests for VAD, subtitles, recognizers, and session management ### Dependencies & Deployment - **requirements.txt**: Added `websockets` for WebSocket support (required by recent Gradio versions) - **Dockerfile**: Multi-stage build with Node.js for the Discord bridge - **discord_bridge/package.json**: Node dependencies (@discordjs/voice, @snazzah/davey, etc.) - **docker-compose.yml**: Added optional `DISCORD_TOKEN` environment variable ## Notable Implementation Details - Discord sends each speaker's audio as a separate stream, eliminating the need for speaker diarization or mixing - The bridge uses Node.js specifically because @discordjs/voice is the only library that currently supports decrypting Discord's DAVE end-to-end voice encryption for received audio - LiveWhisperWorker prevents out-of-order results and bounded lag by merging pending segments instead of queueing them - Audio resampling happens in the input callback to handle devices that don't support 16 kHz natively - VAD now supports max segment length to provide captions during long utterances rather than waiting for pauses - Session management now properly handles network reconnections and Gradio's lifecycle events https://claude.ai/code/session_01Hr7di2bGC8qCojX7XXQbGx