--- title: Local audio perception description: Event tagging, selective speech recognition, and spoken feedback for recorded files and live WebRTC audio. slug: audio --- Recorded files and live WebRTC sessions can run local audio analysis. Recorded MP4 windows and live Opus windows both become mono 16 kHz PCM WAV. The API sends the WAV bytes to a bounded TCP sidecar and receives MessagePack metadata. WAV bytes never enter JSON or base64. ``` Recorded MP4 window Live WebRTC Opus window | | +----------> mono PCM WAV <------------+ | +---------+----------+ | | Silero VAD EfficientAT | sound labels speech intervals | | | Whisper | +---------+----------+ | timestamped WAL moments ``` Silero VAD decides whether speech work is needed. EfficientAT labels non-speech sounds on every window. The selected ASR model runs only when Silero finds a speech interval. A provider such as Gemini may inspect the synchronized MP4, but local transcripts remain the source of speech wording. Provider-only audio claims are discarded when the local pass has no matching observation. ## Model roles The default local stack uses [Silero VAD v6](https://github.com/snakers4/silero-vad), [EfficientAT](https://github.com/fschmid56/EfficientAT) `dymn10_as`, and [Whisper large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) for selective transcription. The other speech models are explicit deployment choices: | Engine | Use it for | Cost and scope | |---|---|---| | [Whisper large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | English speech where wording matters | Default selective ASR | | [SenseVoice](https://github.com/QwenAudio/SenseVoice) | Multilingual speech, language ID, and vocal emotion | Optional compact speech model | | [Moonshine Streaming Tiny](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny) | English screen recordings on small edge hardware | 34M parameters and English only | | [Qwen3-ASR 0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) | Multilingual speech, accents, singing, and noisy audio | Larger runtime with Transformers 5.13 or newer | | [LFM2.5-Audio 1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) | English speech input and optional spoken feedback | Optional because it is much larger than the event front end | LFM is never loaded by the default stack. `voice_feedback: true` asks that backend to synthesize a short summary. The resulting PCM WAV is stored in the content-addressed media directory and the event carries its hash and reference. ## Install and run The runtime manager places Python packages, model source, and downloaded weights in the user cache. It provisions Python 3.12 and installs `uv` into the same cache when neither is already available: ```bash python3 scripts/audio_runtime.py check --profile whisper --json python3 scripts/audio_runtime.py run --profile whisper ``` The first `run` syncs the checked-in `audio/uv.lock`, checks out the pinned EfficientAT revision, and starts the sidecar. Later runs skip setup when the profile, lock hash, Python version, and source revision still match. Package and model downloads are reused across checkouts. Use the setup-only compatibility command when a service manager will start the sidecar: ```bash scripts/setup_audio_models.sh whisper ``` Point the API at the sidecar: ```bash export VIDARAX_AUDIO_SIDECAR_ADDR=127.0.0.1:7790 ``` `core`, `sensevoice`, `moonshine`, `qwen`, `lfm`, and `all` are also valid profiles. Each profile gets its own environment so switching engines cannot silently remove another deployment's dependencies. Set `VIDARAX_CACHE_DIR` to move all runtime caches. `VIDARAX_AUDIO_VENV_DIR`, `VIDARAX_MODEL_CACHE_DIR`, and `VIDARAX_UV` remain explicit overrides. Use `--offline` only after the selected profile and pinned source have been cached. The server defaults to one active inference request and eight queued requests. Use `--max-in-flight` and `--max-queued` to set both bounds after measuring memory and latency on the target device. A full queue returns a typed `overloaded` failure. A request that exhausts its queue deadline returns `timeout`. ## Request Local audio requires `media.mode: "audio_video"`. ```json { "source_uri": "/srv/vidarax-media/gameplay.mp4", "model": "gemini-3.5-flash-lite", "semantic_inference": true, "media": { "mode": "audio_video", "window_ms": 20000, "persist_evidence": true }, "local_audio": { "profile": "gameplay", "speech_engine": "whisper", "min_confidence": 0.35, "max_events": 32, "voice_feedback": false } } ``` Set `semantic_inference: false` for a local-only pass. That mode emits audio moments without making a VLM call. The profile controls label normalization: - `gameplay` keeps explosions, gunfire, impacts, alarms, vehicles, engines, whooshes, hisses, scrapes, music, speech, typing, and similar game cues. - `screen_recording` favors speech, typing, clicks, notifications, and music. - `physical_world` and `general` retain normalized AudioSet labels. The sidecar limits each request to 4 MiB of WAV, 64 observations, and one minute of source time. A model error stays on its chunk. Decode continues and an MP4 window already retained remains available. Sound descriptions remain conservative. A provider moment classified as speech, sound effect, music, ambience, or machinery must overlap a compatible local observation, even when the provider labels its own moment as video-only. Unsupported moments are removed. Supported sound wording is replaced by the local label before the event is written. ## Triggers Audio observations can be replayed through the trigger VM: ```text trigger game-highlight version 1 when audio_event:explosion >= 0.75 and cognition_gate_score >= 0.60 cooldown 3000ms emit game_highlight capture clip 2000ms 4000ms notify webhook end ``` `audio_event: