Quickstart

Build the API server, analyze a local video, and read the resulting event snapshot.

Prerequisites

• A Rust toolchain.
• ffmpeg and ffprobe on PATH (or paths set through VIDARAX_FFMPEG_PATH and VIDARAX_FFPROBE_PATH).
• jq, used below to pull the run ID out of JSON responses.
• An OpenAI-compatible VLM backend, such as vLLM, SGLang, or MLX, reachable at a URL you control. Gemini is configured through the TOML backend file. Without a backend, inference routes fail.
• A video file to analyze. The examples below assume demo.mp4 in a directory the server is allowed to read.

Run the server

The server only reads local files from directories listed in VIDARAX_INGEST_FILE_ROOTS (the list defaults to empty, so local paths are rejected unless you set it or upload the file through POST /v1/upload). Create a root and put a video in it:

mkdir -p /srv/vidarax-media
cp demo.mp4 /srv/vidarax-media/

git clone https://github.com/Cosmin-B/vidarax.git && cd vidarax
cargo build --release -p vidarax-api
VIDARAX_API_KEYS=dev-key \
VIDARAX_VLLM_BASE_URL=http://localhost:8000 \
VIDARAX_INGEST_FILE_ROOTS=/srv/vidarax-media \
cargo run --release -p vidarax-api

The server binds 127.0.0.1:8080 by default. Check readiness:

curl -fsS http://127.0.0.1:8080/v1/health

API keys are on by default

VIDARAX_REQUIRE_API_KEY defaults to true, so VIDARAX_API_KEYS must contain at least one key and every request except /v1/health needs an x-api-key header. For local open-mode development you can disable checks explicitly with VIDARAX_REQUIRE_API_KEY=false and VIDARAX_METRICS_REQUIRE_API_KEY=false.

To run the Vue 3 frontend in a separate terminal:

cd ui && npm install && npm run dev

Analyze a file

The TypeScript SDK wraps the whole flow. It lives in packages/vidarax-sdk/ in the repository. Build it and link it into your project:

cd packages/vidarax-sdk && npm install && npm run build && npm link
cd /path/to/your-project && npm link vidarax

import { Vidarax } from 'vidarax'

const v = new Vidarax('http://localhost:8080', { apiKey: 'dev-key' })

const run = await v.analyze('/srv/vidarax-media/demo.mp4')

for (const event of await v.getEvents(run.runId)) {
  console.log(event.kind, event.payload)
}

The SDK also supports WHIP/WebRTC, batch inference, structured JSON output via output_schema, interactions, and snapshot reads of events and markers. See Events and SDK for the method list.

Use the CLI

The CLI covers run lifecycle, events, markers, search, file analysis, feedback, trigger compilation, edge updates, models, health, and diagnostics.

cargo build --release -p vidarax-cli
export VIDARAX_API_URL=http://127.0.0.1:8080
export VIDARAX_API_KEY=dev-key

target/release/vidarax doctor \
  --url "$VIDARAX_API_URL" \
  --api-key "$VIDARAX_API_KEY"

target/release/vidarax analyze /srv/vidarax-media/demo.mp4
target/release/vidarax runs list
target/release/vidarax events <run_id>

The commands above use the OpenAI-compatible backend configured when the server started.

Optional synchronized audio and video

Native audio-video analysis requires a configured Gemini backend. Uncomment the Gemini entry in vidarax.toml, set GEMINI_API_KEY, and restart the server without VIDARAX_VLLM_BASE_URL or VIDARAX_SGLANG_BASE_URL. Either explicit URL causes the server to skip the TOML backend list.

For local sound events and selective speech, start the audio process in another terminal and restart the API with VIDARAX_AUDIO_SIDECAR_ADDR=127.0.0.1:7790:

python3 scripts/audio_runtime.py run --profile whisper

Then run:

target/release/vidarax analyze /srv/vidarax-media/demo.mp4 \
  --media audio-video \
  --model gemini-3.5-flash-lite \
  --media-window-ms 20000 \
  --media-resolution low \
  --local-audio \
  --audio-profile screen-recording \
  --speech-engine whisper

This sends raw MP4 windows through Gemini File API and passes a mono PCM WAV to the local audio sidecar. It prints multimodal_moment entries with source timestamps, sound and video descriptions, optional speech intent, and their relationship. Retained MP4 windows and optional spoken feedback stay in the binary media store. The first audio process start provisions the locked Python environment automatically.

See Local audio perception for setup and local-only analysis.

Global connection values resolve in this order: command-line flags, VIDARAX_API_URL / VIDARAX_API_KEY / VIDARAX_TENANT_ID, the file selected by VIDARAX_CLI_CONFIG, then the default URL http://127.0.0.1:8080. Without VIDARAX_CLI_CONFIG, the CLI reads ~/.config/vidarax/config when it exists. That file uses KEY=VALUE lines for api_url, api_key, and tenant_id.

Run vidarax config show to inspect the resolved source of each value. Add --json for machine-readable command output.

The same flow over plain HTTP: create a run and capture its ID, then run semantic reasoning against a source the server can read.

RUN_ID=$(curl -s -X POST http://127.0.0.1:8080/v1/runs \
  -H 'x-api-key: dev-key' \
  -H 'content-type: application/json' \
  -d '{ "mode": "balanced", "model": "Qwen/Qwen3-VL-4B-Instruct" }' \
  | jq -r .run_id)
echo "$RUN_ID"

curl -s -X POST http://127.0.0.1:8080/v1/runs/$RUN_ID/reason \
  -H 'x-api-key: dev-key' \
  -H 'content-type: application/json' \
  -d '{
    "source_uri": "/srv/vidarax-media/demo.mp4",
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "semantic_inference": true,
    "semantic_frames_per_chunk": 2,
    "chunk_size": 30,
    "fixed_fps": 5.0,
    "sampling_policy": "fixed"
  }'

mode must be one of balanced, detailed, efficiency, or custom, and model must be in the supported model contract (see GET /v1/models). Local file paths are only readable when they sit under a directory listed in VIDARAX_INGEST_FILE_ROOTS, or when the file was uploaded through POST /v1/upload (the upload response returns the server-side file_path to use as source_uri).

Read events

Every run writes sequence-numbered events to its timeline. This endpoint returns the current snapshot. Use GET /v1/runs/:id/events/stream or the SDK's subscribeEvents() for replayable incremental delivery.

curl -s http://127.0.0.1:8080/v1/runs/$RUN_ID/events -H 'x-api-key: dev-key'

Markers (frame-range annotations derived from the analysis pass) have their own endpoint with filters:

curl -s "http://127.0.0.1:8080/v1/runs/$RUN_ID/markers?event_type=scene_cut" -H 'x-api-key: dev-key'

See Events and SDK for the event shapes and kinds, and API reference for the route contracts and selected configuration variables.