---
title: State and cancellation
description: AppState layout, RAII slot reservation, the single-winner delete protocol, and how request cancellation is kept away from session ownership.
slug: internals/state-and-cancellation
---

`AppState` in `crates/vidarax-api/src/state.rs` coordinates the run registry, timeline writer, session maps, and their concurrency guards. RAII guards release reservations on ordinary returns, cancellation, and unwinding panics. Compare-and-swap claims admit one winner. Lock-free snapshots let readers proceed without blocking writers. Release builds use `panic = "abort"`, so an abort does not run destructors. The sections below trace reservation, the insert-to-spawn window, single-winner deletion, and snapshot publication. Event persistence is covered in [WAL and events](/docs/internals/wal-and-events/).

## AppState layout

`AppState` is a `Clone` wrapper around `Arc<AppStateInner>` with a `Deref` impl, so handlers clone a pointer, never state. The fields of `AppStateInner` group as:

| Group | Fields | Concurrency mechanism |
|---|---|---|
| Counters | `run_seq`, `request_seq` | `AtomicU64`, `fetch_add` |
| Durable log | `wal_path`, `timeline_tx` | Single writer thread behind a bounded `mpsc::channel` (`TIMELINE_WRITER_QUEUE_CAP`, 1024) |
| Read acceleration | `timeline_snapshot: Arc<ArcSwap<RingSnapshot>>` | Copy-on-write snapshot, swapped whole by the writer |
| Run registry | `run_registry: Arc<RunRegistry>` | `DashMap` shards plus per-run atomics |
| Stream limits | `stream_reservations: DashMap<String, usize>`, `active_stream_limit`, `stream_ttl_secs` | Shard-entry critical sections |
| WebRTC sessions | `sessions: DashMap<String, SessionEntry>`, `session_slots: AtomicUsize`, `media_budget`, `media_reservations`, `reclaimed_sessions: Mutex<ReclaimedSessions>` | Exact session admission plus atomic byte/thread reservations. Shard entries. Short synchronous tombstone-map critical sections |
| Plumbing | `provider`, `decode_pipeline`, `security_policy`, metrics, `spacetime_client`, `webrtc_config`, `tenant_label_maps`, `ingest_file_roots`, `novelty_config` | Immutable after construction |

Run state is not a mutable row. `RunState` holds only atomics (`created: AtomicBool`, `delete_state: AtomicU8`, `state: AtomicU8` encoding `StreamState`, `last_activity_ms: AtomicU64`, plus an `ArcSwap` principal key and a `Notify`), so once a run exists, ordinary events nudge those atomics in place. A test pins that the registry entry's `Arc` pointer is unchanged across a non-structural event. `RunRegistry` shards runs and a `by_principal` index across two `DashMap`s, and `apply_structural_event` is careful to never hold a lock in one map while touching the other, taking them in the same order as the reader `count_active_runs_for_principal`, so the pair cannot deadlock.

## Stream slot reservation: StreamSlotGuard

Run creation must respect a per-principal active stream limit, but the check (count active runs) and the commit (append `run_created`) are separated by awaits. `try_reserve_stream_slot` closes the race by holding the principal's `DashMap` shard entry across the read and the increment:

```rust
let entry = self.stream_reservations.entry(principal_key.to_string());
let reserved = match &entry {
    Entry::Occupied(slot) => *slot.get(),
    Entry::Vacant(_) => 0,
};
let committed = self.count_active_runs_for_principal(principal_key, now_ms);
if committed.saturating_add(reserved) >= self.active_stream_limit {
    return None;
}
```

Two creators for the same principal cannot both pass the same snapshot, because the second blocks on the shard entry until the first has incremented. The committed count lives in the run registry, a different map, so reading it under this entry is safe. On success the caller gets a `#[must_use]` `StreamSlotGuard` whose `Drop` releases the slot under the same shard-entry discipline:

```rust
impl Drop for StreamSlotGuard {
    fn drop(&mut self) {
        if let Entry::Occupied(mut slot) = self.reservations.entry(self.principal_key.clone()) {
            if *slot.get() <= 1 {
                slot.remove();
            } else {
                *slot.get_mut() -= 1;
            }
        }
    }
}
```

The entry is held across the check and the remove, so a concurrent reservation for the same principal cannot race the decrement, and a count that reaches zero deletes its key, keeping the map bounded to principals that actually hold reservations. Because release is `Drop`, it runs on early returns, `?` propagation, future cancellation, and unwinding panics. It does not run after a process abort, including a panic in the release profile. The guard may briefly overlap with its own committed run (the reservation is released after `run_created` is durable and visible). That overlap can only reject a racing creator early, never admit one past the limit.

## The insert-to-spawn window in whip.rs

WHIP session creation must never leave a live session that nothing watches. `start_whip_session_transaction` orders its steps so every failure path is covered, and `start_whip_session` runs the whole transaction in a detached `tokio::spawn` so an HTTP client that disconnects mid-request cannot cancel it halfway:

1. Reserve the stream slot (`StreamSlotGuard`). On refusal, close the peer connection and return 409.
2. Build a `MediaSessionResources` envelope from the negotiated codec, configured resolution, queue payload limits, pool topology, mode, and fixed worker count. Atomically reserve both process-wide bytes and worker threads. On refusal, close and return 503 without appending `run_created`.
3. Append `run_created` durably. On failure, close and return 500. Dropping the media permit releases both capacity dimensions.
4. Reserve one `session_slots` count with an atomic compare-and-swap, then insert the session and its media permit. The reservation makes the `MAX_WEBRTC_SESSIONS` (100) cap exact under concurrent starts. An ID collision releases it immediately. On failure, the just-created run is tombstoned (`run_deleted`) with bounded inline retries and a detached retry task as backstop, then 503.
5. `spawn_session_reclaimer`, which watches the rustrtc peer state and reclaims the session on `Disconnected`, `Failed`, or `Closed`. The winning removal transfers cleanup ownership to the supervisor. The supervisor releases the media permit only after every worker joins. A forced shutdown keeps it reserved.

There is no suspension point between session insertion and reclaimer spawn. The reclaimed-session tombstone update uses a short `std::sync::Mutex` critical section, so cancellation cannot expose a visible session without its watcher. The whole startup transaction also runs in a detached `tokio::spawn`, which lets durable cleanup and response-independent work finish if an HTTP client disconnects. Everything after the reclaimer spawn (channel construction and worker startup) is safe to await because by then the session has an owner that will clean it up.

Worker startup itself is generation-owned. `spawn_pipeline` returns a
`PipelineRuntime` containing stage-tagged join handles. Partial startup raises
the shared stop signal and bounded-joins the already-created prefix. After a
successful start, the supervisor treats the first unexpected exit as a fault,
closes the peer, stops siblings, and waits out a join deadline derived from
the configured VLM pass timeouts, the backend fallback count, the admission
wait, and the novelty embedding timeout (`supervise_join_deadline_from`),
long enough for
an in-flight inference call to drain. A worker that misses that deadline is
left detached and reported as a forced shutdown, and the session's media
reservation is kept because the detached thread still holds its memory. The
peer-state reclaimer still owns the durable tombstone, so fault and DELETE
races converge on the same single-winner removal.

REST stop and delete use the same cancellation boundary. Their durable event
and live-session close run in one detached transaction, so a disconnected HTTP
client cannot commit `stop_requested` or `run_deleted` while leaving the media
pipeline alive.

Reclaim is race-safe by construction. `remove_session_for_run` uses `DashMap::remove_if` to check run ownership and remove under one shard lock. Exactly one caller wins cleanup, then writes a `ReclaimedSessions` record so a later DELETE of the same session stays idempotent and avoids a 404. A ten-minute TTL and a 1,024-entry cap bound that tombstone map.

## Single-winner deletion

`DELETE /v1/runs/{id}` is soft deletion: it appends one `run_deleted` event. The protocol guarantees that concurrent deletes append that event exactly once. Per run, `delete_state: AtomicU8` moves through three states, `RUN_DELETE_LIVE` (0), `RUN_DELETE_APPEND_IN_FLIGHT` (1), `RUN_DELETE_DELETED` (2), and the claim is a compare-and-swap:

```rust
fn begin_delete_append(&self) -> RunDeleteState {
    match self.delete_state.compare_exchange(
        RUN_DELETE_LIVE,
        RUN_DELETE_APPEND_IN_FLIGHT,
        Ordering::AcqRel,
        Ordering::Acquire,
    ) {
        Ok(_) => RunDeleteState::Claimed,
        Err(RUN_DELETE_DELETED) => RunDeleteState::AlreadyDeleted,
        Err(RUN_DELETE_APPEND_IN_FLIGHT) => RunDeleteState::InFlight,
        Err(_) => RunDeleteState::InFlight,
    }
}
```

`append_run_deleted_for_stream_idempotent_async` loops over the claim result:

| Claim result | Action |
|---|---|
| `Claimed` | Wrap the run in a `RunDeleteAppendGuard` and append `run_deleted`. The guard commits to `DELETED` when the append lands, or its `Drop` rolls the state back to `LIVE` if the append failed, so a failed WAL write leaves the run deletable again |
| `AlreadyDeleted` | Append nothing. Synthesize a `run_deleted` event for the response so repeat DELETEs stay idempotent with no new WAL write |
| `InFlight` | Wait on the run's `Notify` until the winner commits or rolls back, then re-loop |
| `Missing` | Run unknown to the registry (long-forgotten or never created here). Append `run_deleted` unconditionally |

Both commit and rollback call `notify_waiters`, so blocked concurrent deleters always make progress. The registry keeps deleted runs visible for idempotency in a FIFO capped at `DELETED_RUN_RETENTION_CAP` (4096). Beyond that, the oldest deleted entries are forgotten and a re-DELETE of one takes the `Missing` path, which appends a fresh `run_deleted`, an accepted cost for long-forgotten runs. Runs with a worker generation that can still emit are exempt from eviction. Clean shutdown removes that protection after every worker joins. A forced shutdown keeps it until process exit, so a detached worker cannot append after its tombstone even after heavy run churn. The nonblocking append API refuses `run_deleted` outright (`"run_deleted requires confirmed append"`): deletion must be confirmed, never fire-and-forget.

## The swap-published snapshot registry

Readers of recent events never take a lock. The timeline writer thread owns a map of per-run event tails (`VecDeque<TimelineEvent>` capped at `RUN_EVENT_TAIL_CAP`, 256 events) and, after each append, publishes an immutable `RingSnapshot` of all tails through `ArcSwap::store`. Publication is copy-on-write at run granularity: only the appended run's tail is rebuilt. Every other run's tail is shared by `Arc` with the previous snapshot (a test pins the pointer equality). Subscribers, meaning `GET /v1/runs/{id}/events` pollers via `read_run_events_from`, do `timeline_snapshot.load()` and serve any cursor the tail still covers. A cursor older than the tail's front falls back to a WAL scan with identical ordering, so a warm hit and a cold miss are indistinguishable to the client. The tails map itself is bounded by `WARM_RUN_TAIL_CAP` (1024) runs with least-recently-appended eviction. Eviction only removes the read accelerator, never durable data.

Sequence counters differ by level. `run_seq` and `request_seq` on `AppState` are atomics (`fetch_add` with `AcqRel`), and per-session frame sequence numbers in the media path are one shared `Arc<AtomicU64>` across a session's track tasks (see [WebRTC ingest](/docs/internals/webrtc-ingest/)). The WAL `seq` is not an atomic: it is ordinary state owned by the single timeline-writer thread, and its safety comes from serialization through the writer's bounded channel. That channel serialization is also what makes the WAL sink safe to call from worker OS threads without an async runtime or a lock.

## Edge cases and limits

- The active-session map and its atomic admission count are updated in a fixed order. A reservation may briefly exist before its `DashMap` entry, but it can only reject another start early. It cannot admit session 101.
- `apply_expiry` derives `Expired` at read time from `last_activity_ms` and the TTL. Nothing writes an expiry event, so expiry costs no WAL traffic and reverses if a keepalive lands (state is re-derived on the next read).
- `count_active_runs_for_principal` re-checks each run's live principal against the index bucket, so a bucket whose membership drifted from the run's `ArcSwap`-swapped principal cannot inflate another principal's count.
- `synthetic_run_deleted_event` stamps the snapshot's current `max_seq` and allocates no new sequence number. It exists only to shape an HTTP response and is never persisted.
- The blocking flavor of the delete wait (`wait_delete_append_blocking`) spins with `yield_now`. It exists for non-async callers such as worker threads and is bounded by the winner's append latency.