Cache a copy of the conv tail, not a view of in_proj_qkv (semantic-router#5891)

#4

Fixes https://github.com/vllm-project/semantic-router/issues/5891.

The state pass cached each Gated DeltaNet layer's convolution tail as a slice of that layer's whole in_proj_qkv output, so the cache kept the full [1, conv_dim, S] projection on the GPU until the request ended. This caches a copy of the tail instead, in both paths:

  • StateOnce._delta: tail = mixed[:, :, -window:].clone()
  • FusedStateOnce._fused_delta: entry = (tail.clone(), final). The .contiguous() it replaces is a no-op on the batch-1 slice, so it returned the same view.

MODEL_MANIFEST.json updates files_sha256, runtime.files_sha256 and files_bytes for d3_runtime.py; identity is unchanged. The repeated K/V on the fused path is left for its own issue.

Memory, measured on d3-ultra BF16 through d3_server.py (unfused StateOnce path, one 96 GB RTX PRO 6000): the cached state drops from 1 MiB to 64 KiB per state token (16x). Unpatched, a 34,090-token state block with 19 questions ran out of memory; patched, state blocks up to the 131,072-token limit run, peaking at 35.0 GiB at 129,897 tokens. Probabilities and answers were bitwise identical on 21 requests.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment