Instructions to use vllm-sr/d3-ultra with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vllm-sr/d3-ultra with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-classification", model="vllm-sr/d3-ultra", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("vllm-sr/d3-ultra", trust_remote_code=True) model = AutoModel.from_pretrained("vllm-sr/d3-ultra", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Cache a copy of the conv tail, not a view of in_proj_qkv (semantic-router#5891)
Fixes https://github.com/vllm-project/semantic-router/issues/5891.
The state pass cached each Gated DeltaNet layer's convolution tail as a slice of that layer's whole in_proj_qkv output, so the cache kept the full [1, conv_dim, S] projection on the GPU until the request ended. This caches a copy of the tail instead, in both paths:
StateOnce._delta:tail = mixed[:, :, -window:].clone()FusedStateOnce._fused_delta:entry = (tail.clone(), final). The.contiguous()it replaces is a no-op on the batch-1 slice, so it returned the same view.
MODEL_MANIFEST.json updates files_sha256, runtime.files_sha256 and files_bytes for d3_runtime.py; identity is unchanged. The repeated K/V on the fused path is left for its own issue.
Memory, measured on d3-ultra BF16 through d3_server.py (unfused StateOnce path, one 96 GB RTX PRO 6000): the cached state drops from 1 MiB to 64 KiB per state token (16x). Unpatched, a 34,090-token state block with 19 questions ran out of memory; patched, state blocks up to the 131,072-token limit run, peaking at 35.0 GiB at 129,897 tokens. Probabilities and answers were bitwise identical on 21 requests.