Repository navigation
Releases
cortiq 0.9.3
Added
- Skills on the CUDA hybrid arm, every way the host applies them, on the card:
- a pinned skill (
run/ppl --skill) was already read by the plans through the skill's tensors; now measured
against the float host: Qwen3.5-0.8B q4tp with a 96-tensor skill (BotPsy-0.8B FFN and full attention),
64 greedy tokens on four prompts, top-1 64/64 and the same text, logits within 0.011; perplexity the
CPU's to three digits with and without the skill (38.689 / 30.868); - a soft blend (
--blend): its f32 working tensors go to the card once as q8_2f int8 tiles, encoded on the
host's cores in parallel (before: every graph declined and the blend ran per op, 9.8 tok/s on that 0.8B;
now 346 tok/s, the same text as the CPU's blend); - per-token routing (
--route-dynamic): the token and batch graphs keep the router layer's row (the layer
taps the DFlash drafter uses), so φ moves on the card; before, the graph ran the router's layer and φ never
changed (the router never left the backbone, silently), and MTP drafts took the router away altogether.
Same E values and switch points as the CPU, the prompt read batched (first token 0.3 s against the CPU's
6.9 s on a 90-token prompt). - the skill's speed is the backbone's: A40, medians of three, 128 decode tokens without drafts and a
4096-token prompt — Qwen3.5-0.8B 429 / 415 tok/s and 20 895 / 20 150 tok/s without / with the skill,
Qwen3.5-4B (a 128-tensor skill from NeoHorse-1-4B) 159 / 157 and 5 950 / 5 950; the 4B's greedy text and
logits match the float host with and without it (top-1 64/64, within 0.043).
- a pinned skill (
cortiq skill bakeon CUDA (CMF_GPU=cuda): a Phase-A step of a Qwen3.5-class model runs whole on the card
— embeddings, every layer (frozen GDN and gated full attention, the masked FFN, both norms), the loss over
the scored rows on the full vocabulary and the FFN path's backward down to every layer's mask gradient —
with the ids up and the per-row losses and mask gradients down. The GDN recurrence (conv, delta rule,
BPTT), the exact attention core, the FFN chain and the LM-head cross-entropy have CUDA kernels of their own,
checked against the f64 host operators (relative 1e-7 to 4e-7); the layer-by-layer path keeps them for
Phase B and the through-attention backward. Qwen3.5-0.8B q4tp on an A40, 64 records of up to 128 tokens a
step: 0.2 s a step against 30–41 s on 0.9.2 (Vulkan chains with the GDN layers in f64 on a 7-core host), a
held-out pass of 400 records in seconds instead of 5 minutes, the same masked PPL and token accuracy as the
host path (1.614 / 84.09 %).CMF_BAKE_RESIDENT=0keeps the layer-by-layer path.cortiq skill apply --overlay: a standalone skill file goes beside the base asskill.<id>.*instead of
replacing its tensors, so one file serves the backbone and, under--skill <id>, the specialist. Masks the
skill's record does not link stay out (a mask in the catalog is what a file without its default task falls
back to: the base itself would serve masked, on the host's sparse FFN arm).cortiq bench --skill <id>andcortiq serve --skill <id>(every slot runs that skill).
Fixed
- Dynamic skill routing: a switch mid-generation cleared the sequence and the next token was decoded against an
empty cache (fluent nonsense after the switch, on every backend). The context so far is now read again under
the new weights (the batch graph where generation takes it), andppl --route-dynamicdoes the same. skill bakewithCMF_BAKE_SKIP_BASELINE=1compared the result with the skipped (NaN) baseline and printed
"did not beat baseline"; it now says the baseline was skipped.- Server: the generic
<think>dialect (LFM3, Qwen and every other family that opens its own block) now
releases thinking asreasoning_content— streaming deltas and the non-streaming message field — in the
shape DeepSeek made the convention for OpenAI-compatible servers. Previously only DeepSeek V4.1 and
MiniMax M3 had protocol-level reasoning; the other families streamed<think>...</think>inline as
user-visible text whenever thinking was on. DeepSeek V4.1's stream, which also carried its reasoning
inline, now matches its non-streaming reply. Replies without thinking are byte-identical to before, and
enable_thinking=falsekeeps stripping the prefilled empty block exactly as it did. A client that reads
the inline block can keep it:"reasoning_format": "none"(llama.cpp's spelling) leaves the thinking in
content, streamed or not.
cortiq 0.9.2
Changed
- CUDA hybrid: the RAM tier and the host arenas take VMM host memory (
cuMemCreateat a host location, copies
by address) when a container's limit counts page-locked memory and not that (a 256 MB probe of each); the tier
then holds every expert, as Vulkan's system-memory banks do, laid out at load. On an A40 (46 GB container),
decode after a 512-token prompt and that prompt's first token: Ornith-122B q4tp within 12 / 16 / 24 / 32 GB
and on the whole card 9.0–10.5 / 10.9 / 40.1 / 43.5 / 44.6 tokens/s and 1.8–3.4 s (0.9.0: 6.8 / 7.1 / 28.7 / –
/ 37.3 and about 10 s), perplexity the CPU's (3.049); MiMo-V2.6-Flash on the whole card 24.0 and 5.3 s (0.9.0:
11.3); MiniMax-M3.1-Flash on the whole card 10.0 and 6.9 s (0.9.0: 1.0 and 363 s); both answer "2+2" as the
CPU does.CMF_HYB_PINNED_PROBE=0keeps the page-locked tier. - CUDA hybrid: the card's reserve beside a plan is an eighth of the card, 1–3 GiB (was 3 GiB;
CMF_HYB_RESERVE_MBsets it). On an A40, runs interleaved with 3 GiB: Apex-35B q4tp within 8 GB 75–82
tokens/s against 35.0 (2840 slots against 1627), 83.7 after a 4096-token prompt; Ornith-122B q4tp within
12 GB 9.0 against 8.5. - CUDA hybrid: a prompt chunk on the small path hands the rest of its rows to the prompt path, its streamed
layers staged, once two of its 16-row pieces ran and the stream's count says most picks were built. On an A40
within 24 GB with page-locked memory, the first token after a 512-token prompt: MiniMax-M3.1-Flash 64 s (the
small path: 398 s), MiMo-V2.6-Flash 32 s (297 s). - GLM-5.3 on CUDA: its prompt takes the prompt path up to the DSA top-k (the grouped experts' mixture into the
mHC merge, the indexer's caches written on the way); a chunk where DSA selects still goes the small path. On
an A40, decode after a 512-token prompt and its first token: GLM-5.3-Flash q2tp within 24 GB 8.8–9.7
tokens/s and 5.4–6.0 s (0.9.0: 6.4 and 100 s), on the whole card 22.7 and 5.6 s (0.9.0: 17.1 and 38 s);
perplexity the CPU's (2.715; with the selection cut to 64 positions 2.730), "2+2" answered as the CPU does.
Fixed
- Vulkan on a phone driver: every process that answered once compiled the batch graph's kernels again. They are
built at first use, afterinitwrote the pipeline cache, and the only later save was at the start of a second
generation, whichcortiq run -p, a bench or an app closed after one reply never reaches. A generation's end
now saves the cache when the driver's blob differs from the file (a CPU generation does not bring the device
up for it). On a Snapdragon 778G (Adreno 642L), lfm2.5-230m q4tp with the whole-token graph forced: the first
token of a one-shot run in 6.9 s instead of 41.4 s once the cache holds them. - Vulkan: the token graph compiled GLM's DSA kernels for every model, with no pipeline cache — it asked for the
kernels before asking whether any layer has DSA. The layers are asked first and the DSA pipelines take the
cache: on the same phone decode over 40 tokens at 10.4 tokens/s from the first run instead of 2.4. - GLM-5.3 on CUDA: a prompt past 1536 tokens on a card the expert slots filled fell to the host (its DSA
layers' K/V mirrors grew past the reserve; a 3072-token prompt on the whole A40 ran past an hour). A DSA
layer's mirror stops at top-k rows: it is now sized to that at once and the plan's reserve holds it beside the
slots (1.7 GB for GLM-5.3-Flash). That prompt's first token on the whole card 28.8 s, decode 20.3 tokens/s;
within 24 GB 86 s and 9.6. - The container-limit probes (CUDA page-locked and VMM host memory, Vulkan GPU-visible system memory) read the
container's usage less its page cache: at the limit a charge evicts as much page cache, the usage did not
grow, and the CUDA RAM tier took page-locked memory past the limit (Ornith-122B on the whole A40: killed). - CUDA: the pair micro-benchmark is skipped while routed experts stream through card slots, as on Vulkan: the
per-op path has no route for them.
cortiq 0.9.1
Fixed
- Vulkan, latent-expert
qwen4_expfiles (Edge0-145B) with part of the expert banks in system memory: a prompt
chunk went out as one submission for every layer, long enough for the driver to give up on the device
(within 32 GB on an A40 the device was lost after ~17 s in the chunk past 2048 positions; amdgpu resets its
ring after 10 s). Such a chunk now goes out four layers a submission (CMF_QWEN_CHAINstill overrides), and
the 4.7K-token prompt runs through to decode. - After a lost device the host fallback built its buffers on the dead device, panicked inside wgpu and hung the
process; a failed frame readback now turns the GPU path off for the rest of the run, which goes on on the CPU.
cortiq 0.9.0
Added
- MoE placement for any card size, CUDA hybrid arm: the dense matrices and the attention go to the card first;
routed experts past it stream through card slots (least recently used giving way) when they are more than an
eighth of the experts or past the host's room; a small spill stays in mapped host memory read over the bus,
and a dense part past a small card goes there too. The slots are built from a page-locked RAM tier of raw
experts (DMA, no page faults; a call's missing experts copied in on several host threads, their pages asked
for at once withMADV_WILLNEED), which keeps each expert's card layout after its first build, so the next
one is a single copy; the tier holds every expert when they fit three quarters of the host's room, filled and
laid out at load, else half of it. A prompt chunk stages a streamed layer whole when the streamed experts are
within twice the host's room or once the small path's chunks built most of their picks; each layer then keeps
the experts of the chunk's last rows for the tokens after it. In a container whose limit page-locked memory
falls outside (a 256 MB probe against its usage), the host's room is its available memory.
CMF_GPU_VRAM_MBcaps every placement decision, so a smaller card's runs on a bigger one as it would there.
On an A40 (46 GB container, which counts page-locked memory), decode after a 512-token prompt and that
prompt's first token: Ornith-1.5-122B q4tp within 12 / 16 / 24 GB and on the whole card 6.8 / 7.1 / 28.7 /
37.3 tokens/s and 10 s (0.8.99: 3.4 / 4.4 / 31.2 / 15.4), perplexity the CPU's (3.049); Apex-35B q4tp within
12 GB 88.7 tokens/s and 0.85 s (the tier filled as it goes: 75.4 and 4.65 s), on the whole card 107.8 and
0.36 s, perplexity 4.186 against the CPU's 4.188; MiMo-V2.6-Flash (164 GB) and MiniMax-M3.1-Flash (157 GB) on
the whole card 11.3 and 1.0 (their prompts on the small path: 46 and 363 s), their greedy texts the CPU's.
Knobs:CMF_HYB_STREAM=0,CMF_HYB_STREAM_SPILL=<n>,CMF_HYB_STREAM_STAGE=0|1,CMF_HYB_RAM_TIER=0,
CMF_HYB_TIER_PACK=0,CMF_HYB_TIER_PREPACK=0,CMF_HYB_HOST_TIER=1,CMF_HYB_RAM_FILL_THREADS=<n>,
CMF_HYB_PINNED_PROBE=0. - Vulkan: routed experts past the card live in GPU-visible system memory and the token graph keeps going instead
of ending at a device prefix; the dense part of every layer and the head are placed first. When more than an
eighth of the experts are past the card, the banks go to system memory and the card's room left after the
layers that stay whole becomes LRU slots over them, kept by the token graph itself: after the select one
workgroup maps the picks to slots and a copy kernel brings the misses in (an indirect dispatch: a token whose
picks all hit copies nothing); nothing waits for the host. The prompt runs on the batch graph with the same
placement (it refused any model past the card and went a position at a time), each system-memory bank copied
whole to a staging bank on the card for a chunk of 512 positions. The banks' room is the host's RAM; in a
container whose limit the driver's system memory falls outside (a 256 MB probe against its usage), the host's
available memory. On an A40 (46 GB container), decode after a 512-token prompt and that prompt's first token:
Ornith-122B q4tp on the whole card 31.9 tokens/s and 5.4 s (0.8.99: 4.3 tokens/s), within 12 / 16 / 24 GB
10.6 / 12.6 / 30.6 and 5.5 s (the banks within the container's room: 7.4 / 7.8 / 8.7);
MiMo-V2.6-Flash on the whole card 13.2 and 12.1 s (its own MoE bank: 1.9 and 189 s), within 24 GB 6.5;
MiniMax-M3.1-Flash on the whole card 6.7 and 23.7 s, within 24 GB 4.8 and 24.7 s (CPU 0.65). Perplexity is the
CPU's (Ornith q4tp 3.050 against 3.049); MiMo and MiniMax answer as the CPU does. Knobs:
CMF_WGPU_EXPERT_CACHE=0,CMF_WGPU_EXPERT_SPILL=<n>,CMF_WGPU_EXPERT_STAGE=0,CMF_SYS_EXPERTS=0,
CMF_WGPU_EXPERT_OUT=fit(the container's room). - Vulkan: routed experts past the card AND past the host's memory stream through card slots: pools of slots
(a group of layers each, every buffer within one binding) over a RAM tier of raw experts in GPU-visible
system memory, the least recently used giving way in both. The token graph ends its submission after a
streamed layer's select, the host brings the picks' missing experts into slots (a DMA copy from the tier, or
the file into the tier first, on several threads) and the graph goes on; a prompt chunk stages each streamed
layer whole and leaves the experts of its last rows in the slots, so decode starts warm. Such a file used to
fall to the host for the layers past the room. Its perplexity is the CPU's (Ornith q4tp 3.050).
CMF_WGPU_EXPERT_STREAM=0,CMF_WGPU_EXPERT_TIER=0,CMF_WGPU_EXPERT_WARM=0,
CMF_WGPU_EXPERT_FILL_THREADS=<n>,CMF_STREAM_STATS=1. - Vulkan: a mixed q2tp-v2 file's q2tp gate/up rides the grouped prompt path (
CMF_MOEG_Q2=0: the per-row
loop): Ornith-122B q2tp-v2 on an A40, the first token of a 512-token prompt in 3.5 s instead of 7.0 on the
whole card and 5.2 instead of 8.7 within 32 GB (decode 25-29 tokens/s), perplexity 3.180 on both paths (CPU
3.179). - Vulkan: the MoE select takes 257-512 experts (two a thread of the workgroup, ties to the lower one), and
routed experts with a SwiGLU clamp ride the expert kernels' limit word instead of being refused. - CUDA hybrid arm: MLA with a low-rank query (q_a, its norm, q_b), SwiGLU-OAI (MiniMax M3: α 1.702, β 1, clamp 7)
in dense and routed FFNs, and a V narrower than the head (MiMo: 128 of 192), widened with zeros and cut back
before the output projection. wgpu's token and batch graphs take the low-rank query too. - Vulkan: LFM's short convolutions on the batch graph (a thread a channel walks the chunk's rows over the conv
ring), with their sandwich norms: an LFM3 prompt is read in chunks instead of a position at a time. LFM3-1.7B
q4tp on an A40 reads a 4096-token prompt at 2013 tokens/s instead of 467 and decodes after it at 154 instead
of 103; the greedy text over a long prompt is the CPU's.
Changed
- OmniVoice: the codec's transposed convolutions run as one GEMM and an overlap-add (on the card when one is up)
and its Snake activations over the thread pool: decoding 4.5 s of speech on an NVIDIA A40 takes 0.82 s instead
of 1.85 on CUDA and 1.19 s instead of 2.04 on Vulkan (15.6 s of speech: 3.2 s instead of 6.5). The samples
match the previous path within 3·10⁻⁵ on the CPU. The card's commands on CUDA: 4.5 s of speech in 3.4 s instead
of 4.4, a 34 s text in 22 s instead of 32; Whisper still reads every sample as its text.
Fixed
- Vulkan, Edge0-145B and other latent-expert
qwen4_expfiles: loading the expert banks held every layer's bytes
twice, once in the queue's staging and once on the card, until the first frame. On a Radeon 8060S (121 GiB of
unified memory) the load pushed the GTT to 90.6 GB and filled swap before settling at 59 GB. The upload now
hands its writes to the GPU every 192 MiB, as the expert pool's prefill already did. - Vulkan on an integrated GPU under Linux: with no
CMF_GPU_VRAM_MBthe weight budget was unlimited, so a model
larger than the machine was all planned onto the "card" and only the allocator's refusals placed the rest
(a Ryzen AI Max+ 395 needed the budget set by hand). It is now three quarters of what the GPU can address: its
largest device-local heap (an APU's GTT) within the host's memory, or the host's memory when the driver reports
only a small carve-out.CMF_GPU_VRAM_MBstill overrides it; Android and macOS keep their own rules. - GLM-5.3-Flash (
glm5_next) files ran as a plain transformer and answered with noise: the release had no
runtime for them. Theglm5_nextstack — four-stream mHC residuals, KDA recurrent mixers and NoPE MLA with
the DSA indexer — is in: the published q2tp file answers the card's check ("What is 2+2?" → "4"). DSA's
attention runs absorbed (kv_b through the query and the output instead of over every selected position; the
CPU text is the expanded form's to the byte). On CUDA the whole stack runs on the card's graphs: the KDA and
NoPE MLA mixers, DeepSeek-V4's four-stream mHC (its Sinkhorn on the card), the dense and routed FFNs with
their SwiGLU clamp, the experts streamed through card slots; the prompt on the batch graph. Past 2048
positions, where DSA stops attending to every position, the indexer runs on the card too: its pooled keys,
the pool scores, the top-k selection and the attention over the selected latents (CMF_GLM_GRAPH=0keeps
the host stack). On an A40 pod with 46 GB of memory for the 116 GB file, decode after a 512-token prompt:
17.7 tokens/s (the host stack 1.9, the CPU 1.2), the prompt at 14 tokens/s; within 16 / 24 GB 4.1 / 5.7
(1.1 / 1.3); after a 3072-token prompt 19.7. The perplexity is the CPU's (2.715 against 2.715). On Vulkan the
same stack runs on the wgpu graphs (four-stream mHC kernels, the 288-expert select, DSA past 2048 positions on
the token graph, the experts in system-memory banks under card slots): on the A40 pod 15.5 tokens/s and the
first token of a 512-token prompt in 15 s, within 24 GB 9.8 tokens/s, perplexity 2.715; with the selection cut
to 64 positions, so that DSA selects within a short text, 2.728 as on the CPU (CUDA 2.730). - Vulkan: a model whose attention groups more than eight query heads on a K/V head (MiniMax-M3.1-Flash) kept an
f16 K/V mirror no attention kernel takes, so both graphs declined it and its experts ran on the host: within
24 GB on an A40 0.17 tokens/s and its first token in 1508 s, now 4.8 and 24.7 s. The mirror is f16 only for
a geometry the flash-decode kernels serve. cortiq benchran its untimed pair micro-benchmark per op with the routed experts on the card, which a model
whose experts live past the card ran out of memory on (MiniMax-M3.1-Flash on a whole A40 under Vulkan); it is
...
cortiq 0.8.99
Added
- OmniVoice (k2-fsa), zero-shot text-to-speech in 600+ languages, native in the engine:
cortiq convertpacks
the checkpoint and its Higgs audio tokenizer v2 into one file (the Qwen3 stack in q8_2f, the audio embeddings
and heads in f16), andcortiq tts FILE TEXTspeaks it —--lang,--instruct(voice design: "female, low
pitch, british accent", "男,四川话"),--ref-audiowith--ref-text(voice cloning),--steps,--duration,
--speed; text over 30 s of speech goes in ~15 s pieces joined by short fades. The 32 unmasking passes run
both ways over the row through d1's set forward (the conditional and the unconditional rows as two groups of
one forward), the head as one device GEMM; the codec's decoder (RVQ, DAC) and encoder (HuBERT, the semantic
and acoustic encoders, nearest codes) and the release's duration estimate and post-processing (pydub's silence
trimming) are ported term for term. Against the float32 release the prompt and the post-processed length are
identical, the first pass's log-probabilities within 0.011 on average, and Whisper transcribes our audio as the
release's. On an NVIDIA A40, 4.5 s of speech take 4.4 s on CUDA and 10.6 s on Vulkan; a MacBook Air M4
(Metal) 14.3 s. - CUDA hybrid arm: Ripple's value residual (V reads
lerp(e0, normed, blend)) and clamped SwiGLU, on the token
and the prompt paths. Ripple-65B q4tp on an A40 decodes at 13.3 tokens/s instead of 0.6–0.9 (Vulkan 13.5) and
reads a 4096-token prompt at 428 tokens/s instead of 9.4 (Vulkan 172); the greedy text is Vulkan's. - CUDA hybrid arm: weights past the card go to mapped host memory, routed experts first, so a MoE larger than
the card runs on it: a plan's arena takes the card's free memory less a reserve (CMF_HYB_RESERVE_MB, 3 GiB,
plus the head when the plan has none), the rest is pinned and read over the bus (CMF_HYB_HOST_EXPERTS=0
turns it off; a spill past the container's memory declines instead). On a 46 GB A40: Ornith-1.5-122B q2tp-v2
(48 GB) decodes at 38.5–38.9 tokens/s instead of 1.5–2.4 (Vulkan 14.4) and reads a 4096-token prompt at 774
tokens/s instead of 4.1; q4tp (64 GB, 23 GB of it pinned) 15.4–15.9 instead of 1.9–4.3 (Vulkan 4.5).
Fixed
- Windows: models larger than the card on Vulkan ran entirely on the CPU (#151). The expert banks a card has no
room for live in host-visible system memory that the kernels read over the bus, and that allocation was built
for Linux only; on Windows the device path refused ("latent experts do not fit the card and system memory cannot
hold them for it") and Edge0-145B on a 32 GB Intel Arc Pro B70 stayed on the CPU. The host-memory buffers now
build on Windows too;CMF_SYSMEM=0turns them off. cortiq servestatus and dashboard: the execution mode always read "CPU Only" and the GPU memory 0.0 GB off
macOS, whatever ran. They now name the backend and the card (Gpu { backend, device, budget_mb }) and show the
memory in use on it.- Qwen3.8-Flash-Next / Edge0 on CUDA in a container with a cgroup v1 memory limit: the free-memory estimate read
only cgroup v2, took the host's 426 GB for the container's 46.6 GiB, pinned the n-gram table past the limit and
the OOM killer ended the run. It now reads v1 (memory.limit_in_bytesless RSS and unevictable pages) too:
Edge0-120B on a 46 GB A40 decodes at 52–56 tokens/s (106 after a 4096-token prompt read at 2041 tokens/s),
Edge0-145B at 18–20 (36 after the prompt at 1208). - CUDA: the Gemma-4 / Ripple prompt path took 128-row chunks, tuned for wgpu, and the CUDA batch graph expands every
weight once a chunk; it now takes the CUDA chunk (CMF_CUDA_BATCH, 1024). A 4096-token prompt on an A40:
humanizer-12B (Gemma-4) 778 → 2095 tokens/s with the same greedy text, Ripple-65B 141 → 428. - Vulkan: Edge0 (latent experts) panicked with wgpu's "Out of Memory" when a layer's expert bank did not fit the
card after all: the allocation is now a refusal, and that layer and the later ones go to system memory.
Edge0-120B on a 46 GB A40 decodes at 32.7 tokens/s on Vulkan.
cortiq 0.8.98
Added
- CUDA: binary low-rank (
blr, NVG) weights in the hybrid arm. Each one is repacked at load into binary tile
matrices of the q1 kernels — a V' block per path (group scales ℓ) overg ⊙ x, then U' (group scales h) over
the latents; a gate/up pair shares one interleaved U' — so both stages run on the tensor cores. Qwen3.8-27B-NVG
on an RTX A5000 decodes at 55–58 tokens/s instead of 2.5 (Vulkan: 44) and reads a 4096-token prompt at 282
tokens/s instead of timing out (Vulkan: 48); the batch graph builder takesblron CUDA, whose hybrid arm runs
such stacks the small path's 16 rows at a time. - CUDA: d1's one-pass
setforward on the card (h3_lfm_set_forward, the Metal port's kernels over the weights'
f32 planes). A warm d1-2B question through/v1/system_onetakes 40 ms on an RTX A5000 instead of 850 ms on the
CPU (Vulkan: 65 ms); the first call pays the planes' upload (1.7 s). - Kimi-Linear on the Vulkan / wgpu token and prompt graphs (it used to fall back to the CPU there). KDA layers
run in WGSL (gpu_wgpu/kimi.rs): the short conv with SiLU and the q/k L2 norm, the low-rank decay and β
and the output gate, the delta-rule scan over 128-wide state columns, the gated RMS norm; their state lives
in the GDN maps, laid out as on CUDA. MLA layers take the full-attention arm with their own head geometry
(192-wide keys, 128-wide values, no RoPE) behind a latent RMS norm kernel. q4_tiled experts are repacked on
the device into a nibble plane and an f16 scale plane (gpu_wgpu/moe_q4ts.rs,CMF_MOE_Q4TS=0keeps the
old layout) and run through the 4-row decode and the grouped prefill kernels; prompts of such models go in
256-row chunks. Kimi-Linear-48B-A3B on an RTX A5000: decode 7.7 → 97 tok/s, a 4096-token prompt 4.3 → 507
tok/s; the text equals the CUDA arm's byte for byte.
Fixed
- MiniMax-Music-3 follows ComfyUI's
ar.pywhere the port did not: the RVQ depth decoder runs as the CFG pair
(each stream from its own branch's hidden) and samples every level from the guided logits, and the first
sample's frame is fed back but not kept in the DiT's condition. - MiniMax-Music-3 on CUDA: 10 s of song at 8 steps on an RTX A5000 in 115 s instead of 1126 s (Vulkan: 333 s).
The AR runs as the CUDA token graph by default (CMF_MUSIC3_GRAPH=0opts out), which now reads the hidden state
back alongside the logits — without it the depth decoder read the frame's input and songs ended a few frames
in; the feed-forward half of the DiT runs on the card (h3_music3_ffn); the plane cache may grow into free VRAM
for this pipeline, so the DiT's and the depth decoder's planes stay cached between steps. - CUDA per-op products: a decode-narrow batch (under 8 rows) takes the dense plane only when it is cached or fits
without evicting another, instead of rebuilding a whole stack's planes every token, and batches under 32 rows
multiply in f32 instead of TF32.CMF_CUDA_MM_VERIFY=1checks device products against f32 rows of the weight.
cortiq 0.8.97
Added
- FastH3 (FastVideo's 4-step MiniMax-H3 students,
FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree):
animate-pack --dit-diffusers <transformer/>packs a full MiniMax-H3 DiT in the diffusers layout —to_q | to_k | to_vstacked into the fused panel, the SwiGLU halves swapped back,rope_inv_freqrecomputed, every
block's full adaLN matrix folded into the curve form through one low-rank basis of the student's own time
embedding (ranked by its spectrum) — with the trained VSA gates and the repository's few-step schedule and
sparsity indit.config_json;--dit-from <cmf>takes the DiT of another container by reference over an
--infile's encoder and decoders. The runtime follows the student's own sigma schedule when the step
count matches it. - Video sparse attention (VSA) as FastVideo's VSA-H3 runs it: the packed sequence in tiles of 4×4×4 video
tokens and 64-row chunks of the other streams, per head tile scores from the tile means of q and k, each
video query tile attending its top-k video tiles (10 % at sparsity 0.9) and every non-video tile, non-video
queries attending everything, and the trained gate on the compression branch (softmax over the tile scores
times the tile means of v). The CUDA chain runs it on the card (gpu_cuda/h3_vsa.cu: tile means, scores, a
radix top-k select, the flash kernel over a query tile's key-tile list, the compression branch, the gather
back to row order with the gate); the Vulkan / wgpu chain runs the same kernels in WGSL
(gpu_wgpu/h3_vsa.rs) and the Metal chain in MSL (gpu_metal/h3_vsa.metal,CMF_H3_METAL_VSA=0turns it off;
on an M4 within 0.55 % of the host reference at sparsity 0, 512×288×39 denoised in 129 s against 775 s
on the host).CMF_VSA=0forces dense attention,CMF_VSA_SPARSITYoverrides the packed sparsity,
CMF_VSA_GATE=0drops the compression branch (a diagnostic). - Qwen3-TTS 12 Hz (
Qwen3TTSForConditionalGeneration, the CustomVoice models) runs natively:converttakes
the HF snapshot (the talker as a Qwen3 stack, its text embedding, text projection, code predictor and the
speech tokenizer's decoder asqtts.*, atokenizer.jsonbuilt fromvocab.json+merges.txt), and
cortiq tts <file> "text" --lang ru --speaker serenaspeaks: the talker and the code predictor (a second
pipeline over the same file) on the engine's backends, the decoder (RVQ, a sliding-window transformer,
ConvNeXt upsampling, a SnakeBeta vocoder) on the host with its big products on the device. Greedy codes
matchqwen_tts0.1.1 frame for frame; the decoder on its codes is within 9.3e-4 of its waveform. An
M4 speaks 1.7–1.9× faster than real time, an RTX A5000 1.2× (q4tp). - Haru (higurashi-ai/Haru-4B, full-duplex spoken dialogue at 7.5 Hz):
converttakes the checkpoint (the
Qwen3.5 language model, its audio projectors, the MeanFlow head and the VibeVoice codec asharu.*), and
cortiq duplex <file> user.wav reply.wav --reference voice.wavplays a recording to the model frame by frame,
as in a live call, and writes its answer: the three streams summed into one position, the text token and
the next latent from each final-normed state (DRY, PAD / EOP in float32, one MeanFlow step), the causal
codec over the whole recording and the whole reply. CmfModel::open_view: a file opened again under other tensor names and another architecture (its own uid),
so a stack inside a file can run as a pipeline of its own.- CUDA: the hybrid arm's gate/up pair takes the exact erf GELU (Spark-X2.5), and a stack whose call-wide
rotary table is narrower than its rotary width takes its first attention layer's own rotation as the
call's: Spark-X2.5-4B decodes at 116 tokens/s instead of 94 (q4mix) and 101 instead of 85 (q8_2f) on an
RTX A5000.
Fixed
- CUDA: a short convolution (LFM2/LFM3) and an f32 head-wise output gate now run on the hybrid arm's prompt
path, and the batched prompt graph takes the short-conv layers and the sandwich norms on CUDA, so such a
stack reads a long prompt in one batch: LFM3-1.7B on an RTX A5000 reads 4096 tokens at 9 300–10 200
tokens/s instead of 247. - CUDA: KDA and MLA layers (Kimi Linear) run on the hybrid arm's prompt path — q | k | v, the latent and
kv_b as GEMMs, the KDA delta rule over any chunk 16 rows a pass — instead of 16-row chunks through the
small path: Kimi-Linear-48B-A3B reads 4096 tokens at 2 210–2 255 tokens/s instead of 378 (first token
1.8 s instead of 10.8) on an RTX A5000. - CUDA: routed experts in q8_2f run on the hybrid arm (the tile kernel applies the column field itself)
instead of the CPU: Unlimited-OCR-2 q8_2f reads a page in 7.4 s instead of 33–58 s.
cortiq 0.8.96
Added
-
The resident Embryo graph on CUDA (
gpu_cuda/embryo.rs,embryo.cu): the wide token path of
gpu_wgpu/embryo_wide.wgsl(GDN mixers, bounded anchors with their sinks, the resonance router, the
hierarchical head with its cluster gate) over the same packed model, q8_2f or f16, recurrent state and rings
on the card for the whole sequence;CMF_EMBRYO_RESIDENT=1withCMF_GPU=cuda, the chunked prefill
included. Embryo-56M, which ran op by op on CUDA at 390-410 tok/s, decodes at 884-957 tok/s against the
Vulkan resident graph's 661-720 and the CPU's 424-443 on an RTX A5000 (fast build); the greedy text
identical to the CPU's. -
The Embryo trainer on CUDA (
cortiq-embryo --features cuda): the Vulkan backend's operation recording with
a CUDA device layer (src/cuda_dev.rs), its WGSL kernels translated to CUDA C at start (src/wgsl_cuda.rs,
one source for both backends) and compiled by NVRTC, the PTX kept on disk. Every backend andvulkan_*suite
passes on it. On an RTX A5000, B8×T1024: an Embryo-0 step 339 ms against Vulkan's 708 (24 100 against 11 600
tok/s), a GDN genome's 350 against 511, the first step 0.36 against 2.4 s; the losses of the first six steps
equal Vulkan's (Embryo-0) or within 0.003 (GDN). cuBLAS for the plain GEMMs is opt-in (CMF_CUDA_CUBLAS=1):
no faster here, and it gives up the bit identity of a GEMM's loop and table forms. -
q2tp weights and Prism's activation boundary on the CUDA hybrid arm. q2tp tiles (job kinds 1 and 9) carry a
lane's 16 two-bit codes a 32-column group beside q4tp's ladder and rung codes (rung 0 a zero group), the
registers built exactly as bf16 (the mid-rise 2c - 3 with the half in the scale, or the affine correction's
c - 1);hy_repack2, andhy_expand_q2rfor the prompt path. A Prism file's forward matrices read their
input throughhy_prism_p(the planes in place) orhy_prism_h(f16 rows): every 1024-block of a row as
FWHT(x * signs) / 32 cast to f16, the host's butterflies in their order, at every input site of the token
graph and the prompt path (the GDN gates keep the untouched rows); a stack whose matrices are not all in
the forward basis declines. Ternary-Bonsai-2-27B (q2tp affine, prism_hadamard_qwen35), which ran op by op
on CUDA at 0.44 tok/s, runs the whole token graph: RTX A5000, fast build, CUDA against Vulkan on the same
card, decode 26.1 against 24.5 tok/s, a 4096-token prompt 740 against 117 tok/s (first token 5.5 against
34.9 s); the greedy text identical to the CPU's. -
A routed FFN with q2tp gate/up experts beside q4tp down ones (the
gu_q2profile) on the CUDA hybrid arm,
each expert loaded by its file codec: Qwen3.6-35B-A3B-Escha-W2, which ran op by op at 9 tok/s, decodes at
131.4 tok/s against Vulkan's 99.0 on an RTX A5000 (fast build), a 4096-token prompt 3109 against 177 tok/s
(first token 1.3 against 23.2 s); the greedy text identical to Vulkan's. -
Learned attention sinks and a rotary width of a layer's own on the CUDA hybrid arm. A sink logit joins the
softmax's max and denominator with no value row (hy_valueson the token graph,hy_softmax_hon the
prompt path); each extra rotation table keeps its layer's own width (a NoPE layer's is empty), the layer's
rotary pairs and table offset passed tohy_qk. flint (q2tp; global NoPE layers, sliding RoPE layers of
513 with sinks), which ran op by op on CUDA at 4.4 tok/s, decodes at 125.2 tok/s against Vulkan's 72.3 on an
RTX A5000 (fast build), a 4096-token prompt 3322 against 149 tok/s (first token 1.23 against 27.5 s) and
109.9 against 72.1 tok/s after it; the greedy text identical to the CPU's on both. -
q4_tiled weights on the CUDA hybrid arm (job kind 10): a lane's 16 nibbles a 32-column group (8 bytes, as q4tp's
tiles) and a (row, group)'s f16 scale (hy_repack4t,hy_expand_q4gr), matrices, gate/up pairs, heads and
routed experts (q4tp: false). KAT-Coder-V2.5 (every matrix q4_tiled), which ran op by op on CUDA at 3.3 tok/s,
decodes at 135.3 tok/s against Vulkan's 56.5 on an RTX A5000 (fast build), a 4096-token prompt 3320 against
90 tok/s (first token 1.23 against 45.5 s); the greedy text identical to Vulkan's. -
Gemma-4 on the CUDA hybrid arm: a layer's own head geometry (the global layers' one KV head of 512 beside the
sliding layers' eight of 256: their attention kernels a module of their own, the K/V mirror's rows, the
buffers at the widest layer), the scale-less V norm before the mirror (hy_qk), GeGLU (tanh GELU) in the
gate/up pair's epilogue and on the prompt path, and the sandwich norms and layer scale on the prompt path.
humanizer-12B, which ran op by op on CUDA at 5.1 tok/s, decodes at 48.0 tok/s against Vulkan's 42.1 on an RTX
A5000 (fast build), a 4096-token prompt 779 against 597 tok/s (first token 5.3 against 6.9 s); the greedy text
identical to the CPU's on both. -
A Looped Transformer's mid-stack norms on the CUDA hybrid arm (the final norm on the stream after each
loop's last layer,loop_norm_at), which now takes such stacks from the dense arm;CMF_CUDA_HYB_ALL=1sends
every stack to the hybrid arm. Nanbeige4.2-3B (q4_tiled, 22 layers run twice), which ran op by op on CUDA at
5.2 tok/s, decodes at 68.5 tok/s against Vulkan's 61.7 on an RTX A5000 (fast build); its greedy text
identical to the CPU's. -
q8_2f gate/up pairs (two column fields: a tile's up rows read the input under their own, job kind 15) on
the CUDA hybrid arm, and a launch's input rows put under its jobs' column fields once a call
(hy_field_planes) instead of once a tile (one row, 34816 x 5120: one field 312 -> 282 us, a pair's two
404 -> 275). MiniCPM5-2B q8_2f, whose pairs sent it to the dense arm, decodes at 141-144 tok/s against 89-100
and Vulkan's 133-139 on an RTX A5000; its greedy text the CPU's up to a near tie at token 17 (24.224 against
24.211), as Vulkan's. -
SenseNova-Loop's mHC streams on the CUDA hybrid arm: the residual as up to four streams around both sub-layers,
the router's logits a block each (hy_mhc_logits), its sigmoids and log-domain Sinkhorn mix (hy_mhc_route,
which writes the sub-layer's input rows), the merge after it (hy_mhc_merge), the streams' entry and mean
(hy_mhc_enter,hy_mhc_leave); a sub-layer's product goes apart from the stream. SenseNova-Loop-9B, which ran
op by op on CUDA at 4.4 tok/s, decodes at 48.9 tok/s against Vulkan's 35.5 on an RTX A5000 (fast build); the
greedy text identical to the CPU's. The prompt path runs the streams too (a chunk's streams, coefficients and
logits of its own,mhc_in/mhc_out): a 4096-token prompt at 1393 tok/s against 464 on the small path's
16-row chunks and Vulkan's 1335 (first token 2.94 s), a 450-token prompt's greedy text identical to the CPU's. -
Unlimited-OCR's DASH attention on the CUDA hybrid arm (
hy_dash_route,hy_dash_close,hy_dash_attend): the
rows rotated by their own angles and appended to the f16 mirror (hy_qkover the call's table), the router a
row (index rows, entmax-1.5 over the best 16 candidate blocks, the read set and its log priors), a closing
block's summary key, and the attend over the read set and the local window, on the token graph and the prompt
path; the host's candidate lists go up a call at a time. Unlimited-OCR-2 on an RTX A5000, one page: decode
327.9 tok/s against 50.8 before and Vulkan's 210.4, first token after the page's rows 1.1 s against 21.1 (Vulkan
2.5); the text the CPU's but for a box coordinate off by one, as Vulkan's. -
Kimi-Linear on the CUDA hybrid arm: the graph contract carries Kimi Delta Attention (
GraphAttn::Kda) and
Multi-head Latent Attention (GraphAttn::Mla; the wgpu graphs decline them at their entry). KDA rides a GDN
layer's state and front (the conv, SiLU and q / k norms over the q | k | v channels), its low-rank decay, β and
output gate are f32 products,hy_kda_gatesthe per-channel decay,hy_kda_scan4the delta rule with it, the
sigmoid-gated normhy_gdn_out32's; MLA's latent norm, kv_b and the heads'[k_rope | k_nope]keys and padded
values feed the hybrid attention at heads 192 wide (hy_mla_*; the mirror rows of heads that are no multiple of
256 load two halves at a time), cut back to v wide for the output product. Kimi-Linear-48B-A3B on an RTX A5000:
decode 164 tok/s against 8.8 op by op (Vulkan 8.1), a 4096-token prompt 378 tok/s (first token 10.8 s) and
136 tok/s after it against Vulkan's 4.3 and 2.6; the batched prefill bit for bit the per-position one, the text
the CPU's up to a near tie at token 39 (26.678 against 26.620). -
Vision towers on CUDA (
gpu::vit_encode,gpu::sam_encode,gpu_cuda/vit.rs): the pre-norm ViT blocks of a
CLIP-class tower and SAM's blocks (windowed attention with decomposed relative positions, the global blocks'
bias in the softmax) and neck with the activations on the card, cuBLAS f32 products; they ran on the host
before. Unlimited-OCR-2 on an RTX A5000, a one-page run end to end: 8.4 s against 66.7 (Vulkan 16.5), a tile's
SAM 0.11-0.15 s and CLIP 0.015-0.03 s; the text the same as with the host's towers. -
quantize-gptq --codec q2tp: two steps that cost no bits (docs).--channel-scale ALPHA(with--channel-scale-qkv) multiplies every input channel byt_j = σ_j^α / rms_j^(1−α)before the sweep
and foldstinto the RMSNorm gain in front (post_attention_layernormfor gate/up,input_layernormfor
q/k/v): the same function in floating point, checked byCMF_CHSCALE_FP=1.--refine ITERS --refine-ridge LAMBDA(with--refine-q4tpfor 4-bit tensors) moves single codes to the adjacent level after GPTQ where the
ridge-regularized layer error falls. MiniCPM5-2B at--channel-scale 0.75 --channel-scale-qkv --refine 256 --refine-ridge 0.3, the same 1.314 GB file and Hessians: WikiText-2 test 28.431 → 26.426 (−7.05 % ± 1.20),
validation 28.337 → 26.384, documen...
v0.8.95
Release 0.8.95: Edge0-145B and Edge0-120B in CMF, NVFP4 experts, 12-3…
cortiq 0.8.94
Added
- q1t (ternary) weights on the CUDA hybrid arm (job kind 2): a q1t matrix or gate/up pair without outlier rows
is tiled once on the card (hy_repack1t) into a lane's 16 two-bit codes (value + 1) a 32-column group,
built into bf16x2 registers exactly (128 + c, one fma takes 129 off), a (row, group)'s f16 scale the group's
factor; the prompt path expands them to f16 by bands (hy_expand_q1tr) and a q1t head takes the
tensor-core kernels. A q1t matrix with outlier rows still declines. Bonsai-8B, which ran op by op on CUDA at
1.3 tok/s, runs the whole token graph. RTX A5000, release build, CUDA against Vulkan on the same card (the
greedy texts identical to the CPU's): decode 100.4-100.7 against 8.3 tok/s, a 3072-token prompt 2117 against
36 tok/s (first token 1.45 against 74.9 s) and 92.1 against 7.0 tok/s after it.