Single-file CMF build for local inference (Rust; Vulkan, Metal, CPU)

#7
by infosave - opened

We packaged LFM3-VL-2B as single CMF files that run with cortiq, a Rust inference engine with no Python: the vision encoder at native resolution, images and video frames, on Vulkan/DX12, Metal or the CPU.

Files: https://huggingface.co/infosave/LFM3-VL-2B-cmf — q8_2f (2.14 GB) and q4tp (1.31 GB); the vision encoder stays 16-bit, and its rows match Transformers float32 (cosine above 0.9999).

Decoding runs at 167 / 213 tok/s on an NVIDIA A40 and 82 tok/s (q4tp) on an Apple M4, where a 640×480 image encodes in 0.46 s.

cargo install cortiq-cli
cortiq run lfm3-vl-2b-q8_2f.cmf --image photo.jpg --prompt "Describe this image." --max-tokens 1024

(cortiq 0.8.50)

Sign up or log in to comment