Skip to content

Releases

Release v5.23.0

Release v5.23.0 Latest
Latest

Choose a tag to compare

View all tags
@vasqu vasqu released this 13 Nov 14:10
v5.23.0
52b080b

Release v5.23.0

New Model additions

Inkling2

image

Inkling 2 Small is a multimodal model from Thinking Machines Lab that accepts text and image inputs and generates text outputs, and it is intended for use in English and other languages and across multiple coding languages. Its language backbone has 48 mixture-of-experts decoder layers, each routing tokens to 8 of 384 experts, for 594B total and 17B active parameters. It keeps the Inkling decoder (learned relative attention, short causal convolutions, mixture-of-experts layers) and adds block attention residuals, gated attention with a learned sink, and dynamic key/value convolutions. Images are encoded by a Perception Encoder-based vision transformer and projected into the language model embeddings, one embedding per 40x40 pixel patch.

Links: Documentation | Blog Post

JEV

JEV is a decision head that scores single-token labels with a linear layer at the last token of the prompt, soft-capped like the logits of a language model head. JevForDecision is the model of the checkpoint and its head, and it reads the head files shipped next to the weights (head.safetensors and judge_config.json for JEV). The same head is also used by GEV and reads the 255 answer codes of pplx-decider and d3, and it can be run through the experimental decision pipeline or directly with decide.

Note: JEV and the decision pipeline are currently experimental. Their APIs, configuration, and behavior may change as development continues. They should be used with caution in production environments.

Links: Documentation

LFM3

image

LFM3 is the third generation of Liquid Foundation Models from Liquid AI. Like LFM2, it is a hybrid architecture that combines gated short convolutions with grouped-query attention, with the attention layers split between full-attention and sliding-window layers. The attention layers apply RMSNorm to the query and key heads and gate their output with a per-head sigmoid. Every layer applies a sandwich norm to the operator and feed-forward outputs before the residual addition.

Links: Documentation

Dots 3 Note Preview

Dots 3 Note Preview is a mixture-of-experts causal language model that accepts native text, image, video, and audio inputs. It uses a shared vision encoder for images and videos and a Whisper-style audio encoder, and both encoders project their outputs into the language model's hidden space before autoregressive text generation. BF16 and FP8 inference are supported.

Links: Documentation

LFM3-VL

LFM3-VL is a vision-language model family from Liquid AI, built on the LFM3 language model. It takes images and videos at their native resolution together with text. It combines a hybrid vision encoder (a linear patchifier, gated depthwise 2D convolution layers, and bidirectional grouped-query attention layers with 2D rotary embeddings) with a projector and an LFM3 language model. The projector merges each 2x2 block of patch features into one language token through a layer norm and a two-layer MLP.

Links: Documentation

IQuestCoder

IQuest-Coder-V1 is a family of code language models developed by IQuest Lab, available in 7B, 14B and 40B sizes with base, instruct and thinking variants. Its architecture is the same as Llama, so the Transformers auto classes load it with LlamaModel and LlamaForCausalLM, and only IQuestCoderConfig is specific to IQuestCoder. The looped IQuest-Coder-V1-*-Loop-* checkpoints use a different architecture and aren't supported.

Links: Documentation

InternLM3

InternLM3 is an 8B instruction-tuned language model developed by Shanghai AI Laboratory (checkpoint: internlm/internlm3-8b-instruct). Its architecture is the same as Llama, so the auto classes load it with the Llama implementation (LlamaModel and LlamaForCausalLM), and only InternLM3Config is specific to InternLM3. The config names the attention biases qkv_bias and the output projection and MLP biases bias. InternLM3Config loads these as attention_bias and mlp_bias and requires them to be equal, as they are in the released checkpoints.

Links: Documentation

Breaking changes

Exporters: export, share and run any model on its backend (#50029, #50032, #50309, #50311)

exporters.mp4

Every export now returns an ExportedModel. You call it like the original model (same kwargs, same output class), push it to the Hub, and load it back on the backend's own runtime. The same API covers Dynamo, ONNX Runtime, OpenVINO and ExecuTorch.

from transformers import AutoModel, AutoTokenizer
from transformers.exporters import ExportedModel, OnnxConfig, OnnxExporter

model = AutoModel.from_pretrained("google/embeddinggemma-2")
tokenizer = AutoTokenizer.from_pretrained("google/embeddinggemma-2")
inputs = tokenizer(["Exporters are here!"], return_tensors="pt")

exported = OnnxExporter().export(model, inputs, OnnxConfig(dynamic=True, ort_fusions="cuda"))
exported.push_to_hub("my-org/embeddinggemma-2-onnx")

exported_model = ExportedModel.from_pretrained("my-org/embeddinggemma-2-onnx", device="cuda")
outputs = exported_model(**inputs)  # same output class as model(**inputs)
  • Fused for the runtime: OnnxConfig(ort_fusions="cpu" | "cuda" | "webgpu" | "dml") rewrites the graph into ONNX Runtime's fused transformer ops at export time: attention, GroupQueryAttention over the KV cache, rotary, norms and MoE. OpenVINO exports now hit OpenVINO's fused CPU kernels for attention, RoPE, MoE experts and compressed int8/int4 weights, which makes decode 2.4–3.9× faster than before.
  • Faster than torch.compile: on EmbeddingGemma2 (8 × 128 tokens), ONNX Runtime with the TensorRT EP is 7.6× faster than eager PyTorch and 1.4× faster than torch.compile on an A100. OpenVINO is 4.1× faster than eager and 2.2× faster than torch.compile on a Xeon Platinum 8570 (4.9× and 2.6× with int8 weights).
  • Shareable: an export saves and loads like a model, optionally in a subfolder of the model repo. Its export.json records the config, input shapes, package versions and I/O names. Quantized exports (PT2E, NNCF, ONNX Runtime) save and run the same way.

🚨 export() returns an ExportedModel, and an ONNX program is an onnx_ir.Model. OnnxConfig's and OpenVINOConfig's output_path options are gone in favour of save_pretrained. Custom backends register with register_backend.

📚 Docs: https://huggingface.co/docs/transformers/main/en/exporters

Vision

This release fixes a range of vision-model bugs: classification pipelines now respect problem_type, and several models get corrected behavior (Step3.7 RoPE axis order, MiniMax-M3-VL packed attention isolation, Qwen VL positions with reused caches, PaddleOCR-VL interpolation alignment, Kosmos2 SDPA masking, and Emu3 float image handling). Chunked prefill now keeps image inputs on the

  • fix(pipelines): respect problem_type in image and video classification pipelines (#50582) by @surendra902 in [#50582]
  • Fix Step3.7 vision RoPE spatial W/H axis order (#50547) by @caohch-1 in [#50547]
  • Raise ValueError for out-of-range float images in convert_to_rgb (#50487) by @n24q02m in [#50487]
  • CI: install OpenVINO and NNCF in the daily GPU ...
Read more

Contributors

francip, tarekziade, and 74 other contributors
Assets 2
Loading

Patch release v5.22.2

Patch release v5.22.2

Choose a tag to compare

View all tags
@vasqu vasqu released this 06 Nov 15:43
v5.22.2
6d195c6

Patch release v5.22.2

Some in between patch for security and distributed fixes. See the included commits 🤗

Full Changelog: v5.22.1...v5.22.2

Contributors

Rocketknight1, michaelbenayoun, and 3outeille
Loading

Patch release v5.22.1

Patch release v5.22.1

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 30 Oct 09:36
v5.22.1
446a570

Patch release v5.22.1

We just had to fix the documentation!

** [Fix] Remove WIP doc for Ml4 (#50124) from @remi-or

Contributors

remi-or
Loading

v5.22.0: Mistral Large 4 (Le Chonk)

v5.22.0: Mistral Large 4 (Le Chonk)

Choose a tag to compare

View all tags
@LysandreJik LysandreJik released this 29 Oct 14:15
v5.22.0
0f01f27
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

New Models

Mistral Large 4 (Le Chonk)

image

Mistral Large 4 is Mistral's largest and most capable model to date: a 1 trillion-parameter natively multimodal Mixture-of-Experts model with 49 billion active parameters per token (52 billion including embeddings and output layers), trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own datacenters in Europe.

Mistral Large 4 unifies instruction following, reasoning, coding, and agentic workflows in a single set of weights. While a generalist model, it shows especially good performance on workloads such as cybersecurity, finance, and law, and it surpasses frontier closed models on visual grounding. It is natively fluent in 160+ languages, including every official language of the European Union.

Key Features

Mistral Large 4 includes the following architectural choices:

  • ~1T total / ~52B active parameters: granular Mixture-of-Experts with 128 routed experts (4 active per token) and 1 shared expert.
  • 1M context length, for long-context workloads.
  • Multimodal input: accepts text and image input, with text output. Pixtral-style vision encoder trained to handle variable image sizes and aspect ratios.
  • Instruct and Reasoning functionalities with function calls, reasoning effort configurable per request.

Mistral Large 4 offers the following capabilities:

  • Reasoning Mode: three reasoning efforts — low, medium, high (default) — so the same model can answer a quick chat reply or work through a complex agentic run.
  • Vision: analyzes images, charts, technical drawings, and documents, with state-of-the-art visual grounding among open weights.
  • Multilingual: natively fluent in 160+ languages, including all official languages of the European Union.
  • System Prompt: strong adherence and support for system prompts.
  • Agentic: native function calling, JSON output, and strict JSON schema enforcement.
  • Cybersecurity: state of the art among open-weight models for security work, while refusing malicious requests.
  • Large Context Window: supports a 1M context window.

The model is released under the Mistral Ecosystem Open-Weight License: a modified version of the MIT license, for both commercial and non-commercial use — operating a Model-as-a-Service business requires a separate commercial license with Mistral.

Recommended Settings

  • Reasoning Effort:
    • 'low' / 'medium' / 'high' — default is 'high'.
    • Use reasoning_effort="high" for complex prompts and agentic usage, 'low' for fast chat replies and 'medium' for day to day usage.
  • Sampling: for all reasoning efforts, we recommend the following sampling parameters:
    • temperature=1.0
    • top_p=0.95
  • System Prompt: define a clear environment and use case, including guidance on how to effectively leverage tools in agentic systems.
  • Tools: keep the set of tools well-defined and limit their number to the minimum required for the use case.
  • Context: the checkpoint supports a 1M context window.

Breaking changes

Quantization

FP8 quantized models can now be used for adapter training (such as LoRA with PEFT). Backward passes were added to the dense and grouped FP8 matmuls, and is_trainable is now True. Backward also dequantizes experts in chunks and skips autograd at inference, which cuts peak memory in the MoE layer backward from about 28.9 GiB to 2.8 GiB at 8k tokens. Separately, the AMD CI image now install

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @tarekziade
    • Fix _init_weights gaps in 7 audio models and enable test_can_init_all_missing_weights for them (#49386)
    • Fix image backend equivalence tests for Pillow 13 (#50105)
    • [Janus] Fix processor iterating a string prompt character by character (#50044)
    • [timesfm] Fix fp16 overflow on pad_val in _forward_transform (#50021)
    • Pin the shared Hub MIC fleet dataset (#49802)
    • Remove legacy benchmark/ and disable automatic benchmark runs (#50040)
    • [minimax_m3_vl] Gate the integration tests on total accelerator memory (#50025)
    • [moshi] Skip Moshiko integration tests whose checkpoint is gone (#50017)
  • @eustlb
    • Add Nemotron 3.5 Transcribe (#50084)
    • [Nemotron3Diarization] add left context support (#49283)

Contributors

tarekziade, ydshieh, and 21 other contributors
Loading

Patch release v5.21.1

Patch release v5.21.1

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 29 Oct 08:35
v5.21.1
b58f4b3

Patch release v5.21.1

** Fix rope_scaling setter for configs that assign it during init (#50080 )

We broke remote code loading in vllm, this fixes it sorry everyone!

Loading

Release: v5.21.0

Release: v5.21.0

Choose a tag to compare

View all tags
@Cyrilvallez Cyrilvallez released this 26 Oct 18:17
v5.21
dded62b
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Release v5.21.0

New Model additions

MiniCPM-OCR

MiniCPM-OCR is a document-understanding model from OpenBMB's MiniCPM-V family. It keeps the MiniCPM-V 4.6 architecture, which pairs a SigLIP vision encoder and a window-attention merger with a Qwen3.5 language model. It is specialized for reading text-dense images such as scanned pages, screenshots, receipts and tables.

Links: Documentation | Paper

Flint

flint_benchmarks_nvfp4

Flint is Recursive's mixture-of-experts language model. Compared to a standard SwiGLU MoE transformer such as Qwen3-MoE, it uses hybrid attention (sliding-window attention with rotary embeddings over the current token and the previous 512 tokens in most layers, and full attention without positional encoding in the remaining layers). It also has learned attention sinks, as in GPT-OSS, with per-head RMS-normalized queries and keys. Each sparse layer routes a token to 6 of 256 experts by sigmoid score and adds an ungated shared expert, and the first two layers are dense.

Links: Documentation

UltraVBert

UltraVBert extends ModernVBert by pairing the UltraBERT text encoder with a SigLIP vision encoder for visual document understanding and retrieval. Its image processor splits each page into 512-pixel tiles plus a global view. The connector merges each 4 x 4 block of vision patches into one token, so a tile becomes 64 image tokens, and RMS-normalizes them to the scale of UltraBERT's token embeddings. The max_image_tiles setting (27 by default) caps the number of tiles per page.

Links: Documentation

Gradium Phonon TTS

Gradium Phonon TTS is Gradium's family of on-device text-to-speech models, trained with the Pocket TTS recipe. A causal transformer reads a voice prompt and the text, then emits one continuous 32-dimensional audio latent per step at 12.5 Hz through a one-step flow head, and an end-of-speech head decides when the utterance ends. The latents are decoded to 24 kHz audio by a Pocket TTS audio tokenizer. Unlike Pocket TTS, voice conditioning uses 512-dimensional latents from a separate, unreleased speaker codec, so voices come as precomputed latents and cannot be cloned from a recording.

Links: Documentation

Pocket TTS

Pocket TTS is Kyutai's 100M-parameter text-to-speech model, built as a Continuous Audio Language Model that runs faster than real time on CPU. A causal transformer emits one continuous audio latent per frame (12.5 Hz) through a one-step flow head, and an end-of-speech head decides when to stop. A Mimi-based audio tokenizer encodes voice prompts and decodes the generated latents to 24 kHz audio. Two flow heads are released, Lagrangian self-distillation (default) and drifting, and voices can be cloned from a recording, passed as precomputed latents, or loaded from cached KV states.

Links: Documentation | [Paper](https://huggingface.co/papers/2509.06926 (the Continuous Audio Language Models paper, arXiv 2509.06926). It is the only paper the documentation links, and it describes the architecture family Pocket TTS belongs to. The docs do not cite a separate Pocket TTS technical report.) | [Blog Post](NONE. The documentation links no official blog post. It points only to the original code repository at https://github.com/kyutai-labs/pocket-tts.)

Pocket TTS Audio Tokenizer

The Pocket TTS audio tokenizer is the codec used by Pocket TTS. It has Mimi's architecture (SEANet convolutions plus a small causal transformer on each side), but its residual vector quantizer is replaced by a continuous 32-dimensional bottleneck, the VAE-GAN from Continuous Audio Language Models. It converts 24 kHz audio into latents at 12.5 Hz and decodes them back to audio. It serves as the audio_tokenizer of PocketTTSProcessor, encoding voice prompts and decoding generated latents, and its encoder and decoder can also be loaded on their own.

Links: Documentation | Paper

Breaking changes

Fixed Gemma4 processor imports (the Gemma4, Gemma4 unified and EmbeddingGemma2 processors) so from transformers import * no longer fails in environments that have PIL but not torch; no user action is needed.

  • 🚨 Fix gemma4 cross imports resulting in import failing without torch but with PIL (#49748) by @Sai-Suraj-27

Added an "fp8" backend to QuantizedCache, alongside quanto and hqq, that stores KV states in torch.float8_e4m3fn with a single scale per tensor; enable it with cache_implementation="quantized" and cache_config={"backend": "fp8"}.

Tensor parallelism now has a dedicated embedding style, shards every model's input embeddings by default, and supports widths that aren't divisible by the TP size, so ColwiseParallel and RowwiseParallel now apply only to linear layers; custom TP plans that applied them to embeddings must switch to the new embedding style.

  • 🚨 TP: give embeddings their own style, shard input embeddings by default, and shard widths that don't divide by the TP size (#49570) by @IlyasMoutawwakil

Every MoE TP and EP plan now shards its experts, so models no longer keep all experts on every rank, and a new test_moe_parallel_plans_shard_experts test enforces this; custom or downstream MoE parallel plans must add expert sharding to pass it and to avoid wasted memory.

Cache

This release improves cache handling across the library. Flash Attention now works with static caches, BLOOM no longer crashes with static cache or compiled generate, chunked prefill supports cache continuation, and the Qwen4-Exp indexer's position history stays aligned with its cached keys. Linear-attention caches also gain new getters (is_recording_past, get_recurrent_state, conv-state access) that stop exposing cache internals and prepare for paged-c

Vision

Vision image processors now composite PNG transparency onto white instead of exposing black pixels, and background_color is preserved correctly across the DeepSeek-VL and Janus processors. Model fixes cover SmolVLM's inputs_merger when image tokens aren't a multiple of the patch size, HunYuan-VL image assignment during batched beam search, and Ernie4.5-VL-MoE expert sharding under TP/EP. The DETR

Generation

This release fixes a range of generation bugs: BLOOM crashes with static cache or compiled generate, generate dropping the type of dict-like inputs such as ModelOutput, and incorrect image assignment or placeholder handling ...

Read more

Contributors

tarekziade, RyanMullins, and 65 other contributors
Loading

Patch release v5.20.2

Patch release v5.20.2

Choose a tag to compare

View all tags
@Cyrilvallez Cyrilvallez released this 19 Oct 17:45
v5.20.2
18329f1
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Patch release: v5.20.2

This patch notably solves an issue where some inputs could loose their original type inside prepare_inputs_for_generation in generate, leading to problems with multimodal models. It also fixes some other issues listed on the below commits.

It contains the following commits:

Contributors

hmellor, qgallouedec, and Cyrilvallez
Loading

Release v5.20.1

Release v5.20.1

Choose a tag to compare

View all tags
@vasqu vasqu released this 16 Oct 13:02
v5.20.1
0ea8779

Release v5.20.1

Special surprise including the new model Beam from Reflection AI 🥳

Gemini_Generated_Image_ox4hjlox4hjlox4h

Beam is a token-efficient, frontier-scale model focused on coding and agentic use cases. It is built for teams that want to run a powerful model on their own hardware, infrastructure, or cloud, and customize it to their specific needs.

Links: Documentation

Some real patch fixes

We broke loading models on older torch versions 😢 should be fixed now. But here is the full list of fixes

Full Changelog: v5.20.0...v5.20.1

Contributors

IlyasMoutawwakil, yonigozlan, and 3 other contributors
Loading

Release v5.20.0

Release v5.20.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 15 Oct 18:05
v5.20.0
48e4bcd

Release v5.20.0

New Model additions

Muse Spark Open

image

Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats.

Links: Documentation | Blog Post

Ling 3.0

Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as BailingHybridForCausalLM, so it loads without trust_remote_code=True.

Links: Documentation

Kolibri1

Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers.

Links: Documentation

MiniCPM-V 4.7

MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB. Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes.

The main addition over 4.6 is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a (temporal, height, width) position, so slices of the same image keep their spatial relationship and video frames keep their temporal order.

UnlimitedOcr

UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes.

Links: Documentation | Paper

UltraBERT

UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings.

Links: Documentation | Paper

MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as [start][S01]text[end]. It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages.

Links: Documentation

Breaking changes

🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨

Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API.

Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead.

The minimum kernels version is now 0.17.2, which allows use_fallback=KernelizeFallback.ALL so that a failed kernel load falls back to the original implementation instead of raising an error. Upgrade kernels to 0.17.2 or later.

  • 🚨 Bump min kernels to 0.17.2 for better fallback (#49235) by @SunMarc

A new SubConfigSpec centralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism.

The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens (not split, honored by skip_special_tokens, normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior.

Exporters

Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends.

For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization

Parallelization

This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache generate(), and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry, enable_expert_parallel applies the EP plan alone again, TPU loading uses the `tor

Generation

This release fixes static-cache generate() bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings.

...

Read more

Contributors

francip, shadeMe, and 44 other contributors
Loading

Release v5.19.0

Release v5.19.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 06 Oct 16:39
v5.19.0
c4c4605

Release v5.19.0

New Model additions

EmbeddingGemma2

image

EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video, individually or combined in one input, into a shared 768-dimensional vector space for cross-modal retrieval, semantic similarity, clustering, and classification. It uses Matryoshka Representation Learning, so embeddings can be truncated to 512, 256, or 128 dimensions. It also offers configurable visual and video token budgets, and unused vision or audio towers can be disabled at load time to save memory.

Links: Documentation

Breaking changes

All MoE models whose routers compute logits now return them when output_router_logits=True, following the Qwen3-MoE pattern (a router_logits recorder on the base model, MoeModelOutputWithPast from the backbone, and a MoE causal LM output from the head), so code that relied on the previous outputs or their absence should read the router logits from these output classes.

  • 🚨 Return router logits from every MoE model that computes them (#48920) by @qgallouedec

Owlv2ForObjectDetection.embed_image_query now selects the query box with the highest objectness score, as in the original OWLv2 notebook, instead of the OWL-ViT heuristic, so image-guided query embeddings and detections may differ from earlier releases.

The "paged|" prefix for SDPA and flash attention implementations is deprecated, so users should set the regular attention implementation (e.g. sdpa or flash_attention_2) for continuous batching instead of paged|sdpa or paged|flash_attention_2.

  • 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash (#49112) by @remi-or

The regular flash and SDPA attention functions (flash_attention.py, sdpa_attention.py) now support continuous batching directly, and "paged|..." implementations for these are redirected to them, while eager still requires the "paged|eager" prefix.

  • 🚨 Attention 🚨 Make regular attention support CB (#49101) by @remi-or

In continuous batching, the cache update for the index-based and block-table paths is now fused into a single call, which slightly changes the cache update function's behavior and affects any custom code that calls the separate update paths.

  • 🚨 [CB] 🚨 Fuse update for index and block table path (#49088) by @remi-or

Continuous batching internals changed in preparation for removing "paged": `max

  • 🚨 [CB] 🚨 Little fixes before removing "paged" (#49069) by @remi-or

Parallelization

Expert parallelism gains a token-dispatch implementation, selected via the new ep_dispatch_experts plan rule and now the default for Qwen3 MoE and Mellum, which removes the requirement that EP size equal TP size. The Trainer was also adapted to work with expert parallelism, and the docs now note that PEFT adapters support tensor parallelism. A CI-related fix for pipeline-parallel chart2table inference was also included.

Cache

This release fixes quantized cache handling: generate no longer mutates the user's cache_config, and QuantizedLayer.reorder_cache is repaired. It also adds per-layer cache configuration, so DynamicCache and StaticCache initialize each layer from its own config (sliding window, attention chunk size, conv states, and attention head counts) to better support heterogeneous models. Separately, the deprecation cycle on mask and cache

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @vasqu
  • @remi-or
    • [Fix] Remove old test file with two ancient tests (#49280)
    • [CB] Make CB more device agnostic and add support for XPU (#49156)
    • 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash (#49112)
    • 🚨 Attention 🚨 Make regular attention support CB (#49101)
    • 🚨 [CB] 🚨 Fuse update for index and block table path (#49088)
    • [Refactor] Make some flash-attention utils more readable (#49071)
    • 🚨 [CB] 🚨 Little fixes before removing "paged" (#49069)

Contributors

tarekziade, dacorvo, and 21 other contributors
Loading