tml-renderers

tml-renderers turns structured text, image, and audio chat into model-ready tokens and media for training and inference. It supports Thinking Machines Lab models, including Inkling and Inkling-Small, through TMLv0, and the supported third-party models served on Tinker through model-specific renderers.

Installation

Always use the latest release:

pip install --upgrade tml-renderers

tml-renderers is a compiled extension that requires PyTorch 2.10 or newer.

Quickstart

Create a short conversation with native chat objects and render it for inference or training:

from tml_renderers import chat, renderers

user = chat.Message(
    content=chat.Text("Hello, world!"),
    author=chat.Author(chat.AuthorKind.User),
)
assistant = chat.Message(
    content=chat.Text("Hello!"),
    author=chat.Author(chat.AuthorKind.Model),
)
native_messages = chat.MessageList([user, assistant])

renderer = renderers.Qwen3()
spans, parser = renderer.render_for_completion([user])
training_examples = renderer.render_for_sft(native_messages)

To continue from the exact sampled token prefix, bridge the parsed semantic history to the next user or tool messages instead of rerendering earlier turns:

sampled = chat.TokenSpan(chat.EncodedTextTokenSpan(sampled_token_ids))
parsed = parser.parse_tokens(sampled_token_ids)
follow_up = chat.Message(
    content=chat.Text("Tell me more."),
    author=chat.Author(chat.AuthorKind.User),
)

spans, suffix, parser = renderer.bridge_to_next_turn(
    [*spans, sampled],
    [user, *parsed],
    [follow_up],
)

The first return value preserves every token in the supplied prefix and appends suffix. The returned parser is configured for the combined semantic history.

Supported models

Provider / family	Models	Renderer classes
thinkingmachines	Inkling, Inkling-Small	Inkling (TMLv0)
Qwen	Qwen3 (including vision), Qwen3.5, Qwen3.8	Qwen3, Qwen35, Qwen38
DeepSeek	V3, V3.1, V4, V4.1 Flash	DeepSeekV3, DeepSeekV31, DeepSeekV4, DeepSeekV41
Kimi	K2, K2.5, K3.1-Pro	KimiK2, KimiK25, KimiK31Pro
GLM	5.2, 5.3, 5.3 Flash	Glm52, Glm53, Glm53Flash
GPT-OSS	GPT-OSS	GptOss
Nemotron	Nemotron 3, Nemotron 3.5, Nemotron 3.5 Super VL	Nemotron3, Nemotron3VL

Inkling2 renders the same TMLv0 format with the H200K vocabulary.

GLM history defaults match the upstream templates: Glm52 clears earlier-turn reasoning; Glm53 and Glm53Flash preserve it, including their low/high presets. Pass clear_thinking=True to opt into clearing for GLM-5.3 chat, as Z.ai recommends. This changes the previous GLM-5.3 renderer default; callers that need clearing must now request it explicitly. Use the same setting for rollout and SFT.

OpenAI message compatibility

The same conversation can use OpenAI-style dictionaries:

Message	Native chat object	OpenAI-style dict
User	chat.Message(content=chat.Text("Hello, world!"), author=chat.Author(chat.AuthorKind.User))	{"role": "user", "content": "Hello, world!"}
Assistant	chat.Message(content=chat.Text("Hello!"), author=chat.Author(chat.AuthorKind.Model))	{"role": "assistant", "content": "Hello!"}

Convert the dictionaries before passing them to a renderer:

from tml_renderers import chat, renderers

openai_dicts = [
    {"role": "user", "content": "Hello, world!"},
    {"role": "assistant", "content": "Hello!"},
]
openai_messages = chat.OpenAIMessage.from_openai_chat_dicts(openai_dicts)
round_tripped = chat.OpenAIMessage.to_openai_chat_dicts(openai_messages)

renderer = renderers.Qwen3()
training_examples = renderer.render_for_sft(openai_messages)

The renderer also accepts openai_messages for completion. The conversion supports text, reasoning, images, tool calls, and tool results; it handles message formats, not the OpenAI network API. An assistant dict keeps its reasoning in a top-level reasoning_content. As in an OpenAI request, tools are not a message: pass the tools list as chat.OpenAIMessage.from_openai_chat_dicts(openai_dicts, tools=tools).

The from_oss_* and to_oss_* methods read and write tinker_cookbook message dicts. They are deprecated.

Rendering and parsing (TMLv0)

Rendering converts structured chat messages into the token and media-span sequence consumed by the model. renderers.Inkling and renderers.Inkling2 use the same TMLv0 grammar for both inference and training:

• render_for_completion(...) renders an inference prompt and returns a parser for turning generated model tokens back into messages.
• render_for_sft(...) renders supervised-training examples with aligned, per-token loss weights.

The parser reconstructs structured messages from generated token spans:

from tml_renderers import chat, renderers

renderer = renderers.Inkling()
tokenizer = renderer.tokenizer

messages = [
    chat.Message(
        content=chat.Text("Hello"),
        author=chat.Author(chat.AuthorKind.User),
    )
]

spans, parser = renderer.render_for_completion(messages)
parsed = parser.parse(spans)

training_examples = renderer.render_for_sft(messages)

Parser setup is deferred until the first batch parse or stream acquisition, so callers that only inspect the rendered prompt do not pay for unused parser state. For incremental sampling, one parser stream owns the parsing attempt until it finishes or leaves scope. Feed only newly sampled token IDs. Completed Message updates are equivalent to batch parsing the same token sequence:

completed_messages = []
with parser.parse_stream() as stream:
    for token_ids in sampled_token_chunks:
        for update in stream.feed(token_ids):
            event = update.update
            if isinstance(event, chat.StreamingMessageHeader):
                print(f"[{event.author.kind}] ", end="")
            elif isinstance(event, chat.StreamingContent):
                print(event.content.text, end="", flush=True)
            elif isinstance(event, chat.Message):
                completed_messages.append(event)
    for update in stream.finish():
        if isinstance(update.update, chat.Message):
            completed_messages.append(update.update)

finish() validates the end of the stream and releases the parser. Leaving the context unfinished discards partial state and releases the parser. Parse exceptions expose completed_messages, which contains any complete prefix parsed before the error.

The package includes native tokenizers for these renderers. It does not need Hugging Face chat templates or remote code. Each TMLv0 renderer builds the tokenizer for its model's vocabulary: Inkling uses tokenizers.O200kBaseChatTokenizer() and Inkling2 uses tokenizers.H200kBaseChatTokenizer(). Both bundle their ranks for offline use and share the same public special-token IDs. v0.Renderer(tokenizer) accepts either tokenizer.

tokenizer.encode_ordinary("hello")
tokenizer.decode([24912])
tokenizer.encode_special("message_user")   # -> 200000
tokenizer.decode_special(200000)            # -> "message_user"

qwen_tokenizer = tokenizers.Qwen3Tokenizer()
qwen_tokenizer.encode_special("im_start")   # -> 151644

Full method surface:

method	maps	on invalid / partial UTF-8
encode_ordinary(text)	text → ids	— (special-token markup is encoded as literal text)
encode_special(name)	name → id	raises if name isn't a special token
decode(ids)	ids → text	lossy — invalid/partial bytes become U+FFFD, never raises (tiktoken parity)
decode_strict(ids)	ids → text	raises ValueError on invalid/partial UTF-8
decode_bytes(ids)	ids → bytes	raw bytes, no UTF-8 handling
decode_special(id)	id → name	None if id isn't a special token

The tokenizer also provides is_special_token(id), special_tokens(), and eos_token, bos_token, and all_special_tokens. decode is lossy: a truncated multibyte tail renders as � rather than raising. Use decode_strict when invalid UTF-8 must raise an error.

render_for_completion accepts a MessageList, a list[Message], or a list[OpenAIMessage]. For TMLv0, render_for_completion_with_effort(messages, effort) accepts an effort float in [0, 1] and inserts a system message before the first non-system message; input messages must not already contain ThinkingEffort. For SFT, render_for_sft returns TrainingExamples with per-token loss weights.

Multimodal inputs

tml-renderers includes multimodal codecs for the TMLv0 renderer:

• An ImagePointer is rendered as an ImageAssetPointerTokenSpan with the expected image-token count.
• An AudioPointer is rendered as a DmelTokenSpan.

Image and audio messages use the same completion and training APIs as text:

from tml_renderers import chat, renderers

renderer = renderers.Inkling()

image_msg = chat.Message(
    content=chat.ImagePointer(
        location="image.png",
        format=chat.ImageFormat.Png,
        width=512,
        height=512,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

audio_msg = chat.Message(
    content=chat.AudioPointer(
        location="clip.wav",
        format=chat.AudioFormat.Wav,
        num_frames=48_000,
        sample_rate=16_000,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

spans, _ = renderer.render_for_completion([image_msg, audio_msg])
dmel = next(s.span for s in spans if type(s.span).__name__ == "DmelTokenSpan")
print(dmel.dmel.shape)  # [num_audio_tokens, num_dmel_bins], uint8

Media locations are represented as pointers. Audio is read from local files; tml-renderers never fetches remote media.

Token spans

Renderers work with token-span types you can construct directly:

from tml_renderers.chat import (
    EncodedTextTokenSpan,
    ImageAssetPointerTokenSpan,
    ImageFormat,
    PaddingTokenSpan,
    TokenSpan,
)

text = EncodedTextTokenSpan([101, 102])
image = ImageAssetPointerTokenSpan(
    location="image.png", format=ImageFormat.Png, width=512, height=512, num_tokens=256
)
padding = PaddingTokenSpan(4)

# Wrap in the oneof only where a call boundary asks for a TokenSpan.
wrapped = TokenSpan(text)

DmelTokenSpan is the encoded-audio span produced by the renderer. Its dmel field is a uint8 tensor of shape [num_audio_tokens, num_dmel_bins].

Using with Tinker

In Tinker Cookbook, use the recommended renderer for your model. Select tml_v0 for Inkling and Inkling-Small; supported third-party models use their model-specific renderers. Cookbook converts rendered text, image, and audio inputs for sampling and SFT.

The lower-level tml_renderers.tinker helpers are also available for direct Tinker SDK integration. Provide local media files or base64 image data.

Types

• Core: Message, MessageList, Author, AuthorKind, MessageChannel, MessageMetadata
• Content: Text, Thinking, ThinkingEffort, ImagePointer, AudioPointer, InvokeTool, StructuredToolCall, ToolError, ToolDeclareJson, ModelEndSampling
• Token spans: TokenSpan, EncodedTextTokenSpan, ImageAssetPointerTokenSpan, DmelTokenSpan, PaddingTokenSpan
• Formats: ImageFormat, AudioFormat
• Tool args: ToolArg, ToolMetadata. ToolArg.value is a JSON-encoded string, matching the OpenAI function.arguments convention.
• Tool declarations: ToolSpecJson (one declared tool with name, description, type_name, and JSON-schema parameters), grouped into ToolDeclareJson. The constructor accepts a dict for parameters and serializes it.

Examples

• examples/run_tml_v0.py — low-level rendering and parsing of text and audio.
• examples/cookbook_tinker_demo.py — native and OpenAI-style messages, including tool calls and results. See the public tool-call example.

License

Apache-2.0. The wheel includes LICENSE, NOTICE, and THIRD_PARTY_LICENSES.md.