Skip to content

Multimodal Live API

Gunpal Jain edited this page Nov 15, 2026 · 2 revisions

Multimodal Live API

The Live API is a low-latency, bidirectional WebSocket API for real-time voice and video conversations with Gemini. The SDK implements it in a separate package, Google_GenerativeAI.Live, with two clients:

  • MultiModalLiveClient: Live sessions with Gemini Live models (text, audio and video in; audio out).
  • LiveMusicClient: real-time music generation with Lyria RealTime (Live Music).

Both are in the GenerativeAI.Live namespace.

Feature Gemini API Vertex AI
MultiModalLiveClient Yes Yes (via VertextPlatformAdapter)
SendAudioStreamEndAsync Yes No (Gemini API only)
Ephemeral auth tokens (AuthTokensClient) Yes No (NotSupportedException)
LiveMusicClient Yes No

Prerequisites

dotnet add package Google_GenerativeAI.Live
  • A Gemini API key, or Vertex AI credentials. See Authentication.
  • A Live model, for example GoogleAIModels.Gemini38Live (Gemini API) or VertexAIModels.Gemini.Gemini38Live (Vertex AI). Other Live models include GoogleAIModels.Gemini38LiveExtendedThinking and GoogleAIModels.Gemini31FlashLivePreview.

Response modality: use AUDIO

Current Live models answer with audio. Set ResponseModalities to Modality.AUDIO. If you pass no GenerationConfig, the client defaults to Modality.TEXT, which Live audio models reject. To also get the answer as text, turn on output transcription.

using GenerativeAI;
using GenerativeAI.Live;
using GenerativeAI.Types;

var config = new GenerationConfig
{
    ResponseModalities = new List<Modality> { Modality.AUDIO }
};

using var client = new MultiModalLiveClient(
    new GoogleAIPlatformAdapter(apiKey),
    GoogleAIModels.Gemini38Live,
    config,
    systemInstruction: "You are a helpful assistant.")
{
    OutputAudioTranscriptionEnabled = true,   // text transcript of the model's audio
    InputAudioTranscriptionEnabled = true     // text transcript of your audio
};

You can also create a client from an existing model with the extension method generativeModel.CreateMultiModalLiveClient(...) (namespace GenerativeAI.Live.Extensions).

Events

client.Connected += (_, _) => Console.WriteLine("Connected");
client.Disconnected += (_, e) =>
{
    // When the server closes the session with a reason, e is GenerativeAI.Live.Events.ErrorMessageEventArgs.
    if (e is GenerativeAI.Live.Events.ErrorMessageEventArgs closed)
        Console.WriteLine($"Closed by server: {closed.ErrorMessage}");
};
client.ErrorOccurred += (_, e) => Console.WriteLine($"Error: {e.GetException().Message}");

client.AudioChunkReceived += (_, e) => Play(e.Buffer, e.HeaderInfo.SampleRate);   // raw PCM chunk
client.AudioReceiveCompleted += (_, e) => Console.WriteLine($"Turn audio: {e.Buffer.Length} bytes");
client.TextChunkReceived += (_, e) => Console.Write(e.Text);                       // e.IsTurnFinish at the end
client.OutputTranscriptionReceived += (_, t) => Console.Write(t.Text);
client.InputTranscriptionReceived += (_, t) => Console.WriteLine($"You said: {t.Text}");
client.TurnCompleted += (_, _) => Console.WriteLine("\n[turn complete]");
client.GenerationInterrupted += (_, _) => StopPlayback();                           // the user barged in
client.MessageReceived += (_, e) => { /* raw LiveServerMessage in e.Payload */ };
client.GoAwayReceived += (_, goAway) => Console.WriteLine($"Server closing soon, time left: {goAway.TimeLeft}");
client.SessionResumableUpdateReceived += (_, update) => { /* update.NewHandle, update.Resumable */ };

Play and StopPlayback stand for your own audio code. AudioHelper.AddWaveHeader(...) can wrap PCM data in a WAV header.

Connect and send input

ConnectAsync opens the WebSocket and, by default, sends the setup message built from the client's properties.

await client.ConnectAsync();

// A complete user turn (client content).
await client.SentTextAsync("Hi! What's the weather like on Mars?");

// Realtime text: processed like streamed audio, and can interrupt the model.
await client.SendRealtimeTextAsync("Actually, tell me about Venus instead.");

// Realtime audio: 16-bit PCM, 16 kHz mono by default ("audio/pcm;rate=16000").
await client.SendAudioAsync(pcmChunk);
await client.SendAudioStreamEndAsync();   // microphone turned off (Gemini API only)

// Video frames (camera or screen capture), JPEG by default.
await client.SendVideoFrameAsync(jpegBytes);

await client.DisconnectAsync();

For full control, use SendClientContentAsync(LiveClientContent), SendRealtimeInputAsync(LiveClientRealtimeInput), SendToolResponseAsync(...) and SendSetupAsync(LiveClientSetup) (call ConnectAsync(autoSendSetup: false) first).

Function calling and built-in tools

Set FunctionTools (or call AddFunctionTools(functionTools, toolConfig)) before connecting. The client calls your functions when the model requests them and sends the results back. Set UseGoogleSearch or UseCodeExecutor to add those tools. See Function Calling.

Transcription

  • InputAudioTranscriptionEnabled / OutputAudioTranscriptionEnabled turn transcription on.
  • InputAudioTranscriptionConfig / OutputAudioTranscriptionConfig (AudioTranscriptionConfig) set options such as LanguageCodes. Setting a config enables transcription even if the matching ...Enabled flag is false.
  • Transcripts arrive through InputTranscriptionReceived / OutputTranscriptionReceived as Transcription objects (Text, Finished, LanguageCode, SpeakerLabel).

Manual activity detection

By default the server detects when the user starts and stops speaking. To signal turns yourself, disable automatic activity detection and wrap each spoken turn in activity markers:

client.RealtimeInputConfig = new RealtimeInputConfig
{
    AutomaticActivityDetection = new AutomaticActivityDetection { Disabled = true }
};
await client.ConnectAsync();

await client.SendActivityStartAsync();
await client.SendAudioAsync(pcmChunk);      // one or more audio chunks
await client.SendActivityEndAsync();

Activity markers delimit audio. The server rejects realtime text sent between SendActivityStartAsync and SendActivityEndAsync with "Precondition check failed". Send realtime text with SendRealtimeTextAsync outside the markers.

RealtimeInputConfig also has ActivityHandling and TurnCoverage, and AutomaticActivityDetection has StartOfSpeechSensitivity and EndOfSpeechSensitivity for tuning automatic detection.

Session resumption

Set SessionResumption to receive resumption handles. The client stores the latest resumable handle in LastSessionResumptionHandle. To resume later, pass it as SessionResumptionConfig.Handle:

client.SessionResumption = new SessionResumptionConfig();
await client.ConnectAsync();
// ... later, before the session ends:
string? handle = client.LastSessionResumptionHandle;

// New client, same conversation:
using var resumed = new MultiModalLiveClient(new GoogleAIPlatformAdapter(apiKey), GoogleAIModels.Gemini38Live, config)
{
    SessionResumption = new SessionResumptionConfig { Handle = handle }
};
await resumed.ConnectAsync();

The server sends handles at its own pace, so LastSessionResumptionHandle can stay null for a while. When the WebSocket reconnects, the client re-sends the setup message automatically (ResendSetupOnReconnect, default true), using the latest handle if one was received.

GoAway

The server sends a GoAway message shortly before it ends a connection. Handle GoAwayReceived (LiveServerGoAway.TimeLeft) to finish the turn, save LastSessionResumptionHandle and reconnect.

Context window compression

Long sessions can exceed the context window. Enable a sliding window so the server compresses older turns:

client.ContextWindowCompression = new ContextWindowCompressionConfig
{
    TriggerTokens = 100_000,
    SlidingWindow = new SlidingWindow { TargetTokens = 50_000 }
};

Update context mid-session

SendContextUpdateAsync changes the system instruction and/or tools of a running session without reconnecting:

await client.SendContextUpdateAsync("You are a pirate. Answer in one word.");
// Pass tools as the second argument; an empty collection removes all tools, null keeps the current ones.

Changing tools this way does not change FunctionTools, which the client uses to execute function calls; update it too if you add or remove function declarations.

Ephemeral tokens

Client-side apps (browsers, mobile apps) should not hold your API key. Create a short-lived token on your server with AuthTokensClient (Gemini API only), then connect with the token in place of the API key:

using GenerativeAI.Clients;

// Server side
var tokens = new AuthTokensClient(new GoogleAIPlatformAdapter(apiKey));
AuthToken token = await tokens.CreateAsync(new CreateAuthTokenConfig
{
    Uses = 1,
    ExpireTime = DateTime.UtcNow.AddMinutes(10),
    NewSessionExpireTime = DateTime.UtcNow.AddMinutes(2),
    LiveConnectConstraints = new LiveClientSetup
    {
        Model = GoogleAIModels.Gemini38Live,
        GenerationConfig = config
    }
});

// Client side: token.Name ("auth_tokens/...") replaces the API key.
using var live = new MultiModalLiveClient(new GoogleAIPlatformAdapter(token.Name), GoogleAIModels.Gemini38Live, config);
await live.ConnectAsync();

Tokens expire on their own; there is no delete API. LockAdditionalFields can lock more setup fields.

Live Music (Lyria RealTime)

LiveMusicClient streams generated music over the Gemini API Live Music endpoint (experimental, Gemini API only). Audio arrives as raw 16-bit PCM, 48 kHz stereo by default. You can change prompts and config while it plays.

using var music = new LiveMusicClient(new GoogleAIPlatformAdapter(apiKey), GoogleAIModels.Music.LyriaRealtimeExp);

music.AudioChunkReceived += (_, e) => Play(e.Data, e.SampleRate, e.Channels);
music.FilteredPromptReceived += (_, e) => Console.WriteLine($"Prompt filtered: {e.Text} ({e.FilteredReason})");
music.ErrorOccurred += (_, e) => Console.WriteLine(e.GetException().Message);

await music.ConnectAsync();   // waits for setupComplete (TimeoutException after SetupTimeout, default 30 s)
await music.SetWeightedPromptsAsync(new[] { new WeightedPrompt { Text = "calm piano", Weight = 1.0 } });
await music.SetMusicGenerationConfigAsync(new LiveMusicGenerationConfig { Bpm = 90 });
await music.PlayAsync();

// Steer while playing, then stop.
await music.SetWeightedPromptsAsync("upbeat jazz");
await music.PauseAsync();
await music.StopAsync();
await music.DisconnectAsync();

ResetContextAsync resets the music context. There is also a LiveMusicClient(string? apiKey, ...) constructor.

Error handling

  • Sending before ConnectAsync, or after the socket closed, throws InvalidOperationException ("The WebSocket client is not connected.") and raises ErrorOccurred.
  • Server-side rejections (an invalid setup, an unsupported modality, text between activity markers) close the session: handle Disconnected and check for ErrorMessageEventArgs.
  • AuthTokensClient.CreateAsync throws GenerativeAI.Exceptions.ApiException for API errors and NotSupportedException on Vertex AI.
  • SendSetupAsync throws ArgumentException if the model name has no models/ (or other /) prefix; the constructor adds it for you.

Related pages

Clone this wiki locally