Repository navigation
Multimodal Live API
The Live API is a low-latency, bidirectional WebSocket API for real-time voice and video conversations with Gemini.
The SDK implements it in a separate package, Google_GenerativeAI.Live, with two clients:
-
MultiModalLiveClient: Live sessions with Gemini Live models (text, audio and video in; audio out). -
LiveMusicClient: real-time music generation with Lyria RealTime (Live Music).
Both are in the GenerativeAI.Live namespace.
| Feature | Gemini API | Vertex AI |
|---|---|---|
MultiModalLiveClient |
Yes | Yes (via VertextPlatformAdapter) |
SendAudioStreamEndAsync |
Yes | No (Gemini API only) |
Ephemeral auth tokens (AuthTokensClient) |
Yes | No (NotSupportedException) |
LiveMusicClient |
Yes | No |
dotnet add package Google_GenerativeAI.Live- A Gemini API key, or Vertex AI credentials. See Authentication.
- A Live model, for example
GoogleAIModels.Gemini38Live(Gemini API) orVertexAIModels.Gemini.Gemini38Live(Vertex AI). Other Live models includeGoogleAIModels.Gemini38LiveExtendedThinkingandGoogleAIModels.Gemini31FlashLivePreview.
Current Live models answer with audio. Set ResponseModalities to Modality.AUDIO. If you pass no
GenerationConfig, the client defaults to Modality.TEXT, which Live audio models reject. To also get the
answer as text, turn on output transcription.
using GenerativeAI;
using GenerativeAI.Live;
using GenerativeAI.Types;
var config = new GenerationConfig
{
ResponseModalities = new List<Modality> { Modality.AUDIO }
};
using var client = new MultiModalLiveClient(
new GoogleAIPlatformAdapter(apiKey),
GoogleAIModels.Gemini38Live,
config,
systemInstruction: "You are a helpful assistant.")
{
OutputAudioTranscriptionEnabled = true, // text transcript of the model's audio
InputAudioTranscriptionEnabled = true // text transcript of your audio
};You can also create a client from an existing model with the extension method
generativeModel.CreateMultiModalLiveClient(...) (namespace GenerativeAI.Live.Extensions).
client.Connected += (_, _) => Console.WriteLine("Connected");
client.Disconnected += (_, e) =>
{
// When the server closes the session with a reason, e is GenerativeAI.Live.Events.ErrorMessageEventArgs.
if (e is GenerativeAI.Live.Events.ErrorMessageEventArgs closed)
Console.WriteLine($"Closed by server: {closed.ErrorMessage}");
};
client.ErrorOccurred += (_, e) => Console.WriteLine($"Error: {e.GetException().Message}");
client.AudioChunkReceived += (_, e) => Play(e.Buffer, e.HeaderInfo.SampleRate); // raw PCM chunk
client.AudioReceiveCompleted += (_, e) => Console.WriteLine($"Turn audio: {e.Buffer.Length} bytes");
client.TextChunkReceived += (_, e) => Console.Write(e.Text); // e.IsTurnFinish at the end
client.OutputTranscriptionReceived += (_, t) => Console.Write(t.Text);
client.InputTranscriptionReceived += (_, t) => Console.WriteLine($"You said: {t.Text}");
client.TurnCompleted += (_, _) => Console.WriteLine("\n[turn complete]");
client.GenerationInterrupted += (_, _) => StopPlayback(); // the user barged in
client.MessageReceived += (_, e) => { /* raw LiveServerMessage in e.Payload */ };
client.GoAwayReceived += (_, goAway) => Console.WriteLine($"Server closing soon, time left: {goAway.TimeLeft}");
client.SessionResumableUpdateReceived += (_, update) => { /* update.NewHandle, update.Resumable */ };Play and StopPlayback stand for your own audio code. AudioHelper.AddWaveHeader(...) can wrap PCM data in a WAV
header.
ConnectAsync opens the WebSocket and, by default, sends the setup message built from the client's properties.
await client.ConnectAsync();
// A complete user turn (client content).
await client.SentTextAsync("Hi! What's the weather like on Mars?");
// Realtime text: processed like streamed audio, and can interrupt the model.
await client.SendRealtimeTextAsync("Actually, tell me about Venus instead.");
// Realtime audio: 16-bit PCM, 16 kHz mono by default ("audio/pcm;rate=16000").
await client.SendAudioAsync(pcmChunk);
await client.SendAudioStreamEndAsync(); // microphone turned off (Gemini API only)
// Video frames (camera or screen capture), JPEG by default.
await client.SendVideoFrameAsync(jpegBytes);
await client.DisconnectAsync();For full control, use SendClientContentAsync(LiveClientContent),
SendRealtimeInputAsync(LiveClientRealtimeInput), SendToolResponseAsync(...) and
SendSetupAsync(LiveClientSetup) (call ConnectAsync(autoSendSetup: false) first).
Set FunctionTools (or call AddFunctionTools(functionTools, toolConfig)) before connecting. The client calls
your functions when the model requests them and sends the results back. Set UseGoogleSearch or UseCodeExecutor
to add those tools. See Function Calling.
-
InputAudioTranscriptionEnabled/OutputAudioTranscriptionEnabledturn transcription on. -
InputAudioTranscriptionConfig/OutputAudioTranscriptionConfig(AudioTranscriptionConfig) set options such asLanguageCodes. Setting a config enables transcription even if the matching...Enabledflag is false. - Transcripts arrive through
InputTranscriptionReceived/OutputTranscriptionReceivedasTranscriptionobjects (Text,Finished,LanguageCode,SpeakerLabel).
By default the server detects when the user starts and stops speaking. To signal turns yourself, disable automatic activity detection and wrap each spoken turn in activity markers:
client.RealtimeInputConfig = new RealtimeInputConfig
{
AutomaticActivityDetection = new AutomaticActivityDetection { Disabled = true }
};
await client.ConnectAsync();
await client.SendActivityStartAsync();
await client.SendAudioAsync(pcmChunk); // one or more audio chunks
await client.SendActivityEndAsync();Activity markers delimit audio. The server rejects realtime text sent between SendActivityStartAsync and
SendActivityEndAsync with "Precondition check failed". Send realtime text with SendRealtimeTextAsync outside
the markers.
RealtimeInputConfig also has ActivityHandling and TurnCoverage, and AutomaticActivityDetection has
StartOfSpeechSensitivity and EndOfSpeechSensitivity for tuning automatic detection.
Set SessionResumption to receive resumption handles. The client stores the latest resumable handle in
LastSessionResumptionHandle. To resume later, pass it as SessionResumptionConfig.Handle:
client.SessionResumption = new SessionResumptionConfig();
await client.ConnectAsync();
// ... later, before the session ends:
string? handle = client.LastSessionResumptionHandle;
// New client, same conversation:
using var resumed = new MultiModalLiveClient(new GoogleAIPlatformAdapter(apiKey), GoogleAIModels.Gemini38Live, config)
{
SessionResumption = new SessionResumptionConfig { Handle = handle }
};
await resumed.ConnectAsync();The server sends handles at its own pace, so LastSessionResumptionHandle can stay null for a while. When the
WebSocket reconnects, the client re-sends the setup message automatically (ResendSetupOnReconnect, default
true), using the latest handle if one was received.
The server sends a GoAway message shortly before it ends a connection. Handle GoAwayReceived
(LiveServerGoAway.TimeLeft) to finish the turn, save LastSessionResumptionHandle and reconnect.
Long sessions can exceed the context window. Enable a sliding window so the server compresses older turns:
client.ContextWindowCompression = new ContextWindowCompressionConfig
{
TriggerTokens = 100_000,
SlidingWindow = new SlidingWindow { TargetTokens = 50_000 }
};SendContextUpdateAsync changes the system instruction and/or tools of a running session without reconnecting:
await client.SendContextUpdateAsync("You are a pirate. Answer in one word.");
// Pass tools as the second argument; an empty collection removes all tools, null keeps the current ones.Changing tools this way does not change FunctionTools, which the client uses to execute function calls; update it
too if you add or remove function declarations.
Client-side apps (browsers, mobile apps) should not hold your API key. Create a short-lived token on your server
with AuthTokensClient (Gemini API only), then connect with the token in place of the API key:
using GenerativeAI.Clients;
// Server side
var tokens = new AuthTokensClient(new GoogleAIPlatformAdapter(apiKey));
AuthToken token = await tokens.CreateAsync(new CreateAuthTokenConfig
{
Uses = 1,
ExpireTime = DateTime.UtcNow.AddMinutes(10),
NewSessionExpireTime = DateTime.UtcNow.AddMinutes(2),
LiveConnectConstraints = new LiveClientSetup
{
Model = GoogleAIModels.Gemini38Live,
GenerationConfig = config
}
});
// Client side: token.Name ("auth_tokens/...") replaces the API key.
using var live = new MultiModalLiveClient(new GoogleAIPlatformAdapter(token.Name), GoogleAIModels.Gemini38Live, config);
await live.ConnectAsync();Tokens expire on their own; there is no delete API. LockAdditionalFields can lock more setup fields.
LiveMusicClient streams generated music over the Gemini API Live Music endpoint (experimental, Gemini API only).
Audio arrives as raw 16-bit PCM, 48 kHz stereo by default. You can change prompts and config while it plays.
using var music = new LiveMusicClient(new GoogleAIPlatformAdapter(apiKey), GoogleAIModels.Music.LyriaRealtimeExp);
music.AudioChunkReceived += (_, e) => Play(e.Data, e.SampleRate, e.Channels);
music.FilteredPromptReceived += (_, e) => Console.WriteLine($"Prompt filtered: {e.Text} ({e.FilteredReason})");
music.ErrorOccurred += (_, e) => Console.WriteLine(e.GetException().Message);
await music.ConnectAsync(); // waits for setupComplete (TimeoutException after SetupTimeout, default 30 s)
await music.SetWeightedPromptsAsync(new[] { new WeightedPrompt { Text = "calm piano", Weight = 1.0 } });
await music.SetMusicGenerationConfigAsync(new LiveMusicGenerationConfig { Bpm = 90 });
await music.PlayAsync();
// Steer while playing, then stop.
await music.SetWeightedPromptsAsync("upbeat jazz");
await music.PauseAsync();
await music.StopAsync();
await music.DisconnectAsync();ResetContextAsync resets the music context. There is also a LiveMusicClient(string? apiKey, ...) constructor.
- Sending before
ConnectAsync, or after the socket closed, throwsInvalidOperationException("The WebSocket client is not connected.") and raisesErrorOccurred. - Server-side rejections (an invalid setup, an unsupported modality, text between activity markers) close the
session: handle
Disconnectedand check forErrorMessageEventArgs. -
AuthTokensClient.CreateAsyncthrowsGenerativeAI.Exceptions.ApiExceptionfor API errors andNotSupportedExceptionon Vertex AI. -
SendSetupAsyncthrowsArgumentExceptionif the model name has nomodels/(or other/) prefix; the constructor adds it for you.
- Authentication
- Function Calling
- Multimodal Features
- Agent APIs: custom voices
Getting Started
- Initialization
- Authentication
- Configuration & Error Handling
- Supported Models
- Migration Guide (v4)
- Release Notes (v4)
Core Features
- Generate Content
- Chat Session
- Streaming
- Multimodal Features
- JSON Mode
- Function Calling
- Embeddings
- Context Caching
Gemini APIs
- Interactions API
- Multimodal Live API
- File Search
- Batch API
- Model Tuning
- Webhooks
- Agent APIs
- Image Generation
- Video Generation
- Speech Generation
- Files Client
- Model Client
Vertex AI
Integrations
Guides
Legacy