A Realtime Voice Agent for Delphi and .NET

5 October 2026 · Components

Typing a question and reading the answer is how most applications talk to a language model today. The realtime APIs change that. OpenAI Realtime and Gemini Live take the user's voice as it is spoken and answer with a voice of their own, fast enough to hold a conversation. Getting there by hand means streaming audio in both directions, deciding when the user has finished, stopping the agent when it is interrupted and running tools in the middle of it all.

sgcWebSockets 2026.10 adds TsgcAIVoiceAgent, one component that does all of that. It listens, detects when the user has finished a turn, talks back and calls your tools. It is available for Delphi 7 to 13 and for .NET.

TsgcAIVoiceAgent in action: turn detection, barge-in, live transcripts and tool calls. Also on YouTube.

Quick Start

Choose the provider, set the API key, a voice and the instructions, handle OnTranscript and call Start. That is a working voice assistant.

oVoice := TsgcAIVoiceAgent.Create(nil);
oVoice.Provider := vapOpenAI;
oVoice.OpenAIOptions.ApiKey := 'API_KEY';
oVoice.OpenAIOptions.Voice := 'alloy';
oVoice.Instructions := 'You are a friendly assistant. Answer briefly.';
oVoice.OnTranscript := oVoiceTranscript;
oVoice.OnError := oVoiceError;
oVoice.Start;

OnTranscript delivers live transcripts of both sides of the conversation, the user and the agent, so the screen can follow what is being said:

procedure TForm1.oVoiceTranscript(Sender: TObject;
  aRole: TsgcAIVoiceAgentRole; const aText: string; aFinal: Boolean);
begin
  if aFinal then
  begin
    if aRole = varUser then
      Memo1.Lines.Add('You: ' + aText)
    else
      Memo1.Lines.Add('Agent: ' + aText);
  end;
end;

OpenAI Realtime or Gemini Live, WebSocket or WebRTC

Two providers are supported, OpenAI Realtime and Gemini Live. Switching is one property and the matching options:

oVoice.Provider := vapGemini;
oVoice.GeminiOptions.ApiKey := 'GEMINI_API_KEY';
oVoice.Start;

The session runs over WebSocket. With OpenAI it can also run over WebRTC:

oVoice.Provider := vapOpenAI;
oVoice.Transport := vatWebRTC;
oVoice.OpenAIOptions.ApiKey := 'API_KEY';
oVoice.Start;

Built-in Audio, or Your Own

On Windows, Android and iOS the microphone and the speaker are built in, so the quick start above is all you need. On any other platform, or when the audio comes from somewhere else, turn the device off. Your application feeds audio with SendAudio and receives the answer in OnAudioOutput.

oVoice.Audio.UseDevice := False;
oVoice.Audio.Format := vafPCM16;
oVoice.Audio.SampleRate := 16000;
oVoice.Start;

// ... 20 ms of PCM16 mono at 16 kHz = 640 bytes
SetLength(vChunk, 640);
FillChar(vChunk[0], Length(vChunk), 0);
oVoice.SendAudio(vChunk);

Answering the Phone

Telephony speaks G.711, and so does the voice agent. Set the audio format to mu-law or A-law at 8 kHz and the bytes of a call go straight in and straight out, with no conversion in your code.

oVoice.Provider := vapOpenAI;
oVoice.OpenAIOptions.ApiKey := 'API_KEY';
oVoice.Instructions := 'You answer the phone of a restaurant and take bookings.';
oVoice.Audio.UseDevice := False;
oVoice.Audio.Format := vafG711ULaw;
oVoice.OnAudioOutput := oVoicePhoneAudioOutput;
oVoice.Start;

procedure TForm1.DoPhoneAudioReceived(const aData: TBytes);
begin
  // ... G.711 mu-law, 8 kHz, as it arrives from the RTP stream of the call
  oVoice.SendAudio(aData);
end;

procedure TForm1.oVoicePhoneAudioOutput(Sender: TObject;
  const aData: TBytes);
begin
  // ... already G.711 mu-law at 8 kHz: send it back to the caller
  SendToPhone(aData);
end;

Turn Detection

Turn detection decides when the user has finished speaking and the agent should answer. There are four modes.

Server VAD lets the provider detect the end of the turn with voice activity detection.
Semantic VAD listens to what is said, not only to silence, so a pause in the middle of a sentence does not cut the user off.
Client VAD detects the turn on the client.
Manual is push to talk: your application decides where the turn ends.

oVoice.TurnDetection.Mode := vtdServerVAD;
oVoice.TurnDetection.Threshold := 0.6;
oVoice.TurnDetection.PrefixPaddingMs := 300;
oVoice.TurnDetection.SilenceDurationMs := 700;

oVoice.TurnDetection.Mode := vtdSemanticVAD;
oVoice.TurnDetection.Eagerness := 'low';

With push to talk, commit the audio and ask for a response when the user releases the button:

procedure TForm1.btnTalkMouseUp(Sender: TObject);
begin
  // ... TurnDetection.Mode is vtdManual
  oVoice.CommitAudio;
  oVoice.CreateResponse;
end;

Barge-in

People interrupt, and a good voice agent lets them. With barge-in, when the user starts talking the agent stops speaking and the answer is cancelled. The answer is also truncated to what the user actually heard, so the conversation history does not claim the agent said words that were never played.

oVoice.BargeIn := True;
oVoice.OnTurnStarted := oVoiceTurnStarted;
oVoice.OnTurnEnded := oVoiceTurnEnded;
oVoice.OnBargeIn := oVoiceBargeIn;

procedure TForm1.oVoiceBargeIn(Sender: TObject);
begin
  Memo1.Lines.Add('The user interrupted the agent.');
end;

Tools

A voice agent becomes useful when it can do things. Declare a tool with a name, a description and a JSON schema for its parameters, and answer it in OnToolCall.

procedure TForm1.StartWeatherAssistant;
var
  oTool: TsgcAIVoiceAgentTool;
begin
  oVoice.Provider := vapOpenAI;
  oVoice.OpenAIOptions.ApiKey := 'API_KEY';
  oVoice.Instructions := 'You are a weather assistant. ' +
    'Call get_weather to know the weather of a city before answering.';

  oVoice.Tools.Clear;
  oTool := oVoice.Tools.Add;
  oTool.Name := 'get_weather';
  oTool.Description := 'Returns the current weather of a city.';
  oTool.Parameters := '{"type":"object","properties":' +
    '{"city":{"type":"string","description":"Name of the city"}},' +
    '"required":["city"]}';

  oVoice.OnToolCall := oVoiceWeatherToolCall;
  oVoice.OnTranscript := oVoiceTranscript;
  oVoice.Start;
end;

procedure TForm1.oVoiceWeatherToolCall(Sender: TObject;
  const aCallId, aName, aArguments: string; var aResult: string;
  var aHandled: Boolean);
var
  oJSON: TsgcJSON;
  vCity: string;
begin
  if aName <> 'get_weather' then
    Exit; // ... not mine: the MCP client, if any, gets the call

  oJSON := TsgcJSON.Create(nil);
  Try
    oJSON.Read(aArguments);
    vCity := '';
    if Assigned(oJSON.Node['city']) then
      vCity := oJSON.Node['city'].Value;
  Finally
    oJSON.Free;
  End;

  // ... replace with a call to your weather service
  aResult := '{"city":"' + vCity + '","temperature":21,"sky":"clear"}';
  aHandled := True;
end;

Tools can also come from an MCP server. Assign an MCP client and its tools are offered to the agent automatically:

oMCP := TsgcWSAPIClient_MCP.Create(nil);
oMCP.MCPOptions.HttpOptions.URL := 'https://localhost:5001/mcp';

oVoice.MCPClient := oMCP;
oVoice.Start;

The Demo: a Voice Shop Assistant

The Delphi demo in Demos\15.AI\02.Applications\09.VoiceAgent is a shop assistant that you talk to. It works over a product catalog held in a TDataSet and gives the agent four tools: find_products to search the catalog, get_product to read one product, update_stock to change the stock, and count_low_stock to list the products that need restocking. Ask it what is running low, then tell it to restock a product, and watch the grid change while it answers.

Availability

TsgcAIVoiceAgent arrives in sgcWebSockets 2026.10, for Delphi 7 to 13 and for .NET, in the Enterprise edition. What else is new in the release is in the sgcWebSockets 2026.10 post, and the rest of the AI components are on the AI product page.