Reconnect and Retry: Exponential Backoff, Jitter and Retry-After | eSeGeCe Blog

Reconnect and Retry: Exponential Backoff, Jitter and Retry-After

· Components
Reconnect and Retry: Exponential Backoff, Jitter and Retry-After

Restart a WebSocket server that has ten thousand clients attached and something unpleasant happens. Every client notices the disconnection at roughly the same moment, every client waits the same fixed number of seconds, and every client dials back on the same tick. The server comes up and is immediately hit by ten thousand simultaneous TLS handshakes. It falls over, or it throttles, and the clients retry again in lockstep. This is the thundering herd, and a fixed reconnect interval guarantees it.

sgcWebSockets 2026.7 addresses the two places where clients hammer a server that is already in trouble. The WatchDog can now reconnect with exponential backoff plus jitter instead of a fixed interval, and the HTTP client can retry a failed request by itself, honouring the server's Retry-After hint, using the same backoff maths. Both are opt-in and off by default, so existing applications behave exactly as they did before until you enable them.

The problem with a fixed reconnect interval

The WatchDog has always been the standard way to keep a client connected. You set Enabled, you set an Interval in seconds, and when the connection drops the component retries every Interval seconds until it is back.

That is fine for one client on a flaky Wi-Fi link. It is the wrong behaviour for a fleet. With a fixed interval every client in the fleet is a metronome ticking at the same rate, and a server restart synchronises all of them: they all disconnected at the same instant, so they all reconnect at the same instant, forever. Worse, the interval does not adapt. If the server is down for ten minutes, a client with Interval = 1 makes six hundred pointless connection attempts, and so does everybody else.

What you want is the opposite. Try again quickly the first time, because most disconnections are transient and a fast reconnect is invisible to the user. Then back off, so a server that is genuinely down is not being pounded. And spread the clients out, so they do not all arrive together.

Exponential backoff on the WatchDog

The WatchDog options object gained four properties. The type lives in sgcTCP_Classes:

type
  TsgcWatchDogBackoff = (wdbFixed, wdbExponential);

On any WebSocket component the options are reached as Client.WatchDog, and the backoff settings are these:

uses
  sgcWebSocket, sgcTCP_Classes;

begin
  sgcWebSocketClient1.WatchDog.Enabled           := True;
  sgcWebSocketClient1.WatchDog.Attempts          := 0;     // 0 = keep trying forever
  sgcWebSocketClient1.WatchDog.Interval          := 1;     // seconds, the first delay
  sgcWebSocketClient1.WatchDog.Backoff           := wdbExponential;
  sgcWebSocketClient1.WatchDog.BackoffMultiplier := 2.0;   // double it every attempt
  sgcWebSocketClient1.WatchDog.MaxInterval       := 60;    // seconds, the ceiling
  sgcWebSocketClient1.WatchDog.Jitter            := 0.2;   // up to 20% random spread
end;

The maths is deliberately boring. The first reconnect attempt waits Interval. Each subsequent attempt multiplies the previous delay by BackoffMultiplier, and the result is clamped to MaxInterval. With the values above the delays go 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s, 60s and so on. The counter resets as soon as the client is connected again, so a brief network blip costs you one second, not a minute.

Interval and MaxInterval are both expressed in seconds, as the WatchDog always has been. A MaxInterval of 0 means no ceiling, which is the default. If you enable exponential backoff, set a ceiling, otherwise the delay keeps doubling until it is measured in hours.

The important detail for existing users: Backoff defaults to wdbFixed. If you upgrade and change nothing, your client reconnects on the same fixed interval it always did. The delay computation is only used when you switch to wdbExponential or set a Jitter, and it is the same helper the HTTP retry uses, sgcNextBackoffMs in sgcBase_Helpers.

Jitter, and why it matters at scale

Exponential backoff alone does not break the herd. It only makes the herd arrive less often. Ten thousand clients that all wait 1s, then 2s, then 4s are still ten thousand clients arriving together, just in bigger and bigger waves.

Jitter is what actually spreads them out. It is a fraction of the computed delay, applied randomly per client and per attempt. With Jitter = 0.2 a nominal 8 second delay becomes a random value in a 20% band around 8 seconds, so client A wakes at 7.1s, client B at 8.6s, client C at 8.0s. The fleet arrives as a smear instead of a spike, and the server's accept queue can keep up.

// A conservative fleet profile: fast first retry, hard ceiling, generous spread.
sgcWebSocketClient1.WatchDog.Enabled           := True;
sgcWebSocketClient1.WatchDog.Interval          := 2;
sgcWebSocketClient1.WatchDog.Backoff           := wdbExponential;
sgcWebSocketClient1.WatchDog.BackoffMultiplier := 1.5;   // gentler than doubling
sgcWebSocketClient1.WatchDog.MaxInterval       := 120;
sgcWebSocketClient1.WatchDog.Jitter            := 0.5;   // spread over half the delay

Jitter is orthogonal to the backoff mode. You can leave Backoff on wdbFixed and set only Jitter, which gives you the classic fixed interval with the clients desynchronised. That is often enough on its own, and it is the smallest possible change to an existing application.

Retrying an HTTP request automatically

The second half of the feature is on the HTTP side. A single HTTP request that fails with a 503 because the server is redeploying, or a 429 because you went over a rate limit, is not really an error. It is a request that should be sent again in a moment. Writing that loop by hand is tedious and everybody writes it slightly wrong.

TsgcIdHTTP now has a RetryOptions object, declared in sgcHTTP_Client as TsgcIdHTTPRetry_Options. Enable it and the request retries itself:

uses
  sgcHTTP_Client;

var
  oHTTP: TsgcIdHTTP;
begin
  oHTTP := TsgcIdHTTP.Create(nil);
  try
    oHTTP.RetryOptions.Enabled         := True;    // default False
    oHTTP.RetryOptions.MaxRetries      := 3;
    oHTTP.RetryOptions.InitialInterval := 500;     // milliseconds
    oHTTP.RetryOptions.MaxInterval     := 30000;   // milliseconds
    oHTTP.RetryOptions.Multiplier      := 2.0;
    oHTTP.RetryOptions.Jitter          := 0.2;
    oHTTP.RetryOptions.StatusCodes     := '429,502,503,504';

    ShowMessage(oHTTP.Get('https://api.example.com/v1/orders'));
  finally
    oHTTP.Free;
  end;
end;

The intervals here are in milliseconds, not seconds, because HTTP retries live on a much shorter timescale than a reconnect. The delays follow the same curve as the WatchDog: 500ms, 1s, 2s, 4s, capped at MaxInterval and spread by Jitter.

StatusCodes is the list of HTTP status codes that count as retryable, as a comma separated string. The default is 429,502,503,504, which are the codes that mean "not now, try again" rather than "your request is wrong". A 400 or a 401 is never retried, because sending the same broken request four more times helps nobody. Connection level failures such as a refused socket or a read timeout are retried too.

The same object is on TsgcHTTPAPI_client in sgcHTTP_API, which is the base class of every ready made REST client in the library. So the retry policy is available on all of them without any per client plumbing:

// Any API client that descends from TsgcHTTPAPI_client
sgcHTTPAPI_Client1.RetryOptions.Enabled     := True;
sgcHTTPAPI_Client1.RetryOptions.MaxRetries  := 5;
sgcHTTPAPI_Client1.RetryOptions.StatusCodes := '429,500,502,503,504';

Retry-After: let the server set the pace

Computing your own backoff is a guess. When a server returns 429 or 503 it often tells you exactly how long to wait, in the Retry-After response header, either as a number of seconds or as an HTTP date. That number is not a suggestion. It is the server telling you when it will accept you again, and a client that ignores it and retries on its own schedule is simply going to be rejected again.

HonorRetryAfter is True by default. When the response carries a Retry-After header, that value wins over the computed backoff, and the client waits as long as the server asked:

oHTTP.RetryOptions.Enabled         := True;
oHTTP.RetryOptions.HonorRetryAfter := True;   // default True
oHTTP.RetryOptions.InitialInterval := 500;
oHTTP.RetryOptions.MaxRetries      := 3;

// Server replies 429 with "Retry-After: 12".
// The computed backoff would have been 500ms, the client waits 12 seconds instead.

Both forms of the header are parsed, the delta seconds form and the HTTP date form. This is exactly what OpenAI and Anthropic ask their clients to do when they return a 429, and it is the difference between a client that recovers from a rate limit and a client that stays rate limited.

Retries on the AI clients

The AI clients are where retries earn their keep, because rate limits there are routine rather than exceptional. Each of them exposes its own retry options object on its *Options property, so the settings are visible in the Object Inspector alongside the model and the API key.

On TsgcHTTP_API_OpenAI the path is OpenAIOptions.RetryOptions, of class TsgcHTTPOpenAIRetry_Options:

uses
  sgcHTTP_API_OpenAI;

begin
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.Enabled         := True;
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.Retries         := 3;
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.Wait            := 10000;  // ms, first delay
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.MaxInterval     := 30000;  // ms, ceiling
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.Multiplier      := 2.0;
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.Jitter          := 0.2;
  sgcHTTPOpenAI1.OpenAIOptions.RetryOptions.HonorRetryAfter := True;
end;

Anthropic is the same shape under AnthropicOptions.RetryOptions, and Gemini under GeminiOptions.RetryOptions. The same pattern exists on the Grok, Mistral, DeepSeek and Ollama clients, and on the Stripe and Paddle clients, where a retried payment call is a very different kind of important.

// Anthropic returns 429 and 529 under load, both are treated as retryable.
sgcHTTPAnthropic1.AnthropicOptions.RetryOptions.Enabled         := True;
sgcHTTPAnthropic1.AnthropicOptions.RetryOptions.Retries         := 3;
sgcHTTPAnthropic1.AnthropicOptions.RetryOptions.HonorRetryAfter := True;

Note the property names on the AI objects are Retries and Wait rather than MaxRetries and InitialInterval, to stay consistent with the naming already used elsewhere on those components. The behaviour is identical, and internally they configure the same TsgcIdHTTPRetry_Options on the underlying HTTP client.

Knowing why it failed: Connect and LastError

A retry loop is only as good as its error handling. Before you can decide whether a failure is worth retrying, you have to know what the failure was, and until now the only way to find out on a WebSocket client was to wire an OnError handler and stash the message in a field somewhere.

2026.7 adds a Connect overload that hands you the error directly, in sgcWebSocket_Client_Base:

function Connect(const aTimeout: Integer = 10000): Boolean; overload;
function Connect(out AError: string; const aTimeout: Integer = 10000): Boolean; overload;

So a manual reconnect loop can inspect the reason and decide for itself:

var
  vError: string;
begin
  if not sgcWebSocketClient1.Connect(vError, 5000) then
  begin
    // A DNS failure or a refused socket is worth retrying.
    // A rejected certificate or a 401 on the handshake is not.
    Memo1.Lines.Add('connect failed: ' + vError);
    Exit;
  end;
end;

The last error is also kept on the component itself. TsgcWSComponent_Base in sgcWebSocket_Classes publishes a read only LastError: string property and a ClearLastError method, so you can read the reason after the fact, and reset it before an attempt you want to measure:

sgcWebSocketClient1.ClearLastError;
if not sgcWebSocketClient1.Connect(5000) then
  ShowMessage('connect failed: ' + sgcWebSocketClient1.LastError);

None of this replaces OnError, which still fires as it always has. It just means a simple, synchronous piece of code no longer has to set up an event handler to answer a simple question.

Availability

WatchDog backoff, jitter, HTTP retries and the new Connect overload are available from sgcWebSockets 2026.7 across Delphi 7 through 13 and C++Builder, on Win32/Win64, Linux64, macOS, Android and iOS. Everything described here is off by default. Backoff is wdbFixed, Jitter is 0 and RetryOptions.Enabled is False, so an existing application upgrades with no behavioural change at all and you opt in where it makes sense.

Customers with an active subscription can download the new build from the customer area. Trial users can grab the updated installer at esegece.com/products/websockets/download.

Questions, feedback or help picking a backoff profile for your fleet? Get in touch, you will get a reply from the people who wrote the code.