Overview

When a service entry is determined to execute remotely (IsLocal = false), or when IInvokeTemplate is used for direct cross-service calls, the framework uses DefaultRemoteExecutor to complete the full RPC call chain.

Build RemoteInvokeMessage
    │
    ▼
Compose Polly governance policies (Timeout + Retry + CircuitBreaker + Fallback)
    │
    ▼
Execute IRemoteCaller.InvokeAsync() within policy protection
    │  1. Find all healthy endpoints for the target service
    │  2. Select an endpoint by load balancing strategy
    │  3. Get or reuse a DotNetty TransportClient
    │  4. Run client filter pipeline
    │  5. Send RPC message, await response (UUID correlation)
    │
    ▼
Parse response, return result

RemoteInvokeMessage — RPC Message Structure

Each remote call constructs a RemoteInvokeMessage as the network transport payload:

The message is wrapped in a TransportMessage and encoded/decoded by DotNetty codec (JSON format).

Polly Policy Composition

Before executing the remote call, IInvokePolicyBuilder.Build() composes an async Polly policy chain. The composition order (inner-to-outer wrapping) determines execution order:

Outermost: Fallback policy (catches all failures, returns default/fallback result)
    └── CircuitBreaker policy (opens circuit after N consecutive failures)
        └── Retry policy (retries on transient failures)
            └── Timeout policy (cancels if call exceeds TimeoutMillSeconds)
                └── Actual RPC call

Timeout Policy

Uses Polly Optimistic timeout (relies on CancellationToken, doesn't forcibly abort the Task):

Policy.TimeoutAsync(
    TimeSpan.FromMilliseconds(governanceOptions.TimeoutMillSeconds),
    TimeoutStrategy.Optimistic);

Not created when TimeoutMillSeconds <= 0.

Retry Policy (Overflow Failover)

Triggers when the server returns OverflowMaxServerHandleException (concurrent requests exceed MaxConcurrentHandlingCount):

• RetryTimes retry attempts with RetryIntervalMillSeconds wait between each

Circuit Breaker Policy

Triggers on non-business exceptions:

• Opens after ExceptionsAllowedBeforeBreaking consecutive failures
• Stays open for BreakerSeconds seconds
• Transitions to Half-Open state and allows one test call
• Returns to Closed on success, reopens on failure

Fallback Policy

Executes when circuit is open or all retries exhausted:

[ServiceRoute]
public interface IOrderAppService
{
    [Fallback(typeof(OrderFallbackService))]
    Task<OrderDto> GetByIdAsync(long id);
}

// Fallback implementation (same interface)
public class OrderFallbackService : IOrderAppService
{
    public Task<OrderDto> GetByIdAsync(long id)
        => Task.FromResult(new OrderDto { Id = id, Status = "Unavailable" });
}

Load Balancing Strategies

Consistent Hash Key

[Governance(ShuntStrategy = ShuntStrategy.HashAlgorithm)]
Task<OrderDto> GetByIdAsync([HashKey] long id);

The [HashKey] attribute marks the parameter (or property) used as the hash input.

TransportClient — UUID Correlation

Each RPC call is assigned a UUID (Guid.NewGuid().ToString()), attached to TransportMessage.Id. The TransportClient uses a ConcurrentDictionary<string, TaskCompletionSource<TransportMessage>> to correlate responses:

// Send
var message = new TransportMessage(Guid.NewGuid().ToString(), invokeMessage);
_callbacks[message.Id] = new TaskCompletionSource<TransportMessage>();
await _channel.WriteAndFlushAsync(message);

// Receive (on DotNetty IO thread)
if (_callbacks.TryRemove(received.Id, out var tcs))
    tcs.SetResult(received);

// Await the correlated response
var response = await _callbacks[message.Id].Task.WaitAsync(cancellationToken);

This pattern supports multiple concurrent in-flight calls over a single DotNetty connection (connection pool).