Ice may automatically retry a proxy invocation after a failure. This is a powerful feature that, when used in the proper situations, can significantly improve the robustness of your application without any additional programming effort. The retry facility is governed by one overriding principle: always respect
at-most-once semantics. As explained on
page 18, at-most-once semantics dictate that the Ice run time in the client must never retry a failed proxy invocation unless Ice guarantees that the server has not already received the request, or unless the application declares that it is safe for Ice to violate at-most-once semantics for the request.
interface Account {
long withdraw(long amount);
};
The withdraw operation removes funds from an account. If an invocation of
withdraw fails, automatically retrying the request introduces the risk of a duplicate withdrawal unless Ice is absolutely sure that the server has not already executed the request.
Ice also never retries exceptions that derive from RequestFailedException because they indicate a permanent failure. One such subclass is
OperationNotExistException, whose occurrence signals a serious problem in the application. For instance, it might mean that the client and server are using incompatible Slice definitions, or that the client is trying to invoke operations on the wrong object. The exception to this rule is
ObjectNotExistException, which Ice does consider to be worthy of retry if the proxy in question is indirect (see
page 15) because it gives an application the ability to transparently migrate an Ice object.
In addition to user exceptions and subclasses of RequestFailedException, a server can also return an instance of
UnknownException,
UnknownLocalException, or
UnknownUserException to indicate that it encountered an unexpected exception while dispatching the request. These exceptions
are eligible for retry.
If the error is still a candidate for retry, Ice needs to know whether the server has received the request. Naturally, the Ice run time in the client cannot possibly know that information until the server confirms it by sending a reply. However, to be conservative Ice assumes that the server has received the request as soon as Ice has written the
entire protocol message to the client’s local transport buffers. If the error occurred before Ice managed to write the complete message, retrying the request would not violate at-most-once semantics.
The Ice run time in the server also has the ability to notify the client that a request was not dispatched and therefore that it is safe for the Ice run time in the client to retry the request without violating at-most-once semantics. For example, this situation can occur when the server is shutting down while there are pending requests that have yet to be executed. Sending this notification allows a client to transparently fail over to another server.
If Ice determines that an invocation cannot be retried, it raises the exception that caused the request failure to the application. On the other hand, if Ice does retry the invocation and the subsequent retries also fail, Ice raises the
last exception to the application. For example, if the first attempt fails with
ConnectionRefusedException and the retry fails with
ConnectTimeoutException, the invocation raises
ConnectTimeoutException to the application.
Annotating a Slice operation with the idempotent keyword notifies Ice that it can safely violate at-most-once semantics:
interface Account {
long withdraw(long amount);
idempotent long getBalance();
};
Although withdraw clearly requires the stricter treatment, there is no harm in automatically retrying the
getBalance operation even if the server executes the same request more than once.
In general, "read-only" operations are good candidates for the idempotent keyword whereas many mutating operations are not. However, the risk of duplicate requests is acceptable even for some kinds of mutating operations:
interface Account {
long withdraw(long amount);
idempotent long getBalance();
idempotent void changeAddress(string newAddress);
};
Here we have marked changeAddress as idempotent because executing the request twice has the same effect as executing it only once.
The benefit of the idempotent keyword and the associated relaxation of retry semantics is that an invocation that otherwise might have raised an exception has at least one more chance to succeed. Furthermore, the application does not need to initiate the retry, and in fact the retry activities are completely transparent: if a subsequent retry succeeds, the application receives its results as if nothing went wrong. The invocation only raises an exception once Ice has reached its configured retry limits.
The Ice.RetryIntervals property configures the retry behavior for a communicator and affects invocations on every proxy created by that communicator. (Retry behavior cannot be configured on a per-proxy basis.) The value of this property consists of a series of integers separated by whitespace. The number of integers determines how many retry attempts Ice makes, and the value of each entry represents a delay in milliseconds. If this property is not defined, the default behavior is to retry once immediately after the first failure, which is equivalent to the following property definition:
With this setting, Ice retries immediately as in the default case. If the first retry attempt also fails, Ice waits 100 milliseconds before trying again, then 500 milliseconds, and finally tries one more time after waiting one second.
In some situations you may need to disable retries completely. For example, an application might implement its own retry logic and therefore require immediate notification when a failure occurs. Clients that establish a session with a Glacier2 router also need to disable retries (see
Section 43.3.4). To prevent automatic retries, use a value of
-1:
When retry tracing is enabled, Ice logs a message each time it attempts a retry; the log message includes a description of the exception that prompted the retry. Ice also logs a message when it reaches the retry limit.
If a proxy invocation fails due to a timeout, the application must be prepared for Ice to raise a
TimeoutException (see
Section 32.13). However, a developer that is testing timeouts in an application may be initially confused to discover that it is taking twice as long as expected for Ice to raise the
TimeoutException. Automatic retries are usually the reason for this situation.
For example, suppose that a proxy is configured with a one-second timeout and automatic retries are enabled with the default setting. If an invocation on that proxy fails due to a timeout and Ice determines that the invocation is eligible for retry (using the criteria in
Section 32.23.1), Ice immediately tries the invocation again and waits for another timeout period to expire before finally raising
TimeoutException. From the application’s perspective, the invocation fails after approximately two seconds.
where t is the timeout value,
N is the number of retry intervals, and
D is the sum of the retry intervals (the total delay between retries). For example, consider our example from
Section 32.23.3 again:
The behavior of automatic retries is intimately tied to the presence (and absence) of connections. This section describes the errors that cause Ice to close connections, and provides more details about how connections influence retries.
Ice automatically closes a connection in response to certain fatal error conditions. Of these, the one that is the most likely to affect Ice applications is a timeout (see
Section 32.23.4). Other errors that prompt Ice to close a connection include the following:
When Ice closes a connection in response to one of these errors, all other outstanding requests on the same connection also fail and may be retried if eligible.
One factor that influences retry behavior is the status of the connection on which the failed request was attempted. If the failure caused Ice to close the connection (as discussed in the previous section), or if the request failed because Ice could not establish a connection, Ice must try to obtain another connection before it can retry the request.
Section 37.3 describes the semantics of connection establishment.
It is also important to understand that Ice may not retry the invocation on the original endpoint
even if the connection that was used for the initial request remains open. The retry behavior in this case depends on several criteria:
