Summary: About failure detection, reducing failure detection time, and relevant parameters.
OverviewFailure detection is the time it takes for the space and the client to detect that failure has occurred. Failure detection consists of two main phases:
One of two main failure scenarios might occur:
It takes GigaSpaces a few seconds to recover from process failure or a machine crash. In case of network cable disconnection, the client first has to detect that it has been disconnected from the machine running the space. Therefore, recovery time in this case is longer. For details on how network failure is detected and handled, see the Communication Protocol section. Reducing Failure Detection TimeConfiguring failure detection time can help you handle extreme failure scenarios more effectively. For example, in extreme cases of network disconnection, you might want the failover process to take 2-3 seconds. Here is a good combination for the space settings you may use to reduce the failover time - these should be used with a fast network: cluster-config.groups.group.fail-over-policy.active-election.yield-time=300 cluster-config.groups.group.fail-over-policy.active-election.fault-detector.invocation-delay=300 cluster-config.groups.group.fail-over-policy.active-election.fault-detector.retry-count=2 The following should be specified as system properties: -Dcom.gs.transport_protocol.lrmi.connect_timeout=3 -Dcom.gs.transport_protocol.lrmi.request_timeout=3 -Dcom.gs.jini.config.maxLeaseDuration=2000 -Dcom.gs.jini.config.roundTripTime=1000 By default, the maximum time it takes for a backup space to switch into a primary mode takes ~6-8 seconds. 100 [ms] + (yield-time * 7) + invocation-delay + (retry-count * retry-timeout) = failover time
Change the default settings only if you have a special need to reduce the failover duration. (~1 second). In this case:
Failure Detection ParametersSpace Side ParametersActive Election ParametersThe following parameters in the cluster schema active election block regard failure detection and recovery:
Client Side ParametersProxy ConnectivityThe Proxy Connectivity defines the settings in which the system checks the liveness of space members. Watchdog ParametersSee the com.gs.transport_protocol.lrmi.idle_connection_timeout and the com.gs.transport_protocol.lrmi.request_timeout watchdog properties at the Communication Protocol section. Service Grid ParametersThe Service Grid uses two complementary mechanisms for service detections – the Lookup Service and fault-detection handlers.
The fault-detection handlers check periodically if a service is alive, and in case of failure, how many times to retry and how often. The GSM and GSC fault-detection handler settings are controlled via the relevant properties. The PUFaultDetectionHandler is configurable using the SLA - member alive indicator. For logging information, it is advised to monitor service failure by setting the logging level to Level.FINE. # ServiceGrid FaultDetectionHandler logging com.gigaspaces.grid.gsc.GSCFaultDetectionHandler.level = INFO com.gigaspaces.grid.gsm.GSMFaultDetectionHandler.level = INFO org.openspaces.pu.container.servicegrid.PUFaultDetectionHandler.level = INFO Jini Lookup Service ParametersThe LeaseRenewalManager in the advanced-space.config file is also related to failure detection and recovery:
Lookup Service Unicast discovery parametersWhen a Jini Lookup Service fails and is brought back online, a client (such as a GSC, space or a client with a space proxy) needs to re-discover it. It uses Jini unicast discovery retrying to connect to the failed remote lookup service. The default unicast retry protocol provides a graduating approach, increasing the amount of time to wait before the next discovery attempts are made - upon each invocation, eventually reaching a maximum time interval over which discovery is re-tried. In this way, the network is not flooded with unicast discovery requests referencing a lookup service that may not be available for quite some time (if ever). The downside is that it may delay the discovery of services if these are not brought up quickly. A discovery can be delayed us much as 15 minutes. If you have two GSMs and one fails, but it will be brought back up only in the next hour, then it's discovery will take ~15 minutes after it has loaded. These settings can be configured - see How to Configure Unicast Discovery. |
![]() |
GigaSpaces.com - Legal Notice - 3rd Party Licenses - Site Map - API Docs - Forum - Downloads - Blog - White Papers - Contact Tech Writing - Gen. by Atlassian Confluence |