Summary: Setting communication protocol using exporters; GenericExporter configuration - ports, connections, and threads.


Overview

GigaSpaces provides a pluggable communication adaptor, LRMI (Light Remote Method Invocation). GigaSpaces LRMI is built on top of NIO communication protocol.


The GenericExporter, that implements the net.jini.export.Exporter interface, controls the whole network infrastructure, wraps the LRMI layer, and allows the application to use LRMI transparently.

All of GigaSpaces components that represent remote objects/services (for example: Transaction Manager, Lookup Service, Mahalo, GSM, GSC, space, and container) use the GenericExporter (that can be easily replaced with a custom, user-defined exporter).

GigaSpaces provides the GenericExporter as its default exporter, which works with LRMI. However, it is possible to use GigaSpaces with other exporters instead:

Changing the default exporter is done by editing the <GigaSpaces Root>\config\services\services.config file.

For example, to change the default exporter from GenericExporter to JrmpExporter:

Change:

{
...
    // uncomment the following next line to use Jrmp exporter instead of GenericExporter
    // defaultExporter = new net.jini.jrmp.JrmpExporter();
    defaultExporter = new com.gigaspaces.lrmi.GenericExporter( nioConfig );
}

to:

{
...
    // uncomment the following next line to use Jrmp exporter instead of GenericExporter
    defaultExporter = new net.jini.jrmp.JrmpExporter();
}

GenericExporter Configuration


GigaSpaces GenericExporter configuration is defined in the <GigaSpaces Root>\config\services\services.config file:

com.gigaspaces.transport {
    bindHost = BootUtil.getHostAddressFromProperty("java.rmi.server.hostname");
    //To define the port range change the following line to:
	//bindPort = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.bind-port", BootUtil.getPortInRange(10000, 20000) ) );
	bindPort = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.bind-port", "0") );

	minThreads  = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.min-threads", "1") );
	maxThreads  = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.max-threads", "128") );
	maxConnPool = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.max-conn-pool", "1024") );
	readSelectorThreads = Integer.parseInt( System.getProperty("com.gs.transport_protocol.lrmi.selector.threads", "1") );

	//LRMI Watchdog parameters
	watchdogRequestTimeout 	 = System.getProperty("com.gs.transport_protocol.lrmi.request_timeout", "30s");
	watchdogListeningTimeout = System.getProperty("com.gs.transport_protocol.lrmi.listening_timeout", "5m");
	watchdogIdleConnectionTimeout = System.getProperty("com.gs.transport_protocol.lrmi.idle_connection_timeout", "15m");

	//LRMI ThreadPool idle timeout default is 5 min.
	threadPoolIdleTimeout = Long.parseLong(System.getProperty("com.gs.transport_protocol.lrmi.threadpool.idle_timeout", "300000"));

	//LRMI timout on the Socket connect. Default connect timeout in seconds is 0 - infinite timeout
	socketConnectTimeout = System.getProperty("com.gs.transport_protocol.lrmi.connect_timeout", "0");

    nioConfig = new com.gigaspaces.config.lrmi.nio.NIOConfiguration( minThreads, /* min executors threads */
                                                                     maxThreads, /* max executors threads */
                                                                     maxConnPool, /* maxConnPool */
                                                                     bindHost, /*  if null resolves to the localhost IP address */
                                                                     bindPort, /* if 0 any next free port will be used for incoming client requests.*/
                                                                     readSelectorThreads,
                                                                     watchdogRequestTimeout,
                                                                     watchdogListeningTimeout,
                                                                     watchdogIdleConnectionTimeout,
                                                                     threadPoolIdleTimeout,
                                                                     socketConnectTimeout
																	);

    // uncomment the following next line to use Jrmp (RMI) exporter instead of GenericExporter (NIO based)
    // defaultExporter = new net.jini.jrmp.JrmpExporter();
    defaultExporter = new com.gigaspaces.lrmi.GenericExporter( nioConfig );
}


As discussed above, LRMI is built on top of NIO communication protocol. In this sense, the JVM is built on top of LRMI, and holds all of GigaSpaces remote services. Therefore, the GenericExporter is configured in the JVM level, and not in the level of each specific remote service.

This means that all remote services run by GigaSpaces (Transaction Manager, Lookup Service, Mahalo, GSM, GSC, space, container, etc.), even if they are running on different machines and/or different JVMs, receive the same values defined in services.config.

You might want to define different options for different remote services. This can be done using system properties. For example, if you are working with NFS (Network File System), where all machines in the network are using one GigaSpaces installation installed on a central machine, the client and server use the same port defined in the GenericExporter bindPort argument. To override this, you can define the java.rmi.server.hostname system property as the bindPort value in the central machine, thus enabling the client and server to define their own port.

You don't necessarily have to define a certain system property in all remote services - if the system property is not defined, the default value is used.

Options

The GenericExporter configuration options are detailed below. Remember that you can override all of them using system properties, as described in the section above.

For the available system properties used for overrides, see Javadoc .


Option Description Default value
MaxConnPool The maximum amount of connections to the space server remote services that can work simultaneously in a client connection pool. Starts with 1 connection. Defined per each remote service (by default, each remote service has 1024 maximum connections). 1024
minThreads GigaSpaces maintains a thread pool in the client and server side, that manages incoming remote requests. The thread pool size is increased each time with one additional thread and shrinks when existing threads are not used for 5 minutes. This parameter specifies the minimum size of this thread pool. 1
maxThreads This parameter specifies the maximum size of the thread pool.
Make sure the maximum size of the thread pool accommodates the maximum number of concurrent requests to the space. The client uses this pool for server requests into the client side - i.e. notify callbacks. When the pool is exhausted and all threads are consumed to process incoming requests, additional requests are blocked until existing requested processing are complete.
128
bindPort Server port used for incoming client requests, or notifications from server to client. The server port is set by default to 0, which means next free port. This means that whenever GigaSpaces is launched, it allocates one of the available ports. Define a specific port value to enforce a specific port on the space server or client side (see how to override this option above). Alternatively, you can define a range of ports, see below. 0
bindHost Binds the GigaSpaces Server on a specified network interface, according to the java.rmi.server.hostname property (see below). The system takes the host defined in the java.rmi.server.hostname property. If java.rmi.server.hostname is null, the system sets the localhost IP address.

Alternatively, you can define the host directly in the services.config file (in case you do not have permissions to do this using the system property). For example:
com.gigaspaces.transport {
     host="192.168.33.3";
...
}

host

If you are using the Slow Consumer mechanism, you need to define this option in your space schema, as well as in the services.config file. Refer to the Slow Consumer section for more details.

Defining Socket Port Range per JVM

It is possible to define a range of ports using the bindPort argument, meaning that every JVM acquires a bind port from a defined port range. This functionality is useful when having multiple JVMs on the same machine (clients or servers), and simplifies firewall setup.

The following method in BootUtils is a utility method which helps to retrieve a free port from a port range:

BootUtil.getPortInRange(10000, 20000)

To define the port range change the following line in the com.gigaspaces.transport block:

bindPort = Integer.parseInt( System.getProperty("com.gigaspaces.transport.bind-port", "0") );

to:

bindPort = Integer.parseInt( System.getProperty("com.gigaspaces.transport.bind-port", BootUtil.getPortInRange(10000, 20000) ) );

Instead of 1000, 2000, specify the desired port range (in this case, the port range is between 1000 and 2000).

When the JVM is started, the system checks if the com.gigaspaces.transport.bind-port system property is defined. If not, the defined port range is used.

Even if you define a port range, the returned port might still be acquired by another process - BootUtil.getPortInRange can't lock the free port. To prevent this, if you are running multiple JVMs on the same machine, run them one after the other, and not all at once.

Configuration with Multi-Homed Machines

When working with a multi-homed machine (a computer with multiple network cards), use the following system property to bind the GigaSpaces Server on a specified network interface:

-Djava.rmi.server.hostname=<hostname or IP address>

The value of this property represents the host name string that must be associated with the network interface. The default value of this property is the IP address of the local host, in "dotted-quad" format.

Troubleshooting

You can troubleshoot the space activity using LRMI logging. You can turn on LRMI logging in the following modes:

Max buffer size

The NIO internal cache (a DirectByteBuffer) might cause an OutOfMemoryError due to direct memory exhaustion.
To avoid such a scenario, the LRMI layer breaks the outgoing buffer into a small buffer.
By doing so, the NIO internal cache is kept small, and may not cause any error.
The size of these chunks can be determined by the following system property:

-Dcom.gs.lrmi.maxBufferSize (default=640kb)

Offline mode – via gs_logging.properties

  1. Open /config/gs_logging.properties
  2. Locate the following line:
    com.gigaspaces.lrmi.level = INFO
  3. Set debug logger granularity level:
    • ALL – all debug messages available for LRMI.
    • FINE – configuration, watchdog, client connections (connect/disconnect).
    • FINER – content and execution state of remote method invocation on remote object.
    • SEVER – caught exceptions by LRMI on server and client side.
  4. Save and close the gs_logging.properties file.
  5. Start a space: /bin/gsInstance.

LRMI communication transport protocol debug messages are displayed.

During Runtime – using JMX

  1. Start gsInstance using the following Java system properties:
    -Dcom.sun.management.jmxremote.port=5001
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false
  2. Start jconsole – copy the JMX URL to the JConsole.
  3. Click Connect.

  4. Set com.gigaspaces.lrmi with a desired log level.
  5. Click setLoggerLevel.
  6. A new log level is defined.

When LRMI logging is turned on, the space displays the following when started:

To test LRMI logging, you can run the space ping utility using the following command:

/bin/gs.bat/space ping mySpace_container mySpace -r -i 1

The space displays the following:

The client displays the following:

Troubleshooting Tips

If you encounter the following exception:

CONFIG [com.gigaspaces.grid.gsc]: initialServiceLoadDelay=5000
Exception in thread "LRMI Connection--pool-1-thread-1" java.lang.NullPointerException
at sun.nio.ch.PollSelectorImpl.wakeup(PollSelectorImpl.java:84)
at com.gigaspaces.lrmi.nio.SelectorThread.registerKey(SelectorThread.java:250)
at com.gigaspaces.lrmi.nio.ChannelEntry.workerFinishedReading(ChannelEntry.java:131)
at com.gigaspaces.lrmi.nio.Pivot$Worker.dispatch(Pivot.java:111)
at com.j_spaces.kernel.WorkingGroup$TaskWrapper.run(WorkingGroup.java:62)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

note that this is a bug in Sun. For more details, see http://bugs.sun.com.

If your client application loses its connection to the server, you can follow a simple procedure to check if the server erased any of your notify templates in the interim. For each notify template, write an Entry to the space that matches the template and see if you receive a notification. If you do not receive a notification, this means that while you were disconnected, new Entries matching the notify template entered the space (you can try to find them – depending on their lease time, they may still exist). As a result, your notify template was erased.


GigaSpaces.com - Legal Notice - 3rd Party Licenses - Site Map - API Docs - Forum - Downloads - Blog - White Papers - Contact Tech Writing - Gen. by Atlassian Confluence