Coherence: POF Exceptions

If you are trying to use POF for object serialization as recommended for better performance and less RAM utilization with Coherence, there are 2 ways to implement it:

1. Implement PortableObject interface for all objects that will be stored in the Coherence Grid - Need to modify the object class and implement readExternal and writeExternal methods.
2. Externalize the serialization logic by creating classes that implement PofSerializer interface and logic for serialization and de-serialization of your objects is implemented in this class using the serialize/deserialize methods. Later, you can map this class as the serializer for your object in pof-config.xml

More details on both the above ways can be found here.

If you are getting an error while using the PortableObject interface as below:

Caused by: java.io.EOFException: user type POF stream terminated at com.tangosol.io.pof.PofBufferWriter$UserTypeWriter.writeUserTypeInfo(PofBufferWriter.java:2621) at com.tangosol.io.pof.PofBufferWriter$UserTypeWriter.writeRemainder(PofBufferWriter.java:2497) at com.tangosol.io.pof.PortableObjectSerializer.serialize(PortableObjectSerializer.java:96) at com.tangosol.io.pof.PofBufferWriter.writeUserType(PofBufferWriter.java:1667)
... 13 more

Remember to remove the readRemainder() and writeRemainder() from the readExternal/writeExternal methods as they are used with POFSerializer interface.

Coherence Random Questions?

What will happen if the eviction policy dictates that we evict an entry from the backing map that is still in the write-behind queue (and therefore has not been flushed to the database)?

In this situation, the read-write backing map will synchronously invoke the store operation on any entries about to be evicted. The implication here is that the client thread performing the put operation will be blocked while evicted entries are flushed to the database. It is an unfortunate side effect for the client thread, as its operation will experience a higher than expected latency, but it acts as a necessary throttle to avoid losing data. This edge condition highlights the necessity to configure a worker thread pool even for caches that are strictly performing write behind in order to prevent this flush from occurring on the service thread. It is important to keep in mind that the store operation will not always necessarily be performed by the write-behind thread. Note that this can also occur with caches that have expiry configured. The likelihood of this occurring will decrease if there is a large difference between expiry time and write-behind time.

What are service threads and how is the PUT operation managed in the Coherence Grid?

Each clustered service in Coherence is represented by a service thread at each JVM participating in the cluster. This thread is responsible for communicating with other nodes and providing the functionality exposed via the NamedCache API along with system level functionality such as life-cycle, distribution and fail-over. As a rule, all communications between the service threads are done in an asynchronous (non-blocking) mode, allowing for a minimum processing latency at this tier. On the other hand, the client functionality (for example a NamedCache.put call) is quite often implemented using a synchronous (blocking) approach using the internal poll API. Naturally, this poll API is not allowed to be used by the service thread, since this could lead to a high latency at best and deadlock at worst. When a listener is added to a local map that is used as a primary storage for the partitioned cache service, the events that such a listener receives are sent synchronously on that same service thread. It is discouraged to perform any operations that have a potential of blocking during such an event processing. A best practice is to queue the event and process it asynchronously on a different thread.

Coherence - Large clusters hanging in queries

There will be instances where you might need to query the entire coherence cluster spanning 100+ nodes and you might experience that the queries are not completed in hours.

Filter filter =
Set entries = CacheFactory.getCache(cacheName).entrySet(filter);

The filters' should be constructed optimally and if the queries are spanning 100+ nodes then try using the PartitionedFilter for executing such queries. The Paritition filter limits the scope of another filter to those entries that have keys that belong to the specified partition set. This approach may complicate the client code, but can dramatically reduce the memory footprint used by the requestor.

Another approach to using PartitionedFilter is PartitionedIterator as below

PartitionedIterator iter = new PartitionedIterator(cache, filter, setPartitions,PartitionedIterator.OPT_ENTRIES | PartitionedIterator.OPT_BY_MEMBER);

while (iter.hasNext())
{
Map.Entry entry = (Map.Entry) iter.next();
}
}

What should be the value of thread-count in distributed cache?

The thread-count value specifies the number of daemon threads used by the distributed cache service. If zero, all relevant tasks are performed on the service thread. The Default value is 0 and can be override using the system property tangosol.coherence.distributed.threads. This value will increase the parallelism of the processing in the Coherence Grid provided the edition used in Enterprise or Grid and not Standard.

It is recommended to set the value to 0 for scenarios with purely in-memory data (no read-through, write-through, or write-behind) and simple access (no entry processors, aggregators, and so on). For heavy compute scenarios (such as aggregators), the number of threads should be the number of available cores for that compute. For example, if you run 4 nodes on a 16 core box, then there should be roughly 4 threads in the pool. For IO intensive scenarios (such as read through, write-through, and write-behind), the number of threads must be higher. In this case, increase the threads just to the point that the box is saturated.

Remember, each service instance has its own primary thread. This thread has the option of using its own isolated thread pool if the thread-count is greater than zero. If the thread-count is zero, then all work will be performed by the primary service thread. If the thread-count is greater than zero, then all work will be performed by the thread pool (the primary thread acts as a task coordinator). The thread-count is per-service and per-cluster-member. Each cache service has a unique name. The CacheFactory class uses a single cache service instance for each cache type (Replicated/Distributed/etc). If you manually create additional cache services, they will each have their own isolated thread pools.

Oracle Coherence - Split Brain Scenario

Please refer to Oracle Notes :

Oracle Coherence and Split-Brain FAQ [ID 1069132.1]

Oracle Coherence, Split-Brain, and Recovery Protocols Example In Detail [ID 1069429.1]

Witness Protocol: The Coherence clustering protocol (TCMP) is a reliable transport mechanism built on UDP. In order for the protocol to be reliable, it requires an acknowledgement (ACK) for each packet delivered. If a packet fails to be acknowledged within the configured timeout period, the Coherence cluster member will log a packet timeout. When this occurs, the cluster member will consult with other members to determine who is at fault for the communication failure. If the witness members agree that the suspect member is at fault, the suspect is removed. If the witnesses unanimously disagree, the accuser is removed. This process is known as the witness protocol.

Panic Protocol: When the presence of more than one cluster (i.e. Split-Brain) is detected by a Coherence member, the panic protocol is invoked in order to resolve the conflicting clusters and consolidate into a single cluster. The protocol consists of the removal of smaller clusters until there is one cluster remaining. In the case of equal size clusters, the one with the older Senior Member will survive.

How to use ExtensibleEnvironment in Coherence Cache Configuration?

The ExtensibleEnvironment is an enhanced ConfigurableCacheFactory implementation that allows developers to independently create custom configurations and runtime extensions to Coherence. In other words, it allows you to include multiple configurations for your Coherence Grid. The steps that shall be followed to use it are as under:

1. Download the coherence-common jar from the Incubator site

2. Create the parent configuration file with the introduce:config pointing to other configuration that you want to include in the Coherence Grid configuration as below:

<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config xmlns:introduce="class://com.oracle.coherence.environment.extensible.namespaces.IntroduceNamespaceContentHandler">
<introduce:config file="C:\Oracle\Coherence\Files\IncludeExample\IncludeConfiguration\include-config.xml"/>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>example-distributed</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<!--
Distributed caching scheme.
-->
<distributed-scheme>
<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
<local-scheme>
<scheme-ref>example-binary-backing-map</scheme-ref>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
</caching-schemes>
</cache-config>

Please note, it is very important to specify the namespace xmlns:introduce="class://com.oracle.coherence.environment.extensible.namespaces.IntroduceNamespaceContentHandler and also the file location needs to be complete path as otherwise it look in META-INF folder of the project.

3. Create other configuration files as mentioned below:

<?xml version="1.0"?>

<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-schemes>
<local-scheme>
<scheme-name>example-binary-backing-map</scheme-name>
<eviction-policy>HYBRID</eviction-policy>
<high-units>{back-size-limit 0}</high-units>
<unit-calculator>BINARY</unit-calculator>
<expiry-delay>{back-expiry 1h}</expiry-delay>
<cachestore-scheme></cachestore-scheme>
</local-scheme>
</caching-schemes>
</cache-config>

Now in the above example, we can see that parent configuration file has a reference to the "example-binary-backing-map" cache scheme which is available in the child configuration scheme and is included using the introduce element.

Using Log4j for Coherence

In order to start using Log4j for Coherence Grid Log Management,

Step 1: Pass the following system properties:

-Dlog4j.configuration=file:${DIR}/log4j.xml [Location of log4j.xml]
-Dtangosol.coherence.log.logger=Coherence
-Dtangosol.coherence.log=log4j

Step2: Add the log4j library in the classpath

Step 3: In order to create seperate files for multiple nodes running in the cluster, create a system variable that will be passed to the log4j.xml

-DlogFileName=${LOG_DIR}/.${INSTANCE}_${TIMESTAMP}

The sample Log4j.xml that can be used is as under,

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<!-- Pattern to output the caller's file name and line number -->
<param name="ConversionPattern" value="%5p [%t] (%F:%L) - %m%n"/>
</layout>
</appender>
<appender name="FileRollbySize" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="${logFileName}"/>
<param name="MaxFileSize" value="10000KB"/>
<!-- Keep 5 backup file -->
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%p %t %c - %m%n"/>
</layout>
</appender>
<root>
<appender-ref ref="FileRollbySize" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>