In Oracle Real Application Clusters (RAC), two or more instances of an Oracle DBMS manage a single database. It means that the same data can be queried and updated simultaneously by users connected to distinct servers. Although such shared data architecture may lead to high availability, scalability and performance, it also imposes additional challenges to concurrency and integrity control.
Some of these additional challenges of Oracle’s RAC architecture are addressed by the use of an interconnect network, which is a private network that connects participating cluster nodes. Such network is a key component in RAC and must be configured carefully. Token-rings and crossover cables are not supported and the use of a dedicated switch is recommended. The use of Jumbo Frames should be considered.
The interconnect network is used by RAC as part of the global shared cache mechanism. Also, it is used by nodes to send heartbeat messages to other participating nodes. If the interconnect network does not function properly, available nodes would reformate the cluster, thinking that other unreachable nodes are dead (even though they are not) leading to a “split brain” situation. Nodes become uncoordinated in their access to shared files and may overwrite each other’s data, causing data loss or corruption.
Detecting Split Brain Situations
Oracle have some specific security mechanisms to avoid split brain. If heartbeat messages between nodes fails for a certain time (timeout interval – less than 30 seconds), a node eviction process is started. In such process, a node is rebooted and removed from the cluster.
Hence, when network communication fails for a certain time, Oracle clusterware places alerts in the log. It starts with CRS-1612 (network communication missing for 50% of timeout interval), then it alerts when communication is missing for 75% of timeout interval (CRS-01611) and another alert is placed when communication missing reaches 90% of timeout interval (CRS-01610). When 100% of timeout interval is reached, node eviction process starts (see our article on shutdown in progress in Oracle).
The voting disk (in shared storage) is used to detect alive nodes: the node with the highest number would be the one to be evicted. The CRS-1609 (“This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity”) message is places in evicted node’s log. One may also find “clssnmCheckDskInfo: Aborting local node to avoid splitbrain.” in the evicted node CSSD log. In the surviving nodes, you may find CRS-1607 message (“Node % is being evicted in cluster incarnation”).
If you get any of the above messages in your log, you should check the interconnect network used by cluster. Look if it is functioning properly, verifying logical configurations and physical components.