Split Brain: What’s new in Oracle Database 12.1.0.2c?

In an Oracle cluster prior to version 12.1.0.2c, when a split brain problem occurs, the node with lowest node number survives. However, starting from Oracle Database 12.1.0.2c, the node with higher weight will survive during split brain resolution. In this article I will explore this new feature for one of the possible factors contributing to the node weight, i.e. the

In an Oracle cluster prior to version 12.1.0.2c, when a split brain problem occurs, the node with lowest node number survives. However, starting from Oracle Database 12.1.0.2c, the node with higher weight will survive during split brain resolution.

In this article I will explore this new feature for one of the possible factors contributing to the node weight, i.e. the number of database services executing on a node.

What is Split Brain?

In a cluster, a private interconnect is used by cluster nodes to monitor each node’s status and communicate with each other. When two or more nodes fail to ping or connect to each other via this private interconnect, the cluster gets partitioned into two or more smaller sub-clusters each of which cannot talk to others over the interconnect. Oblivious of the existence of other cluster fragments, each sub-cluster continues to operate independently of the others. This is called “Split Brain”. In such a scenario, integrity of the cluster and its data might be compromised due to uncoordinated writes to shared data by independently operating nodes. Hence, to protect the integrity of the cluster and its data, the split-brain must be resolved.

How does the Oracle Grid Infrastructure Clusterware resolve a “split brain” situation?

Voting disk is used by Oracle Cluster Synchronization Services Daemon (ocssd) on each node, to mark its own attendance and also to record the nodes it can communicate with. In a “split brain” situation, voting disk is used to determine which node(s) will survive and which node(s) will be evicted.

Prior to Oracle Database 12.1.0.2c, the algorithm to determine the node(s) to be retained / evicted is as follows:

  • If the sub-clusters are of the different sizes, the clusterware identifies the largest sub-cluster, and aborts all the nodes which do not belong to that sub-cluster.
  • If all the sub-clusters are of the same size, the sub-cluster having the lowest numbered node survives so that, in a 2-node cluster, the node with the lowest node number will survive.

However, starting from 12.1.0.2c, in case of split brain, some improvement has been made to node eviction algorithm. In order to make largest number of resources available to the users, the node weight is computed for each node based on number of the resource executing on it and the sub-cluster with higher weight will survive.

Starting in Oracle Database 12.1.0.2c, the new algorithm to determine the node(s) to be retained / evicted is as follows:

  • If the sub-clusters are of the different sizes, the functionality is same as earlier i.e. the clusterware identifies the largest sub-cluster, and aborts all the nodes which do NOT belong to that sub-cluster.
  • If all the sub-clusters are of the same size, the functionality has been modified as:
    • If the sub-clusters have equal node weights, the sub-cluster with the lowest numbered node in it survives so that, in a 2-node cluster, the node with the lowest node number will survive.
    • If the sub-clusters have unequal node weights, the sub-cluster having the higher weight survives so that, in a 2-node cluster, the node with the lowest node number might be evicted if it has a lower weight.

Now I will demonstrate this new feature in an Oracle 12.1.0.2c standard 3 node cluster, using an RAC database called admindb for one of the possible factors contributing to the node weight, i.e. the number of database services executing on a node. Since I will only explore the scenarios for which functionality has been modified, i.e. sub-clusters are of equal size, I have shut down one of the nodes so that there are only 2 active nodes in the cluster.

Current scenario:

Name of the cluster: Cluster01.example.com

Number of nodes: 3 (host01, host02, host03)

Name of RAC database: admindb

Instances of RAC database: admindb1 on host01

admindb2 on host02

Overview

  • Check that only two nodes (host01 and host02) are active and host01 has lower node number
  • Create two singleton services for the RAC database admindb
    • Serv1 : Preferred instance admindb1
    • Serv2: Preferred instance admindb2

Case-I: Equal number of database services executing on both the nodes

  • Start both the services for database admindb so that equal number of database services execute on both the nodes.
  • Simulate loss of connectivity between two nodes.
  • Verify that:
    • host01 is retained as it has a lower node number.
    • host02 is evicted.

Case-II: Unequal number of database services executing on both the nodes

  • Stop the service serv1 so that host01 is not hosting any service and service serv2 executes on host02. As a result, unequal number of database services execute on both the nodes.
  • Simulate loss of connectivity between two nodes.
  • Verify that:
    • host02 is retained as it has higher number of database services executing.
    • host01 is evicted although it has a lower node number.

Demonstration

  • Check that only two nodes (host01 and host02) are active and host01 has lower node number:

  • Create two singleton services for the RAC database admindb:
    • Serv1 : Preferred instance admindb1
    • Serv2: Preferred instance admindb2

Case-I: Equal number of database services executing on both the nodes

We will verify that when an equal number of database services are running on both nodes, the node with lower node number (host01) survives.

  • Verify that admindb is the only database in the cluster having its instances executing on host01 and host02.

  • Start both the services for database admindb so that serv1 executes on host01 and serv2 executes on host02. As a result, equal number of database services execute on both the nodes.

  • Find out name of Private network

  • To simulate loss of connectivity between two nodes, stop the private network service on one of the nodes:

  • Verify that host01 is retained as it has a lower node number and host02 is evicted:

Hence, we observed that when an equal number of database services were running on both nodes, the node with lower node number (host01) survives.

Case-II: Unequal numbers of database services executing on both the nodes

We will verify that when an unequal number of database services are running on the two nodes, the node hosting the higher number of database services survives even if it has a higher node number.

  • Stop the service serv1 so that host01 is not hosting any service and service serv2 executes on host02. As a result, unequal number of database services execute on both the nodes.

  • To simulate loss of connectivity between two nodes, stop private network service on one of the nodes:

  • Verify that host02 is retained as it has higher number of database services executing and host01 is evicted although it has a lower node number:

  • OCSSD Log of host02:

Thus, we observed that when unequal number of database services are running on the two nodes, the node with higher number of database services survives even though it has a higher node number.

Summary:

Starting from 12.1.0.2, during split brain resolution, the new algorithm followed to decide the nodes to be evicted/retained is as follows:

  • If the sub-clusters are of the different sizes, the functionality is same as earlier, i.e. the clusterware identifies the largest sub-cluster, and aborts all the nodes which do
    not belong to that sub-cluster
  • If all the sub-clusters are of the same size, the functionality has been modified as:
  • If the sub-clusters have equal node weights, the sub-cluster with the lowest numbered node in it survives so that, in a 2-node cluster, the node with the lowest node number will survive.
  • If the sub-clusters have unequal node weights, the sub-cluster having the higher weight survives so that, in a 2-node cluster, the node with the lowest node number might be evicted if it has a lower weight.