In part 1 I explained how to duplicate a single instance database using RMAN. This article will focus on converting the single instance to RAC.

2. Convert Single instance to RAC

We have migrated the database successfully from Non-ASM to ASM using RMAN; the database is still running as single instance.  Now we need to convert the single instance database to RAC.

 2.1 Create redo and undo for second instance

Each individual instance requires an independent set of redo and undo segments. Generally speaking, redo and undo are handled on a per instance basis. You must create two redo log groups for each instance and all redo groups must be stored on shared devices for an instance or crash recovery purpose. Each instance can have more than two redo log groups, but two are minimum per instance.

Each instance is assigned a thread number starting at 1, when you add more logfiles make sure you mention the THREAD, so that logfiles will be added to right instance.

Create second thread of online redo logs in order to start instance 2 and enable thread 2. It is always recommended to create redo log groups across disk groups for redundancy:

SYS@ORADB> alter database add logfile thread 2 group 3 ('+DATA’,'+FLASH’) size 50m reuse;

Database altered.
SYS@ORADB> alter database add logfile thread 2 group 4 ('+DATA’,'+FLASH’) size 50m reuse;

Database altered.

SYS@ORADB>  alter database enable public thread 2;

Database altered.

Create undo tablespace for second instance using recommended automatic undo management feature. Each instance undo tablespace must be shared by all other instances for recovery:

SYS@ORADB> create undo tablespace UNDOTBS2 datafile  '+DATA' size 500M;

If you have more than two nodes then you need to repeat the steps on each node in the cluster.

2.2 Add cluster related parameters

The duplicated instance does not have any cluster related parameters, we need to add the cluster parameter in order to convert single instance to RAC. The CLUSTER_DATABASE=true parameter needs to set before a database can be started in cluster mode.

The CLUSTER_DATABASE_INSTANCES parameter specifies the number of instances configured as part of the cluster database. You should set this parameter value to the number of instances in your Real Application Cluster.

The INSTANCE_NUMBER is a unique number that maps instances to database.

Add below cluster related parameters to initORADB.ora:

*.cluster_database_instances=2
*.cluster_database=true
*.remote_listener='LISTENERS_ORADB’
ORADB1.instance_number=1
ORADB2.instance_number=2
ORADB1.thread=1
ORADB2.thread=2
ORADB1.undo_tablespace='UNDOTBS1'
ORADB2.undo_tablespace='UNDOTBS2'
#update the actual controlfile path
*.control_files='+DATA/ORADB/controlfile/current.256.666342941','+FLASH/ORADB/controlfile/current.256.662312941'

Copy the updated init.ora file to node2 and rename the files as per instance name.

[oracle@orarac1]$ mv initORADB.ora initORADB1.ora
[oracle@orarac2]$ mv initORADB.ora initORADB2.ora

To make it easier to configure the tnsnames.ora and listener.ora manually for RAC environment, use Network Configuration Assistant (NETCA) to configure these files.

Drop the static listener created during the duplication process and launch Network Configuration Assistant (NETCA) on node1 and create the listener and tnsnames.ora entries.  The netca starts the listener at the end of the configuration.

Prior 11g R2, it is recommended to run the listener from the ASM home and from11gR2 it is recommended to run the listener from the Grid Infrastructure home.

You can refer to the link below for configuring and administering Oracle net listener:
http://docs.oracle.com/cd/E11882_01/network.112/e10836/listenercfg.htm

Shutdown the database
SYS@ORADB> > shutdown immediate

2.3 Update the environment and start the database

Set the environment variable for each instance, recreate the password file and start the instances individually on two different nodes.

Environment variable setup on node1:

[oracle@orarac1]$ export ORACLE_SID=ORADB1
[oracle@orarac1]$ export ORACLE_HOME=/home/oracle/product/v10204
[oracle@orarac1]$ orapwd file= $ORACLE_HOME/dbs/orapwORADB1 password=xxxxxxx
[oracle@orarac1]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Jan 19 19:01:29 2012

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

SYS@ORADB1> connect /as sysdba
Connected to an idle instance.
SQL>startup
ORACLE instance started.

Total System Global Area  541065216 bytes
Fixed Size                  2085288 bytes
Variable Size             289410648 bytes
Database Buffers          239075328 bytes
Redo Buffers               10493952 bytes
Database mounted.

Database opened.

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
ORADB1

SQL>

Environment variable setup on node2:

[oracle@orarac2]$ export ORACLE_SID=ORADB2
[oracle@orarac2]$ export ORACLE_HOME=/home/oracle/product/v10204
[oracle@orarac2]$ orapwd file= $ORACLE_HOME/dbs/orapwORADB2 password=xxxxxxx
oracle@orarac2]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Jan 19 19:02:29 2012

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

SYS@ORADB1> connect /as sysdba

Connected to an idle instance.

SQL>startup
ORACLE instance started.

Total System Global Area  541065216 bytes
Fixed Size                  2085288 bytes
Variable Size             289410648 bytes
Database Buffers          239075328 bytes
Redo Buffers               10493952 bytes
Database mounted.
Database opened.
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
ORADB2

SQL>

Users may receive the below error, which is the most common error:

[oracle@orarac1]$sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Jan 19 20:26:33 2012

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

SQL> connect / as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/home/oracle/product/v10204/dbs/initORADB.ora'

To avoid this error make sure to set the environment variable correctly.

2.4 Register the RAC instances with CRS

The Server Control Utility(SRVCTL) is a command line interface that you can use to manage RAC databases, ASM instances etc. from a single point. Using SRVCTL, you can stop and start the database and instances, delete or move instances and services. The SRVCTL stores configuration data in the Oracle Cluster Registry (OCR), you can also use this tool to add services and manage service configuration.

In order to manage Oracle instances using SRVCTL, you need to register the RAC instances with CRS. The CRS  should be able to manage the RAC instances when you register. If the servers are rebooted due to planned or unplanned maintenance, the CRS automatically starts up the instances during the reboot.

Register the database instances with CRS framework using this command:

[oracle@orarac1]$ srvctl add database -d ORADB -o /home/oracle/product/v10204
[oracle@orarac1]$ srvctl add instance -d ORADB -i ORADB1 -n orarac1
[oracle@orarac1]$ srvctl add instance -d ORADB -i ORADB2 -n orarac2

Now the instances are registered with CRS, use SRVCTL to stop and start the database. Once the daetabase has been started with SRVCTL the conversion process is complete.

Stop the database using this command:

[oracle@orarac1]$ srvctl stop database -d ORADB

Start the database using this command:

[oracle@orarac1]$ srvctl start database -d ORADB

Use the following command to check the status of all instances converted RAC database:

SQL> select * from v$active_instances

INST_NUMBER INST_NAME         

-----------  -----------------
           1 ORADB1 
           2 ORADB2 

Check the status of all resources:

[oracle@orarac1]$ crs_stat -t

Name           Type           Target    State     Host
------------------------------------------------------------
ora....B1.inst application    ONLINE    ONLINE    orarac1
ora....B2.inst application    ONLINE    ONLINE    orarac2
ora.ORADB.db application      ONLINE    ONLINE    orarac2
ora....SM1.asm application    ONLINE    ONLINE    orarac1
ora....r1.lsnr application    ONLINE    ONLINE    orarac1
ora....ac1.gsd application    ONLINE    ONLINE    orarac1
ora....ac1.ons application    ONLINE    ONLINE    orarac1
ora....ac1.vip application    ONLINE    ONLINE    orarac1
ora....SM2.asm application    ONLINE    ONLINE    orarac2
ora....r2.lsnr application    ONLINE    ONLINE    orarac2
ora....ac2.gsd application    ONLINE    ONLINE    orarac2
ora....ac2.ons application    ONLINE    ONLINE    orarac2
ora....ac2.vip application    ONLINE    ONLINE    orarac2

The State must be “ONLINE” for all resources.

The crs_stat command deprecated in 11gR2, if your database is 11g R2 then use this command to check the status:

$ crsctl status resource –t

In order to shutdown CRS you can run the following command on each node in the cluster. This command will stop Oracle HA Services and Clusterware Stack in a single command:

#crsctl stop crs

From 11g R2 onwards, you can do this in two stops:

  1. Stop Clustwerware stack on local node:
    #crsctl stop cluster
  2. You can stop the clusterware stack on all nodes in the cluster:
    # Crsctl stop cluster –all

2.5 Create the spfile in ASM 

SPFILE simplifies administration, and maintains consistent parameter settings. SPFILE is a binary file and lets you make persistent changes to individual parameters. Use the CREATE SPFILE statement to create a server parameter file from PFILE with SYSDBA/SYSOPER privilege. By default PFILE or SPFILE default location is “$ORACLE_HOME/dbs” for UNIX and LINUX, “%ORACLE_HOME%\database” for Windows. In the case of RAC, the best practice is creating SPFILE in ASM shared storeage:

SQL> create spfile=’+FLASH/spfileORADB.ora from pfile;

Restart the database in order to take effect of spfile.:

[oracle@orarac1]$ srvctl stop instance  -d ORADB
[oracle@orarac1]$ srvctl start database -d ORADB

2.7 Cluster Verify

The Cluster Verification Utility will be used to perform system checks in preparation for installation, system changes or patch updates. Run CLUVFY utility to check everything is fine after converting single instance to RAC:

Verify your Oracle Clusterware using “cluvfy” and fix issues, if any.

For ex: – The following command checks the complete oracle clusterware stack:
$ cluvfy comp crs- n all – verbose

Where -n all option is to verify all of the cluster nodes – verbose argument produces detailed output of individual checks.

Please refer Oracle documentation on Cluster Verify Utility for more information:
http://docs.oracle.com/cd/B28359_01/rac.111/b28255/cvu.htm

2.6  Check the logs

It is strongly advised to check all the logs related to Cluster, Database and instances when you perform installation, system changes or patch updates…etc. Make sure to check the log files to see the unexpected issues, if any.

CRS_HOME/log/hostname/crsd/ – The log files for the CRS daemon
CRS_HOME/log/hostname/cssd/ – The log files for the CSS daemon
CRS_HOME/log/hostname/evmd/ – The log files for the EVM daemon
CRS_HOME/log/hostname/client/ – The log files for the Oracle Cluster Registry (OCR)
CRS_HOME/log/hostname/racg/ – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostname/racg – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostanme/alert.log – The alert.log for Clusterware issues.

Please note that the CRS_HOME is the directory in which the Oracle Clusterware software was installed and hostname is the name of the node.

2.7  Setup Backup for RAC Database Using RMAN

Using RMAN for Oracle RAC database backup is a recommended and easy method as it does not require any separate installation. RMAN enables you to backup, restore and recover data files, control files, redo log files and spfile.  RAC database backup is the same as backing up single instance but the best practice is to backup into a shared file system.

You can setup the backup from any instance in the Cluster and you are able to restore and recovery database from any instance in the cluster.

You can configure multiple channels and scale up the database backup load on multiple RAC instances:

CONFIGURE CHANNEL DEVICE TYPE [DISK|SBT] CONNECT '@RAC1'
CONFIGURE CHANNEL DEVICE TYPE [DISK|SBT]CONNECT '@RAC2'

You can configure channels to use automatic load balancing:

CONFIGURE DEVICE TYPE [DISK|SBT] PARALLELISM no_of_channels;

Take advantage of parallelism for faster restore/recovery operations by setting RECOVERY_PARALLELISM

RMAN can be configured to automatically backup the control file and spfile whenever the database structure in the control file changes and whenever a backup record is added. The autobackup feature enables RMAN to recover the database even if the current control file, catalog, and server parameter file are lost:

Sample script to backup database:

RMAN> run {

   allocate channel d1 type disk;
   allocate channel d2 type disk;
   backup  tag 'full_backup'   database
   format 'shared_storage/ORACLE_SID_%d_t%t_s%s_p%p.rmn';
   release channel d1;
   release channel d2;
    }
exit;

Oracles 11g introduced many exciting new features and information can be found here:
Oracle RMAN New Features in 11g

If you need to backup your database to tape then you need Media management library. Please refer to the following document for configuring RMAN to backup database to a media manager:
http://docs.oracle.com/cd/B28359_01/backup.111/b28270/rcmconfb.htm#i1006526

In many situations DBAs end up with an incomplete or failed recovery. The main reason for this issue is users don’t know what was backed up and what needs to be backed up for a successful recovery. Please refer to the following document to understand Oracle RMAN Reporting:
http://www.oracleracexpert.com/2011/06/understand-oracle-rman-reporting.html

2.8  Workload Management

Server pools (From 11g R2) – You can define a database to run in a server pool.  A Server pool is a logical entity, an administrator can allocate resources to specific applications. Refer to the following document on server pools:
http://docs.oracle.com/cd/E11882_01/server.112/e24611/apqos_intro.htm#APQOS117

Service – Use service for workload management and you can hide the Oracle RAC complexity by providing a single system image to manage work load.   A service can span one of more instances of a database and an instance can support multiple services. Refer to the following document to understand Oracle Net Service architecture:
http://docs.oracle.com/cd/E11882_01/network.112/e10836/intro.htm#g462598

Load Balancing – Oracle Net service provides the connection load balancing and from 11g R2 onwards you can use SCAN (single client access name). Based on goal defined for the service, the listener chooses the database instance that will best meet the goal and the connection is routed to that instance through the local listener.

Connection pool - RAC database

Refer to the following document to understand SCAN and Client service connections:
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/admin.htm#autoId29

Fast Application Notification (FAN) –  FAN provides integration between RAC database and application. It allows the application to be aware of the current configuration of the server pools so that application connections are made to those RAC instances that are currently able to respond. Refer to the following document to understand FAN:
http://docs.oracle.com/cd/E11882_01/rac.112/e17264/configwlm.htm#TDPRC295

Alternative methods to convert single instance to RAC

Oracle provides the following methods to convert a single instance database to RAC:

  1. DBCA
  2. RCONFIG (from 10gR2)
  3. Enterprise Manager
  4. Manual (Using RMAN)

 Refer to the following document to learn about these alternate methods: http://docs.oracle.com/cd/E11882_01/install.112/e18069/cvrt2rac.htm

Conclusion

Oracle Real application cluster is designed for Scalability and high availability and many customers implement RAC for their mission critical applications. Oracle RAC provides continuous service for both unplanned and planned outages and it allows the enterprise applications to grow in any direction by protecting from software and hardware failures, and ensures continuous data access.