Showing posts with label RAC Internals. Show all posts
Showing posts with label RAC Internals. Show all posts

Wednesday, November 17, 2010

Convert a Single Instance Database to Oracle RAC 10g on RHEL3

Learn how to convert a single-instance database to Oracle Real Application Clusters (RAC) 10g on Red Hat Enterprise Linux 3 (RHEL3), step by step.
Contents
Overview
Step 1: Preliminary Installation
Step 2: Migrate Your Database to ASM
Step 3: Install Cluster Ready Services (CRS) Software
Step 4: Install Oracle RAC Software
Step 5: Post Installation
Step 6: Test Transparent Application Failover (TAF)
Conclusion

Download for this article:
Oracle Database 10g Enterprise Edition and Clusterware for Linux x86
Oracle ASMlib Drivers



For those with limited hands-on experience, implementing Oracle RAC 10g can be an intimidating prospect. But it doesn't have to be that way.
In this guide, I'll provide the precise procedures for converting a single-instance Oracle 10g database to Oracle RAC on RHEL3. We'll use Oracle Automatic Storage Management (ASM), an integrated file system and a volume management solution built into the Oracle database kernel, for RAC database storage because it simplifies database storage management tasks while offering storage reliability and performance.
To summarize on a very high level, there are several ways to convert your database to RAC. If the database is small, you can consider installing a new RAC database on ASM and export/import your data from your current database to the RAC database. For a larger database, the preferred method would be to use RMAN to migrate the database files over to ASM. The method demonstrated here is a two-phase approach: first, migrating the data to ASM, and second, converting the single-instance database to RAC. If you are new to ASM, I would recommend taking this migration path to get familiar with ASM before leaping into ASM and RAC at the same time. (For more information about Oracle ASM, visit the Oracle ASM Product Center or refer to the documentation.)
This guide requires a basic understanding of RAC architecture and some familiarity with managing and administering Oracle Database and Red Hat Enterprise Linux. Refer to the documentation for details.

________________________________________
Overview
The RAC cluster comprises two Intel x86 servers running on RHEL3 (Kernel 2.4.21-27). Each node has access to a shared storage and connectivity to the public and private network.
This guide is structured into the following steps:
1. Preliminary Installation
2. Migrating Your Database to ASM
3. Installing Oracle Cluster Ready Services (CRS) Software
4. Installing Oracle RAC Software
5. Post Installation
6. Testing Transparent Application Failover (TAF)
Unless otherwise specified, you should execute all steps on both nodes.
Here's an overview of our single-instance database environment before converting to RAC:
Host Name Instance Name Database Name $ORACLE_BASE Database File Storage
salmon1 prod1 prod1 /u01/app/oracle ext3

And an overview of the RAC database environment:
Host Name Instance Name Database Name $ORACLE_BASE Database File Storage OCR & CRS Voting Disk
salmon1 prod1a prod1 /u01/app/oracle ASM Raw
salmon2 prod1b prod1 /u01/app/oracle ASM Raw

You'll install the Oracle Home on each node for redundancy.The ASM and RAC instances share the same Oracle Home on each node.
Step 1: Preliminary Installation

1a. Verify software package versions.
Install the required packages. Additional information can be obtained from the documentation.
[root@salmon1]# rpm -qa | grep -i compat
compat-libstdc++-7.3-2.96.128
compat-gcc-c++-7.3-2.96.128
compat-libstdc++-devel-7.3-2.96.128
compat-db-4.0.14-5
compat-glibc-7.x-2.2.4.32.6
compat-slang-1.4.5-5
compat-gcc-7.3-2.96.128
compat-pwdb-0.62-3
[root@salmon1]#
[root@salmon1]# rpm -qa | grep openmotif
openmotif-2.2.3-3.RHEL3
[root@salmon1]#
[root@salmon1]# rpm -qa | grep -i gcc
gcc-gnat-3.2.3-42
gcc-c++-ssa-3.5ssa-0.20030801.48
compat-gcc-c++-7.3-2.96.128
libgcc-ssa-3.5ssa-0.20030801.48
gcc-3.2.3-42
gcc-g77-3.2.3-42
gcc-java-3.2.3-42
gcc-ssa-3.5ssa-0.20030801.48
gcc-g77-ssa-3.5ssa-0.20030801.48
gcc-objc-ssa-3.5ssa-0.20030801.48
libgcc-3.2.3-42
gcc-c++-3.2.3-42
gcc-objc-3.2.3-42
gcc-java-ssa-3.5ssa-0.20030801.48
compat-gcc-7.3-2.96.128

1b. Verify kernel parameters.
Verify the following kernel parameters. Additional information can be obtained from the documentation.
[root@salmon1]# sysctl -a | grep shm
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 2147483648
[root@salmon1]# sysctl -a | grep sem
kernel.sem = 250 32000 100 128
[root@salmon1]# sysctl -a | grep -i ip_local
net.ipv4.ip_local_port_range = 1024 65000
[root@salmon1]# sysctl -a | grep -i file-max
fs.file-max = 65536
1c. Create the Oracle Base directory, oracle user, and groups.
Using the information below on node 1, create the oracle user and the oinstall and dba groups on the second node.
[oracle@salmon1]$ hostname
salmon1.dbsconsult.com
[oracle@salmon1]$
[oracle@salmon1]$ id
uid=500(oracle) gid=500(dba) groups=500(dba),501(oinstall)
[oracle@salmon1]$
[oracle@salmon1]$ echo $ORACLE_BASE
/u01/app/oracle

1d. Edit the oracle user environment file.
[oracle@salmon1]$ more .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

export PATH=$PATH:$HOME/bin
export ORACLE_SID=prod1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.1.0
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
umask 022

1e. Configure the oracle user shell limits.
[root@salmon1]# more /etc/security/limits.conf
* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536
[root@salmon1]# grep pam_limits /etc/pam.d/login
session required /lib/security/pam_limits.so
1f. Configure public and private network.
Using the information below, make the necessary changes to network interface devices eth0 (public) and eth1 (private).
[root@salmon1]# redhat-config-network
Host Name IP Address Type
salmon1.dbsconsult.com 192.168.0.184 Public (eth0)
salmon2.dbsconsult.com 192.168.0.185 Public (eth0)
salmon1.dbsconsult.com 10.10.10.84 Private (eth1)
salmon2.dbsconsult.com 10.10.10.85 Private (eth1)
salmon1-vip.dbsconsult.com 192.168.0.186 Virtual
salmon2-vip.dbsconsult.com 192.168.0.187 Virtual
1g. Edit the /etc/hosts file.
127.0.0.1 localhost.localdomain localhost
10.10.10.84 sallocal1.dbsconsult.com sallocal1
10.10.10.85 sallocal2.dbsconsult.com sallocal2
192.168.0.184 salmon1.dbsconsult.com salmon1
192.168.0.185 salmon2.dbsconsult.com salmon2
192.168.0.186 salmon1-vip.dbsconsult.com salmon1-vip
192.168.0.187 salmon2-vip.dbsconsult.com salmon2-vip
Verify the hostname and the configured network interface devices.
[root@salmon1]# hostname
salmon1.dbsconsult.com
[root@salmon1]# /sbin/ifconfig
1h. Establish user equivalence with SSH.
During the Cluster Ready Services (CRS) and RAC installation, the Oracle Universal Installer (OUI) has to be able to copy the software as oracle to all RAC nodes without being prompted for a password. In Oracle 10g, this can be accomplished using ssh instead of rsh.
To establish user equivalence, generate the user's public and private keys as the oracle user on both nodes.
[oracle@salmon1]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
5d:8c:42:97:eb:42:ae:52:52:e9:59:20:2a:d3:6f:59 oracle@salmon1.dbsconsult.com
Test the connection on every node. Verify that you are not prompted for password when you run the following the second time.
ssh salmon1 date
ssh salmon2 date
ssh sallocal1 date
ssh sallocal2 date
ssh salmon1.dbsconsult.com date
ssh salmon2.dbsconsult.com date
ssh sallocal1.dbsconsult.com date
ssh sallocal2.dbsconsult.com date
1i. Configure hangcheck timer kernel module.
The hangcheck timer kernel module monitors the system's health and restarts a failing RAC node. It uses two parameters, hangcheck_tick (defines the system checks frequency) and hangcheck_margin (defines the maximum hang delay before a RAC node is reset), to determine if a node is failing.
Add the following line in /etc/rc.d/rc.local to load the hangcheck module automatically.
[root@salmon1]# grep insmod /etc/rc.d/rc.local
insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
1j. Recreate database control file.
Make sure that the entries below are sized appropriately in the control file before converting to RAC. If required, recreate the database control file with the right settings.
MAXLOGFILES
MAXLOGMEMBERS
MAXDATAFILES
MAXINSTANCES
MAXLOGHISTORY
1k. Resize the database buffer cache.
When transitioning from a single instance database to RAC, additional memory is required for the database buffer cache. In RAC, space is allocated for the Global Cache Service (GCS) in every block buffer cache. The amount of additional memory requires depends on how the application accesses the data—that is, if the same block is cached in more than one instance.
I observed an increase of about 8% buffer cache usage during an experimental demonstration. Use the buffer cache advisory to determine an optimal buffer cache size or let Oracle take control by switching to Oracle Automatic Shared Memory Management (ASMM).

________________________________________
Step 2: Migrate Your Database to ASM
The Oracle Cluster Synchronization Services (CSS) daemon is required for synchronization between the ASM instance and the database instances. The CSS daemon must be up before the ASM instance can be started. When you installed or upgraded to Oracle Database 10g, the CSS should have already been configured for the single-node version and it should start automatically when the system reboots.

[root@salmon1]# ps –ef | grep ocssd.bin | wc –l
17
2a. Download ASM RPMs.
Download the following ASM RPMs from OTN:
• oracleasm-support-2.0.0-1.i386.rpm
• oracleasm-2.4.21-27.EL-1.0.4-2.i686.rpm (driver for UP kernel) or oracleasm-2.4.21-27.ELsmp-1.0.4-1.i686.rpm (driver for SMP kernel)
• oracleasmlib-2.0.0-1.i386.rpm
2b. Install the ASM RPMs.
Install the ASM RPMs as the root user.
[root@salmon1]# rpm -i oracleasm-support-2.0.0-1.i386.rpm
[root@salmon1]# rpm -i oracleasm-2.4.21-27.EL-1.0.4-2.i686.rpm
Linking module oracleasm.o into the module path [ OK ]
[root@salmon1]# rpm -i oracleasmlib-2.0.0-1.i386.rpm
[root@salmon1]# rpm -qa | grep oracleasm
oracleasm-2.4.21-27.EL-1.0.4-2
oracleasm-support-2.0.0-1
oracleasmlib-2.0.0-1
2c. Configure ASM.
Configure the ASMLib as the root user.
[root@salmon1]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration [ OK ]
Creating /dev/oracleasm mount point [ OK ]
Loading module "oracleasm" [ OK ]
Mounting ASMlib driver filesystem [ OK ]
Scanning system for ASM disks
2d. Create ASM disks.
Create the ASM disks on any one node as the root user.
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdg5
Marking disk "/dev/sdg5" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL2 /dev/sdg6
Marking disk "/dev/sdg6" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL3 /dev/sdg7
Marking disk "/dev/sdg7" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL4 /dev/sdg8
Marking disk "/dev/sdg8" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL5/dev/sdg9
Marking disk "/dev/sdg9" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL6 /dev/sdg10
Marking disk "/dev/sdg10" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL7 /dev/sdg11
Marking disk "/dev/sdg11" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL8 /dev/sdg12
Marking disk "/dev/sdg12" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL9 /dev/sdg13
Marking disk "/dev/sdg13" as an ASM disk [ OK ]
[root@salmon1]# /etc/init.d/oracleasm createdisk VOL10 /dev/sdg14
Marking disk "/dev/sdg14" as an ASM disk [ OK ]
Verify that the ASM disks are visible from every node.
[root@salmon1]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
VOL5
VOL6
VOL7
VOL8
VOL9
VOL10
[root@salmon1]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks [ OK ]
2e. Create ASM init.ora.
On the first node, create an init+ASM1A.ora file in $ORACLE_HOME/dbs with the following parameters:
#asm_diskgroups='DG1', ‘DG2’, ‘RECOVERYDEST’
asm_diskstring='ORCL:VOL*'
background_dump_dest=/u01/app/oracle/admin/+ASM/bdump
core_dump_dest=/u01/app/oracle/admin/+ASM/cdump
user_dump_dest=/u01/app/oracle/admin/+ASM/udump
instance_type=asm
large_pool_size=16M
remote_login_passwordfile=exclusive
+ASM1A.instance_number=1
+ASM1B.instance_number=2
2f. Create ASM password file.
Using the orapwd utility, create an orapw+ASM1A file in $ORACLE_HOME/dbs on the first node.
[oracle@salmon1]$ cd $ORACLE_HOME/dbs
[oracle@salmon1]$ orapwd file=orapw+ASM1A password=sys entries=5
2g. Create the first ASM instance.
Create the first ASM instance on the first node. The second ASM instance will be created in Post Installation after the CRS software is installed on the second node.
[oracle@salmon1]$ export ORACLE_SID=+ASM1A
[oracle@salmon1]$ sqlplus / as sysdba

SQL*Plus: Release 10.1.0.3.0 - Production on Thu May 26 05:51:07 2005

Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 104857600 bytes
Fixed Size 777616 bytes
Variable Size 104079964 bytes
Database Buffers 0 bytes
Redo Buffers 0 bytes
ORA-15110: no diskgroups mounted
2h. Create the ASM instance spfile.
Create a spfile immediately after the ASM instance starts. With spfile, any newly created disk groups are automatically added to the spfile.
SQL> create spfile from pfile;

File created.
2i. Create disk groups.
Create three disk groups: DG1, DG2, and RECOVERYDEST. DG1 and DG2 will be used to store Oracle data files and redo logs. RECOVERYDEST will be used as the flash recovery area.
SQL> create diskgroup dg1 normal redundancy
2 failgroup fg1a disk
3 'ORCL:VOL1','ORCL:VOL2'
4 failgroup fg1b disk
5 'ORCL:VOL3','ORCL:VOL4';

Diskgroup created.

SQL> create diskgroup dg2 normal redundancy
2 failgroup fg2a disk
3 'ORCL:VOL5','ORCL:VOL6'
4 failgroup fg2b disk
5 'ORCL:VOL7','ORCL:VOL8';

Diskgroup created.

SQL> create diskgroup recoverydest normal redundancy
2 failgroup fgrd1 disk
3 'ORCL:VOL9'
4 failgroup fgrd2 disk
5 'ORCL:VOL10';

Diskgroup created.

SQL> show parameter diskgroup

NAME TYPE VALUE
------------------------------ ----------- -----------------------------
asm_diskgroups string DG1, DG2, RECOVERYDEST

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB
------------------------- -------------------
DG1 36864
DG2 36864
RECOVERYDEST 73728

3 rows selected.

SQL> select name,path,failgroup from v$asm_disk;

NAME PATH FAILGROUP
-------- --------------- ----------------
VOL1 ORCL:VOL1 FG1A
VOL10 ORCL:VOL10 FGRD2
VOL2 ORCL:VOL2 FG1A
VOL3 ORCL:VOL3 FG1B
VOL4 ORCL:VOL4 FG1B
VOL5 ORCL:VOL5 FG2A
VOL6 ORCL:VOL6 FG2A
VOL7 ORCL:VOL7 FG2B
VOL8 ORCL:VOL8 FG2B
VOL9 ORCL:VOL9 FGRD1

10 rows selected.
2j. Configure flash recovery area.

SQL> connect sys/sys@prod1 as sysdba
Connected.
SQL> alter database disable block change tracking;

Database altered.

SQL> alter system set db_recovery_file_dest_size=72G;

System altered.

SQL> alter system set db_recovery_file_dest=’+RECOVERYDEST’;

System altered.
2k. Migrate data files to ASM.
You must use RMAN to migrate the data files to ASM disk groups. All data files will be migrated to the newly created disk group, DG1. The redo logs and control files are created in DG1 and DG2. In a production environment, you should store redo logs on different set of disks and disk controllers from the rest of the Oracle data files.
SQL> connect sys/sys@prod1 as sysdba
Connected.
SQL> alter system set db_create_file_dest=’+DG1’;

System altered.

SQL> alter system set control_files='+DG1/cf1.dbf' scope=spfile;

System altered.

SQL> shutdown immediate;

[oracle@salmon1]$ rman target /

RMAN> startup nomount;

Oracle instance started

Total System Global Area 419430400 bytes
Fixed Size 779416 bytes
Variable Size 128981864 bytes
Database Buffers 289406976 bytes
Redo Buffers 262144 bytes

RMAN> restore controlfile from '/u02/oradata/prod1/control01.ctl';

Starting restore at 26-MAY-05
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=160 devtype=DISK

channel ORA_DISK_1: copied controlfile copy
output filename=+DG1/cf1.dbf
Finished restore at 26-MAY-05

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN> backup as copy database format '+DG1';

Starting backup at 26-MAY-05
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=160 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00001 name=/u02/oradata/prod1/system01.dbf
output filename=+DG1/prod1/datafile/system.257.1 tag=TAG20050526T073442 recid=1 stamp=559294642
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:49
channel ORA_DISK_1: starting datafile copy
input datafile fno=00003 name=/u02/oradata/prod1/sysaux01.dbf
output filename=+DG1/prod1/datafile/sysaux.258.1 tag=TAG20050526T073442 recid=2 stamp=559294735
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:26
channel ORA_DISK_1: starting datafile copy
input datafile fno=00002 name=/u02/oradata/prod1/undotbs01.dbf
output filename=+DG1/prod1/datafile/undotbs1.259.1 tag=TAG20050526T073442 recid=3 stamp=559294750
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=/u02/oradata/prod1/users01.dbf
output filename=+DG1/prod1/datafile/users.260.1 tag=TAG20050526T073442 recid=4 stamp=559294758
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
copying current controlfile
output filename=+DG1/prod1/controlfile/backup.261.1 tag=TAG20050526T073442 recid=5 stamp=559294767
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:08
Finished backup at 26-MAY-05

RMAN> switch database to copy;

datafile 1 switched to datafile copy "+DG1/prod1/datafile/system.257.1"
datafile 2 switched to datafile copy "+DG1/prod1/datafile/undotbs1.259.1"
datafile 3 switched to datafile copy "+DG1/prod1/datafile/sysaux.258.1"
datafile 4 switched to datafile copy "+DG1/prod1/datafile/users.260.1"

RMAN> alter database open;

database opened

RMAN> exit

SQL> connect sys/sys@prod1 as sysdba
Connected.
SQL> select tablespace_name, file_name from dba_data_files;

TABLESPACE FILE_NAME
--------------------- -----------------------------------------
USERS +DG1/prod1/datafile/users.260.1
SYSAUX +DG1/prod1/datafile/sysaux.258.1
UNDOTBS1 +DG1/prod1/datafile/undotbs1.259.1
SYSTEM +DG1/prod1/datafile/system.257.1
2l. Migrate temp tablespace to ASM.
SQL> alter tablespace temp add tempfile size 100M;

Tablespace altered.

SQL> select file_name from dba_temp_files;

FILE_NAME
-------------------------------------
+DG1/prod1/tempfile/temp.264.3
2m. Migrate redo logs to ASM.
Drop existing redo logs and recreate them in ASM disk groups, DG1 and DG2.
SQL> alter system set db_create_online_log_dest_1='+DG1';

System altered.

SQL> alter system set db_create_online_log_dest_2='+DG2';

System altered.

SQL> select group#, member from v$logfile;

GROUP# MEMBER
--------------- ----------------------------------
1 /u03/oradata/prod1/redo01.log
2 /u03/oradata/prod1/redo02.log

SQL> alter database add logfile group 3 size 10M;

Database altered.

SQL> alter system switch logfile;

System altered.

SQL> alter database drop logfile group 1;

Database altered.

SQL> alter database add logfile group 1 size 100M;

Database altered.

SQL> alter database drop logfile group 2;

Database altered.

SQL> alter database add logfile group 2 size 100M;

Database altered.

SQL> alter system switch logfile;

System altered.

SQL> alter database drop logfile group 3;

Database altered.

SQL> select group#, member from v$logfile;

GROUP# MEMBER
--------------- ----------------------------------------
1 +DG1/prod1/onlinelog/group_1.265.3
1 +DG2/prod1/onlinelog/group_1.257.1
2 +DG1/prod1/onlinelog/group_2.266.3
2 +DG2/prod1/onlinelog/group_2.258.1
2n. Create pfile from spfile.
Create and retain a copy of the database pfile. You'll add more RAC specific parameters to the pfile later, in the Post Installation.
SQL> connect sys/sys@prod1 as sysdba
Connected.
SQL> create pfile='/tmp/tmppfile.ora' from spfile;

File created.
2o. Add additional control file.
If an additional control file is required for redundancy, you can create it in ASM as you would on any other filesystem.
SQL> connect sys/sys@prod1 as sysdba
Connected to an idle instance.
SQL> startup mount
ORACLE instance started.

Total System Global Area 419430400 bytes
Fixed Size 779416 bytes
Variable Size 128981864 bytes
Database Buffers 289406976 bytes
Redo Buffers 262144 bytes
Database mounted.

SQL> alter database backup controlfile to '+DG2/cf2.dbf';

Database altered.

SQL> alter system set control_files='+DG1/cf1.dbf ','+DG2/cf2.dbf' scope=spfile;

System altered.

SQL> shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 419430400 bytes
Fixed Size 779416 bytes
Variable Size 128981864 bytes
Database Buffers 289406976 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.
SQL> select name from v$controlfile;

NAME
---------------------------------------
+DG1/cf1.dbf
+DG2/cf2.dbf
After successfully migrating all the data files over to ASM, the old data files are no longer needed and can be removed. Your single-instance database is now running on ASM!

________________________________________
Step 3: Install Cluster Ready Services (CRS) Software
CRS requires two files—the Oracle Cluster Registry (OCR) and the Voting Disk—on shared raw devices or Oracle Cluster File System (OCFS). These files must be accessible to all nodes in the cluster. Raw devices are used here to house both files.
3a. Create OCR and Voting Disk.
The storage size for the OCR should be at least 100MB and the storage size for the voting disk should be at least 20MB.
File Raw Device Disk Partition Filename Size (MB)
OCR /dev/raw/raw11 /dev/sde1 /u02/oradata/prod1/ocr 100
Voting Disk /dev/raw/raw12 /dev/sde2 /u02/oradata/prod1/vdisk 20
[root@salmon1]# more /etc/sysconfig/rawdevices
/dev/raw/raw11 /dev/sde1
/dev/raw/raw12 /dev/sde2
[root@salmon1]# chown oracle:dba /dev/raw/raw11
[root@salmon1]# chown oracle:dba /dev/raw/raw12
[root@salmon1]# /sbin/service rawdevices restart
Assigning devices:
/dev/raw/raw11 --> /dev/sde1
/dev/raw/raw11: bound to major 8, minor 65
/dev/raw/raw12 --> /dev/sde2
/dev/raw/raw12: bound to major 8, minor 66
done
[root@salmon1]# su - oracle
[oracle@salmon1]$ ln -s /dev/raw/raw11 /u02/oradata/prod1/ocr
[oracle@salmon1]$ ln -s /dev/raw/raw12 /u02/oradata/prod1/vdisk
3b. Install CRS software.
Before installing the CRS software, shut down the listener, database, and ASM instance. Mount the CRS CD or download the software from OTN. The OUI should be launched on only the first node. During installation, the installer automatically copies the software to the second node.
[oracle@salmon1]$ export ORACLE_BASE=/u01/app/oracle
[oracle@salmon1]$ /mnt/cdrom/runInstaller
1. Welcome - Click on "Next."
2. Specify File Locations:
a. Name: OraCr10g_home1
b. Path: /u01/app/oracle/product/10.1.0/crs_1
3. Language Selection - English
4. Cluster Configuration:
a. Cluster Name: crs
b. Public Node Name: salmon1, Private Node Name: sallocal1
c. Public Node Name: salmon2, Private Node Name: sallocal2
5. Private Interconnect Enforcement:
a. Interface Name: eth0, Subnet: 192.168.0.0, Interface Type: Public
b. Interface Name: eth1, Subnet: 10.10.10.0, Interface Type: Private
6. Oracle Cluster Registry:
a. Specify OCR Location: /u02/oradata/prod1/ocr
7. Voting Disk:
a. Enter Voting disk filename: /u02/oradata/prod1/vdisk
8. Execute /u01/app/oracle/oraInventory/orainstRoot.sh script from another window as the root user on the first node.
9. Execute /u01/app/oracle/oraInventory/orainstRoot.sh script from another window as the root user on the second node.
10. Click on "Continue" after the orainstRoot.sh has been run on both nodes.
11. Summary - Click on "Install"
12. Execute /u01/app/oracle/product/10.1.0/crs_1/root.sh script from another window as the root user on the first node. Do not run the next step until this step completes.
13. Execute /u01/app/oracle/product/10.1.0/crs_1/root.sh script from another window as the root user on the second node.
14. Click on "OK" after the root.sh has been run on both nodes.
15. End of Installation - Click on "Exit."
[oracle@salmon1]$ /u01/app/oracle/product/10.1.0/crs_1/bin/olsnodes -n
salmon1 1
salmon2 2
[oracle@salmon1]$ ps -ef | egrep "css|crs|evm"

________________________________________
Step 4: Install Oracle RAC Software
4a. Edit the oracle user environment file.
On node 1, set ORACLE_SID=prod1a.
On node 2, set ORACLE_SID=prod1b.
[oracle@salmon1]$ more .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

export PATH=$PATH:$HOME/bin
export ORACLE_SID=prod1a
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.1.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.1.0/crs_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
umask 022
4b. Install RAC software.
Mount the Oracle Database 10g Enterprise Edition CD or download the software from OTN. Launch the OUI on only the first node. During installation, the installer automatically copies the software to the second node.
[oracle@salmon1]$ . ~/.bash_profile
[oracle@salmon1]$ /mnt/cdrom/runInstaller
1. Welcome - Click on "Next."
2. Specify File Locations:
a. Name: OraDB10g_home1
b. Path: /u01/app/oracle/product/10.1.0/db_1
3. Specify Hardware Cluster Installation Mode:
a. Select "Cluster Installation"
b. Click on "Select All"
4. Select Installation Type:
a. Select "Enterprise Edition"
5. Product-specific Prerequisite Checks:
a. Verify that all checks are successful before proceeding. Ignore the openmotif-2.1.30-11 warning.
6. Select Database Configuration:
a. Select "Do not create a starter database"
7. Summary - Click on "Install"
8. Execute /u01/app/oracle/product/10.1.0/db_1/root.sh script from another window as the root user on the first node. Do not run root.sh on the second node until the VIP configuration completes.
9. VIPCA Welcome - Click on "Next"
10. VIP Configuration Assistance, 1 of 2: Network Interfaces:
a. Select "eth0"
11. VIP Configuration Assistance, 2 of 2: Virtual IPs for cluster nodes:
a. Node Name: salmon1
b. IP Alias Name: salmon1-vip
c. IP address: 192.168.0.186
d. Subnet Mask: 255.255.255.0
e. Node Name: salmon2
f. IP Alias Name: salmon2-vip
g. IP address: 192.168.0.187
h. Subnet Mask: 255.255.255.0
12. Summary- Click on "Finish"
13. Configuration Assistant Progress Dialog - Click on "OK"
14. Configuration Results - Click on "Exit"
15. Execute /u01/app/oracle/product/10.1.0/db_1/root.sh script from another window as the root user on the second node.
16. Click on "OK" after the root.sh has run on both nodes.
17. End of Installation - Click on "Exit"
4c. Configure Oracle Listener.
The Network Configuration Assistant (NETCA) should only be launched and configured on one node. At the end of the configuration process, the NETCA starts up the Oracle listener on both nodes.
[oracle@salmon1]$ netca
1. Oracle Net Configuration Assistant: Real Application Clusters, Configuration:
a. Select "Cluster configuration"
2. Oracle Net Configuration Assistant: TOPSNodes:
a. Click "Select all nodes"
3. Oracle Net Configuration Assistant: Welcome:
a. Select "Listener configuration"
4. Oracle Net Configuration Assistant: Listener Configuration, Listener:
a. Select "Add"
5. Oracle Net Configuration Assistant: Listener Configuration, Listener Name:
a. Listener Name: LISTENER
6. Oracle Net Configuration Assistant: Listener Configuration, Select Protocols
a. Selected Protocols: TCP
7. Oracle Net Configuration Assistant: Listener Configuration, TCP/IP Protocol:
a. Select "Use the standard port number of 1521"
8. Oracle Net Configuration Assistant: Listener Configuration, More Listeners?
a. Select "No"
9. Oracle Net Configuration Assistant: Listener Configuration Done:
a. Click on "Next"
10. Oracle Net Configuration Assistant: Welcome
a. Select "Naming Methods configuration"
b. Click on "Next"
11. Oracle Net Configuration Assistant: Naming Methods Configuration, Select Naming:
a. Select "Local Naming"
12. Oracle Net Configuration Assistant: Naming Methods Configuration Done:
a. Click on "Next"
13. Oracle Net Configuration Assistant: Welcome
a. Click on "Finish"
4d. Verify status of services.
On node 1:
[oracle@salmon1]$ srvctl status nodeapps -n salmon1
VIP is running on node: salmon1
GSD is running on node: salmon1
Listener is running on node: salmon1
ONS daemon is running on node: salmon1
On node 2:
[oracle@salmon2]$ srvctl status nodeapps -n salmon2
VIP is running on node: salmon2
GSD is running on node: salmon2
Listener is running on node: salmon2
ONS daemon is running on node: salmon2
4e. listener.ora file
On node 1:
[oracle@salmon1]$ more $ORACLE_HOME/network/admin/listener.ora

LISTENER_SALMON1 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon1-vip)(PORT = 1521))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.184)(PORT = 1521))
)
)
)

SID_LIST_LISTENER_SALMON1 =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.1.0/db_1)
(PROGRAM = extproc)
)
)
On node 2:
[oracle@salmon2]$ more $ORACLE_HOME/network/admin/listener.ora

LISTENER_SALMON2 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon2-vip)(PORT = 1521))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.185)(PORT = 1521))
)
)
)

SID_LIST_LISTENER_SALMON2 =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.1.0/db_1)
(PROGRAM = extproc)
)
)
4f. tnsnames.ora file
On both nodes:
[oracle@salmon1]$ more $ORACLE_HOME/network/admin/tnsnames.ora
LISTENERS_PROD1 =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon2-vip)(PORT = 1521))
)

PROD1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVICE_NAME = PROD1)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 200)
(DELAY = 5)
)
)
)
PROD1A =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon1-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = PROD1)
(INSTANCE_NAME = PROD1A)
)
)

PROD1B =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = salmon2-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = PROD1)
(INSTANCE_NAME = PROD1B)
)
)

________________________________________
Step 5: Post Installation
5a. Relocate the ASM instance files on the first node.
Copy the ASM instance password file and spfile+ASM1A.ora to the new Oracle Home.
[oracle@salmon1]$ cp /u01/app/oracle/product/10.1.0/dbs/orapw+ASM1A
/u01/app/oracle/product/10.1.0/db_1/dbs
[oracle@salmon1]$ cp /u01/app/oracle/product/10.1.0/dbs/spfile+ASM1A.ora
/u01/app/oracle/product/10.1.0/db_1/dbs
5b. Create init.ora for the second ASM instance.
On the second node, create an init+ASM1B.ora file in $ORACLE_HOME/dbs.
asm_diskgroups='DG1', ‘DG2’, ‘RECOVERYDEST’
asm_diskstring='ORCL:VOL*'
background_dump_dest=/u01/app/oracle/admin/+ASM/bdump
core_dump_dest=/u01/app/oracle/admin/+ASM/cdump
user_dump_dest=/u01/app/oracle/admin/+ASM/udump
instance_type=asm
large_pool_size=16M
remote_login_passwordfile=exclusive
+ASM1A.instance_number=1
+ASM1B.instance_number=2
5c. Create password file for the second ASM instance.
Using the orapwd utility, create an orapw+ASM1B file in $ORACLE_HOME/dbs on the second node.
5d. Create the second ASM instance.
Create the second ASM instance on the second node.
[oracle@salmon2]$ export ORACLE_SID=+ASM1B
[oracle@salmon2]$ sqlplus / as sysdba

SQL*Plus: Release 10.1.0.3.0 - Production on Thu May 26 18:43:14 2005

Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 104857600 bytes
Fixed Size 777616 bytes
Variable Size 104079964 bytes
Database Buffers 0 bytes
Redo Buffers 0 bytes
ASM diskgroups mounted
SQL> create spfile from pfile;

File created.

SQL> shutdown immediate;
5e. Register the ASM instances with CRS.
For higher availability, register the ASM instances under the CRS framework. When registered, the CRS should detect any failed instances and automatically attempt to start up the instances. The CRS should also automatically start up the instances when the servers are rebooted.
On node 1:

[oracle@salmon1]$ srvctl add asm -n salmon1 -i +ASM1A -o /u01/app/oracle/product/10.1.0/db_1
[oracle@salmon1]$ srvctl start asm -n salmon1
[oracle@salmon1]$ srvctl status asm -n salmon1
ASM instance +ASM1A is running on node salmon1.
On node 2:
[oracle@salmon2]$ srvctl add asm -n salmon2 -i +ASM1B -o /u01/app/oracle/product/10.1.0/db_1
[oracle@salmon2]$ srvctl start asm -n salmon2
[oracle@salmon2]$ srvctl status asm -n salmon2
ASM instance +ASM1B is running on node salmon2.

[oracle@salmon1]$ ps -ef | grep dbw | grep -v grep
oracle 8918 1 0 May 26 ? 00:00:01 asm_dbw0_+ASM1A
[oracle@salmon1]$ kill -9 8918
[oracle@salmon1]$ srvctl status asm –n salmon1
ASM instance +ASM1A is not running on node salmon1.
[oracle@salmon1]$ ps -ef | grep dbw | grep -v grep
[oracle@salmon1]$ srvctl status asm –n salmon1
ASM instance +ASM1A is running on node salmon1.
[oracle@salmon1]$ ps -ef | grep dbw | grep -v grep
oracle 8920 1 0 May 26 ? 00:00:01 asm_dbw0_+ASM1A
5f. Create password files for both RAC instances.
Using the orapwd utility, create an orapwprod1a file in $ORACLE_HOME/dbs on the first node and an orapwprod1b file in $ORACLE_HOME/dbs on the second node.
5g. Add RAC specific parameters in pfile.
Modify the pfile saved in Step 2n. Add and modify the following parameters:
*.cluster_database_instances=2
*.cluster_database=true
*.remote_listener='LISTENERS_PROD1’
prod1a.thread=1
prod1a.instance_number=1
prod1a.undo_tablespace='UNDOTBS1'
prod1b.thread=2
prod1b.instance_number=2
prod1b.undo_tablespace='UNDOTBS2'
5h. Modify init.ora of both RAC instances.
On node 1:
[oracle@salmon1]$ more $ORACLE_HOME/dbs/initprod1a.ora
spfile='+DG1/spfileprod1.ora'
On node 2:
[oracle@salmon2]$ more $ORACLE_HOME/dbs/initprod1b.ora
spfile='+DG1/spfileprod1.ora'
5i. Migrate database spfile to ASM.
SQL> !echo $ORACLE_SID
prod1a

SQL> connect / as sysdba
Connected to an idle instance.
SQL> startup pfile=/tmp/tmppfile.ora
ORACLE instance started.

Total System Global Area 419430400 bytes
Fixed Size 779416 bytes
Variable Size 128981864 bytes
Database Buffers 289406976 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.

SQL> create spfile='+DG1/spfileprod1.ora' from pfile='/tmp/tmppfile.ora';

File created.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 419430400 bytes
Fixed Size 779416 bytes
Variable Size 128981864 bytes
Database Buffers 289406976 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.
SQL> show parameter spfile

NAME TYPE VALUE
---------------------- ------------ ---------------------------
spfile string +DG1/spfileprod1.ora
5j. Create RAC Data Dictionary Views.
Create the RAC data dictionary views on the first RAC instance.
SQL> !echo $ORACLE_SID
prod1a

SQL> spool /tmp/catclust.log
SQL> @$ORACLE_HOME/rdbms/admin/catclust
...
...
...
SQL> spool off
SQL> shutdown immediate;
5k. Register the RAC instances with CRS.
On node 1:
[oracle@salmon1]$ srvctl add database -d prod1 -o $ORACLE_HOME
[oracle@salmon1]$ srvctl add instance -d prod1 -i prod1a -n salmon1
[oracle@salmon1]$ srvctl add instance -d prod1 -i prod1b -n salmon2
[oracle@salmon1]$ srvctl start instance -d prod1 -i prod1a
5l. Create redo logs for the second RAC instance.
SQL> connect sys/sys@prod1a as sysdba
Connected.
SQL> alter database add logfile thread 2 group 3 size 100M;
SQL> alter database add logfile thread 2 group 4 size 100M;
SQL> select group#, member from v$logfile;

GROUP# MEMBER
--------------- ----------------------------------------
1 +DG1/prod1/onlinelog/group_1.265.3
1 +DG2/prod1/onlinelog/group_1.257.1
2 +DG1/prod1/onlinelog/group_2.266.3
2 +DG2/prod1/onlinelog/group_2.258.1
3 +DG1/prod1/onlinelog/group_3.268.1
3 +DG2/prod1/onlinelog/group_3.259.1
4 +DG1/prod1/onlinelog/group_4.269.1
4 +DG2/prod1/onlinelog/group_4.260.1

8 rows selected.

SQL> alter database enable thread 2;

Database altered.
5m. Create undo tablespace for the second RAC instance.
SQL> create undo tablespace UNDOTBS2 datafile size 200M;

SQL> select tablespace_name, file_name from dba_data_files
2 where tablespace_name=’UNDOTBS2’;

TABLESPACE FILE_NAME
--------------------- --------------------------------------
UNDOTBS2 +DG1/prod1/datafile/undotbs2.270.1
5n. Start up the second RAC instance.
[oracle@salmon1]$ srvctl start instance -d prod1 -i prod1b
[oracle@salmon1]$ crs_stat -t
Name Type Target State Host
-----------------------------------------------------------------------
ora....1a.inst application ONLINE ONLINE salmon1
ora....1b.inst application ONLINE ONLINE salmon2
ora.prod1.db application ONLINE ONLINE salmon1
ora....M1A.asm application ONLINE ONLINE salmon1
ora....M1B.asm application ONLINE ONLINE salmon2
ora....N1.lsnr application ONLINE ONLINE salmon1
ora....on1.gsd application ONLINE ONLINE salmon1
ora....on1.ons application ONLINE ONLINE salmon1
ora....on1.vip application ONLINE ONLINE salmon1
ora....N2.lsnr application ONLINE ONLINE salmon2
ora....on2.gsd application ONLINE ONLINE salmon2
ora....on2.ons application ONLINE ONLINE salmon2
ora....on2.vip application ONLINE ONLINE salmon2
[oracle@salmon1]$ srvctl status database -d prod1
Instance prod1a is running on node salmon1
Instance prod1b is running on node salmon2
[oracle@salmon1]$ srvctl stop database -d prod1
[oracle@salmon1]$ srvctl start database -d prod1
[oracle@salmon1]$ sqlplus system/system@prod1

SQL*Plus: Release 10.1.0.3.0 - Production on Fri May 27 05:53:21 2005

Copyright (c) 1982, 2004, Oracle. All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> select instance_number instance#, instance_name, host_name, status
2 from gv$instance;

INSTANCE# INSTANCE_NAME HOST_NAME STATUS
----------------- ----------------- ------------------------- -----------
1 prod1a salmon1.dbsconsult.com OPEN
2 prod1b salmon2.dbsconsult.com OPEN
Congratulations, you have converted your single-instance database to RAC!

________________________________________
Step 6: Test Transparent Application Failover (TAF)
The failover mechanism in Oracle TAF enables any failed database connections to reconnect to another node within the cluster. The failover is transparent to the user. Oracle re-executes the query on the failed over instance and continues to display the remaining results to the user.
To demonstrate the failover mechanism of the TAF option, connect to two different database sessions and execute these steps:
6a. Connect the first session using the PROD1 service.
If the returned output of failover_type and failover_mode is 'NONE', verify that the PROD1 service is configured correctly in tnsnames.ora.
SQL> connect system/system@prod1 as sysdba
Connected.

SQL> select instance_number instance#, instance_name, host_name, status
2 from v$instance;

INSTANCE# INSTANCE_NAME HOST_NAME STATUS
----------------- -------------- ------------------------- ----------------
1 prod1a salmon1.dbsconsult.com OPEN

SQL> select failover_type, failover_method, failed_over
2 from v$session where username='SYSTEM';

FAILOVER_TYPE FAILOVER_MODE FAILED_OVER
------------------------- -------------------- ----------------------
SELECT BASIC NO
6b. Shut down the instance from another session.
Connect as the sys user on prod1a instance and shut down the instance.
SQL> connect sys/sys@prod1a as sysdba
Connected.

SQL> select instance_number instance#, instance_name, host_name, status
2 from v$instance;

INSTANCE# INSTANCE_NAME HOST_NAME STATUS
----------------- ---------------- ------------------------ ----------------
1 prod1a salmon1.dbsconsult.com OPEN

SQL> shutdown abort;
ORACLE instance shut down.
6c. Verify that the session has failed over.
From the same session in Step 6a, execute the queries below to verify that the session has failed over to another instance.
SQL> select instance_number instance#, instance_name, host_name, status
2 from v$instance;

INSTANCE# INSTANCE_NAME HOST_NAME STATUS
----------------- ---------------- ------------------------ ----------------
2 prod1b salmon2.dbsconsult.com OPEN

SQL> select failover_type, failover_method, failed_over
2 from v$session where username='SYSTEM';

FAILOVER_TYPE FAILOVER_MODE FAILED_OVER
----------------- -------------- ----------------------
SELECT BASIC YES

Sunday, July 26, 2009

Oracle 10g Laptop/nodeless RAC Howto

Oracle 10g Laptop/nodeless RAC Howto
This procedure is only for experimaental purpose for testing RAC features.

I tried it on a Desktop it works
=====================

Oracle's model of clustering involves multiple instances (software processes)talking to a single database (physical datafiles). .
Let's get started, what will we need to do? Here's a quick outline of thesteps involved:
1. setup ip addresses of the virtual servers
2. setup ssh and rsh with autologin configured
3. setup the raw devices Oracle's ASM software will use
4. install the clusterware softare, and then Oracle's 10g software
5. setup the listener and an ASM instance
6. create an instance, start it, and register with srvct
7. create a second instance & undo tablespace, & register it

1. Setup IP Addresses
----------------------
Oracle wants to have a few interfaces available to it. To follow our analogyof a hitchhiker traveling across America, we'll name our server route66.

So add that name to your /etc/hosts/ file along with the private and vipnames:
192.168.0.19 route66
192.168.0.75 route66-priv
#192.168.0.76 route66-vip

Notice that we've commented out route66-vip. We'll explain more aboutthis later, but suffice it to say now that the clusterware installeris very finicky about this.

In order for these two additional names to be reachable, we needethernet devices to associate with those IPs.
It's a fairly straightforwardthing to create with ifconfig as follows:

$ /sbin/ifconfig eth0:1 192.168.0.75 netmask 255.255.255.0 broadcast 192.168.0.255$ /sbin/ifconfig eth0:2 192.168.0.76 netmask 255.255.255.0 broadcast 192.168.0.255

If your IPs, or network is configured differently, adjust the IP or broadcastaddress accordingly.

2. setup ssh and rsh with autologin
------------------------------------
Most modern Linux systems do *NOT* come with rsh installed. That's forgood reason, because it's completely insecure, and shouldn't be used atall. Why Oracle's installer requires it is beyond me, but you'll needit. You can probably disable it once the clusterware is installed.
Head over to http://rpmfind.net and see if you can find a copy foryour distro. You might also have luck using up2date or yumm if youalready have those configured, as they handle dependencies, and alwaysdownload the *right* version. With rpm, install this way:

$ rpm Uvh rsh-server-0.17-34.1.i386.rpm$ rpm Uvh rsh-0.17-34.1.i386.rpm
Next enable autologin by adding names to your /home/oracle/.rhosts file.After starting rsh, you should be able to login as follows:
$ rsh route66-priv

Once that works, move on the the sshd part. Most likely ssh is already onyour system, so just start it (as root):
$ /etc/rc.d/init.d/sshd start
Next, as the "oracle" user, generate the keys:
$ ssh-keygen -t dsa
Normally you would copy id_dsa.pub to a remote system, but for us wejust want to login to self. So copy as follows:

$ cd .ssh
$ cp id_dsa.pub authorized_keys
$ chmod 644 authorized_keys

Verify that you can login now:
$ ssh route66-priv
3. setup the raw devices
-------------------------
Most of the time when you think of files on a Unix system, you'rethinking of files as represented through a filesystem. A filesystemprovides you a way to interact with the underlying disk hardwarethrough the use of files. A filesystem provides buffering, toimprove I/O performance automatically. However in the case of adatabase like Oracle, it already has a sophisticated mechanism forbuffering which is smart in that it knows everything about it'sfiles, and how it wants to read and write to them. So for anapplication like Oracle unbuffered I/O is ideal. It bypasses awhole layer of software, making your overall throughput faster!You achieve this feat of magic using raw devices. We're going tohand them over to Oracle's Automatic Storage Manager in a minutebut first let's get to work creating the device files for ourRAC setup.
Create three 2G disks. These will be used as general storage spacefor our ASM instance:
$ mkdir /asmdisks
$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=2000
$ dd if=/dev/zero of=/asmdisks/disk2 bs=1024k count=2000
$ dd if=/dev/zero of=/asmdisks/disk3 bs=1024k count=2000

Create two more smaller disks, one for the Oracle Cluster Registry,and another for the voting disk:
$ dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100
$ dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20

Now we use a loopback device to make Linux treat these FILES asraw devices.
$ /sbin/losetup /dev/loop1 /asmdisks/disk1
$ raw /dev/raw/raw1 /dev/loop1
$ chown oracle.dba /dev/raw/raw1
You'll want to run those same three commands on disk2 through disk5 now.

4. Install the Clusterware & Oracle's 10g Software
--------------------------------------------------
Finally we're done with the Operating System setup, and we can move onto Oracle. The first step will be to install the clusterware. I'lltell you in advance that this was the most difficult step in the entireRAC on a laptop saga. Oracle's installer tries to *HELP* you all alongthe way, which really means standing in front of you!

First let's make a couple of symlinks to our OCR and voting disks:
$ ln -sf /dev/raw/raw4 /home/oracle/product/disk_ocr
$ ln -sf /dev/raw/raw5 /home/oracle/product/disk_vot

As with any Oracle install, you'll need a user, and group alreadycreated, and you'll want to set the usual environment variables suchas ORACLE_HOME, ORACLE_SID, etc. Remember that previous to this pointyou already have ssh and rsh autologin working. If you're not surego back and test again. That will certainly hold you up here, andgive you all sorts of confusing error messages.
If you're running on an uncertified version of Linux, you may wantto fire up the clusterware installer as follows:

$ ./runInstaller -ignoreSysPrereqs

If your Linux distro is still giving you trouble, you might trydownloading from centos.org where you can find complete ISOs forRHEL, various versions. You can also safely ignore memory warningsduring startup. If you're short on memory, it will certainly slow thingsdown, but we're hitchhikers right?

You'll be asked to specify the cluster configuration details. You'llwant route66-vip to be commented out, so if you haven't done that andget an error to the affect of route66-vip already in use go ahead and edit your /etc/hosts file.
I also got messages saying "route66-priv not reachable". Check againthat sshd is running, and possibly disable your firewall rules:
$ /etc/rc.d/init.d/iptables stop
Also verify that eth0:1, and eth0:2 are created. Have you rebootedsince you created them? Be sure they're still there with:
$ /sbin/ifconfig -a
Specify the network interface. This defaults to PRIVATE, just editand specify PUBLIC.
The next two steps ask for the OCR disk and voting disk. Be sure tospecify external redundancy. This is your way of telling Oracle thatyou'll take care of mirroring these important disks yourself, as lossof either of them will get you in deep doodoo. Of course we'rehitchhikers so we're not trying to build a system that is never goingto breakdown, but rather we want to get the feeling of the wind blowingin our hair. Click through to install and you should be in good shape.At the completion, the installer will ask you to run the root.shscript. I found this worked fine up until the vipca (virtual ipconfiguration assistant). I then ran this one manually. You'll needto uncomment route66-vip from your /etc/hosts file as well. Onceall configuration assistants have completed successfully, return tothe installer and click continue, and it will do various other sanitychecks of your cluster configuration.
Since the clusterware install is rather testy, you'll probably be doingit a few times before you get it right.

Saturday, July 11, 2009


Oracle Real Application Clusters is a resource-sharing system that increases availability and performance by distributing the workload across multiple nodes.


Oracle Real Application Clusters allows database files to be accessed from multiple instances running on different nodes of a cluster. These nodes are connected to each other by a high speed interconnect.

Each machine or node performs database processing, and all nodes share access to the same database.


This configuration can be used on systems that share resources such as disks, and that have very fast communication between machines or nodes.Failure of one node does not make data inaccessible for all users; the system ensures continued data availability.

=====================================
Let's begin with a brief overview of RAC architecture.
A cluster is a set of 2 or more machines (nodes) that share or coordinate resources to perform the same task.
A RAC database is 2 or more instances running on a set of clustered nodes, with all instances accessing a shared set of database files.
Depending on the O/S platform, a RAC database may be deployed on a cluster that uses vendor clusterware plus Oracle's own clusterware (Cluster Ready Services), or on a cluster that solely uses Oracle's own clusterware.

Thus, every RAC sits on a cluster that is running Cluster Ready Services. srvctl is the primary tool DBAs use to configure CRS for their RAC database and processes.
Cluster Ready Services, or CRS, is a new feature for 10g RAC. Essentially, it is Oracle's own clusterware. On most platforms, Oracle supports vendor clusterware; in these cases, CRS interoperates with the vendor clusterware, providing high availability support and service and workload management.
On Linux and Windows clusters, CRS serves as the sole clusterware. In all cases, CRS provides a standard cluster interface that is consistent across all platforms.
CRS consists of four processes (crsd, occsd, evmd, and evmlogger) and two disks: the Oracle Cluster Registry (OCR), and the voting disk.
CRS manages the following resources:
The ASM instances on each node
Databases
The instances on each node
Oracle Services on each node

The cluster nodes themselves, including the following processes, or "nodeapps":
VIP
GSD
The listener
The ONS daemon

CRS stores information about these resources in the OCR. If the information in the OCR for one of these resources becomes damaged or inconsistent, then CRS is no longer able to manage that resource. Fortunately, the OCR automatically backs itself up regularly and frequently.
Interacting with CRS and the OCR:
srvctl is the tool Oracle recommends that DBAs use to interact with CRS and the cluster registry. Oracle does provide several tools to interface with the cluster registry and CRS more directly, at a lower level, but these tools are deliberately undocumented and intended only for use by Oracle Support. srvctl, in contrast, is well documented and easy to use. Using other tools to modify the OCR or manage CRS without the assistance of Oracle Support runs the risk of damaging the OCR.

2) Important Components in RAC
a)Node
b)Interconnect
c)Shared Storage
d)Clusterware


3) Interconnect


The interconnect links the nodes together, RAC needs a high speed interconnect network for cluster communication and cache fusion. The interconnect must be low latency.Some of the interconnect used in RAC are, Gigabit Ethernet, High Speed Switch, Memory Channels or Infiniband (IB).


4) Shared Disk Storage.


RAC requires shared disk access to the following files

controlfiles, datafiles, redolog files, tempfile, undofile, OCR and Voting disk etc.


To be able to read/write by all members in a cluster at the same time a shared disk storage system must be used. Available options in this case are:


Raw volumes: These are directly attached raw devices that require storage that operates in block mode such as fiber channel or iSCSI.


NFS attached storage: Network file storage can be used in a supported configuration to provide a shared repository for all RAC database files, preferably through a high-speed private network. For example, Netapp Filer offers CFS functionality via NFS to the server machines. These file systems are mounted by using special mount options.Cluster File System: One or more cluster file systems can be used to hold all RAC files. Cluster file systems require block mode storage such as fiber channel or iSCSI. It can not be used on top of NAS (NFS), (some options: AIX GPFS, RedHat GFS,OCFS2,Veritas Storage Solution)


Automatic Storage Management (ASM): is a portable, dedicated, and optimized storage for Oracle database files.


Storage Area Network (SAN): is a shared dedicated high-speed network connectingstorage elements and the backend of the servers.


5) Clusterware


To make sure Real Application Cluster can work properly on the nodes as a single database, Clusterware must be installed. Before Oracle 10g vendor Clusterware was needed except for windows and Linux. With the introduction of Oracle 10g, oracle introduce Oracle Clusterware it offers a complete, integrated Clusterware management solution on all platforms Oracle Database 10g runs on.

This clusterware functionality includes mechanisms for cluster connectivity, messaging and locking, cluster control and recovery, and a services provisioning framework. No 3rd party clusterware management software need be purchased. But if wanted it can still be used, but still Oracle Clusterware needs to be installed.

6)RAC Software Principles for Oracle 10g
A few additional background processes associated with a RAC instance that are not there for a single-instance database. These processes are primarily used to maintain database coherency among each instance. They manage what is called the global resources:LMON: Global Enqueue Service MonitorLMD0: Global Enqueue Service DaemonLMSx: Global Cache Service Processes, where x can range from 0 to 10LCK0: Lock processDIAG: Diagnosibility processAt the cluster level, you find the main processes of the Clusterware software. They provide a standard cluster interface on all platforms and perform high-availability operations. You find these processes on each node of the cluster:

How to Clean Up After a Failed Oracle Clusterware (CRS) Installation

Source:-
Please refers to steps in Metalink Note:
239998.1
Broad Level Steps:-
=================
rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/*
rm -f /etc/inittab.crs
cp /etc/inittab.orig /etc/inittab
rm -rf /oracrs/oracle/product
rm -rf /oracrs/oracle/oraInventory

Clusterware installation Steps:

http://onlineappsdba.com/index.php/2009/06/06/oracle-rac-clusterware-installation-overview/

Listener Configuration for RAC LOCAL_LISTENER REMOTE_LISTENER

The TNS listener (tnslsnr process on *nix) process listens on a specific network address for connection requests to one of the services from one of the database instances that it services. When requested, it either spawns a server process (dedicated server environment) and connects the user to that process or forwards the connection request to a dispatcher (shared server environment) for service to the database service requested.

Alternatively, if the listener knows of more than one instance providing the requested service, it may direct the client to an alternate listener (usually on a different node) that will service the request.

In any Oracle database configuration, listeners define the instances as local or remote (in single-instance environments, normally everything is local). You can see this behavior when examining the “lsnrctl services ” output (lsnrctl syntax reference here). A listener’s services are those services that have been registered with it by instances. A listener will accept registration from any instance (this may be a weak point of security, but that’s another topic) and listeners have no outbound communication with any other entity in the Oracle environment (or beyond).

The remote_listener parameter specifies a list of listening endpoints that the local instance should contact to register its services. This list is usually defined in a TNS entry in the tnsnames.ora file and then the TNS alias set as the value of the remote_listener parameter. Here’s a sample of what that entry might look like:
LISTENERS_CLUSTERNAME= (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip)(PORPORT=1521)) ) (ADDRESS=(PROTOCOL=TCP)(HOST=node3-vip)(PORT=1521)) )

The local_listener parameter is sometimes confusing. It defines where to connect to the local instance, but its most important function is related to remote listeners. The contents of the local_listener parameter are passed along to the remote listeners during remote registration so that when those remote listeners wish to refer a connection request to the local instance, they refer the client (requestor) to the proper listening endpoint so it can get connected.

The local_listener should contain the ADDRESS section of the TNS entry and the HOST portion should reference the VIP address, like this:
(ADDRESS=(PROTOCOL=TCP)(HOST=node3-vip)(PORT=1521))

To properly (manually) configure listeners in a RAC environment, follow the steps like the ones below. Note that in most cases, the Oracle Net Configuration Assistant (netca) will do this for you as part of the database creation process.

Create individual listener.ora files for each listener. Make sure that the HOST= lines in the listener.ora definition reference the VIP addresses (and only the VIP address). I prefer to specify IP addresses instead of hostnames or DNS names here to avoid possible lookup issues and/or confusion.

Create a TNS entry (on each node) that looks like the one below to specify a single TNS entry that references all the listeners in the cluster. Note that the HOST= parts reference the VIP addresses of each node (I used names instead of IP addresses here to avoid reader confusion–I’d put in IP addresses in the HOST= attributes when using this for a real configuration.). 3. LISTENERS_CLUSTERNAME = (ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521)) )

Set the remote_listener parameter in the instances (a global parameter, not an instance-specific parameter) to be the name of the TNS entry you created in the previous step. This is done with “alter system set remote_listener = 'LISTENERS_CLUSTERNAME'; “

Set the local_listener parameter to be the ADDRESS string for the local instance. This parameter must be an instance-specific parameter with each instance having a similar, but unique value since each instance runs on a different HOST. If the local instance (called inst1 in the example here) runs on a node with the node VIP of 10.3.121.54, then set the local_listener parameter accordingly for each instance (it is instance-specific, so use the sid= syntax):
alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=10.3.121.54)(PORT=1521))' sid='inst1';

On each instance, you can run “alter system register;” to force immediate registration with the listeners. If you don’t do this, the listener registration will usually be updated within a minute or two anyway (automatically), but this command can help shorten debugging cycles when necessary.

Friday, July 10, 2009

Oracle Clusterware and network bond for redundancy i.e bond0,bond2 etc

The Private and Public interface used in case of RAC 10g should have redundancy
in network interface so that in case a network interface fails the backup interface can take over.

To create a channel bonding interface, first create a file in the /etc/sysconfig/network-scripts/ directory.

You can copy an existing ifcfg-eth# to ifcfg-bond#, where # is a number for the interface.

In general # start with 0. Update the ifcfg-bond#,
change the DEVICE= directive must be bond#.

When the bonding device config file is created you need to update the interfaces which you want to bond and add the master=bond# to the interface configuration fileas well as add slave=yes.

Remove the IP adress information. Repeat this step for 2 interfaces.Below is an example of the configuration file of the interfaces.


Example: interface bond0
GATEWAY=192.168.100.1
DEVICE=bond0
BONDIG_OPTS="mode=1 miimon=500
"BOOTPROTO=none
NETMASK=255.255.255.0
IPADDR=192.168.100.10
ONBOOT=yesUSERCTL=no
Example: interface ifcfg-eth2
GATEWAY=192.168.100.1
TYPE=Ethernet
DEVICE=eth2
BOOTPROTO=none
NETMASK=255.255.255.0
MASTER=bond0 <=== here the bond0 is defined
SLAVE=yes <== here the interface role is defined
ONBOOT=yes
USERCTL=no
IPV6
INIT=no
PEERDNS=yes
Beside the interface configuration you need to update the /etc/modprobe.conf and add the following line:alias bond0 bondinginstall bond0 /sbin/modprobe bonding -o bonding0 mode=1 miimon=500
When the above steps are performed you can start the interfaces and check the result.ifup bond0ifup eth0ifup eth2
cat /proc/net/bonding/bond0
example output:
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)
Bonding Mode: load balancing (round-robin)MII Status: upMII Polling Interval (ms): 0Up Delay (ms): 0Down Delay (ms): 0
Slave Interface: eth0MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:8a:5e:21
Slave Interface: eth2MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:8a:5e:0d

Wednesday, July 8, 2009

Viewing definition of an object using DBMS_METADATA

set long 50000
set pagesiz 50

select dbms_metadata.get_ddl('PACKAGE', 'ZPB_BUILD_METADATA', 'APPS') from dual;
select dbms_metadata.get_ddl('PACKAGE_BODY', 'ZPB_BUILD_METADATA', 'APPS') from dual;

also using

select text from dba_source where name='' and owner='';

ipcrs ipcrm shmid semid

When you are forced to terminate Oracle on a UNIX server, you must perform the following steps:

1)Kill all Oracle processes associated with the ORACLE_SID

2)Use the ipcs –pmb command to identify all held RAM memory segments.

3)ipcs -a gives the following information
a)------ Shared Memory Segments --------
b)------ Semaphore Arrays --------

4)ipcrm -m shmid removes the shared memory segments.

5)ipcrm -s semid removes the semaphores

6)we can create a single command to terminate all Oracle processes associated with your hung database instance.
root> ps -efgrep $ORACLE_SID \grep -v grepawk '{print $2}'xargs -i kill -9 {}

Tuesday, July 7, 2009

The Voting disk is one of the required parts of the Oracle Cluster environment.
Another name for the Voting disk is quorum disk.

The voting disk is used to determine quorum in case of failure and provides a second heartbeat mechanism to validate cluster health. The Disk heartbeat is maintained in the voting disk.

The voting disk is used to ascertain cluster state.

If a node eviction needs to take place the voting disk is updated with the "eviction message".

If we look at the structure of the voting disk, each node of the cluster has his own part in the voting disk. When a new node is added it will add the information to a new part of the voting disk.

From Oracle 10g release 2 you can define multiple voting disks. Where in Oracle 10g release 1, you need to make sure the voting disk is mirrored to make sure the voting disk is not the single point of failure.

How to add, remove or backup the voting disk see:
http://download-east.oracle.com/docs/cd/B19306_01/rac.102/b14197/votocr.htm

Cluster Registry is one of the required components in an Oracle Cluster environment.

It is a registry which contains all the information about the cluster environment. You need to think of node names, ip addresses, an application resources like listener,vip, gsd but also the databases/instance. Also the parameters like need to startup, dependencies are stored in the OCR.


The OCR is created during the CRS installation when the root.sh script is executed. When root.sh is executed it will read the ocr.loc file which is create during installation and pointing the the OCR file /device.

To make sure all the nodes in the cluster can read the OCR the ocr location must be on shared storage.

The location of the ocr.loc depended on the platform used.
Linux: /etc/oracle/ocr.loc
Aix: /etc/oracle/ocr.loc
Solaris : /var/opt/oracle
Windows : HKEY_LOCAL_MACHINE\SOFTWARE\Oracle\OCR

If we look in the ocr.loc file we see the following.

bash-3.00$ cat /etc/oracle/ocr.loc ocrconfig_loc=/oracrs/oradata/data01/ocrdisk1

ocrmirrorconfig_loc=/oracrs/oradata/data02/ocrdisk2
local_only=FALSE

The value of local_only = true indicates "Single instance only" and false means using "RAC"


The orc.loc is the location where the CRS stack will check for the OCR during startup. When the OCR is found it will be read for the voting disk location and the other information. If for some reason the orc.loc or the location in the ocr.loc is not available the cluster will not be started.

From oracle 10g release 2, it is possible to define more OCR locations (mirroring).
Clients of the OCR are srvctl, css, crs, dbua, vipca and em
.

Tools which can be used:
ocrconfig - configuration tool for Oracle Cluster Registry
ocrdump – utility to dump the contents of the OCR in a file.
ocrcheck – utility to verify the OCR integrity.

Ocrconfig:
http://download-uk.oracle.com/docs/cd/B19306_01/rac.102/b14197/ocrsyntax.htm#RACAD835

ocrdump:
http://download-uk.oracle.com/docs/cd/B19306_01/rac.102/b14197/appsupport.htm#sthref1123

ocrcheck:
http://download-uk.oracle.com/docs/cd/B19306_01/rac.102/b14197/appsupport.htm#BEHJIJIB

VIPCA
=========
Oracle Clusterware requires a virtual IP address for each node in the cluster.

This IP address must be on the same subnet as the public IP address for the node and should be an address that is assigned a name in the Domain Name Service, but is unused and cannot be pinged in the network before installation of Oracle Clusterware.

The VIP is a node application (nodeapp) defined in the OCR that is managed by Oracle Clusterware. The VIP is configured with the VIPCA utility. The root script calls the VIPCA utility in silent mode.


Why Oracle 10g has a VIP?

Protects database clients from long TCP IP timeouts (>10 minutes).

During normal operation works the same as hostname.

During failure it removes network timeout from connection request time ,client fails immediately to the next address in the list.

Oracle RAC Details

=================

About Virtual IP
Why is there a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node fails?
It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen.

The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately. This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used.
Going one step further is making use of Transparent Application Failover (TAF). With TAF successfully configured, it is possible to completely avoid ORA-3113 errors alltogether! TAF will be discussed in more detail in Section 28 ("Transparent Application Failover - (TAF)").
Without using VIPs, clients connected to a node that died will often wait a 10-minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution without using VIPs (Source - Metalink Note 220970.1).

Background Processes in a RAC database

LMS

This background process copy read consistent blocks from the holding instance buffer cache to the requesting instance. LMSn also performs rollback on uncommitted transactions for blocks that are being requested for consistent read by another instance.

This background process is also called Global Cache Services.
This is the name you often see back in wait events (GCS).
Default 2 LMS background processes are started.

LMON

This background process monitors the entire cluster database. LMON checks and manages instance deaths and perform recovery for the Global Cache Service//LMS.
Joining an leaving instances are managed by LMON. LMON manage also all the global resource in the RAC database. LMON register the instance/database with the node monitoring part of the cluster (CSSD).

This background process is also called Global Enqueue Monitoring.
LMON provide services are also referred to cluster group service (CGS).

LMD

This background process manage access to the blocks and global enqueues. Also global deadlock detection and remote resource request are handled by LMD. LMD also manage lock requests for GCS /LMS.

This background process is also called Global Enqueue Service Deamon. In wait events you will see GES.

DIAG

The diagnostic Daemon (DIAG) captures diagnostic data in case there is a process failure within the instance. The diag log can be used to investigate why there was a process failure.

LCK

The Lock Process is also for non-RAC environments LCK manage local noncache requests (row cache, lock requests, library locks). And it also managed shared resource requests cross instance. It keeps a list of invalid and valid lock elements. And if needed past information to the GCS.

Interconnect setup in RAC environment.

The interconnect is a very important part of the cluster environment it is on of the aorta’s of a cluster environment. The interconnect is used as physical layer between the cluster nodes to perform heartbeats as well as the cache fusion is using it. The interconnect must be a private connection. Cross over cable is not support.

In a day to day operation it is proven that when the interconnect is configured correctly the interconnect will not be the bottleneck in case of performance issues. In the rest of this article will be focus on the how to validate the interconnect is really used. A DBA must be able to validate the interconnect settings in case of performance problems. Out of scope is the physical attachment of the interconnect.

Although you should thread performance issues in a Cluster environment the way you would normally also do in no-cluster environments here some area’s you can focus on.
Normally the average interconnect latency using gigabit must be < 5ms.

Latency around 2ms are normal.

Solution Validation of the interconnect ?

There are several ways to validate the interconnect.

This can be down using the x$ksxpia table, by using oradebug, or using queries on gv$ views(not possible in Oracle 9i). Besides the queries it is also possible to validate the use of the interconnect from the alert.log of your instance. Below we will list the options how to use them. Available interfaces? This query shows all the interfaces which are know within the Oracle database instances.

The query will work on Oracle 10g and 11g, not in 9i.

set linesize 120
col name for a22
col ip_address for a15

select inst_id,name,ip_address,is_public from gv$configured_interconnects order by 1,2; This

set linesize 120
col name for a22
col ip_address for a15

select inst_id,name,ip_address,is_public from gv$cluster_interconnects order by 1,2;

SQL> connect / as sysdba
SQL> alter session set tracefile_identifier=oradebug_interc
SQL> oradebug setmypid
SQL> oradebug ipc
SQL> exit

Now if you open the tracefile, in the bdump location, you can find the IP address used for the interconnect. Here is the result of the above oradebug ipc command.

dmno 0x7902775e admport:
SSKGXPT 0x10569c44 flags SSKGXPT_READPENDING active network 0
info for network 0
socket no 7 IP 145.72.220.83 UDP 53032
HACMP network_id 0 sflags SSKGXPT_UP
context timestamp 0
no ports
sconno accono ertt state seq# sent async sync rtrans acks

Query x$ksxpia
The last option is to query the x$ksxpia, which is instance specific query. A query on this view is providing the information from which setting the interconnect is picked up. Depending on the environment this can be useful to indicate if an where the configuration went wrong. This query will work in Oracle 9i,10g and 11g.

Below is an example of the output.

col picked_ksxpia format a15
col indx format 9999
col name_ksxpia format a5
col ip_ksxpia format a20

select * from x$KSXPIA

ADDR INDX INST_ID PUB_KSXPIA PICKED_KSXPIA NAME_ IP_KSXPIA ---------------- ----- ---------- ------------------------------ --------------- ----- -------------------- 00000001105D6540 0 1 N OCR en7 145.72.220.83 00000001105D6540 1 1 Y OCR en6 145.72.220.10

Note: The pub_ksxpia indicate if the interface is a public or private one. The picked_ksxpia indicated from where the information is collect. In the example the interface information from the OCR is used. Other values are OSD which is meaning third party clusterware is used. And CI is also possible which means the cluster_interconnect parameter is set. (Last I recommend not to do).

Using Alert.log
Instead of using one or more of the above queries you can also check the alert.log of the instances involved. During the startup of the RDBMS instance the interface used for the public and private connect is mentioned. Note this start from Oracle 10g, and is not available in Oracle 9i.

Example: alert.log

Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.2.0 configured from OCR for use as a public interface

Using Oracle Cluster Registry to validate settings.
This method can be used to validate the settings in the OCR. But it does not mean it is also used by the database instances. To validate the settings of ocr you need to use the oifcfg command to retrieve the information from the OCR.

The oifcfg iflist list all the interfaces available on the Operating system and will not get this from the OCR.
The oifcfg getif list the configuration from the OCR.

An example:

racworkshop1:/export/home/oracle$ oifcfg iflist
eth0 192.168.2.0
eth1 10.10.10.0
racworkshop1:/export/home/oracle$ oifcfg getif
eth0 192.168.2.0 global public
eth1 10.10.10.0 global cluster_interconnect

Source: http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=35

Oracle Cluster Ready Services (CRS)

Oracle Database 10g contains many enhancements, but one of the most interesting is the introduction of Oracle Cluster Ready Services (CRS), Oracle's full-stack clusterware.

Oracle CRS is Oracle's own clusterware tightly coupled with Oracle Real Application Clusters (RAC).

CRS must be installed prior to the installation of Oracle RAC.

It can also work over any third-party clustering software but there is no longer a requirement to buy and deploy such software.

In short, Oracle CRS is primarily responsible for managing the high-availability (HA) architecture of Oracle RAC with the help of Cluster Ready Services Daemon (CRSD), Oracle Cluster Synchronization Server Daemon (OCSSD) and the Event Manager Daemon (EVMD).

The CRSD manages the HA functionality by starting, stopping, and failing over the application resources and maintaining the profiles and current states in the Oracle Cluster Registry (OCR).

The OCSSD manages the participating nodes in the cluster by using the voting disk. The OCSSD also protects against the data corruption potentially caused by "split brain" syndrome by forcing a machine to reboot.

Although Oracle CRS replaces the Oracle Cluster Manager (ORACM) in Oracle9i RAC, it does continue support for the Global Services Daemon (GSD), which in Oracle9i is responsible for communicating with the Oracle RAC database.
In Oracle 10g, GSD's sole purpose is to serve Oracle9i clients (such as SRVCTL, Database Configuration Assistant, and Oracle Enterprise Manager).

Source :-
http://download.oracle.com/docs/html/B10766_08/toc.htm

Monday, July 6, 2009

How does Oracle Clustware start?

Note#
=======
From 10.1.0.4
===============================
/etc/init.d/init.crs start
/etc/init.d/init.crs stop
/etc/init.d/init.crs enable
/etc/init.d/init.crs disable

starts crsd,cssd,evmd daemons.

From 10.2.X
===============================

crsctl stop crs
crsctl start crs
crsctl enable crs
crsctl disable crs

The startup of the Oracle clusterware daemons is based on start scripts that are executed as root user.

The environment variables are set in these scripts.
The clusterware start scripts can be found in:
· /etc/init.d (Sun, Linux)
· /sbin (HPUX, HP Itanium, Tru64)
· /etc (AIX)
They are named:
· init.cssd: starts ocssd.bin, oclsomon, oprocd and oclsvmon daemons
· init.evmd: starts evmd.bin daemon
. init.crsd: starts crsd.bin daemon
· init.crs : enabler/start/disabler script


Automatic startup of the clusterware daemons rely on these 3 steps (two Unix OS system mechanisms and a check of whether the clusterware is startable.

1. Execution of the init.d rc*.d scripts that enable the clusterware to start or not after a reboot

Note: The init.* scripts should never normally be run from the command line, if you do require manual disabling or enabling of CRS startup other than for the diagnostic purposes listed in this note then invoke crsctl as the root user as follows:

$ crsctl disable crs
Oracle Clusterware is disabled for start-up after a reboot.

$ crsctl enable crs
Oracle Clusterware is enabled for start-up after a reboot.

Running 'init.crs enable' allows the clusterware to autostart at system reboot ( this is the default
setting).

Running 'init.crs disable' (does the reverse) prevents the clusterware from autostarting at system reboot.

When the rc*.d scripts are executed at the correct OS runlevel, then the 'init.crs start' execution
(executed in run level 3 or 5 for the Unix versions) will check the automatic setting (enabled or
disabled) and set the clusterware in startable or non startable mode accordingly.



2. The inittab mechanism via the three respawnable clusterware scripts
A successful installation will populate the inittab file with entries to start the CRS stack:



3. The prerequisite check with the 'init.cssd startcheck' execution

The clusterware scripts further run a check script to know whether the clusterware is startable before launching the clusterware processes, that is, to check whether basic prerequisites are met and permit the clusterware to start :

init.cssd startcheck
That last script needs to return code 0 to permit the clusterware to start. In case of errors
/tmp/crsctl.xxxx logging files are written with the error message.

It executes as user oracle this command via a
'su -l oracle'
crsctl check boot


Execution of this command should return nothing in order to permit the clusterware to start.
Note: 'crsctl check boot' may return other errors such as:
'no read access to the ocr', 'clustered ip is not defined' or '$CRS_HOME is not mounted'
and prevent the clusterware from starting.
Once the above three prerequisites are met, the clusterware starts via *.bin executables which can be confirmed using the command:

'ps -ef grep .bin':
oracle 19611 19610 0 Dec17 /opt/app/oracle/product/crs/bin/oclsomon.bin
oracle 19547 18245 0 Dec17 /opt/app/oracle/product/crs/bin/ocssd.bin
oracle 18215 18005 0 Dec17 /opt/app/oracle/product/crs/bin/evmd.bin
root 18649 16555 0 Dec17 /opt/app/oracle/product/crs/bin/crsd.bin