Installing Oracle Database 10.2.0.5 RAC on RHEL 5.7

Operating System: RHEL 5.7
Grid Infrastructure: 10201_clusterware_linux_x86_64.cpio
Database Software: 10201_database_linux_x86_64.cpio
Patches: p8202632_10205_Linux-x86-64.zip

Pre-Installation Preparation and Verification

Verifying and Updating the redhat-release File.

~]# rpm -q redhat-release
~]# cp /etc/redhat-release /etc/redhat-release.orig
~]# cat > /etc/redhat-release << EOF
Red Hat Enterprise Linux AS release 3 (Taroon)
EOF

Verifying Memory and Swap Space.

Oracle states that the system must have at least 512MB of RAM and 1GB of swap space or a swap space twice the size of the RAM. For systems with more than 2 GB of RAM it states the swap space can be between one and two times the size of the RAM.

~]# grep MemTotal /proc/meminfo
~]# grep SwapTotal /proc/meminfo

Verifying Temporary(/tmp) Space.

The Oracle Universal Installer (OUI) requires up to 400 MB of free space in the /tmp directory.

~]# df /tmp

~]# mkdir /<AnotherFilesystem> /tmp
~]# chown root.root /<AnotherFilesystem> /tmp
~]# chmod 1777 /<AnotherFilesystem> /tmp
~]# export TEMP=/<AnotherFilesystem>
~]# export TMPDIR=/<AnotherFilesystem>

Verifying Hosts Resolve.

~]# cat >> /etc/hosts << EOF
# Oracle 10g
192.168.184.201 rac1
192.168.184.202 rac2
192.168.184.203 rac1-vip
192.168.184.204 rac2-vip
100.100.100.201 rac1-priv
100.100.100.202 rac2-priv
EOF

Verifying Software Packages (RPMs).

~]# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}  (%{ARCH})\n' \
    binutils compat-db control-center gcc gcc-c++ glibc glibc-common \
    gnome-libs libstdc++ libstdc++-devel make pdksh sysstat xscreensaver \
    libaio

~]# mkdir /mnt/cdrom
~]# mount /dev/cdrom /mnt/cdrom
~]# cat > /etc/yum.repos.d/ol57.repo << EOF
[ol57]
name = ol57
baseurl = file:///mnt/cdrom/Server
gpgcheck = 0
enabled = 1
EOF

~]# yum -y install binutils compat-db control-center gcc gcc-c++ glibc glibc-common \
    gnome-libs libstdc++ libstdc++-devel make pdksh sysstat xscreensaver \
    libaio

Verifying Kernel Parameters.

~]# cat > /etc/sysctl.d/97-oracle-database-10g-sysctl.conf << EOF
kernel.shmall = 4294967296
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144
# the hugetlb_shm_group is dba group id
vm.hugetlb_shm_group = 1001
EOF

~]# cat > /etc/security/limits.d/97-oracle-database-10g-limits.conf << EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

Configuring Users, Groups and Environments for Oracle Grid Infrastructure and Oracle Database

Creating Oracle User Accounts.

groupadd -g 1000 oinstall
groupadd -g 1001 dba
useradd -u 1000 -c "Oracle software owner" -g oinstall -G dba oracle
echo oracle | passwd --stdin oracle

Creating Oracle Directories.

mkdir -p /u01/app/oracle
chown oracle.oinstall /u01/app/oracle

Setting Oracle Environments.

cat >> /home/oracle/.bash_profile << EOF
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=coco
EOF

Installation Errors

Failed to upgrade Oracle Cluster Registry configuration

节点 1 执行 root.sh 时出现报错：Failed to upgrade Oracle Cluster Registry configuration

[root@rac1 ~]# /u01/app/oracle/product/10.2.0/crs_1/root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

Workaround:

1) 更新补丁包 p4679769_10201_Linux-x86-64.zip，具体更新方式参考更新包 README.txt

2) 重新格式化磁盘

dd if=/dev/zero of=/dev/raw/raw1 bs=1M count=300
dd if=/dev/zero of=/dev/raw/raw2 bs=1M count=300
dd if=/dev/zero of=/dev/raw/raw3 bs=1M count=300

Failure at final check of Oracle CRS stack

节点 2 执行 root.sh 时出现报错：Failure at final check of Oracle CRS stack

[root@rac2 ~]# /u01/app/oracle/product/10.2.0/crs_1/root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10

crsd.log 出现如下错误：

[root@rac2 ~]# tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac2/crsd/crsd.log
2022-07-12 15:13:44.119: [ COMMCRS][1113364800]clsc_connect: (0x23f3630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac2_crs))
2022-07-12 15:13:44.119: [ CSSCLNT][3461711584]clsssInitNative: connect failed, rc 9
2022-07-12 15:13:44.120: [  CRSRTI][3461711584]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2022-07-12 15:13:45.122: [ CRSMAIN][3461711584][PANIC]0CRSD exiting: Could not init the CSS context
2022-07-12 15:13:45.123: [ default][3461711584]Terminating clsd session

Workaround:

大意了，没有关防火墙，关了重新执行就可以了。

service iptables stop
chkconfig iptables off

error while loading shared libraries: libpthread.so.0

节点 2 执行 root.sh 时出现报错：error while loading shared libraries: libpthread.so.0

[root@rac2 ~]# /u01/app/oracle/product/10.2.0/crs_1/install/rootdelete.sh
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
[root@rac2 ~]# /u01/app/oracle/product/10.2.0/crs_1/root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        rac1
        rac2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
/u01/app/oracle/product/10.2.0/crs_1/jdk/jre//bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

Workaround:

1) 在每个节点上，修改 $ORA_CRS_HOME/bin 目录下的 srvctl 和 vipca 文件。在 vipca 文件 ARGUMENTS="" 行之前和 srvctl 文件的 export LD_ASSUME_KERNEL 行之后增加 unset LD_ASSUME_KERNEL 语句

sed -i '/^export LD_ASSUME_KERNEL/a unset LD_ASSUME_KERNEL' srvctl
sed -i '/^ARGUMENTS=""/i unset LD_ASSUME_KERNEL' vipca

2) 在任意一个节点上使用 oifcfg 配置 public 和 vip 网络

[root@rac1 ~]# cd /u01/app/oracle/product/10.2.0/crs_1/bin/
[root@rac1 bin]# ./oifcfg setif -global eth0/192.168.184.0:public
[root@rac1 bin]# ./oifcfg setif -global eth1/100.100.100.0:cluster_interconnect
[root@rac1 bin]# ./oifcfg getif
eth0  192.168.184.0  global  public
eth1  100.100.100.0  global  cluster_interconnect
[root@rac1 bin]# ./oifcfg iflist
eth0  192.168.184.0
eth1  100.100.100.0

3) 在任意一个节点用 root 用户身份手动执行 $ORA_CRS_HOME/bin/vipca 工具，配置 IP 信息，配置完成后执行./crs_stat -t，状态正常

[root@rac1 bin]# ./vipca
[root@rac1 bin]# ./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.rac1.gsd   application    ONLINE    ONLINE    rac1
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   application    ONLINE    ONLINE    rac1
ora.rac2.gsd   application    ONLINE    ONLINE    rac2
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   application    ONLINE    ONLINE    rac2