Installing Oracle Database 11.2.0.4 RAC on RHEL 7.9

Operating System: Red Hat Enterprise Linux Server release 7.9 (Maipo)
Oracle Database: p13390677_112040_Linux-x86-64_{1..3}of7.zip

Packages Requirements

rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}  (%{ARCH})\n' \
bc binutils elfutils-libelf elfutils-libelf-devel fontconfig-devel \
glibc glibc-devel ksh libaio libaio-devel libXrender libX11 libXau \
libXi libXtst libgcc libnsl librdmacm libstdc++ libstdc++-devel gcc \
libxcb libibverbs make smartmontools sysstat pdksh gcc-c++ compat-libstdc++-33 \
compat-libcap1 perl-Env unzip xorg-x11-utils xorg-x11-xauth net-tools

・ Configuring Yum Repositories.

Installing Operating System Packages.

yum -y install \
bc binutils elfutils-libelf elfutils-libelf-devel fontconfig-devel \
glibc glibc-devel ksh libaio libaio-devel libXrender libX11 libXau \
libXi libXtst libgcc libnsl librdmacm libstdc++ libstdc++-devel gcc \
libxcb libibverbs make smartmontools sysstat pdksh gcc-c++ compat-libstdc++-33 \
compat-libcap1 perl-Env unzip xorg-x11-utils xorg-x11-xauth net-tools

curl -o /soft/compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm https://meno.lostyu.qzz.io/assets/software/linux/rpm/compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm
yum -y localinstall /soft/compat-libstdc++-33-3.2.3-72.el7.x86_64.rpm

Configuring Operating Systems for Oracle Grid Infrastructure on Linux

Create software directory.

mkdir /soft

Disabling Firewall and SElinux.

systemctl disable firewalld --now
sed -i '/SELINUX=/{s/enforcing/disabled/}' /etc/selinux/config

Disabling Zero Configuration Network.

cp -p /etc/sysconfig/network{,.bak.`date +%m%d`}
echo "NOZEROCONF=yes" >>/etc/sysconfig/network

Disabling Transparent Huagepages.

・ How to disable transparent hugepages (THP) on Red Hat Enterprise Linux 7, 8 and 9

Configuring Public, Private, Virtual and SCAN IP names and Addresses.

cat >>/etc/hosts <<EOF
# Oracle 11g
192.168.3.121   oracle11grac01
192.168.3.122   oracle11grac02
192.168.3.123   oracle11grac01-vip
192.168.3.124   oracle11grac02-vip
192.168.3.125   rac-scan
100.100.100.121   oracle11grac01-priv
100.100.100.122   oracle11grac02-priv
EOF

The SCAN name Cannot more than 15 characters.

Configuring Users, Groups and Environments for Oracle Grid Infrastructure and Oracle Database

Create Oracle software owner user accounts on each cluster node.

groupadd -g 1000 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
groupadd -g 1300 dba
groupadd -g 1301 oper
useradd -u 1101 -g oinstall -G asmadmin,asmdba,asmoper,dba -m -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid
useradd -u 1100 -g oinstall -G dba,oper,asmdba,asmadmin -m -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
echo "grid" | passwd --stdin grid
echo "oracle" | passwd --stdin oracle

Create Grid home and Oracle home directories on each cluster node.

mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle/product/11.2.0/db_1
chown -R grid:oinstall /u01
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01

Configure Oracle Software owner user account environment on each cluster node.

cat >> /root/.bash_profile << "EOF"
# Oracle 11g
export GRID_HOME=/u01/app/11.2.0/grid
EOF

cat >> /home/grid/.bash_profile <<"EOF"
# Oracle 11g
export ORACLE_SID=+ASM
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
EOF

cat >> /home/oracle/.bash_profile <<"EOF"
# Oracle 11g
export ORACLE_SID=orcl
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
EOF

sed -i "/ORACLE_SID/s/ASM/ASM`echo ${HOSTNAME##*0}`/" /home/grid/.bash_profile
sed -i "/ORACLE_SID/s/orcl/orcl`echo ${HOSTNAME##*0}`/" /home/oracle/.bash_profile

(Option) Installing rlwrap package.

wget --user sakura --ask-password -P /soft https://meno.lostyu.qzz.io/assets/software/linux/rpm/rlwrap/rlwrap-0.43-2.el7.x86_64.rpm
yum -y localinstall /soft/rlwrap-0.43-2.el7.x86_64.rpm

cat >> /home/grid/.bash_profile << "EOF"
if command -v rlwrap >/dev/null 2>&1; then
  alias sqlplus='rlwrap sqlplus'
  alias asmcmd='rlwrap asmcmd'
fi
EOF

cat >> /home/oracle/.bash_profile << "EOF"
if command -v rlwrap >/dev/null 2>&1; then
  alias sqlplus='rlwrap sqlplus'
  alias rman='rlwrap rman'
fi
EOF

Completing Preinstallation Tasks Manually

Configuring NTP Synchronization (Opation)

yum -y install chrony
systemctl start chronyd
chronyc sources -v

Configuring SSH User Equivalency

ssh-keygen -t rsa

Copy keys to a remote authorized_keys file.

ssh-copy-id oracle11grac01
ssh-copy-id oracle11grac02

Verifying ssh user equivalency on cluster nodes without being prompted for a password.

ssh oracle11grac01 date
ssh oracle11grac02 date

Repeat as oracle user.

Configuring Kernel Parameters for Linux

cat > /etc/sysctl.d/97-oracle-database-11g-sysctl.conf <<EOF
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
EOF

sysctl -p

Configuring Resource Limits for Oracle Software Installation Users

cat > /etc/security/limits.d/97-oracle-database-11g-limits.conf <<EOF
oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728
grid   soft   nofile    1024
grid   hard   nofile    65536
grid   soft   nproc    16384
grid   hard   nproc    16384
grid   soft   stack    10240
grid   hard   stack    32768
grid   hard   memlock    134217728
grid   soft   memlock    134217728
EOF

Configuring Storage Device Path Persistence Using Oracle ASMLIB

Install kmod-oracleasm,oracleasm-support,oracleasmlib rpm package.

curl -o /soft/kmod-oracleasm-2.0.8-28.el7.x86_64.rpm https://meno.lostyu.qzz.io/assets/res/linux/rpm/oracle/kmod-oracleasm-2.0.8-28.el7.x86_64.rpm
curl -o /soft/oracleasm-support-2.1.11-2.el7.x86_64.rpm https://meno.lostyu.qzz.io/assets/res/linux/rpm/oracle/oracleasm-support-2.1.11-2.el7.x86_64.rpm
curl -o /soft/oracleasmlib-2.0.12-1.el7.x86_64.rpm https://meno.lostyu.qzz.io/assets/res/linux/rpm/oracle/oracleasmlib-2.0.12-1.el7.x86_64.rpm

yum -y localinstall /soft/kmod-oracleasm-2.0.8-28.el7.x86_64.rpm
yum -y localinstall /soft/oracleasm-support-2.1.11-2.el7.x86_64.rpm
yum -y localinstall /soft/oracleasmlib-2.0.12-1.el7.x86_64.rpm

Configure the Oracle Linux ASMLib driver.

~]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

~]# oracleasm configure -x "rhel sda"
Writing Oracle ASM library driver configuration: done

~]# oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE="rhel sda"
ORACLEASM_SCAN_DIRECTORIES=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

~]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

~]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

Allocate a device for Oracle ASMLib use.

~]# echo -e "n\n\n\n\n\nw\n" > /tmp/create_new_partition

~]# for i in b c d e f g; do fdisk /dev/sd$i < /tmp/create_new_partition; done

~]# oracleasm createdisk OCR01 /dev/sdb1
~]# oracleasm createdisk OCR02 /dev/sdc1
~]# oracleasm createdisk DATA01 /dev/sdd1
~]# oracleasm createdisk DATA02 /dev/sde1
~]# oracleasm createdisk RECO01 /dev/sdf1
~]# oracleasm createdisk RECO02 /dev/sdg1
~]# oracleasm createdisk OCR03 /dev/sdh1
~]# oracleasm createdisk DATA03 /dev/sdi1

~]# oracleasm scandisks
~]# oracleasm listdisks
DATA01
DATA02
DATA03
OCR01
OCR02
OCR03
RECO01
RECO02

The configuration file in /etc/sysconfig/oracleasm.

Installing Oracle Grid Infrastructure for a New Cluster

Upload GI software file to software directory.

scp /mnt/e/Download/p13390677_112040_Linux-x86-64_3of7.zip root@192.168.3.121:/soft

Unzip GI software file.

unzip -q -d /soft /soft/p13390677_112040_Linux-x86-64_3of7.zip
chown -R grid:oinstall /soft/grid

Installing the cvuqdisk RPM.

set CVUQDISK_GRP=oinstall
rpm -ivh /soft/grid/rpm/cvuqdisk-1.0.9-1.rpm

/soft/grid/runcluvfy.sh stage -pre crsinst -fixup -n oracle11grac01,oracle11grac02 -verbose

The pdksh-5.2.14 package missing can be ignore.

export DISPLAY=192.168.3.86:0.0
/soft/grid/runInstaller

Installing the Oracle Database Software

Upload Database software file to software directory.

scp /mnt/e/Download/p13390677_112040_Linux-x86-64_[12]of7.zip root@192.168.3.121:/soft

Unzip Database software file.

unzip -q -d /soft/ /soft/p13390677_112040_Linux-x86-64_1of7.zip; unzip -q -d /soft/ /soft/p13390677_112040_Linux-x86-64_2of7.zip
chown -R oracle:oinstall /soft/database

export DISPLAY=192.168.3.86:0.0
/soft/database/runInstaller

Patch p19955028_112040_Linux-x86-64

・ p19955028_112040_Linux-x86-64

Installation Errors of Oracle 11.2.0.4 on RHEL 7.9

・ runInstaller start failed due could not execute xdpyinfo
・ Package compat-libstdc++-33-3.2.3 missing on rhel 7
・ CRS-4124 Oracle High Availability Services startup failed
・ INS-20802 Oracle Cluster Verification Utility failed
・ Error in invoking target agent nmhs of makerfile ins_emagent.mk

Appendix

Setting Up Grid Infrastructure.

Installing database.

Creating database.