Oracle Exadata Virtual Machine 18.1.20.0.0

System Operation: Oracle Linux Release 7 Update 4
Exadata Softeware: V983555-01.zip
Oracle Database: p10404530_112030_Linux-x86-64_{1..3}of7.zip
System Memory: 16GB
X3-2

Cell Node

建议 GI 软件安装 11.2.0.3.0 版本，11.2.0.4.0 版本存在 BUG，暂时无法排除处理。具体情况查看 Installation Errors 小节。

Setting network and hostname.

# 网卡ens32作为service网络，ens35作为infiniband网络
nmcli con modify ens32 ipv4.add 192.168.3.110/24 ipv4.gate 192.168.3.1 ipv4.dns 114.114.114.114 ipv4.method manual autoconnect yes
nmcli con modify ens35 ipv4.add 100.100.100.110/24 ipv4.method manual autoconnect yes

# 将ifcfg-ens35网卡配置文件中的PREFIX=24参数变更为NETMASK=255.255.255.0，具体情况查看 Installation Errors 小节
sed -i '/PREFIX=24/s/PREFIX=24/NETMASK=255.255.255.0/' /etc/sysconfig/network-scripts/ifcfg-ens35

sed -i '/::1/s/::1/# ::1/' /etc/hosts
echo "100.100.100.110 exacell01" >>/etc/hosts

systemctl disable firewalld --now
sed -i '/^SELINUX=/s/enforcing/disabled/' /etc/selinux/config

Configuring kernel parameters

cat >>/etc/sysctl.conf <<EOF
fs.file-max = 65536
fs.aio-max-nr = 500000000
vm.min_free_kbytes = 512000
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.core.wmem_max = 4194304
EOF

cat >> /etc/security/limits.conf << EOF
root soft nofile 131072
root hard nofile 131072
celladmin soft nproc 2047
celladmin hard nproc 16384
celladmin soft nofile 131072
celladmin hard nofile 131072
EOF

cat >> /etc/modprobe.d/rds.conf << EOF
install rds /sbin/modprobe --ignore -install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma
EOF

Creating log directories.

mkdir -p /var/log/oracle
chmod -R 775 /var/log/oracle

Installing Cell software.

mkdir /tmp/V983555-01
unzip -d /tmp/V983555-01 /tmp/V983555-01.zip
cd /tmp/V983555-01
tar -xvf nfsimg-18.1.20.0.0-190911-cell.tar
cd cellbits/
tar -jxvf cellrpms.tbz
yum -y localinstall jdk1.8-1.8.0_221-fcs.x86_64.rpm
unzip cell.bin
yum -y localinstall cell-18.1.20.0.0_LINUX.X64_190911-1.x86_64.rpm

Starting Cell services.

# cd /opt/oracle/cell/cellsrv/bin/
# ./cellcli

CellCLI> alter cell restart services all
Stopping RS services...
CELL-01509: Restart Server (RS) not responding.
Starting the RS services...
Getting the state of RS services...  running
Restarting CELLSRV services...
The RESTART of CELLSRV services was not successful.
CELL-01531: Unable to parse the cellinit.ora file due to incorrect parameters in the file.
Restarting MS services...
The RESTART of MS services was successful.

CellCLI> create cell exacell01 interconnect1=ens35

Creating virtual disks.

mkdir -p /opt/oracle/cell/disks/raw

cat >/tmp/disk.sh <<EOF
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk01 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk02 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk03 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk04 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk05 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk06 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk07 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk08 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk09 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk10 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk11 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/disk12 bs=1M count=5120
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/FLASH01 bs=1M count=1024
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/FLASH02 bs=1M count=1024
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/FLASH03 bs=1M count=1024
dd if=/dev/zero of=/opt/oracle/cell/disks/raw/FLASH04 bs=1M count=1024
EOF

sh /tmp/disk.sh
chmod 660 /opt/oracle/cell/disks/raw/*

Creating Cell grid disks.

cellcli -e create celldisk all
cellcli -e create flashcache all size=2G
cellcli -e create flashlog all
cellcli -e create griddisk all harddisk prefix=data,size=2G
cellcli -e create griddisk all harddisk prefix=arch,size=1G
cellcli -e create griddisk all harddisk prefix=ocr,size=500M
cellcli -e list griddisk

DB Node

Installing Oracle 11g RAC

・ Installing Oracle 11g RAC

Setting Up Configuration Files for a Database Server Host

mkdir -p /etc/oracle/cell/network-config

echo "ipaddress1=100.100.100.210/24" > /etc/oracle/cell/network-config/cellinit.ora
cat >/etc/oracle/cell/network-config/cellip.ora <<EOF
cell="100.100.100.110","100.100.100.111","100.100.100.112"
EOF

chown -R grid:oinstall /etc/oracle/cell/network-config
chmod -R 775 /etc/oracle/cell/network-config

kfod verify.

# Login as grid
# Path is GI Software Unzip Location
export LD_LIBRARY_PATH=/soft/grid/stage/ext/lib:/lib:/usr/lib
/soft/grid/stage/ext/bin/kfod disks=all dscvgroup=true

Installation Errors

cell alert log: /opt/oracle/cell/log/diag/asm/cell/exacell01/trace/alert.log

CELL-02598

执行 create cell cell1 interconnect1=ens35 时，出现如下错误

CellCLI> create cell exacell01 interconnect1=ens35
CELL-02598: Ipaddress/Netmask attribute is not properly configured for interconnect ens35.

Solution:

将 ifcfg-ens35 网卡配置文件中的 PREFIX=24 参数变更为 NETMASK=255.255.255.0。

sed -i '/PREFIX=24/s/PREFIX=24/NETMASK=255.255.255.0/' /etc/sysconfig/network-scripts/ifcfg-ens35

CELL-02018

执行 create cell cell1 interconnect1=ens35 时，出现如下错误

CellCLI> create cell exacell01 interconnect1=ens35
Cell exacell03 successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-02018: Internal error detected: Not enough open file descriptors.

Solution:

资源限制 nofile 参数设置不足，增加该参数值。

[celladmin@exacell03 ~]$ ulimit -Sn
131072
[celladmin@exacell03 ~]$ ulimit -Hn
131072

CELL-01554

重启 cell 服务，MS 服务无法启动

CellCLI> alter cell restart services all

Stopping RS services...
The SHUTDOWN of RS services was successful.
Starting the RS services...
Getting the state of RS services...  running
Restarting CELLSRV services...
The RESTART of CELLSRV services was successful.
Restarting MS services...
The RESTART of MS services was not successful.
CELL-01554: MS startup failed for unknown reasons.

查看 /opt/oracle/cell/log/diag/asm/cell/exacell01/trace/alert.log 日志文件

2023-06-05T21:14:33.160755+08:00
[RS] Start service MS failed with error: -74.
2023-06-05T21:14:33.165961+08:00
[RS] Monitoring process /opt/oracle/cell18.1.20.0.0_LINUX.X64_190911/cellsrv/bin/cellrsmmt (pid: 1042) returned with error: 162
2023-06-05T21:14:33.166253+08:00
[RS] Service MS is no longer present
Errors in file /opt/oracle/cell18.1.20.0.0_LINUX.X64_190911/log/diag/asm/cell/exacell01/trace/rstrc_556_mmt.trc  (incident=137):
RS-7445 [Serv MS is absent] [It will not be restarted] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/cell18.1.20.0.0_LINUX.X64_190911/log/diag/asm/cell/exacell01/incident/incdir_137/rstrc_556_mmt_i137.trc

[RS] Stopped Service MS

查看提示的 trace 文件

2023-06-05 21:14:23.037 :00000038: MS process is not alive. Pid is missing.
2023-06-05 21:14:23.037 :00000039: Missed a heartbeat for process MS or leaking memory, error: -75
2023-06-05 21:14:23.037 :0000003A: Service MS was not alive, try starting
...
2023-06-05 21:14:23.255 :00000048: OS system dependent operation:connect_error failed with status: 111
2023-06-05 21:14:23.255 :00000049: OS failure message: Connection
2023-06-05 21:14:23.255 :0000004A: failure occurred at: sosstcpconne
2023-06-05 21:14:23.255 :0000004B: Failed to heartbeat MS (port: 5043 timeout: 60 sec)
2023-06-05 21:14:23.255 :0000004C: socket open error: Port no: 8888. Received errorno 111. Connection refused
2023-06-05 21:14:23.356 :0000004D: mon_proc_pid oldpid: 1097

Solution:

原因暂时未排查清楚，用 root 用户可正常重启

[root@exacell01 ~]# cd /opt/oracle/cell/cellsrv/bin
[root@exacell01 bin]#
[root@exacell01 bin]# ./cellcli -e alter cell restart services all
Stopping RS services...
The SHUTDOWN of RS services was successful.
Starting the RS services...
Getting the state of RS services...  running
Restarting CELLSRV services...
The RESTART of CELLSRV services was successful.
Restarting MS services...
The RESTART of MS services was successful.

CSS startup failed and Network initialization failed

CSS startup failed when running root.sh on the first node.

[root@exadb01 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2023-06-04 22:57:07.442:
[client(23969)]CRS-2101:The OLR was formatted using version 3.
2023-06-04 23:01:36.538:
[client(24889)]CRS-2101:The OLR was formatted using version 3.

CRS-2672: Attempting to start 'ora.mdnsd' on 'exadb01'
CRS-2676: Start of 'ora.mdnsd' on 'exadb01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exadb01'
CRS-2676: Start of 'ora.gpnpd' on 'exadb01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exadb01'
CRS-2672: Attempting to start 'ora.gipcd' on 'exadb01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exadb01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exadb01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exadb01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exadb01'
CRS-2676: Start of 'ora.diskmon' on 'exadb01' succeeded
CRS-2674: Start of 'ora.cssd' on 'exadb01' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'exadb01'
CRS-2681: Clean of 'ora.cssd' on 'exadb01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'exadb01'
CRS-2677: Stop of 'ora.gipcd' on 'exadb01' succeeded
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
The exlusive mode cluster start failed, see Grid Infrastructure alert log for more information
Initial cluster configuration failed.  See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_exadb01.log for details
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

See /u01/app/11.2.0/grid/log/exadb01/cssd/ocssd.log

2023-06-04 23:08:37.504: [   SKGFD][315557632]ERROR: -8(OS Error 1 (bind_fail,skgxpvifconf,requested interface 100.100.100.210 failed bind.
Check output from ifconfig command,Error 0)
)
2023-06-04 23:08:37.504: [   SKGFD][315557632]ERROR: -10(OSS Operation oss_initialize failed with error 4 [Network initialization failed]
)
2023-06-04 23:08:37.504: [    CSSD][315557632]clsssnmvDDiscThread: Unable to create clsf context
2023-06-04 23:08:37.504: [    CSSD][315557632]###################################
2023-06-04 23:08:37.504: [    CSSD][315557632]clssscExit: CSSD aborting from thread clssnmvDDiscThread
2023-06-04 23:08:37.504: [    CSSD][315557632]###################################
2023-06-04 23:08:37.504: [    CSSD][315557632](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

Solution:

安装 GI 软件 11.2.0.3.0 版本，无此报错。