Reflection of my thoughts and technical expriences: Adding New Node to Existing Cluster (Two Node RAC) using silent option - Oracle 12cR2 (12.2) GI and Oracle 12cR1 (12.1.0.2) RDBMS

This article we are going to demonstrate how to add new node to the existing two node RAC. Adding New Node to Existing Cluster (Two Node RAC) using silent option - Oracle 12cR2 (12.2) GI and Oracle 12cR1 (12.1.0.2) RDBMS.

Environment Setup:

GRID Infrasturcture: Oracle 12cR2 (12.2.0.1.0)
RDBMS: Oracle 12cR2 (12.1.0.2.0)
Operating System: Oracle Enterprise Linux 7.2
Used features: Automatic Storage Management Filter Driver (ASMFD)
Node-1: oel72-rac1
Node-2: oel72-rac2

Node-3: oel72-rac3 (New Node)

Assumptions:

a. New node will be added with same OS configurations, User Groups.
b. Public, Private and VIPs are configured.
c. Set the Kernel parameters (/etc/sysctl.conf).
d. Set the limits (/etc/security/limits.conf).
e. Disable the firewall.
f. Setup SSH between the cluster nodes.
g. Check the accessibility between the cluster nodes.
h. Shared Storage using "ASMFD" without /udev rules.

Hosts file: /etc/hosts

[root@oel72-rac3 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# Public
192.168.2.122 oel72-rac1.localdomain oel72-rac1
192.168.2.123 oel72-rac2.localdomain oel72-rac2
192.168.2.130 oel72-rac3.localdomain oel72-rac3
# Private
10.1.4.122 oel72-rac1-priv.localdomain oel72-rac1-priv
10.1.4.123 oel72-rac2-priv.localdomain oel72-rac2-priv
10.1.4.130 oel72-rac3-priv.localdomain oel72-rac3-priv
# Virtual
192.168.2.124 oel72-rac1-vip.localdomain oel72-rac1-vip
192.168.2.125 oel72-rac2-vip.localdomain oel72-rac2-vip
192.168.2.131 oel72-rac3-vip.localdomain oel72-rac3-vip
# SCAN
192.168.2.126 racp-scan.localdomain racp-scan
192.168.2.127 racp-scan.localdomain racp-scan
192.168.2.128 racp-scan.localdomain racp-scan

Login into Node-1 (oel72-rac1) and check the permissions for the following file

[root@oel72-rac1 ~]# cd /u01/app/12.2.0/grid/network/log
[root@oel72-rac1 log]# ls -lrth
total 4.0K
-rw-r-----. 1 root root 3.0K Jul 14 23:03 sqlnet.log
[root@oel72-rac1 log]#
[root@oel72-rac1 log]# chown oracle:oinstall sqlnet.log
[root@oel72-rac1 log]# chmod 777 sqlnet.log
[root@oel72-rac1 log]# ls -lrth
total 4.0K
-rwxrwxrwx. 1 oracle oinstall 3.0K Jul 14 23:03 sqlnet.log
[root@oel72-rac1 log]#

Copy "cvuqdisk-1.0.10-1.rpm" to the Node-3 (oel72-rac3) as a pre-requisite

[oracle@oel72-rac1 ~]$ cd /u01/app/12.2.0/grid/cv/rpm/
[oracle@oel72-rac1 rpm]$ scp cvuqdisk-1.0.10-1.rpm oracle@oel72-rac3:/home/oracle
The authenticity of host 'oel72-rac3 (192.168.2.130)' can't be established.
ECDSA key fingerprint is 1f:d9:85:ba:ed:db:0e:b6:ea:c5:2a:02:74:7d:62:3b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'oel72-rac3,192.168.2.130' (ECDSA) to the list of known hosts.
oracle@oel72-rac3's password:
cvuqdisk-1.0.10-1.rpm 100% 8860 8.7KB/s 00:00
[oracle@oel72-rac1 rpm]$

Login to Node-3 (oel72-rac3) as a 'root' user and install "cvuqdisk-1.0.10-1.rpm"

[oracle@oel72-rac3 ~]$ su - root
Password:
Last login: Wed Oct 7 16:55:12 EDT 2020 on pts/0

[root@oel72-rac3 ~]# chmod u+x /home/oracle/cvuqdisk-1.0.10-1.rpm
[root@oel72-rac3 ~]# cd /home/oracle

[root@oel72-rac3 oracle]# rpm -ivh cvuqdisk-1.0.10-1.rpm
Preparing... ################################# [100%]
Using default group oinstall to install package
Updating / installing...
1:cvuqdisk-1.0.10-1 ################################# [100%]
[root@oel72-rac3 oracle]#

Login to Node-1 (oel72-rac1) and execute cluvfy with "-pre nodeadd" option.

[oracle@oel72-rac1 ~]$ cluvfy stage -pre nodeadd -n oel72-rac3 -verbose
Verifying User Equivalence ...FAILED (PRVG-2019, PRKN-1038)
User equivalence unavailable on all the specified nodes
Verification cannot proceed
Pre-check for node addition was unsuccessful on all the nodes.
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
Verifying User Equivalence ...FAILED
oel72-rac3: PRVG-2019 : Check for equivalence of user "oracle" from node
"oel72-rac1" to node "oel72-rac3" failed
PRKN-1038 : The command "/usr/bin/ssh -o FallBackToRsh=no -o
PasswordAuthentication=no -o StrictHostKeyChecking=yes -o
NumberOfPasswordPrompts=0 oel72-rac3 -n /bin/true" run on node
"oel72-rac1" gave an unexpected output: "Permission denied
(publickey,gssapi-keyex,gssapi-with-mic,password)."

CVU operation performed: stage -pre nodeadd
Date: Oct 7, 2020 5:04:48 PM
CVU home: /u01/app/12.2.0/grid/
User: oracle
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ cd $ORACLE_HOME
[oracle@oel72-rac1 grid]$ cd addnode/
[oracle@oel72-rac1 addnode]$
[oracle@oel72-rac1 addnode]$ ls -lrth
total 36K
-rw-r-----. 1 oracle oinstall 13K Aug 5 2016 addnode.pl
-rw-r-----. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini.sbs.ouibak
-rw-r-----. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini.sbs
-rw-r-----. 1 oracle oinstall 3.7K Jan 26 2017 addnode.sh.ouibak
-rw-r--r--. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini
-rwxr-x---. 1 oracle oinstall 3.7K Jul 9 11:09 addnode.sh
[oracle@oel72-rac1 addnode]$

[oracle@oel72-rac1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={oel72-rac3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={oel72-rac3-vip}" "CLUSTER_NEW_NODE_ROLES={hub}"
Logfile Location : /u01/app/oraInventory/logs/sshsetup1_2020-10-07_05-07-20-PM.log
ClusterLogger - log file location: /u01/app/oraInventory/logs/remoteInterfaces2020-10-07_05-07-20-PM.log
[oracle@oel72-rac1 addnode]$

Note: Just for SSH Setup. Once done come out of the installation of GUI window because we are planning to use "-silent" option.

[oracle@oel72-rac1 ~]$ cluvfy stage -pre nodeadd -n oel72-rac3 -verbose
Verifying Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 12.1688GB (1.2759932E7KB) 8GB (8388608.0KB) passed
oel72-rac1 12.1688GB (1.2759932E7KB) 8GB (8388608.0KB) passed
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 11.6764GB (1.2243572E7KB) 50MB (51200.0KB) passed
oel72-rac1 9.3639GB (9818812.0KB) 50MB (51200.0KB) passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 9.8125GB (1.0289148E7KB) 12.1688GB (1.2759932E7KB) failed
oel72-rac1 9.8125GB (1.0289148E7KB) 12.1688GB (1.2759932E7KB) failed
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: oel72-rac3:/usr,oel72-rac3:/var,oel72-rac3:/etc,oel72-rac3:/u01/app/12.2.0/grid,oel72-rac3:/sbin,oel72-rac3:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr oel72-rac3 / 46.5283GB 25MB passed
/var oel72-rac3 / 46.5283GB 5MB passed
/etc oel72-rac3 / 46.5283GB 25MB passed
/u01/app/12.2.0/grid oel72-rac3 / 46.5283GB 6.9GB passed
/sbin oel72-rac3 / 46.5283GB 10MB passed
/tmp oel72-rac3 / 46.5283GB 1GB passed
Verifying Free Space: oel72-rac3:/usr,oel72-rac3:/var,oel72-rac3:/etc,oel72-rac3:/u01/app/12.2.0/grid,oel72-rac3:/sbin,oel72-rac3:/tmp ...PASSED
Verifying Free Space: oel72-rac1:/usr,oel72-rac1:/var,oel72-rac1:/etc,oel72-rac1:/u01/app/12.2.0/grid,oel72-rac1:/sbin,oel72-rac1:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr oel72-rac1 / 21.584GB 25MB passed
/var oel72-rac1 / 21.584GB 5MB passed
/etc oel72-rac1 / 21.584GB 25MB passed
/u01/app/12.2.0/grid oel72-rac1 / 21.584GB 6.9GB passed
/sbin oel72-rac1 / 21.584GB 10MB passed
/tmp oel72-rac1 / 21.584GB 1GB passed
Verifying Free Space: oel72-rac1:/usr,oel72-rac1:/var,oel72-rac1:/etc,oel72-rac1:/u01/app/12.2.0/grid,oel72-rac1:/sbin,oel72-rac1:/tmp ...PASSED
Verifying User Existence: oracle ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists(54321)
oel72-rac1 passed exists(54321)
Verifying Users With Same UID: 54321 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying User Existence: root ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists(0)
oel72-rac1 passed exists(0)
Verifying Users With Same UID: 0 ...PASSED
Verifying User Existence: root ...PASSED
Verifying Group Existence: asmadmin ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists
oel72-rac1 passed exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists
oel72-rac1 passed exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists
oel72-rac1 passed exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed exists
oel72-rac1 passed exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmadmin ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 yes yes yes passed
oel72-rac1 yes yes yes passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: asmdba ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 yes yes yes passed
oel72-rac1 yes yes yes passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: oinstall ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 yes yes yes passed
oel72-rac1 yes yes yes passed
Verifying Group Membership: oinstall ...PASSED
Verifying Group Membership: asmoper ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 yes yes yes passed
oel72-rac1 yes yes yes passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 5 3,5 passed
oel72-rac1 5 3,5 passed
Verifying Run Level ...PASSED
Verifying Architecture ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 x86_64 x86_64 passed
oel72-rac1 x86_64 x86_64 passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 3.8.13-98.7.1.el7uek.x86_64 3.8.13 passed
oel72-rac1 3.8.13-98.7.1.el7uek.x86_64 3.8.13 passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 250 250 250 passed
oel72-rac1 250 250 250 passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 32000 32000 32000 passed
oel72-rac1 32000 32000 32000 passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 100 100 100 passed
oel72-rac1 100 100 100 passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 128 128 128 passed
oel72-rac1 128 128 128 passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 4398046511104 4398046511104 6533085184 passed
oel72-rac1 4398046511104 4398046511104 6533085184 passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 4096 4096 4096 passed
oel72-rac1 4096 4096 4096 passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 1073741824 1073741824 1275993 passed
oel72-rac1 1073741824 1073741824 1275993 passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 6815744 6815744 6815744 passed
oel72-rac1 6815744 6815744 6815744 passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
oel72-rac1 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 262144 262144 262144 passed
oel72-rac1 262144 262144 262144 passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 4194304 4194304 4194304 passed
oel72-rac1 4194304 4194304 4194304 passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 262144 262144 262144 passed
oel72-rac1 262144 262144 262144 passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 1048576 1048576 1048576 passed
oel72-rac1 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 1048576 1048576 1048576 passed
oel72-rac1 1048576 1048576 1048576 passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
oel72-rac3 1 1 1 passed
oel72-rac1 1 1 1 passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 binutils-2.27-43.base.0.1.el7_8.1 binutils-2.23.52.0.1 passed
oel72-rac1 binutils-2.27-43.base.0.1.el7_8.1 binutils-2.23.52.0.1 passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
oel72-rac1 compat-libcap1-1.10-7.el7 compat-libcap1-1.10 passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 libgcc(x86_64)-4.8.5-39.0.3.el7 libgcc(x86_64)-4.8.2 passed
oel72-rac1 libgcc(x86_64)-4.8.5-39.0.3.el7 libgcc(x86_64)-4.8.2 passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 libstdc++(x86_64)-4.8.5-39.0.3.el7 libstdc++(x86_64)-4.8.2 passed
oel72-rac1 libstdc++(x86_64)-4.8.5-39.0.3.el7 libstdc++(x86_64)-4.8.2 passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 libstdc++-devel(x86_64)-4.8.5-39.0.3.el7 libstdc++-devel(x86_64)-4.8.2 passed
oel72-rac1 libstdc++-devel(x86_64)-4.8.5-39.0.3.el7 libstdc++-devel(x86_64)-4.8.2 passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 sysstat-10.1.5-19.el7 sysstat-10.1.5 passed
oel72-rac1 sysstat-10.1.5-19.el7 sysstat-10.1.5 passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 ksh ksh passed
oel72-rac1 ksh ksh passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 make-3.82-24.el7 make-3.82 passed
oel72-rac1 make-3.82-24.el7 make-3.82 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 glibc(x86_64)-2.17-307.0.1.el7.1 glibc(x86_64)-2.17 passed
oel72-rac1 glibc(x86_64)-2.17-307.0.1.el7.1 glibc(x86_64)-2.17 passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 glibc-devel(x86_64)-2.17-307.0.1.el7.1 glibc-devel(x86_64)-2.17 passed
oel72-rac1 glibc-devel(x86_64)-2.17-307.0.1.el7.1 glibc-devel(x86_64)-2.17 passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
oel72-rac1 libaio(x86_64)-0.3.109-13.el7 libaio(x86_64)-0.3.109 passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
oel72-rac1 libaio-devel(x86_64)-0.3.109-13.el7 libaio-devel(x86_64)-0.3.109 passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 nfs-utils-1.3.0-0.21.el7 nfs-utils-1.2.3-15 passed
oel72-rac1 nfs-utils-1.3.0-0.21.el7 nfs-utils-1.2.3-15 passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 smartmontools-6.2-4.el7 smartmontools-6.2-4 passed
oel72-rac1 smartmontools-6.2-4.el7 smartmontools-6.2-4 passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
oel72-rac1 net-tools-2.0-0.17.20131004git.el7 net-tools-2.0-0.17 passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
Node Name Status
------------------------------------ ------------------------
oel72-rac3 passed
oel72-rac1 passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
oel72-rac2 cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
oel72-rac1 cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Node Addition ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying '/u01/app/12.2.0/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Node Connectivity ...
Verifying Hosts File ...
Node Name Status
------------------------------------ ------------------------
oel72-rac1 passed
oel72-rac2 passed
oel72-rac3 passed
Verifying Hosts File ...PASSED
Interface information for node "oel72-rac3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.130 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.130 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Interface information for node "oel72-rac2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.123 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.125 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.126 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.123 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Interface information for node "oel72-rac1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.122 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.124 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.128 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.127 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.122 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Check: MTU consistency on the private interfaces of subnet "10.1.4.0"
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 enp0s9 10.1.4.130 10.1.4.0 1500
oel72-rac2 enp0s9 10.1.4.123 10.1.4.0 1500
oel72-rac1 enp0s9 10.1.4.122 10.1.4.0 1500
Check: MTU consistency of the subnet "192.168.2.0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 enp0s8 192.168.2.130 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.123 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.125 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.126 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.122 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.124 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.128 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.127 192.168.2.0 1500
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.123] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.125] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac2[enp0s8:192.168.2.125] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac1[enp0s8:192.168.2.124] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac1[enp0s8:192.168.2.124] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac1[enp0s8:192.168.2.128] oel72-rac1[enp0s8:192.168.2.127] yes
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oel72-rac1[enp0s9:10.1.4.122] oel72-rac2[enp0s9:10.1.4.123] yes
oel72-rac1[enp0s9:10.1.4.122] oel72-rac3[enp0s9:10.1.4.130] yes
oel72-rac2[enp0s9:10.1.4.123] oel72-rac3[enp0s9:10.1.4.130] yes
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying subnet mask consistency for subnet "192.168.2.0" ...PASSED
Verifying subnet mask consistency for subnet "10.1.4.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...
Checking subnet "10.1.4.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast check ...PASSED
Verifying ASM Integrity ...
Verifying Node Connectivity ...
Verifying Hosts File ...
Node Name Status
------------------------------------ ------------------------
oel72-rac3 passed
Verifying Hosts File ...PASSED
Interface information for node "oel72-rac3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.130 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.130 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Interface information for node "oel72-rac2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.123 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.125 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.126 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.123 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Interface information for node "oel72-rac1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
enp0s8 192.168.2.122 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.124 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.128 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s8 192.168.2.127 192.168.2.0 0.0.0.0 10.0.2.2 08:00:27:C2:FA:2B 1500
enp0s9 10.1.4.122 10.1.4.0 0.0.0.0 10.0.2.2 08:00:27:19:3E:9E 1500
Check: MTU consistency on the private interfaces of subnet "10.1.4.0"
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 enp0s9 10.1.4.130 10.1.4.0 1500
oel72-rac2 enp0s9 10.1.4.123 10.1.4.0 1500
oel72-rac1 enp0s9 10.1.4.122 10.1.4.0 1500
Check: MTU consistency of the subnet "192.168.2.0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
oel72-rac3 enp0s8 192.168.2.130 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.123 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.125 192.168.2.0 1500
oel72-rac2 enp0s8 192.168.2.126 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.122 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.124 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.128 192.168.2.0 1500
oel72-rac1 enp0s8 192.168.2.127 192.168.2.0 1500
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.123] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.125] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac1[enp0s8:192.168.2.122] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac2[enp0s8:192.168.2.125] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.123] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac2[enp0s8:192.168.2.126] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.125] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac3[enp0s8:192.168.2.130] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac2[enp0s8:192.168.2.126] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.124] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac3[enp0s8:192.168.2.130] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac1[enp0s8:192.168.2.124] oel72-rac1[enp0s8:192.168.2.128] yes
oel72-rac1[enp0s8:192.168.2.124] oel72-rac1[enp0s8:192.168.2.127] yes
oel72-rac1[enp0s8:192.168.2.128] oel72-rac1[enp0s8:192.168.2.127] yes
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oel72-rac1[enp0s9:10.1.4.122] oel72-rac2[enp0s9:10.1.4.123] yes
oel72-rac1[enp0s9:10.1.4.122] oel72-rac3[enp0s9:10.1.4.130] yes
oel72-rac2[enp0s9:10.1.4.123] oel72-rac3[enp0s9:10.1.4.130] yes
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying subnet mask consistency for subnet "192.168.2.0" ...PASSED
Verifying subnet mask consistency for subnet "10.1.4.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...Disks "/dev/sdp,/dev/sdq,/dev/sdj1,/dev/sdk1,/dev/sdl1,/dev/sdm1,/dev/sdn1,/dev/sdo1,/dev/sdr,/dev/sds" are managed by ASM.
Verifying Device Checks for ASM ...PASSED
Verifying Database home availability ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Network Time Protocol (NTP) ...
Verifying '/etc/ntp.conf' ...
Node Name File exists?
------------------------------------ ------------------------
oel72-rac3 yes
oel72-rac1 yes
Verifying '/etc/ntp.conf' ...PASSED
Verifying '/etc/chrony.conf' ...
Node Name File exists?
------------------------------------ ------------------------
oel72-rac3 yes
oel72-rac1 yes
Verifying '/etc/chrony.conf' ...PASSED
Verifying '/var/run/chronyd.pid' ...
Node Name File exists?
------------------------------------ ------------------------
oel72-rac3 yes
oel72-rac1 yes
Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Daemon 'chronyd' ...
Node Name Running?
------------------------------------ ------------------------
oel72-rac3 yes
oel72-rac1 yes
Verifying Daemon 'chronyd' ...PASSED
Verifying NTP daemon or service using UDP port 123 ...
Node Name Port Open?
------------------------------------ ------------------------
oel72-rac3 yes
oel72-rac1 yes
Verifying NTP daemon or service using UDP port 123 ...PASSED
Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
Verifying Network Time Protocol (NTP) ...FAILED (PRVG-1063)
Verifying User Not In Group "root": oracle ...
Node Name Status Comment
------------ ------------------------ ------------------------
oel72-rac3 passed does not exist
oel72-rac1 passed does not exist
Verifying User Not In Group "root": oracle ...PASSED
Verifying resolv.conf Integrity ...
Node Name Status
------------------------------------ ------------------------
oel72-rac3 failed
oel72-rac1 failed
checking response for name "oel72-rac3" from each of the name servers specified
in "/etc/resolv.conf"
Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
oel72-rac3 192.168.2.1 IPv4 failed
checking response for name "oel72-rac1" from each of the name servers specified
in "/etc/resolv.conf"
Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
oel72-rac1 192.168.2.1 IPv4 failed
Verifying resolv.conf Integrity ...FAILED (PRVF-5636, PRVG-10048)
Verifying DNS/NIS name service ...PASSED
Verifying User Equivalence ...
Node Name Status
------------------------------------ ------------------------
oel72-rac3 passed
oel72-rac2 passed
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying /boot mount ...PASSED
Verifying zeroconf check ...PASSED
Pre-check for node addition was unsuccessful.
Checks did not pass for the following nodes:
oel72-rac3,oel72-rac1
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
Verifying Swap Size ...FAILED
oel72-rac3: PRVF-7573 : Sufficient swap size is not available on node
"oel72-rac3" [Required = 12.1688GB (1.2759932E7KB) ; Found =
9.8125GB (1.0289148E7KB)]
oel72-rac1: PRVF-7573 : Sufficient swap size is not available on node
"oel72-rac1" [Required = 12.1688GB (1.2759932E7KB) ; Found =
9.8125GB (1.0289148E7KB)]
Verifying Network Time Protocol (NTP) ...FAILED
PRVG-1063 : configuration files for more than one time synchronization service
were found on nodes of the cluster
configuration file "/etc/chrony.conf" was found on nodes "oel72-rac3,oel72-rac1"
configuration file "/etc/ntp.conf" was found on nodes "oel72-rac3,oel72-rac1"
Verifying chrony daemon is synchronized with at least one external time
source ...FAILED
oel72-rac3: PRVG-13606 : chrony daemon is not synchronized with any external
time source on node "oel72-rac3".
oel72-rac1: PRVG-13606 : chrony daemon is not synchronized with any external
time source on node "oel72-rac1".
Verifying resolv.conf Integrity ...FAILED
oel72-rac3: PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel72-rac3,oel72-rac1
oel72-rac3: PRVG-10048 : Name "oel72-rac3" was not resolved to an address of
the specified type by name servers o"192.168.2.1".
oel72-rac1: PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel72-rac3,oel72-rac1
oel72-rac1: PRVG-10048 : Name "oel72-rac1" was not resolved to an address of
the specified type by name servers o"192.168.2.1".

CVU operation performed: stage -pre nodeadd
Date: Oct 7, 2020 5:10:54 PM
CVU home: /u01/app/12.2.0/grid/
User: oracle
[oracle@oel72-rac1 ~]$

Note: Some failures are ignore because of "SWAP" and resolv.conf file.

Execute the "addnode.sh" option with the following parameters

[oracle@oel72-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel72-rac1 ~]$
[oracle@oel72-rac1 ~]$ cd $ORACLE_HOME
[oracle@oel72-rac1 grid]$ cd addnode/
[oracle@oel72-rac1 addnode]$
[oracle@oel72-rac1 addnode]$ ls -lrth
total 36K
-rw-r-----. 1 oracle oinstall 13K Aug 5 2016 addnode.pl
-rw-r-----. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini.sbs.ouibak
-rw-r-----. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini.sbs
-rw-r-----. 1 oracle oinstall 3.7K Jan 26 2017 addnode.sh.ouibak
-rw-r--r--. 1 oracle oinstall 2.1K Jan 26 2017 addnode_oraparam.ini

[oracle@oel72-rac1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={oel72-rac3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={oel72-rac3-vip}" "CLUSTER_NEW_NODE_ROLES={hub}" -silent -ignorePrereq
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 17% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2020-10-07_05-17-40-PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 49% Done.
Saving cluster inventory in progress.
.................................................. 83% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.2.0/grid was successful.
Please check '/u01/app/12.2.0/grid/inventory/silentInstall2020-10-07_5-17-36-PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 90% Done.
Update Inventory in progress.
Update Inventory successful.
.................................................. 97% Done.
As a root user, execute the following script(s):
1. /u01/app/oraInventory/orainstRoot.sh
2. /u01/app/12.2.0/grid/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[oel72-rac3]
Execute /u01/app/12.2.0/grid/root.sh on the following nodes:
[oel72-rac3]
The scripts can be executed in parallel on all the nodes.
.................................................. 100% Done.
Successfully Setup Software.
[oracle@oel72-rac1 addnode]$

Login to Node-3 (oel72-rac3) as a 'root' user and execute the above mentioned scripts

[oracle@oel72-rac3 ~]$ su - root
Password:
Last login: Wed Oct 7 17:02:44 EDT 2020 on pts/0
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# /u01/app/12.2.0/grid/root.sh
Check /u01/app/12.2.0/grid/install/root_oel72-rac3_2020-10-07_17-28-44-261433436.log for the output of root script
[root@oel72-rac3 ~]#
[root@oel72-rac3 ~]# tail -100f /u01/app/12.2.0/grid/install/root_oel72-rac3_2020-10-07_17-28-44-261433436.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.2.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/oel72-rac3/crsconfig/rootcrs_oel72-rac3_2020-10-07_05-29-00PM.log
2020/10/07 17:29:07 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/10/07 17:29:07 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2020/10/07 17:29:42 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/10/07 17:29:42 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/10/07 17:29:56 CLSRSC-363: User ignored prerequisites during installation
2020/10/07 17:29:56 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/10/07 17:29:56 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/10/07 17:30:02 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2020/10/07 17:30:05 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2020/10/07 17:30:07 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2020/10/07 17:30:07 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2020/10/07 17:30:09 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/10/07 17:30:18 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/10/07 17:30:18 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/10/07 17:30:20 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/10/07 17:30:35 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/10/07 17:31:14 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/10/07 17:31:47 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2020/10/07 17:32:12 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/10/07 17:32:13 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel72-rac3'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel72-rac3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel72-rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2020/10/07 17:32:33 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac3'
CRS-2676: Start of 'ora.mdnsd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel72-rac3'
CRS-2676: Start of 'ora.gpnpd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel72-rac3'
CRS-2676: Start of 'ora.gipcd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac3'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac3'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac3'
CRS-2676: Start of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac3'
CRS-2676: Start of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel72-rac3'
CRS-2676: Start of 'ora.crf' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac3'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac3' succeeded
CRS-6017: Processing resource auto-start for servers: oel72-rac3
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.chad' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.net1.network' on 'oel72-rac3'
CRS-2676: Start of 'ora.net1.network' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'oel72-rac3'
CRS-2676: Start of 'ora.chad' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.ons' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel72-rac3'
CRS-2676: Start of 'ora.scan2.vip' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.DATADG.dg' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.ACFSDG.dg' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'oel72-rac3'
CRS-2676: Start of 'ora.DATADG.dg' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.ACFSDG.dg' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.proxy_advm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3'
CRS-2676: Start of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3'
CRS-2676: Start of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3' succeeded
CRS-6016: Resource auto-start has completed for server oel72-rac3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/10/07 17:36:36 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/10/07 17:36:36 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2020/10/07 17:36:58 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/10/07 17:37:53 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Check the ASM instance status from Node-3 (oel72-rac3)

[root@oel72-rac3 ~]# ps -ef | grep pmon
oracle 14672 1 0 17:35 ? 00:00:00 asm_pmon_+ASM3
oracle 15195 1 0 17:36 ? 00:00:00 apx_pmon_+APX3
root 16471 1938 0 17:38 pts/1 00:00:00 grep --color=auto pmon
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM3
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_GG.advm
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.ACFSDG.dg
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.DATADG.dg
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.OCRVD.dg
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.acfsdg.acfs_gg.acfs
ONLINE ONLINE oel72-rac1 mounted on /mnt/acfs_gg,STABLE
ONLINE ONLINE oel72-rac2 mounted on /mnt/acfs_gg,STABLE
ONLINE ONLINE oel72-rac3 mounted on /mnt/acfs_gg,STABLE
ora.chad
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.net1.network
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.ons
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
ora.proxy_advm
ONLINE ONLINE oel72-rac1 STABLE
ONLINE ONLINE oel72-rac2 STABLE
ONLINE ONLINE oel72-rac3 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel72-rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel72-rac3 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel72-rac1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE oel72-rac2 169.254.12.248 10.1.
4.123,STABLE
ora.asm
1 ONLINE ONLINE oel72-rac1 Started,STABLE
2 ONLINE ONLINE oel72-rac2 Started,STABLE
3 ONLINE ONLINE oel72-rac3 Started,STABLE
ora.cvu
1 ONLINE ONLINE oel72-rac1 STABLE
ora.mgmtdb
1 ONLINE ONLINE oel72-rac2 Open,STABLE
ora.oel72-rac1.vip
1 ONLINE ONLINE oel72-rac1 STABLE
ora.oel72-rac2.vip
1 ONLINE ONLINE oel72-rac2 STABLE
ora.oel72-rac3.vip
1 ONLINE ONLINE oel72-rac3 STABLE
ora.orcl.db
1 ONLINE ONLINE oel72-rac1 Open,HOME=/u01/app/o
racle/product/12.1.0
.2/db_1,STABLE
2 ONLINE ONLINE oel72-rac2 Open,HOME=/u01/app/o
racle/product/12.1.0
.2/db_1,STABLE
ora.qosmserver
1 ONLINE ONLINE oel72-rac1 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel72-rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE oel72-rac3 STABLE
ora.scan3.vip
1 ONLINE ONLINE oel72-rac1 STABLE
--------------------------------------------------------------------------------
[root@oel72-rac3 ~]#

Once GI done, we are going to add the RDBMS home to the newly added node-3 (oel72-rac3). This process also we are going to use "silent"option. Create ORACLE_HOME directory in Node-3 (oel72-rac3).

[oracle@oel72-rac3 ~]$ cd /u01/app/oracle
[oracle@oel72-rac3 oracle]$

[oracle@oel72-rac3 oracle]$ mkdir -p product/12.1.0.2/db_1
[oracle@oel72-rac3 oracle]$

Login to Node-1 (oel72-rac1) and execute "addnode.sh" from ORACLE_HOME

[oracle@oel72-rac1 ~]$ ps -ef | grep pmon
oracle 6601 15385 0 21:02 pts/0 00:00:00 grep --color=auto pmon
oracle 26408 1 0 20:58 ? 00:00:00 asm_pmon_+ASM1
oracle 26574 1 0 20:58 ? 00:00:00 ora_pmon_orcl1
oracle 26662 1 0 20:58 ? 00:00:00 apx_pmon_+APX1
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcl
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ cd $ORACLE_HOME
[oracle@oel72-rac1 db_1]$ cd addnode/
[oracle@oel72-rac1 addnode]$

[oracle@oel72-rac1 addnode]$ ls -lrth
total 12K
-rw-r--r--. 1 oracle oinstall 2.0K Jul 7 2014 addnode_oraparam.ini.sbs
-rwxr-xr-x. 1 oracle oinstall 3.6K Jul 16 01:51 addnode.sh
-rw-r--r--. 1 oracle oinstall 2.0K Jul 16 01:51 addnode_oraparam.ini
[oracle@oel72-rac1 addnode]$

[oracle@oel72-rac1 addnode]$ ./addnode.sh "CLUSTER_NEW_NODES={oel72-rac3}" -silent -ignorePrereq
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 20941 MB Passed
Checking swap space: must be greater than 150 MB. Actual 10047 MB Passed
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 8% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2020-10-07_09-06-36PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 14% Done.
Copying files to node in progress.
Copying files to node successful.
.................................................. 73% Done.
Saving cluster inventory in progress.
.................................................. 80% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/12.1.0.2/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 88% Done.
As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/db_1/root.sh
Execute /u01/app/oracle/product/12.1.0.2/db_1/root.sh on the following nodes:
[oel72-rac3]
..........
Update Inventory in progress.
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
[oracle@oel72-rac1 addnode]$

Login to Node-3 (oel72-rac3) and execute the above script as a 'root' user

[root@oel72-rac3 ~]# sh /u01/app/oracle/product/12.1.0.2/db_1/root.sh
Check /u01/app/oracle/product/12.1.0.2/db_1/install/root_oel72-rac3_2020-10-07_21-14-49.log for the output of root script
[root@oel72-rac3 ~]#

[oracle@oel72-rac3 ~]$ su - root
Password:
Last login: Wed Oct 7 21:10:47 EDT 2020
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# tail -100f /u01/app/oracle/product/12.1.0.2/db_1/install/root_oel72-rac3_2020-10-07_21-14-49.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/12.1.0.2/db_1
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Login to Node-1 (oel72-rac1) prepare 3rd instance for the Node-3 (oel72-rac3)

[oracle@oel72-rac1 ~]$ . oraenv
ORACLE_SID = [orcl] ? orcl1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Wed Oct 7 21:31:16 2020
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> set lines 100 pages 200
SQL> select group#,thread#,bytes/1024/1024,members from v$log;
GROUP# THREAD# BYTES/1024/1024 MEMBERS
---------- ---------- --------------- ----------
1 1 50 2
2 1 50 2
3 2 50 2
4 2 50 2
SQL>

SQL> select group#,member from v$logfile order by group#;
GROUP# MEMBER
-----------------------------------------------------------------
1 +DATADG/ORCL/ONLINELOG/group_1.264.1045907497
1 +DATADG/ORCL/ONLINELOG/group_1.263.1045907495
2 +DATADG/ORCL/ONLINELOG/group_2.266.1045907497
2 +DATADG/ORCL/ONLINELOG/group_2.265.1045907497
3 +DATADG/ORCL/ONLINELOG/group_3.270.1045907699
3 +DATADG/ORCL/ONLINELOG/group_3.271.1045907699
4 +DATADG/ORCL/ONLINELOG/group_4.272.1045907699
4 +DATADG/ORCL/ONLINELOG/group_4.273.1045907699
8 rows selected.

Add LOGFILE with thread 3 for the 3rd Instance

SQL> alter database add logfile thread 3 group 5 size 50M;
Database altered.
SQL>

SQL> alter database add logfile thread 3 group 6 size 50M;
Database altered.
SQL>

SQL> select group#,member from v$logfile order by group#;
GROUP# MEMBER
-----------------------------------------------------------------------
1 +DATADG/ORCL/ONLINELOG/group_1.263.1045907495
1 +DATADG/ORCL/ONLINELOG/group_1.264.1045907497
2 +DATADG/ORCL/ONLINELOG/group_2.265.1045907497
2 +DATADG/ORCL/ONLINELOG/group_2.266.1045907497
3 +DATADG/ORCL/ONLINELOG/group_3.270.1045907699
3 +DATADG/ORCL/ONLINELOG/group_3.271.1045907699
4 +DATADG/ORCL/ONLINELOG/group_4.272.1045907699
4 +DATADG/ORCL/ONLINELOG/group_4.273.1045907699
5 +DATADG/ORCL/ONLINELOG/group_5.293.1053207997
5 +DATADG/ORCL/ONLINELOG/group_5.294.1053207997
6 +DATADG/ORCL/ONLINELOG/group_6.295.1053208013
6 +DATADG/ORCL/ONLINELOG/group_6.296.1053208013
12 rows selected.
SQL>

Add "UNDO TABLESPACE" for the 3rd Instance

SQL> select inst_id,name,value from gv$parameter where name like 'undo_tablespace%';
INST_ID NAME VALUE
-------------------------------------------------------------------------------------
1 undo_tablespace UNDOTBS1
2 undo_tablespace UNDOTBS2

SQL> create undo tablespace UNDOTBS3 datafile size 400M;
Tablespace created.
SQL>

SQL> alter system set undo_tablespace=UNDOTBS3 scope=spfile sid='orcl3';
System altered.
SQL>

SQL> alter system set instance_number=3 scope=spfile sid='orcl3';
System altered.
SQL>

SQL> alter system set thread=3 scope=spfile sid='orcl3';
System altered.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl config database -d orcl
Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATADG/ORCL/PARAMETERFILE/spfile.274.1045907699
Password file: +DATADG/ORCL/PASSWORD/pwdorcl.256.1045907323
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATADG
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: orcl1,orcl2
Configured nodes: oel72-rac1,oel72-rac2
Database is administrator managed
[oracle@oel72-rac1 ~]$

Add 3rd instance using "srvctl" command

[oracle@oel72-rac1 ~]$ srvctl add instance -d orcl -instance orcl3 -node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl config database -d orcl
Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATADG/ORCL/PARAMETERFILE/spfile.274.1045907699
Password file: +DATADG/ORCL/PASSWORD/pwdorcl.256.1045907323
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATADG
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: orcl1,orcl2,orcl3
Configured nodes: oel72-rac1,oel72-rac2,oel72-rac3
Database is administrator managed
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ cd $ORACLE_HOME/dbs
[oracle@oel72-rac1 dbs]$

[oracle@oel72-rac1 dbs]$ ls -lrth initorcl*
-rw-r-----. 1 oracle oinstall 37 Jul 16 09:55 initorcl1.ora
[oracle@oel72-rac1 dbs]$

[oracle@oel72-rac1 dbs]$ cat initorcl1.ora
SPFILE='+DATADG/orcl/spfileorcl.ora'
[oracle@oel72-rac1 dbs]$

[oracle@oel72-rac1 dbs]$ scp initorcl1.ora oracle@oel72-rac3:/u01/app/oracle/product/12.1.0.2/db_1/dbs/initorcl3.ora
initorcl1.ora 100% 37 0.0KB/s 00:00
[oracle@oel72-rac1 dbs]$
Check in Node-3 (ole72-rac3):

[oracle@oel72-rac3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/dbs
[oracle@oel72-rac3 dbs]$
[oracle@oel72-rac3 dbs]$ ls -lrth
total 528K
-rw-rw----. 1 oracle oinstall 512K Oct 7 21:10 id_orcl1.dat
-rw-r-----. 1 oracle oinstall 37 Oct 7 21:10 initorcl1.ora
-rw-rw----. 1 oracle oinstall 1.6K Oct 7 21:10 hc_orcl1.dat
-rw-r--r--. 1 oracle oinstall 3.0K Oct 7 21:10 init.ora
-rw-r-----. 1 oracle oinstall 37 Oct 7 22:29 initorcl3.ora
[oracle@oel72-rac3 dbs]$

[oracle@oel72-rac3 dbs]$ cat initorcl3.ora
SPFILE='+DATADG/orcl/spfileorcl.ora'
[oracle@oel72-rac3 dbs]$

[oracle@oel72-rac3 dbs]$ ps -ef | grep pmon
oracle 12444 1 0 22:32 ? 00:00:00 ora_pmon_orcl3
oracle 13046 1392 0 22:33 pts/1 00:00:00 grep --color=auto pmon
oracle 14180 1 0 20:53 ? 00:00:00 asm_pmon_+ASM3
oracle 15649 1 0 20:55 ? 00:00:00 apx_pmon_+APX3
oracle 17831 1 0 20:55 ? 00:00:00 mdb_pmon_-MGMTDB
[oracle@oel72-rac3 dbs]$

[oracle@oel72-rac3 dbs]$ ps -ef | grep tns
root 15 2 0 19:05 ? 00:00:00 [netns]
oracle 14264 1 0 19:50 ? 00:00:01 /u01/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
oracle 15338 1 0 20:55 ? 00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
oracle 15500 1 0 20:55 ? 00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
oracle 15868 1 0 19:52 ? 00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle 16200 1392 0 22:42 pts/1 00:00:00 grep --color=auto tns
[oracle@oel72-rac3 dbs]$

[oracle@oel72-rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Wed Oct 7 22:30:57 2020
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> alter database enable public thread 3;
Database altered.
SQL>

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel72-rac1
Instance orcl2 is running on node oel72-rac2
Instance orcl3 is not running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl start instance -d orcl -instance orcl3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel72-rac1
Instance orcl2 is running on node oel72-rac2
Instance orcl3 is running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl stop database -d orcl
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is not running on node oel72-rac1
Instance orcl2 is not running on node oel72-rac2
Instance orcl3 is not running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl start database -d orcl
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl config database -d orcl
Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATADG/ORCL/PARAMETERFILE/spfile.274.1045907699
Password file: +DATADG/ORCL/PASSWORD/pwdorcl.256.1045907323
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATADG
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: orcl1,orcl2,orcl3
Configured nodes: oel72-rac1,oel72-rac2,oel72-rac3
Database is administrator managed
[oracle@oel72-rac1 ~]$

Stop the complete cluster and start from the newly added Node-3 (oel72-rac3)
[root@oel72-rac3 ~]# . oraenv
ORACLE_SID = [+ASM2] ? +ASM3
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac1'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac2.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac3'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.oel72-rac2.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cvu' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac1.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.oel72-rac1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ons' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac2'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac2' has completed
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'oel72-rac3'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac2'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac3.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.oel72-rac3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac1' has completed
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac3'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac1'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac3'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac3' succeeded
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac3'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac1'
CRS-2676: Start of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac1'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2676: Start of 'ora.cssd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac2'
CRS-2676: Start of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac1'
CRS-2676: Start of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac2'
CRS-2676: Start of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac1'
CRS-2676: Start of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac1'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac2'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.crsd' on 'oel72-rac3' succeeded
[root@oel72-rac3 ~]#

Post check using "cluvfy"

[oracle@oel72-rac1 ~]$ cluvfy stage -post nodeadd -n oel72-rac3
Verifying Node Connectivity ...
Verifying Hosts File ...PASSED
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying subnet mask consistency for subnet "192.168.2.0" ...PASSED
Verifying subnet mask consistency for subnet "10.1.4.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying Node Addition ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying '/u01/app/12.2.0/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Multicast check ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
Verifying DNS/NIS name service 'racp-scan' ...
Verifying Name Service Switch Configuration File Integrity ...PASSED
Verifying DNS/NIS name service 'racp-scan' ...FAILED (PRVG-1101)
Verifying Single Client Access Name (SCAN) ...FAILED (PRVG-1101)
Verifying User Not In Group "root": oracle ...PASSED
Verifying Clock Synchronization ...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Verifying Network Time Protocol (NTP) ...
Verifying '/etc/ntp.conf' ...PASSED
Verifying '/etc/chrony.conf' ...PASSED
Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Daemon 'chronyd' ...PASSED
Verifying NTP daemon or service using UDP port 123 ...PASSED
Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
Verifying Network Time Protocol (NTP) ...FAILED (PRVG-1063)
Verifying Clock Synchronization ...FAILED (PRVG-1063, PRVG-13606)
Post-check for node addition was unsuccessful.
Checks did not pass for the following nodes:
oel72-rac3,oel72-rac1
Failures were encountered during execution of CVU verification request "stage -post nodeadd".
Verifying Single Client Access Name (SCAN) ...FAILED
Verifying DNS/NIS name service 'racp-scan' ...FAILED
PRVG-1101 : SCAN name "racp-scan" failed to resolve
Verifying Clock Synchronization ...FAILED
Verifying Network Time Protocol (NTP) ...FAILED
PRVG-1063 : configuration files for more than one time synchronization
service were found on nodes of the cluster
configuration file "/etc/chrony.conf" was found on nodes "oel72-rac3"
configuration file "/etc/ntp.conf" was found on nodes "oel72-rac3"
Verifying chrony daemon is synchronized with at least one external time
source ...FAILED
oel72-rac3: PRVG-13606 : chrony daemon is not synchronized with any
external time source on node "oel72-rac3".

CVU operation performed: stage -post nodeadd
Date: Oct 8, 2020 1:07:13 AM
CVU home: /u01/app/12.2.0/grid/
User: oracle
[oracle@oel72-rac1 ~]$

Hope it helps...

Reflection of my thoughts and technical expriences

Wednesday, October 7, 2020

Adding New Node to Existing Cluster (Two Node RAC) using silent option - Oracle 12cR2 (12.2) GI and Oracle 12cR1 (12.1.0.2) RDBMS

No comments:

Post a Comment