3-Node RAC (Newly Added Node) - CRS-4404: The following nodes did not reply within the allotted time: Compute Node 3

Existing Cluster: oel72-rac1 and oel72-rac2

Newly Added Node: oel72-rac3

Checking the Cluster Database Status:

[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel72-rac1
Instance orcl2 is running on node oel72-rac2
Instance orcl3 is running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl stop database -d orcl
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is not running on node oel72-rac1
Instance orcl2 is not running on node oel72-rac2
Instance orcl3 is not running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ srvctl start database -d orcl
[oracle@oel72-rac1 ~]$
[oracle@oel72-rac1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel72-rac1
Instance orcl2 is running on node oel72-rac2
Instance orcl3 is running on node oel72-rac3
[oracle@oel72-rac1 ~]$

[oracle@oel72-rac1 ~]$ su - root
Password:
Last login: Sun Oct  4 22:00:52 EDT 2020
[root@oel72-rac1 ~]#

[root@oel72-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel72-rac1 ~]#

[root@oel72-rac1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac1'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.oel72-rac2.vip' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.oel72-rac1.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.oel72-rac1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.oel72-rac2.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac2'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac2'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac2'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac1'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-4404: The following nodes did not reply within the allotted time:
oel72-rac3
[root@oel72-rac1 ~]# 

Note: Not able to stop the newly added node (oel72-rac3). Stopped from the node oel72-rac3

Stopped cluster from oel72-rac3:

[root@oel72-rac3 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.oel72-rac3.vip' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel72-rac3'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.cvu' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.oel72-rac3.vip' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac2.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.scan3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac3'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'oel72-rac3'
CRS-2677: Stop of 'ora.oel72-rac2.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac1.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.oel72-rac1.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac3'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac3'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac3' succeeded
[root@oel72-rac3 ~]#

Again started start cluster completely from all the three nodes (oel72-rac1 & oel72-rac2) and separately (oel72-rac3).

Checked cluster status from all the three nodes

oel72-rac1:
===========
[root@oel72-rac1 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
CRS-4404: The following nodes did not reply within the allotted time:
oel72-rac3
[root@oel72-rac1 ~]#

 
oel72-rac2:
===========
[root@oel72-rac2 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel72-rac2 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel72-rac2 ~]#

oel72-rac3:
===========
[root@oel72-rac3 ~]# crsctl check cluster -all
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
CRS-4404: The following nodes did not reply within the allotted time:
oel72-rac1
[root@oel72-rac3 ~]#

So, even cluster status not able to check from the nodes (oel72-rac1 & oel72-rac3) but able to check to from the second node (oel72-rac2).

Note: I am able to stop and start complete cluster from the node ole72-rac2

[root@oel72-rac2 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac3'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.oel72-rac1.vip' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.oel72-rac2.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.oel72-rac2.vip' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.oel72-rac1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac2'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac1'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac1'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac2'
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac2'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac2' has completed
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac2'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.scan3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac3'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3'
CRS-2677: Stop of 'ora.qosmserver' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac3.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.oel72-rac3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.ons' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac3'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac3'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac3' succeeded
[root@oel72-rac2 ~]#

[root@oel72-rac2 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
oel72-rac2:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
oel72-rac3:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************

[root@oel72-rac2 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac2'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac1'
CRS-2676: Start of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac3'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac1'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac2'
CRS-2676: Start of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac2'
CRS-2676: Start of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac2'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac1'
CRS-2676: Start of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac1'
CRS-2676: Start of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac1'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac3'
CRS-2676: Start of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac3'
CRS-2676: Start of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac3'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac3' succeeded
[root@oel72-rac2 ~]#

Checked the clusterware alert.log (gpnpd is stuck). I have restarted gpnpd on the bad node (oel72-rac3)

Note: After a few minutes, gpnpd.bin should be automatically restarted by GI and check the status again

[root@oel72-rac3 ~]# ps -ef | grep gpnpd.bin
oracle    5326     1  0 16:57 ?        00:00:29 /u01/app/12.2.0/grid/bin/gpnpd.bin
root     22582 27233  0 23:16 pts/0    00:00:00 grep --color=auto gpnpd.bin
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# ps -ef | grep pnp
oracle    5326     1  0 16:57 ?        00:00:29 /u01/app/12.2.0/grid/bin/gpnpd.bin
root     22624 27233  0 23:16 pts/0    00:00:00 grep --color=auto pnp
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# kill -HUP 5326
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# ps -ef | grep gpnpd.bin
oracle   22719     1  0 23:17 ?        00:00:00 /u01/app/12.2.0/grid/bin/gpnpd.bin
root     22806 27233  0 23:18 pts/0    00:00:00 grep --color=auto gpnpd.bin
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# ps -ef | grep pnp
oracle   22719     1  0 23:17 ?        00:00:00 /u01/app/12.2.0/grid/bin/gpnpd.bin
root     22819 27233  0 23:18 pts/0    00:00:00 grep --color=auto pnp
[root@oel72-rac3 ~]#

[root@oel72-rac3 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel72-rac3 ~]#

Now, try to stop the complete cluster from the first node (oel72-rac1)

[root@oel72-rac1 ~]# ps -ef | grep gpnpd.bin
root      3472 30446  0 23:17 pts/0    00:00:00 grep --color=auto gpnpd.bin
oracle    5976     1  0 16:56 ?        00:00:30 /u01/app/12.2.0/grid/bin/gpnpd.bin
[root@oel72-rac1 ~]#

[root@oel72-rac1 ~]# ps -ef | grep pnp
root      3504 30446  0 23:18 pts/0    00:00:00 grep --color=auto pnp
oracle    5976     1  0 16:56 ?        00:00:30 /u01/app/12.2.0/grid/bin/gpnpd.bin
[root@oel72-rac1 ~]#

[root@oel72-rac1 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel72-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel72-rac1 ~]#

Finally, I was able to stop the complete cluster from all the nodes using oel72-rac1

[root@oel72-rac1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel72-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel72-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac1'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel72-rac2'
CRS-2677: Stop of 'ora.acfsdg.acfs_gg.acfs' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ACFSDG.ACFS_GG.advm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3'
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac1'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.oel72-rac1.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac3.vip' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac1'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2677: Stop of 'ora.oel72-rac1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.oel72-rac3.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.ACFSDG.dg' on 'oel72-rac2'
CRS-2677: Stop of 'ora.scan2.vip' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.ACFSDG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.cvu' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.scan3.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac1'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac1' has completed
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac3'
CRS-2677: Stop of 'ora.chad' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.ons' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac3'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac1'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'oel72-rac2'
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac3'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac3'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac3'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'oel72-rac2'
CRS-2677: Stop of 'ora.DATADG.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.oel72-rac2.vip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.oel72-rac2.vip' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac1'
CRS-2677: Stop of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac3'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ons' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel72-rac2'
CRS-2677: Stop of 'ora.net1.network' on 'oel72-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel72-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel72-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'oel72-rac2'
CRS-2677: Stop of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel72-rac2'
CRS-2677: Stop of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel72-rac2'
CRS-2677: Stop of 'ora.cssd' on 'oel72-rac2' succeeded
[root@oel72-rac1 ~]#

[root@oel72-rac1 ~]# crsctl check cluster -all
**************************************************************
oel72-rac1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
oel72-rac2:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
oel72-rac3:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************

[root@oel72-rac1 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel72-rac3'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel72-rac2'
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.diskmon' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac1'
CRS-2676: Start of 'ora.cssd' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel72-rac3'
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.ctssd' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac1'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac3'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel72-rac2'
CRS-2676: Start of 'ora.asm' on 'oel72-rac3' succeeded
CRS-2676: Start of 'ora.asm' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac3'
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac2'
CRS-2676: Start of 'ora.asm' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel72-rac1'
CRS-2676: Start of 'ora.storage' on 'oel72-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac1'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac1' succeeded
CRS-2676: Start of 'ora.storage' on 'oel72-rac2' succeeded
CRS-2676: Start of 'ora.storage' on 'oel72-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac2'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel72-rac3'
CRS-2676: Start of 'ora.crsd' on 'oel72-rac3' succeeded
[root@oel72-rac1 ~]#

Please refer for more details: "crsctl check cluster -all" command gives CRS-4404, CRS-4405 errors (Doc ID 1392934.1).

Hope it helps...