Configuring 3-Node RAC - Oracle Database 19c (19.3) - Oracle Enterprise Linux 8.0

Configuring 3-Node RAC for Oracle Database 19c (19.3) with April 2020 RU on Oracle Enterprise Linux 8.0 

Oracle 19c - GI and RDBMS Softwares with April 2020 RU

1. Oracle 19c RDBMS: LINUX.X64_193000_db_home.zip
2. Oracle 19c APR 2020 RU:p30869156_190000_Linux-x86-64.zip
3. Oracle 19c GI: LINUX.X64_193000_grid_home.zip
4. Oracle 19c APR 2020 RU:p30899722_190000_Linux-x86-64.zip

I have shared in this blog post major steps.

1. Installed Oracle Linux 8.0 (Server with GUI) and configured 3 Network Interfaces

Go to “19c-rac1” VM Settings, 

Click on Network 

1. Select “Adapter 1” and Click on “Enable Network Adapter” and Select “NAT” as Attached to 
2. Select “Adapter 2” and Click on “Enable Network Adapter” and Select “Host-Only Adapter” as Attached to 
3. Select “Adapter 3” and Click on “Enable Network Adapter” and Select “Internal Network” as Attached to

a. NAT
b. “Host-Only Adapter” - Public IP: 192.168.2.10/255.255.255.0
c. “Internal Network”  - Private IP: 10.1.4.10/255.255.255.0

Repeat the step:1 for other two nodes and finally NAT, Public IP, Private IP and SCAN IPs 

19c-rac1:
---------
a. NAT
b. Public IP: 192.168.2.10/255.255.255.0
c. Private IP: 10.1.4.10/255.255.255.0

19c-rac2:
--------
a. NAT
b. Public IP: 192.168.2.20/255.255.255.0
c. Private IP: 10.1.4.20/255.255.255.0

19c-rac3:
---------
a. NAT
b. Public IP: 192.168.2.30/255.255.255.0
c. Private IP: 10.1.4.30/255.255.255.0

[root@19c-rac1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.10 19c-rac1.localdomain          19c-rac1
10.1.4.10    19c-rac1-priv.localdomain     19c-rac1-priv
192.168.2.11 19c-rac1-vip.localdomain      19c-rac1-vip

192.168.2.20 19c-rac2.localdomain          19c-rac2
10.1.4.20    19c-rac2-priv.localdomain     19c-rac2-priv
192.168.2.21 19c-rac2-vip.localdomain      19c-rac2-vip

192.168.2.30 19c-rac3.localdomain          19c-rac3
10.1.4.30    19c-rac3-priv.localdomain     19c-rac3-priv
192.168.2.31 19c-rac3-vip.localdomain      19c-rac3-vip

192.168.2.40 rac-scan.localdomain          rac-scan
192.168.2.50 rac-scan.localdomain          rac-scan
192.168.2.60 rac-scan.localdomain          rac-scan
[root@19c-rac1 ~]# 

Note-1: Information from /etc/hosts
Note-2: Same for 19c-rac2 and 19c-rac3 from /etc/hosts

How to install and activate Cockpit web console on OEL 8


[root@19c-rac1 ~]# systemctl enable --now cockpit.socket
Created symlink /etc/systemd/system/sockets.target.wants/cockpit.socket ? /usr/lib/systemd/system/cockpit.socket.

[root@19c-rac1 ~]# sudo ss -tulpn | grep :9090
tcp    LISTEN   0        128                     *:9090                 *:*      users:(("systemd",pid=1,fd=23))
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# sudo firewall-cmd --add-service=cockpit
Warning: ALREADY_ENABLED: 'cockpit' already in 'public'
success

[root@19c-rac1 ~]# sudo firewall-cmd --add-service=cockpit --permanent
Warning: ALREADY_ENABLED: cockpit
success
[root@19c-rac1 ~]#

Pre-installation tasks - - > Repeat 19c-rac2 and 19c-rac3 nodes also

[root@19c-rac1 ~]# dnf install -y https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm
Last metadata expiration check: 5:55:02 ago on Tue 12 May 2020 10:11:38 AM EDT.
oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm                                                                                   46 kB/s |  24 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================
 Package                                            Architecture               Version                                   Repository                             Size
=====================================================================================================================================================================
Installing:
 oracle-database-preinstall-19c                     x86_64                     1.0-1.el8                                 @commandline                           24 k
Installing dependencies:
 ksh                                                x86_64                     20120801-254.0.1.el8                      ol8_appstream                         927 k
 libaio-devel                                       x86_64                     0.3.112-1.el8                             ol8_baseos_latest                      19 k
 libnsl                                             x86_64                     2.28-101.0.1.el8                          ol8_baseos_latest                      97 k

Transaction Summary
=====================================================================================================================================================================
Install  4 Packages

Total size: 1.0 M
Total download size: 1.0 M
Installed size: 3.5 M
Downloading Packages:
(1/3): libaio-devel-0.3.112-1.el8.x86_64.rpm         107 kB/s |  19 kB     00:00
(2/3): libnsl-2.28-101.0.1.el8.x86_64.rpm            446 kB/s |  97 kB     00:00
(3/3): ksh-20120801-254.0.1.el8.x86_64.rpm           1.8 MB/s | 927 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                2.0 MB/s | 1.0 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                          1/1
  Installing       : ksh-20120801-254.0.1.el8.x86_64                          1/4
  Running scriptlet: ksh-20120801-254.0.1.el8.x86_64                          1/4
  Installing       : libaio-devel-0.3.112-1.el8.x86_64                        2/4
  Installing       : libnsl-2.28-101.0.1.el8.x86_64                           3/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Installing       : oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Verifying        : libnsl-2.28-101.0.1.el8.x86_64                           1/4
  Verifying        : libaio-devel-0.3.112-1.el8.x86_64                        2/4
  Verifying        : ksh-20120801-254.0.1.el8.x86_64                          3/4
  Verifying        : oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4

Installed:
ksh-20120801-254.0.1.el8.x86_64            
libaio-devel-0.3.112-1.el8.x86_64            
libnsl-2.28-101.0.1.el8.x86_64            
oracle-database-preinstall-19c-1.0-1.el8.x86_64

Complete!
[root@19c-rac1 ~]#

Note: If you have not used the "oracle-database-preinstall-19c" package to perform all prerequisites, 
you will need to manually perform the setup tasks.


[root@rac2-19cr7-oel82 ~]# cat /etc/sysctl.conf
[root@19c-rac1 ~]# cat /etc/security/limits.d/oracle-database-preinstall-19c.conf
[root@19c-rac1 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

[root@19c-rac1 ~]# vi /etc/selinux/config
[root@19c-rac1 ~]# setenforce Permissive

[root@19c-rac1 ~]# systemctl stop firewalld
[root@19c-rac1 ~]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# mkdir -p /u01/app/19.3.0/grid
[root@19c-rac1 ~]# mkdir -p /u01/app/oracle/product/19.3.0/dbhome_1
[root@19c-rac1 ~]# chown -R oracle:oinstall /u01
[root@19c-rac1 ~]# chmod -R 775 /u01/

[oracle@19c-rac1 ~]$ vi grid_env
[oracle@19c-rac1 ~]$ cat grid_env
export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ vi db_env
[oracle@19c-rac1 ~]$ cat db_env
export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

[oracle@19c-rac1 ~]$
[oracle@19c-rac1 ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs

export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=19c-rac1
export ORACLE_UNQNAME=cdbrac
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/19.3.0/grid
export DB_HOME=$ORACLE_BASE/product/19.3.0/dbhome_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
[oracle@19c-rac1 ~]$

[root@19c-rac1 rpm]# pwd
/u01/app/19.3.0/grid/cv/rpm

[root@19c-rac1 rpm]# export CVUQDISK_GRP=dba
[root@19c-rac1 rpm]# rpm -ivh cvuqdisk-1.0.10-1.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:cvuqdisk-1.0.10-1                ################################# [100%]
[root@19c-rac1 rpm]#

[root@19c-rac1 rpm]# rpm -qa | grep cvuqdisk
cvuqdisk-1.0.10-1.x86_64
[root@19c-rac1 rpm]#

2. Create Shared Disks 

1. Choose “Fixed size” 
2. Best practice to Choose File location outside of existing VM directory and input “12 GB” as storage space 
3. Click “Create

3. Sharing Shared Disks 

1. Go to “File” > “Virtual Media Manager” 
2. Right click on Shared disk and make it Shareable. 
3. Repeat the task for all eight shared disk


4. Formatting Shared Disks Start the “19c-rac1” VM and login a root user. Now we have to configure the Shared disks.

1. Go to /dev file system and look for sd* using ls command. Since we have added four shared disks, 
you can see sdb,sdc,sdd,sde,sdf,sdg,sdh and sdi 

2. Format shared disks using “fdisk” command as shown below. 
3. Once format is completed, you can see sdb1,sdc1,sdd1,sde1,sdf1,sdg1,sdh1 and sdi1


[root@19c-rac1 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x3c82e2b8.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-25165823, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-25165823, default 25165823):

Created a new partition 1 of type 'Linux' and of size 12 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[root@19c-rac1 ~]#

Note: Repeat the same process for sdc,sde,sdf,sdg,sdh and sdi

5. Configure your UDEV rules 

1. Add “options=-g” to "/etc/scsi_id.config" file. 
2. Find SCSI ID of all eight disk using “scsi_id” command as shown above. 

[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VB422666b7-031705cb
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdd
1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sde
1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdf
1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdg
1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdh
1ATA_VBOX_HARDDISK_VBa855f734-9a09207e
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi
1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d
[root@19c-rac1 ~]#

3. Using scsi_id, edit "/etc/udev/rules.d/99-oracle-asmdevices.rules" file as shown above. 

[root@19c-rac1~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
[root@19c-rac1~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"

4. Load updated block device partition tables.

Note: Clone 19c-rac1 node as 19c-rac2 and 19c-rac3.

19c-rac2

[oracle@19c-rac2 ~]$ su - root
Password:
[root@19c-rac2 ~]# cat /etc/scsi_id.config
options=-g

[root@19c-rac2 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"
[root@19c-rac2 ~]#

19c-rac3:

[oracle@19c-rac3 ~]$ su - root
Password:
[root@19c-rac3 ~]# cat /etc/scsi_id.config
options=-g

[root@19c-rac3 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"
[root@19c-rac3 ~]#


Installation Oracle GRID Infrastructure 19c (19.3.0) and applying APRIL 2020 RU 19c (19.7.0) 

[oracle@19c-rac1 ~]$ export CV_ASSUME_DISTID=OEL7.6
[oracle@19c-rac1 grid]$ ./gridSetup.sh -applyPSU /u01/app/19.3.0/grid/30899722/

Preparing the home to patch...
Applying the patch /u01/app/19.3.0/grid/30899722/...
Successfully applied the patch.
The log can be found at: /tmp/GridSetupActions2020-05-16_08-47-40PM/installerPatchActions_2020-05-16_08-47-40PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at:
 /u01/app/19.3.0/grid/install/response/grid_2020-05-16_08-47-40PM.rsp

You can find the log of this install session at:
 /tmp/GridSetupActions2020-05-16_08-47-40PM/gridSetupActions2020-05-16_08-47-40PM.log
Moved the install session logs to:
 /u01/app/oraInventory/logs/GridSetupActions2020-05-16_08-47-40PM
[oracle@19c-rac1 grid]$









[root@19c-rac1 ~]# sh /tmp/GridSetupActions2020-05-18_01-13-22AM/CVU_19.0.0.0.0_oracle/runfixup.sh
All Fix-up operations were completed successfully.
[root@19c-rac1 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac3 ~]#


[root@19c-rac1 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac1/crsconfig/rootcrs_19c-rac1_2020-05-18_01-45-03AM.log
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:45:14 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:45:16 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:45:46 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:45:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:46:06 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:46:12 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:46:12 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:46:17 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:46:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:46:37 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:46:37 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:46:41 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:46:46 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /u01/app/oracle/cfgtoollogs/asmca/asmca-200518AM014717.log for details.

2020/05/18 01:48:14 CLSRSC-482: Running command: '/u01/app/19.3.0/grid/bin/ocrconfig -upgrade oracle oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk 0f7b1a25f7bf4f32bf8fa0f50f003b75.
Successfully replaced voting disk group with +OCRVD.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0f7b1a25f7bf4f32bf8fa0f50f003b75 (/dev/sdb1) [OCRVD]
Located 1 voting disk(s).
2020/05/18 01:49:09 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:50:12 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:50:12 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:51:36 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:51:59 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac2/crsconfig/rootcrs_19c-rac2_2020-05-18_01-52-26AM.log
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:52:33 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:52:36 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:52:39 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:52:39 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:52:57 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:52:57 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:52:59 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:53:00 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:53:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:53:20 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:53:20 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:53:22 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:53:23 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service
2020/05/18 01:53:32 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:54:18 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:54:18 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:54:35 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:54:41 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac3/crsconfig/rootcrs_19c-rac3_2020-05-18_01-55-10AM.log
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:55:17 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:55:19 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:55:21 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:55:21 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:55:39 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:55:40 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:55:42 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:55:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:56:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:56:03 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:56:03 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:56:05 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:56:06 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service
2020/05/18 01:56:15 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:57:01 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:57:01 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:57:22 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:57:30 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac3 ~]#

Configuring diskgroups DATADG and RECODG through ASMCA  





Check the status of ASM Instance from 19c-rac1, 19c-rac2 and 19c-rac3 nodes

[oracle@19c-rac1 ~]$ ps -ef | grep pmon
oracle    2071  2333  0 09:41 pts/0    00:00:00 grep --color=auto pmon
oracle    7081     1  0 01:52 ?        00:00:01 asm_pmon_+ASM1
oracle    5698     1  0 09:41 ?        00:00:00 mdb_pmon_-MGMTDB
[oracle@19c-rac1 ~]$

[oracle@19c-rac2 ~]$ ps -ef | grep pmon
oracle    5936  5895  0 09:43 pts/0    00:00:00 grep --color=auto pmon
oracle    8270     1  0 01:54 ?        00:00:01 asm_pmon_+ASM2
[oracle@19c-rac2 ~]$ 

[oracle@19c-rac3 ~]$ ps -ef | grep pmon
oracle    2371  2333  0 09:44 pts/0    00:00:00 grep --color=auto pmon
oracle    7981     1  0 01:57 ?        00:00:01 asm_pmon_+ASM3
[oracle@19c-rac3 ~]$

Installation of Oracle Database 19c (19.3.0) Software 

Note: Select "Setup Software Only" Option



[root@19c-rac1 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac3 ~]#

Creation Oracle Database 19c (Container Database with 2 PDBs) 









Check ASM Instances and RDBMS Instances

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [cdbrac1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$ cat /etc/os-release | grep PRETTY
PRETTY_NAME="Oracle Linux Server 8.0"
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon May 18 09:48:51 2020
Version 19.7.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
+ASM2                          2
+ASM3                          3
+ASM1                          1

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? cdbrac
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/19.3.0/dbhome_1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus sys@cdbrac as sysdba


SQL*Plus: Release 19.0.0.0.0 - Production on Mon May 18 09:50:31 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Enter password:
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 PDB2                           READ WRITE NO
SQL>

SQL> select instance_name,instance_number from gv$instance order by instance_number;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
cdbrac1                        1
cdbrac2                        2
cdbrac3                        3

SQL>

Applying APRIL 2020 RU (p30869156_190000_Linux-x86-64.zip) for Oracle 19c (19.3.0)

[root@19c-rac1 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 09:59:03 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_09-59-09AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_09-59-44AM.log
The id for this session is 7R66

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac1
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-02-12AM_1.log

OPatchauto session completed at Mon May 18 10:08:41 2020
Time taken to complete the session 9 minutes, 38 seconds
[root@19c-rac1 ~]#


[root@19c-rac2 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 10:10:57 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_10-11-04AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_10-11-22AM.log
The id for this session is YZ9L

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac2
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-13-50AM_1.log
OPatchauto session completed at Mon May 18 10:20:10 2020
Time taken to complete the session 9 minutes, 13 seconds
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 10:20:34 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_10-20-42AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_10-21-03AM.log
The id for this session is 3BT9

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1

SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac3
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-23-31AM_1.log
OPatchauto session completed at Mon May 18 11:04:59 2020
Time taken to complete the session 44 minutes, 26 seconds
[root@19c-rac3 ~]#


Check the status, stop and start cluster for all the nodes

[root@19c-rac1 ~]# crsctl check cluster -all
**************************************************************
19c-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
19c-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
19c-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.qosmserver' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac3'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac3'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac2' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '19c-rac2'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac3' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac3'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac3'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac1'
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.19c-rac2.vip' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.scan1.vip' on '19c-rac2'
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac3.vip' on '19c-rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.19c-rac2.vip' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.cvu' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.19c-rac3.vip' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.scan1.vip' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac2'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac2' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac2' succeeded.
CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac2'
CRS-2677: Stop of 'ora.ons' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac2'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac2'
CRS-2677: Stop of 'ora.storage' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac3'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac3' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac3' succeeded.
CRS-2677: Stop of 'ora.ctssd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac3'
CRS-2677: Stop of 'ora.ons' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac3'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac3'
CRS-2677: Stop of 'ora.storage' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac2'
CRS-2677: Stop of 'ora.cssd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac3'
CRS-2677: Stop of 'ora.evmd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac3'
CRS-2677: Stop of 'ora.cssd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on '19c-rac1'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1'
CRS-2677: Stop of 'ora.MGMTLSNR' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac1' succeeded.
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac1.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.19c-rac1.vip' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac1'
CRS-2677: Stop of 'ora.ons' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac1'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac1'
CRS-2677: Stop of 'ora.storage' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac1'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac1'
CRS-2677: Stop of 'ora.cssd' on '19c-rac1' succeeded
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac3'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac2'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac3'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac3'
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac1'
CRS-2676: Start of 'ora.diskmon' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.diskmon' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac3'
CRS-2676: Start of 'ora.evmd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.diskmon' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.cssd' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac3'
CRS-2676: Start of 'ora.cssd' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac3'
CRS-2676: Start of 'ora.cssd' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac1'
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac1'
CRS-2676: Start of 'ora.ctssd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.ctssd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.ctssd' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac3'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac2'
CRS-2676: Start of 'ora.asm' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac1'
CRS-2672: Attempting to start 'ora.storage' on '19c-rac2'
CRS-2676: Start of 'ora.asm' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on '19c-rac1'
CRS-2676: Start of 'ora.asm' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on '19c-rac3'
CRS-2676: Start of 'ora.storage' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac3'
CRS-2676: Start of 'ora.crsd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.storage' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac2'
CRS-2676: Start of 'ora.storage' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac1'
CRS-2676: Start of 'ora.crsd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.crsd' on '19c-rac1' succeeded
[root@19c-rac1 ~]#

Check the Status as "Oracle" User

[oracle@19c-rac1 ~]$ srvctl status database -db cdbrac
Instance cdbrac1 is running on node 19c-rac1
Instance cdbrac2 is running on node 19c-rac2
Instance cdbrac3 is running on node 19c-rac3
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ srvctl config database -d cdbrac
Database unique name: cdbrac
Database name: cdbrac
Oracle home: /u01/app/oracle/product/19.3.0/dbhome_1
Oracle user: oracle
Spfile: +DATADG/CDBRAC/PARAMETERFILE/spfile.275.1040722631
Password file: +DATADG/CDBRAC/PASSWORD/pwdcdbrac.256.1040721721
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECODG,DATADG
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: cdbrac1,cdbrac2,cdbrac3
Configured nodes: 19c-rac1,19c-rac2,19c-rac3
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
[oracle@19c-rac1 ~]$

Hope it helps...........Thank You