Sunday, November 29, 2020

PRCR-1079 CRS-5017 ORA-00821 CRS-2674 CRS-2632 -> ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

 [oracle@rac-p1 ~]$ . oraenv
ORACLE_SID = [racp1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:07:01 2020
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> show parameter streams_pool

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
streams_pool_size                    big integer 0
SQL>

SQL> alter system set streams_pool_size=1024M scope=spfile sid='*';
System altered.

After changing streams_pool_size

[oracle@rac-p1 ~]$ srvctl stop database -d racp
[oracle@rac-p1 ~]$ srvctl start database -d racp
PRCR-1079 : Failed to start resource ora.racp.db
CRS-5017: The resource action "ora.racp.db start" encountered the following error:
ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac-p1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.racp.db' on 'rac-p1' failed
CRS-5017: The resource action "ora.racp.db start" encountered the following error:
ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac-p2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.racp.db' on 'rac-p2' failed
CRS-2632: There are no more servers to try to place resource 'ora.racp.db' on that would satisfy its placement policy
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ . oraenv
ORACLE_SID = [racp] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:12:11 2020
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup mount;
ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M
SQL> exit
Disconnected
[oracle@rac-p1 ~]$

The instance doesn’t come up! This is easy to fix without having to restore the spfile from backup:

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:28:35 2020
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M
SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora' from spfile='+DATA/racp/spfileracp.ora';
File created.

SQL> host vi /u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora

Now correct the value in the text file. I just changed the sga_target parameter from it here.

racp1.__sga_target=3072M
racp2.__sga_target=3072M
*.sga_target=3072M
*.streams_pool_size=256M

SQL> create spfile='+DATA/racp/spfileracp.ora' from pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora';

File created.

SQL> startup force;
ORACLE instance started.

Total System Global Area 3221225472 bytes
Fixed Size                  2929552 bytes
Variable Size             822086768 bytes
Database Buffers         2382364672 bytes
Redo Buffers               13844480 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp
Instance racp1 is running on node rac-p1
Instance racp2 is not running on node rac-p2
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl stop database -d racp
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp
Instance racp1 is not running on node rac-p1
Instance racp2 is not running on node rac-p2
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl start database -d racp
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp
Instance racp1 is running on node rac-p1
Instance racp2 is running on node rac-p2
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ . oraenv
ORACLE_SID = [racp1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:43:57 2020
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
READ WRITE           PRIMARY
READ WRITE           PRIMARY

SQL> show parameter sga_

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_max_size                         big integer 3G
sga_target                           big integer 3G
unified_audit_sga_queue_size         integer     1048576
SQL>

SQL> show parameter streams_p

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
streams_pool_size                    big integer 256M
SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1130.ora' from spfile='+DATA/racp/spfileracp.ora';
File created.
SQL>

Checking Node-2 (rac-p2)

[oracle@rac-p2 ~]$ . oraenv
ORACLE_SID = [racp2] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac-p2 ~]$

[oracle@rac-p2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:19:00 2020
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
READ WRITE           PRIMARY
READ WRITE           PRIMARY

SQL> show parameter sga_

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_max_size                         big integer 3G
sga_target                           big integer 3G
unified_audit_sga_queue_size         integer     1048576
SQL>

SQL> show parameter streams_p

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
streams_pool_size                    big integer 256M
SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp2_1130.ora' from spfile='+DATA/racp/spfileracp.ora';
File created.

SQL>


Tuesday, November 17, 2020

Oracle 19c (19.9.0) with ASMFD Driver, Kernel 5.4.17 and Oracle Enterprise Linux 8.2 (3-Node RAC)

1. Installed OEL 8.2
2. Upgraded Kernel "kernel-uek-4.18.0-193.13.2.el8_2.x86_64" to kernel-uek-5.4.17-2036.100.6.1.el8uek.x86_64
3. Installed GI 19.3 (while GI installation applied Oracle 19c GI (19.8) RU)

[oracle@19c-rac1 ~]$ cd /u01/app/19.3.0/grid/

[oracle@19c-rac1 grid]$ ./gridSetup.sh -applyPSU /home/oracle/31305339/
Preparing the home to patch...
Applying the patch /home/oracle/31305339/...
Successfully applied the patch.
The log can be found at: /tmp/GridSetupActions2020-11-12_09-01-20PM/installerPatchActions_2020-11-12_09-01-20PM.log
Launching Oracle Grid Infrastructure Setup Wizard...
The response file for this session can be found at:
 /u01/app/19.3.0/grid/install/response/grid_2020-11-12_09-01-20PM.rsp
You can find the log of this install session at:
 /tmp/GridSetupActions2020-11-12_09-01-20PM/gridSetupActions2020-11-12_09-01-20PM.log
Moved the install session logs to:
 /u01/app/oraInventory/logs/GridSetupActions2020-11-12_09-01-20PM
[oracle@19c-rac1 grid]$

[oracle@19c-rac1 bin]$ afddriverstate installed
AFD-620: AFD is not supported on this operating system version: 'unknown'
AFD-9204: AFD device driver installed status: 'false'
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ afddriverstate loaded
AFD-620: AFD is not supported on this operating system version: 'unknown'
AFD-9206: AFD device driver loaded status: 'false'
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ afddriverstate version
AFD-620: AFD is not supported on this operating system version: 'unknown'
AFD-642: AFD not installed
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ cat /etc/oracle-release
Oracle Linux Server release 8.2
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ uname -r
5.4.17-2036.100.6.1.el8uek.x86_64
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ acfsload start
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
ACFS-9130: Root access required
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ su - root
Password:
[root@19c-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./afddriverstate supported
AFD-620: AFD is not supported on this operating system version: 'unknown'
AFD-9201: Not Supported
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
ACFS-9204: false
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
ACFS-9204: false
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
ACFS-9201: Not Supported
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
ACFS-9129: ADVM/ACFS not installed
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_configure
ASMCMD-9520: AFD is not 'supported'
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# acfsroot install
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# cat /etc/oracle-release
Oracle Linux Server release 8.2
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# uname -r
5.4.17-2036.100.6.1.el8uek.x86_64
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# uname -a
Linux 19c-rac1.localdomain 5.4.17-2036.100.6.1.el8uek.x86_64 #2 SMP Thu Oct 29 17:06:00 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@19c-rac1 bin]#
[root@19c-rac1 bin]# exit
logout

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [cdbrac1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME
[oracle@19c-rac1 grid]$ cd OPatch/

[oracle@19c-rac1 OPatch]$ ./opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
OPatch succeeded.
[oracle@19c-rac1 OPatch]$

Monday, November 9, 2020

Oracle 19c (19.9.0) with ASMFD Driver, Kernel 4.1.12 and Oracle Linux 7.8 (3-Node RAC)

Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a multi-platform, scalable file system, and storage management technology that extends Oracle Automatic Storage Management (Oracle ASM) functionality to support customer files maintained outside of Oracle Database.

Environment:
  • GRID Version: Oracle 19c (19.3.0) 
    • Applied RU on GRID: p31750108_190000_Linux-x86-64.zip (RU Oracle 19c (19.9.0) Oct 2020)
  • RDBMS Version: Oracle 19c (19.3.0)
    • Applied RU on RDBMS: p31771877_190000_Linux-x86-64.zip (RU Oracle 19c (19.9.0) Oct 2020)
  • OEL Version: Oracle Linux Server 7.8
  • Kernel Version: 4.1.12-124.41.5.el7uek.x86_64
Please refer for more details: 
ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Please refer for more details:  (19c ASMFD Driver and Oracle Linux 8)
https://community.oracle.com/tech/apps-infra/discussion/4341677/19c-asmfd-driver-and-oracle-linux-8

Added ss5.vdi and ss4.vdi disks 10 GB each for ACFS file system with ACFS volume

[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# export ORACLE_BASE=/u01/app/oracle
[root@rac1-19c ~]# export ORACLE_HOME=/u01/app/19.3.0/grid
[root@rac1-19c ~]# cd $ORACLE_HOME/bin
[root@rac1-19c bin]#

[root@rac1-19c bin]# asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     61432    26704                0           26704              0             Y  DATADG/

[root@rac1-19c bin]# ls -alrt /dev/oracleafd/disks/*
-rw-rw-r--. 1 oracle oinstall 10 Nov  8 20:30 /dev/oracleafd/disks/DATADG02
-rw-rw-r--. 1 oracle oinstall 10 Nov  8 20:30 /dev/oracleafd/disks/DATADG01
[root@rac1-19c bin]#

[root@rac1-19c bin]# ./acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9325:     Driver OS kernel version = 4.1.12-112.16.4.el7uek.x86_64.
ACFS-9326:     Driver build number = 190222.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9547:     Driver available build number = 190222.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9549:     Kernel and command versions.
Kernel:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
Commands:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9322: completed
[root@rac1-19c bin]#

[root@rac1-19c bin]# ./acfsdriverstate loaded
ACFS-9203: true
[root@rac1-19c bin]#

[root@rac1-19c bin]# ./acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 4.1.12-112.16.4.el7uek.x86_64.
ACFS-9326:     Driver build number = 190222.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9547:     Driver available build number = 190222.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.3.0.0.0).
[root@rac1-19c bin]#

[root@rac1-19c bin]# ./acfsdriverstate supported
ACFS-9200: Supported
[root@rac1-19c bin]#

Stamping with AFD label for newly added disks

[root@rac1-19c bin]# ./asmcmd afd_label ACFSDG01 /dev/sdd
[root@rac1-19c bin]# ./asmcmd afd_label ACFSDG02 /dev/sde

[root@rac1-19c bin]# ls -alrt /dev/oracleafd/disks/
total 16
drwxrwx---. 3 oracle asmadmin  80 Nov  8 20:30 ..
-rw-rw-r--. 1 oracle oinstall  10 Nov  8 20:30 DATADG02
-rw-rw-r--. 1 oracle oinstall  10 Nov  8 20:30 DATADG01
-rw-rw-r--. 1 oracle oinstall   9 Nov  8 20:45 ACFSDG01
-rw-rw-r--. 1 oracle oinstall   9 Nov  8 20:45 ACFSDG02
drwxrwx---. 2 oracle asmadmin 120 Nov  8 20:45 .
[root@rac1-19c bin]#
[root@rac1-19c bin]# exit
logout

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Nov 8 20:47:05 2020
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBILITY        DATABASE_COMPATIBILI
------------ ---------- -------------------- --------------------
           1 DATADG     19.0.0.0.0           10.1.0.0.0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ cd $ORACLE_HOME/bin
[oracle@rac1-19c bin]$ pwd
/u01/app/19.3.0/grid/bin
[oracle@rac1-19c bin]$ ./asmca -silent -createDiskGroup 
-diskGroupName ACFSDG -disk 'AFD:ACFS*' 
-redundancy EXTERNAL -au_size 4 
-compatible.asm 19.0.0.0 
-compatible.rdbms 10.1.0.0

[oracle@rac1-19c ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     20480    20292                0           20292              0             N  ACFSDG/
MOUNTED  EXTERN  N         512             512   4096  4194304     61432    26696                0           26696              0             Y  DATADG/
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Nov 8 20:50:52 2020
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBILITY        DATABASE_COMPATIBILI
------------ ---------- -------------------- --------------------
           1 DATADG     19.0.0.0.0           10.1.0.0.0
           2 ACFSDG     19.0.0.0.0           10.1.0.0.0

SQL> select dg.name as diskgroup, a.name as name,a.value 
from v$asm_diskgroup dg, v$asm_attribute a 
where dg.name='ACFSDG' and 
dg.group_number=a.group_number and 
a.name like '%compatible%';

DISKGRO NAME                                     VALUE
------- ---------------------------------------- ---------------
ACFSDG  compatible.asm                           19.0.0.0.0
ACFSDG  compatible.rdbms                         10.1.0.0.0
ACFSDG  compatible.advm                          19.0.0.0.0

SQL> alter diskgroup ACFSDG set attribute 'compatible.rdbms'='19.0.0.0.0';
Diskgroup altered.

SQL> select dg.name as diskgroup, a.name as name,a.value 
from v$asm_diskgroup dg, v$asm_attribute a 
where dg.name='ACFSDG' and 
dg.group_number=a.group_number and 
a.name like '%compatible%';

DISKGRO NAME                                     VALUE
------- ---------------------------------------- ---------------
ACFSDG  compatible.asm                           19.0.0.0.0
ACFSDG  compatible.rdbms                         19.0.0.0.0
ACFSDG  compatible.advm                          19.0.0.0.0
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

[oracle@rac1-19c ~]$ asmcmd
ASMCMD>

Creating ACFS volume using ACFSDG

ASMCMD> volcreate -G ACFSDG -s 19G acfs_vol
ASMCMD> volinfo --all

Diskgroup Name: ACFSDG

         Volume Name: ACFS_VOL
         Volume Device: /dev/asm/acfs_vol-480
         State: ENABLED
         Size (MB): 19456
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:
ASMCMD> exit
[oracle@rac1-19c ~]$

[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# modprobe  oracleacfs
[root@rac1-19c ~]# modprobe  oracleadvm
[root@rac1-19c ~]# modprobe  oracleoks

[root@rac1-19c ~]# lsmod | grep oracle
oracleafd             217088  1
oracleacfs           5607424  1
oracleadvm           1241088  7
oracleoks             724992  2 oracleacfs,oracleadvm
[root@rac1-19c ~]#

[root@rac1-19c ~]# mkfs -t acfs /dev/asm/acfs_vol-480
mkfs.acfs: version                   = 19.0.0.0.0
mkfs.acfs: on-disk version           = 46.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol-480
mkfs.acfs: volume size               = 20401094656  (  19.00 GB )
mkfs.acfs: Format complete.
[root@rac1-19c ~]#

[root@rac1-19c /]# chown -R oracle:oinstall /acfs_vol
[root@rac1-19c /]# chmod -R 777 /acfs_vol
[root@rac1-19c /]#

[root@rac1-19c /]# acfsutil registry -a /dev/asm/acfs_vol-480 /acfs_vol
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry

[root@rac1-19c /]# acfsutil registry -c /dev/asm/acfs_vol-480 /acfs_vol -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-480'
[root@rac1-19c /]#

[root@rac1-19c /]# mount -t acfs /dev/asm/acfs_vol-480 /acfs_vol
[root@rac1-19c /]#

[root@rac1-19c /]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               7.1G   16K  7.1G   1% /dev
tmpfs                  7.1G  1.1G  6.0G  16% /dev/shm
tmpfs                  7.1G  9.6M  7.1G   1% /run
tmpfs                  7.1G     0  7.1G   0% /sys/fs/cgroup
/dev/mapper/ol-root     84G   32G   53G  38% /
/dev/mapper/ol-home     50G  9.4G   41G  19% /home
/dev/sda1              497M  171M  327M  35% /boot
tmpfs                  1.5G   12K  1.5G   1% /run/user/42
tmpfs                  1.5G     0  1.5G   0% /run/user/54321
/dev/asm/acfs_vol-480   19G  828M   19G   5% /acfs_vol
[root@rac1-19c /]#

Add the volume in fstab

[root@rac1-19c /]# mount | grep acfs
/dev/asm/acfs_vol-480 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@rac1-19c /]#

[root@rac1-19c /]# cat /etc/fstab

# /etc/fstab
# Created by anaconda on Tue Aug 25 17:29:44 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=c9d497e2-5283-4e44-a4ca-a2fdcaa0cf86 /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
[root@rac1-19c /]#
[root@rac1-19c /]# vi /etc/fstab
[root@rac1-19c /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Aug 25 17:29:44 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                               xfs     defaults        0 0
UUID=c9d497e2-5283-4e44-a4ca-a2fdcaa0cf86 /boot         xfs     defaults        0 0
/dev/mapper/ol-home     /home                           xfs     defaults        0 0
/dev/mapper/ol-swap     swap                            swap    defaults        0 0
/dev/asm/acfs_vol-480   /acfs_vol                       acfs    defaults        0 0
[root@rac1-19c /]#

[root@rac1-19c /]# ssh rac2-19c
[root@rac2-19c ~]# vi /etc/fstab
[root@rac2-19c ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Aug 25 17:29:44 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=c9d497e2-5283-4e44-a4ca-a2fdcaa0cf86 /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/dev/asm/acfs_vol-480   /acfs_vol               acfs    defaults        0 0
[root@rac2-19c ~]#
[root@rac2-19c ~]# exit
logout
Connection to rac2-19c closed.
[root@rac1-19c /]#

[root@rac1-19c /]# ssh rac3-19c
[root@rac3-19c ~]# vi /etc/fstab
[root@rac3-19c ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Aug 25 17:29:44 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=c9d497e2-5283-4e44-a4ca-a2fdcaa0cf86 /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/dev/asm/acfs_vol-480   /acfs_vol               acfs    defaults        0 0
[root@rac3-19c ~]#

[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.DATADG.GHCHKPT.advm
               OFFLINE OFFLINE      rac1-19c                 STABLE
               OFFLINE OFFLINE      rac2-19c                 STABLE
               OFFLINE OFFLINE      rac3-19c                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.acfsdg.acfs_vol.acfs
               ONLINE  ONLINE       rac1-19c                 mounted on /acfs_vol
                                                             ,STABLE
               ONLINE  ONLINE       rac2-19c                 mounted on /acfs_vol
                                                             ,STABLE
               ONLINE  ONLINE       rac3-19c                 mounted on /acfs_vol
                                                             ,STABLE
ora.chad
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.datadg.ghchkpt.acfs
               OFFLINE OFFLINE      rac1-19c                 STABLE
               OFFLINE OFFLINE      rac2-19c                 STABLE
               OFFLINE OFFLINE      rac3-19c                 STABLE
ora.helper
               OFFLINE OFFLINE      rac1-19c                 IDLE,STABLE
               OFFLINE OFFLINE      rac2-19c                 IDLE,STABLE
               OFFLINE OFFLINE      rac3-19c                 IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.ons
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.proxy_advm
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.DATADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac2-19c                 169.254.19.103 10.1.4.123,STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 Started,STABLE
      2        ONLINE  ONLINE       rac2-19c                 Started,STABLE
      3        ONLINE  ONLINE       rac3-19c                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.cvu
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac2-19c                 Open,STABLE
ora.orcl.db
      1        ONLINE  ONLINE       rac3-19c                 Open,HOME=/u01/app/oracle/product/19.3.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       rac2-19c                 Open,HOME=/u01/app/oracle/product/19.3.0
                                                             /db_1,STABLE
      3        ONLINE  ONLINE       rac1-19c                 Open,HOME=/u01/app/oracle/product/19.3.0
                                                             /db_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.rac1-19c.vip
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.rac2-19c.vip
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.rac3-19c.vip
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac2-19c                 STABLE
--------------------------------------------------------------------------------

[root@rac1-19c ~]# crsctl stat res -t | grep ACFS
ora.ACFSDG.ACFS_VOL.advm
ora.ACFSDG.dg(ora.asmgroup)
[root@rac1-19c ~]#

[root@rac1-19c ~]# srvctl status diskgroup -diskgroup ACFSDG -verbose
Disk Group ACFSDG is running on rac1-19c,rac2-19c,rac3-19c
[root@rac1-19c ~]#

Applying 19c (19.9.0) patch on 19c (19.3.0) for GRID and RDBMS - 3 Node RAC

[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/31750108/

OPatchauto session is initiated at Mon Nov  9 10:47:39 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-11-09_10-47-44AM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-11-09_10-48-38AM.log
The id for this session is YMIR

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-19c
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-54-34AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-54-34AM_1.log

Host:rac1-19c
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-58-46AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-58-46AM_1.log

Patch: /home/oracle/31750108/31773437
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-58-46AM_1.log

Patch: /home/oracle/31750108/31780966
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_10-58-46AM_1.log

OPatchauto session completed at Mon Nov  9 11:16:54 2020
Time taken to complete the session 29 minutes, 15 seconds
[root@rac1-19c ~]#

rac2-19c

[root@rac2-19c ~]# . oraenv
ORACLE_SID = [+ASM2] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been changed from /home/oracle to /u01/app/oracle
[root@rac2-19c ~]#

[root@rac2-19c ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/31750108/

OPatchauto session is initiated at Mon Nov  9 11:17:20 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-11-09_11-17-30AM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-11-09_11-18-34AM.log
The id for this session is E371

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac2-19c
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-35-24AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-35-24AM_1.log


Host:rac2-19c
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-40-02AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-40-02AM_1.log

Patch: /home/oracle/31750108/31773437
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-40-02AM_1.log

Patch: /home/oracle/31750108/31780966
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_11-40-02AM_1.log

OPatchauto session completed at Mon Nov  9 11:58:15 2020
Time taken to complete the session 40 minutes, 56 seconds
[root@rac2-19c ~]#

rac3-19c

[oracle@rac3-19c ~]$ su - root
Password:
Last login: Sun Nov  8 21:37:30 CST 2020 on pts/0
[root@rac3-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM3
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac3-19c ~]#

[root@rac3-19c ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/31750108/

OPatchauto session is initiated at Mon Nov  9 11:59:55 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-11-09_12-00-00PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-11-09_12-01-07PM.log
The id for this session is 9RF7

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac3-19c
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-08-53PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-08-53PM_1.log

Host:rac3-19c
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-13-35PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-13-35PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-13-35PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-11-09_12-13-35PM_1.log


OPatchauto session completed at Mon Nov  9 13:53:06 2020
Time taken to complete the session 113 minutes, 12 seconds
[root@rac3-19c ~]#

Check the GI and RDBMS patches after applying patches:

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$ cd $ORACLE_HOME/OPatch

[oracle@rac1-19c OPatch]$ ./opatch lspatches
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31773437;ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
OPatch succeeded.
[oracle@rac1-19c OPatch]$

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcl1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$
[oracle@rac1-19c ~]$ cd $ORACLE_HOME/OPatch

[oracle@rac1-19c OPatch]$ ./opatch lspatches
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)
OPatch succeeded.
[oracle@rac1-19c OPatch]$

Check the PATH and Libraries for ACFSDG

SQL> col name for a20
col path for a18
col library for a60
set lines 160
SQL> select name, path, library from v$asm_disk where group_number <> 0;

NAME                     PATH                           LIBRARY
--------------------       ------------------            ------------------------------------------------------------
ACFSDG02             AFD:ACFSDG02        AFD Library - Generic , version 3 (KABI_V3)
ACFSDG01             AFD:ACFSDG01        AFD Library - Generic , version 3 (KABI_V3)
DATADG01             AFD:DATADG01       AFD Library - Generic , version 3 (KABI_V3)
DATADG02             AFD:DATADG02       AFD Library - Generic , version 3 (KABI_V3)

SQL>

[oracle@rac1-19c ~]$ sqlplus sys@orcl as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 9 14:51:47 2020
Version 19.9.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
-------------------------    ------------------------------
orcl1                             1
orcl2                             2
orcl3                             3

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$
[oracle@rac1-19c ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.DATADG.GHCHKPT.advm
               OFFLINE OFFLINE      rac1-19c                 STABLE
               OFFLINE OFFLINE      rac2-19c                 STABLE
               OFFLINE OFFLINE      rac3-19c                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.acfsdg.acfs_vol.acfs
               ONLINE  ONLINE       rac1-19c                 mounted on /acfs_vol,STABLE
               ONLINE  ONLINE       rac2-19c                 mounted on /acfs_vol,STABLE
               ONLINE  ONLINE       rac3-19c                 mounted on /acfs_vol,STABLE
ora.chad
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.datadg.ghchkpt.acfs
               OFFLINE OFFLINE      rac1-19c                 STABLE
               OFFLINE OFFLINE      rac2-19c                 STABLE
               OFFLINE OFFLINE      rac3-19c                 STABLE
ora.helper
               OFFLINE OFFLINE      rac1-19c                 STABLE
               OFFLINE OFFLINE      rac2-19c                 STABLE
               OFFLINE OFFLINE      rac3-19c                 IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.ons
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
ora.proxy_advm
               ONLINE  ONLINE       rac1-19c                 STABLE
               ONLINE  ONLINE       rac2-19c                 STABLE
               ONLINE  ONLINE       rac3-19c                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.DATADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac3-19c                 169.254.10.67 10.1.4.124,STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 Started,STABLE
      2        ONLINE  ONLINE       rac2-19c                 Started,STABLE
      3        ONLINE  ONLINE       rac3-19c                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       rac1-19c                 STABLE
      2        ONLINE  ONLINE       rac2-19c                 STABLE
      3        ONLINE  ONLINE       rac3-19c                 STABLE
ora.cvu
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac3-19c                 Open,STABLE
ora.orcl.db
      1        ONLINE  ONLINE       rac3-19c                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /db_1,STABLE
      2        ONLINE  ONLINE       rac1-19c                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /db_1,STABLE
      3        ONLINE  ONLINE       rac2-19c                 Open,HOME=/u01/app/o
                                                             racle/product/19.3.0
                                                             /db_1,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.rac1-19c.vip
      1        ONLINE  ONLINE       rac1-19c                 STABLE
ora.rac2-19c.vip
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.rac3-19c.vip
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac3-19c                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac2-19c                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac1-19c                 STABLE
--------------------------------------------------------------------------------
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ cd $ORACLE_HOME/bin
[oracle@rac1-19c bin]$ pwd
/u01/app/19.3.0/grid/bin

[oracle@rac1-19c bin]$ ./afddriverstate supported
AFD-9200: Supported
[oracle@rac1-19c bin]$

[oracle@rac1-19c bin]$ ./asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1-19c.localdomain'
[oracle@rac1-19c bin]$

[oracle@rac1-19c bin]$ ./afdroot version_check
AFD-616: Valid AFD distribution media detected at: '/u01/app/19.3.0/grid/usm/install/Oracle/EL7UEK/x86_64/4.1.12-112.16.4/4.1.12-112.16.4-x86_64/bin'
[oracle@rac1-19c bin]$

[oracle@rac1-19c bin]$ ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
ACFS-9203: true

[oracle@rac1-19c bin]$ ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded
ACFS-9203: true

[oracle@rac1-19c bin]$ ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported
ACFS-9200: Supported

[oracle@rac1-19c bin]$ ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9325:     Driver OS kernel version = 4.1.12-112.16.4.el7uek.x86_64.
ACFS-9326:     Driver build number = 200813.1.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = 200813.1.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[oracle@rac1-19c bin]$

Trying to destroy the disk using "dd" command

Now let's test the feature where it won't let any non-oracle I/O to go through and corrupt the disks managed by ASMFD:

[oracle@rac1-19c ~]$ su - root
Password:
Last login: Mon Nov  9 16:16:59 CST 2020

[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# cd $ORACLE_HOME/bin

[root@rac1-19c bin]# ./asmcmd afd_lsdsk
-----------------------------------------------
Label                               Filtering          Path
===============================================
ACFSDG01                    ENABLED   /dev/sdd
ACFSDG02                    ENABLED   /dev/sde
DATADG01                    ENABLED   /dev/sdb1
DATADG02                    ENABLED   /dev/sdc1
[root@rac1-19c bin]#

[root@rac1-19c bin]# exit
logout

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 9 16:20:57 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select name, path, total_mb, free_mb from v$asm_disk;

NAME                               PATH                                        TOTAL_MB    FREE_MB
------------------------------ ------------------------------             --------- ----------
ACFSDG02                       AFD:ACFSDG02                        10240        412
ACFSDG01                       AFD:ACFSDG01                        10240        416
DATADG01                       AFD:DATADG01                        30716      11584
DATADG02                       AFD:DATADG02                        30716      11624

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ dd if=/dev/zero of=/dev/sdd bs=1M count=1000
dd: failed to open ‘/dev/sdd’: Permission denied
[oracle@rac1-19c ~]$

Login as 'root' user and execute "dd" command.

[oracle@rac1-19c ~]$ su - root
Password:
Last login: Mon Nov  9 16:19:23 CST 2020
[root@rac1-19c ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-19c ~]#

[root@rac1-19c ~]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.88603 s, 270 MB/s
[root@rac1-19c ~]#

[root@rac1-19c ~]# exit
logout
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-19c ~]$

[oracle@rac1-19c ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 9 16:22:57 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select name, path, total_mb, free_mb from v$asm_disk;

NAME                               PATH                                          TOTAL_MB    FREE_MB
------------------------------   ------------------------------            ----------           ----------
ACFSDG02                       AFD:ACFSDG02                        10240             412
ACFSDG01                       AFD:ACFSDG01                        10240             416
DATADG01                       AFD:DATADG01                        30716            11588
DATADG02                       AFD:DATADG02                        30716            11628
SQL>

One of the ASMFD feature: As you can see, the "dd" command did succeed but it didn't actually wipe the disks. Oracle rejected all the I/O which was coming from "dd" command either used by the 'oracle' or 'root' doesn't matter because it was not coming through Oracle software stack.

Working Screenshots: