Oracle 19c (19.8) with ASMFD, Kernel 5.4.17 & Oracle Linux 8.2 (3-Node RAC)

Oracle has released the patches for Oracle 19c (19.8) and 19c (19.9) for ACFS/AFD drivers.

1. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.9.0.0.0ACFSRU

2. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.8.0.0.0ACFSRU

Please refer for more details:

ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Click Additional notes: 

OL8 UEK6: 30590023 OL8 UEK6 SUPPORT FOR ACFS/AFD (NOTE: 19.x and later Grid only)

Environment:

1. Installed OEL 8.2

2. Upgraded Kernel "kernel-uek-4.18.0-193.13.2.el8_2.x86_64" to kernel-uek-5.4.17-2036.100.6.1.el8uek.x86_64

3. Installed GI 19.3 (while GI installation applied Oracle 19c GI (19.8) RU)

4. Applied "p30590023_198000ACFSRU_Linux-x86-64.zip" patch on Oracle 19c (19.8) for GRID

19c-rac1:

[oracle@19c-rac1 ~]$ ps -ef | grep pmon

oracle     12651       1  0 18:30 ?        00:00:00 asm_pmon_+ASM1

oracle     14408       1  0 18:31 ?        00:00:00 mdb_pmon_-MGMTDB

oracle     35737   35364  0 18:39 pts/1    00:00:00 grep --color=auto pmon

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ uname -r

5.4.17-2036.100.6.1.el8uek.x86_64

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ uname -a

Linux 19c-rac1.localdomain 5.4.17-2036.100.6.1.el8uek.x86_64 #2 SMP Thu Oct 29 17:06:00 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cat /etc/oracle-release

Oracle Linux Server release 8.2

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ echo $ORACLE_HOME

/u01/app/19.3.0/grid

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch

[oracle@19c-rac1 OPatch]$ ./opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac1 OPatch]$

[oracle@19c-rac1 ~]$ chmod 777 p30590023_198000ACFSRU_Linux-x86-64.zip

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ scp p30590023_198000ACFSRU_Linux-x86-64.zip oracle@19c-rac2:/home/oracle/

p30590023_198000ACFSRU_Linux-x86-64.zip        100%  686MB  36.5MB/s   00:18

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ scp p30590023_198000ACFSRU_Linux-x86-64.zip oracle@19c-rac3:/home/oracle/

p30590023_198000ACFSRU_Linux-x86-64.zip        100%  686MB  37.3MB/s   00:18

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ unzip p30590023_198000ACFSRU_Linux-x86-64.zip

Archive:  p30590023_198000ACFSRU_Linux-x86-64.zip

   creating: 30590023/

   creating: 30590023/automation/

  inflating: 30590023/automation/messages.properties

  inflating: 30590023/automation/bp1-out-of-place-non-rolling-switchback.xml

  inflating: 30590023/automation/bp1-out-of-place-switchback.xml

[oracle@19c-rac1 ~]$ echo $ORACLE_HOME

/u01/app/19.3.0/grid

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch

[oracle@19c-rac1 OPatch]$ ./opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac1 OPatch]$

[oracle@19c-rac1 OPatch]$ ./opatch version

OPatch Version: 12.2.0.1.21

OPatch succeeded.

[oracle@19c-rac1 OPatch]$

[oracle@19c-rac1 OPatch]$ cd

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ su - root

Password:

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:24:30 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-24-35PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-25-12PM.log

The id for this session is P258

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid

CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid

Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid

CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac1

CRS Home:/u01/app/19.3.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023

Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-27-43PM_1.log

OPatchauto session completed at Fri Dec 18 19:31:02 2020

Time taken to complete the session 6 minutes, 32 seconds

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# exit

logout

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch

[oracle@19c-rac1 OPatch]$ ./opatch lspatches

30590023;ACFS Interim patch for 30590023

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac1 OPatch]$ cd

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ ps -ef | grep pmon

oracle     87342       1  0 19:30 ?        00:00:00 asm_pmon_+ASM1

oracle     97416   35364  0 19:48 pts/1    00:00:00 grep --color=auto pmon

[oracle@19c-rac1 ~]$

19c-rac2:

[oracle@19c-rac2 ~]$ . oraenv

ORACLE_SID = [cdbrac2] ? +ASM2

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ cd $ORACLE_HOME/OPatch

[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ ./opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ ./opatch version

OPatch Version: 12.2.0.1.21

OPatch succeeded.

[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ cd

[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ ps -ef | grep pmon

oracle     15057       1  0 18:32 ?        00:00:00 asm_pmon_+ASM2

oracle     48512   29638  0 19:24 pts/0    00:00:00 grep --color=auto pmon

[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ su - root

Password:

[root@19c-rac2 ~]# . oraenv

ORACLE_SID = [root] ? +ASM2

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac2 ~]#

[root@19c-rac2 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:47:58 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-48-02PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-48-14PM.log

The id for this session is ZJLN

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid

CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid

Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid

CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac2

CRS Home:/u01/app/19.3.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023

Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-49-48PM_1.log

OPatchauto session completed at Fri Dec 18 19:53:11 2020

Time taken to complete the session 5 minutes, 13 seconds

[root@19c-rac2 ~]#

19c-rac3:

[oracle@19c-rac3 ~]$ . oraenv

ORACLE_SID = [cdbrac3] ? +ASM3

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ cd $ORACLE_HOME/OPatch

[oracle@19c-rac3 OPatch]$

[oracle@19c-rac3 OPatch]$ ./opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac3 OPatch]$

[oracle@19c-rac3 ~]$ ps -ef | grep pmon

oracle     14810       1  0 18:32 ?        00:00:00 asm_pmon_+ASM3

oracle     48694   29385  0 19:24 pts/0    00:00:00 grep --color=auto pmon

[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ su - root

Password:

[root@19c-rac3 ~]# . oraenv

ORACLE_SID = [root] ? +ASM3

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac3 ~]#

[root@19c-rac3 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:54:35 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-54-39PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-54-50PM.log

The id for this session is KS92

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid

Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid

CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid

Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid

CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac3

CRS Home:/u01/app/19.3.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023

Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-57-06PM_1.log

OPatchauto session completed at Fri Dec 18 20:03:37 2020

Time taken to complete the session 9 minutes, 2 seconds

[root@19c-rac3 ~]#

Checking drivers in 19c-rac1:

[oracle@19c-rac1 ~]$ su - root

Password:

[root@19c-rac1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin

[root@19c-rac1 bin]# ./afddriverstate supported

AFD-9200: Supported

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed

ACFS-9203: true

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded

ACFS-9203: true

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported

ACFS-9200: Supported

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version

ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.

ACFS-9326:     Driver build number = RELEASE.

ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9547:     Driver available build number = RELEASE.

ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).

[root@19c-rac1 bin]#

[root@19c-rac1 bin]#  asmcmd afd_configure

ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac1'

CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac1'

CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac1'

CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'

CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.19c-rac1.vip' on '19c-rac1'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac1'

CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1'

CRS-2677: Stop of 'ora.19c-rac1.vip' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.cvu' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac1'

CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac1' succeeded

CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac1' succeeded.

CRS-2672: Attempting to start 'ora.19c-rac1.vip' on '19c-rac3'

CRS-2672: Attempting to start 'ora.scan3.vip' on '19c-rac2'

CRS-2672: Attempting to start 'ora.cvu' on '19c-rac2'

CRS-2676: Start of 'ora.cvu' on '19c-rac2' succeeded

CRS-2676: Start of 'ora.19c-rac1.vip' on '19c-rac3' succeeded

CRS-2676: Start of 'ora.scan3.vip' on '19c-rac2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2'

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.ons' on '19c-rac1'

CRS-2677: Stop of 'ora.ons' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac1'

CRS-2677: Stop of 'ora.net1.network' on '19c-rac1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac1' has completed

CRS-2677: Stop of 'ora.crsd' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.crf' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac1'

CRS-2677: Stop of 'ora.crf' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac1'

CRS-2677: Stop of 'ora.ctssd' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.evmd' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac1'

CRS-2677: Stop of 'ora.cssd' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac1'

CRS-2677: Stop of 'ora.gpnpd' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.gipcd' on '19c-rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_configure

AFD-627: AFD distribution files found.

AFD-634: Removing previous AFD installation.

AFD-635: Previous AFD components successfully removed.

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

AFD-636: Installing requested AFD software.

AFD-637: Loading installed AFD drivers.

AFD-9321: Creating udev for AFD.

AFD-9323: Creating module dependencies - this may take some time.

AFD-9154: Loading 'oracleafd.ko' driver.

AFD-649: Verifying AFD devices.

AFD-9156: Detecting control device '/dev/oracleafd/admin'.

AFD-638: AFD installation correctness verified.

Modifying resource dependencies - this may take some time.

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac1.localdomain'

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@19c-rac1 bin]#

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ls -lrt afd*

-rwxr-xr-x 1 root oinstall     1000 Jun 20  2016 afdroot

-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin

-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot

-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:07 afdtool

-rwxr-xr-x 1 root oinstall      760 Dec 18 20:07 afdload

-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:07 afddriverstate

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# lsmod|grep oracle

oracleafd             221184  1

oracleacfs           4997120  0

oracleadvm            745472  0

oracleoks             729088  2 oracleadvm,oracleacfs

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# cat /etc/os-release | head -n2

NAME="Oracle Linux Server"

VERSION="8.2"

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version

ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.

ACFS-9326:     Driver build number = RELEASE.

ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9547:     Driver available build number = RELEASE.

ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).

[root@19c-rac1 bin]#

Checking drivers in 19c-rac2:

[root@19c-rac2 ~]# cd $ORACLE_HOME/bin

[root@19c-rac2 bin]# ./afddriverstate supported

AFD-9200: Supported

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed

ACFS-9203: true

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded

ACFS-9203: true

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported

ACFS-9200: Supported

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version

ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.

ACFS-9326:     Driver build number = RELEASE.

ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9547:     Driver available build number = RELEASE.

ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_configure

ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up

[root@19c-rac2 bin]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac2'

CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac2'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.qosmserver' on '19c-rac2'

CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac2'

CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'

CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.19c-rac2.vip' on '19c-rac2'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac2'

CRS-2677: Stop of 'ora.cvu' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.19c-rac2.vip' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2'

CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.qosmserver' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac2'

CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac2' succeeded

CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac2' succeeded.

CRS-2672: Attempting to start 'ora.qosmserver' on '19c-rac1'

CRS-2672: Attempting to start 'ora.19c-rac2.vip' on '19c-rac3'

CRS-2672: Attempting to start 'ora.scan3.vip' on '19c-rac1'

CRS-2672: Attempting to start 'ora.cvu' on '19c-rac1'

CRS-2676: Start of 'ora.cvu' on '19c-rac1' succeeded

CRS-2676: Start of 'ora.19c-rac2.vip' on '19c-rac3' succeeded

CRS-2676: Start of 'ora.scan3.vip' on '19c-rac1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded

CRS-2676: Start of 'ora.qosmserver' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on '19c-rac2'

CRS-2677: Stop of 'ora.ons' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac2'

CRS-2677: Stop of 'ora.net1.network' on '19c-rac2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac2' has completed

CRS-2677: Stop of 'ora.crsd' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.storage' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.crf' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac2'

CRS-2677: Stop of 'ora.crf' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.storage' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'

CRS-2677: Stop of 'ora.mdnsd' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac2'

CRS-2677: Stop of 'ora.ctssd' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.evmd' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac2'

CRS-2677: Stop of 'ora.cssd' on '19c-rac2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac2'

CRS-2677: Stop of 'ora.gpnpd' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.gipcd' on '19c-rac2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_configure

AFD-627: AFD distribution files found.

AFD-634: Removing previous AFD installation.

AFD-635: Previous AFD components successfully removed.

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

AFD-636: Installing requested AFD software.

AFD-637: Loading installed AFD drivers.

AFD-9321: Creating udev for AFD.

AFD-9323: Creating module dependencies - this may take some time.

AFD-9154: Loading 'oracleafd.ko' driver.

AFD-649: Verifying AFD devices.

AFD-9156: Detecting control device '/dev/oracleafd/admin'.

AFD-638: AFD installation correctness verified.

Modifying resource dependencies - this may take some time.

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac2.localdomain'

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# ls -lrt afd*

-rwxr-xr-x 1 root oinstall     1000 Nov 12 21:27 afdroot

-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin

-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot

-rwxr-xr-x 1 root oinstall      760 Dec 18 20:16 afdload

-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:16 afdtool

-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:16 afddriverstate

[root@19c-rac2 bin]#

[root@19c-rac2 bin]# lsmod|grep oracle

oracleafd             221184  1

oracleacfs           4997120  0

oracleadvm            745472  0

oracleoks             729088  2 oracleadvm,oracleacfs

[root@19c-rac2 bin]# cat /etc/os-release | head -n2

NAME="Oracle Linux Server"

VERSION="8.2"

[root@19c-rac2 bin]#

Checking drivers in 19c-rac3:

[root@19c-rac3 ~]# cd $ORACLE_HOME/bin

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# ./afddriverstate supported

AFD-9200: Supported

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed

ACFS-9203: true

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded

ACFS-9203: true

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported

ACFS-9200: Supported

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version

ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.

ACFS-9326:     Driver build number = RELEASE.

ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9547:     Driver available build number = RELEASE.

ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_configure

ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up

[root@19c-rac3 bin]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac3'

CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac3'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac3'

CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac3'

CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3'

CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on '19c-rac3'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'

CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'

CRS-2673: Attempting to stop 'ora.chad' on '19c-rac3'

CRS-2677: Stop of 'ora.scan2.vip' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3'

CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded

CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded

CRS-2677: Stop of 'ora.chad' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.mgmtdb' on '19c-rac3'

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac3'

CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac3' succeeded

CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac3' succeeded.

CRS-2677: Stop of 'ora.mgmtdb' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.MGMTLSNR' on '19c-rac3'

CRS-2677: Stop of 'ora.MGMTLSNR' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.19c-rac3.vip' on '19c-rac3'

CRS-2672: Attempting to start 'ora.MGMTLSNR' on '19c-rac1'

CRS-2677: Stop of 'ora.19c-rac3.vip' on '19c-rac3' succeeded

CRS-2676: Start of 'ora.MGMTLSNR' on '19c-rac1' succeeded

CRS-2672: Attempting to start 'ora.mgmtdb' on '19c-rac1'

CRS-2676: Start of 'ora.mgmtdb' on '19c-rac1' succeeded

CRS-2672: Attempting to start 'ora.chad' on '19c-rac1'

CRS-2672: Attempting to start 'ora.chad' on '19c-rac2'

CRS-2676: Start of 'ora.chad' on '19c-rac2' succeeded

CRS-2676: Start of 'ora.chad' on '19c-rac1' succeeded

CRS-2672: Attempting to start 'ora.19c-rac3.vip' on '19c-rac2'

CRS-2672: Attempting to start 'ora.scan2.vip' on '19c-rac1'

CRS-2676: Start of 'ora.19c-rac3.vip' on '19c-rac2' succeeded

CRS-2676: Start of 'ora.scan2.vip' on '19c-rac1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on '19c-rac1'

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on '19c-rac3'

CRS-2677: Stop of 'ora.ons' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac3'

CRS-2677: Stop of 'ora.net1.network' on '19c-rac3' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac3' has completed

CRS-2677: Stop of 'ora.crsd' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.crf' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac3'

CRS-2677: Stop of 'ora.crf' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.mdnsd' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac3'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac3'

CRS-2677: Stop of 'ora.ctssd' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.evmd' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac3'

CRS-2677: Stop of 'ora.cssd' on '19c-rac3' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac3'

CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac3'

CRS-2677: Stop of 'ora.gpnpd' on '19c-rac3' succeeded

CRS-2677: Stop of 'ora.gipcd' on '19c-rac3' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac3' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_configure

AFD-627: AFD distribution files found.

AFD-634: Removing previous AFD installation.

AFD-635: Previous AFD components successfully removed.

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

AFD-636: Installing requested AFD software.

AFD-637: Loading installed AFD drivers.

AFD-9321: Creating udev for AFD.

AFD-9323: Creating module dependencies - this may take some time.

AFD-9154: Loading 'oracleafd.ko' driver.

AFD-649: Verifying AFD devices.

AFD-9156: Detecting control device '/dev/oracleafd/admin'.

AFD-638: AFD installation correctness verified.

Modifying resource dependencies - this may take some time.

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac3.localdomain'

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# ls -lrt afd*

-rwxr-xr-x 1 root oinstall     1000 Nov 12 21:27 afdroot

-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin

-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot

-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:25 afdtool

-rwxr-xr-x 1 root oinstall      760 Dec 18 20:25 afdload

-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:25 afddriverstate

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# lsmod|grep oracle

oracleafd             221184  1

oracleacfs           4997120  0

oracleadvm            745472  0

oracleoks             729088  2 oracleadvm,oracleacfs

[root@19c-rac3 bin]#

[root@19c-rac3 bin]# cat /etc/os-release | head -n2

NAME="Oracle Linux Server"

VERSION="8.2"

[root@19c-rac3 bin]#

Creating diskgroup ACFSDG using asmca:

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [cdbrac1] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmcmd

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     49136    25104                0           25104              0             Y  OCRVD/

ASMCMD>

ASMCMD> exit

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmca &

[1] 166449

[oracle@19c-rac1 ~]$

Screenshots: 19cr8-ASMFD-1.JPG to 19cr8-ASMFD-4.JPG

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/bin

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ pwd

/u01/app/19.3.0/grid/bin

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsget

parameter:/dev/sd*

profile:/dev/sd*

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsset /dev/sd*,'AFD:*'

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsget

parameter:/dev/sd*, AFD:*

profile:/dev/sd*,AFD:*

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 ~]$ su - root

Password:

[root@19c-rac1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin

[root@19c-rac1 bin]#

[root@19c-rac1 bin]#  ls -alrt /dev/oracleafd/disks/*

ls: cannot access '/dev/oracleafd/disks/*': No such file or directory

[root@19c-rac1 bin]# ./acfsload start

ACFS-9391: Checking for existing ADVM/ACFS installation.

ACFS-9392: Validating ADVM/ACFS installation files for operating system.

ACFS-9393: Verifying ASM Administrator setup.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.

ACFS-9326:     Driver build number = RELEASE.

ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9547:     Driver available build number = RELEASE.

ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).

ACFS-9549:     Kernel and command versions.

Kernel:

    Build version: 19.0.0.0.0

    Build full version: 19.8.0.0.0

    Build hash:    9256567290

    Bug numbers:   NoTransactionInformation

Commands:

    Build version: 19.0.0.0.0

    Build full version: 19.8.0.0.0

    Build hash:    9256567290

    Bug numbers:   NoTransactionInformation

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.

ACFS-9156: Detecting control device '/dev/ofsctl'.

ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf

ACFS-9322: completed

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ls -alrt /dev/oracleafd/disks/*

ls: cannot access '/dev/oracleafd/disks/*': No such file or directory

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./asmcmd afd_label ACFSDG01 /dev/sdp1

[root@19c-rac1 bin]# ./asmcmd afd_label ACFSDG02 /dev/sdq1

[root@19c-rac1 bin]# ls -alrt /dev/oracleafd/disks/

total 8

drwxrwx--- 3 oracle dba      80 Dec 18 20:07 ..

-rw-rw-r-- 1 oracle oinstall 10 Dec 18 20:51 ACFSDG01

-rw-rw-r-- 1 oracle oinstall 10 Dec 18 20:51 ACFSDG02

drwxrwxr-x 2 oracle dba      80 Dec 18 20:51 .

[root@19c-rac1 bin]#

Note: Again executed ASMCA and created ACFSDG

[root@19c-rac1 bin]# exit

logout

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 20:52:11 2020

Version 19.8.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

SQL>  SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME

------------ ------------------------------

COMPATIBILITY

------------------------------------------------------------

DATABASE_COMPATIBILITY

------------------------------------------------------------

           1 OCRVD

19.0.0.0.0

10.1.0.0.0

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 20:59:54 2020

Version 19.8.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

SQL> set lines 100 pages 100

SQL> col GROUP_NUMBER format 99

SQL> col NAME format a10

SQL> col COMPATIBILITY format a10

SQL> col DATABASE_COMPATIBILITY format a20

SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBIL DATABASE_COMPATIBILI

------------ ---------- ---------- --------------------

           1 OCRVD      19.0.0.0.0 10.1.0.0.0

           2 ACFSDG     19.0.0.0.0 10.1.0.0.0

SQL>  alter diskgroup ACFSDG set attribute 'compatible.rdbms'='19.0.0.0.0';

Diskgroup altered.

SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBIL DATABASE_COMPATIBILI

------------ ---------- ---------- --------------------

           1 OCRVD      19.0.0.0.0 10.1.0.0.0

           2 ACFSDG     19.0.0.0.0 19.0.0.0.0

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmcmd

ASMCMD> volcreate -G ACFSDG -s 19G acfs_vol

ASMCMD> volinfo --all

Diskgroup Name: ACFSDG

         Volume Name: ACFS_VOL

         Volume Device: /dev/asm/acfs_vol-296

         State: ENABLED

         Size (MB): 19456

         Resize Unit (MB): 64

         Redundancy: UNPROT

         Stripe Columns: 8

         Stripe Width (K): 1024

         Usage:

         Mountpath:

ASMCMD> exit

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ su - root

Password:

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# modprobe  oracleacfs

[root@19c-rac1 ~]# modprobe  oracleadvm

[root@19c-rac1 ~]# modprobe  oracleoks

[root@19c-rac1 ~]# lsmod | grep oracle

oracleafd             221184  1

oracleacfs           4997120  1

oracleadvm            745472  7

oracleoks             729088  2 oracleadvm,oracleacfs

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# mkfs -t acfs /dev/asm/acfs_vol-296

mkfs.acfs: version                   = 19.0.0.0.0

mkfs.acfs: on-disk version           = 46.0

mkfs.acfs: volume                    = /dev/asm/acfs_vol-296

mkfs.acfs: volume size               = 20401094656  (  19.00 GB )

mkfs.acfs: Format complete.

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# chown -R oracle:oinstall /acfs_vol

chown: cannot access '/acfs_vol': No such file or directory

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd /

[root@19c-rac1 /]# mkdir acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# chown -R oracle:oinstall /acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# chmod -R 777 /acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# acfsutil registry -a /dev/asm/acfs_vol-296 /acfs_vol

acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry

[root@19c-rac1 /]#

[root@19c-rac1 /]# acfsutil registry -c /dev/asm/acfs_vol-296 /acfs_vol -u oracle

acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-296'

[root@19c-rac1 /]#

[root@19c-rac1 /]# mount -t acfs /dev/asm/acfs_vol-296 /acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# df -h

Filesystem             Size  Used Avail Use% Mounted on

devtmpfs               4.8G  8.0K  4.8G   1% /dev

tmpfs                  4.8G  845M  3.9G  18% /dev/shm

tmpfs                  4.8G   18M  4.8G   1% /run

tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup

/dev/mapper/ol-root     50G   26G   25G  51% /

/dev/mapper/ol-home     42G  7.4G   35G  18% /home

/dev/sda1             1014M  291M  724M  29% /boot

tmpfs                  968M  1.2M  967M   1% /run/user/42

tmpfs                  968M  4.6M  964M   1% /run/user/54321

/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# mount | grep acfs

/dev/asm/acfs_vol-296 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)

[root@19c-rac1 /]#

[root@19c-rac1 /]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

[root@19c-rac1 /]#

[root@19c-rac1 /]# vi /etc/fstab

[root@19c-rac1 /]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0

[root@19c-rac1 /]#

[root@19c-rac1 /]# crsctl stat res -t | grep acfs

ora.acfsdg.acfs_vol.acfs

               ONLINE  ONLINE       19c-rac1                 mounted on /acfs_vol

               ONLINE  ONLINE       19c-rac2                 mounted on /acfs_vol

               ONLINE  ONLINE       19c-rac3                 mounted on /acfs_vol

[root@19c-rac1 /]#

[root@19c-rac1 /]# crsctl stat res -t | grep ACFS

ora.ACFSDG.ACFS_VOL.advm

ora.ACFSDG.dg(ora.asmgroup)

[root@19c-rac1 /]#

[root@19c-rac1 /]# srvctl status diskgroup -diskgroup ACFSDG -verbose

Disk Group ACFSDG is running on 19c-rac1,19c-rac3,19c-rac2

[root@19c-rac1 /]#

[root@19c-rac1 /]# exit

logout

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 21:10:08 2020

Version 19.8.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

SQL> set lines 100 pages 100

SQL>  col name for a20

col path for a18

col library for a60

set lines 160

SQL> select name, path, library from v$asm_disk where group_number <> 0;

NAME                 PATH               LIBRARY

-------------------- ------------------ ------------------------------------------------------------

OCRVD_0002           /dev/sdl1          System

OCRVD_0003           /dev/sdm1          System

OCRVD_0000           /dev/sdj1          System

OCRVD_0001           /dev/sdk1          System

ACFSDG01             AFD:ACFSDG01       AFD Library - Generic , version 3 (KABI_V3)

ACFSDG02             AFD:ACFSDG02       AFD Library - Generic , version 3 (KABI_V3)

6 rows selected.

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/bin

[oracle@19c-rac1 bin]$ ./asmcmd afd_lsdsk

--------------------------------------------------------------------------------

Label                     Filtering   Path

=========================================================================ACFSDG01                    ENABLED   /dev/sdp1

ACFSDG02                    ENABLED   /dev/sdq1

[oracle@19c-rac1 bin]$

[oracle@19c-rac2 ~]$ . oraenv

ORACLE_SID = [+ASM2] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ cd $ORACLE_HOME/bin

[oracle@19c-rac2 bin]$ asmcmd dsget

parameter:/dev/sd*, AFD:*

profile:/dev/sd*,AFD:*

[oracle@19c-rac2 bin]$

[oracle@19c-rac2 bin]$ cd

[oracle@19c-rac2 ~]$ su - root

Password:

[root@19c-rac2 ~]#

[root@19c-rac2 /]# df -h

Filesystem             Size  Used Avail Use% Mounted on

devtmpfs               4.8G  8.0K  4.8G   1% /dev

tmpfs                  4.8G  845M  3.9G  18% /dev/shm

tmpfs                  4.8G   26M  4.7G   1% /run

tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup

/dev/mapper/ol-root     50G   25G   26G  49% /

/dev/sda1             1014M  290M  725M  29% /boot

/dev/mapper/ol-home     42G  1.8G   40G   5% /home

tmpfs                  968M  1.2M  967M   1% /run/user/42

tmpfs                  968M     0  968M   0% /run/user/54321

/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol

[root@19c-rac2 /]#

[root@19c-rac2 /]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

[root@19c-rac2 /]#

[root@19c-rac2 /]# vi /etc/fstab

[root@19c-rac2 /]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0

[root@19c-rac2 /]#

[oracle@19c-rac3 ~]$ . oraenv

ORACLE_SID = [+ASM3] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ cd $ORACLE_HOME/bin

[oracle@19c-rac3 bin]$

[oracle@19c-rac3 bin]$ asmcmd dsget

parameter:/dev/sd*, AFD:*

profile:/dev/sd*,AFD:*

[oracle@19c-rac3 ~]$ df -h

Filesystem             Size  Used Avail Use% Mounted on

devtmpfs               4.8G  8.0K  4.8G   1% /dev

tmpfs                  4.8G  845M  3.9G  18% /dev/shm

tmpfs                  4.8G   34M  4.7G   1% /run

tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup

/dev/mapper/ol-root     50G   25G   26G  49% /

/dev/mapper/ol-home     42G  1.8G   40G   5% /home

/dev/sda1             1014M  290M  725M  29% /boot

tmpfs                  968M  1.2M  967M   1% /run/user/42

tmpfs                  968M     0  968M   0% /run/user/54321

/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol

[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ su - root

Password:

[root@19c-rac3 ~]#

[root@19c-rac3 ~]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

[root@19c-rac3 ~]#

[root@19c-rac3 ~]# vi /etc/fstab

[root@19c-rac3 ~]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun May 17 21:14:39 2020

#

# Accessible filesystems, by reference, are maintained under '/dev/disk/'.

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.

#

# After editing this file, run 'systemctl daemon-reload' to update systemd

# units generated from this file.

#

/dev/mapper/ol-root     /                       xfs     defaults        0 0

UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0

/dev/mapper/ol-home     /home                   xfs     defaults        0 0

/dev/mapper/ol-swap     swap                    swap    defaults        0 0

/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0

[root@19c-rac3 ~]#

Final Status

Hope it helps...Thank You for referring my blog.

Stop/Start all Oracle RAC databases at once - It's safe to apply the patches

 Applied OCW Patch 30304402 

Node-1:

[oracle@rac-p1 ~]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch lspatches

30340202;Database Patch Set Update : 12.1.0.2.200114 (30340202)

25869825;OCW Patch Set Update : 12.1.0.2.170718 (25869825)

OPatch succeeded.

[oracle@rac-p1 ~]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ cd 30464119/30304402/

[oracle@rac-p1 30304402]$ cd custom/scripts

[oracle@rac-p1 scripts]$ ls

postpatch.sh  prepatch.sh

[oracle@rac-p1 scripts]$ pwd

/home/oracle/30464119/30304402/custom/scripts

[oracle@rac-p1 scripts]$

Stopping all RAC databases of an ORACLE_HOME

This command will stop all databases running on the ORACLE_HOME specified and save the state before the shutdown in a state file.

[oracle@rac-p1 scripts]$ srvctl stop home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p1_stop_3.dmp -n rac-p1

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ srvctl status database -d racp

Instance racp1 is not running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ ./prepatch.sh -dbhome /u01/app/oracle/product/12.1.0/db_1

./prepatch.sh completed successfully.

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1/ -local /home/oracle/30464119/30304402/

Oracle Interim Patch Installer version 12.2.0.1.23

Copyright (c) 2020, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/db_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/12.1.0/db_1//oraInst.loc

OPatch version    : 12.2.0.1.23

OUI version       : 12.1.0.2.0

Log file location : /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_10-38-56AM_1.log

Verifying environment and performing prerequisite checks...

--------------------------------------------------------------------------

Start OOP by Prereq process.

Launch OOP...

Oracle Interim Patch Installer version 12.2.0.1.23

Copyright (c) 2020, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/db_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/12.1.0/db_1//oraInst.loc

OPatch version    : 12.2.0.1.23

OUI version       : 12.1.0.2.0

Log file location : /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_10-39-13AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   30304402

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/oracle/product/12.1.0/db_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '30304402' to OH '/u01/app/oracle/product/12.1.0/db_1'

ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ] , [ oracle.has.cvu, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.has.db, 12.1.0.2.0...

Patching component oracle.has.common, 12.1.0.2.0...

Patching component oracle.has.deconfig, 12.1.0.2.0...

Patching component oracle.has.common.cvu, 12.1.0.2.0...

Patching component oracle.has.rsf, 12.1.0.2.0...

Patch 30304402 successfully applied.

Sub-set patch [25869825] has become inactive due to the application of a super-set patch [30304402].

Please refer to Doc ID 2161861.1 for any possible further required actions.

Log file location: /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_10-39-13AM_1.log

OPatch succeeded.

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ ./postpatch.sh -dbhome /u01/app/oracle/product/12.1.0/db_1

Reading /u01/app/oracle/product/12.1.0/db_1/install/params.ora..

Reading /u01/app/oracle/product/12.1.0/db_1/install/params.ora..

Found uninstantiated /u01/app/oracle/product/12.1.0/db_1/crs/sbs/srvctl.sbs file, copying to /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/racgwrap

Skipping the missing file /u01/app/oracle/product/12.1.0/db_1/bin/racgwrap

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/diskmon.bin

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/lsnodes

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/osdbagrp

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/rawutl

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/srvm/admin/ractrans

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/srvm/admin/getcrshome

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/crsdiag.pl

Postpatch completed successfully

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ srvctl status database -d racp

Instance racp1 is not running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 scripts]$

Starting all RAC databases of an ORACLE_HOME

This command uses a state file to restart all databases previously running on a node.

[oracle@rac-p1 scripts]$ srvctl start home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p1_stop_3.dmp -n rac-p1

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 scripts]$

[oracle@rac-p1 scripts]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch lspatches

30304402;OCW PATCH SET UPDATE 12.1.0.2.200114 (30304402)

30340202;Database Patch Set Update : 12.1.0.2.200114 (30340202)

OPatch succeeded.

[oracle@rac-p1 scripts]$

Node-2:

[oracle@rac-p2 ~]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch lspatches

30340202;Database Patch Set Update : 12.1.0.2.200114 (30340202)

25869825;OCW Patch Set Update : 12.1.0.2.170718 (25869825)

OPatch succeeded.

[oracle@rac-p2 ~]$

[oracle@rac-p2 ~]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p2 ~]$

[oracle@rac-p2 ~]$ cd 30464119/30304402/custom/scripts/

[oracle@rac-p2 scripts]$ srvctl stop home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p2_stop_3.dmp -n rac-p2

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is not running on node rac-p2

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ ./prepatch.sh -dbhome /u01/app/oracle/product/12.1.0/db_1

./prepatch.sh completed successfully.

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch apply -oh /u01/app/oracle/product/12.1.0/db_1/ -local /home/oracle/30464119/30304402/

Oracle Interim Patch Installer version 12.2.0.1.23

Copyright (c) 2020, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/db_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/12.1.0/db_1//oraInst.loc

OPatch version    : 12.2.0.1.23

OUI version       : 12.1.0.2.0

Log file location : /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_02-33-15AM_1.log

Verifying environment and performing prerequisite checks...

-------------------------------------------------------------------------

Start OOP by Prereq process.

Launch OOP...

Oracle Interim Patch Installer version 12.2.0.1.23

Copyright (c) 2020, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/db_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/12.1.0/db_1//oraInst.loc

OPatch version    : 12.2.0.1.23

OUI version       : 12.1.0.2.0

Log file location : /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_02-33-34AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   30304402

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/oracle/product/12.1.0/db_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '30304402' to OH '/u01/app/oracle/product/12.1.0/db_1'

ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ] , [ oracle.has.cvu, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.has.db, 12.1.0.2.0...

Patching component oracle.has.common, 12.1.0.2.0...

Patching component oracle.has.deconfig, 12.1.0.2.0...

Patching component oracle.has.common.cvu, 12.1.0.2.0...

Patching component oracle.has.rsf, 12.1.0.2.0...

Patch 30304402 successfully applied.

Sub-set patch [25869825] has become inactive due to the application of a super-set patch [30304402].

Please refer to Doc ID 2161861.1 for any possible further required actions.

Log file location: /u01/app/oracle/product/12.1.0/db_1/cfgtoollogs/opatch/opatch2020-12-03_02-33-34AM_1.log

OPatch succeeded.

[oracle@rac-p2 scripts]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is not running on node rac-p2

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ ./postpatch.sh -dbhome /u01/app/oracle/product/12.1.0/db_1

Reading /u01/app/oracle/product/12.1.0/db_1/install/params.ora..

Reading /u01/app/oracle/product/12.1.0/db_1/install/params.ora..

Found uninstantiated /u01/app/oracle/product/12.1.0/db_1/crs/sbs/srvctl.sbs file, copying to /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Parsing file /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/racgwrap

Skipping the missing file /u01/app/oracle/product/12.1.0/db_1/bin/racgwrap

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Verifying file /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/srvctl

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/srvconfig

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/cluvfy

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/kfod

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/diskmon.bin

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/lsnodes

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/osdbagrp

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/rawutl

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/srvm/admin/ractrans

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/srvm/admin/getcrshome

Reapplying file permissions on /u01/app/oracle/product/12.1.0/db_1/bin/crsdiag.pl

Postpatch completed successfully

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ srvctl start home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p2_stop_3.dmp -n rac-p2

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p2 scripts]$ sh /u01/app/oracle/product/12.1.0/db_1/OPatch/opatch lspatches

30304402;OCW PATCH SET UPDATE 12.1.0.2.200114 (30304402)

30340202;Database Patch Set Update : 12.1.0.2.200114 (30340202)

OPatch succeeded.

[oracle@rac-p2 scripts]$

Final Status:

[oracle@rac-p1 scripts]$ srvctl status home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p1_stop_3.dmp -n rac-p1

Database racp is running on node rac-p1

[oracle@rac-p1 scripts]$ srvctl status home -o /u01/app/oracle/product/12.1.0/db_1 -n rac-p1

PRKO-2082 : Missing mandatory option -statefile

[oracle@rac-p1 scripts]$

[oracle@rac-p2 scripts]$ srvctl status home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p2_stop_4.dmp -n rac-p2

Database racp is running on node rac-p2

[oracle@rac-p2 scripts]$

[oracle@rac-p2 scripts]$ srvctl status home -o /u01/app/oracle/product/12.1.0/db_1 -s /tmp/rac-p2_stop_4.dmp

PRKO-2082 : Missing mandatory option -node

[oracle@rac-p2 scripts]$

Hope it helps....

PRCR-1079 CRS-5017 ORA-00821 CRS-2674 CRS-2632 -> ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

 [oracle@rac-p1 ~]$ . oraenv

ORACLE_SID = [racp1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:07:01 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> show parameter streams_pool

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

streams_pool_size                    big integer 0

SQL>

SQL> alter system set streams_pool_size=1024M scope=spfile sid='*';

System altered.

After changing streams_pool_size

[oracle@rac-p1 ~]$ srvctl stop database -d racp

[oracle@rac-p1 ~]$ srvctl start database -d racp

PRCR-1079 : Failed to start resource ora.racp.db

CRS-5017: The resource action "ora.racp.db start" encountered the following error:

ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac-p1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.racp.db' on 'rac-p1' failed

CRS-5017: The resource action "ora.racp.db start" encountered the following error:

ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac-p2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.racp.db' on 'rac-p2' failed

CRS-2632: There are no more servers to try to place resource 'ora.racp.db' on that would satisfy its placement policy

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ . oraenv

ORACLE_SID = [racp] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:12:11 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;

ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

SQL> exit

Disconnected

[oracle@rac-p1 ~]$

The instance doesn’t come up! This is easy to fix without having to restore the spfile from backup:

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:28:35 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;

ORA-00821: Specified value of sga_target 900M is too small, needs to be at least 1384M

SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora' from spfile='+DATA/racp/spfileracp.ora';

File created.

SQL> host vi /u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora

Now correct the value in the text file. I just changed the sga_target parameter from it here.

racp1.__sga_target=3072M

racp2.__sga_target=3072M

*.sga_target=3072M

*.streams_pool_size=256M

SQL> create spfile='+DATA/racp/spfileracp.ora' from pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1129.ora';

File created.

SQL> startup force;

ORACLE instance started.

Total System Global Area 3221225472 bytes

Fixed Size                  2929552 bytes

Variable Size             822086768 bytes

Database Buffers         2382364672 bytes

Redo Buffers               13844480 bytes

Database mounted.

Database opened.

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is not running on node rac-p2

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl stop database -d racp

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp

Instance racp1 is not running on node rac-p1

Instance racp2 is not running on node rac-p2

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl start database -d racp

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ srvctl status database -d racp

Instance racp1 is running on node rac-p1

Instance racp2 is running on node rac-p2

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ . oraenv

ORACLE_SID = [racp1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac-p1 ~]$

[oracle@rac-p1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:43:57 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE

-------------------- ----------------

READ WRITE           PRIMARY

READ WRITE           PRIMARY

SQL> show parameter sga_

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

sga_max_size                         big integer 3G

sga_target                           big integer 3G

unified_audit_sga_queue_size         integer     1048576

SQL>

SQL> show parameter streams_p

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

streams_pool_size                    big integer 256M

SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp1_1130.ora' from spfile='+DATA/racp/spfileracp.ora';

File created.

SQL>

Checking Node-2 (rac-p2)

[oracle@rac-p2 ~]$ . oraenv

ORACLE_SID = [racp2] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac-p2 ~]$

[oracle@rac-p2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 29 15:19:00 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE

-------------------- ----------------

READ WRITE           PRIMARY

READ WRITE           PRIMARY

SQL> show parameter sga_

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

sga_max_size                         big integer 3G

sga_target                           big integer 3G

unified_audit_sga_queue_size         integer     1048576

SQL>

SQL> show parameter streams_p

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

streams_pool_size                    big integer 256M

SQL>

SQL> create pfile='/u01/app/oracle/product/12.1.0/db_1/dbs/initracp2_1130.ora' from spfile='+DATA/racp/spfileracp.ora';

File created.

SQL>

Oracle 19c (19.9.0) with ASMFD Driver, Kernel 5.4.17 and Oracle Enterprise Linux 8.2 (3-Node RAC)

1. Installed OEL 8.2

2. Upgraded Kernel "kernel-uek-4.18.0-193.13.2.el8_2.x86_64" to kernel-uek-5.4.17-2036.100.6.1.el8uek.x86_64

3. Installed GI 19.3 (while GI installation applied Oracle 19c GI (19.8) RU)

[oracle@19c-rac1 ~]$ cd /u01/app/19.3.0/grid/

[oracle@19c-rac1 grid]$ ./gridSetup.sh -applyPSU /home/oracle/31305339/

Preparing the home to patch...

Applying the patch /home/oracle/31305339/...

Successfully applied the patch.

The log can be found at: /tmp/GridSetupActions2020-11-12_09-01-20PM/installerPatchActions_2020-11-12_09-01-20PM.log

Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at:

 /u01/app/19.3.0/grid/install/response/grid_2020-11-12_09-01-20PM.rsp

You can find the log of this install session at:

 /tmp/GridSetupActions2020-11-12_09-01-20PM/gridSetupActions2020-11-12_09-01-20PM.log

Moved the install session logs to:

 /u01/app/oraInventory/logs/GridSetupActions2020-11-12_09-01-20PM

[oracle@19c-rac1 grid]$

[oracle@19c-rac1 bin]$ afddriverstate installed

AFD-620: AFD is not supported on this operating system version: 'unknown'

AFD-9204: AFD device driver installed status: 'false'

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ afddriverstate loaded

AFD-620: AFD is not supported on this operating system version: 'unknown'

AFD-9206: AFD device driver loaded status: 'false'

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ afddriverstate version

AFD-620: AFD is not supported on this operating system version: 'unknown'

AFD-642: AFD not installed

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ cat /etc/oracle-release

Oracle Linux Server release 8.2

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ uname -r

5.4.17-2036.100.6.1.el8uek.x86_64

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ acfsload start

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

ACFS-9130: Root access required

[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ su - root

Password:

[root@19c-rac1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base has been set to /u01/app/oracle

[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./afddriverstate supported

AFD-620: AFD is not supported on this operating system version: 'unknown'

AFD-9201: Not Supported

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

ACFS-9204: false

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

ACFS-9204: false

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

ACFS-9201: Not Supported

ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

ACFS-9129: ADVM/ACFS not installed

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_configure

ASMCMD-9520: AFD is not 'supported'

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# acfsroot install

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'unknown'

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# cat /etc/oracle-release

Oracle Linux Server release 8.2

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# uname -r

5.4.17-2036.100.6.1.el8uek.x86_64

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# uname -a

Linux 19c-rac1.localdomain 5.4.17-2036.100.6.1.el8uek.x86_64 #2 SMP Thu Oct 29 17:06:00 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@19c-rac1 bin]#

[root@19c-rac1 bin]# exit

logout

[oracle@19c-rac1 ~]$ . oraenv

ORACLE_SID = [cdbrac1] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME

[oracle@19c-rac1 grid]$ cd OPatch/

[oracle@19c-rac1 OPatch]$ ./opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[oracle@19c-rac1 OPatch]$