Tuesday, June 29, 2021

ORA-16692: operation disallowed for a database or far sync instance that sends redo data - Converting from Physical Standby Database to Snapshot Standby Database

Configured two physical standby databases for primary database and also setting the RedoRoutes property between these databases.


Primary Database: orcldb

Physical Standby Database: orcldbs 

Physical Standby Database: orcldbp


1. Check the configuration

DGMGRL> show configuration;

Configuration - dg_rac

  Protection Mode: MaxPerformance

  Members:

  orcldb  - Primary database

    orcldbs - Physical standby database

      orcldbp - Physical standby database (receiving current redo)

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 35 seconds ago)

DGMGRL>


2. Check the RedoRoutes property

DGMGRL> show database orcldb redoroutes;

  RedoRoutes = '(LOCAL : orcldbs SYNC)'

DGMGRL>

DGMGRL> show database orcldbs redoroutes;

  RedoRoutes = '(LOCAL : orcldb SYNC, orcldbp SYNC)(orcldb : orcldbp ASYNC)(orcldbp : orcldb ASYNC)'

DGMGRL>

DGMGRL> show database orcldbp redoroutes;

  RedoRoutes = '(LOCAL : orcldbs SYNC)'

DGMGRL>

3. Configured using following commands

DGMGRL>edit database orcldb set property redoroutes='(LOCAL : orcldbs SYNC)';

Property "redoroutes" updated

DGMGRL>edit database orcldbs set property redoroutes='(LOCAL : orcldb SYNC, orcldbp SYNC)(orcldb : orcldbp ASYNC)(orcldbp : orcldb ASYNC)';

Property "redoroutes" updated

DGMGRL>edit database orcldbp set property redoroutes='(LOCAL : orcldbs SYNC)';

Property "redoroutes" updated

While trying to convert one of the physical standby database to snapshot standby database getting the following error.

DGMGRL> convert database orcldbs to snapshot standby;

Converting database "orcldbs" to a Snapshot Standby database, please wait...

Error: ORA-16692: operation disallowed for a database or far sync instance that sends redo data

Failed.

Failed to convert database "orcldbs"

DGMGRL>

Cause: An attempt was made to delete, disable, or convert (physical standby database) a member that 

has a non-empty value set for its RedoRoutes property.

Action: Reset the value of the RedoRoutes property for the member to be deleted, disabled, or 

converted (physical standby database) to the empty string and then retry the operation.

4. Resetting RedoRoutes property for orcldbs database

DGMGRL> edit database orcldbs set property redoroutes='';

Property "redoroutes" updated

DGMGRL>

DGMGRL> show configuration;

Configuration - dg_rac

  Protection Mode: MaxPerformance

  Members:

  orcldb  - Primary database

    orcldbs - Physical standby database

  Members Not Receiving Redo:

  orcldbp - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 25 seconds ago)

DGMGRL>

5. Converted to Snapshot Standby Database

DGMGRL> convert database orcldbs to snapshot standby;

Converting database "orcldbs" to a Snapshot Standby database, please wait...

Database "orcldbs" converted successfully

DGMGRL>

6. Login to orcldbs database and check the database role

[oracle@standby ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jun 29 19:39:09 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

ORCLDB    READ WRITE           SNAPSHOT STANDBY

SQL> exit

7. Converted back to physical standby database from snapshot standby database

DGMGRL> convert database orcldbs to physical standby;

Converting database "orcldbs" to a Physical Standby database, please wait...

Operation requires shut down of instance "orcldbs" on database "orcldbs"

Shutting down instance "orcldbs"...

Database closed.

Database dismounted.

ORACLE instance shut down.

Operation requires start up of instance "orcldbs" on database "orcldbs"

Starting instance "orcldbs"...

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Database mounted.

Continuing to convert database "orcldbs" ...

Database "orcldbs" converted successfully

DGMGRL>

DGMGRL> edit database orcldbs set property redoroutes='(LOCAL : orcldb SYNC, orcldbp SYNC)(orcldb : orcldbp ASYNC)(orcldbp : orcldb ASYNC)';

Property "redoroutes" updated

DGMGRL>


DGMGRL> show configuration;

Configuration - dg_rac

  Protection Mode: MaxPerformance

  Members:

  orcldb  - Primary database

    orcldbs - Physical standby database

      orcldbp - Physical standby database (receiving current redo)

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 7 seconds ago)

DGMGRL>

8. Login to orcldbs database and check the database role

[oracle@standby ~]$ . oraenv

ORACLE_SID = [orcldbs] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@standby ~]$

[oracle@standby ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jun 29 19:59:41 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select open_mode,database_role from v$database;

OPEN_MODE            DATABASE_ROLE

-------------------            ------------------------

READ ONLY WITH APPLY PHYSICAL STANDBY

SQL>


Saturday, December 19, 2020

Oracle 19c (19.9) with ASMFD, Kernel 5.4.17 & Oracle Linux 8.3 (2-Node RAC)

Oracle has released the patches for Oracle 19c (19.8) and 19c (19.9) for ACFS/AFD drivers.

1. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.9.0.0.0ACFSRU
2. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.8.0.0.0ACFSRU

Please refer for more details:
ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Click Additional notes: 
OL8 UEK6: 30590023 OL8 UEK6 SUPPORT FOR ACFS/AFD (NOTE: 19.x and later Grid only)

Environment:

1. Installed OEL 8.3
2. Upgraded Kernel "kernel-uek-4.18.0-193.13.2.el8_2.x86_64" to kernel-uek-5.4.17-2036.100.6.1.el8uek.x86_64
3. Installed GI 19.3 and applied patch 19.7 and applied Oracle 19c (19.9)
4. Applied "p30590023_199000ACFSRU_Linux-x86-64.zip" patch on Oracle 19c (19.9) for GRID

[oracle@rac2-oel77-123 ~]$ . oraenv
ORACLE_SID = [orcldb2] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac2-oel77-123 ~]$
[oracle@rac2-oel77-123 ~]$ cd $ORACLE_HOME/OPatch
[oracle@rac2-oel77-123 OPatch]$ ./opatch lspatches
30898856;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30898856)
30894985;OCW RELEASE UPDATE 19.7.0.0.0 (30894985)
30869304;ACFS RELEASE UPDATE 19.7.0.0.0 (30869304)
30869156;Database Release Update : 19.7.0.0.200414 (30869156)

OPatch succeeded.
[oracle@rac2-oel77-123 OPatch]$


[oracle@rac1-oel77-122 ~]$ su - root
Password:
[root@rac1-oel77-122 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# ps -ef | grep pmon
oracle      6506       1  0 15:21 ?        00:00:00 asm_pmon_+ASM1
oracle      8961       1  0 15:22 ?        00:00:00 mdb_pmon_-MGMTDB
oracle      9092       1  0 15:22 ?        00:00:00 ora_pmon_orcldb1
root       38171   37950  0 16:23 pts/1    00:00:00 grep --color=auto pmon
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# sh /u01/app/19.7.0/grid/OPatch/opatchauto apply /home/oracle/31750108/

OPatchauto session is initiated at Sat Dec 19 16:24:20 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-19_04-24-23PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-19_04-24-58PM.log
The id for this session is SKYE

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/19.7.0/grid
Binary patch applied successfully on home /u01/app/19.7.0/grid

Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-oel77-122
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-30-30PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-30-30PM_1.log

Host:rac1-oel77-122
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-39-54PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-39-54PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-39-54PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_16-39-54PM_1.log


OPatchauto session completed at Sat Dec 19 16:54:51 2020
Time taken to complete the session 30 minutes, 31 seconds
[root@rac1-oel77-122 ~]# 


[root@rac2-oel77-123 ~]# . oraenv
ORACLE_SID = [+ASM2] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# sh /u01/app/19.7.0/grid/OPatch/opatchauto apply /home/oracle/31750108/

OPatchauto session is initiated at Sat Dec 19 17:19:31 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-19_05-19-35PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-19_05-19-56PM.log
The id for this session is BKPL

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/d                                                                                       b_1
Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product                                    /19.3.0/db_1

Start applying binary patch on home /u01/app/19.7.0/grid
Binary patch applied successfully on home /u01/app/19.7.0/grid

Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac2-oel77-123
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-28-51PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-28-51PM_1.log

Host:rac2-oel77-123
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-38-02PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-38-02PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-38-02PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_17-38-02PM_1.log

OPatchauto session completed at Sat Dec 19 18:04:34 2020
Time taken to complete the session 45 minutes, 3 seconds
[root@rac2-oel77-123 ~]#


[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ cd $ORACLE_HOME/OPatch

[oracle@rac1-oel77-122 OPatch]$ ./opatch lspatches
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31773437;ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

OPatch succeeded.
[oracle@rac1-oel77-122 OPatch]$


[oracle@rac1-oel77-122 ~]$ su - root
Password:
[root@rac1-oel77-122 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# sh /u01/app/19.7.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Sat Dec 19 18:29:38 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-19_06-29-40PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-19_06-30-18PM.log
The id for this session is 1WBA

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.7.0/grid


Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
No step execution required.........


Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service


Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid


Start applying binary patch on home /u01/app/19.7.0/grid
Binary patch applied successfully on home /u01/app/19.7.0/grid


Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid


Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-oel77-122
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/30590023/30590023
Reason: This patch is not applicable to this specified target type - "rac_database"


Host:rac1-oel77-122
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_18-32-14PM_1.log

OPatchauto session completed at Sat Dec 19 18:37:26 2020
Time taken to complete the session 7 minutes, 49 seconds
[root@rac1-oel77-122 ~]#

[root@rac2-oel77-123 ~]# sh /u01/app/19.7.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Sat Dec 19 18:42:26 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-19_06-42-30PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-19_06-42-51PM.log
The id for this session is 1VGP

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.7.0/grid

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
No step execution required.........

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service

Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid

Start applying binary patch on home /u01/app/19.7.0/grid
Binary patch applied successfully on home /u01/app/19.7.0/grid

Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac2-oel77-123
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/30590023/30590023
Reason: This patch is not applicable to this specified target type - "rac_database"


Host:rac2-oel77-123
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023
Log: /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-19_18-45-41PM_1.log


OPatchauto session completed at Sat Dec 19 18:54:07 2020
Time taken to complete the session 11 minutes, 41 seconds
[root@rac2-oel77-123 ~]#

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ cd $ORACLE_HOME/OPatch
[oracle@rac1-oel77-122 OPatch]$

[oracle@rac1-oel77-122 OPatch]$ ./opatch lspatches
30590023;ACFS Interim patch for 30590023
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

OPatch succeeded.

[oracle@rac1-oel77-122 ~]$ su - root
Password:

[root@rac1-oel77-122 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# cd $ORACLE_HOME/bin
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# ./afddriverstate supported
AFD-9200: Supported
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
[WARNING] 09389: ORACLE_HOME is not set to the location of the Grid Infrastructure home.
ACFS-9203: true
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid installed
ACFS-9203: true
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid loaded
ACFS-9203: true
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid supported
ACFS-9200: Supported
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# asmcmd afd_configure
ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1-oel77-122'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.chad' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.orcldb.db' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.orcldb.db' on 'rac1-oel77-122' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.rac1-oel77-122.vip' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.rac1-oel77-122.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac1-oel77-122' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac1-oel77-122' succeeded.
CRS-2677: Stop of 'ora.chad' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.cvu' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.rac1-oel77-122.vip' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.cvu' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.rac1-oel77-122.vip' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.scan3.vip' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ons' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.net1.network' on 'rac1-oel77-122' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1-oel77-122' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.crf' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ctssd' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.cssd' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.gipcd' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac1-oel77-122' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1-oel77-122' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1-oel77-122'
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1-oel77-122 bin]#
[root@rac1-oel77-122 bin]# ls -lrt afd*
-rwxr-xr-x 1 root oinstall     1000 Jun 20  2016 afdroot
-rwxr-xr-x 1 root oinstall 87863448 Nov 25 00:33 afdboot
-rwxr-xr-x 1 root oinstall   284992 Nov 25 00:33 afdtool.bin
-rwxr-xr-x 1 root oinstall      760 Dec 19 19:01 afdload
-rwxr-xr-x 1 root oinstall     3084 Dec 19 19:01 afddriverstate
-rwxr-xr-x 1 root oinstall     2862 Dec 19 19:01 afdtool
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# lsmod|grep oracle
oracleafd             221184  1
oracleacfs           5001216  0
oracleadvm            745472  0
oracleoks             729088  2 oracleadvm,oracleacfs
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# cat /etc/os-release | head -n2
NAME="Oracle Linux Server"
VERSION="8.3"
[root@rac1-oel77-122 bin]#

[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac1-oel77-122 bin]#

[oracle@rac2-oel77-123 ~]$ . oraenv
ORACLE_SID = [+ASM2] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac2-oel77-123 ~]$

[oracle@rac2-oel77-123 ~]$ cd $ORACLE_HOME/OPatch
[oracle@rac2-oel77-123 OPatch]$ ./opatch lspatches
30590023;ACFS Interim patch for 30590023
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

OPatch succeeded.

[oracle@rac2-oel77-123 ~]$ su - root
Password:
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# cd $ORACLE_HOME/OPatch
[root@rac2-oel77-123 OPatch]#
[root@rac2-oel77-123 OPatch]# ./opatch lspatches
The user is root. OPatch cannot continue if the user is root.

OPatch failed with error code 255
[root@rac2-oel77-123 OPatch]#
[root@rac2-oel77-123 OPatch]# cd
[root@rac2-oel77-123 ~]#
[root@rac2-oel77-123 ~]# cd $ORACLE_HOME/bin
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# ./afddriverstate supported
AFD-9200: Supported
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid installed
ACFS-9203: true
[root@rac2-oel77-123 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid loaded
ACFS-9203: true
[root@rac2-oel77-123 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid supported
ACFS-9200: Supported
[root@rac2-oel77-123 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# asmcmd afd_configure
ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2-oel77-123'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.orcldb.db' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.chad' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.orcldb.db' on 'rac2-oel77-123' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cvu' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.qosmserver' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac2-oel77-123' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac2-oel77-123' succeeded.
CRS-2677: Stop of 'ora.mgmtdb' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.rac2-oel77-123.vip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.rac2-oel77-123.vip' on 'rac2-oel77-123' succeeded
CRS-2675: Stop of 'ora.chad' on 'rac2-oel77-123' failed
CRS-2679: Attempting to clean 'ora.chad' on 'rac2-oel77-123'
CRS-2681: Clean of 'ora.chad' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.cvu' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.rac2-oel77-123.vip' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.cvu' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.rac2-oel77-123.vip' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.scan2.vip' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.scan3.vip' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.ons' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.net1.network' on 'rac2-oel77-123' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2-oel77-123' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.crf' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.ctssd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cssd' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac2-oel77-123' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2-oel77-123' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2-oel77-123 bin]#

[root@rac2-oel77-123 bin]# asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac2-oel77-123'
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# ls -lrt afd*
-rwxr-xr-x 1 root oinstall     1000 Aug  1 11:59 afdroot
-rwxr-xr-x 1 root oinstall 87863448 Nov 25 00:33 afdboot
-rwxr-xr-x 1 root oinstall   284992 Nov 25 00:33 afdtool.bin
-rwxr-xr-x 1 root oinstall      760 Dec 19 20:39 afdload
-rwxr-xr-x 1 root oinstall     3084 Dec 19 20:39 afddriverstate
-rwxr-xr-x 1 root oinstall     2862 Dec 19 20:39 afdtool
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# lsmod|grep oracle
oracleafd             221184  1
oracleacfs           5001216  0
oracleadvm            745472  0
oracleoks             729088  2 oracleadvm,oracleacfs
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac2-oel77-123 bin]#
[root@rac2-oel77-123 bin]#

Final Check:

[root@rac1-oel77-122 ~]# uname -r
5.4.17-2036.100.6.1.el8uek.x86_64
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# cat /etc/oracle-release
Oracle Linux Server release 8.3
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# cd $ORACLE_HOME/bin
[root@rac1-oel77-122 bin]#
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid installed
ACFS-9203: true
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid loaded
ACFS-9203: true
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid supported
ACFS-9200: Supported
[root@rac1-oel77-122 bin]# ./acfsdriverstate -orahome /u01/app/19.7.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.9.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.9.0.0.0).
[root@rac1-oel77-122 bin]#
[root@rac1-oel77-122 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1-oel77-122'
[root@rac1-oel77-122 bin]#
[root@rac1-oel77-122 bin]# exit
logout
[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$
[oracle@rac1-oel77-122 ~]$ cd $ORACLE_HOME/OPatch
[oracle@rac1-oel77-122 OPatch]$
[oracle@rac1-oel77-122 OPatch]$ ./opatch lspatches
30590023;ACFS Interim patch for 30590023
31780966;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)
31771877;Database Release Update : 19.9.0.0.201020 (31771877)

OPatch succeeded.
[oracle@rac1-oel77-122 OPatch]$

Friday, December 18, 2020

Oracle 19c (19.8) with ASMFD, Kernel 5.4.17 & Oracle Linux 8.2 (3-Node RAC)

Oracle has released the patches for Oracle 19c (19.8) and 19c (19.9) for ACFS/AFD drivers.

1. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.9.0.0.0ACFSRU
2. 30590023 UEK6 SUPPORT FOR ACFS/AFD (System Patch) - > 19.8.0.0.0ACFSRU

Please refer for more details:
ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Click Additional notes: 
OL8 UEK6: 30590023 OL8 UEK6 SUPPORT FOR ACFS/AFD (NOTE: 19.x and later Grid only)

Environment:

1. Installed OEL 8.2
2. Upgraded Kernel "kernel-uek-4.18.0-193.13.2.el8_2.x86_64" to kernel-uek-5.4.17-2036.100.6.1.el8uek.x86_64
3. Installed GI 19.3 (while GI installation applied Oracle 19c GI (19.8) RU)
4. Applied "p30590023_198000ACFSRU_Linux-x86-64.zip" patch on Oracle 19c (19.8) for GRID

19c-rac1:

[oracle@19c-rac1 ~]$ ps -ef | grep pmon
oracle     12651       1  0 18:30 ?        00:00:00 asm_pmon_+ASM1
oracle     14408       1  0 18:31 ?        00:00:00 mdb_pmon_-MGMTDB
oracle     35737   35364  0 18:39 pts/1    00:00:00 grep --color=auto pmon
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ uname -r
5.4.17-2036.100.6.1.el8uek.x86_64
[oracle@19c-rac1 ~]$
[oracle@19c-rac1 ~]$ uname -a
Linux 19c-rac1.localdomain 5.4.17-2036.100.6.1.el8uek.x86_64 #2 SMP Thu Oct 29 17:06:00 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[oracle@19c-rac1 ~]$
[oracle@19c-rac1 ~]$ cat /etc/oracle-release
Oracle Linux Server release 8.2
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ echo $ORACLE_HOME
/u01/app/19.3.0/grid
[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch
[oracle@19c-rac1 OPatch]$ ./opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@19c-rac1 OPatch]$

[oracle@19c-rac1 ~]$ chmod 777 p30590023_198000ACFSRU_Linux-x86-64.zip
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ scp p30590023_198000ACFSRU_Linux-x86-64.zip oracle@19c-rac2:/home/oracle/
p30590023_198000ACFSRU_Linux-x86-64.zip        100%  686MB  36.5MB/s   00:18
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ scp p30590023_198000ACFSRU_Linux-x86-64.zip oracle@19c-rac3:/home/oracle/
p30590023_198000ACFSRU_Linux-x86-64.zip        100%  686MB  37.3MB/s   00:18
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ unzip p30590023_198000ACFSRU_Linux-x86-64.zip
Archive:  p30590023_198000ACFSRU_Linux-x86-64.zip
   creating: 30590023/
   creating: 30590023/automation/
  inflating: 30590023/automation/messages.properties
  inflating: 30590023/automation/bp1-out-of-place-non-rolling-switchback.xml
  inflating: 30590023/automation/bp1-out-of-place-switchback.xml

[oracle@19c-rac1 ~]$ echo $ORACLE_HOME
/u01/app/19.3.0/grid
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch
[oracle@19c-rac1 OPatch]$ ./opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@19c-rac1 OPatch]$

[oracle@19c-rac1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.
[oracle@19c-rac1 OPatch]$
[oracle@19c-rac1 OPatch]$ cd
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ su - root
Password:
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:24:30 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-24-35PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-25-12PM.log
The id for this session is P258

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac1
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-27-43PM_1.log

OPatchauto session completed at Fri Dec 18 19:31:02 2020
Time taken to complete the session 6 minutes, 32 seconds
[root@19c-rac1 ~]#
[root@19c-rac1 ~]# exit
logout
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/OPatch
[oracle@19c-rac1 OPatch]$ ./opatch lspatches
30590023;ACFS Interim patch for 30590023
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
OPatch succeeded.
[oracle@19c-rac1 OPatch]$ cd
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ ps -ef | grep pmon
oracle     87342       1  0 19:30 ?        00:00:00 asm_pmon_+ASM1
oracle     97416   35364  0 19:48 pts/1    00:00:00 grep --color=auto pmon
[oracle@19c-rac1 ~]$

19c-rac2:

[oracle@19c-rac2 ~]$ . oraenv
ORACLE_SID = [cdbrac2] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ cd $ORACLE_HOME/OPatch
[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ ./opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.
[oracle@19c-rac2 OPatch]$

[oracle@19c-rac2 OPatch]$ cd
[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ ps -ef | grep pmon
oracle     15057       1  0 18:32 ?        00:00:00 asm_pmon_+ASM2
oracle     48512   29638  0 19:24 pts/0    00:00:00 grep --color=auto pmon
[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ su - root
Password:
[root@19c-rac2 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac2 ~]#

[root@19c-rac2 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:47:58 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-48-02PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-48-14PM.log
The id for this session is ZJLN

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac2
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-49-48PM_1.log

OPatchauto session completed at Fri Dec 18 19:53:11 2020
Time taken to complete the session 5 minutes, 13 seconds
[root@19c-rac2 ~]#

19c-rac3:

[oracle@19c-rac3 ~]$ . oraenv
ORACLE_SID = [cdbrac3] ? +ASM3
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ cd $ORACLE_HOME/OPatch
[oracle@19c-rac3 OPatch]$

[oracle@19c-rac3 OPatch]$ ./opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
OPatch succeeded.
[oracle@19c-rac3 OPatch]$

[oracle@19c-rac3 ~]$ ps -ef | grep pmon
oracle     14810       1  0 18:32 ?        00:00:00 asm_pmon_+ASM3
oracle     48694   29385  0 19:24 pts/0    00:00:00 grep --color=auto pmon
[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ su - root
Password:
[root@19c-rac3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM3
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac3 ~]#

[root@19c-rac3 ~]# sh /u01/app/19.3.0/grid/OPatch/opatchauto apply /home/oracle/30590023/

OPatchauto session is initiated at Fri Dec 18 19:54:35 2020

System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-18_07-54-39PM.log.

Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-18_07-54-50PM.log
The id for this session is KS92

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Patch applicability verified successfully on home /u01/app/19.3.0/grid

Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:19c-rac3
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/30590023/30590023
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-18_19-57-06PM_1.log

OPatchauto session completed at Fri Dec 18 20:03:37 2020
Time taken to complete the session 9 minutes, 2 seconds
[root@19c-rac3 ~]#

Checking drivers in 19c-rac1:

[oracle@19c-rac1 ~]$ su - root
Password:
[root@19c-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin
[root@19c-rac1 bin]# ./afddriverstate supported
AFD-9200: Supported

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
ACFS-9203: true

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded
ACFS-9203: true

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported
ACFS-9200: Supported

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).
[root@19c-rac1 bin]#

[root@19c-rac1 bin]#  asmcmd afd_configure
ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac1'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac1.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1'
CRS-2677: Stop of 'ora.19c-rac1.vip' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.cvu' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac1' succeeded.
CRS-2672: Attempting to start 'ora.19c-rac1.vip' on '19c-rac3'
CRS-2672: Attempting to start 'ora.scan3.vip' on '19c-rac2'
CRS-2672: Attempting to start 'ora.cvu' on '19c-rac2'
CRS-2676: Start of 'ora.cvu' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.19c-rac1.vip' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.scan3.vip' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac1'
CRS-2677: Stop of 'ora.ons' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac1'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.crf' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac1'
CRS-2677: Stop of 'ora.crf' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac1'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac1'
CRS-2677: Stop of 'ora.cssd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac1'
CRS-2677: Stop of 'ora.gpnpd' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.gipcd' on '19c-rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac1.localdomain'
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@19c-rac1 bin]#
[root@19c-rac1 bin]#
[root@19c-rac1 bin]# ls -lrt afd*
-rwxr-xr-x 1 root oinstall     1000 Jun 20  2016 afdroot
-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin
-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot
-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:07 afdtool
-rwxr-xr-x 1 root oinstall      760 Dec 18 20:07 afdload
-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:07 afddriverstate
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# lsmod|grep oracle
oracleafd             221184  1
oracleacfs           4997120  0
oracleadvm            745472  0
oracleoks             729088  2 oracleadvm,oracleacfs
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# cat /etc/os-release | head -n2
NAME="Oracle Linux Server"
VERSION="8.2"
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).
[root@19c-rac1 bin]#

Checking drivers in 19c-rac2:

[root@19c-rac2 ~]# cd $ORACLE_HOME/bin
[root@19c-rac2 bin]# ./afddriverstate supported
AFD-9200: Supported
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
ACFS-9203: true

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded
ACFS-9203: true

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported
ACFS-9200: Supported

[root@19c-rac2 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_configure
ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up

[root@19c-rac2 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.qosmserver' on '19c-rac2'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac2.vip' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac2'
CRS-2677: Stop of 'ora.cvu' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.19c-rac2.vip' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2'
CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac2'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac2' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac2' succeeded.
CRS-2672: Attempting to start 'ora.qosmserver' on '19c-rac1'
CRS-2672: Attempting to start 'ora.19c-rac2.vip' on '19c-rac3'
CRS-2672: Attempting to start 'ora.scan3.vip' on '19c-rac1'
CRS-2672: Attempting to start 'ora.cvu' on '19c-rac1'
CRS-2676: Start of 'ora.cvu' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.19c-rac2.vip' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.scan3.vip' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.qosmserver' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac2'
CRS-2677: Stop of 'ora.ons' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac2'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.crf' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac2'
CRS-2677: Stop of 'ora.crf' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.storage' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.mdnsd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac2'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac2'
CRS-2677: Stop of 'ora.cssd' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac2'
CRS-2677: Stop of 'ora.gpnpd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.gipcd' on '19c-rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac2.localdomain'
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@19c-rac2 bin]#
[root@19c-rac2 bin]# ls -lrt afd*
-rwxr-xr-x 1 root oinstall     1000 Nov 12 21:27 afdroot
-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin
-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot
-rwxr-xr-x 1 root oinstall      760 Dec 18 20:16 afdload
-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:16 afdtool
-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:16 afddriverstate
[root@19c-rac2 bin]#

[root@19c-rac2 bin]# lsmod|grep oracle
oracleafd             221184  1
oracleacfs           4997120  0
oracleadvm            745472  0
oracleoks             729088  2 oracleadvm,oracleacfs

[root@19c-rac2 bin]# cat /etc/os-release | head -n2
NAME="Oracle Linux Server"
VERSION="8.2"
[root@19c-rac2 bin]#

Checking drivers in 19c-rac3:

[root@19c-rac3 ~]# cd $ORACLE_HOME/bin
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# ./afddriverstate supported
AFD-9200: Supported

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid installed
ACFS-9203: true

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid loaded
ACFS-9203: true

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid supported
ACFS-9200: Supported

[root@19c-rac3 bin]# ./acfsdriverstate -orahome /u01/app/19.3.0/grid version
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_configure
ASMCMD-9523: command cannot be used when Oracle Clusterware stack is up

[root@19c-rac3 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '19c-rac3'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac3'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on '19c-rac3'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac3'
CRS-2677: Stop of 'ora.scan2.vip' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on '19c-rac3'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac3'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac3' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac3' succeeded.
CRS-2677: Stop of 'ora.mgmtdb' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on '19c-rac3'
CRS-2677: Stop of 'ora.MGMTLSNR' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac3.vip' on '19c-rac3'
CRS-2672: Attempting to start 'ora.MGMTLSNR' on '19c-rac1'
CRS-2677: Stop of 'ora.19c-rac3.vip' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on '19c-rac1'
CRS-2676: Start of 'ora.mgmtdb' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.chad' on '19c-rac1'
CRS-2672: Attempting to start 'ora.chad' on '19c-rac2'
CRS-2676: Start of 'ora.chad' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.chad' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.19c-rac3.vip' on '19c-rac2'
CRS-2672: Attempting to start 'ora.scan2.vip' on '19c-rac1'
CRS-2676: Start of 'ora.19c-rac3.vip' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on '19c-rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac3'
CRS-2677: Stop of 'ora.ons' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac3'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.crf' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.mdnsd' on '19c-rac3'
CRS-2677: Stop of 'ora.crf' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac3'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac3'
CRS-2677: Stop of 'ora.cssd' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.gpnpd' on '19c-rac3'
CRS-2677: Stop of 'ora.gpnpd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.gipcd' on '19c-rac3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '19c-rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host '19c-rac3.localdomain'
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@19c-rac3 bin]#
[root@19c-rac3 bin]# ls -lrt afd*
-rwxr-xr-x 1 root oinstall     1000 Nov 12 21:27 afdroot
-rwxr-xr-x 1 root oinstall   280896 Nov 15 09:17 afdtool.bin
-rwxr-xr-x 1 root oinstall 87596064 Nov 15 09:17 afdboot
-rwxr-xr-x 1 root oinstall     2860 Dec 18 20:25 afdtool
-rwxr-xr-x 1 root oinstall      760 Dec 18 20:25 afdload
-rwxr-xr-x 1 root oinstall     3084 Dec 18 20:25 afddriverstate
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# lsmod|grep oracle
oracleafd             221184  1
oracleacfs           4997120  0
oracleadvm            745472  0
oracleoks             729088  2 oracleadvm,oracleacfs
[root@19c-rac3 bin]#

[root@19c-rac3 bin]# cat /etc/os-release | head -n2
NAME="Oracle Linux Server"
VERSION="8.2"
[root@19c-rac3 bin]#

Creating diskgroup ACFSDG using asmca:

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [cdbrac1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     49136    25104                0           25104              0             Y  OCRVD/
ASMCMD>
ASMCMD> exit
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmca &
[1] 166449
[oracle@19c-rac1 ~]$

Screenshots: 19cr8-ASMFD-1.JPG to 19cr8-ASMFD-4.JPG


















































[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/bin
[oracle@19c-rac1 bin]$
[oracle@19c-rac1 bin]$ pwd
/u01/app/19.3.0/grid/bin
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsget
parameter:/dev/sd*
profile:/dev/sd*
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsset /dev/sd*,'AFD:*'
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 bin]$ asmcmd dsget
parameter:/dev/sd*, AFD:*
profile:/dev/sd*,AFD:*
[oracle@19c-rac1 bin]$

[oracle@19c-rac1 ~]$ su - root
Password:
[root@19c-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# cd $ORACLE_HOME/bin
[root@19c-rac1 bin]#

[root@19c-rac1 bin]#  ls -alrt /dev/oracleafd/disks/*
ls: cannot access '/dev/oracleafd/disks/*': No such file or directory

[root@19c-rac1 bin]# ./acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9325:     Driver OS kernel version = 5.4.17-2011.0.7.el8uek.x86_64.
ACFS-9326:     Driver build number = RELEASE.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9547:     Driver available build number = RELEASE.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.8.0.0.0).
ACFS-9549:     Kernel and command versions.
Kernel:
    Build version: 19.0.0.0.0
    Build full version: 19.8.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
Commands:
    Build version: 19.0.0.0.0
    Build full version: 19.8.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9322: completed
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ls -alrt /dev/oracleafd/disks/*
ls: cannot access '/dev/oracleafd/disks/*': No such file or directory
[root@19c-rac1 bin]#

[root@19c-rac1 bin]# ./asmcmd afd_label ACFSDG01 /dev/sdp1
[root@19c-rac1 bin]# ./asmcmd afd_label ACFSDG02 /dev/sdq1

[root@19c-rac1 bin]# ls -alrt /dev/oracleafd/disks/
total 8
drwxrwx--- 3 oracle dba      80 Dec 18 20:07 ..
-rw-rw-r-- 1 oracle oinstall 10 Dec 18 20:51 ACFSDG01
-rw-rw-r-- 1 oracle oinstall 10 Dec 18 20:51 ACFSDG02
drwxrwxr-x 2 oracle dba      80 Dec 18 20:51 .
[root@19c-rac1 bin]#

Note: Again executed ASMCA and created ACFSDG


















































[root@19c-rac1 bin]# exit
logout
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 20:52:11 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL>  SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME
------------ ------------------------------
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY
------------------------------------------------------------
           1 OCRVD
19.0.0.0.0
10.1.0.0.0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 20:59:54 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL> set lines 100 pages 100
SQL> col GROUP_NUMBER format 99
SQL> col NAME format a10
SQL> col COMPATIBILITY format a10
SQL> col DATABASE_COMPATIBILITY format a20
SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBIL DATABASE_COMPATIBILI
------------ ---------- ---------- --------------------
           1 OCRVD      19.0.0.0.0 10.1.0.0.0
           2 ACFSDG     19.0.0.0.0 10.1.0.0.0

SQL>  alter diskgroup ACFSDG set attribute 'compatible.rdbms'='19.0.0.0.0';
Diskgroup altered.

SQL> SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBIL DATABASE_COMPATIBILI
------------ ---------- ---------- --------------------
           1 OCRVD      19.0.0.0.0 10.1.0.0.0
           2 ACFSDG     19.0.0.0.0 19.0.0.0.0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ asmcmd
ASMCMD> volcreate -G ACFSDG -s 19G acfs_vol

ASMCMD> volinfo --all
Diskgroup Name: ACFSDG

         Volume Name: ACFS_VOL
         Volume Device: /dev/asm/acfs_vol-296
         State: ENABLED
         Size (MB): 19456
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

ASMCMD> exit
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ su - root
Password:
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# modprobe  oracleacfs
[root@19c-rac1 ~]# modprobe  oracleadvm
[root@19c-rac1 ~]# modprobe  oracleoks
[root@19c-rac1 ~]# lsmod | grep oracle
oracleafd             221184  1
oracleacfs           4997120  1
oracleadvm            745472  7
oracleoks             729088  2 oracleadvm,oracleacfs
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# mkfs -t acfs /dev/asm/acfs_vol-296
mkfs.acfs: version                   = 19.0.0.0.0
mkfs.acfs: on-disk version           = 46.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol-296
mkfs.acfs: volume size               = 20401094656  (  19.00 GB )
mkfs.acfs: Format complete.
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# chown -R oracle:oinstall /acfs_vol
chown: cannot access '/acfs_vol': No such file or directory
[root@19c-rac1 ~]#
[root@19c-rac1 ~]# cd /
[root@19c-rac1 /]# mkdir acfs_vol
[root@19c-rac1 /]#
[root@19c-rac1 /]# chown -R oracle:oinstall /acfs_vol
[root@19c-rac1 /]#
[root@19c-rac1 /]# chmod -R 777 /acfs_vol
[root@19c-rac1 /]#
[root@19c-rac1 /]# acfsutil registry -a /dev/asm/acfs_vol-296 /acfs_vol
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry
[root@19c-rac1 /]#
[root@19c-rac1 /]# acfsutil registry -c /dev/asm/acfs_vol-296 /acfs_vol -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-296'
[root@19c-rac1 /]#
[root@19c-rac1 /]# mount -t acfs /dev/asm/acfs_vol-296 /acfs_vol
[root@19c-rac1 /]#

[root@19c-rac1 /]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               4.8G  8.0K  4.8G   1% /dev
tmpfs                  4.8G  845M  3.9G  18% /dev/shm
tmpfs                  4.8G   18M  4.8G   1% /run
tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup
/dev/mapper/ol-root     50G   26G   25G  51% /
/dev/mapper/ol-home     42G  7.4G   35G  18% /home
/dev/sda1             1014M  291M  724M  29% /boot
tmpfs                  968M  1.2M  967M   1% /run/user/42
tmpfs                  968M  4.6M  964M   1% /run/user/54321
/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol
[root@19c-rac1 /]#

[root@19c-rac1 /]# mount | grep acfs
/dev/asm/acfs_vol-296 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@19c-rac1 /]#

[root@19c-rac1 /]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
[root@19c-rac1 /]#
[root@19c-rac1 /]# vi /etc/fstab
[root@19c-rac1 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0
[root@19c-rac1 /]#

[root@19c-rac1 /]# crsctl stat res -t | grep acfs
ora.acfsdg.acfs_vol.acfs
               ONLINE  ONLINE       19c-rac1                 mounted on /acfs_vol
               ONLINE  ONLINE       19c-rac2                 mounted on /acfs_vol
               ONLINE  ONLINE       19c-rac3                 mounted on /acfs_vol
[root@19c-rac1 /]#

[root@19c-rac1 /]# crsctl stat res -t | grep ACFS
ora.ACFSDG.ACFS_VOL.advm
ora.ACFSDG.dg(ora.asmgroup)
[root@19c-rac1 /]#

[root@19c-rac1 /]# srvctl status diskgroup -diskgroup ACFSDG -verbose
Disk Group ACFSDG is running on 19c-rac1,19c-rac3,19c-rac2
[root@19c-rac1 /]#
[root@19c-rac1 /]# exit
logout

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 18 21:10:08 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL> set lines 100 pages 100
SQL>  col name for a20
col path for a18
col library for a60
set lines 160

SQL> select name, path, library from v$asm_disk where group_number <> 0;

NAME                 PATH               LIBRARY
-------------------- ------------------ ------------------------------------------------------------
OCRVD_0002           /dev/sdl1          System
OCRVD_0003           /dev/sdm1          System
OCRVD_0000           /dev/sdj1          System
OCRVD_0001           /dev/sdk1          System
ACFSDG01             AFD:ACFSDG01       AFD Library - Generic , version 3 (KABI_V3)
ACFSDG02             AFD:ACFSDG02       AFD Library - Generic , version 3 (KABI_V3)

6 rows selected.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

[oracle@19c-rac1 ~]$ cd $ORACLE_HOME/bin
[oracle@19c-rac1 bin]$ ./asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
=========================================================================ACFSDG01                    ENABLED   /dev/sdp1
ACFSDG02                    ENABLED   /dev/sdq1
[oracle@19c-rac1 bin]$

[oracle@19c-rac2 ~]$ . oraenv
ORACLE_SID = [+ASM2] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac2 ~]$

[oracle@19c-rac2 ~]$ cd $ORACLE_HOME/bin
[oracle@19c-rac2 bin]$ asmcmd dsget
parameter:/dev/sd*, AFD:*
profile:/dev/sd*,AFD:*
[oracle@19c-rac2 bin]$

[oracle@19c-rac2 bin]$ cd
[oracle@19c-rac2 ~]$ su - root
Password:
[root@19c-rac2 ~]#

[root@19c-rac2 /]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               4.8G  8.0K  4.8G   1% /dev
tmpfs                  4.8G  845M  3.9G  18% /dev/shm
tmpfs                  4.8G   26M  4.7G   1% /run
tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup
/dev/mapper/ol-root     50G   25G   26G  49% /
/dev/sda1             1014M  290M  725M  29% /boot
/dev/mapper/ol-home     42G  1.8G   40G   5% /home
tmpfs                  968M  1.2M  967M   1% /run/user/42
tmpfs                  968M     0  968M   0% /run/user/54321
/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol
[root@19c-rac2 /]#

[root@19c-rac2 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
[root@19c-rac2 /]#

[root@19c-rac2 /]# vi /etc/fstab
[root@19c-rac2 /]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot                   xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0
[root@19c-rac2 /]#

[oracle@19c-rac3 ~]$ . oraenv
ORACLE_SID = [+ASM3] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ cd $ORACLE_HOME/bin
[oracle@19c-rac3 bin]$

[oracle@19c-rac3 bin]$ asmcmd dsget
parameter:/dev/sd*, AFD:*
profile:/dev/sd*,AFD:*

[oracle@19c-rac3 ~]$ df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               4.8G  8.0K  4.8G   1% /dev
tmpfs                  4.8G  845M  3.9G  18% /dev/shm
tmpfs                  4.8G   34M  4.7G   1% /run
tmpfs                  4.8G     0  4.8G   0% /sys/fs/cgroup
/dev/mapper/ol-root     50G   25G   26G  49% /
/dev/mapper/ol-home     42G  1.8G   40G   5% /home
/dev/sda1             1014M  290M  725M  29% /boot
tmpfs                  968M  1.2M  967M   1% /run/user/42
tmpfs                  968M     0  968M   0% /run/user/54321
/dev/asm/acfs_vol-296   19G  828M   19G   5% /acfs_vol
[oracle@19c-rac3 ~]$

[oracle@19c-rac3 ~]$ su - root
Password:
[root@19c-rac3 ~]#

[root@19c-rac3 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
[root@19c-rac3 ~]#

[root@19c-rac3 ~]# vi /etc/fstab
[root@19c-rac3 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun May 17 21:14:39 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=539eae26-4e39-4888-8697-cb027503ef1b /boot xfs     defaults        0 0
/dev/mapper/ol-home     /home                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/dev/asm/acfs_vol-296   /acfs_vol               acfs    defaults        0 0
[root@19c-rac3 ~]#

Final Status





























Hope it helps...Thank You for referring my blog.