Reflection of my thoughts and technical expriences: Out Of Place (OOP) Patching: Oracle 19c RAC (19.7.0) to Oracle 19c RAC (19.9.0) using 2-Step Method with Rollback Option

Before Out of Place Patching:

Node-1: rac1-oel77-122

Node-2: rac2-oel77-123

GI Home: /u01/app/19.7.0/grid

GI Version: Oracle 19c (19.7.0)

RDBMS Home: /u01/app/oracle/product/19.3.0/db_1 (Applied Patch 19.8)

RDBMS Version: Oracle 19c (19.8.0)

After Out of Place Patching:

Node-1: rac1-oel77-122

Node-2: rac2-oel77-123

GI Home: /home/oracle/19.9.0/grid

GI Version: Oracle 19c (19.9.0)

RDBMS Home: /home/oracle/product/19.9.0/db_1

RDBMS Version: Oracle 19c (19.9.0.0.0)

Patching softwares:

Oracle 19c (19.9.0.0.0) - GI - Oct2020

p31750108_190000_Linux-x86-64.zip

Oracle 19c (19.9.0.0.0) - RDBMS - Oct2020

p31771877_190000_Linux-x86-64.zip

Process of 2-Step Method:

1) Prepare Clone is invoked on Node-1 (rac1-oel77-122)

- New GI Home is prepared on Node 1 ( as per the provided location )

- New RAC Home is prepared on Node 1 ( as per the provided location )

- Cloning occurs for the respective homes

- Patching is done on the new homes

- There is no impact on services running from the original homes on any node

2) Prepare Clone is invoked on Node-2 (rac2-oel77-123)

- New GI Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )

- New RAC Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )

- Cloning occurs for the respective homes

- Patching is done on the new homes

- There is no impact on services running from the original homes on any node

3) Switch Clone is invoked on Node-1 (rac1-oel77-122)

- Cloned homes are verified

- Nodelist is updated in the inventory for new homes

- Database Service is brought down

- CRS Services are brought down

- CRS is started from the Cloned Home

- Database is started from Cloned home

- Postpatch operation is performed

4) Switch Clone is invoked on Node-2 (rac2-oel77-123)

- Cloned homes are verified

- Nodelist is updated in the inventory for new homes

- Database Service is brought down

- CRS Services are brought down

- CRS is started from the Cloned Home

- Database is started from Cloned home

- Postpatch operation is performed , Datapatch is invoked

Overall Steps:

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

[root@rac2-oel77-123 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

[root@rac1-oel77-122 OPatch]# ./opatchauto apply -switch-clone

[root@rac2-oel77-123 OPatch]# ./opatchauto apply -switch-clone

Login to Node-1(rac1-oel77-122)

[oracle@rac1-oel77-122 ~]$ . oraenv

ORACLE_SID = [orcldb1] ? +ASM1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 14:54:04 2020

Version 19.7.0.0.0

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME INSTANCE_NUMBER

---------------- ---------------

+ASM2 2

+ASM1 1

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ . oraenv

ORACLE_SID = [+ASM1] ? orcldb1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb

Instance orcldb1 is running on node rac1-oel77-122

Instance orcldb2 is running on node rac2-oel77-123

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 14:54:59 2020

Version 19.8.0.0.0

Enter password:

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME INSTANCE_NUMBER

---------------- ---------------

orcldb2 2

orcldb1 1

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

[oracle@rac1-oel77-122 ~]$

Apply the 2-Step Method Patch process in Node-1(rac1-oel77-122)

[root@rac1-oel77-122 app]# cd $ORACLE_HOME

[root@rac1-oel77-122 grid]# cd OPatch/

[root@rac1-oel77-122 OPatch]#

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

OPatchauto session is initiated at Mon Oct 26 15:06:44 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_03-06-48PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_03-07-09PM.log

The id for this session is PLA4

Please press ENTER button to accept system generated default clone path.

Please enter clone path [/u01/app/19.7.0/grid_2 ] :

/home/oracle/19.9.0/grid

Please enter clone path [/u01/app/oracle/product/19.3.0/db_2 ] :

/home/oracle/product/19.9.0/db_1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...

Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac1-oel77-122

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac1-oel77-122

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.

Failed to unlock crs clone home for home /u01/app/19.7.0/grid.

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:

Patch Target : rac1-oel77-122->/u01/app/19.7.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.7.0/grid, host: rac1-oel77-122.

Command failed: cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/

Command failure output:

cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Mon Oct 26 15:17:49 2020

Time taken to complete the session 11 minutes, 5 seconds

opatchauto failed with error code 42

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

OPatchauto session is initiated at Mon Oct 26 15:19:34 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_03-19-38PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_03-19-58PM.log

The id for this session is 5DCP