Reflection of my thoughts and technical expriences: Out Of Place (OOP) Patching: Oracle 19c RAC (19.8.0) to Oracle 19c RAC (19.9.0) using Single-Session Method

Out of Place Patching

Starting Oracle 12c (12.2.0.1.0), there is a new feature introduced with the latest version of opatchauto.

You can perform out of place ( OOP ) Patching.

This is helpful when the current Oracle Homes do not have sufficient space left and there is a need to shift the Oracle Homes to a new location .
This operation generally requires cloning of the Oracle Homes followed by Patching and then switching the CRS and the Database , including all the services to newly cloned Homes respectively .
All these operations can be performed at a single stretch by making use of OOP Patching .

Before Out of Place Patching:

Node-1: rac1-oel77-122

Node-2: rac2-oel77-123

GI Home: /u01/app/19.7.0/grid

GI Version: Oracle 19c (19.8.0.0.0)

RDBMS Home: /u01/app/oracle/product/19.3.0/db_1

RDBMS Version: Oracle 19c (19.8.0.0.0)

After Out of Place Patching:

Node-1: rac1-oel77-122

Node-2: rac2-oel77-123

GI Home: /home/oracle/19.9.0/grid

GI Version: Oracle 19c (19.9.0.0.0)

RDBMS Home: /home/oracle/product/19.9.0/db_1

RDBMS Version: Oracle 19c (19.9.0.0.0)

Patching softwares:

Oracle 19c (19.9.0.0.0) - GI - Oct2020

p31750108_190000_Linux-x86-64.zip

Oracle 19c (19.9.0.0.0) - RDBMS - Oct2020

p31771877_190000_Linux-x86-64.zip

Single-Session Method:

1) OOP Patching is invoked on Node-1 (rac1-oel77-122)

- MGMT Database is shifted to Node (rac2-oel77-123)

- New GI Home is prepared on Node 1 ( as per the provided location )

- New RAC Home is prepared on Node 1 ( as per the provided location )

- Cloning occurs for the respective homes

- Patching is done on the new homes

- DB goes down on Node 1 (rac1-oel77-122)

- ASM goes down on Node 1 (rac1-oel77-122)

- Switch occurs from old homes to the new homes

- ASM comes up from the New GI Home on Node 1 (rac1-oel77-122)

- MGMT Database is shifted back to Node 1 (rac1-oel77-122)

- Database comes up from the new RAC Home on Node 1 (rac1-oel77-122)

- There is no impact on Node 2 , all the services are running as it is

2) OOP Patching is invoked on Node 2 (rac2-oel77-123)

- New GI Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )

- New RAC Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )

- Cloning occurs for the respective homes

- Patching is done on the new homes

- DB goes down on Node 2 (rac2-oel77-123)

- ASM goes down on Node 2 (rac2-oel77-123)

- Switch occurs from old homes to the new homes

- ASM comes up from the New GI Home on Node 1 (rac1-oel77-122)

- Database comes up from the new RAC Home on Node 1 (rac1-oel77-122)

- Datapatch is called for post patching steps on Database as this is the last Node

- There is no impact on Node 1 all the services are running from the new Cloned + Patched Home

Login to Node-1 (rac1-oel77-122) and unzip the files and copy the files to Node-2 (rac2-oel77-123)

[oracle@rac1-oel77-122 ~]$ scp p317* oracle@rac2-oel77-123:/home/oracle/

p31750108_190000_Linux-x86-64.zip 100% 2724MB 76.1MB/s 00:35

p31771877_190000_Linux-x86-64.zip 100% 1359MB 83.5MB/s 00:16

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ unzip p31750108_190000_Linux-x86-64.zip

Archive: p31750108_190000_Linux-x86-64.zip

creating: 31750108/

creating: 31750108/31773437/

inflating: 31750108/31773437/README.txt

creating: 31750108/31773437/etc/

[oracle@rac1-oel77-122 ~]$ unzip p31771877_190000_Linux-x86-64.zip

Archive: p31771877_190000_Linux-x86-64.zip

creating: 31771877/

inflating: 31771877/README.html

creating: 31771877/etc/

creating: 31771877/etc/config/

rac2-oel77-123:

[oracle@rac2-oel77-123 ~]$ unzip p31750108_190000_Linux-x86-64.zip

Archive: p31750108_190000_Linux-x86-64.zip

creating: 31750108/

[oracle@rac2-oel77-123 ~]$ unzip p31771877_190000_Linux-x86-64.zip

Archive: p31771877_190000_Linux-x86-64.zip

creating: 31771877/

Checking the instance status from Node-1

[oracle@rac1-oel77-122 ~]$ . oraenv

ORACLE_SID = [orcldb1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb

Instance orcldb1 is running on node rac1-oel77-122

Instance orcldb2 is running on node rac2-oel77-123

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 02:17:23 2020

Version 19.8.0.0.0

Enter password:

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME INSTANCE_NUMBER

---------------- ---------------

orcldb1 1

orcldb2 2

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.8.0.0.0

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ su - root

Password:

[root@rac1-oel77-122 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# echo $ORACLE_HOME

/u01/app/19.7.0/grid

[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# cd $ORACLE_HOME

[root@rac1-oel77-122 grid]# cd OPatch/

Login as 'root' user and start OOP in Node-1 (rac1-oel77-122)

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -outofplace

OPatchauto session is initiated at Sat Oct 24 02:27:42 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-24_02-27-43AM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_02-28-02AM.log

The id for this session is B77Y

Please press ENTER button to accept system generated default clone path.

Please enter clone path [/u01/app/oracle/product/19.3.0/db_2 ] :

/home/oracle/product/19.9.0/db_1

Please enter clone path [/u01/app/19.7.0/grid_2 ] :

/home/oracle/19.9.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1

Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...

Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac1-oel77-122

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac1-oel77-122

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.

Failed to unlock crs clone home for home /u01/app/19.7.0/grid.

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:

Patch Target : rac1-oel77-122->/u01/app/19.7.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.7.0/grid, host: rac1-oel77-122.

Command failed: cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/

Command failure output:

cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sat Oct 24 02:41:24 2020

Time taken to complete the session 13 minutes, 42 seconds

opatchauto failed with error code 42

Note: Permissions error for the user 'oracle' and you can resume the patch after corrections.

[root@rac1-oel77-122 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sat Oct 24 03:20:36 2020

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_03-20-36AM.log

Resuming existing session with id B77Y

Unlocking CRS clone home for home /u01/app/19.7.0/grid.

Unlocked CRS clone home successfully for home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/19.7.0/grid.

Clone operation successful for oracle home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/oracle/product/19.3.0/db_1.

Clone operation successful for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation for oracle home /u01/app/19.7.0/grid.

Performing post clone operation was successful for oracle home /home/oracle/19.9.0/grid.

Performing post clone operation for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation was successful for oracle home /home/oracle/product/19.9.0/db_1.

Performing prepatch operation on home /home/oracle/product/19.9.0/db_1

Perpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Start applying binary patch on home /home/oracle/19.9.0/grid

Start applying binary patch on home /home/oracle/product/19.9.0/db_1

Binary patch applied successfully on home /home/oracle/19.9.0/grid

Binary patch applied successfully on home /home/oracle/product/19.9.0/db_1

Update nodelist in the inventory for oracle home /home/oracle/19.9.0/grid.

Update nodelist in the inventory is completed for oracle home /home/oracle/19.9.0/grid.

Bringing down database service on home /home/oracle/product/19.9.0/db_1

Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Bringing down database service on home /home/oracle/product/19.9.0/db_1

Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb

Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Performing postpatch operation on home /home/oracle/product/19.9.0/db_1

Postpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Starting CRS service on home /home/oracle/19.9.0/grid

CRS service started successfully on home /home/oracle/19.9.0/grid

Starting database service on home /home/oracle/product/19.9.0/db_1

Database service successfully started on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted

No step execution required.........

Confirm that all resources have been started from home /home/oracle/19.9.0/grid.

All resources have been started successfully from home /home/oracle/19.9.0/grid.

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-oel77-122

CRS Home:/u01/app/19.7.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877

Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31772784

Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31773437

Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31780966

Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Host:rac1-oel77-122

RAC Home:/u01/app/oracle/product/19.3.0/db_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877

Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31772784

Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Out of place patching clone home(s) summary

____________________________________________

Host : rac1-oel77-122

Actual Home : /u01/app/19.7.0/grid

Version:19.0.0.0.0

Clone Home Path : /home/oracle/19.9.0/grid

Host : rac1-oel77-122

Actual Home : /u01/app/oracle/product/19.3.0/db_1

Version:19.0.0.0.0

Clone Home Path : /home/oracle/product/19.9.0/db_1

OPatchauto session completed at Sat Oct 24 04:03:45 2020

Time taken to complete the session 43 minutes, 10 seconds

[root@rac1-oel77-122 OPatch]#

Checking instances while patching, MGMTDB relocated to Node-2 and once OOP is done ASM, DB instances are UP.

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon

oracle 10393 1 0 Oct23 ? 00:00:00 asm_pmon_+ASM1

oracle 12805 1 0 Oct23 ? 00:00:00 mdb_pmon_-MGMTDB

oracle 13419 1 0 Oct23 ? 00:00:01 ora_pmon_orcldb1

oracle 16525 13143 0 03:27 pts/1 00:00:00 grep --color=auto pmon

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon

oracle 10393 1 0 Oct23 ? 00:00:00 asm_pmon_+ASM1

oracle 14529 13143 0 03:57 pts/1 00:00:00 grep --color=auto pmon

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon

oracle 25255 13143 0 03:59 pts/1 00:00:00 grep --color=auto pmon

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon

oracle 379 13143 0 04:02 pts/1 00:00:00 grep --color=auto pmon

oracle 29798 1 0 04:00 ? 00:00:00 asm_pmon_+ASM1

[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon

oracle 471 1 0 04:03 ? 00:00:00 ora_pmon_orcldb1

oracle 2719 13143 0 04:04 pts/1 00:00:00 grep --color=auto pmon

oracle 29798 1 0 04:00 ? 00:00:00 asm_pmon_+ASM1

[oracle@rac1-oel77-122 ~]$

Login as 'root' user and start OOP in Node-2 (rac2-oel77-123)

Note-1: We no need to specify new home locations for GI and RDBMS. It will take automatically

[root@rac2-oel77-123 ~]# . oraenv

ORACLE_SID = [+ASM1] ? +ASM2

ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid

The Oracle base has been set to /u01/app/oracle

[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# cd $ORACLE_HOME

[root@rac2-oel77-123 grid]# cd OPatch/

[root@rac2-oel77-123 OPatch]#

[root@rac2-oel77-123 OPatch]# ./opatchauto apply /home/oracle/31750108/ -outofplace

OPatchauto session is initiated at Sat Oct 24 04:11:10 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-24_04-11-13AM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_04-11-34AM.log

The id for this session is 4ZFI