Tuesday, October 27, 2020

Out Of Place (OOP) Patching: Oracle 19c RAC (19.7.0) to Oracle 19c RAC (19.9.0) using 2-Step Method with Rollback Option

Before Out of Place Patching:

Node-1: rac1-oel77-122
Node-2: rac2-oel77-123
GI Home: /u01/app/19.7.0/grid
GI Version:  Oracle 19c (19.7.0)
RDBMS Home: /u01/app/oracle/product/19.3.0/db_1 (Applied Patch 19.8)
RDBMS Version: Oracle 19c (19.8.0)

After Out of Place Patching:

Node-1: rac1-oel77-122
Node-2: rac2-oel77-123
GI Home: /home/oracle/19.9.0/grid
GI Version:  Oracle 19c (19.9.0)
RDBMS Home: /home/oracle/product/19.9.0/db_1
RDBMS Version: Oracle 19c (19.9.0.0.0)

Patching softwares:

Oracle 19c (19.9.0.0.0) - GI - Oct2020
p31750108_190000_Linux-x86-64.zip

Oracle 19c (19.9.0.0.0) - RDBMS - Oct2020
p31771877_190000_Linux-x86-64.zip

Process of 2-Step Method:

1) Prepare Clone is invoked on Node-1 (rac1-oel77-122)
    - New GI Home is prepared on Node 1 ( as per the provided location ) 
    - New RAC Home is prepared on Node 1 ( as per the provided location )
    - Cloning occurs for the respective homes
    - Patching is done on the new homes
    - There is no impact on services running from the original homes on any node 
    
2) Prepare Clone is invoked on Node-2 (rac2-oel77-123) 
    - New GI Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically ) 
    - New RAC Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )
    - Cloning occurs for the respective homes
    - Patching is done on the new homes
    - There is no impact on services running from the original homes on any node

3) Switch Clone is invoked on Node-1 (rac1-oel77-122)
    - Cloned homes are verified 
    - Nodelist is updated in the inventory for new homes
    - Database Service is brought down
    - CRS Services are brought down
    - CRS is started from the Cloned Home 
    - Database is started from Cloned home 
    - Postpatch operation is performed
                            
4) Switch Clone is invoked on Node-2 (rac2-oel77-123)
    - Cloned homes are verified 
    - Nodelist is updated in the inventory for new homes
    - Database Service is brought down
    - CRS Services are brought down
    - CRS is started from the Cloned Home 
    - Database is started from Cloned home 
    - Postpatch operation is performed , Datapatch is invoked 

Overall Steps:

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST
[root@rac2-oel77-123 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST
[root@rac1-oel77-122 OPatch]# ./opatchauto apply -switch-clone
[root@rac2-oel77-123 OPatch]# ./opatchauto apply -switch-clone

Login to Node-1(rac1-oel77-122)

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 14:54:04 2020
Version 19.7.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
+ASM2                          2
+ASM1                          1

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcldb1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb
Instance orcldb1 is running on node rac1-oel77-122
Instance orcldb2 is running on node rac2-oel77-123
[oracle@rac1-oel77-122 ~]$
[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 14:54:59 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Enter password:
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
orcldb2                        2
orcldb1                        1

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
[oracle@rac1-oel77-122 ~]$

Apply the 2-Step Method Patch process in Node-1(rac1-oel77-122)

[root@rac1-oel77-122 app]# cd $ORACLE_HOME
[root@rac1-oel77-122 grid]# cd OPatch/

[root@rac1-oel77-122 OPatch]#
[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

OPatchauto session is initiated at Mon Oct 26 15:06:44 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_03-06-48PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_03-07-09PM.log
The id for this session is PLA4
Please press ENTER button to accept system generated default clone path.
Please enter clone path [/u01/app/19.7.0/grid_2 ] :
/home/oracle/19.9.0/grid
Please enter clone path [/u01/app/oracle/product/19.3.0/db_2 ] :
/home/oracle/product/19.9.0/db_1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...
Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Failed to unlock crs clone home for home /u01/app/19.7.0/grid.

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:
Patch Target : rac1-oel77-122->/u01/app/19.7.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0/grid, host: rac1-oel77-122.
Command failed:  cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/
Command failure output:
cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Mon Oct 26 15:17:49 2020
Time taken to complete the session 11 minutes, 5 seconds

opatchauto failed with error code 42

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

OPatchauto session is initiated at Mon Oct 26 15:19:34 2020
System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_03-19-38PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_03-19-58PM.log
The id for this session is 5DCP
Please press ENTER button to accept system generated default clone path.
Please enter clone path [/u01/app/19.7.0/grid_2 ] :
/home/oracle/19.9.0/grid
Please enter clone path [/u01/app/oracle/product/19.3.0/db_2 ] :
/home/oracle/product/19.9.0/db_1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...
Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Unlocked CRS clone home successfully for home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/19.7.0/grid.
Clone operation successful for oracle home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/oracle/product/19.3.0/db_1.
Clone operation successful for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation for oracle home /u01/app/19.7.0/grid.
Performing post clone operation was successful for oracle home /home/oracle/19.9.0/grid.

Performing post clone operation for oracle home /u01/app/oracle/product/19.3.0/db_1.
Performing post clone operation was successful for oracle home /home/oracle/product/19.9.0/db_1.

Performing prepatch operation on home /home/oracle/product/19.9.0/db_1
Perpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Start applying binary patch on home /home/oracle/19.9.0/grid

Start applying binary patch on home /home/oracle/product/19.9.0/db_1
Binary patch applied successfully on home /home/oracle/19.9.0/grid

Binary patch applied successfully on home /home/oracle/product/19.9.0/db_1

Performing postpatch operation on home /home/oracle/product/19.9.0/db_1
Postpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-oel77-122
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-26PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-26PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-26PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-26PM_1.log

Host:rac1-oel77-122
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-27PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_15-38-27PM_1.log

Out of place patching clone home(s) summary
____________________________________________
Host : rac1-oel77-122
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

Host : rac1-oel77-122
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

OPatchauto session completed at Mon Oct 26 16:11:52 2020
Time taken to complete the session 52 minutes, 18 seconds
[root@rac1-oel77-122 OPatch]#

Login to Node-2(rac2-oel77-123)

[root@rac2-oel77-123 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# cd $ORACLE_HOME
[root@rac2-oel77-123 grid]# cd OPatch/
[root@rac2-oel77-123 OPatch]#

[root@rac2-oel77-123 OPatch]# ./opatchauto apply /home/oracle/31750108/ -prepare-clone -logLevel FINEST

OPatchauto session is initiated at Mon Oct 26 16:12:52 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_04-12-57PM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_04-13-47PM.log
The id for this session is 1L82

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...
Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac2-oel77-123
Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac2-oel77-123
Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Unlocked CRS clone home successfully for home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/19.7.0/grid.
Clone operation successful for oracle home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/oracle/product/19.3.0/db_1.
Clone operation successful for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation for oracle home /u01/app/19.7.0/grid.
Performing post clone operation was successful for oracle home /home/oracle/19.9.0/grid.

Performing post clone operation for oracle home /u01/app/oracle/product/19.3.0/db_1.
Performing post clone operation was successful for oracle home /home/oracle/product/19.9.0/db_1.

Performing prepatch operation on home /home/oracle/product/19.9.0/db_1
Perpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Start applying binary patch on home /home/oracle/19.9.0/grid
Start applying binary patch on home /home/oracle/product/19.9.0/db_1
Binary patch applied successfully on home /home/oracle/19.9.0/grid

Binary patch applied successfully on home /home/oracle/product/19.9.0/db_1

Performing postpatch operation on home /home/oracle/product/19.9.0/db_1
Postpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac2-oel77-123
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Host:rac2-oel77-123
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-26_16-45-22PM_1.log

Out of place patching clone home(s) summary
____________________________________________
Host : rac2-oel77-123
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

Host : rac2-oel77-123
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

OPatchauto session completed at Mon Oct 26 17:28:13 2020
Time taken to complete the session 75 minutes, 22 seconds
[root@rac2-oel77-123 OPatch]#

Login to Node-1(rac1-oel77-122)

[root@rac1-oel77-122 OPatch]# ./opatchauto apply -switch-clone

OPatchauto session is initiated at Mon Oct 26 17:53:29 2020
Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_05-53-31PM.log
The id for this session is WW9L

Verifying clone home details before switch for oracle home /home/oracle/19.9.0/grid.
Verified clone home details successfully for oracle home /home/oracle/19.9.0/grid before switch.

Verifying clone home details before switch for oracle home /home/oracle/product/19.9.0/db_1.
Verified clone home details successfully for oracle home /home/oracle/product/19.9.0/db_1 before switch.

Update nodelist in the inventory for oracle home /home/oracle/19.9.0/grid.
Update nodelist in the inventory is completed for oracle home /home/oracle/19.9.0/grid.

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Starting CRS service on home /home/oracle/19.9.0/grid
CRS service started successfully on home /home/oracle/19.9.0/grid

Starting database service on home /home/oracle/product/19.9.0/db_1
Database service successfully started on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /home/oracle/19.9.0/grid.
All resources have been started successfully from home /home/oracle/19.9.0/grid.

OPatchAuto successful.
--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : rac1-oel77-122
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

Host : rac1-oel77-122
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

OPatchauto session completed at Mon Oct 26 18:01:42 2020
Time taken to complete the session 8 minutes, 13 seconds
[root@rac1-oel77-122 OPatch]#

Login to Node-2(rac2-oel77-123)

[root@rac2-oel77-123 OPatch]# ./opatchauto apply -switch-clone

OPatchauto session is initiated at Mon Oct 26 18:02:03 2020
Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_06-02-07PM.log
The id for this session is 1LLT

Verifying clone home details before switch for oracle home /home/oracle/19.9.0/grid.
Verified clone home details successfully for oracle home /home/oracle/19.9.0/grid before switch.

Verifying clone home details before switch for oracle home /home/oracle/product/19.9.0/db_1.
Verified clone home details successfully for oracle home /home/oracle/product/19.9.0/db_1 before switch.

Update nodelist in the inventory for oracle home /home/oracle/19.9.0/grid.
Update nodelist in the inventory is completed for oracle home /home/oracle/19.9.0/grid.

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Starting CRS service on home /home/oracle/19.9.0/grid
CRS service started successfully on home /home/oracle/19.9.0/grid

Starting database service on home /home/oracle/product/19.9.0/db_1
Database service successfully started on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /home/oracle/19.9.0/grid.
All resources have been started successfully from home /home/oracle/19.9.0/grid.

Modifying  the database(s) to use new location home /home/oracle/product/19.9.0/db_1.
All database(s) modified successfully to run from new location home /home/oracle/product/19.9.0/db_1.

Trying to apply SQL patch on home /home/oracle/product/19.9.0/db_1
SQL patch applied successfully on home /home/oracle/product/19.9.0/db_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : rac2-oel77-123
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

Host : rac2-oel77-123
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

OPatchauto session completed at Mon Oct 26 18:29:00 2020
Time taken to complete the session 26 minutes, 57 seconds
[root@rac2-oel77-123 OPatch]#

While Applying on Node-1(rac1-oel77-122)

[root@rac1-oel77-122 ~]# ps -ef | grep pmon
root      7190 18923  0 17:53 pts/2    00:00:00 grep --color=auto pmon
oracle   11502     1  0 14:29 ?        00:00:00 asm_pmon_+ASM1
oracle   31008     1  0 14:34 ?        00:00:00 ora_pmon_orcldb1
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# ps -ef | grep pmon
root     10426 18923  0 17:55 pts/2    00:00:00 grep --color=auto pmon
oracle   11502     1  0 14:29 ?        00:00:00 asm_pmon_+ASM1
oracle   31008     1  0 14:34 ?        00:00:00 ora_pmon_orcldb1
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# ps -ef | grep pmon
root     11268 18923  0 17:55 pts/2    00:00:00 grep --color=auto pmon
oracle   11502     1  0 14:29 ?        00:00:00 asm_pmon_+ASM1
oracle   31008     1  0 14:34 ?        00:00:00 ora_pmon_orcldb1
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# ps -ef | grep pmon
oracle   11502     1  0 14:29 ?        00:00:00 asm_pmon_+ASM1
root     12174 18923  0 17:56 pts/2    00:00:00 grep --color=auto pmon
[root@rac1-oel77-122 ~]#
[root@rac1-oel77-122 ~]# ps -ef | grep pmon
oracle    3924     1  0 18:01 ?        00:00:00 ora_pmon_orcldb1
root      4889 18923  0 18:01 pts/2    00:00:00 grep --color=auto pmon
oracle   32516     1  0 17:59 ?        00:00:00 asm_pmon_+ASM1
[root@rac1-oel77-122 ~]#

While applying on Node-2(rac2-oel77-123)

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
root     28140 17404  0 18:02 pts/1    00:00:00 grep --color=auto pmon
oracle   28503     1  0 14:33 ?        00:00:00 asm_pmon_+ASM2
oracle   30635     1  0 14:34 ?        00:00:00 ora_pmon_orcldb2
oracle   30976     1  0 14:34 ?        00:00:00 mdb_pmon_-MGMTDB
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
oracle   28503     1  0 14:33 ?        00:00:00 asm_pmon_+ASM2
oracle   30635     1  0 14:34 ?        00:00:00 ora_pmon_orcldb2
oracle   30976     1  0 14:34 ?        00:00:00 mdb_pmon_-MGMTDB
root     31082 17404  0 18:04 pts/1    00:00:00 grep --color=auto pmon
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
oracle   28503     1  0 14:33 ?        00:00:00 asm_pmon_+ASM2
oracle   30976     1  0 14:34 ?        00:00:00 mdb_pmon_-MGMTDB
root     32104 17404  0 18:04 pts/1    00:00:00 grep --color=auto pmon
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
root     17191 17404  0 18:08 pts/1    00:00:00 grep --color=auto pmon
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
oracle   17268     1  0 18:08 ?        00:00:00 asm_pmon_+ASM2
root     18267 17404  0 18:08 pts/1    00:00:00 grep --color=auto pmon
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# ps -ef | grep pmon
oracle   17268     1  0 18:08 ?        00:00:00 asm_pmon_+ASM2
oracle   20045     1  0 18:09 ?        00:00:00 mdb_pmon_-MGMTDB
root     20778 17404  0 18:10 pts/1    00:00:00 grep --color=auto pmon
[root@rac2-oel77-123 ~]#

Finally, check the versions and instances

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 18:56:30 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
+ASM1                          1
+ASM2                          2

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcldb1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 26 18:57:05 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Enter password:
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
orcldb1                        1
orcldb2                        2

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb
Instance orcldb1 is running on node rac1-oel77-122
Instance orcldb2 is running on node rac2-oel77-123
[oracle@rac1-oel77-122 ~]$

[root@rac1-oel77-122 ~]# crsctl stat res -t          
------------------------------------------------------------------------
Name           Target  State        Server           State details
------------------------------------------------------------------------
Local Resources                                      
------------------------------------------------------------------------
ora.LISTENER.lsnr                                    
               ONLINE  ONLINE       rac1-oel77-122   STABLE
               ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.chad                                             
               ONLINE  ONLINE       rac1-oel77-122   STABLE
               ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.net1.network                                     
               ONLINE  ONLINE       rac1-oel77-122   STABLE
               ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.ons                                              
               ONLINE  ONLINE       rac1-oel77-122   STABLE
               ONLINE  ONLINE       rac2-oel77-123   STABLE
------------------------------------------------------------------------
Cluster Resources                                    
------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)               
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   STABLE
      3        ONLINE  OFFLINE                       STABLE
ora.DATADG.dg(ora.asmgroup)                          
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   STABLE
      3        OFFLINE OFFLINE                       STABLE
ora.LISTENER_SCAN1.lsnr                              
      1        ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.LISTENER_SCAN2.lsnr                              
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
ora.LISTENER_SCAN3.lsnr                              
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
ora.MGMTLSNR                                         
      1        ONLINE  ONLINE       rac2-oel77-123   169.254.10.240 10.1.4.123,STABLE
ora.OCRVD.dg(ora.asmgroup)                           
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   STABLE
      3        OFFLINE OFFLINE                       STABLE
ora.RECODG.dg(ora.asmgroup)                          
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   STABLE
      3        OFFLINE OFFLINE                       STABLE
ora.asm(ora.asmgroup)                                
      1        ONLINE  ONLINE       rac1-oel77-122   Started,STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   Started,STABLE
      3        OFFLINE OFFLINE                       STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)                 
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   STABLE
      3        OFFLINE OFFLINE                       STABLE
ora.cvu                                              
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
ora.mgmtdb                                           
      1        ONLINE  ONLINE       rac2-oel77-123   Open,STABLE
ora.orcldb.db                                        
      1        ONLINE  ONLINE       rac1-oel77-122   Open,HOME=/home/oracle/product/19.9.0/db_1,STABLE
      2        ONLINE  ONLINE       rac2-oel77-123   Open,HOME=/home/oracle/product/19.9.0/db_1,STABLE
ora.qosmserver                                       
      1        ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.rac1-oel77-122.vip                               
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
ora.rac2-oel77-123.vip                               
      1        ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.scan1.vip                                        
      1        ONLINE  ONLINE       rac2-oel77-123   STABLE
ora.scan2.vip                                        
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
ora.scan3.vip                                        
      1        ONLINE  ONLINE       rac1-oel77-122   STABLE
------------------------------------------------------------------------
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2-oel77-123'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.chad' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.orcldb.db' on 'rac1-oel77-122'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.chad' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.orcldb.db' on 'rac1-oel77-122' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.rac1-oel77-122.vip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.asm' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.orcldb.db' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cvu' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.rac1-oel77-122.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.orcldb.db' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.chad' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac1-oel77-122' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac1-oel77-122' succeeded.
CRS-2677: Stop of 'ora.mgmtdb' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac2-oel77-123'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.asm' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac2-oel77-123' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac2-oel77-123' succeeded.
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.rac2-oel77-123.vip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.rac2-oel77-123.vip' on 'rac2-oel77-123' succeeded
CRS-2675: Stop of 'ora.chad' on 'rac2-oel77-123' failed
CRS-2679: Attempting to clean 'ora.chad' on 'rac2-oel77-123'
CRS-2681: Clean of 'ora.chad' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.ons' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.net1.network' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ons' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.net1.network' on 'rac1-oel77-122' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1-oel77-122' has completed
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2-oel77-123' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2-oel77-123'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.storage' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.crsd' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.storage' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ctssd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2-oel77-123' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2-oel77-123'
CRS-2677: Stop of 'ora.cssd' on 'rac2-oel77-123' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1-oel77-122'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.ctssd' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1-oel77-122' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1-oel77-122' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1-oel77-122'
CRS-2677: Stop of 'ora.cssd' on 'rac1-oel77-122' succeeded
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.evmd' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.evmd' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2-oel77-123'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.diskmon' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.diskmon' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.cssd' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1-oel77-122'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.ctssd' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac2-oel77-123' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.asm' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.asm' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.storage' on 'rac1-oel77-122' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1-oel77-122'
CRS-2676: Start of 'ora.crsd' on 'rac1-oel77-122' succeeded
CRS-2676: Start of 'ora.storage' on 'rac2-oel77-123' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2-oel77-123'
CRS-2676: Start of 'ora.crsd' on 'rac2-oel77-123' succeeded
[root@rac1-oel77-122 ~]#

Option for Rollback

[root@rac1-oel77-122 OPatch]# ./opatchauto rollback -switch-clone

OPatchauto session is initiated at Mon Oct 26 19:37:17 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_07-37-21PM.log.

OPATCHAUTO-72134: OPatchauto execution from the current Grid home is not allowed.
OPATCHAUTO-72134: Multiple Grid homes are identified in the current host and current home is not active grid home.
OPATCHAUTO-72134: Please execute from the active grid home.
OPatchauto session completed at Mon Oct 26 19:37:22 2020
Time taken to complete the session 0 minute, 5 seconds
Topology creation failed.
[root@rac1-oel77-122 OPatch]#

[root@rac1-oel77-122 OPatch]# cd $ORACLE_HOME
[root@rac1-oel77-122 grid]# pwd
/u01/app/19.7.0/grid
[root@rac1-oel77-122 grid]#

[root@rac1-oel77-122 grid]# . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[root@rac1-oel77-122 grid]#
[root@rac1-oel77-122 grid]# pwd
/u01/app/19.7.0/grid
[root@rac1-oel77-122 grid]#

[root@rac1-oel77-122 grid]# cd /home/oracle/19.9.0/grid
[root@rac1-oel77-122 grid]# cd OPatch/
[root@rac1-oel77-122 OPatch]# pwd
/home/oracle/19.9.0/grid/OPatch
[root@rac1-oel77-122 OPatch]#

[root@rac1-oel77-122 OPatch]# ./opatchauto rollback -switch-clone

OPatchauto session is initiated at Mon Oct 26 19:38:36 2020

System initialization log file is /home/oracle/19.9.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_07-38-39PM.log.

Session log file is /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_07-39-41PM.log
The id for this session is A5ZP

Updating datapatch libraries on home /u01/app/oracle/product/19.3.0/db_1
Successfully updated datapatch libraries on home /u01/app/oracle/product/19.3.0/db_1

Update nodelist in the inventory for oracle home /u01/app/19.7.0/grid.
Update nodelist in the inventory is completed for oracle home /u01/app/19.7.0/grid.

Bringing down database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/db_1

Bringing down database service on home /u01/app/oracle/product/19.3.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/db_1

Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid

Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid

Starting database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/db_1

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /u01/app/19.7.0/grid.
All resources have been started successfully from home /u01/app/19.7.0/grid.

OPatchAuto successful.
--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : rac1-oel77-122
Actual Home : /home/oracle/product/19.9.0/db_1
Version:19.0.0.0.0
Clone Home Path : /u01/app/oracle/product/19.3.0/db_1

Host : rac1-oel77-122
Actual Home : /home/oracle/19.9.0/grid
Version:19.0.0.0.0
Clone Home Path : /u01/app/19.7.0/grid

OPatchauto session completed at Mon Oct 26 19:49:05 2020
Time taken to complete the session 10 minutes, 29 seconds
[root@rac1-oel77-122 OPatch]#

[root@rac1-oel77-122 OPatch]# ps -ef | grep pmon
oracle   10187     1  0 19:47 ?        00:00:00 asm_pmon_+ASM1
oracle   13520     1  0 19:48 ?        00:00:00 ora_pmon_orcldb1
root     16075 24809  0 19:49 pts/1    00:00:00 grep --color=auto pmon
[root@rac1-oel77-122 OPatch]#

Login to Node-2 (rac2-oel77-123)

[root@rac2-oel77-123 ~]# . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /u01/app/oracle
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# cd /home/oracle/19.9.0/grid
[root@rac2-oel77-123 grid]#

[root@rac2-oel77-123 grid]# cd OPatch/
[root@rac2-oel77-123 OPatch]#

[root@rac2-oel77-123 OPatch]# ./opatchauto rollback -switch-clone

OPatchauto session is initiated at Mon Oct 26 19:51:32 2020

System initialization log file is /home/oracle/19.9.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-26_07-51-39PM.log.

Session log file is /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-26_07-52-39PM.log
The id for this session is 2D14

Updating datapatch libraries on home /u01/app/oracle/product/19.3.0/db_1
Successfully updated datapatch libraries on home /u01/app/oracle/product/19.3.0/db_1

Update nodelist in the inventory for oracle home /u01/app/19.7.0/grid.
Update nodelist in the inventory is completed for oracle home /u01/app/19.7.0/grid.

Bringing down database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/db_1

Bringing down database service on home /u01/app/oracle/product/19.3.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/db_1

Bringing down CRS service on home /u01/app/19.7.0/grid
CRS service brought down successfully on home /u01/app/19.7.0/grid

Starting CRS service on home /u01/app/19.7.0/grid
CRS service started successfully on home /u01/app/19.7.0/grid

Starting database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/db_1

Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /u01/app/19.7.0/grid.
All resources have been started successfully from home /u01/app/19.7.0/grid.

Modifying  the database(s) to use new location home /u01/app/oracle/product/19.3.0/db_1.
All database(s) modified successfully to run from new location home /u01/app/oracle/product/19.3.0/db_1.

Trying to roll back SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch rolled back successfully on home /u01/app/oracle/product/19.3.0/db_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : rac2-oel77-123
Actual Home : /home/oracle/19.9.0/grid
Version:19.0.0.0.0
Clone Home Path : /u01/app/19.7.0/grid

Host : rac2-oel77-123
Actual Home : /home/oracle/product/19.9.0/db_1
Version:19.0.0.0.0
Clone Home Path : /u01/app/oracle/product/19.3.0/db_1

OPatchauto session completed at Mon Oct 26 20:45:02 2020
Time taken to complete the session 53 minutes, 31 seconds
[root@rac2-oel77-123 OPatch]#

Hope it helps.....With this we have seen two methods of Out Of Place Patching (OOP).

Sunday, October 25, 2020

Out Of Place (OOP) Patching: Oracle 19c RAC (19.8.0) to Oracle 19c RAC (19.9.0) using Single-Session Method

Out of Place Patching

Starting Oracle 12c (12.2.0.1.0), there is a new feature introduced with the latest version of opatchauto.

You can perform out of place ( OOP ) Patching.
  • This is helpful when the current Oracle Homes do not have sufficient space left and there is a need to shift the Oracle Homes to a new location .
  • This operation generally requires cloning of the Oracle Homes followed by Patching and then switching the CRS and the Database , including all the services to newly cloned Homes respectively .
  • All these operations can be performed at a single stretch by making use of OOP Patching .
Before Out of Place Patching:

Node-1: rac1-oel77-122
Node-2: rac2-oel77-123
GI Home: /u01/app/19.7.0/grid
GI Version:  Oracle 19c (19.8.0.0.0)
RDBMS Home: /u01/app/oracle/product/19.3.0/db_1
RDBMS Version: Oracle 19c (19.8.0.0.0)

After Out of Place Patching:

Node-1: rac1-oel77-122
Node-2: rac2-oel77-123
GI Home: /home/oracle/19.9.0/grid
GI Version:  Oracle 19c (19.9.0.0.0)
RDBMS Home: /home/oracle/product/19.9.0/db_1
RDBMS Version: Oracle 19c (19.9.0.0.0)

Patching softwares:

Oracle 19c (19.9.0.0.0) - GI - Oct2020
p31750108_190000_Linux-x86-64.zip

Oracle 19c (19.9.0.0.0) - RDBMS - Oct2020
p31771877_190000_Linux-x86-64.zip

Single-Session Method:

1) OOP Patching is invoked on Node-1 (rac1-oel77-122)
    - MGMT Database is shifted to Node (rac2-oel77-123)
    - New GI Home is prepared on Node 1 ( as per the provided location )
    - New RAC Home is prepared on Node 1 ( as per the provided location )
    - Cloning occurs for the respective homes
    - Patching is done on the new homes
    - DB goes down on Node 1 (rac1-oel77-122)
    - ASM goes down on Node 1 (rac1-oel77-122)
    - Switch occurs from old homes to the new homes
    - ASM comes up from the New GI Home on Node 1 (rac1-oel77-122)
    - MGMT Database is shifted back to Node 1 (rac1-oel77-122)
    - Database comes up from the new RAC Home on Node 1 (rac1-oel77-122)
    - There is no impact on Node 2 , all the services are running as it is

2) OOP Patching is invoked on Node 2 (rac2-oel77-123)
    - New GI Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )
    - New RAC Home is prepared on Node 2 ( No need to provide location, it will be picked up automatically )
    - Cloning occurs for the respective homes
    - Patching is done on the new homes
    - DB goes down on Node 2 (rac2-oel77-123)
    - ASM goes down on Node 2 (rac2-oel77-123)
    - Switch occurs from old homes to the new homes
    - ASM comes up from the New GI Home on Node 1 (rac1-oel77-122)
    - Database comes up from the new RAC Home on Node 1 (rac1-oel77-122)
    - Datapatch is called for post patching steps on Database as this is the last Node
    - There is no impact on Node 1 all the services are running from the new Cloned + Patched Home

Login to Node-1 (rac1-oel77-122) and unzip the files and copy the files to Node-2 (rac2-oel77-123)

[oracle@rac1-oel77-122 ~]$ scp p317* oracle@rac2-oel77-123:/home/oracle/
p31750108_190000_Linux-x86-64.zip         100% 2724MB  76.1MB/s   00:35
p31771877_190000_Linux-x86-64.zip         100% 1359MB  83.5MB/s   00:16
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ unzip p31750108_190000_Linux-x86-64.zip
Archive:  p31750108_190000_Linux-x86-64.zip
   creating: 31750108/
   creating: 31750108/31773437/
  inflating: 31750108/31773437/README.txt
   creating: 31750108/31773437/etc/

[oracle@rac1-oel77-122 ~]$ unzip p31771877_190000_Linux-x86-64.zip
Archive:  p31771877_190000_Linux-x86-64.zip
   creating: 31771877/
  inflating: 31771877/README.html
   creating: 31771877/etc/
   creating: 31771877/etc/config/

rac2-oel77-123:

[oracle@rac2-oel77-123 ~]$ unzip p31750108_190000_Linux-x86-64.zip
Archive:  p31750108_190000_Linux-x86-64.zip
   creating: 31750108/

[oracle@rac2-oel77-123 ~]$ unzip p31771877_190000_Linux-x86-64.zip
Archive:  p31771877_190000_Linux-x86-64.zip
   creating: 31771877/

Checking the instance status from Node-1

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [orcldb1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb
Instance orcldb1 is running on node rac1-oel77-122
Instance orcldb2 is running on node rac2-oel77-123
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 02:17:23 2020
Version 19.8.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Enter password:

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
orcldb1                        1
orcldb2                        2

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ su - root
Password:
[root@rac1-oel77-122 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# echo $ORACLE_HOME
/u01/app/19.7.0/grid
[root@rac1-oel77-122 ~]#

[root@rac1-oel77-122 ~]# cd $ORACLE_HOME
[root@rac1-oel77-122 grid]# cd OPatch/

Login as 'root' user and start OOP in Node-1 (rac1-oel77-122)

[root@rac1-oel77-122 OPatch]# ./opatchauto apply /home/oracle/31750108/ -outofplace

OPatchauto session is initiated at Sat Oct 24 02:27:42 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-24_02-27-43AM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_02-28-02AM.log
The id for this session is B77Y
Please press ENTER button to accept system generated default clone path.
Please enter clone path [/u01/app/oracle/product/19.3.0/db_2 ] :
/home/oracle/product/19.9.0/db_1
Please enter clone path [/u01/app/19.7.0/grid_2 ] :
/home/oracle/19.9.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...
Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac1-oel77-122
Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Failed to unlock crs clone home for home /u01/app/19.7.0/grid.

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:
Patch Target : rac1-oel77-122->/u01/app/19.7.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0/grid, host: rac1-oel77-122.
Command failed:  cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/
Command failure output:
cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sat Oct 24 02:41:24 2020
Time taken to complete the session 13 minutes, 42 seconds

opatchauto failed with error code 42

Note: Permissions error for the user 'oracle' and you can resume the patch after corrections.

[root@rac1-oel77-122 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sat Oct 24 03:20:36 2020
Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_03-20-36AM.log
Resuming existing session with id B77Y

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Unlocked CRS clone home successfully for home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/19.7.0/grid.
Clone operation successful for oracle home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/oracle/product/19.3.0/db_1.
Clone operation successful for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation for oracle home /u01/app/19.7.0/grid.
Performing post clone operation was successful for oracle home /home/oracle/19.9.0/grid.

Performing post clone operation for oracle home /u01/app/oracle/product/19.3.0/db_1.
Performing post clone operation was successful for oracle home /home/oracle/product/19.9.0/db_1.

Performing prepatch operation on home /home/oracle/product/19.9.0/db_1
Perpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Start applying binary patch on home /home/oracle/19.9.0/grid

Start applying binary patch on home /home/oracle/product/19.9.0/db_1
Binary patch applied successfully on home /home/oracle/19.9.0/grid

Binary patch applied successfully on home /home/oracle/product/19.9.0/db_1

Update nodelist in the inventory for oracle home /home/oracle/19.9.0/grid.
Update nodelist in the inventory is completed for oracle home /home/oracle/19.9.0/grid.

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Performing postpatch operation on home /home/oracle/product/19.9.0/db_1
Postpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Starting CRS service on home /home/oracle/19.9.0/grid
CRS service started successfully on home /home/oracle/19.9.0/grid

Starting database service on home /home/oracle/product/19.9.0/db_1
Database service successfully started on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /home/oracle/19.9.0/grid.
All resources have been started successfully from home /home/oracle/19.9.0/grid.

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac1-oel77-122
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31773437
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31780966
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Host:rac1-oel77-122
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_03-24-36AM_1.log

Out of place patching clone home(s) summary
____________________________________________
Host : rac1-oel77-122
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

Host : rac1-oel77-122
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

OPatchauto session completed at Sat Oct 24 04:03:45 2020
Time taken to complete the session 43 minutes, 10 seconds
[root@rac1-oel77-122 OPatch]#

Checking instances while patching, MGMTDB relocated to Node-2 and once OOP is done ASM, DB instances are UP.

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon
oracle   10393     1  0 Oct23 ?        00:00:00 asm_pmon_+ASM1
oracle   12805     1  0 Oct23 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   13419     1  0 Oct23 ?        00:00:01 ora_pmon_orcldb1
oracle   16525 13143  0 03:27 pts/1    00:00:00 grep --color=auto pmon
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon
oracle   10393     1  0 Oct23 ?        00:00:00 asm_pmon_+ASM1
oracle   14529 13143  0 03:57 pts/1    00:00:00 grep --color=auto pmon
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon
oracle   25255 13143  0 03:59 pts/1    00:00:00 grep --color=auto pmon
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon
oracle     379 13143  0 04:02 pts/1    00:00:00 grep --color=auto pmon
oracle   29798     1  0 04:00 ?        00:00:00 asm_pmon_+ASM1
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ ps -ef | grep pmon
oracle     471     1  0 04:03 ?        00:00:00 ora_pmon_orcldb1
oracle    2719 13143  0 04:04 pts/1    00:00:00 grep --color=auto pmon
oracle   29798     1  0 04:00 ?        00:00:00 asm_pmon_+ASM1
[oracle@rac1-oel77-122 ~]$


Login as 'root' user and start OOP in Node-2 (rac2-oel77-123)

Note-1: We no need to specify new home locations for GI and RDBMS. It will take automatically

[root@rac2-oel77-123 ~]# . oraenv
ORACLE_SID = [+ASM1] ? +ASM2
ORACLE_HOME = [/home/oracle] ? /u01/app/19.7.0/grid
The Oracle base has been set to /u01/app/oracle
[root@rac2-oel77-123 ~]#

[root@rac2-oel77-123 ~]# cd $ORACLE_HOME
[root@rac2-oel77-123 grid]# cd OPatch/
[root@rac2-oel77-123 OPatch]#

[root@rac2-oel77-123 OPatch]# ./opatchauto apply /home/oracle/31750108/ -outofplace

OPatchauto session is initiated at Sat Oct 24 04:11:10 2020

System initialization log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-24_04-11-13AM.log.

Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_04-11-34AM.log
The id for this session is 4ZFI

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.7.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1

Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location. Please wait...

Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location. Please wait...
Clone of oracle home /u01/app/19.7.0/grid is /home/oracle/19.9.0/grid on host rac2-oel77-123
Copying the files from the existing oracle home /u01/app/19.7.0/grid to a new location is successful.

Clone of oracle home /u01/app/oracle/product/19.3.0/db_1 is /home/oracle/product/19.9.0/db_1 on host rac2-oel77-123
Copying the files from the existing oracle home /u01/app/oracle/product/19.3.0/db_1 to a new location is successful.

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Failed to unlock crs clone home for home /u01/app/19.7.0/grid.

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:
Patch Target : rac2-oel77-123->/u01/app/19.7.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0/grid, host: rac2-oel77-123.
Command failed:  cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/
Command failure output:
cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sat Oct 24 04:24:46 2020
Time taken to complete the session 13 minutes, 36 seconds

 opatchauto failed with error code 42
[root@rac2-oel77-123 OPatch]#

At the log

Unlocking CRS clone home for home /u01/app/19.7.0/grid.'}
2020-10-24 15:25:43,715 INFO  [67] com.oracle.glcm.patch.auto.db.framework.sdk.patchplanner.SystemModelUtils - Base Class: class dbmodel.common.OracleHome PropertyName: owner Value: oracle
2020-10-24 15:25:43,728 INFO  [67] com.oracle.glcm.patch.auto.db.product.cmdtranslator.commands.RootCrsCommand - rootPlPath: ROOT_CRS_PL_PATH
2020-10-24 15:25:43,730 INFO  [67] com.oracle.glcm.patch.auto.db.framework.sdk.patchplanner.SystemModelUtils - Base Class: class dbmodel.common.OracleHome PropertyName: owner Value: oracle
2020-10-24 15:25:43,730 INFO  [67] com.oracle.glcm.patch.auto.db.integration.controller.action.CRSShutDownStartupAction - Updated root crs location is /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/bootstrap_rac2-oel77-123/patchwork/crs/install
2020-10-24 15:25:43,731 INFO  [67] com.oracle.glcm.patch.auto.db.integration.controller.action.CRSShutDownStartupAction - Updated xag file location is /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/bootstrap_rac2-oel77-123/patchwork/xag
2020-10-24 15:25:43,733 INFO  [67] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
Executing command as oracle:
 cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/
2020-10-24 15:25:43,733 INFO  [67] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: cp -Rf /u01/app/19.7.0/grid/opatchautocfg/db/dbtmp/* /home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/
2020-10-24 15:25:43,762 INFO  [67] com.oracle.glcm.patch.auto.db.product.executor.GISystemCall - Is retry required=false
2020-10-24 15:25:43,765 INFO  [67] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 1
2020-10-24 15:25:43,765 INFO  [67] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
COMMAND EXECUTION FAILURE :

ERROR:
cp: target '/home/oracle/19.9.0/grid/opatchautocfg/db/dbtmp/' is not a directory

[root@rac2-oel77-123 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sat Oct 24 15:29:35 2020
Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_03-29-35PM.log
Resuming existing session with id 4ZFI

Unlocking CRS clone home for home /u01/app/19.7.0/grid.
Unlocked CRS clone home successfully for home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/19.7.0/grid.
Clone operation successful for oracle home /u01/app/19.7.0/grid.

Creating clone for oracle home /u01/app/oracle/product/19.3.0/db_1.
Clone operation successful for oracle home /u01/app/oracle/product/19.3.0/db_1.

Performing post clone operation for oracle home /u01/app/19.7.0/grid.
Performing post clone operation was successful for oracle home /home/oracle/19.9.0/grid.

Performing post clone operation for oracle home /u01/app/oracle/product/19.3.0/db_1.
Performing post clone operation was successful for oracle home /home/oracle/product/19.9.0/db_1.

Performing prepatch operation on home /home/oracle/product/19.9.0/db_1
Perpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Start applying binary patch on home /home/oracle/19.9.0/grid

Start applying binary patch on home /home/oracle/product/19.9.0/db_1
Binary patch applied successfully on home /home/oracle/19.9.0/grid

Failed while applying binary patches on home /home/oracle/product/19.9.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : rac2-oel77-123->/home/oracle/product/19.9.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /home/oracle/product/19.9.0/db_1, host: rac2-oel77-123.
Command failed:  /home/oracle/product/19.9.0/db_1/OPatch/opatchauto  apply /home/oracle/31750108/ -oh /home/oracle/product/19.9.0/db_1 -target_type rac_database -binary -invPtrLoc /u01/app/19.7.0/grid/oraInst.loc -jre /u01/app/19.7.0/grid/OPatch/jre -ignorejdkvalidation -persistresult /u01/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_rac2-oel77-123_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_rac2-oel77-123_rac_2.ser
Command failure output:
==Following patches FAILED in apply:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_15-35-11PM_1.log
Reason: Failed during Analysis: CheckSystemSpace Failed, [ Prerequisite Status: FAILED, Prerequisite output:
The details are:
Required amount of space(5657.764MB) is not available.]

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sat Oct 24 15:56:24 2020
Time taken to complete the session 26 minutes, 50 seconds

 opatchauto failed with error code 42
[root@rac2-oel77-123 OPatch]# 

Ha Ha Ha...OOP is testing me. Space issues in Node-2. Removed GI and RDBMS Zipped Patch files and resume patch again. It's going..

[root@rac2-oel77-123 oracle]# rm -rf p31750108_190000_Linux-x86-64.zip
[root@rac2-oel77-123 oracle]# rm -rf p31771877_190000_Linux-x86-64.zip

[root@rac2-oel77-123 oracle]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                              4.8G     0  4.8G   0% /dev
tmpfs                                 4.8G  893M  4.0G  19% /dev/shm
tmpfs                                 4.8G  122M  4.7G   3% /run
tmpfs                                 4.8G     0  4.8G   0% /sys/fs/cgroup
/dev/mapper/ol_rac1--oel77--122-root   50G   34G   17G  68% /
/dev/mapper/ol_rac1--oel77--122-home   45G   36G  9.0G  80% /home
/dev/sda1                            1014M  170M  845M  17% /boot
tmpfs                                 983M   16K  983M   1% /run/user/42
tmpfs                                 983M  4.0K  983M   1% /run/user/54321
[root@rac2-oel77-123 oracle]#

[root@rac2-oel77-123 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sat Oct 24 15:59:32 2020
Session log file is /u01/app/19.7.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-24_03-59-34PM.log
Resuming existing session with id 4ZFI

Start applying binary patch on home /home/oracle/product/19.9.0/db_1
Binary patch applied successfully on home /home/oracle/product/19.9.0/db_1

Update nodelist in the inventory for oracle home /home/oracle/19.9.0/grid.
Update nodelist in the inventory is completed for oracle home /home/oracle/19.9.0/grid.

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Bringing down database service on home /home/oracle/product/19.9.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: orcldb
Database service successfully brought down on home /home/oracle/product/19.9.0/db_1

Performing postpatch operation on home /home/oracle/product/19.9.0/db_1
Postpatch operation completed successfully on home /home/oracle/product/19.9.0/db_1

Starting CRS service on home /home/oracle/19.9.0/grid
CRS service started successfully on home /home/oracle/19.9.0/grid

Starting database service on home /home/oracle/product/19.9.0/db_1
Database service successfully started on home /home/oracle/product/19.9.0/db_1

Preparing home /home/oracle/product/19.9.0/db_1 after database service restarted
No step execution required.........

Confirm that all resources have been started from home /home/oracle/19.9.0/grid.
All resources have been started successfully from home /home/oracle/19.9.0/grid.

Modifying  the database(s) to use new location home /home/oracle/product/19.9.0/db_1.
All database(s) modified successfully to run from new location home /home/oracle/product/19.9.0/db_1.

Trying to apply SQL patch on home /home/oracle/product/19.9.0/db_1
SQL patch applied successfully on home /home/oracle/product/19.9.0/db_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:rac2-oel77-123
CRS Home:/u01/app/19.7.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_15-35-10PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_15-35-10PM_1.log

Patch: /home/oracle/31750108/31773437
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_15-35-10PM_1.log

Patch: /home/oracle/31750108/31780966
Log: /home/oracle/19.9.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_15-35-10PM_1.log

Host:rac2-oel77-123
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/31750108/31771877
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_16-00-00PM_1.log

Patch: /home/oracle/31750108/31772784
Log: /home/oracle/product/19.9.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-24_16-00-00PM_1.log

Out of place patching clone home(s) summary
____________________________________________
Host : rac2-oel77-123
Actual Home : /u01/app/19.7.0/grid
Version:19.0.0.0.0
Clone Home Path : /home/oracle/19.9.0/grid

Host : rac2-oel77-123
Actual Home : /u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Clone Home Path : /home/oracle/product/19.9.0/db_1

OPatchauto session completed at Sat Oct 24 16:41:14 2020
Time taken to complete the session 41 minutes, 43 seconds
[root@rac2-oel77-123 OPatch]#

[oracle@rac2-oel77-123 ~]$ ps -ef | grep pmon
oracle   10471     1  0 16:19 ?        00:00:00 asm_pmon_+ASM2
oracle   11621     1  0 16:21 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   23757     1  0 16:30 ?        00:00:00 ora_pmon_orcldb2
oracle   29218 22979  0 16:47 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac2-oel77-123 ~]$

Check the instances, GI and RDBMS versions

[oracle@rac1-oel77-122 ~]$ srvctl status database -d orcldb
Instance orcldb1 is running on node rac1-oel77-122
Instance orcldb2 is running on node rac2-oel77-123
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 16:49:46 2020
Version 19.9.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
+ASM1                          1
+ASM2                          2

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcldb1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-oel77-122 ~]$

[oracle@rac1-oel77-122 ~]$ sqlplus sys@orcldb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 16:50:21 2020
Version 19.9.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
orcldb1                        1
orcldb2                        2

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
[oracle@rac1-oel77-122 ~]$

Hope it helps....If anyone tested already in 19c please share if you got the same permission error for the user 'oracle'.