Reflection of my thoughts and technical expriences: July 2022

<pre>

Note: I have shown here high level steps for applying Grid RU and RDBMS RU. Please follow "readme" before applying RUs based on your environment with all pre-checks.

Environment:
------------------
Primary RAC: Oracle 19c (19.12) with two node RAC
Physical Standby RAC: Oracle 19c (19.12) with two node RAC

Required Softwares:
--------------------------
Latest Opatch: p6880880_190000_Linux-x86-64.zip
GI Release Update (July 2022): p34130714_190000_Linux-x86-64.zip
Database Release Update (July 2022): p34133642_190000_Linux-x86-64.zip

Check the status of cluster instances in Physical Standby RAC:

---------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 12:19:27 2022

Version 19.12.0.0.0

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> select name,open_mode,database_role from gv$database;

NAME OPEN_MODE DATABASE_ROLE

--------- -------------------- ----------------

ORCL READ ONLY WITH APPLY PHYSICAL STANDBY

[oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [orcl1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sat Jul 30 14:36:51 2022

Version 19.12.0.0.0

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/oracle

Connected to "orcl"

Connected as SYSDBA.

DGMGRL>

DGMGRL> show configuration;

Configuration - dg_19c

Protection Mode: MaxPerformance

Members:

orcl - Primary database

orcls - Physical standby database

Fast-Start Failover: Disabled

Configuration Status:

SUCCESS (status updated 2 seconds ago)

DGMGRL>

DGMGRL> edit database orcls set state='APPLY-OFF';

Succeeded.

DGMGRL>

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl status database -d orcls

Instance orcls1 is running on node oel70-base3

Instance orcls2 is running on node oel70-base4

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 14:35:37 2022

Version 19.12.0.0.0

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE",

d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME

from GV$DATABASE d, gv$instance i

where i.INST_ID=d.INST_ID;

DATABASE_HOST DB_NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- --------------------

oel70-base3 orcls1 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4 orcls2 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE",

d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME

from GV$DATABASE d, gv$instance i

where i.INST_ID=d.INST_ID; 2

DATABASE_HOST DB_NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- -------------------

oel70-base3 orcls1 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4 orcls2 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> select inst_id,process, status, thread#, sequence#, block#, blocks from gv$managed_standby where process='MRP0';

select a.thread#, (select max (sequence#) from v$archived_log where archived='YES' and thread#=a.thread#) archived,max(a.sequence#) applied,

(select max(sequence#) from v$archived_log where archived='YES' and thread#=a.thread#)-max(a.sequence#)gap

from v$archived_log a where a.applied='YES' group by a.thread# order by thread#;

INST_ID PROCESS STATUS THREAD# SEQUENCE# BLOCK# BLOCKS

---------- --------- ------------ ---------- ---------- ---------- ----------

2 MRP0 APPLYING_LOG 2 48 26627 409600

SQL> 2 3

THREAD# ARCHIVED APPLIED GAP

---------- ---------- ---------- ----------

1 57 57 0

2 47 47 0

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

[oracle@oel70-base3 ~]$

Unzip required patches with OPatch in both the nodes:
----------------------------------------------------------------------

Standby node-1

[oracle@oel70-base3 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/product/19.0.0/dbhome_1/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive: p34133642_190000_Linux-x86-64.zip

Standby node-2

[oracle@oel70-base4 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34130714_190000_Linux-x86-64.zip
Archive: p34130714_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive: p34133642_190000_Linux-x86-64.zip

Applying Grid RU and RDBMS RU in Physical Standby Node-1:

-----------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ su - root

Password:

Last login: Sat Jul 30 14:33:21 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108396

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle 5187 1 0 01:29 ? 00:00:01 asm_pmon_+ASM1

oracle 8157 1 0 01:33 ? 00:00:02 ora_pmon_orcls1

root 24502 24146 0 14:45 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 14:46:45 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_02-46-51PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_02-47-56PM.log

The id for this session is IM3L

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_prepatch_apply_inplace_oel70-base3_2022-07-30_02-51-57PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Failed while applying binary patches on home /u01/app/19.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : oel70-base3->/u01/app/19.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.0.0/grid, host: oel70-base3.

Command failed: /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/ -oh /u01/app/19.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc -jre /u01/app/19.0.0/grid/OPatch/jre -persistresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_oel70-base3_crs_1.ser -analyzedresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_oel70-base3_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sat Jul 30 15:00:17 2022

Time taken to complete the session 13 minutes, 32 seconds

opatchauto failed with error code 42

[root@oel70-base3 ~]#

Got the error and checking the opatch logile

----------------------------------------------------------

[oracle@oel70-base3 ~]$ cat /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO] ENTERING METHOD: NApply(patches,options)

[Jul 30, 2022 2:59:09 PM] [INFO] Checking if the Oracle Home /u01/app/19.0.0/grid is good for OPatch to work.

[Jul 30, 2022 2:59:09 PM] [INFO] Option unsupported:false and isCASEnabled :false

[Jul 30, 2022 2:59:09 PM] [INFO] OPatch invoked as follows: 'napply -phBaseFile /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/SDKWork.txt -local '

[Jul 30, 2022 2:59:09 PM] [INFO] Oracle Home : /u01/app/19.0.0/grid

Central Inventory : /u01/app/oraInventory

from : /u01/app/19.0.0/grid/oraInst.loc

OPatch version : 12.2.0.1.32

OUI version : 12.2.0.7.0

OUI location : /u01/app/19.0.0/grid/oui

Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO] Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch_history.txt

[Jul 30, 2022 2:59:11 PM] [INFO] [OPSR-TIME] Loading raw inventory

[Jul 30, 2022 2:59:11 PM] [INFO] [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 133 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 135 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-TIME] Raw inventory loaded successfully

[Jul 30, 2022 2:59:12 PM] [INFO] NApply::no CAS enabled, OPatch runs with legacy process.

[Jul 30, 2022 2:59:12 PM] [INFO] Verifying environment and performing prerequisite checks...

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-TIME] Running prerequisite checks

[Jul 30, 2022 2:59:12 PM] [INFO] opatch-external.jar is in /u01/app/19.0.0/grid/OPatch/jlib/opatch-external.jar

[Jul 30, 2022 2:59:19 PM] [INFO] [OPSR-TIME] Loading cooked inventory

[Jul 30, 2022 2:59:19 PM] [INFO] [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 107 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO] [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 138 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO] [OPSR-TIME] Cooked inventory loaded successfully

[Jul 30, 2022 2:59:45 PM] [INFO] Checking if Oracle Home has components required by patches...

[Jul 30, 2022 2:59:47 PM] [INFO] CheckMissingComps: Cached file does not exist or is invalid, re-build prereq result.

[Jul 30, 2022 2:59:47 PM] [INFO] Checking conflict among patches...

[Jul 30, 2022 2:59:47 PM] [INFO] Running prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches can be applied: 33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches are not required:

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches are auto rollbackable:

[Jul 30, 2022 2:59:48 PM] [INFO] Finished checking prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO] Conflict checking amongst patches PASSED

[Jul 30, 2022 2:59:48 PM] [INFO] Checking conflicts against Oracle Home...

[Jul 30, 2022 2:59:50 PM] [INFO] Running prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:50 PM] [INFO] CUP_LOG: Found pi CUP 34133642 is a superset of poh CUP: 32904851

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches can be applied: 33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches are not required:

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches are auto rollbackable:

[Jul 30, 2022 2:59:53 PM] [INFO] Finished checking prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:56 PM] [INFO] OCM validation is skipped for SDK mode.

[Jul 30, 2022 2:59:58 PM] [INFO] Execution of 'sh /home/oracle/34130714/34133642/custom/scripts/init -apply 34133642 ':

[Jul 30, 2022 2:59:58 PM] [INFO] Return Code = 0

[Jul 30, 2022 2:59:58 PM] [INFO] Execution of ' Command and arguments of init script is restricted to be viewed in log file ':

Return Code = 0

[Jul 30, 2022 3:00:01 PM] [INFO] Running prerequisite checks...

[Jul 30, 2022 3:00:04 PM] [INFO] Space Needed : 10231.319MB

[Jul 30, 2022 3:00:04 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 33575402

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34133642

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34139601

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34160635

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed on patch :34318175

[Jul 30, 2022 3:00:11 PM] [WARNING] Action file /u01/app/19.0.0/grid/jlib/srvmasm.jar is in the jar list,OOP should be lanched

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Files in use by a process: /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Files in use by a process: /u01/app/19.0.0/grid/lib/libasmclntsh19.so PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:14 PM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:16 PM] [INFO] UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [INFO] Finishing UtilSession at Sat Jul 30 15:00:16 CDT 2022

[Jul 30, 2022 3:00:16 PM] [INFO] Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1: 6136m

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep 6136

oracle 6136 1 0 01:30 ? 00:00:40 /u01/app/19.0.0/grid/jdk/bin/java -server -Xms128M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/u01/app/oracle/crsdata/oel70-base3/qos/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=oel70-base3 -Doracle.http.port=8888 -Doracle.jmx.port=8895 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=false -Djava.security.manager -Djava.security.policy=/u01/app/oracle/crsdata/oel70-base3/qos/conf/catalina.policy -Djava.security.egd=file:/dev/urandom -XX:ParallelGCThreads=2 -Dcatalina.home=/u01/app/19.0.0/grid/tomcat -Dcatalina.base=/u01/app/oracle/crsdata/oel70-base3/qos -Djava.io.tmpdir=/u01/app/oracle/crsdata/oel70-base3/qos/temp -Doracle.home=/u01/app/19.0.0/grid -Doracle.jwc.mode=CLUSTERWARE -classpath /u01/app/19.0.0/grid/jlib/cryptoj.jar:/u01/app/19.0.0/grid/jlib/oraclepki.jar:/u01/app/19.0.0/grid/jlib/osdt_core.jar:/u01/app/19.0.0/grid/jlib/osdt_cert.jar:/u01/app/19.0.0/grid/tomcat/lib/tomcat-juli.jar:/u01/app/19.0.0/grid/tomcat/lib/bootstrap.jar:/u01/app/19.0.0/grid/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

oracle 23827 21496 0 15:13 pts/1 00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ kill -9 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep 6136

oracle 24100 21496 0 15:13 pts/1 00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

After killing process and resuming the patch:

----------------------------------------------------------

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto resume

OPatchauto session is initiated at Sat Jul 30 15:33:25 2022

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-33-25PM.log

Resuming existing session with id IM3L

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_postpatch_apply_inplace_oel70-base3_2022-07-30_03-39-19PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base3

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Host:oel70-base3

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patching session reported following warning(s):

______________________________________

[WARNING] The database instance 'cdbc1' from '/u01/app/oracle/product/19.0.0/dbhome_1', in host'oel70-base3' is not running. SQL changes, if any, will not be applied.

To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).

Refer to the readme to get the correct steps for applying the sql changes.

OPatchauto session completed at Sat Jul 30 15:46:34 2022

Time taken to complete the session 13 minutes, 10 seconds

[root@oel70-base3 ~]#

Check the process:

------------------------

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle 14834 1 0 15:42 ? 00:00:00 asm_pmon_+ASM1

oracle 15294 1 0 15:43 ? 00:00:00 ora_pmon_orcls1

root 22124 24146 0 15:46 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

Applying Grid RU and RDBMS RU in Physical Standby Node-2:

-----------------------------------------------------------------------------------

[oracle@oel70-base4 ~]$ su - root

Password:

Last login: Sat Jul 30 15:49:07 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108437

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# . oraenv

ORACLE_SID = [root] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# ps -ef | grep pmon

oracle 7280 1 0 01:32 ? 00:00:01 asm_pmon_+ASM2

oracle 7788 1 0 01:33 ? 00:00:02 ora_pmon_orcls2

root 28280 27450 0 15:50 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 15:51:30 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_03-51-35PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-52-40PM.log

The id for this session is 8SKS