Applying the first Monthly Recommended Patches (MRP) for Oracle 19c (19.17) with Physical Standby Databases

Environment:
------------------------------------------------------------------------------
Primary Database: orcldb
ORACLE_HOME: /u01/app/oracle/product/19.3.0.0/dbhome_1
GSM_HOME: /u01/app/oracle/product/19.3.0.0/gsmhome_1
AGENT_HOME: /u01/app/oracle/agent/agent_13.5.0.0.0
Database Name: orcldb

Physical Standby Database-1: orcldbp
ORACLE_HOME: /u01/app/oracle/product/19.3.0.0/dbhome_1 
GSM_HOME: /u01/app/oracle/product/19.3.0.0/gsmhome_1
AGENT_HOME: /u01/app/oracle/agent/agent_13.5.0.0.0
Database Name: orcldbp

Physical Standby Database-2: orcldbs
ORACLE_HOME: /u01/app/oracle/product/19.3.0.0/dbhome_1 
GSM_HOME: /u01/app/oracle/product/19.3.0.0/gsmhome_1
AGENT_HOME: /u01/app/oracle/agent/agent_13.5.0.0.0
Database Name: orcldbs

Note-1: The above databases are Oracle 19c with applied 19.17 RU.
Note-2: You must use the OPatch utility version 12.2.0.1.32 or later to apply this patch, and ensure to satisfy the following prerequisites prior to installation.

Please refer for more details: Primary Note for Database Proactive Patch Program (Doc ID 888.1)

Monthly Recommended Patch (MRP) for Oracle 19c (19.17)
------------------------------------------------------------------------------
Date Published: November 15, 2022
Version: 19.17.0.0.221115 - DATABASE MRP
Patch Number: Patch 34737974
Name: DATABASE MRP 19.17.0.0.221115
Prerequisite: Oracle Database RU 19.17 

What is in the MRP1 for Oracle Database 19c (19.17)
----------------------------------------------------------------------
The following bugs are fixed by this patch:
33896423, 34333986, 30691454, 34538232, 34574048, 34366627, 34724125

Ensure that you verify the Oracle Inventory because OPatch accesses it to install the patches. To verify the inventory, run the following command.

Login to Physical Standby Database-1
-------------------------------------------------
[oracle@stbyh01 ~]$ cd $ORACLE_HOME/OPatch

[oracle@stbyh01 OPatch]$ ./opatch lsinventory -detail -oh /u01/app/oracle/product/19.3.0.0/dbhome_1
Oracle Interim Patch Installer version 12.2.0.1.33
Copyright (c) 2022, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/product/19.3.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.33
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatch/opatch2022-11-17_12-21-48PM_1.log

(Output truncated for better visibility).

Maintain a location (UNZIPPED_PATCH_LOCATION) for storing the contents of the patch ZIP file. 
Extract the contents of the patch ZIP file by running the following command:

[oracle@stbyh01 ~]$ unzip p34737974_1917000DBRU_Linux-x86-64.zip
Archive:  p34737974_1917000DBRU_Linux-x86-64.zip
   creating: 34737974/
   creating: 34737974/34366627/
   creating: 34737974/34366627/etc_online/
   creating: 34737974/34366627/etc_online/config/
  inflating: 34737974/34366627/etc_online/config/inventory.xml
  inflating: 34737974/34366627/etc_online/config/actions.xml
   creating: 34737974/34366627/files/
   creating: 34737974/34366627/files/lib/
   creating: 34737974/34366627/files/lib/libgeneric19.a/
  inflating: 34737974/34366627/files/lib/libgeneric19.a/kgnfs.o

(Output truncated for better visibility).

Determine whether any currently installed interim patches conflict with MRP patch by executing the opatch command shown below

[oracle@stbyh01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -analyze -online
Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1
OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-23-40_binary.log
Patch selected: /home/oracle/34737974

Analysing this list of patches :

[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-23-43PM_1.log
opatchauto SUCCEEDED.

[oracle@stbyh01 OPatch]$

How do you apply an MRP on Oracle Database 19c (19.17)
----------------------------------------------------------------------------
Before apply, check the status of databases in dgmgrl and and also stop all the dependencies

DGMGRL> show configuration lag;
Configuration - orcldbcfg
  Protection Mode: MaxAvailability
  Members:
  orcldb  - Primary database
    orcldbp - (*) Physical standby database
              Transport Lag:      0 seconds (computed 0 seconds ago)
              Apply Lag:          0 seconds (computed 0 seconds ago)
      orcldbs - Physical standby database (receiving current redo)
                Transport Lag:      0 seconds (computed 1 second ago)
                Apply Lag:          0 seconds (computed 1 second ago)
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
SUCCESS   (status updated 36 seconds ago)

DGMGRL> edit database orcldbp set state='APPLY-OFF';
Succeeded.
DGMGRL>

DGMGRL> edit database orcldbs set state='APPLY-OFF';
Succeeded.
DGMGRL>

Stop the OEM 13c agent 
-------------------------------
[oracle@stbyh01 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
[oracle@stbyh01 bin]$

In my environment Oracle Global Data Services (GDS) are configured
------------------------------------------------------------------------------------------
[oracle@stbyh01 ~]$ ./gsm_setup.sh
GDSCTL: Version 19.0.0.0.0 - Production on Thu Nov 17 12:06:24 CST 2022
Copyright (c) 2011, 2019, Oracle.  All rights reserved.
Welcome to GDSCTL, type "help" for information.
Current GSM is set to GSMNORTH
GDSCTL> stop gsm -gsm GSMNORTH
GSM is stopped successfully
GDSCTL> exit

Shut down all instances and listeners associated with the Oracle home that you are updating
---------------------------------------------------------------------------------------------------------------------
[oracle@stbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbp] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ sqlplus sys/welcome@orcldbp as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 12:15:04 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbp] ? catgdsdr
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ sqlplus sys/welcome@catgdsdr as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 12:18:51 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@stbyh01 ~]$
[oracle@stbyh01 ~]$ lsnrctl stop
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 17-NOV-2022 12:15:46
Copyright (c) 1991, 2022, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=stbyh01.localdomain)(PORT=1521)))
The command completed successfully
[oracle@stbyh01 ~]$

Run the OPatchauto utility by entering the following command:
-----------------------------------------------------------------------------------
[oracle@stbyh01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -oh /u01/app/oracle/product/19.3.0.0/dbhome_1

Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1
OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-31-31_binary.log
Patch selected: /home/oracle/34737974

Analysing this list of patches :
[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

Applying the patches ...

Patches successfully applied.

==Following patches were SUCCESSFULLY applied:
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-31-34PM_1.log
opatchauto SUCCEEDED.
[oracle@stbyh01 OPatch]$

How does the inventory tell you that you’ve applied an MRP
------------------------------------------------------------------------------
[oracle@stbyh01 OPatch]$ ./opatch lsinventory | grep -i "Patch Description"
Patch description:  "Fix for bug 34724125"
Patch description:  "POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB"
Patch description:  "AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE."
Patch description:  "DNFS IO HANG  DURING STRESS TEST"
Patch description:  "AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1"
Patch description:  "FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT"
Patch description:  "SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG"
Patch description:  "Database Release Update : 19.17.0.0.221018 (34419443)"
Patch description:  "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
[oracle@stbyh01 OPatch]$

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Note: At this point, you will have no indication that an MRP has been applied.

These are the applied using MRP 

Patch description:  "Fix for bug 34724125"
Patch description:  "POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB"
Patch description:  "AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE."
Patch description:  "DNFS IO HANG  DURING STRESS TEST"
Patch description:  "AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1"
Patch description:  "FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT"
Patch description:  "SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG"
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[oracle@stbyh01 OPatch]$ ./opatch lspatches
34724125;Fix for bug 34724125
34574048;POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB
34538232;AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE.
34366627;DNFS IO HANG  DURING STRESS TEST
34333986;AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1
33896423;FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT
30691454;SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@stbyh01 OPatch]$

Start all instances and listeners associated with the Oracle home
-----------------------------------------------------------------------------------
[oracle@stbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbp] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ sqlplus sys/welcome as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 16:07:29 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORACLE instance started.
Total System Global Area 3053449816 bytes
Fixed Size                  8901208 bytes
Variable Size             637534208 bytes
Database Buffers         2399141888 bytes
Redo Buffers                7872512 bytes
Database mounted.
Database opened.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbp] ? catgdsdr
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ sqlplus sys/welcome as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 16:08:03 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORACLE instance started.
Total System Global Area 3053449816 bytes
Fixed Size                  8901208 bytes
Variable Size             620756992 bytes
Database Buffers         2415919104 bytes
Redo Buffers                7872512 bytes
Database mounted.
Database opened.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@stbyh01 ~]$

[oracle@stbyh01 ~]$ ps -ef | grep pmon
oracle    9725     1  0 16:07 ?        00:00:00 ora_pmon_orcldbp
oracle    9953     1  0 16:08 ?        00:00:00 ora_pmon_catgdsdr
oracle   12397  3775  0 16:46 pts/0    00:00:00 grep --color=auto pmon
[oracle@stbyh01 ~]$

Login to Physical Standby Database-2
-------------------------------------------------
Execute the following steps:

1. [oracle@cstbyh01 OPatch]$ ./opatch lsinventory -detail -oh /u01/app/oracle/product/19.3.0.0/dbhome_1

2. [oracle@cstbyh01 ~]$ unzip p34737974_1917000DBRU_Linux-x86-64.zip

Determine whether any currently installed interim patches conflict with MRP patch by executing the opatch command shown below

[oracle@cstbyh01 ~]$ cd $ORACLE_HOME/OPatch
[oracle@cstbyh01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -analyze -online

Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1
OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-37-05_binary.log
Patch selected: /home/oracle/34737974

Analysing this list of patches :

[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-37-08PM_1.log
opatchauto SUCCEEDED.
[oracle@cstbyh01 OPatch]$

Stop the OEM 13c agent 
-----------------------------------
[oracle@cstbyh01 ~]$ cd /u01/app/oracle/agent/agent_13.5.0.0.0/bin/

[oracle@cstbyh01 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
[oracle@cstbyh01 bin]$

In my environment Oracle Global Data Services (GDS) are configured
------------------------------------------------------------------------------------------
[oracle@cstbyh01 ~]$ ./gsm_setup.sh
GDSCTL: Version 19.0.0.0.0 - Production on Tue Nov 15 18:34:14 CST 2022
Copyright (c) 2011, 2019, Oracle.  All rights reserved.
Welcome to GDSCTL, type "help" for information.
Current GSM is set to GSMSOUTH

GDSCTL> stop gsm -gsm GSMSOUTH
GSM is stopped successfully

GDSCTL> exit

Shut down all instances and listeners associated with the Oracle home that you are updating
----------------------------------------------------------------------------------------------------------------------
[oracle@cstbyh01 ~]$ sqlplus sys/welcome@orcldbs as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 12:15:25 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@cstbyh01 ~]$

[oracle@cstbyh01 ~]$ lsnrctl stop
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 17-NOV-2022 12:15:59
Copyright (c) 1991, 2022, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cstbyh01.localdomain)(PORT=1521)))
The command completed successfully
[oracle@cstbyh01 ~]$

Run the OPatchauto utility by entering the following command:
-----------------------------------------------------------------------------------
[oracle@cstbyh01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -oh /u01/app/oracle/product/19.3.0.0/dbhome_1

Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1
OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-39-44_binary.log
Patch selected: /home/oracle/34737974
Analysing this list of patches :
[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

Applying the patches ...

Patches successfully applied.
==Following patches were SUCCESSFULLY applied:
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-39-47PM_1.log
opatchauto SUCCEEDED.

How does the inventory tell you that you’ve applied an MRP
-------------------------------------------------------------------------------
[oracle@cstbyh01 OPatch]$ ./opatch lsinventory | grep -i "Patch Description"

Patch description:  "Fix for bug 34724125"
Patch description:  "POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB"
Patch description:  "AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE."
Patch description:  "DNFS IO HANG  DURING STRESS TEST"
Patch description:  "AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1"
Patch description:  "FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT"
Patch description:  "SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG"
Patch description:  "Database Release Update : 19.17.0.0.221018 (34419443)"
Patch description:  "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
[oracle@cstbyh01 OPatch]$

[oracle@cstbyh01 OPatch]$ ./opatch lspatches
34724125;Fix for bug 34724125
34574048;POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB
34538232;AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE.
34366627;DNFS IO HANG  DURING STRESS TEST
34333986;AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1
33896423;FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT
30691454;SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@cstbyh01 OPatch]$

Start all instances and listeners associated with the Oracle home
-----------------------------------------------------------------------------------
[oracle@cstbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbs] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@cstbyh01 ~]$

[oracle@cstbyh01 ~]$ sqlplus sys/welcome as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 16:07:46 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORACLE instance started.
Total System Global Area 3053449816 bytes
Fixed Size                  8901208 bytes
Variable Size             637534208 bytes
Database Buffers         2399141888 bytes
Redo Buffers                7872512 bytes
Database mounted.
Database opened.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

[oracle@cstbyh01 ~]$ . oraenv
ORACLE_SID = [orcldbs] ?
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@cstbyh01 ~]$ lsnrctl start
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 17-NOV-2022 16:09:27
Copyright (c) 1991, 2022, Oracle.  All rights reserved.
Starting /u01/app/oracle/product/19.3.0.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /u01/app/oracle/product/19.3.0.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/cstbyh01/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cstbyh01.localdomain)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cstbyh01.localdomain)(PORT=1521)))

STATUS of the LISTENER
----------------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                17-NOV-2022 16:09:27
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/19.3.0.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/cstbyh01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cstbyh01.localdomain)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "orcldbs" has 1 instance(s).
  Instance "orcldbs", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@cstbyh01 ~]$

Applying MRP in Primary Database:
-------------------------------------------------
Execute the following steps:

1. [oracle@prim01 OPatch]$ ./opatch lsinventory -detail -oh /u01/app/oracle/product/19.3.0.0/dbhome_1

2. [oracle@prim01 ~]$ unzip p34737974_1917000DBRU_Linux-x86-64.zip

Determine whether any currently installed interim patches conflict with MRP patch by executing the opatch command shown below

[oracle@prim01 ~]$ cd $ORACLE_HOME/OPatch

[oracle@prim01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -analyze -online
Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1

OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-41-39_binary.log

Patch selected: /home/oracle/34737974

Analysing this list of patches :
[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-41-42PM_1.log
opatchauto SUCCEEDED.
[oracle@prim01 OPatch]$

Shutting down Primary Database and GDS Catalog Database

----------------------------------------------------------------------

[oracle@prim01 ~]$ . oraenv

ORACLE_SID = [orcldb] ?
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@prim01 ~]$ sqlplus sys/welcome@orcldb as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 12:17:10 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@prim01 ~]$

[oracle@prim01 ~]$ sqlplus sys/welcome@catgds as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 12:18:17 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

[oracle@prim01 ~]$ . oraenv
ORACLE_SID = [orcldb] ?
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@prim01 ~]$ lsnrctl stop
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 17-NOV-2022 12:40:38
Copyright (c) 1991, 2022, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=prim01.localdomain)(PORT=1521)))
The command completed successfully
[oracle@prim01 ~]$

Stop the OEM 13c agent
---------------------------------
[oracle@prim01 ~]$ cd /u01/app/oracle/agent/agent_13.5.0.0.0/bin/
[oracle@prim01 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ...
 stopped.
[oracle@prim01 bin]$

Stop the GSM in Primary Database
----------------------------------------------
[oracle@prim01 ~]$ ./gsm_setup.sh
GDSCTL: Version 19.0.0.0.0 - Production on Thu Nov 17 12:06:06 CST 2022
Copyright (c) 2011, 2019, Oracle.  All rights reserved.
Welcome to GDSCTL, type "help" for information.
Current GSM is set to GSMEAST
GDSCTL> stop gsm -gsm GSMEAST
GSM is stopped successfully
GDSCTL> exit
[oracle@prim01 ~]$

Run the OPatchauto utility by entering the following command:
------------------------------------------------------------------------------------
[oracle@prim01 OPatch]$ ./opatchauto apply -binary /home/oracle/34737974/ -oh /u01/app/oracle/product/19.3.0.0/dbhome_1

Oracle Home : /u01/app/oracle/product/19.3.0.0/dbhome_1
OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 13.9.5.0.0
Running from       : /u01/app/oracle/product/19.3.0.0/dbhome_1
opatchauto log file: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2022-11-17_12-42-23_binary.log

Patch selected: /home/oracle/34737974

Analysing this list of patches :
[/home/oracle/34737974/33896423, /home/oracle/34737974/34333986, /home/oracle/34737974/30691454, /home/oracle/34737974/34538232, /home/oracle/34737974/34574048, /home/oracle/34737974/34366627, /home/oracle/34737974/34724125] ...

Analysis completed.

Applying the patches ...

Patches successfully applied.

==Following patches were SUCCESSFULLY applied:
Patch: /home/oracle/34737974/30691454
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/33896423
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/34333986
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/34366627
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/34538232
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/34574048
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
Patch: /home/oracle/34737974/34724125
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-11-17_12-42-27PM_1.log
opatchauto SUCCEEDED.

[oracle@prim01 OPatch]$ ./opatch lsinventory | grep -i "Patch Description"
Patch description:  "Fix for bug 34724125"
Patch description:  "POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB"
Patch description:  "AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE."
Patch description:  "DNFS IO HANG  DURING STRESS TEST"
Patch description:  "AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1"
Patch description:  "FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT"
Patch description:  "SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG"
Patch description:  "Database Release Update : 19.17.0.0.221018 (34419443)"
Patch description:  "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
[oracle@prim01 OPatch]$

[oracle@prim01 OPatch]$ ./opatch lspatches
34724125;Fix for bug 34724125
34574048;POD EEHO-DEV5 UNABLE TO SWITCHOVER ODS DB
34538232;AFTER GI PATCHING AND DB UPGRADE TO 19.16 SESSIONS ARE CONSUMING TEMPTABLESPACE.
34366627;DNFS IO HANG  DURING STRESS TEST
34333986;AIM ORA-600 [KTUSCV1 CV BUF TOO BIG] - KTUSCV1
33896423;FLUSH OUT STALE ANTILOCKS AND CONVERT KCLCLS_2 AND  KCLANTILOCK_17 TO SOFT ASSERT
30691454;SYD  E1POD DBHOME PATCHING COMPLETELY HUNG WITH KPDBHASHTABLE_FIND MULTIPLE INSTANCE HANG
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@prim01 OPatch]$

Starting the Primary Database and GDS Catalog Database

----------------------------------------------------------------------

[oracle@prim01 ~]$ . oraenv

ORACLE_SID = [orcldb] ?
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@prim01 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 15:52:57 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORACLE instance started.
Total System Global Area 3053449816 bytes
Fixed Size                  8901208 bytes
Variable Size             654311424 bytes
Database Buffers         2382364672 bytes
Redo Buffers                7872512 bytes
Database mounted.
Database opened.
SQL>

[oracle@prim01 ~]$ lsnrctl start
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 17-NOV-2022 15:53:45
Copyright (c) 1991, 2022, Oracle.  All rights reserved.
Starting /u01/app/oracle/product/19.3.0.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /u01/app/oracle/product/19.3.0.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/prim01/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=prim01.localdomain)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=prim01.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                17-NOV-2022 15:53:45
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/19.3.0.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/prim01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=prim01.localdomain)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "orcldb" has 1 instance(s).
  Instance "orcldb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@prim01 ~]$

[oracle@prim01 ~]$ . oraenv
ORACLE_SID = [orcldb] ? catgds
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@prim01 ~]$

[oracle@prim01 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 17 15:54:19 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup;
ORACLE instance started.
Total System Global Area 3053449816 bytes
Fixed Size                  8901208 bytes
Variable Size             704643072 bytes
Database Buffers         2332033024 bytes
Redo Buffers                7872512 bytes
Database mounted.
Database opened.

SQL> exit
[oracle@prim01 ~]$ . oraenv
ORACLE_SID = [orcldb] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@prim01 ~]$

Login to Data Guard Broker

---------------------------------------

[oracle@prim01 ~]$ dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Nov 17 16:08:19 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/welcome
Connected to "orcldb"
Connected as SYSDBA.
DGMGRL>

DGMGRL> show configuration;
Configuration - orcldbcfg
  Protection Mode: MaxAvailability
  Members:
  orcldb  - Primary database
    Warning: ORA-16824: multiple warnings, including fast-start failover-related warnings, detected for the database
    orcldbp - (*) Physical standby database
      orcldbs - Physical standby database (receiving current redo)
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
WARNING   (status updated 43 seconds ago)

DGMGRL> edit database orcldbp set state='APPLY-ON';
Succeeded.

DGMGRL> edit database orcldbs set state='APPLY-ON';
Succeeded.

DGMGRL> show configuration lag;
Configuration - orcldbcfg
  Protection Mode: MaxAvailability
  Members:
  orcldb  - Primary database
    orcldbp - (*) Physical standby database
              Transport Lag:      0 seconds (computed 0 seconds ago)
              Apply Lag:          0 seconds (computed 0 seconds ago)
      orcldbs - Physical standby database (receiving current redo)
                Transport Lag:      0 seconds (computed 1 second ago)
                Apply Lag:          0 seconds (computed 1 second ago)
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
SUCCESS   (status updated 36 seconds ago)
DGMGRL>

Started the Observer in other window in Background Mode
----------------------------------------------------------------------------
[oracle@prim01 ~]$ . oraenv
ORACLE_SID = [orcldb] ?
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@prim01 ~]$ dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Nov 17 16:10:49 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/welcome
Connected to "orcldb"
Connected as SYSDBA.
DGMGRL>

DGMGRL> start observer orcldbcfg_obs in BACKGROUND file is '/u01/orcldbcfg_obs.dat' logfile is '/u01/orcldbcfg_obs.log' connect identifier is orcldb;

Connected to "orcldb"
Submitted command "START OBSERVER" using connect identifier "orcldb"
DGMGRL> DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Nov 17 16:11:05 2022
Version 19.17.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "orcldb"
Connected as SYSDBA.
Succeeded in opening the observer file "/u01/orcldbcfg_obs.dat".
[W000 2022-11-17T16:11:06.702-06:00] FSFO target standby is orcldbp
Observer 'orcldbcfg_obs' started
The observer log file is '/u01/orcldbcfg_obs.log'.
DGMGRL>

Hope it helps................

Dynamic Services Fallback Option in Oracle 19c

Introduction of Dynamic Services Fallback:

1. A Service fails over to an available instance when there are no preferred instances available.

2. For a dynamic service that is placed using preferred and available settings, you can now specify that the service should fall back to a preferred instance when it becomes available.

3. This option allows for more control in placing dynamic database services and ensures that a given service is available on a preferred instance as much as possible.

Environment Setup:

Primary RAC         : orcl (orcl1 and orcl2)
GRID Home         : /u01/app/19.0.0/grid
RDBMS Home : /u01/app/oracle/product/19.0.0/dbhome_1
Version                 : Oracle Database 19c EE - Production Version 19.12.0.0.0

Scan IP         : oel70p-scan

Step-1: Check the status of the 'orcl' database.

[oracle@oel70-base1 ~]$ srvctl status database -d orcl

Instance orcl1 is running on node oel70-base1

Instance orcl2 is running on node oel70-base2

[oracle@oel70-base1 ~]$

Step-2: Add the service 'serv1' with 'failback' option and start the service.

Step-3: Check the status of 'serv1' service and shutdown the database, startup the database and check the status of 'serv1' service.

 

Step-4: srvctl config service does not report state of failback when it is "NO" and created new service 'serv2' checking with 'failback' option enabling and checking the configuration.

[oracle@oel70-base1 ~]$ srvctl config service -d orcl -s serv1

Service name: serv1

Server pool:

Cardinality: 1

Service role: PRIMARY

Management policy: AUTOMATIC

DTP transaction: false

AQ HA notifications: false

Global: false

Commit Outcome: false

Failover type:

Failover method:

Failover retries:

Failover delay:

Failover restore: NONE

Connection Load Balancing Goal: LONG

Runtime Load Balancing Goal: NONE

TAF policy specification: NONE

Edition:

Pluggable database name:

Hub service:

Maximum lag time: ANY

SQL Translation Profile:

Retention: 86400 seconds

Failback :  true

Replay Initiation Time: 300 seconds

Drain timeout:

Stop option:

Session State Consistency: DYNAMIC

GSM Flags: 0

Service is enabled

Preferred instances: orcl1

Available instances: orcl2

CSS critical: no

Service uses Java: false

[oracle@oel70-base1 ~]$

Note: Please have a look at these properties

1. Session State Consistency: DYNAMIC

2. Failback                            : true

Thanks for looking this feature and blog. Happy learning.

Testing Application Continuity in Oracle 19c RAC (19.12) using Java Application

Introduction to Application Continuity

Application Continuity (AC) is a feature available with the Oracle Real Application Clusters (RAC), Oracle RAC One Node and Oracle Active Data Guard (ADG) options that masks outages from end users and applications by recovering the in-flight database sessions following recoverable outages. Application Continuity performs this recovery beneath the application so that the outage appears to the application as a slightly delayed execution.

Application Continuity improves the user experience for both planned maintenance and unplanned outages. Application Continuity enhances the fault tolerance of systems and applications that use an Oracle database.

Oracle MAA is a collection of architecture, configuration, and life cycle best practices and blueprints. It provides Oracle’s customers with valuable insights and expert recommendations which have been validated and tested working with enterprise customers. This is also an outcome of ongoing communication, with the community of database architects, software engineers, and database strategists, that helps Oracle develop a deep and complete understanding of various kinds of events that can affect availability or data integrity. Over the years, this led to the development and natural evolution of an array of availability reference architectures.

Application Continuity extends the MAA architecture to the application tier.

Environment Setup:

Primary RAC Database: orcl (orcl1 and orcl2)
GRID Home : /u01/app/19.0.0/grid
RDBMS Home: /u01/app/oracle/product/19.0.0/dbhome_1
Version    : Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.12.0.0.0
Scan        : oel70p-scan

Pre-requisites for Application Continuity:

1. Database and client (driver) version >= Oracle 12c
2. Enable Application Continuity on the Service
[oracle@oel70-base1 ~]$ srvctl modify service -d ORCL -s acservice -failovertype TRANSACTION -commit_outcome TRUE
3. Use the recommended tnsnames.ora or JDBC URL
4. Use Orachk: ($orachk -accheck)
5. Check in AWR Reports: (User Calls in = User Calls Protected)

Before Application Continuity:

Database outages cause in-flight work to be lost, leaving users and applications in-doubt

1. Developer pains
2. Duplicate submissions
3. Cancelled work
4. Errors even when planned 
5. Restart applications and mid-tiers

Step-1: Login as 'oracle' user check the database status (orcl)

[oracle@oel70-base1 ~]$ . oraenv
ORACLE_SID = [orcl1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel70-base1
Instance orcl2 is running on node oel70-base2
[oracle@oel70-base1 ~]$

Step-2: Create service (acservice) and check the status of the service

[oracle@oel70-base1 ~]$ srvctl add service -s acservice -d orcl -r orcl1,orcl2
[oracle@oel70-base1 ~]$ srvctl start service -s acservice -d orcl
[oracle@oel70-base1 ~]$ srvctl status service -s acservice -d orcl
Service acservice is running on instance(s) orcl1,orcl2
[oracle@oel70-base1 ~]$

Step-3: Modify the service to check 'Application Continuity' property

[oracle@oel70-base1 ~]$ srvctl modify service -d ORCL -s acservice -failovertype TRANSACTION -replay_init_time 300 -failoverretry 30 -failoverdelay 3 -notification TRUE -commit_outcome TRUE

[oracle@oel70-base1 ~]$

a. failovertype = AUTO or TRANSACTION for Application Continuity
b. commit_outcome = TRUE for Transaction Guard
c. failoverretry = Number of connection retries per replay
d. failoverdelay = Delay in seconds between connection retries

[oracle@oel70-base1 ~]$ srvctl config service -d  `srvctl config database` -s ac_service
Service name: ac_service
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: true
Global: false
Commit Outcome: true
Failover type: TRANSACTION
Failover method:
Failover retries:
Failover delay:
Failover restore: LEVEL1
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb1
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: orcl1
Available instances: orcl2
CSS critical: no
Service uses Java: false
[oracle@oel70-base1 ~]$

Step-4: Install JDK as a 'root' user and download ojdbc.jar file and place it to lib folder.

[root@oel70-base1 oracle]# rpm -ivh jdk-12.0.2_linux-x64_bin.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:jdk-12.0.2-2000:12.0.2-ga        ################################# [100%]
[root@oel70-base1 oracle]#
[root@oel70-base1 oracle]# chmod 755 ojdbc8.jar
[root@oel70-base1 oracle]# mv ojdbc8.jar /lib/

Step-5: Update bash_profile accordingly and execute the bash_profile

[oracle@oel70-base1 ~]$ cat .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export TMP=/tmp
export TMPDIR=$TMP
export JAVA_HOME=/usr/java/jdk-12.0.2/bin
export ORACLE_HOSTNAME=oel70-base1.localdomain
export ORACLE_UNQNAME=orcl
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/19.0.0/grid
export DB_HOME=$ORACLE_BASE/product/19.0.0/dbhome_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=orcl1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH:$JAVA_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:/lib/ojdbc8.jar
[oracle@oel70-base1 ~]$

Step-6: Create 'Actest.java' program to test the functionality of 'Application Continuity'

[oracle@oel70-base1 ~] cat AcTest.java
import java.sql.*;
import oracle.jdbc.*;
import oracle.jdbc.pool.*;
import oracle.jdbc.replay.*;
public class AcTest
{
public static void main(String[] args) throws SQLException,java.lang.InterruptedException
{
oracle.jdbc.replay.OracleDataSource  AcDatasource = oracle.jdbc.replay.OracleDataSourceFactory.getOracleDataSource();
AcDatasource.setURL("jdbc:oracle:thin:@racs-scan:1521/acservice");
AcDatasource.setUser("system");
AcDatasource.setPassword("oracle");
Connection conn = AcDatasource.getConnection();
conn.setAutoCommit(false);
PreparedStatement stmt = conn.prepareStatement("select instance_name from v$instance");
ResultSet rset = stmt.executeQuery();
while (rset.next())
{
System.out.println("You are Connected to RAC Instance - "+ rset.getString(1));
}
Thread.currentThread().sleep(60000);
((oracle.jdbc.replay.ReplayableConnection)conn).beginRequest();
PreparedStatement stmt1 = conn.prepareStatement("select instance_name from v$instance");
ResultSet rset1 = stmt1.executeQuery();
while (rset1.next())
{
System.out.println("After Replay Connected to RAC Instance - "+rset1.getString(1));
}
rset.close();
stmt.close();
rset1.close();
stmt1.close();
conn.close();
((oracle.jdbc.replay.ReplayableConnection)conn).endRequest();
}
}
[oracle@oel70-base1]#

Note: Configure connection string and sleep (60000) based on your environment/testing cycles.

Step-7: Login as 'root' user and compile 'Actest.java' program and execute the 'Actest.java' program using 'oracle' user.

[root@oel70-base1 ~]# javac /home/oracle/AcTest.java
[root@oel70-base1 ~]#

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl1
- - -> Waiting 

Actest.java program connected to 'orcl1' instance. We will stop instance using srvctl with force option and check the 'Actest.java' program output in the other screen.

[oracle@oel70-base1 ~]$ srvctl stop instance -i orcl1 -d orcl -force

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl1
After Replay Connected to RAC Instance - orcl2
[oracle@oel70-base1 ~]$

Note: It will be reconnected to 'orcl2' instance automatically based on 'Actest.java' program.

Step-8: Execute 'Actest.java' program again and it will be connected to 'orcl2' instance because 'orcl1' is not yet started. 

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl2
Start the 'orcl1' instance and check the output of 'Actest.java' program.

[oracle@oel70-base1 ~]$ srvctl start instance -i orcl1 -d orcl

Note: The above command is taking time to start 'orcl1' instance mean time 'Actest.java' program connected to again 'orcl2' instance.

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl2
After Replay Connected to RAC Instance - orcl2
[oracle@oel70-base1 ~]$

Now, both the instances are UP and try to connect 'Actest.java' program.

[oracle@oel70-base1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel70-base1
Instance orcl2 is running on node oel70-base2
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ java AcTest.java

You are Connected to RAC Instance - orcl2

So, it connected to 'orcl2' instance.

Additional notes:  Configuring your Service

When using Oracle Database on premise, you can create services on Oracle RAC that use Transparent Application Continuity, or Application Continuity, or TAF. You can use roles to distinguish whether the services are active on Active Data Guard or the primary database. The following examples illustrate this:

Basic Service Creation

$ srvctl add service -db mydb -service MYSERVICE –preferred inst1 -available
inst2 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE -role PRIMARY

Transparent Application Continuity

$ srvctl add service -db mydb -service GOLD -preferred serv1 -available serv2 -
failover_restore AUTO -failoverretry 30 -failoverdelay 10 -commit_outcome TRUE -
failovertype AUTO -replay_init_time 1800 -retention 86400 -notification TRUE -
drain_timeout 300 -stopoption IMMEDIATE

Application Continuity

$ srvctl add service -db mydb -service SILVER -preferred serv1 -available serv2
-failover_restore LEVEL1 -failoverretry 30 -failoverdelay 10 -commit_outcome
TRUE -failovertype TRANSACTION -replay_init_time 1800 -retention 86400 -
notification TRUE -drain_timeout 300 -stopoption IMMEDIATE

Transparent Application Failover

$ srvctl add service -db mydb -service BRONZE -preferred serv1 -available serv2
-failover_restore LEVEL1 -failoverretry 30 -failoverdelay 10 -commit_outcome
TRUE -failovertype SELECT -retention 86400 -notification TRUE -drain_timeout 300
-stopoption IMMEDIATE

To add with the Data Guard role, here is the TAC example:

$ srvctl add service -db mydb -service GOLD -preferred serv1 -available serv2 -
failover_restore AUTO -failoverretry 30 -failoverdelay 10 -commit_outcome TRUE -
failovertype AUTO -replay_init_time 1800 -retention 86400 -notification TRUE -
role PHYSICAL_STANDBY -drain_timeout 300 -stopoption IMMEDIATE

SQL Queries to check the status of services

Query-1: To report protection by PDB, use the following example:

set lines 85
col Service_name format a30 trunc heading "Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
select con_id, total_requests,
total_calls,total_protected,total_protected*100/NULLIF(total_calls,0) as Protected
from(
select * from
(select s.con_id, s.name, s.value
FROM GV$CON_SYSSTAT s, GV$STATNAME n
WHERE s.inst_id = n.inst_id
AND s.statistic# = n.statistic#
AND s.value != 0 )
pivot(
sum(value)
for name in ('cumulative begin requests' as total_requests, 'cumulative end requests' as
Total_end_requests, 'cumulative user calls in requests' as Total_calls, 'cumulative user
calls protected by Application Continuity' as total_protected)
))
order by con_id;

Query-2: To report protection by service, use the following example:

set pagesize 60
set lines 120
col Service_name format a30 trunc heading "Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
select con_id, service_name,total_requests,
total_calls,total_protected,total_protected*100/NULLIF(total_calls,0) as Protected
from(
select * from
(select a.con_id, a.service_name, c.name,b.value
FROM gv$session a, gv$sesstat b, gv$statname c
WHERE a.sid = b.sid
AND a.inst_id = b.inst_id
AND b.value != 0
AND b.statistic# = c.statistic#
AND b.inst_id = c.inst_id
AND a.service_name not in ('SYS$USERS','SYS$BACKGROUND'))
pivot(
sum(value)
for name in ('cumulative begin requests' as total_requests, 'cumulative end requests' as
Total_end_requests, 'cumulative user calls in requests' as Total_calls, 'cumulative user
calls protected by Application Continuity' as total_protected) ))
order by con_id, service_name; 

Query-3: To report protection history over last three days, use the following example:

set lines 85
col Service_name format a30 trunc heading"Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
set lines 85
col Service_name format a30 trunc heading"Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
select a.instance_number,begin_interval_time, total_requests, total_calls, total_protected,
total_protected*100/NULLIF(total_calls,0) as Protected
from(
select * from
(select a.snap_id, a.instance_number,a.stat_name, a.value
FROM dba_hist_sysstat a
WHERE a.value != 0 )
pivot(
sum(value)
for stat_name in ('cumulative begin requests' as total_requests, 'cumulative end requests' as
Total_end_requests, 'cumulative user calls in requests' as Total_calls, 'cumulative user calls
protected by Application Continuity' as total_protected)
)) a,
dba_hist_snapshot b
where a.snap_id=b.snap_id
and a.instance_number=b.instance_number
and begin_interval_time>systimestamp - interval '3' day
order by a.snap_id,a.instance_number;

Please refer more details:

https://www.youtube.com/watch?v=kBG5DxZ3szg
https://www.oracle.com/database/technologies/high-availability/app-continuity.html
https://www.youtube.com/watch?v=DOtNjigyjV4

Goldengate DBLOGIN Issue OCI Error ORA (status = 12545-ORA-12545) and OGG-03542 Failed to connect to the database. Oracle Golden Gate 21c

Oracle Database 12c: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle Golden Gate 21c: /u01/app/ogg/12c_21c

[oracle@primary ~]$ cd /u01/app/ogg/12c_21c/

[oracle@primary 12c_21c]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047_FBO
Oracle Linux 7, x64, 64bit (optimized), Oracle Database 21c and lower supported versions on Jul 29 2021 03:59:23
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.
GGSCI (primary) 1> dblogin userid ogguser@db12c, password oracle
2022-08-13 20:07:33  INFO    OGG-03542  Failed to connect to the database. Verify that the connection string and following environment variables are correct:
LD_LIBRARY_PATH = /u01/app/oracle/product/12.1.0.2/dbhome_1/lib:/lib:/usr/lib:/u01/app/ogg/12c_21c.
Error: OCI Error ORA (status = 12154-ORA-12154: TNS:could not resolve the connect identifier specified
)
GGSCI (primary) 2> exit

[oracle@primary ~]$ vi .bash_profile
[oracle@primary ~]$ . ./.bash_profile
[oracle@primary ~]$
[oracle@primary ~]$ cd /u01/app/ogg/12c_21c/

[oracle@primary 12c_21c]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047_FBO
Oracle Linux 7, x64, 64bit (optimized), Oracle Database 21c and lower supported versions on Jul 29 2021 03:59:23
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.

GGSCI (primary) 1> dblogin userid ogguser@db12c, password oracle
Successfully logged into database.
GGSCI (primary as ogguser@db12c) 2>

Solution:
Apart from other entries in .bash_profile added below "TNS_ADMIN" and specified in "LD_LIBRARY_PATH"
export TNS_ADMIN=$ORACLE_BASE/product/12.1.0.2/dbhome_1/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/u01/app/ogg/12c_21c:$TNS_ADMIN

Applying GRID and RDBMS RU (19.16.0.0.220719) to Oracle 19c (19.12) with two node Primary RAC with Physical Standby RAC

<pre>

Note: I have shown here high level steps for applying Grid RU and RDBMS RU. Please follow "readme" before applying RUs based on your environment with all pre-checks.

Environment:
------------------
Primary RAC: Oracle 19c (19.12) with two node RAC
Physical Standby RAC: Oracle 19c (19.12) with two node RAC

Required Softwares:
--------------------------
Latest Opatch:                                        p6880880_190000_Linux-x86-64.zip
GI Release Update (July 2022):        p34130714_190000_Linux-x86-64.zip
Database Release Update (July 2022): p34133642_190000_Linux-x86-64.zip

Check the status of cluster instances in Physical Standby RAC:

---------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 12:19:27 2022

Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> select name,open_mode,database_role from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

ORCL      READ ONLY WITH APPLY PHYSICAL STANDBY

ORCL      READ ONLY WITH APPLY PHYSICAL STANDBY

[oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [orcl1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sat Jul 30 14:36:51 2022

Version 19.12.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/oracle

Connected to "orcl"

Connected as SYSDBA.

DGMGRL>

DGMGRL> show configuration;

Configuration - dg_19c

  Protection Mode: MaxPerformance

  Members:

  orcl  - Primary database

    orcls - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:

SUCCESS   (status updated 2 seconds ago)

DGMGRL>

DGMGRL> edit database orcls set state='APPLY-OFF';

Succeeded.

DGMGRL>

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl status database -d orcls

Instance orcls1 is running on node oel70-base3

Instance orcls2 is running on node oel70-base4

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 14:35:37 2022

Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

             d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE", 

             d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME 

           from GV$DATABASE d, gv$instance i 

           where i.INST_ID=d.INST_ID;

DATABASE_HOST             DB_NAME          DB_UNIQUE_NAME             DATABASE_ROLE    OPEN_MODE           STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- --------------------

oel70-base3               orcls1           orcls                     PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4               orcls2           orcls                     PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

            d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE", 

            d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME

      from GV$DATABASE d, gv$instance i 

      where i.INST_ID=d.INST_ID;  2

DATABASE_HOST             DB_NAME          DB_UNIQUE_NAME             DATABASE_ROLE    OPEN_MODE           STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- -------------------

oel70-base3       orcls1           orcls  PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4       orcls2           orcls  PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> select inst_id,process, status, thread#, sequence#, block#, blocks from gv$managed_standby where process='MRP0';

select a.thread#, (select max (sequence#) from v$archived_log where archived='YES' and thread#=a.thread#) archived,max(a.sequence#) applied,

(select max(sequence#) from v$archived_log where archived='YES' and thread#=a.thread#)-max(a.sequence#)gap

from v$archived_log a where a.applied='YES' group by a.thread# order by thread#;

   INST_ID PROCESS   STATUS          THREAD#  SEQUENCE#     BLOCK#     BLOCKS

---------- --------- ------------ ---------- ---------- ---------- ----------

         2 MRP0      APPLYING_LOG          2         48      26627     409600

SQL>   2    3

   THREAD#   ARCHIVED    APPLIED        GAP

---------- ---------- ---------- ----------

         1         57         57          0

         2         47         47          0

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

[oracle@oel70-base3 ~]$

Unzip required patches with OPatch in both the nodes:
----------------------------------------------------------------------

Standby node-1

[oracle@oel70-base3 ~]$  unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive:  p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$  unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/product/19.0.0/dbhome_1/
Archive:  p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive:  p34133642_190000_Linux-x86-64.zip

Standby node-2

[oracle@oel70-base4 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive:  p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34130714_190000_Linux-x86-64.zip
Archive:  p34130714_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive:  p34133642_190000_Linux-x86-64.zip

Applying Grid RU and RDBMS RU in Physical Standby Node-1:

-----------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ su - root

Password:

Last login: Sat Jul 30 14:33:21 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108396

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle    5187     1  0 01:29 ?        00:00:01 asm_pmon_+ASM1

oracle    8157     1  0 01:33 ?        00:00:02 ora_pmon_orcls1

root     24502 24146  0 14:45 pts/0    00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 14:46:45 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_02-46-51PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_02-47-56PM.log

The id for this session is IM3L

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_prepatch_apply_inplace_oel70-base3_2022-07-30_02-51-57PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Failed while applying binary patches on home /u01/app/19.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : oel70-base3->/u01/app/19.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.0.0/grid, host: oel70-base3.

Command failed:  /u01/app/19.0.0/grid/OPatch/opatchauto  apply /home/oracle/34130714/ -oh /u01/app/19.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc -jre /u01/app/19.0.0/grid/OPatch/jre -persistresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_oel70-base3_crs_1.ser -analyzedresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_oel70-base3_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sat Jul 30 15:00:17 2022

Time taken to complete the session 13 minutes, 32 seconds

opatchauto failed with error code 42

[root@oel70-base3 ~]#

Got the error and checking the opatch logile

----------------------------------------------------------

[oracle@oel70-base3 ~]$ cat /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO]    ENTERING METHOD: NApply(patches,options)

[Jul 30, 2022 2:59:09 PM] [INFO]    Checking if the Oracle Home /u01/app/19.0.0/grid is good for OPatch to work.

[Jul 30, 2022 2:59:09 PM] [INFO]    Option unsupported:false and isCASEnabled :false

[Jul 30, 2022 2:59:09 PM] [INFO]    OPatch invoked as follows: 'napply -phBaseFile /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/SDKWork.txt -local '

[Jul 30, 2022 2:59:09 PM] [INFO]    Oracle Home       : /u01/app/19.0.0/grid

                                    Central Inventory : /u01/app/oraInventory

                                       from           : /u01/app/19.0.0/grid/oraInst.loc

                                    OPatch version    : 12.2.0.1.32

                                    OUI version       : 12.2.0.7.0

                                    OUI location      : /u01/app/19.0.0/grid/oui

                                    Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO]    Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch_history.txt

[Jul 30, 2022 2:59:11 PM] [INFO]    [OPSR-TIME] Loading raw inventory

[Jul 30, 2022 2:59:11 PM] [INFO]    [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 133 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO]    [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 135 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO]    [OPSR-TIME] Raw inventory loaded successfully

[Jul 30, 2022 2:59:12 PM] [INFO]    NApply::no CAS enabled, OPatch runs with legacy process.

[Jul 30, 2022 2:59:12 PM] [INFO]    Verifying environment and performing prerequisite checks...

[Jul 30, 2022 2:59:12 PM] [INFO]    [OPSR-TIME] Running prerequisite checks

[Jul 30, 2022 2:59:12 PM] [INFO]    opatch-external.jar is in /u01/app/19.0.0/grid/OPatch/jlib/opatch-external.jar

[Jul 30, 2022 2:59:19 PM] [INFO]    [OPSR-TIME] Loading cooked inventory

[Jul 30, 2022 2:59:19 PM] [INFO]    [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 107 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO]    [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 138 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO]    [OPSR-TIME] Cooked inventory loaded successfully

[Jul 30, 2022 2:59:45 PM] [INFO]    Checking if Oracle Home has components required by patches...

[Jul 30, 2022 2:59:47 PM] [INFO]    CheckMissingComps: Cached file does not exist or is invalid, re-build prereq result.

[Jul 30, 2022 2:59:47 PM] [INFO]    Checking conflict among patches...

[Jul 30, 2022 2:59:47 PM] [INFO]    Running prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO]    Following patches can be applied:  33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:48 PM] [INFO]    Following patches are not required:

[Jul 30, 2022 2:59:48 PM] [INFO]    Following patches are auto rollbackable:

[Jul 30, 2022 2:59:48 PM] [INFO]    Finished checking prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO]    Conflict checking amongst patches PASSED

[Jul 30, 2022 2:59:48 PM] [INFO]    Checking conflicts against Oracle Home...

[Jul 30, 2022 2:59:50 PM] [INFO]    Running prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:50 PM] [INFO]    CUP_LOG: Found pi CUP 34133642 is a superset of  poh CUP: 32904851

[Jul 30, 2022 2:59:53 PM] [INFO]    Following patches can be applied:  33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:53 PM] [INFO]    Following patches are not required:

[Jul 30, 2022 2:59:53 PM] [INFO]    Following patches are auto rollbackable:

[Jul 30, 2022 2:59:53 PM] [INFO]    Finished checking prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:56 PM] [INFO]    OCM validation is skipped for SDK mode.

[Jul 30, 2022 2:59:58 PM] [INFO]    Execution of 'sh /home/oracle/34130714/34133642/custom/scripts/init -apply 34133642 ':

[Jul 30, 2022 2:59:58 PM] [INFO]    Return Code = 0

[Jul 30, 2022 2:59:58 PM] [INFO]    Execution of ' Command and arguments of init script is restricted to be viewed in log file ':

                                    Return Code = 0

[Jul 30, 2022 3:00:01 PM] [INFO]    Running prerequisite checks...

[Jul 30, 2022 3:00:04 PM] [INFO]    Space Needed : 10231.319MB

[Jul 30, 2022 3:00:04 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 33575402

[Jul 30, 2022 3:00:08 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34133642

[Jul 30, 2022 3:00:08 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34139601

[Jul 30, 2022 3:00:08 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34160635

[Jul 30, 2022 3:00:08 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed on patch :34318175

[Jul 30, 2022 3:00:11 PM] [WARNING] Action file /u01/app/19.0.0/grid/jlib/srvmasm.jar is in the jar list,OOP should be lanched

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Files in use by a process: /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Files in use by a process: /u01/app/19.0.0/grid/lib/libasmclntsh19.so PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO]    Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

                                    /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

                                    /u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:14 PM] [INFO]    Prerequisite check "CheckActiveFilesAndExecutables" failed.

                                    The details are:

Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

                                    /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

                                    /u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:16 PM] [INFO]    UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [INFO]    Finishing UtilSession at Sat Jul 30 15:00:16 CDT 2022

[Jul 30, 2022 3:00:16 PM] [INFO]    Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1:  6136m

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep  6136

oracle    6136     1  0 01:30 ?        00:00:40 /u01/app/19.0.0/grid/jdk/bin/java -server -Xms128M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/u01/app/oracle/crsdata/oel70-base3/qos/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=oel70-base3 -Doracle.http.port=8888 -Doracle.jmx.port=8895 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=false -Djava.security.manager -Djava.security.policy=/u01/app/oracle/crsdata/oel70-base3/qos/conf/catalina.policy -Djava.security.egd=file:/dev/urandom -XX:ParallelGCThreads=2 -Dcatalina.home=/u01/app/19.0.0/grid/tomcat -Dcatalina.base=/u01/app/oracle/crsdata/oel70-base3/qos -Djava.io.tmpdir=/u01/app/oracle/crsdata/oel70-base3/qos/temp -Doracle.home=/u01/app/19.0.0/grid -Doracle.jwc.mode=CLUSTERWARE -classpath /u01/app/19.0.0/grid/jlib/cryptoj.jar:/u01/app/19.0.0/grid/jlib/oraclepki.jar:/u01/app/19.0.0/grid/jlib/osdt_core.jar:/u01/app/19.0.0/grid/jlib/osdt_cert.jar:/u01/app/19.0.0/grid/tomcat/lib/tomcat-juli.jar:/u01/app/19.0.0/grid/tomcat/lib/bootstrap.jar:/u01/app/19.0.0/grid/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

oracle   23827 21496  0 15:13 pts/1    00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ kill -9 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep  6136

oracle   24100 21496  0 15:13 pts/1    00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

After killing process and resuming the patch:

----------------------------------------------------------

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto resume

OPatchauto session is initiated at Sat Jul 30 15:33:25 2022

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-33-25PM.log

Resuming existing session with id IM3L

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_postpatch_apply_inplace_oel70-base3_2022-07-30_03-39-19PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base3

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Host:oel70-base3

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patching session reported following warning(s):

______________________________________

[WARNING] The database instance 'cdbc1' from '/u01/app/oracle/product/19.0.0/dbhome_1', in host'oel70-base3' is not running. SQL changes, if any,  will not be applied.

To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).

Refer to the readme to get the correct steps for applying the sql changes.

OPatchauto session completed at Sat Jul 30 15:46:34 2022

Time taken to complete the session 13 minutes, 10 seconds

[root@oel70-base3 ~]#

Check the process:

------------------------

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle   14834     1  0 15:42 ?        00:00:00 asm_pmon_+ASM1

oracle   15294     1  0 15:43 ?        00:00:00 ora_pmon_orcls1

root     22124 24146  0 15:46 pts/0    00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

Applying Grid RU and RDBMS RU in Physical Standby Node-2:

-----------------------------------------------------------------------------------

[oracle@oel70-base4 ~]$ su - root

Password:

Last login: Sat Jul 30 15:49:07 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108437

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# . oraenv

ORACLE_SID = [root] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# ps -ef | grep pmon

oracle    7280     1  0 01:32 ?        00:00:01 asm_pmon_+ASM2

oracle    7788     1  0 01:33 ?        00:00:02 ora_pmon_orcls2

root     28280 27450  0 15:50 pts/0    00:00:00 grep --color=auto pmon

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 15:51:30 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_03-51-35PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-52-40PM.log

The id for this session is 8SKS

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base4/crsconfig/crs_prepatch_apply_inplace_oel70-base4_2022-07-30_03-57-08PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base4/crsconfig/crs_postpatch_apply_inplace_oel70-base4_2022-07-30_04-11-04PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base4

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-00-07PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-00-07PM_1.log

Host:oel70-base4

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-05-11PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-05-11PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-05-11PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-05-11PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_16-05-11PM_1.log

Patching session reported following warning(s):

_____________________________________

[WARNING] The database instance 'cdbc2' from '/u01/app/oracle/product/19.0.0/dbhome_1', in host'oel70-base4' is not running. SQL changes, if any,  will not be applied.

To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).

Refer to the readme to get the correct steps for applying the sql changes.

OPatchauto session completed at Sat Jul 30 16:18:20 2022

Time taken to complete the session 26 minutes, 50 seconds

[root@oel70-base4 ~]#

Check the process:

------------------------

[root@oel70-base4 ~]# ps -ef | grep pmon

oracle     826     1  0 16:14 ?        00:00:00 asm_pmon_+ASM2

oracle    1316     1  0 16:14 ?        00:00:00 ora_pmon_orcls2

root     20587 27450  0 16:29 pts/0    00:00:00 grep --color=auto pmon

[root@oel70-base4 ~]#

Applying Grid RU and RDBMS RU in Primary RAC:

----------------------------------------------------------------------

Primary RAC - Node1:

[oracle@oel70-base1 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/

Archive:  p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base1 ~]$ unzip p34130714_190000_Linux-x86-64.zip

Archive:  p34130714_190000_Linux-x86-64.zip

[oracle@oel70-base1 ~]$ unzip p34133642_190000_Linux-x86-64.zip

Archive:  p34133642_190000_Linux-x86-64.zip

Note-2: Do it for Node-2

Applying Grid RU and RDBMS RU in Primary Node-1:

------------------------------------------------------------------------

[oracle@oel70-base1 ~]$ su - root

Password:

Last login: Sat Jul 30 17:33:33 CDT 2022

[root@oel70-base1 ~]#

[root@oel70-base1 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[root@oel70-base1 ~]#

[root@oel70-base1 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 17:34:54 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_05-34-58PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_05-35-31PM.log

The id for this session is TTT2

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base1/crsconfig/crs_prepatch_apply_inplace_oel70-base1_2022-07-30_05-40-31PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base1/crsconfig/crs_postpatch_apply_inplace_oel70-base1_2022-07-30_05-52-50PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base1

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-43-35PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-43-35PM_1.log

Host:oel70-base1

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-47-42PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-47-42PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-47-42PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-47-42PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_17-47-42PM_1.log

OPatchauto session completed at Sat Jul 30 18:02:22 2022

Time taken to complete the session 27 minutes, 29 seconds

[root@oel70-base1 ~]#

[root@oel70-base1 ~]# ps -ef | grep pmon

root      1033 29000  0 18:02 pts/0    00:00:00 grep --color=auto pmon

oracle   21877     1  0 17:55 ?        00:00:00 asm_pmon_+ASM1

oracle   22265     1  0 17:56 ?        00:00:00 ora_pmon_orcl1

[root@oel70-base1 ~]#

Applying Grid RU and RDBMS RU in Primary Node-2:

------------------------------------------------------------------------

[oracle@oel70-base2 ~]$ su - root

Password:

Last login: Sat Jul 30 17:48:50 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659060080

[root@oel70-base2 ~]#

[root@oel70-base2 ~]# . oraenv

ORACLE_SID = [root] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[root@oel70-base2 ~]#

[root@oel70-base2 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 18:03:15 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_06-03-23PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_06-03-56PM.log

The id for this session is 5F8B

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base2/crsconfig/crs_prepatch_apply_inplace_oel70-base2_2022-07-30_06-10-34PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Failed while applying binary patches on home /u01/app/19.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : oel70-base2->/u01/app/19.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.0.0/grid, host: oel70-base2.

Command failed:  /u01/app/19.0.0/grid/OPatch/opatchauto  apply /home/oracle/34130714/ -oh /u01/app/19.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc -jre /u01/app/19.0.0/grid/OPatch/jre -persistresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_oel70-base2_crs_1.ser -analyzedresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_oel70-base2_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sat Jul 30 18:20:32 2022

Time taken to complete the session 17 minutes, 17 seconds

opatchauto failed with error code 42

[root@oel70-base2 ~]#

[oracle@oel70-base2 ~]$ cat /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

[Jul 30, 2022 6:19:10 PM] [INFO]    ENTERING METHOD: NApply(patches,options)

[Jul 30, 2022 6:19:10 PM] [INFO]    Checking if the Oracle Home /u01/app/19.0.0/grid is good for OPatch to work.

[Jul 30, 2022 6:19:10 PM] [INFO]    Option unsupported:false and isCASEnabled :false

[Jul 30, 2022 6:19:10 PM] [INFO]    OPatch invoked as follows: 'napply -phBaseFile /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/SDKWork.txt -local '

[Jul 30, 2022 6:19:10 PM] [INFO]    Oracle Home       : /u01/app/19.0.0/grid

                                    Central Inventory : /u01/app/oraInventory

                                       from           : /u01/app/19.0.0/grid/oraInst.loc

                                    OPatch version    : 12.2.0.1.32

                                    OUI version       : 12.2.0.7.0

                                    OUI location      : /u01/app/19.0.0/grid/oui

                                    Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

[Jul 30, 2022 6:19:10 PM] [INFO]    Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch_history.txt

[Jul 30, 2022 6:19:12 PM] [INFO]    [OPSR-TIME] Loading raw inventory

[Jul 30, 2022 6:19:12 PM] [INFO]    [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 132 (MB)

[Jul 30, 2022 6:19:12 PM] [INFO]    [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 135 (MB)

[Jul 30, 2022 6:19:12 PM] [INFO]    [OPSR-TIME] Raw inventory loaded successfully

[Jul 30, 2022 6:19:12 PM] [INFO]    NApply::no CAS enabled, OPatch runs with legacy process.

[Jul 30, 2022 6:19:12 PM] [INFO]    Verifying environment and performing prerequisite checks...

[Jul 30, 2022 6:19:12 PM] [INFO]    [OPSR-TIME] Running prerequisite checks

[Jul 30, 2022 6:19:12 PM] [INFO]    opatch-external.jar is in /u01/app/19.0.0/grid/OPatch/jlib/opatch-external.jar

[Jul 30, 2022 6:19:20 PM] [INFO]    [OPSR-TIME] Loading cooked inventory

[Jul 30, 2022 6:19:20 PM] [INFO]    [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 107 (MB)

[Jul 30, 2022 6:19:23 PM] [INFO]    [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 101 (MB)

[Jul 30, 2022 6:19:23 PM] [INFO]    [OPSR-TIME] Cooked inventory loaded successfully

[Jul 30, 2022 6:19:47 PM] [INFO]    Checking if Oracle Home has components required by patches...

[Jul 30, 2022 6:19:49 PM] [INFO]    CheckMissingComps: Cached file does not exist or is invalid, re-build prereq result.

[Jul 30, 2022 6:19:49 PM] [INFO]    Checking conflict among patches...

[Jul 30, 2022 6:19:49 PM] [INFO]    Running prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 6:19:50 PM] [INFO]    Following patches can be applied:  33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 6:19:50 PM] [INFO]    Following patches are not required:

[Jul 30, 2022 6:19:50 PM] [INFO]    Following patches are auto rollbackable:

[Jul 30, 2022 6:19:50 PM] [INFO]    Finished checking prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 6:19:50 PM] [INFO]    Conflict checking amongst patches PASSED

[Jul 30, 2022 6:19:50 PM] [INFO]    Checking conflicts against Oracle Home...

[Jul 30, 2022 6:19:53 PM] [INFO]    Running prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 6:19:54 PM] [INFO]    CUP_LOG: Found pi CUP 34133642 is a superset of  poh CUP: 32904851

[Jul 30, 2022 6:19:57 PM] [INFO]    Following patches can be applied:  33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 6:19:57 PM] [INFO]    Following patches are not required:

[Jul 30, 2022 6:19:57 PM] [INFO]    Following patches are auto rollbackable:

[Jul 30, 2022 6:19:57 PM] [INFO]    Finished checking prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 6:20:02 PM] [INFO]    OCM validation is skipped for SDK mode.

[Jul 30, 2022 6:20:07 PM] [INFO]    Execution of 'sh /home/oracle/34130714/34133642/custom/scripts/init -apply 34133642 ':

[Jul 30, 2022 6:20:07 PM] [INFO]    Return Code = 0

[Jul 30, 2022 6:20:07 PM] [INFO]    Execution of ' Command and arguments of init script is restricted to be viewed in log file ':

                                    Return Code = 0

[Jul 30, 2022 6:20:10 PM] [INFO]    Running prerequisite checks...

[Jul 30, 2022 6:20:12 PM] [INFO]    Space Needed : 10231.319MB

[Jul 30, 2022 6:20:12 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 33575402

[Jul 30, 2022 6:20:16 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34133642

[Jul 30, 2022 6:20:16 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34139601

[Jul 30, 2022 6:20:16 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34160635

[Jul 30, 2022 6:20:16 PM] [INFO]    Prereq checkPatchApplicableOnCurrentPlatform Passed on patch :34318175

[Jul 30, 2022 6:20:20 PM] [WARNING] Action file /u01/app/19.0.0/grid/jlib/srvmasm.jar is in the jar list,OOP should be lanched

[Jul 30, 2022 6:20:24 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 18:20:24 CDT 2022

[Jul 30, 2022 6:20:24 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 18:20:24 CDT 2022

[Jul 30, 2022 6:20:24 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 18:20:24 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Files in use by a process: /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 PID( 8097 )

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Files in use by a process: /u01/app/19.0.0/grid/lib/libasmclntsh19.so PID( 8097 )

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:25 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 18:20:25 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:26 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 18:20:26 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 18:20:27 CDT 2022

[Jul 30, 2022 6:20:27 PM] [INFO]    Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

                                    /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

                                    /u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 6:20:27 PM] [INFO]    Prerequisite check "CheckActiveFilesAndExecutables" failed.

                                    The details are:

                                    Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

                                    /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

                                    /u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 6:20:31 PM] [INFO]    UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 6:20:31 PM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 6:20:31 PM] [INFO]    Finishing UtilSession at Sat Jul 30 18:20:31 CDT 2022

[Jul 30, 2022 6:20:31 PM] [INFO]    Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-18-57PM_1.log

[oracle@oel70-base2 ~]$

[oracle@oel70-base2 ~]$ /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1:  8097m

[oracle@oel70-base2 ~]$

[oracle@oel70-base2 ~]$ ps -ef | grep 8097

oracle    8097     1  0 01:51 ?        00:00:45 /u01/app/19.0.0/grid/jdk/bin/java -server -Xms128M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/u01/app/oracle/crsdata/oel70-base2/qos/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=oel70-base2 -Doracle.http.port=8888 -Doracle.jmx.port=8895 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=false -Djava.security.manager -Djava.security.policy=/u01/app/oracle/crsdata/oel70-base2/qos/conf/catalina.policy -Djava.security.egd=file:/dev/urandom -XX:ParallelGCThreads=2 -Dcatalina.home=/u01/app/19.0.0/grid/tomcat -Dcatalina.base=/u01/app/oracle/crsdata/oel70-base2/qos -Djava.io.tmpdir=/u01/app/oracle/crsdata/oel70-base2/qos/temp -Doracle.home=/u01/app/19.0.0/grid -Doracle.jwc.mode=CLUSTERWARE -classpath /u01/app/19.0.0/grid/jlib/cryptoj.jar:/u01/app/19.0.0/grid/jlib/oraclepki.jar:/u01/app/19.0.0/grid/jlib/osdt_core.jar:/u01/app/19.0.0/grid/jlib/osdt_cert.jar:/u01/app/19.0.0/grid/tomcat/lib/tomcat-juli.jar:/u01/app/19.0.0/grid/tomcat/lib/bootstrap.jar:/u01/app/19.0.0/grid/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

oracle   29628 29100  0 19:01 pts/1    00:00:00 grep --color=auto 8097

[oracle@oel70-base2 ~]$

[oracle@oel70-base2 ~]$ kill -9 8097

[oracle@oel70-base2 ~]$

[oracle@oel70-base2 ~]$ /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

[oracle@oel70-base2 ~]$

[root@oel70-base2 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto resume

OPatchauto session is initiated at Sat Jul 30 19:02:29 2022

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_07-02-30PM.log

Resuming existing session with id 5F8B

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Checking shared status of home.....

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base2/crsconfig/crs_postpatch_apply_inplace_oel70-base2_2022-07-30_07-08-42PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base2

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-13-39PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_18-13-39PM_1.log

Host:oel70-base2

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_19-03-03PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_19-03-03PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_19-03-03PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_19-03-03PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_19-03-03PM_1.log

OPatchauto session completed at Sat Jul 30 19:42:17 2022

Time taken to complete the session 39 minutes, 48 seconds

[root@oel70-base2 ~]#

Enabling Archives through DGMGRL:

--------------------------------------------------

[oracle@oel70-base1 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sat Jul 30 19:52:16 2022

Version 19.16.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/oracle

Connected to "orcl"

Connected as SYSDBA.

DGMGRL> edit database orcls set state='APPLY-ON';

Succeeded.

DGMGRL>

DGMGRL> show configuration;

Configuration - dg_19c

  Protection Mode: MaxPerformance

  Members:

  orcl  - Primary database

    orcls - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:

SUCCESS   (status updated 58 seconds ago)

DGMGRL>

Testing Phase b/n Primary RAC and Standby RAC:

-------------------------------------------------------------------

oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [orcl1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 19:51:18 2022

Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.16.0.0.0

SQL> select name,open_mode,database_role from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

ORCL      READ WRITE           PRIMARY

ORCL      READ WRITE           PRIMARY

SQL> select * from dgtest;

        NO NAME

---------- --------------------

         1 ORACLE

         2 SQLSERVER

         3 IBM DB2

         4 SYBASE

         5 POSTGRES

         6 MONGODB

         7 CASSANDRA

         8 TERADATA

         9 MYSQL

        10 COCKROCH

        11 SNOWFLAKE

        12 FOXPRO

12 rows selected.

SQL> insert into dgtest values (13,'DBASE');

1 row created.

SQL> commit;

Commit complete.

SQL> alter system switch all logfile;

System altered.

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 19:51:52 2022

Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.16.0.0.0

SQL> select name,open_mode,database_role from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

ORCL      READ ONLY            PHYSICAL STANDBY

ORCL      READ ONLY            PHYSICAL STANDBY

SQL>

SQL> select * from dgtest;

        NO NAME

---------- --------------------

         1 ORACLE

         2 SQLSERVER

         3 IBM DB2

         4 SYBASE

         5 POSTGRES

         6 MONGODB

         7 CASSANDRA

         8 TERADATA

         9 MYSQL

        10 COCKROCH

        11 SNOWFLAKE

        12 FOXPRO

12 rows selected.

SQL> select name,open_mode,database_role from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE

--------- -------------------- ----------------

ORCL      READ ONLY WITH APPLY PHYSICAL STANDBY

ORCL      READ ONLY WITH APPLY PHYSICAL STANDBY

SQL>

SQL> select * from dgtest;

        NO NAME

---------- --------------------

         1 ORACLE

         2 SQLSERVER

         3 IBM DB2

         4 SYBASE

         5 POSTGRES

         6 MONGODB

         7 CASSANDRA

         8 TERADATA

         9 MYSQL

        10 COCKROCH

        11 SNOWFLAKE

        NO NAME

---------- --------------------

        12 FOXPRO

        13 DBASE

13 rows selected.

SQL>

Checking applied Patches:

-----------------------------------

Primary RAC

[oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [orcl1] ? +ASM1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$ sh /u01/app/19.0.0/grid/OPatch/opatch lspatches

34318175;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34318175)

34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)

34139601;ACFS RELEASE UPDATE 19.16.0.0.0 (34139601)

34133642;Database Release Update : 19.16.0.0.220719 (34133642)

33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [+ASM1] ? orcl1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sh /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch lspatches

34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)

34133642;Database Release Update : 19.16.0.0.220719 (34133642)

OPatch succeeded.

[oracle@oel70-base1 ~]$

Standby RAC

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ? +ASM1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$ sh /u01/app/19.0.0/grid/OPatch/opatch lspatches

34318175;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34318175)

34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)

34139601;ACFS RELEASE UPDATE 19.16.0.0.0 (34139601)

34133642;Database Release Update : 19.16.0.0.220719 (34133642)

33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [+ASM1] ? orcls1

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sh /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch lspatches

34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)

34133642;Database Release Update : 19.16.0.0.220719 (34133642)

OPatch succeeded.

[oracle@oel70-base3 ~]$

</pre>