Reflection of my thoughts and technical expriences

Tuesday, October 25, 2022

Dynamic Services Fallback Option in Oracle 19c

Introduction of Dynamic Services Fallback:

1. A Service fails over to an available instance when there are no preferred instances available.

2. For a dynamic service that is placed using preferred and available settings, you can now specify that the service should fall back to a preferred instance when it becomes available.

3. This option allows for more control in placing dynamic database services and ensures that a given service is available on a preferred instance as much as possible.

Environment Setup:

Primary RAC : orcl (orcl1 and orcl2)
GRID Home : /u01/app/19.0.0/grid
RDBMS Home : /u01/app/oracle/product/19.0.0/dbhome_1
Version : Oracle Database 19c EE - Production Version 19.12.0.0.0

Scan IP : oel70p-scan

Step-1: Check the status of the 'orcl' database.

[oracle@oel70-base1 ~]$ srvctl status database -d orcl

Instance orcl1 is running on node oel70-base1

Instance orcl2 is running on node oel70-base2

[oracle@oel70-base1 ~]$

Step-2: Add the service 'serv1' with 'failback' option and start the service.

Step-3: Check the status of 'serv1' service and shutdown the database, startup the database and check the status of 'serv1' service.

Step-4: srvctl config service does not report state of failback when it is "NO" and created new service 'serv2' checking with 'failback' option enabling and checking the configuration.

[oracle@oel70-base1 ~]$ srvctl config service -d orcl -s serv1

Service name: serv1

Server pool:

Cardinality: 1

Service role: PRIMARY

Management policy: AUTOMATIC

DTP transaction: false

AQ HA notifications: false

Global: false

Commit Outcome: false

Failover type:

Failover method:

Failover retries:

Failover delay:

Failover restore: NONE

Connection Load Balancing Goal: LONG

Runtime Load Balancing Goal: NONE

TAF policy specification: NONE

Edition:

Pluggable database name:

Hub service:

Maximum lag time: ANY

SQL Translation Profile:

Retention: 86400 seconds

Failback : true

Replay Initiation Time: 300 seconds

Drain timeout:

Stop option:

Session State Consistency: DYNAMIC

GSM Flags: 0

Service is enabled

Preferred instances: orcl1

Available instances: orcl2

CSS critical: no

Service uses Java: false

[oracle@oel70-base1 ~]$

Note: Please have a look at these properties

1. Session State Consistency: DYNAMIC

2. Failback : true

Thanks for looking this feature and blog. Happy learning.

Saturday, October 15, 2022

Testing Application Continuity in Oracle 19c RAC (19.12) using Java Application

Introduction to Application Continuity

Application Continuity (AC) is a feature available with the Oracle Real Application Clusters (RAC), Oracle RAC One Node and Oracle Active Data Guard (ADG) options that masks outages from end users and applications by recovering the in-flight database sessions following recoverable outages. Application Continuity performs this recovery beneath the application so that the outage appears to the application as a slightly delayed execution.

Application Continuity improves the user experience for both planned maintenance and unplanned outages. Application Continuity enhances the fault tolerance of systems and applications that use an Oracle database.

Oracle MAA is a collection of architecture, configuration, and life cycle best practices and blueprints. It provides Oracle’s customers with valuable insights and expert recommendations which have been validated and tested working with enterprise customers. This is also an outcome of ongoing communication, with the community of database architects, software engineers, and database strategists, that helps Oracle develop a deep and complete understanding of various kinds of events that can affect availability or data integrity. Over the years, this led to the development and natural evolution of an array of availability reference architectures.

Application Continuity extends the MAA architecture to the application tier.

Environment Setup:

Primary RAC Database: orcl (orcl1 and orcl2)
GRID Home : /u01/app/19.0.0/grid
RDBMS Home: /u01/app/oracle/product/19.0.0/dbhome_1
Version : Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.12.0.0.0
Scan : oel70p-scan

Pre-requisites for Application Continuity:

1. Database and client (driver) version >= Oracle 12c
2. Enable Application Continuity on the Service
[oracle@oel70-base1 ~]$ srvctl modify service -d ORCL -s acservice -failovertype TRANSACTION -commit_outcome TRUE
3. Use the recommended tnsnames.ora or JDBC URL
4. Use Orachk: ($orachk -accheck)
5. Check in AWR Reports: (User Calls in = User Calls Protected)

Before Application Continuity:

Database outages cause in-flight work to be lost, leaving users and applications in-doubt

1. Developer pains
2. Duplicate submissions
3. Cancelled work
4. Errors even when planned
5. Restart applications and mid-tiers

Step-1: Login as 'oracle' user check the database status (orcl)

[oracle@oel70-base1 ~]$ . oraenv
ORACLE_SID = [orcl1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel70-base1
Instance orcl2 is running on node oel70-base2
[oracle@oel70-base1 ~]$

Step-2: Create service (acservice) and check the status of the service

[oracle@oel70-base1 ~]$ srvctl add service -s acservice -d orcl -r orcl1,orcl2
[oracle@oel70-base1 ~]$ srvctl start service -s acservice -d orcl
[oracle@oel70-base1 ~]$ srvctl status service -s acservice -d orcl
Service acservice is running on instance(s) orcl1,orcl2
[oracle@oel70-base1 ~]$

Step-3: Modify the service to check 'Application Continuity' property

[oracle@oel70-base1 ~]$ srvctl modify service -d ORCL -s acservice -failovertype TRANSACTION -replay_init_time 300 -failoverretry 30 -failoverdelay 3 -notification TRUE -commit_outcome TRUE

[oracle@oel70-base1 ~]$

a. failovertype = AUTO or TRANSACTION for Application Continuity
b. commit_outcome = TRUE for Transaction Guard
c. failoverretry = Number of connection retries per replay
d. failoverdelay = Delay in seconds between connection retries

[oracle@oel70-base1 ~]$ srvctl config service -d `srvctl config database` -s ac_service
Service name: ac_service
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: true
Global: false
Commit Outcome: true
Failover type: TRANSACTION
Failover method:
Failover retries:
Failover delay:
Failover restore: LEVEL1
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb1
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: orcl1
Available instances: orcl2
CSS critical: no
Service uses Java: false
[oracle@oel70-base1 ~]$

Step-4: Install JDK as a 'root' user and download ojdbc.jar file and place it to lib folder.

[root@oel70-base1 oracle]# rpm -ivh jdk-12.0.2_linux-x64_bin.rpm
Preparing... ################################# [100%]
Updating / installing...
1:jdk-12.0.2-2000:12.0.2-ga ################################# [100%]
[root@oel70-base1 oracle]#
[root@oel70-base1 oracle]# chmod 755 ojdbc8.jar
[root@oel70-base1 oracle]# mv ojdbc8.jar /lib/

Step-5: Update bash_profile accordingly and execute the bash_profile

[oracle@oel70-base1 ~]$ cat .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export TMP=/tmp
export TMPDIR=$TMP
export JAVA_HOME=/usr/java/jdk-12.0.2/bin
export ORACLE_HOSTNAME=oel70-base1.localdomain
export ORACLE_UNQNAME=orcl
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/19.0.0/grid
export DB_HOME=$ORACLE_BASE/product/19.0.0/dbhome_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=orcl1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH:$JAVA_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:/lib/ojdbc8.jar
[oracle@oel70-base1 ~]$

Step-6: Create 'Actest.java' program to test the functionality of 'Application Continuity'

[oracle@oel70-base1 ~] cat AcTest.java
import java.sql.*;
import oracle.jdbc.*;
import oracle.jdbc.pool.*;
import oracle.jdbc.replay.*;
public class AcTest
{
public static void main(String[] args) throws SQLException,java.lang.InterruptedException
{
oracle.jdbc.replay.OracleDataSource AcDatasource = oracle.jdbc.replay.OracleDataSourceFactory.getOracleDataSource();
AcDatasource.setURL("jdbc:oracle:thin:@racs-scan:1521/acservice");
AcDatasource.setUser("system");
AcDatasource.setPassword("oracle");
Connection conn = AcDatasource.getConnection();
conn.setAutoCommit(false);
PreparedStatement stmt = conn.prepareStatement("select instance_name from v$instance");
ResultSet rset = stmt.executeQuery();
while (rset.next())
{
System.out.println("You are Connected to RAC Instance - "+ rset.getString(1));
}
Thread.currentThread().sleep(60000);
((oracle.jdbc.replay.ReplayableConnection)conn).beginRequest();
PreparedStatement stmt1 = conn.prepareStatement("select instance_name from v$instance");
ResultSet rset1 = stmt1.executeQuery();
while (rset1.next())
{
System.out.println("After Replay Connected to RAC Instance - "+rset1.getString(1));
}
rset.close();
stmt.close();
rset1.close();
stmt1.close();
conn.close();
((oracle.jdbc.replay.ReplayableConnection)conn).endRequest();
}
}
[oracle@oel70-base1]#

Note: Configure connection string and sleep (60000) based on your environment/testing cycles.

Step-7: Login as 'root' user and compile 'Actest.java' program and execute the 'Actest.java' program using 'oracle' user.

[root@oel70-base1 ~]# javac /home/oracle/AcTest.java
[root@oel70-base1 ~]#

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl1
- - -> Waiting

Actest.java program connected to 'orcl1' instance. We will stop instance using srvctl with force option and check the 'Actest.java' program output in the other screen.

[oracle@oel70-base1 ~]$ srvctl stop instance -i orcl1 -d orcl -force

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl1
After Replay Connected to RAC Instance - orcl2
[oracle@oel70-base1 ~]$

Note: It will be reconnected to 'orcl2' instance automatically based on 'Actest.java' program.

Step-8: Execute 'Actest.java' program again and it will be connected to 'orcl2' instance because 'orcl1' is not yet started.

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl2
Start the 'orcl1' instance and check the output of 'Actest.java' program.

[oracle@oel70-base1 ~]$ srvctl start instance -i orcl1 -d orcl

Note: The above command is taking time to start 'orcl1' instance mean time 'Actest.java' program connected to again 'orcl2' instance.

[oracle@oel70-base1 ~]$ java AcTest.java
You are Connected to RAC Instance - orcl2
After Replay Connected to RAC Instance - orcl2
[oracle@oel70-base1 ~]$

Now, both the instances are UP and try to connect 'Actest.java' program.

[oracle@oel70-base1 ~]$ srvctl status database -d orcl
Instance orcl1 is running on node oel70-base1
Instance orcl2 is running on node oel70-base2
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ java AcTest.java

You are Connected to RAC Instance - orcl2

So, it connected to 'orcl2' instance.

Additional notes: Configuring your Service

When using Oracle Database on premise, you can create services on Oracle RAC that use Transparent Application Continuity, or Application Continuity, or TAF. You can use roles to distinguish whether the services are active on Active Data Guard or the primary database. The following examples illustrate this:

Basic Service Creation

$ srvctl add service -db mydb -service MYSERVICE –preferred inst1 -available
inst2 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE -role PRIMARY

Transparent Application Continuity

$ srvctl add service -db mydb -service GOLD -preferred serv1 -available serv2 -
failover_restore AUTO -failoverretry 30 -failoverdelay 10 -commit_outcome TRUE -
failovertype AUTO -replay_init_time 1800 -retention 86400 -notification TRUE -
drain_timeout 300 -stopoption IMMEDIATE

Application Continuity

$ srvctl add service -db mydb -service SILVER -preferred serv1 -available serv2
-failover_restore LEVEL1 -failoverretry 30 -failoverdelay 10 -commit_outcome
TRUE -failovertype TRANSACTION -replay_init_time 1800 -retention 86400 -
notification TRUE -drain_timeout 300 -stopoption IMMEDIATE

Transparent Application Failover

$ srvctl add service -db mydb -service BRONZE -preferred serv1 -available serv2
-failover_restore LEVEL1 -failoverretry 30 -failoverdelay 10 -commit_outcome
TRUE -failovertype SELECT -retention 86400 -notification TRUE -drain_timeout 300
-stopoption IMMEDIATE

To add with the Data Guard role, here is the TAC example:

SQL Queries to check the status of services

Query-1: To report protection by PDB, use the following example:

set pagesize 60
set lines 120
col Service_name format a30 trunc heading "Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
select con_id, service_name,total_requests,
total_calls,total_protected,total_protected*100/NULLIF(total_calls,0) as Protected
from(
select * from
(select a.con_id, a.service_name, c.name,b.value
FROM gv$session a, gv$sesstat b, gv$statname c
WHERE a.sid = b.sid
AND a.inst_id = b.inst_id
AND b.value != 0
AND b.statistic# = c.statistic#
AND b.inst_id = c.inst_id
AND a.service_name not in ('SYS$USERS','SYS$BACKGROUND'))
pivot(
sum(value)
for name in ('cumulative begin requests' as total_requests, 'cumulative end requests' as
Total_end_requests, 'cumulative user calls in requests' as Total_calls, 'cumulative user
calls protected by Application Continuity' as total_protected) ))
order by con_id, service_name;

Query-3: To report protection history over last three days, use the following example:

set lines 85
col Service_name format a30 trunc heading"Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
set lines 85
col Service_name format a30 trunc heading"Service"
break on con_id skip1
col Total_requests format 999,999,9999 heading "Requests"
col Total_calls format 9,999,9999 heading "Calls in requests"
col Total_protected format 9,999,9999 heading "Calls Protected"
col Protected format 999.9 heading "Protected %"
select a.instance_number,begin_interval_time, total_requests, total_calls, total_protected,
total_protected*100/NULLIF(total_calls,0) as Protected
from(
select * from
(select a.snap_id, a.instance_number,a.stat_name, a.value
FROM dba_hist_sysstat a
WHERE a.value != 0 )
pivot(
sum(value)
for stat_name in ('cumulative begin requests' as total_requests, 'cumulative end requests' as
Total_end_requests, 'cumulative user calls in requests' as Total_calls, 'cumulative user calls
protected by Application Continuity' as total_protected)
)) a,
dba_hist_snapshot b
where a.snap_id=b.snap_id
and a.instance_number=b.instance_number
and begin_interval_time>systimestamp - interval '3' day
order by a.snap_id,a.instance_number;

Please refer more details:

https://www.youtube.com/watch?v=kBG5DxZ3szg
https://www.oracle.com/database/technologies/high-availability/app-continuity.html
https://www.youtube.com/watch?v=DOtNjigyjV4

Saturday, August 13, 2022

Goldengate DBLOGIN Issue OCI Error ORA (status = 12545-ORA-12545) and OGG-03542 Failed to connect to the database. Oracle Golden Gate 21c

Oracle Database 12c: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle Golden Gate 21c: /u01/app/ogg/12c_21c

[oracle@primary ~]$ cd /u01/app/ogg/12c_21c/

[oracle@primary 12c_21c]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047_FBO
Oracle Linux 7, x64, 64bit (optimized), Oracle Database 21c and lower supported versions on Jul 29 2021 03:59:23
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.
GGSCI (primary) 1> dblogin userid ogguser@db12c, password oracle
2022-08-13 20:07:33 INFO OGG-03542 Failed to connect to the database. Verify that the connection string and following environment variables are correct:
LD_LIBRARY_PATH = /u01/app/oracle/product/12.1.0.2/dbhome_1/lib:/lib:/usr/lib:/u01/app/ogg/12c_21c.
Error: OCI Error ORA (status = 12154-ORA-12154: TNS:could not resolve the connect identifier specified
)
GGSCI (primary) 2> exit

[oracle@primary ~]$ vi .bash_profile
[oracle@primary ~]$ . ./.bash_profile
[oracle@primary ~]$
[oracle@primary ~]$ cd /u01/app/ogg/12c_21c/

GGSCI (primary) 1> dblogin userid ogguser@db12c, password oracle
Successfully logged into database.
GGSCI (primary as ogguser@db12c) 2>

Solution:
Apart from other entries in .bash_profile added below "TNS_ADMIN" and specified in "LD_LIBRARY_PATH"
export TNS_ADMIN=$ORACLE_BASE/product/12.1.0.2/dbhome_1/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/u01/app/ogg/12c_21c:$TNS_ADMIN

Saturday, July 30, 2022

Applying GRID and RDBMS RU (19.16.0.0.220719) to Oracle 19c (19.12) with two node Primary RAC with Physical Standby RAC

<pre>

Note: I have shown here high level steps for applying Grid RU and RDBMS RU. Please follow "readme" before applying RUs based on your environment with all pre-checks.

Environment:
------------------
Primary RAC: Oracle 19c (19.12) with two node RAC
Physical Standby RAC: Oracle 19c (19.12) with two node RAC

Required Softwares:
--------------------------
Latest Opatch: p6880880_190000_Linux-x86-64.zip
GI Release Update (July 2022): p34130714_190000_Linux-x86-64.zip
Database Release Update (July 2022): p34133642_190000_Linux-x86-64.zip

Check the status of cluster instances in Physical Standby RAC:

---------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 12:19:27 2022

Version 19.12.0.0.0

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> select name,open_mode,database_role from gv$database;

NAME OPEN_MODE DATABASE_ROLE

--------- -------------------- ----------------

ORCL READ ONLY WITH APPLY PHYSICAL STANDBY

[oracle@oel70-base1 ~]$ . oraenv

ORACLE_SID = [orcl1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sat Jul 30 14:36:51 2022

Version 19.12.0.0.0

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/oracle

Connected to "orcl"

Connected as SYSDBA.

DGMGRL>

DGMGRL> show configuration;

Configuration - dg_19c

Protection Mode: MaxPerformance

Members:

orcl - Primary database

orcls - Physical standby database

Fast-Start Failover: Disabled

Configuration Status:

SUCCESS (status updated 2 seconds ago)

DGMGRL>

DGMGRL> edit database orcls set state='APPLY-OFF';

Succeeded.

DGMGRL>

[oracle@oel70-base3 ~]$ . oraenv

ORACLE_SID = [orcls1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl status database -d orcls

Instance orcls1 is running on node oel70-base3

Instance orcls2 is running on node oel70-base4

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jul 30 14:35:37 2022

Version 19.12.0.0.0

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE",

d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME

from GV$DATABASE d, gv$instance i

where i.INST_ID=d.INST_ID;

DATABASE_HOST DB_NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- --------------------

oel70-base3 orcls1 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4 orcls2 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> set lines 200

col DATABASE_HOST for a25;

col DB_UNIQUE_NAME for a25

col HOST_NAME for a15;

col DATABASE_ROLE for a10

col OPEN_MODE for a10

col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",

d.db_unique_name "DB_UNIQUE_NAME" , d.DATABASE_ROLE " DATABASE_ROLE",

d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME

from GV$DATABASE d, gv$instance i

where i.INST_ID=d.INST_ID; 2

DATABASE_HOST DB_NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE STARTUP_TIME

------------------------- ---------------- ------------------------- ---------------- -------------------- -------------------

oel70-base3 orcls1 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

oel70-base4 orcls2 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 29-JUL-22

SQL> select inst_id,process, status, thread#, sequence#, block#, blocks from gv$managed_standby where process='MRP0';

select a.thread#, (select max (sequence#) from v$archived_log where archived='YES' and thread#=a.thread#) archived,max(a.sequence#) applied,

(select max(sequence#) from v$archived_log where archived='YES' and thread#=a.thread#)-max(a.sequence#)gap

from v$archived_log a where a.applied='YES' group by a.thread# order by thread#;

INST_ID PROCESS STATUS THREAD# SEQUENCE# BLOCK# BLOCKS

---------- --------- ------------ ---------- ---------- ---------- ----------

2 MRP0 APPLYING_LOG 2 48 26627 409600

SQL> 2 3

THREAD# ARCHIVED APPLIED GAP

---------- ---------- ---------- ----------

1 57 57 0

2 47 47 0

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.12.0.0.0

[oracle@oel70-base3 ~]$

Unzip required patches with OPatch in both the nodes:
----------------------------------------------------------------------

Standby node-1

[oracle@oel70-base3 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/product/19.0.0/dbhome_1/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base3 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive: p34133642_190000_Linux-x86-64.zip

Standby node-2

[oracle@oel70-base4 ~]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid/
Archive: p6880880_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34130714_190000_Linux-x86-64.zip
Archive: p34130714_190000_Linux-x86-64.zip

[oracle@oel70-base4 ~]$ unzip p34133642_190000_Linux-x86-64.zip
Archive: p34133642_190000_Linux-x86-64.zip

Applying Grid RU and RDBMS RU in Physical Standby Node-1:

-----------------------------------------------------------------------------------

[oracle@oel70-base3 ~]$ su - root

Password:

Last login: Sat Jul 30 14:33:21 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108396

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle 5187 1 0 01:29 ? 00:00:01 asm_pmon_+ASM1

oracle 8157 1 0 01:33 ? 00:00:02 ora_pmon_orcls1

root 24502 24146 0 14:45 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 14:46:45 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_02-46-51PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_02-47-56PM.log

The id for this session is IM3L

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch applicability verified successfully on home /u01/app/19.0.0/grid

Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Executing patch validation checks on home /u01/app/19.0.0/grid

Patch validation checks successfully completed on home /u01/app/19.0.0/grid

Executing patch validation checks on home /u01/app/oracle/product/19.0.0/dbhome_1

Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1

Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0/grid

Prepatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_prepatch_apply_inplace_oel70-base3_2022-07-30_02-51-57PM.log

CRS service brought down successfully on home /u01/app/19.0.0/grid

Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1

Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

Start applying binary patch on home /u01/app/19.0.0/grid

Failed while applying binary patches on home /u01/app/19.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : oel70-base3->/u01/app/19.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.0.0/grid, host: oel70-base3.

Command failed: /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/ -oh /u01/app/19.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc -jre /u01/app/19.0.0/grid/OPatch/jre -persistresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_oel70-base3_crs_1.ser -analyzedresult /u01/app/19.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_oel70-base3_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sat Jul 30 15:00:17 2022

Time taken to complete the session 13 minutes, 32 seconds

opatchauto failed with error code 42

[root@oel70-base3 ~]#

Got the error and checking the opatch logile

----------------------------------------------------------

[oracle@oel70-base3 ~]$ cat /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO] ENTERING METHOD: NApply(patches,options)

[Jul 30, 2022 2:59:09 PM] [INFO] Checking if the Oracle Home /u01/app/19.0.0/grid is good for OPatch to work.

[Jul 30, 2022 2:59:09 PM] [INFO] Option unsupported:false and isCASEnabled :false

[Jul 30, 2022 2:59:09 PM] [INFO] OPatch invoked as follows: 'napply -phBaseFile /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/SDKWork.txt -local '

[Jul 30, 2022 2:59:09 PM] [INFO] Oracle Home : /u01/app/19.0.0/grid

Central Inventory : /u01/app/oraInventory

from : /u01/app/19.0.0/grid/oraInst.loc

OPatch version : 12.2.0.1.32

OUI version : 12.2.0.7.0

OUI location : /u01/app/19.0.0/grid/oui

Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[Jul 30, 2022 2:59:09 PM] [INFO] Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch_history.txt

[Jul 30, 2022 2:59:11 PM] [INFO] [OPSR-TIME] Loading raw inventory

[Jul 30, 2022 2:59:11 PM] [INFO] [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 133 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 135 (MB)

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-TIME] Raw inventory loaded successfully

[Jul 30, 2022 2:59:12 PM] [INFO] NApply::no CAS enabled, OPatch runs with legacy process.

[Jul 30, 2022 2:59:12 PM] [INFO] Verifying environment and performing prerequisite checks...

[Jul 30, 2022 2:59:12 PM] [INFO] [OPSR-TIME] Running prerequisite checks

[Jul 30, 2022 2:59:12 PM] [INFO] opatch-external.jar is in /u01/app/19.0.0/grid/OPatch/jlib/opatch-external.jar

[Jul 30, 2022 2:59:19 PM] [INFO] [OPSR-TIME] Loading cooked inventory

[Jul 30, 2022 2:59:19 PM] [INFO] [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 107 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO] [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 138 (MB)

[Jul 30, 2022 2:59:22 PM] [INFO] [OPSR-TIME] Cooked inventory loaded successfully

[Jul 30, 2022 2:59:45 PM] [INFO] Checking if Oracle Home has components required by patches...

[Jul 30, 2022 2:59:47 PM] [INFO] CheckMissingComps: Cached file does not exist or is invalid, re-build prereq result.

[Jul 30, 2022 2:59:47 PM] [INFO] Checking conflict among patches...

[Jul 30, 2022 2:59:47 PM] [INFO] Running prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches can be applied: 33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches are not required:

[Jul 30, 2022 2:59:48 PM] [INFO] Following patches are auto rollbackable:

[Jul 30, 2022 2:59:48 PM] [INFO] Finished checking prereq checkConflictAmongPatchesWithDetail

[Jul 30, 2022 2:59:48 PM] [INFO] Conflict checking amongst patches PASSED

[Jul 30, 2022 2:59:48 PM] [INFO] Checking conflicts against Oracle Home...

[Jul 30, 2022 2:59:50 PM] [INFO] Running prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:50 PM] [INFO] CUP_LOG: Found pi CUP 34133642 is a superset of poh CUP: 32904851

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches can be applied: 33575402 34133642 34139601 34160635 34318175

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches are not required:

[Jul 30, 2022 2:59:53 PM] [INFO] Following patches are auto rollbackable:

[Jul 30, 2022 2:59:53 PM] [INFO] Finished checking prereq checkConflictAgainstOHWithDetail

[Jul 30, 2022 2:59:56 PM] [INFO] OCM validation is skipped for SDK mode.

[Jul 30, 2022 2:59:58 PM] [INFO] Execution of 'sh /home/oracle/34130714/34133642/custom/scripts/init -apply 34133642 ':

[Jul 30, 2022 2:59:58 PM] [INFO] Return Code = 0

[Jul 30, 2022 2:59:58 PM] [INFO] Execution of ' Command and arguments of init script is restricted to be viewed in log file ':

Return Code = 0

[Jul 30, 2022 3:00:01 PM] [INFO] Running prerequisite checks...

[Jul 30, 2022 3:00:04 PM] [INFO] Space Needed : 10231.319MB

[Jul 30, 2022 3:00:04 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 33575402

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34133642

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34139601

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 34160635

[Jul 30, 2022 3:00:08 PM] [INFO] Prereq checkPatchApplicableOnCurrentPlatform Passed on patch :34318175

[Jul 30, 2022 3:00:11 PM] [WARNING] Action file /u01/app/19.0.0/grid/jlib/srvmasm.jar is in the jar list,OOP should be lanched

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/suptools/oratop/oratop at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/proc at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/sdk/procob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Files in use by a process: /u01/app/19.0.0/grid/lib/libclntsh.so.19.1 PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libsqlplus.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libagtsh.so.1.0 at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/lib/libasmclntsh19.so at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Files in use by a process: /u01/app/19.0.0/grid/lib/libasmclntsh19.so PID( 6136 )

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/oracle at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/rman at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrap at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/renamedg at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/plshprof at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/amdu at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfed at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kfod at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orion at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjob at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/extjobo at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/setasmgid at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:13 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:13 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/lsnrctl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnslsnr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/orapwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sqlldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/proc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/procob at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/wrc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/impdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/ldapsearch at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/expdp at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dgmgrl at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/trcldr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tnsping at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/hsodbc at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/sbttest at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/kgmgr at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dg4pwd at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tkprof at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/nid at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/tstshm at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Start fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Finish fuser command /sbin/fuser /u01/app/19.0.0/grid/bin/dbv at Sat Jul 30 15:00:14 CDT 2022

[Jul 30, 2022 3:00:14 PM] [INFO] Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:14 PM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following active files/executables/libs are used by ORACLE_HOME :/u01/app/19.0.0/grid

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libasmclntsh19.so

[Jul 30, 2022 3:00:16 PM] [INFO] UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

[Jul 30, 2022 3:00:16 PM] [INFO] Finishing UtilSession at Sat Jul 30 15:00:16 CDT 2022

[Jul 30, 2022 3:00:16 PM] [INFO] Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-58-58PM_1.log

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ /sbin/fuser /u01/app/19.0.0/grid/lib/libclntsh.so.19.1

/u01/app/19.0.0/grid/lib/libclntsh.so.19.1: 6136m

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep 6136

oracle 6136 1 0 01:30 ? 00:00:40 /u01/app/19.0.0/grid/jdk/bin/java -server -Xms128M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/u01/app/oracle/crsdata/oel70-base3/qos/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=oel70-base3 -Doracle.http.port=8888 -Doracle.jmx.port=8895 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=false -Djava.security.manager -Djava.security.policy=/u01/app/oracle/crsdata/oel70-base3/qos/conf/catalina.policy -Djava.security.egd=file:/dev/urandom -XX:ParallelGCThreads=2 -Dcatalina.home=/u01/app/19.0.0/grid/tomcat -Dcatalina.base=/u01/app/oracle/crsdata/oel70-base3/qos -Djava.io.tmpdir=/u01/app/oracle/crsdata/oel70-base3/qos/temp -Doracle.home=/u01/app/19.0.0/grid -Doracle.jwc.mode=CLUSTERWARE -classpath /u01/app/19.0.0/grid/jlib/cryptoj.jar:/u01/app/19.0.0/grid/jlib/oraclepki.jar:/u01/app/19.0.0/grid/jlib/osdt_core.jar:/u01/app/19.0.0/grid/jlib/osdt_cert.jar:/u01/app/19.0.0/grid/tomcat/lib/tomcat-juli.jar:/u01/app/19.0.0/grid/tomcat/lib/bootstrap.jar:/u01/app/19.0.0/grid/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

oracle 23827 21496 0 15:13 pts/1 00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ kill -9 6136

[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ ps -ef | grep 6136

oracle 24100 21496 0 15:13 pts/1 00:00:00 grep --color=auto 6136

[oracle@oel70-base3 ~]$

After killing process and resuming the patch:

----------------------------------------------------------

[root@oel70-base3 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto resume

OPatchauto session is initiated at Sat Jul 30 15:33:25 2022

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-33-25PM.log

Resuming existing session with id IM3L

Start applying binary patch on home /u01/app/19.0.0/grid

Binary patch applied successfully on home /u01/app/19.0.0/grid

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0/grid

Postpatch operation log file location: /u01/app/oracle/crsdata/oel70-base3/crsconfig/crs_postpatch_apply_inplace_oel70-base3_2022-07-30_03-39-19PM.log

CRS service started successfully on home /u01/app/19.0.0/grid

Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted

No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1

Skipping SQL patch step execution on standby database : orcls

SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:oel70-base3

RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /home/oracle/34130714/34139601

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/33575402

Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /home/oracle/34130714/34318175

Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/34133642

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_14-54-28PM_1.log

Host:oel70-base3

CRS Home:/u01/app/19.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/34130714/33575402

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34133642

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34139601

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34160635

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patch: /home/oracle/34130714/34318175

Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-07-30_15-33-59PM_1.log

Patching session reported following warning(s):

______________________________________

[WARNING] The database instance 'cdbc1' from '/u01/app/oracle/product/19.0.0/dbhome_1', in host'oel70-base3' is not running. SQL changes, if any, will not be applied.

To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).

Refer to the readme to get the correct steps for applying the sql changes.

OPatchauto session completed at Sat Jul 30 15:46:34 2022

Time taken to complete the session 13 minutes, 10 seconds

[root@oel70-base3 ~]#

Check the process:

------------------------

[root@oel70-base3 ~]# ps -ef | grep pmon

oracle 14834 1 0 15:42 ? 00:00:00 asm_pmon_+ASM1

oracle 15294 1 0 15:43 ? 00:00:00 ora_pmon_orcls1

root 22124 24146 0 15:46 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base3 ~]#

Applying Grid RU and RDBMS RU in Physical Standby Node-2:

-----------------------------------------------------------------------------------

[oracle@oel70-base4 ~]$ su - root

Password:

Last login: Sat Jul 30 15:49:07 CDT 2022

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1659108437

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# . oraenv

ORACLE_SID = [root] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# ps -ef | grep pmon

oracle 7280 1 0 01:32 ? 00:00:01 asm_pmon_+ASM2

oracle 7788 1 0 01:33 ? 00:00:02 ora_pmon_orcls2

root 28280 27450 0 15:50 pts/0 00:00:00 grep --color=auto pmon

[root@oel70-base4 ~]#

[root@oel70-base4 ~]# sh /u01/app/19.0.0/grid/OPatch/opatchauto apply /home/oracle/34130714/

OPatchauto session is initiated at Sat Jul 30 15:51:30 2022

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2022-07-30_03-51-35PM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2022-07-30_03-52-40PM.log

The id for this session is 8SKS