Wednesday, September 15, 2021

Oracle 12cR2/18c/19c - Generate AWRs in Active Data Guard (ADG) Physical Standby Databases

Generate AWRs in Active Data Guard (ADG) Physical Standby Databases

Starting with Oracle 12cR2 (12.2), Automatic Workload Repository (AWR) data can be captured for ADG standby databases. This feature enables analyzing any performance-related issues for ADG standby databases.

Environment: 
------------------

1. Primary RAC: Oracle 19c (19.3) GI, Oracle 19c (19.12) RU and same for RDBMS
2. Standby RAC: Oracle 19c (19.3) GI, Oracle 19c (19.12) RU and same for RDBMS
3. Operating System: Oracle Linux Server 7.9

1. Confirm the open mode and the database role on the standby:

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status from gv$database,gv$instance;

INSTANCE_NAME   DATABASE_ROLE        PROTECTION_MODE      FLASHBACK_ON         STATUS
--------------- -------------------- -------------------- -------------------- ----------
orcls1          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN
orcls2          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN

2. On Primary RAC Database 

The SYS$UMF user is the default database user that has all the privileges to access the system-level Remote Management Framework (RMF) views and tables. 

All the AWR related operations in RMF can be performed only by the SYS$UMF user. The SYS$UMF user is locked by default and it must be unlocked before deploying the RMF topology:

SQL> alter user sys$umf identified by sysumf account unlock;
User altered.

3. Create the database link between the primary database and the standby database and vice versa:

SQL> create database link dbl_orcl_to_orcls CONNECT TO sys$umf IDENTIFIED BY sysumf using 'orcls';
Database link created.

SQL> create database link dbl_orcls_to_orcl CONNECT TO sys$umf IDENTIFIED BY sysumf using 'orcl';
Database link created.

4. We need to configure database nodes to add to the topology. Each database node in a topology must be assigned a unique name (default is DB_UNIQUE_NAME):

Example we have chosen "primary" for primary and "stanby" for standby.

SQL> alter system set "_umf_remote_enabled"=TRUE scope=BOTH;
System altered.

SQL> exec dbms_umf.configure_node ('primary');
PL/SQL procedure successfully completed.

5. The standby database is remote to the target system (the primary database). We can register it via the corresponding database link.

On Physical Standby Database:

SQL> exec dbms_umf.configure_node ('stanby','dbl_orcls_to_orcl');
PL/SQL procedure successfully completed.
SQL>

6. Create the RMF topology:

SQL> exec DBMS_UMF.create_topology ('Topology1');
PL/SQL procedure successfully completed.
SQL>

7. Verify the steps done at primary RAC database

SQL> select * from dba_umf_topology;
TOPOLOGY_NAME    TARGET_ID TOPOLOGY_VERSION TOPOLOGY
--------------- ---------- ---------------- --------
Topology1        509936611                1 ACTIVE

SQL> select * from dba_umf_registration;
TOPOLOGY_NAME   NODE_NAME          NODE_ID  NODE_TYPE AS_SO AS_CA STATE
--------------- --------------- ---------- ---------- ----- ----- --------------------
Topology1       primary          509936611          0 FALSE FALSE OK
SQL>

8. Register the standby database with the topology

SQL> exec DBMS_UMF.register_node ('Topology1', 'stanby', 'dbl_orcl_to_orcls', 'dbl_orcls_to_orcl', 'FALSE', 'FALSE');
PL/SQL procedure successfully completed.
SQL>

9. Enable the AWR service on the node:

SQL> exec DBMS_WORKLOAD_REPOSITORY.register_remote_database(node_name=>'stanby');
PL/SQL procedure successfully completed.

10. Verify all the queries 

SQL> select * from dba_umf_topology;
TOPOLOGY_NAME    TARGET_ID TOPOLOGY_VERSION TOPOLOGY
--------------- ---------- ---------------- --------
Topology1        509936611                7 ACTIVE
SQL>

SQL> select * from dba_umf_registration;
TOPOLOGY_NAME   NODE_NAME          NODE_ID  NODE_TYPE AS_SO AS_CA STATE
--------------- --------------- ---------- ---------- ----- ----- --------------------
Topology1       primary          509936611          0 FALSE FALSE OK
Topology1       stanby          3184583509          0 FALSE FALSE OK
SQL>

SQL> select * from dba_umf_service;
TOPOLOGY_NAME      NODE_ID SERVICE
--------------- ---------- -------
Topology1       3184583509 AWR
SQL>

11. Create a remote snapshot using the RMF

SQL> exec dbms_workload_repository.create_remote_snapshot('stanby');
PL/SQL procedure successfully completed.
Note: We need to run at least two to get the begin_snap and end_snap.

12. Switch the logfiles at Primary RAC Database

SQL> alter system switch all logfile;
System altered.

SQL> alter system switch all logfile;
System altered.

SQL> alter system switch all logfile;
System altered.

13. Login to Physical Standby Database and execute AWR reports

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status from gv$database,gv$instance;

INSTANCE_NAME   DATABASE_ROLE        PROTECTION_MODE      FLASHBACK_ON         STATUS
--------------- -------------------- -------------------- -------------------- ----------
orcls1          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN
orcls2          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN

SQL> @?/rdbms/admin/awrrpti.sql

Specify the Report Type
~~~~~~~~~~~~~~~~~~~~~~~
AWR reports can be generated in the following formats.  Please enter the
name of the format at the prompt. Default value is 'html'.
   'html'          HTML format (default)
   'text'          Text format
   'active-html'   Includes Performance Hub active report

Enter value for report_type: html
Type Specified: html
Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  DB Id      Inst Num   DB Name      Instance     Host
------------ ---------- ---------    ----------   ------
  1610181442     1      ORCL         orcl1        oel70-base1
  1610181442     2      ORCL         orcl2        oel70-base2
  3184583509     1      ORCL         orcls1       oel70-base3
  3184583509     2      ORCL         orcls2       oel70-base4
Enter value for dbid: 3184583509
Using 3184583509 for database Id
Enter value for inst_num: 1
Using 1 for instance number
Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days: 5
Listing the last 5 days of Completed Snapshots
Instance     DB Name      Snap Id       Snap Started    Snap Level
------------ ------------ ---------- ------------------ ----------
orcls1       ORCL                 1  14 Sep 2021 23:44    1
                                  2  15 Sep 2021 01:01    1
Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 1
Begin Snapshot Id specified: 1
Enter value for end_snap: 2
End   Snapshot Id specified: 2
Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_1_1_2.html.  To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: /home/oracle/awrrpt_orcls1.html
14. Execute for Instance-2:

SQL> @?/rdbms/admin/awrrpti.sql

Specify the Report Type
~~~~~~~~~~~~~~~~~~~~~~~
AWR reports can be generated in the following formats.  Please enter the
name of the format at the prompt. Default value is 'html'.
   'html'          HTML format (default)
   'text'          Text format
   'active-html'   Includes Performance Hub active report

Enter value for report_type: html
Type Specified: html
Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  DB Id      Inst Num   DB Name      Instance     Host
------------ ---------- ---------    ----------   ------
  1610181442     1      ORCL         orcl1        oel70-base1
  1610181442     2      ORCL         orcl2        oel70-base2
  3184583509     1      ORCL         orcls1       oel70-base3
  3184583509     2      ORCL         orcls2       oel70-base4
Enter value for dbid: 3184583509
Using 3184583509 for database Id
Enter value for inst_num: 2
Using 2 for instance number
Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days: 2
Listing the last 2 days of Completed Snapshots
Instance     DB Name      Snap Id       Snap Started    Snap Level
------------ ------------ ---------- ------------------ ----------
orcls2       ORCL                 1  14 Sep 2021 23:47    1
                                  2  15 Sep 2021 01:03    1
Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 1
Begin Snapshot Id specified: 1
Enter value for end_snap: 2
End   Snapshot Id specified: 2
Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_2_1_2.html.  To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: /home/oracle/awrrpt_orcls2.html
Using the report name /home/oracle/awrrpt_orcls2.html











Hope it helps....


Please refer for more details:
How to Generate AWRs in Active Data Guard Standby Databases (Doc ID 2409808.1)

Tuesday, September 14, 2021

Oracle 19c Restore Point Replication From Primary Database To Physical Standby Database

Oracle Active Data Guard and Flashback enhancements

1. Today, restore points are set on each Data Guard database individually

    Requires multiple operations if the same restore point across the configuration is desired

2. With Oracle Database 19c, the primary restore points are automatically created on each standby

    Identified by a suffix to the name of “_PRIMARY”


Environment:

Primary RAC: Oracle 19c (19.3) Grid Infrastructure and 19.12 RU
                  Oracle 19c (19.3) RDBMS and 19.12 RU

Physical Standby RAC: Oracle 19c (19.3) Grid Infrastructure and 19.12 RU
                                  Oracle 19c (19.3) RDBMS and 19.12 RU

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status 
from gv$database,gv$instance;

INSTANCE_N DATABASE_ROLE   PROTECTION_MODE      FLASHBACK_ON         STATUS
---------- --------------- -------------------- -------------------- ----------
orcl2      PRIMARY         MAXIMUM PERFORMANCE  NO                   OPEN
orcl1      PRIMARY         MAXIMUM PERFORMANCE  NO                   OPEN
SQL>

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl stop database -d orcl

[oracle@oel70-base1 ~]$ srvctl start database -d orcl -startoption mount

[oracle@oel70-base1 ~]$ . oraenv
ORACLE_SID = [orcl1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:45:09 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database flashback on;
Database altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl stop database -d orcl
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl start database -d orcl
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:50:25 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> set lines 100 pages 100
col status format a10
col instance_name format a15
col database_role format a15
col protection_mode format a20
col flashback_on format a20

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status 
from gv$database,gv$instance;

INSTANCE_NAME   DATABASE_ROLE   PROTECTION_MODE      FLASHBACK_ON         STATUS
--------------- --------------- -------------------- -------------------- ----------
orcl2           PRIMARY         MAXIMUM PERFORMANCE  YES                  OPEN
orcl1           PRIMARY         MAXIMUM PERFORMANCE  YES                  OPEN

SQL> alter system switch all logfile;
System altered.

SQL> alter system switch all logfile;
System altered.

SQL> create restore point grp_test guarantee flashback database;
Restore point created.

SQL> set lines 100 pages 100
SQL> col SCN format 99999999
SQL> col GUARANTEE_FLASHBACK_DATABASE format a10
SQL> col TIME format a20
SQL> col NAME format a10
SQL> col REPLICATED format a10
SQL> col TIME format a32

SQL> select SCN, GUARANTEE_FLASHBACK_DATABASE, TIME, NAME, REPLICATED 
from v$restore_point;

      SCN GUARANTEE_ TIME                             NAME       REPLICATED
--------- ---------- -------------------------------- ---------- ----------
  3190291 YES        14-SEP-21 04.07.09.000000000 PM  GRP_TEST   NO

Physical Standby Database:
-----------------------------------

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status 
from gv$database,gv$instance;

INSTANCE_NAME   DATABASE_ROLE        PROTECTION_MODE      FLASHBACK_ON    STATUS
--------------- -------------------- -------------------- --------------- ----------
orcls2          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  NO              OPEN
orcls1          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  NO              OPEN

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl stop database -d orcls

[oracle@oel70-base3 ~]$ srvctl start database -d orcls -startoption mount

[oracle@oel70-base3 ~]$ . oraenv
ORACLE_SID = [orcls1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:58:11 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database flashback on;
alter database flashback on
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

SQL> alter database recover managed standby database cancel;
Database altered.

SQL> alter database flashback on;
Database altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl stop database -d orcls

[oracle@oel70-base3 ~]$ srvctl start database -d orcls
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 16:02:02 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> set lines 100 pages 100
col status format a10
col instance_name format a15
col database_role format a15
col protection_mode format a20
col flashback_on format a20

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status 
from gv$database,gv$instance;

INSTANCE_NAME   DATABASE_ROLE        PROTECTION_MODE      FLASHBACK_ON         STATUS
--------------- -------------------- -------------------- -------------------- ----------
orcls1          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN
orcls2          PHYSICAL STANDBY     MAXIMUM PERFORMANCE  YES                  OPEN

SQL> set lines 200
col DATABASE_HOST for a25;
col DB_UNIQUE_NAME for a25
col HOST_NAME for a15;
col DATABASE_ROLE for a10
col OPEN_MODE for a10
col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",d.db_unique_name "DB_UNIQUE_NAME", 
d.DATABASE_ROLE " DATABASE_ROLE", d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME
from GV$DATABASE d, gv$instance i where i.INST_ID=d.INST_ID;
select inst_id,process, status, thread#, sequence#, block#, blocks from gv$managed_standby where process='MRP0';
select a.thread#, (select max (sequence#) 
                            from v$archived_log 
                            where archived='YES' and thread#=a.thread#) archived,max(a.sequence#) applied,
                          (select max(sequence#) 
                           from v$archived_log 
                   where archived='YES' and thread#=a.thread#)-max(a.sequence#)gap
from v$archived_log a 
where a.applied='YES' 
group by a.thread# 
order by thread# 

DATABASE_HOST             DB_NAME          DB_UNIQUE_NAME             DATABASE_ROLE    OPEN_MODE           STARTUP_TIME
------------------------- ---------------- ------------------------- ---------------- -------------------- -------------------
oel70-base3               orcls1           orcls                     PHYSICAL STANDBY READ ONLY WITH APPLY 14-SEP-21
oel70-base4               orcls2           orcls                     PHYSICAL STANDBY READ ONLY WITH APPLY 14-SEP-21

SQL>
   INST_ID PROCESS   STATUS        THREAD#  SEQUENCE#     BLOCK#     BLOCKS
---------- --------- ---------- ---------- ---------- ---------- ----------
         1 MRP0      APPLYING_L          1         41       7668     409600
                     OG

   THREAD#   ARCHIVED    APPLIED        GAP
---------- ---------- ---------- ----------
         1         40         40          0
         2         30         30          0
SQL>

SQL> set lines 100 pages 100
col SCN format 99999999
col GUARANTEE_FLASHBACK_DATABASE format a10
col TIME format a33
col NAME format a10
col REPLICATED format a10
SQL> col name format a20

SQL> select SCN, GUARANTEE_FLASHBACK_DATABASE, TIME, NAME, REPLICATED 
from v$restore_point;

      SCN GUARANTEE_ TIME                              NAME                 REPLICATED
--------- ---------- --------------------------------- -------------------- ----------
  3190291 NO         14-SEP-21 04.07.09.000000000 PM   GRP_TEST_PRIMARY     YES
SQL>


DGMGRL> show configuration lag;
Configuration - dg_19c
  Protection Mode: MaxPerformance
  Members:
  orcl  - Primary database
    orcls - Physical standby database
            Transport Lag:      0 seconds (computed 0 seconds ago)
            Apply Lag:          0 seconds (computed 1 second ago)
Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS   (status updated 31 seconds ago)
DGMGRL>

Saturday, September 11, 2021

Prepare a primary database for a Data Guard environment in Oracle 21c (21.3.0)

Prepare a primary database for a Data Guard environment in Oracle 21c (21.3.0)

Environment:
==========
First Container Database: orcl
Pluggable Databases: PDB & PDB1
GI Version: 21c (21.3.0)
RDBMS Version: 21c (21.3.0)

[oracle@rac1-21c ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-21c ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     36852    23188                0           23188              0             N  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304     24568    24172                0           24172              0             Y  OCRVD/
MOUNTED  EXTERN  N         512             512   4096  4194304     36852    33244                0           33244              0             N  RECO/
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ ps -ef | grep pmon
oracle      8272       1  0 14:33 ?        00:00:00 asm_pmon_+ASM1
oracle     11342       1  0 14:34 ?        00:00:00 ora_pmon_orcl1
oracle     69999       1  0 15:10 ?        00:00:00 ora_pmon_orcldb1
oracle     87499   19459  0 15:22 pts/0    00:00:00 grep --color=auto pmon
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcl1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ sqlplus / as sysdba
SQL*Plus: Release 21.0.0.0.0 - Production on Sat Sep 11 15:22:35 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
         4 PDB2                           READ WRITE NO

SQL> archive log list;
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     24
Current log sequence           25

SQL> select force_logging from gv$database;
FORCE_LOGGING
---------------------------------------
NO
NO
NO

SQL> select flashback_on from gv$database;
FLASHBACK_ON
------------------
NO
NO
NO

SQL> exit
Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ dgmgrl /
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Sat Sep 11 15:23:36 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "orcl"
Connected as SYSDG.
DGMGRL>

DGMGRL> help
The following commands are available:
@              Execute DGMGRL script file
!              Host operating system command
/              Repeat the last command
--             Comment to be ignored by DGMGRL
add            Adds a member to the broker configuration
connect        Connects to an Oracle database instance
convert        Converts a database from one type to another
create         Creates a broker configuration or far sync instance
disable        Disables a configuration, a member, or fast-start failover
edit           Edits a configuration or a member
enable         Enables a configuration, a member, or fast-start failover
exit           Exits the program
export         Export Data Guard Broker configuration to a file.
failover       Changes a standby database to be the primary database
help           Displays description and syntax for a command
host           Host operating system command
import         Import Data Guard Broker configuration from a file.
migrate        Migrate a pluggable database from one configuration to another.
prepare        Prepare a primary database for a Data Guard environment.
quit           Exits the program
reinstate      Changes a database marked for reinstatement into a viable standby
rem            Comment to be ignored by DGMGRL
remove         Removes a configuration or a member
set            Set a DGMGRLI CLI property to a specified value
show           Displays information about a configuration or a member
shutdown       Shuts down a currently running Oracle database instance
spool          store input and output of DGMGRL CLI in a file
sql            Executes a SQL statement
start          Starts the fast-start failover observer
startup        Starts an Oracle database instance
stop           Stops the fast-start failover observer
switchover     Switches roles between a primary and standby database
validate       Performs an exhaustive set of validations for a member
Use "help <command>" to see syntax for individual commands

DGMGRL> help prepare
Prepare a primary database for a Data Guard environment.
Syntax:
  PREPARE DATABASE FOR DATA GUARD
    [WITH [DB_UNIQUE_NAME IS <db-unique-name>]
          [DB_RECOVERY_FILE_DEST IS <directory-location>]
          [DB_RECOVERY_FILE_DEST_SIZE IS <size>]
          [BROKER_CONFIG_FILE_1 IS <broker-config-file-1-location>]
          [BROKER_CONFIG_FILE_2 IS <broker-config-file-2-location>]];

DGMGRL>

DGMGRL> PREPARE DATABASE FOR DATA GUARD 
> WITH 
> DB_UNIQUE_NAME IS ORCL 
> DB_RECOVERY_FILE_DEST IS "+RECO" 
> DB_RECOVERY_FILE_DEST_SIZE IS "10G" 
> BROKER_CONFIG_FILE_1 IS "+DATA" 
> BROKER_CONFIG_FILE_2 IS "+RECO";
Preparing database "orcl" for Data Guard.
Primary database must be restarted to enable archivelog mode.
Shutting down database "orcl".
Database stopped.
Starting database "orcl" to mounted mode.
Database started.
Initialization parameter DB_RECOVERY_FILE_DEST_SIZE set to '10G'.
Initialization parameter DB_RECOVERY_FILE_DEST set to '+RECO'.
Initialization parameter DG_BROKER_CONFIG_FILE1 set to '+DATA'.
Initialization parameter DG_BROKER_CONFIG_FILE2 set to '+RECO'.
LOG_ARCHIVE_DEST_n initialization parameter already set for local archival.
Initialization parameter LOG_ARCHIVE_DEST_2 set to 'location=use_db_recovery_file_dest valid_for=(all_logfiles, all_roles)'.
Initialization parameter LOG_ARCHIVE_DEST_STATE_2 set to 'Enable'.
Adding standby log group size 209715200 and assigning it to thread 3.
Adding standby log group size 209715200 and assigning it to thread 3.
Adding standby log group size 209715200 and assigning it to thread 2.
Adding standby log group size 209715200 and assigning it to thread 2.
Adding standby log group size 209715200 and assigning it to thread 1.
Adding standby log group size 209715200 and assigning it to thread 1.
Initialization parameter STANDBY_FILE_MANAGEMENT set to 'AUTO'.
Initialization parameter DG_BROKER_START set to TRUE.
Database set to FORCE LOGGING.
Database set to ARCHIVELOG.
Database set to FLASHBACK ON.
Shutting down database "orcl".
Database stopped.
Starting database "orcl".
Database started.
DGMGRL>


[oracle@rac1-21c ~]$ . oraenv
ORACLE_SID = [orcl1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ sqlplus / as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on Sat Sep 11 15:42:43 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO
         4 PDB2                           READ WRITE NO
SQL>

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     24
Next log sequence to archive   25
Current log sequence           25
SQL>

SQL> select force_logging from gv$database;

FORCE_LOGGING
---------------------------------------
YES
YES
YES

SQL> select flashback_on from gv$database;

FLASHBACK_ON
------------------
YES
YES
YES

SQL>

SQL> select group#,type,member from v$logfile where type like 'STAN%';

GROUP# TYPE            MEMBER
------ --------------- ------------------------------------------------------------
     7 STANDBY         +DATA/ORCL/ONLINELOG/group_7.324.1082993621
     7 STANDBY         +RECO/ORCL/ONLINELOG/group_7.310.1082993621
     8 STANDBY         +DATA/ORCL/ONLINELOG/group_8.325.1082993621
     8 STANDBY         +RECO/ORCL/ONLINELOG/group_8.311.1082993623
     9 STANDBY         +DATA/ORCL/ONLINELOG/group_9.326.1082993623
     9 STANDBY         +RECO/ORCL/ONLINELOG/group_9.312.1082993625
    10 STANDBY         +DATA/ORCL/ONLINELOG/group_10.327.1082993625
    10 STANDBY         +RECO/ORCL/ONLINELOG/group_10.313.1082993625
    11 STANDBY         +DATA/ORCL/ONLINELOG/group_11.328.1082993627
    11 STANDBY         +RECO/ORCL/ONLINELOG/group_11.314.1082993627
    12 STANDBY         +DATA/ORCL/ONLINELOG/group_12.329.1082993629
    12 STANDBY         +RECO/ORCL/ONLINELOG/group_12.315.1082993629

12 rows selected.

SQL> select group#,type,member from v$logfile where type like 'ONLIN%';

GROUP# TYPE            MEMBER
------ --------------- ------------------------------------------------------------
     2 ONLINE          +DATA/ORCL/ONLINELOG/group_2.263.1080574111
     2 ONLINE          +RECO/ORCL/ONLINELOG/group_2.258.1080574111
     1 ONLINE          +DATA/ORCL/ONLINELOG/group_1.262.1080574111
     1 ONLINE          +RECO/ORCL/ONLINELOG/group_1.257.1080574111
     3 ONLINE          +DATA/ORCL/ONLINELOG/group_3.271.1080574361
     3 ONLINE          +RECO/ORCL/ONLINELOG/group_3.259.1080574361
     4 ONLINE          +DATA/ORCL/ONLINELOG/group_4.272.1080574363
     4 ONLINE          +RECO/ORCL/ONLINELOG/group_4.260.1080574363
     5 ONLINE          +DATA/ORCL/ONLINELOG/group_5.273.1080574363
     5 ONLINE          +RECO/ORCL/ONLINELOG/group_5.261.1080574365
     6 ONLINE          +DATA/ORCL/ONLINELOG/group_6.274.1080574365
     6 ONLINE          +RECO/ORCL/ONLINELOG/group_6.262.1080574365

12 rows selected.

SQL>

Friday, September 3, 2021

PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current && PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old

After upgrading (using AutoUpgrade) from Oracle 12c (12.1.0.2) to Oracle 19c (19.3) and applied patch Oracle 19c (19.12) . Below are the messages in alert log.
  • PL/SQL package SYS.DBMS_BACKUP_RESTORE version  is not current
  • PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old
Try to connect rman 

[oracle@primary ~]$ rman target / nocatalog
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Sep 3 04:49:15 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
PL/SQL package SYS.DBMS_BACKUP_RESTORE version 19.03.00.00 in TARGET database is not current
PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 in TARGET database is not current
connected to target database: ORCLDB (DBID=2506522584)
using target database control file instead of recovery catalog
RMAN>

Connect to the target database and execute the following scripts:

[oracle@primary ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 3 04:50:07 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> @$ORACLE_HOME/rdbms/admin/dbmsrman.sql
SQL> @$ORACLE_HOME/rdbms/admin/prvtrmns.plb
SQL> @$ORACLE_HOME/rdbms/admin/dbmsbkrs.sql
SQL> @$ORACLE_HOME/rdbms/admin/prvtbkrs.plb

[oracle@primary ~]$ rman target / nocatalog
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Sep 3 04:50:50 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
connected to target database: ORCLDB (DBID=2506522584)
using target database control file instead of recovery catalog
RMAN>

Hope it helps...

Wednesday, September 1, 2021

CLSRSC-614: failed to get the list of configured diskgroups - Oracle 12cR2 (12.2.0) - Grid Installation

Environment: 

Operating System: Oracle Linux 6.5
RDBMS Version: Oracle 12c (12.2.0) 
RDBMS Patch: p32928749_122010_Linux-x86-64.zip (July 2021)
Instances: Two Node 

Scenario: While executing 'root.sh' as a 'root' user for Oracle 12c R2 (12.2.0) Grid Installation on 2-node RAC. 



Step-1: Node-1: While executing "root.sh" of Grid Installation got the following errors after "step 19 of 19"

2021/09/01 16:03:18 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/09/01 16:08:56 CLSRSC-614: failed to get the list of configured diskgroups
Died at /u01/app/12.2.0/grid/crs/install/oraasm.pm line 2069.
The command '/u01/app/12.2.0/grid/perl/bin/perl -I/u01/app/12.2.0/grid/perl/lib -I/u01/app/12.2.0/grid/crs/install /u01/app/12.2.0/grid/crs/install/rootcrs.pl ' execution failed

Step-2: Check 'asmcmd' login

[root@primary-rac1 ~]# ps -ef | grep pmon
oracle   10473     1  0 16:00 ?        00:00:00 asm_pmon_+ASM1
root     25874  9531  0 16:09 pts/1    00:00:00 grep pmon
[root@primary-rac1 ~]#


[root@primary-rac1 ~]# asmcmd
kgfnGetConnDetails requires 4 parameters at /u01/app/12.2.0/grid/lib/asmcmdbase.pm line 5704.
[root@primary-rac1 ~]#

Step-3: Execute the following in both the nodes one by one.

[root@primary-rac1 ~]# export ORACLE_HOME=/u01/app/12.2.0/grid
[root@primary-rac1 ~]# . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[root@primary-rac1 ~]#

[root@primary-rac1 ~]# /usr/bin/make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk client_sharedlib libasmclntsh12.ohso libasmperl12.ohso ORACLE_HOME=/u01/app/12.2.0/grid

/u01/app/12.2.0/grid/bin/genclntsh
`if [ -d /u01/app/12.2.0/grid/lib32 ]; then rm -f /u01/app/12.2.0/grid/lib32/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib32/libclntsh.so.11.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; else rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; fi `
`if [ -d /u01/app/12.2.0/grid/lib32 ]; then ln -s libclntsh.so /u01/app/12.2.0/grid/lib32/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib32/libclntsh.so.11.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; else ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; fi `
/u01/app/12.2.0/grid/bin/genoccish
/u01/app/12.2.0/grid/bin/genagtsh /u01/app/12.2.0/grid/lib/libagtsh.so 1.0
make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ALWAYS=always /u01/app/12.2.0/grid/lib/libasmclntsh12.so
make[1]: Entering directory `/root'
rm -f /u01/app/12.2.0/grid/lib/libasmclntsh12.so
/u01/app/12.2.0/grid/bin/linkshlib /u01/app/12.2.0/grid/lib/libasmclntsh12.so /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk so ld_shlib LIBS
+ PATH=/bin:/usr/bin:/usr/ccs/bin
+ export PATH
+ lib=/u01/app/12.2.0/grid/lib/libasmclntsh12.so
+ makefile=/u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk
+ so_ext=so
+ target=ld_shlib
++ basename /u01/app/12.2.0/grid/lib/libasmclntsh12.so .so
+ libname=libasmclntsh12
++ dirname /u01/app/12.2.0/grid/lib/libasmclntsh12.so
+ sodir=/u01/app/12.2.0/grid/lib
+ ardir=/u01/app/12.2.0/grid/lib/
+ '[' var = ld_shlib ']'
+ suffix=LIBS
+ var=
+ '[' '!' -f /u01/app/12.2.0/grid/lib/libasmclntsh12.a ']'
+ '[' '' '!=' '' ']'
+ make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ld_shlib _FULL_LIBNAME=/u01/app/12.2.0/grid/lib/libasmclntsh12.so _LIBNAME=libasmclntsh12 _LIBDIR=/u01/app/12.2.0/grid/lib/ '_LIBNAME_LIBS=$(libasmclntsh12LIBS)' '_LIBNAME_EXTRALIBS=$(libasmclntsh12EXTRALIBS)'
make[2]: Entering directory `/root'
/u01/app/12.2.0/grid/bin/orald -o /u01/app/12.2.0/grid/lib/libasmclntsh12.so -shared -z noexecstack -Wl,--disable-new-dtags -L/tmp/bootstraplib/ -L/u01/app/12.2.0/grid/lib/ -L/u01/app/12.2.0/grid/rdbms/lib/ -L/u01/app/12.2.0/grid/lib/stubs/ -Wl,--version-script=/u01/app/12.2.0/grid/rdbms/admin/libasmclntsh12.def -Wl,--whole-archive /u01/app/12.2.0/grid/lib/libasmclntsh12.a -Wl,--no-whole-archive    -lirc
make[2]: Leaving directory `/root'
make[1]: Leaving directory `/root'
make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ALWAYS=always /u01/app/12.2.0/grid/lib/libasmperl12.so
make[1]: Entering directory `/root'
rm -f /u01/app/12.2.0/grid/lib/libasmperl12.so
/u01/app/12.2.0/grid/bin/linkshlib /u01/app/12.2.0/grid/lib/libasmperl12.so /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk so dlopenlib LIBS
+ PATH=/bin:/usr/bin:/usr/ccs/bin
+ export PATH
+ lib=/u01/app/12.2.0/grid/lib/libasmperl12.so
+ makefile=/u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk
+ so_ext=so
+ target=dlopenlib
++ basename /u01/app/12.2.0/grid/lib/libasmperl12.so .so
+ libname=libasmperl12
++ dirname /u01/app/12.2.0/grid/lib/libasmperl12.so
+ sodir=/u01/app/12.2.0/grid/lib
+ ardir=/u01/app/12.2.0/grid/lib/
+ '[' var = dlopenlib ']'
+ suffix=LIBS
+ var=
+ '[' '!' -f /u01/app/12.2.0/grid/lib/libasmperl12.a ']'
+ '[' '' '!=' '' ']'
+ make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk dlopenlib _FULL_LIBNAME=/u01/app/12.2.0/grid/lib/libasmperl12.so _LIBNAME=libasmperl12 _LIBDIR=/u01/app/12.2.0/grid/lib/ '_LIBNAME_LIBS=$(libasmperl12LIBS)' '_LIBNAME_EXTRALIBS=$(libasmperl12EXTRALIBS)'
make[2]: Entering directory `/root'
/u01/app/12.2.0/grid/bin/orald -o /u01/app/12.2.0/grid/lib/libasmperl12.so -shared -z noexecstack -Wl,--disable-new-dtags -L/tmp/bootstraplib/ -L/u01/app/12.2.0/grid/lib/ -L/u01/app/12.2.0/grid/rdbms/lib/ -L/u01/app/12.2.0/grid/lib/stubs/ -Wl,--version-script=/u01/app/12.2.0/grid/rdbms/admin/libasmperl12.def -Wl,--whole-archive /u01/app/12.2.0/grid/lib/libasmperl12.a -Wl,--no-whole-archive -lclntsh -lclntshcore -lasmclntsh12 -lhasgen12 -locr12   -lirc
make[2]: Leaving directory `/root'
make[1]: Leaving directory `/root'
[root@primary-rac1 ~]#

Step-4: Executed again "root.sh" from Node-1

[root@primary-rac1 ~]# /u01/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/primary-rac1/crsconfig/rootcrs_primary-rac1_2021-09-01_04-21-46PM.log
2021/09/01 16:21:52 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/09/01 16:21:52 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2021/09/01 16:25:27 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2021/09/01 16:25:27 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/09/01 16:25:37 CLSRSC-363: User ignored prerequisites during installation
2021/09/01 16:25:37 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/09/01 16:25:43 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/09/01 16:25:49 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2021/09/01 16:25:55 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2021/09/01 16:25:59 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2021/09/01 16:26:01 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2021/09/01 16:26:16 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/09/01 16:26:20 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/09/01 16:26:21 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/09/01 16:26:24 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/09/01 16:26:27 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/09/01 16:26:31 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/09/01 16:26:36 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/09/01 16:26:49 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'primary-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'primary-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.primary-rac1.vip' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'primary-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'primary-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'primary-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'primary-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'primary-rac1'
CRS-2677: Stop of 'ora.cvu' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac1'
CRS-2677: Stop of 'ora.primary-rac1.vip' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'primary-rac1'
CRS-2677: Stop of 'ora.ons' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'primary-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'primary-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'primary-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary-rac1'
CRS-2677: Stop of 'ora.storage' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'primary-rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'primary-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary-rac1'
CRS-2677: Stop of 'ora.ctssd' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'primary-rac1'
CRS-2677: Stop of 'ora.cssd' on 'primary-rac1' succeeded
CRS-2673: Attempting to stop 'ora.driver.afd' on 'primary-rac1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary-rac1'
CRS-2677: Stop of 'ora.driver.afd' on 'primary-rac1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'primary-rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2021/09/01 16:27:18 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'primary-rac1'
CRS-2676: Start of 'ora.mdnsd' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary-rac1'
CRS-2676: Start of 'ora.gpnpd' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'primary-rac1'
CRS-2676: Start of 'ora.gipcd' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary-rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary-rac1'
CRS-2676: Start of 'ora.diskmon' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary-rac1'
CRS-2676: Start of 'ora.ctssd' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary-rac1'
CRS-2676: Start of 'ora.asm' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'primary-rac1'
CRS-2676: Start of 'ora.storage' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'primary-rac1'
CRS-2676: Start of 'ora.crf' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'primary-rac1'
CRS-2676: Start of 'ora.crsd' on 'primary-rac1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 01-SEP-2021 16:28:27
Copyright (c) 1991, 2016, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=23.202.231.168)(PORT=1521)))
TNS-12535: TNS:operation timed out
 TNS-12560: TNS:protocol adapter error
  TNS-00505: Operation timed out
Connecting to (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=23.195.69.108)(PORT=1521)))
CRS-5014: Agent "ORAAGENT" timed out starting process "/u01/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/u01/app/oracle/diag/crs/primary-rac1/crs/trace/crsd_oraagent_oracle.trc"
CRS-5017: The resource action "ora.MGMTLSNR check" encountered the following error:
(:CLSN00009:)Command Aborted. For details refer to "(:CLSN00109:)" in "/u01/app/oracle/diag/crs/primary-rac1/crs/trace/crsd_oraagent_oracle.trc".
CRS-2664: Resource 'ora.OCRVD.dg' is already running on 'primary-rac1'
CRS-6017: Processing resource auto-start for servers: primary-rac1
CRS-2672: Attempting to start 'ora.scan3.vip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.primary-rac1.vip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.ons' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.qosmserver' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.cvu' on 'primary-rac1'
CRS-2676: Start of 'ora.cvu' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1'
CRS-2676: Start of 'ora.scan2.vip' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1'
CRS-2676: Start of 'ora.primary-rac1.vip' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'
CRS-2676: Start of 'ora.ons' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'primary-rac1' succeeded
CRS-6016: Resource auto-start has completed for server primary-rac1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2021/09/01 16:30:57 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/09/01 16:30:57 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/09/01 16:31:03 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/09/01 16:32:09 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@primary-rac1 ~]#

Step-5: Check the asmcmd as a 'oracle' user

[oracle@primary-rac1 ~]$ . oraenv
ORACLE_SID = [orcldb1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@primary-rac1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     40952    40660                0           40660              0             Y  OCRVD/
[oracle@primary-rac1 ~]$

Step-6: Execute "root.sh" in Node-2 

[root@primary-rac2 grid]# /u01/app/12.2.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/primary-rac2/crsconfig/rootcrs_primary-rac2_2021-09-01_04-43-15PM.log
2021/09/01 16:43:18 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/09/01 16:43:18 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2021/09/01 16:48:16 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2021/09/01 16:48:16 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/09/01 16:48:19 CLSRSC-363: User ignored prerequisites during installation
2021/09/01 16:48:19 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/09/01 16:48:20 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/09/01 16:48:20 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2021/09/01 16:48:22 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2021/09/01 16:48:23 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2021/09/01 16:48:24 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2021/09/01 16:48:26 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/09/01 16:48:27 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/09/01 16:49:13 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/09/01 16:49:14 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/09/01 16:49:15 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/09/01 16:49:17 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2021/09/01 16:49:34 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/09/01 16:49:36 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary-rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'primary-rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2021/09/01 16:49:45 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'primary-rac2'
CRS-2676: Start of 'ora.evmd' on 'primary-rac2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary-rac2'
CRS-2676: Start of 'ora.gpnpd' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'primary-rac2'
CRS-2676: Start of 'ora.gipcd' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary-rac2'
CRS-2676: Start of 'ora.diskmon' on 'primary-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary-rac2'
CRS-2676: Start of 'ora.ctssd' on 'primary-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary-rac2'
CRS-2676: Start of 'ora.asm' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'primary-rac2'
CRS-2676: Start of 'ora.storage' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'primary-rac2'
CRS-2676: Start of 'ora.crf' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'primary-rac2'
CRS-2676: Start of 'ora.crsd' on 'primary-rac2' succeeded
CRS-6017: Processing resource auto-start for servers: primary-rac2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.net1.network' on 'primary-rac2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded
CRS-2676: Start of 'ora.net1.network' on 'primary-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'primary-rac1'
CRS-2672: Attempting to start 'ora.ons' on 'primary-rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'primary-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary-rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac2'
CRS-2676: Start of 'ora.ons' on 'primary-rac2' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary-rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac2' succeeded
CRS-2676: Start of 'ora.asm' on 'primary-rac2' succeeded
CRS-6016: Resource auto-start has completed for server primary-rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2021/09/01 16:53:15 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/09/01 16:53:15 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/09/01 16:53:29 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/09/01 16:53:42 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
You have mail in /var/spool/mail/root
[root@primary-rac2 grid]#

Please refer for more details:

ASMCMD Failing With "KGFNGETCONNDETAILS Requires 4 Parameters at <GI_HOME>/lib/asmcmdbase.pm (Doc ID 2748316.1)


Hope it helps...