Reflection of my thoughts and technical expriences

Sunday, October 3, 2021

Oracle 19c (19.12) - Oracle ASM Cluster File System (ACFS) Replication.

Oracle 19c - Oracle ASM Cluster File System (ACFS) Replication.

Primary RAC:
------------------
a. Operating System: Oracle Linux 8.2
b. Kernel: 5.4.17-2011.1.2.el8uek.x86_64
c. Grid Infra: Oracle 19c (19.3) + RU 19.12

Standby RAC:
-------------------
a. Operating System: Oracle Linux 8.2
b. Kernel: 5.4.17-2011.1.2.el8uek.x86_64
c. Grid Infra: Oracle 19c (19.3) + RU 19.12

Oracle ACFS, as part of Oracle Grid Infrastructure, is integrated with Oracle ASM, Oracle ADVM and Oracle Clusterware. Oracle ACFS will provide the following capabilities.

» Security
» Audit
» Compression
» Snapshots
» Replication:
» Tagging
» Plugins
» Accelerator Volumes
» NAS Maximum Availability eXtensions
» ACFS Remote Service

ACFS Replication:

Oracle ACFS provides the capability to replicate an entire file system from one location to a different location so that customers can provide for test and development and disaster recovery. Starting in Oracle 12cR2, Oracle ACFS replication evolved to a snapshot-based replication solution, enhancing its already existing capabilities and providing maximum efficiency and flexibility in its replication method. Starting in Oracle 18c, Oracle ACFS introduces role reversal, allowing a standby site to become an active site and the active site to become a standby site in case this is required.
The primary and standby sites can have differing configurations; i.e., primary can be multi-node cluster and standby can be a single-node cluster.

Oracle 19c (19.5) release, replication supports both switchover and failover. In these operations, the existing standby is converted to be the new primary, and the existing primary may either become the new standby, or may be replaced as the new standby by a different storage location.
In all cases, the standby file system is a read-only file system. One use-case for the standby file system is that it can be the source for backups.

Note that primary site running Linux, Solaris or AIX can replicate to a standby site running any of those operating systems

Configuring Password-less SSH login

In the password-less SSH login method, we’re going to exchange encrypted keys instead of entering the actual password while connecting to the remote systems using SSH. So, nobody can easily hack or guess your password, because we don’t use passwords any more to access the remote systems. And more importantly, hidden key-loggers and brute-force attacks do not work if we use Password-less SSH.

Primary RAC (oel82-rac1 & oel82-rac2)
------------------------------------------------------------
Step-1: If the public key file does not exist, generate a public / private key pair on the primary by running this command as “root”:

[root@oel82-rac1 ~]# cd /root/.ssh
-bash: cd: /root/.ssh: No such file or directory
[root@oel82-rac1 ~]# pwd
/root
[root@oel82-rac1 ~]# mkdir ~/.ssh
[root@oel82-rac1 ~]# cd .ssh/
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# pwd
/root/.ssh

[root@oel82-rac1 .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:ahbBDHCCaM7Jfr0rrcsnjfK4aZhzqm/qdANiHR25BVM root@oel82-rac1.localdomain
The key's randomart image is:
+---[RSA 3072]----+
|..o.o++E |
|o. o.=o. |
|= .. .* |
| =. .. . |
|oo .. . S |
|o... . o |
|.o.o+ = |
|=oB+.B |
|B%==*.. |
+----[SHA256]-----+
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# ls -la
total 12
drwxr-xr-x. 2 root root 38 Sep 28 14:31 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-------. 1 root root 2610 Sep 28 14:31 id_rsa
-rw-r--r--. 1 root root 581 Sep 28 14:31 id_rsa.pub
[root@oel82-rac1 .ssh]# cat id_rsa.pub >> authorized_keys
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# ls -la
total 16
drwxr-xr-x. 2 root root 61 Sep 28 14:32 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-r--r--. 1 root root 581 Sep 28 14:32 authorized_keys
-rw-------. 1 root root 2610 Sep 28 14:31 id_rsa
-rw-r--r--. 1 root root 581 Sep 28 14:31 id_rsa.pub

Step-2: Copy the "authorized_keys" from oel82-rac1 to oel82-rac2 node.

[root@oel82-rac1 .ssh]# scp authorized_keys root@oel82-rac2:/root/.ssh
The authenticity of host 'oel82-rac2 (192.168.2.20)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac2,192.168.2.20' (ECDSA) to the list of known hosts.
root@oel82-rac2's password:
authorized_keys 100% 581 0.5KB/s 00:01
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# exit
logout
[oracle@oel82-rac1 ~]$

Step-3: If the public key file does not exist, generate a public / private key pair on the primary by running this command as “root”:

[oracle@oel82-rac2 ~]$ su - root
Password:
[root@oel82-rac2 ~]# pwd
/root
[root@oel82-rac2 ~]# mkdir ~/.ssh
[root@oel82-rac2 ~]# cd .ssh/
[root@oel82-rac2 .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:L6CIjvWl173vMYPckAobl1VWznW0Ez0omGvz2dHfX8c root@oel82-rac2
The key's randomart image is:
+---[RSA 3072]----+
| o o.oo=|
| o + + o=|
| o . +o.|
| * . . ..|
| + S = o ..o|
| . . . * + * . E|
|. o . o.o.+ = +|
|o. . o. ... + .|
|.. o. .+o |
+----[SHA256]-----+
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# ls -la
total 12
drwxr-xr-x. 2 root root 38 Sep 28 14:32 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-------. 1 root root 2602 Sep 28 14:32 id_rsa
-rw-r--r--. 1 root root 569 Sep 28 14:32 id_rsa.pub
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# ls -la
total 16
drwxr-xr-x. 2 root root 61 Sep 28 14:34 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-r--r--. 1 root root 581 Sep 28 14:34 authorized_keys
-rw-------. 1 root root 2602 Sep 28 14:32 id_rsa
-rw-r--r--. 1 root root 569 Sep 28 14:32 id_rsa.pub
[root@oel82-rac2 .ssh]# cat id_rsa.pub >> authorized_keys
[root@oel82-rac2 .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2

Standby RAC (oel82-rac3 & oel82-rac4)
--------------------------------------------------
Step-4:

[root@oel82-rac2 .ssh]# scp authorized_keys oracle@oel82-rac3:/home/oracle/.ssh
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 1150 0.4KB/s 00:03
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# cd
[root@oel82-rac2 ~]#
[root@oel82-rac2 ~]# vi /etc/fstab
[root@oel82-rac2 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac2 ~]#

[root@oel82-rac2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.7G 8.0K 4.7G 1% /dev
tmpfs 4.8G 845M 4.0G 18% /dev/shm
tmpfs 4.8G 18M 4.8G 1% /run
tmpfs 4.8G 0 4.8G 0% /sys/fs/cgroup
/dev/mapper/ol-root 50G 19G 32G 38% /
/dev/mapper/ol-home 65G 607M 64G 1% /home
/dev/sda1 1014M 295M 720M 30% /boot
tmpfs 968M 1.2M 967M 1% /run/user/42
tmpfs 968M 4.0K 968M 1% /run/user/54321
/dev/asm/acfs_vol-204 10G 570M 9.5G 6% /acfs_vol
[root@oel82-rac2 ~]#
[root@oel82-rac2 ~]# exit
logout
[oracle@oel82-rac2 ~]$ pwd
/home/oracle
[oracle@oel82-rac2 ~]$
[oracle@oel82-rac2 ~]$ cd .ssh/
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 10:29 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-r--r--. 1 oracle oinstall 1737 Sep 28 15:12 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 10:28 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 10:28 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 738 Sep 28 10:31 known_hosts
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac2 .ssh]$

Step-5:

-------------

[oracle@oel82-rac2 .ssh]$ scp authorized_keys oel82-rac3:/home/oracle/.ssh/
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 2308 0.2KB/s 00:12
[oracle@oel82-rac2 .ssh]$
Step-5:
[oracle@oel82-rac3 ~]$ su - root
Password:
pwd
[root@oel82-rac3 ~]# pwd
/root
[root@oel82-rac3 ~]# mkdir ~/.ssh
[root@oel82-rac3 ~]# exit
logout
[oracle@oel82-rac3 ~]$ clear
[oracle@oel82-rac3 ~]$ pwd
/home/oracle
[oracle@oel82-rac3 ~]$ cd .ssh/
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1142 Sep 28 13:19 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1150 Sep 28 14:36 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3

Step-6:

-----------

[oracle@oel82-rac3 .ssh]$ scp authorized_keys oracle@oel82-rac4:/home/oracle/.ssh

authorized_keys 100% 1721 0.4KB/s 00:03
[oracle@oel82-rac3 .ssh]$
Step-6
[oracle@oel82-rac4 ~]$ su - root
Password:
pwd
[root@oel82-rac4 ~]# pwd
/root
[root@oel82-rac4 ~]# mkdir ~/.ssh
[root@oel82-rac4 ~]# exit
logout
[oracle@oel82-rac4 ~]$ cd .ssh/
[oracle@oel82-rac4 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1142 Sep 28 13:19 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-------. 1 oracle oinstall 1694 Sep 28 13:19 known_hosts
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6wxULv13UOhUrRy/5dN5NiY48KAM/xLAsO4d0Ea+pdaLTWRt2hreGhE4eEuRZEes4BP5eoVSv/HvrS3xvBJcZVZLDx8wktVODqFH9P9ggLRpi/fLn20memLNz4EwFJt5DRv2UJAeJIQbycHUxt8N/R8h1z8H39V7KVbbUC+PZW6reVYgSSoyczwFj5tdzalwwoPPf4OwyYPb6EXhHWQZ5jFgwxlNfQaS7jGzcjgvKrNDVy+x9Lam7KrMB3IbhaCwZ0i0UrcFusF3PocONK0xbRThY4cztyVKyKVCkpG8/2VCv7yO2EUGc9P4f37DGj6D5WtBvCmWCqJ5S3AwRyyG7ZxWqSlvSjL6OGHFP2xatkr96jq456fbfejL8ZF7UHA886e/OZub1H6zwCJGnEe47oebhyBPrGEWukisZ5YttTuYzOOVWp5MclCaGxGRog10sJrutcX1+4dNqseFaZ11nubwJrLnFIL8Op+JndlewQotGjdgo4dKDF8OpMzuR0/s= oracle@oel82-rac4
[oracle@oel82-rac4 .ssh]$ cd
[oracle@oel82-rac4 ~]$ su - root
Password:
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# vi /etc/fstab
[root@oel82-rac4 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# exit
logout

Step-7: Create volume "acfs_vol" in Primary RAC
-------------------------------------------------------------
[oracle@oel82-rac1 ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 12284 12144 0 12144 0 N ACFSDG/
MOUNTED EXTERN N 512 512 4096 4194304 12284 11944 0 11944 0 Y OCRVD/
ASMCMD> volcreate -G ACFSDG -s 10G acfs_vol
ASMCMD> volinfo --all
Diskgroup Name: ACFSDG
Volume Name: ACFS_VOL
Volume Device: /dev/asm/acfs_vol-204
State: ENABLED
Size (MB): 10240
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
ASMCMD>
ASMCMD> exit
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ su - root
Password:
[root@oel82-rac1 ~]# . oraenv
ORACLE_SID = [/u01/app/19.3.0/grid] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been changed from /home/oracle to /u01/app/oracle
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# modprobe oracleacfs
[root@oel82-rac1 ~]# modprobe oracleadvm
[root@oel82-rac1 ~]# modprobe oracleoks
[root@oel82-rac1 ~]# lsmod | grep oracle
oracleacfs 5025792 1
oracleadvm 712704 7
oracleoks 733184 2 oracleadvm,oracleacfs
oracleafd 221184 1
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# cd /
[root@oel82-rac1 /]# mkdir acfs_vol
[root@oel82-rac1 /]# chown -R oracle:oinstall /acfs_vol
[root@oel82-rac1 /]# mkfs -t acfs /dev/asm/acfs_vol-204
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/acfs_vol-204
mkfs.acfs: volume size = 10737418240 ( 10.00 GB )
mkfs.acfs: Format complete.
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# acfsutil registry -a /dev/asm/acfs_vol-204 /acfs_vol/
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# acfsutil registry -c /dev/asm/acfs_vol-204 /acfs_vol/ -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-204'
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# mount -t acfs /dev/asm/acfs_vol-204 /acfs_vol/
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# mount | grep acfs
/dev/asm/acfs_vol-204 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# vi /etc/fstab
[root@oel82-rac1 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# srvctl config filesystem -d /dev/asm/acfs_vol-204
Volume device: /dev/asm/acfs_vol-204
Diskgroup name: acfsdg
Volume name: acfs_vol
Canonical volume device: /dev/asm/acfs_vol-204
Accelerator volume devices:
Mountpoint path: /acfs_vol
Mount point owner: root
Mount point group: root
Mount permissions: owner:root:rwx,pgrp:root:r-x,other::r-x
Mount users:
Type: ACFS
Mount options:
Description:
ACFS file system is enabled
ACFS file system is individually enabled on nodes:
ACFS file system is individually disabled on nodes:
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# srvctl status filesystem -d /dev/asm/acfs_vol-204
ACFS file system /acfs_vol is mounted on nodes oel82-rac1,oel82-rac2
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.7G 8.0K 4.7G 1% /dev
tmpfs 4.8G 845M 4.0G 18% /dev/shm
tmpfs 4.8G 18M 4.8G 1% /run
tmpfs 4.8G 0 4.8G 0% /sys/fs/cgroup
/dev/mapper/ol-root 50G 19G 32G 38% /
/dev/mapper/ol-home 65G 607M 64G 1% /home
/dev/sda1 1014M 295M 720M 30% /boot
tmpfs 968M 1.2M 967M 1% /run/user/42
tmpfs 968M 4.0K 968M 1% /run/user/54321
/dev/asm/acfs_vol-204 10G 570M 9.5G 6% /acfs_vol
[root@oel82-rac1 /]#

Step-8:
----------
[root@oel82-rac2 ~]# vi /etc/fstab
[root@oel82-rac2 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac2 ~]#

Step-9: Create volume "acfs_vol" in Standby RAC
--------------------------------------------------------------------
[oracle@oel82-rac3 ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 12284 12144 0 12144 0 N ACFSDG/
MOUNTED EXTERN N 512 512 4096 4194304 12284 11944 0 11944 0 Y OCRVD/
ASMCMD>

ASMCMD> volcreate -G ACFSDG -s 10G acfs_vol
ASMCMD> volinfo --all
Diskgroup Name: ACFSDG
Volume Name: ACFS_VOL
Volume Device: /dev/asm/acfs_vol-211
State: ENABLED
Size (MB): 10240
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
ASMCMD>
ASMCMD> exit
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ su - root
Password:
[root@oel82-rac3 ~]#
[root@oel82-rac3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel82-rac3 ~]#

[root@oel82-rac3 ~]# modprobe oracleacfs
[root@oel82-rac3 ~]# modprobe oracleadvm
[root@oel82-rac3 ~]# modprobe oracleoks
[root@oel82-rac3 ~]# lsmod | grep oracle
oracleacfs 5025792 1
oracleadvm 712704 7
oracleoks 733184 2 oracleadvm,oracleacfs
oracleafd 221184 1
[root@oel82-rac3 ~]#
[root@oel82-rac3 ~]# cd /

[root@oel82-rac3 /]# mkdir acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# chown -R oracle:oinstall /acfs_vol
[root@oel82-rac3 /]# chmod -R 777 /acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# mkfs -t acfs /dev/asm/acfs_vol-211
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/acfs_vol-211
mkfs.acfs: volume size = 10737418240 ( 10.00 GB )
mkfs.acfs: Format complete.
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# acfsutil registry -a /dev/asm/acfs_vol-211 /acfs_vol/
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# acfsutil registry -c /dev/asm/acfs_vol-211 /acfs_vol/ -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-211'
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# mount -t acfs /dev/asm/acfs_vol-211 /acfs_vol/
[root@oel82-rac3 /]#
[root@oel82-rac3 /]# mount | grep acfs
/dev/asm/acfs_vol-211 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# vi /etc/fstab
[root@oel82-rac3 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl config filesystem -d /dev/asm/acfs_vol-211
Volume device: /dev/asm/acfs_vol-211
Diskgroup name: acfsdg
Volume name: acfs_vol
Canonical volume device: /dev/asm/acfs_vol-211
Accelerator volume devices:
Mountpoint path: /acfs_vol
Mount point owner: root
Mount point group: root
Mount permissions: owner:root:rwx,pgrp:root:r-x,other::r-x
Mount users:
Type: ACFS
Mount options:
Description:
ACFS file system is enabled
ACFS file system is individually enabled on nodes:
ACFS file system is individually disabled on nodes:
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl status filesystem -d /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3,oel82-rac4
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl stop filesystem -d /dev/asm/acfs_vol-211 -n oel82-rac4
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl status filesystem -d /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3
[root@oel82-rac3 /]#

Step-10:
-----------
[oracle@oel82-rac4 ~]$ su - root
Password:
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# vi /etc/fstab
[root@oel82-rac4 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac4 ~]#

Step-11: Initiate acfs replication in standby
-------------------------------------------------------
[root@oel82-rac3 /]# /sbin/acfsutil repl init standby -u oracle /acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:00:38 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:00:38 2021
Number of Events: 1
[root@oel82-rac3 /]#

Step-12: Start acfs replication in primary rac
--------------------------------------------------------------
[root@oel82-rac1 /]# /sbin/acfsutil repl info -c -v -u oracle oel82-rac3 /acfs_vol/
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
' -- retrying].
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
' -- retrying].
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
'].
acfsutil repl info: ACFS-05914: user root on node oel82-rac1 unable to log in via 'ssh' to the standby node oel82-rac3 as user oracle
[root@oel82-rac1 /]#
[root@oel82-rac1 /]# exit
logout
[oracle@oel82-rac1 ~]$

Note: I have got the above error and I have copied "keys" for Oracle user from primary rac to standby rac

[oracle@oel82-rac1 ~]$ pwd
/home/oracle
[oracle@oel82-rac1 ~]$ cd .ssh/
[oracle@oel82-rac1 .ssh]$
[oracle@oel82-rac1 .ssh]$ ls -la
total 28
drwx------. 2 oracle oinstall 134 Sep 28 10:35 .
drwx------. 7 oracle oinstall 4096 Sep 28 15:06 ..
-rw-r--r--. 1 oracle oinstall 1154 Sep 28 10:29 authorized_keys
-rw-------. 1 oracle oinstall 2622 Sep 28 10:26 id_rsa
-rw-r--r--. 1 oracle oinstall 583 Sep 28 10:26 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 1292 Sep 28 15:08 known_hosts
-rw-------. 1 oracle oinstall 2622 Sep 28 10:35 oel82-rac1.id_rsa
-rw-r--r--. 1 oracle oinstall 583 Sep 28 10:35 oel82-rac1.id_rsa.pub

[oracle@oel82-rac1 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
[oracle@oel82-rac1 .ssh]$

[oracle@oel82-rac1 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac1 .ssh]$

[oracle@oel82-rac1 .ssh]$ scp authorized_keys oel82-rac2:/home/oracle/.ssh/
authorized_keys 100% 1737 13.2KB/s 00:00
[oracle@oel82-rac1 .ssh]$
[oracle@oel82-rac2 ~]$ pwd
/home/oracle
[oracle@oel82-rac2 ~]$
[oracle@oel82-rac2 ~]$ cd .ssh/
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 10:29 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-r--r--. 1 oracle oinstall 1737 Sep 28 15:12 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 10:28 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 10:28 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 738 Sep 28 10:31 known_hosts
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac2 .ssh]$

[oracle@oel82-rac2 .ssh]$ scp authorized_keys oel82-rac3:/home/oracle/.ssh/
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 2308 0.2KB/s 00:12
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ cd .ssh/
[oracle@oel82-rac3 .ssh]$ ls -l
total 16
-rw-------. 1 oracle oinstall 1721 Sep 28 14:38 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$

[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2308 Sep 28 15:13 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ scp authorized_keys oel82-rac4:/home/oracle/.ssh/
authorized_keys 100% 2879 18.0KB/s 00:00
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$

[oracle@oel82-rac4 ~]$ cd .ssh/
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:58 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-------. 1 oracle oinstall 1694 Sep 28 13:19 known_hosts
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6wxULv13UOhUrRy/5dN5NiY48KAM/xLAsO4d0Ea+pdaLTWRt2hreGhE4eEuRZEes4BP5eoVSv/HvrS3xvBJcZVZLDx8wktVODqFH9P9ggLRpi/fLn20memLNz4EwFJt5DRv2UJAeJIQbycHUxt8N/R8h1z8H39V7KVbbUC+PZW6reVYgSSoyczwFj5tdzalwwoPPf4OwyYPb6EXhHWQZ5jFgwxlNfQaS7jGzcjgvKrNDVy+x9Lam7KrMB3IbhaCwZ0i0UrcFusF3PocONK0xbRThY4cztyVKyKVCkpG8/2VCv7yO2EUGc9P4f37DGj6D5WtBvCmWCqJ5S3AwRyyG7ZxWqSlvSjL6OGHFP2xatkr96jq456fbfejL8ZF7UHA886e/OZub1H6zwCJGnEe47oebhyBPrGEWukisZ5YttTuYzOOVWp5MclCaGxGRog10sJrutcX1+4dNqseFaZ11nubwJrLnFIL8Op+JndlewQotGjdgo4dKDF8OpMzuR0/s= oracle@oel82-rac4
[oracle@oel82-rac4 .ssh]$

Step-13:
------------
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac2 date
Tue Sep 28 15:15:38 CDT 2021
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac3 date
Tue Sep 28 15:15:51 CDT 2021
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac4 date
Tue Sep 28 15:16:02 CDT 2021
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c -v -u oracle oel82-rac3 /acfs_vol/
A valid 'ssh' connection was detected for standby node oel82-rac3 as user oracle.
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl init primary -C -s oracle@oel82-rac3 -m /acfs_vol/ /acfs_vol/

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 3
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ echo "Test ACFS Replication" > /acfs_vol/replication_testing.txt
[oracle@oel82-rac1 ~]$ cd /acfs_vol/
[oracle@oel82-rac1 acfs_vol]$ ls -la
total 100
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:20 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:47 ..
drwxrwxrwx. 2 oracle oinstall 65536 Sep 28 14:48 lost+found
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac1 acfs_vol]$

[oracle@oel82-rac1 acfs_vol]$ cat replication_testing.txt

Test ACFS Replication

[oracle@oel82-rac1 acfs_vol]$

[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 20
[oracle@oel82-rac1 acfs_vol]$

Step-14:
-----------
[oracle@oel82-rac3 /]$ cd acfs_vol/
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$ ls -la
total 96
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:20 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:55 ..
drwx------. 2 root root 65536 Sep 28 14:56 lost+found
[oracle@oel82-rac3 acfs_vol]$ ls
lost+found
[oracle@oel82-rac3 acfs_vol]$ ls
lost+found
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$ ls -la
total 100
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:21 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:55 ..
drwx------. 2 root root 65536 Sep 28 14:56 lost+found
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac3 acfs_vol]$

[oracle@oel82-rac3 acfs_vol]$ cat replication_testing.txt
Test ACFS Replication
[oracle@oel82-rac3 acfs_vol]$

[oracle@oel82-rac3 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:21:35 2021
Apply completed at: Tue Sep 28 15:21:35 2021
Time: Tue Sep 28 15:21:34 2021
Event: Clone applied
Session started at: Tue Sep 28 15:21:34 2021
Session ended at: Tue Sep 28 15:21:34 2021
Number of bytes applied: 23148
Time: Tue Sep 28 15:21:17 2021
Apply started at: Tue Sep 28 15:21:17 2021
Time: Tue Sep 28 15:20:45 2021
Apply completed at: Tue Sep 28 15:20:45 2021
Time: Tue Sep 28 15:20:44 2021
Event: Clone applied
Session started at: Tue Sep 28 15:20:44 2021
Session ended at: Tue Sep 28 15:20:44 2021
Number of bytes applied: 21114
Time: Tue Sep 28 15:20:44 2021
Apply started at: Tue Sep 28 15:20:44 2021
Time: Tue Sep 28 15:20:21 2021
Apply completed at: Tue Sep 28 15:20:21 2021
Time: Tue Sep 28 15:20:20 2021
Event: Clone applied
Session started at: Tue Sep 28 15:20:17 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes applied: 21114
Time: Tue Sep 28 15:20:17 2021
Apply started at: Tue Sep 28 15:20:17 2021
Time: Tue Sep 28 15:18:48 2021
Apply completed at: Tue Sep 28 15:18:48 2021
Time: Tue Sep 28 15:18:48 2021
Event: Clone applied
Session started at: Tue Sep 28 15:18:48 2021
Session ended at: Tue Sep 28 15:18:48 2021
Number of bytes applied: 21499
Time: Tue Sep 28 15:18:48 2021
Apply started at: Tue Sep 28 15:18:48 2021
Time: Tue Sep 28 15:00:38 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:00:38 2021
Number of Events: 13
[oracle@oel82-rac3 acfs_vol]$

Step-15: Testing replication:
-------------------------------------
[oracle@oel82-rac1 acfs_vol]$ pwd
/acfs_vol
[oracle@oel82-rac1 acfs_vol]$ cd
[oracle@oel82-rac1 ~]$ ls
p6880880_190000_Linux-x86-64.zip PatchSearch.xml
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ cp p6880880_190000_Linux-x86-64.zip /acfs_vol/
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ cd /acfs_vol/
[oracle@oel82-rac1 acfs_vol]$
[oracle@oel82-rac1 acfs_vol]$ ls -la
total 122960
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:25 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:47 ..
drwxrwxrwx. 2 oracle oinstall 65536 Sep 28 14:48 lost+found
-rwxr-xr-x. 1 oracle oinstall 120998448 Sep 28 15:25 p6880880_190000_Linux-x86-64.zip
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac1 acfs_vol]$
[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:25:23 2021
Create started at: Tue Sep 28 15:25:23 2021
Time: Tue Sep 28 15:25:22 2021
Transfer started at: Tue Sep 28 15:25:22 2021
Time: Tue Sep 28 15:21:35 2021
Transfer completed at: Tue Sep 28 15:21:35 2021
Transfer status: 0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 23

[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:25:23 2021
Create started at: Tue Sep 28 15:25:23 2021
Time: Tue Sep 28 15:25:22 2021
Transfer started at: Tue Sep 28 15:25:22 2021
Time: Tue Sep 28 15:21:35 2021
Transfer completed at: Tue Sep 28 15:21:35 2021
Transfer status: 0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 23

[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Standby
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Last sync time with primary: Tue Sep 28 15:25:19 2021
Receiving primary as of: Tue Sep 28 15:25:17 2021
Status: Receive Completed
Last receive started at: Tue Sep 28 15:27:17 2021
Last receive completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last receive: 0 days, 0 hours, 0 minutes, 25 seconds
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac3 ~]$

Step-16:
------------
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -s /acfs_vol/
-------------------------------------------------------
Tue Sep 28 15:18:25 2021 - Tue Sep 28 15:27:42 2021
-------------------------------------------------------
Data replicated: 0.11GB
Avg. rate of data replication: 12.48MB/minute
Avg. time from capture to apply: 00:00:32
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl bg info /acfs_vol/
Resource: ora.repl.dupd.acfsdg.acfs_vol.acfs
Target State: ONLINE , OFFLINE
Current State: ONLINE on oel82-rac1, OFFLINE on oel82-rac2
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil info fs -s /acfs_vol/
/acfs_vol/
amount of change since mount: 115.53 MB
average rate of change since mount: 31.46 KB/s
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10066976768 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10066976768 ( 9.38 GB )
metadata read I/O count: 18287
metadata write I/O count: 768
total metadata bytes read: 78139392 ( 74.52 MB )
total metadata bytes written: 7852032 ( 7.49 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 393216 ( 384.00 KB )
replication status: primary
compression status: DISABLED
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl info -s /acfs_vol/
-------------------------------------------------------
Tue Sep 28 15:00:38 2021 - Tue Sep 28 15:27:42 2021
-------------------------------------------------------
Data replicated: 0.11GB
Avg. rate of data replication: 4.28MB/minute
Average time to apply snapshot duplicate stream: 00:00:05
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl bg info /acfs_vol/
acfsutil repl bg: ACFS-05058: command issued on the incorrect replication site
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ /sbin/acfsutil info fs -s /acfs_vol/
/acfs_vol/
amount of change since mount: 115.48 MB
average rate of change since mount: 35.89 KB/s
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:55:25 2021
mount time: Tue Sep 28 14:56:28 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 9999724544 ( 9.31 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-211
label:
state: Available
major, minor: 250, 108033
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 9999724544 ( 9.31 GB )
metadata read I/O count: 2343
metadata write I/O count: 731
total metadata bytes read: 32858112 ( 31.34 MB )
total metadata bytes written: 6725632 ( 6.41 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 393216 ( 384.00 KB )
replication status: standby
compression status: DISABLED
[oracle@oel82-rac3 ~]$
Step-17:
---------
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl pause /acfs_vol/
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Paused
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 15:25:17 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 15:25:22 2021
Last send completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last send: 0 days, 0 hours, 2 minutes, 20 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl resume /acfs_vol/
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Running
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 15:25:17 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 15:25:22 2021
Last send completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last send: 0 days, 0 hours, 2 minutes, 20 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil compress on /acfs_vol/
acfsutil compress on: ACFS-09130: Root access required
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl sync apply /acfs_vol
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Running
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 16:01:44 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 16:01:44 2021
Last send completed at: Tue Sep 28 16:01:53 2021
Elapsed time for last send: 0 days, 0 hours, 0 minutes, 9 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ su - root
Password:
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil compress on /acfs_vol/
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10067369984 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10067369984 ( 9.38 GB )
metadata read I/O count: 22542
metadata write I/O count: 958
total metadata bytes read: 96133120 ( 91.68 MB )
total metadata bytes written: 9535488 ( 9.09 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 786432 ( 768.00 KB )
replication status: primary
compression status: ENABLED
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info fs
/acfs_vol
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10067369984 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10067369984 ( 9.38 GB )
metadata read I/O count: 22554
metadata write I/O count: 958
total metadata bytes read: 96182272 ( 91.73 MB )
total metadata bytes written: 9535488 ( 9.09 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 786432 ( 768.00 KB )
replication status: primary
compression status: ENABLED
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil meta -f /home/oracle/mymetafile /dev/asm/acfs_vol-204
Reading configuration information from device /dev/asm/acfs_vol-204 ...
Log file name: /home/oracle/mymetafile.log
Input device name: /dev/asm/acfs_vol-204
Output file name: /home/oracle/mymetafile
Copying first 10 megabytes from device /dev/asm/acfs_vol-204 ...
Continue scanning metadata on device /dev/asm/acfs_vol-204 ...
End of general metadata scan on device /dev/asm/acfs_vol-204 ...
Copying global bit map from device /dev/asm/acfs_vol-204 ...
Copying snap map bit map from device /dev/asm/acfs_vol-204 ...
Copying local bit map from device /dev/asm/acfs_vol-204 ...
Copying local bit map from device /dev/asm/acfs_vol-204 ...
acfsutil meta summary:
Log file name: /home/oracle/mymetafile.log
Input device name: /dev/asm/acfs_vol-204
Output file name: /home/oracle/mymetafile
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info ftrace -s /acfs_vol/
Fileid: 2, Pathname: /acfs_vol
Fileid: 7, Pathname: /acfs_vol/.ACFS
Fileid: 9, Pathname: /acfs_vol/.ACFS/repl
Fileid: 11, Pathname: /acfs_vol/.ACFS/repl/ready
Fileid: 43, Pathname: /acfs_vol/.ACFS/repl/ready/DupdRunning_0
[root@oel82-rac1 ~]#

[oracle@oel82-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid/
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.acfsdg.acfs_vol.acfs
ONLINE ONLINE oel82-rac1 mounted on /acfs_vol
,STABLE
ONLINE ONLINE oel82-rac2 mounted on /acfs_vol
,STABLE
ora.chad
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.net1.network
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.ons
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.proxy_advm
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel82-rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel82-rac1 STABLE
ora.OCRVD.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 Started,STABLE
2 ONLINE ONLINE oel82-rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.cvu
1 ONLINE ONLINE oel82-rac1 STABLE
ora.oel82-rac1.vip
1 ONLINE ONLINE oel82-rac1 STABLE
ora.oel82-rac2.vip
1 ONLINE ONLINE oel82-rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE oel82-rac1 STABLE
ora.repl.dupd.acfsdg.acfs_vol.acfs
1 ONLINE ONLINE oel82-rac1 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel82-rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE oel82-rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE oel82-rac1 STABLE
--------------------------------------------------------------------------------
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac3 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.acfsdg.acfs_vol.acfs
ONLINE ONLINE oel82-rac3 mounted on /acfs_vol
,STABLE
OFFLINE OFFLINE oel82-rac4 admin unmounted /acf
s_vol,STABLE
ora.chad
ONLINE OFFLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.net1.network
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.ons
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.proxy_advm
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel82-rac3 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel82-rac3 STABLE
ora.OCRVD.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 Started,STABLE
2 ONLINE ONLINE oel82-rac4 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.cvu
1 ONLINE ONLINE oel82-rac3 STABLE
ora.oel82-rac3.vip
1 ONLINE ONLINE oel82-rac3 STABLE
ora.oel82-rac4.vip
1 ONLINE ONLINE oel82-rac4 STABLE
ora.qosmserver
1 ONLINE ONLINE oel82-rac3 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel82-rac4 STABLE
ora.scan2.vip
1 ONLINE ONLINE oel82-rac3 STABLE
ora.scan3.vip
1 ONLINE ONLINE oel82-rac3 STABLE
--------------------------------------------------------------------------------
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac1 ~]$ findmnt -t acfs
TARGET SOURCE FSTYPE OPTIONS
/acfs_vol /dev/asm/acfs_vol-204 acfs rw,relatime,device,rootsuid,ordered
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac3 ~]$ findmnt -t acfs
TARGET SOURCE FSTYPE OPTIONS
/acfs_vol /dev/asm/acfs_vol-211 acfs rw,relatime,device,rootsuid,ordered
[oracle@oel82-rac3 ~]$

[root@oel82-rac1 ~]# . oraenv
ORACLE_SID = [+ASM] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid/
The Oracle base has been set to /u01/app/oracle
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# srvctl status filesystem -device /dev/asm/acfs_vol-204
ACFS file system /acfs_vol is mounted on nodes oel82-rac1,oel82-rac2
[root@oel82-rac1 ~]#
[root@oel82-rac3 ~]# srvctl status filesystem -device /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3
[root@oel82-rac3 ~]#

Wednesday, September 15, 2021

Oracle 12cR2/18c/19c - Generate AWRs in Active Data Guard (ADG) Physical Standby Databases

Generate AWRs in Active Data Guard (ADG) Physical Standby Databases

Starting with Oracle 12cR2 (12.2), Automatic Workload Repository (AWR) data can be captured for ADG standby databases. This feature enables analyzing any performance-related issues for ADG standby databases.

Environment:
------------------

1. Primary RAC: Oracle 19c (19.3) GI, Oracle 19c (19.12) RU and same for RDBMS
2. Standby RAC: Oracle 19c (19.3) GI, Oracle 19c (19.12) RU and same for RDBMS

3. Operating System: Oracle Linux Server 7.9

1. Confirm the open mode and the database role on the standby:

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status from gv$database,gv$instance;

INSTANCE_NAME DATABASE_ROLE PROTECTION_MODE FLASHBACK_ON STATUS
--------------- -------------------- -------------------- -------------------- ----------
orcls1 PHYSICAL STANDBY MAXIMUM PERFORMANCE YES OPEN
orcls2 PHYSICAL STANDBY MAXIMUM PERFORMANCE YES OPEN

2. On Primary RAC Database

The SYS$UMF user is the default database user that has all the privileges to access the system-level Remote Management Framework (RMF) views and tables.

All the AWR related operations in RMF can be performed only by the SYS$UMF user. The SYS$UMF user is locked by default and it must be unlocked before deploying the RMF topology:

SQL> alter user sys$umf identified by sysumf account unlock;
User altered.

3. Create the database link between the primary database and the standby database and vice versa:

SQL> create database link dbl_orcl_to_orcls CONNECT TO sys$umf IDENTIFIED BY sysumf using 'orcls';
Database link created.

SQL> create database link dbl_orcls_to_orcl CONNECT TO sys$umf IDENTIFIED BY sysumf using 'orcl';
Database link created.

4. We need to configure database nodes to add to the topology. Each database node in a topology must be assigned a unique name (default is DB_UNIQUE_NAME):

Example we have chosen "primary" for primary and "stanby" for standby.

SQL> alter system set "_umf_remote_enabled"=TRUE scope=BOTH;
System altered.

SQL> exec dbms_umf.configure_node ('primary');
PL/SQL procedure successfully completed.

5. The standby database is remote to the target system (the primary database). We can register it via the corresponding database link.

On Physical Standby Database:

SQL> exec dbms_umf.configure_node ('stanby','dbl_orcls_to_orcl');
PL/SQL procedure successfully completed.
SQL>

6. Create the RMF topology:

SQL> exec DBMS_UMF.create_topology ('Topology1');
PL/SQL procedure successfully completed.
SQL>

7. Verify the steps done at primary RAC database

SQL> select * from dba_umf_topology;
TOPOLOGY_NAME TARGET_ID TOPOLOGY_VERSION TOPOLOGY
--------------- ---------- ---------------- --------
Topology1 509936611 1 ACTIVE

SQL> select * from dba_umf_registration;
TOPOLOGY_NAME NODE_NAME NODE_ID NODE_TYPE AS_SO AS_CA STATE
--------------- --------------- ---------- ---------- ----- ----- --------------------
Topology1 primary 509936611 0 FALSE FALSE OK
SQL>

8. Register the standby database with the topology

SQL> exec DBMS_UMF.register_node ('Topology1', 'stanby', 'dbl_orcl_to_orcls', 'dbl_orcls_to_orcl', 'FALSE', 'FALSE');
PL/SQL procedure successfully completed.
SQL>

9. Enable the AWR service on the node:

SQL> exec DBMS_WORKLOAD_REPOSITORY.register_remote_database(node_name=>'stanby');
PL/SQL procedure successfully completed.

10. Verify all the queries

SQL> select * from dba_umf_topology;
TOPOLOGY_NAME TARGET_ID TOPOLOGY_VERSION TOPOLOGY
--------------- ---------- ---------------- --------
Topology1 509936611 7 ACTIVE
SQL>

SQL> select * from dba_umf_registration;

TOPOLOGY_NAME NODE_NAME NODE_ID NODE_TYPE AS_SO AS_CA STATE
--------------- --------------- ---------- ---------- ----- ----- --------------------
Topology1 primary 509936611 0 FALSE FALSE OK
Topology1 stanby 3184583509 0 FALSE FALSE OK
SQL>

SQL> select * from dba_umf_service;
TOPOLOGY_NAME NODE_ID SERVICE
--------------- ---------- -------
Topology1 3184583509 AWR
SQL>

11. Create a remote snapshot using the RMF

SQL> exec dbms_workload_repository.create_remote_snapshot('stanby');
PL/SQL procedure successfully completed.
Note: We need to run at least two to get the begin_snap and end_snap.

12. Switch the logfiles at Primary RAC Database

SQL> alter system switch all logfile;
System altered.

13. Login to Physical Standby Database and execute AWR reports

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status from gv$database,gv$instance;

SQL> @?/rdbms/admin/awrrpti.sql

Specify the Report Type
~~~~~~~~~~~~~~~~~~~~~~~
AWR reports can be generated in the following formats. Please enter the
name of the format at the prompt. Default value is 'html'.
'html' HTML format (default)
'text' Text format
'active-html' Includes Performance Hub active report

Enter value for report_type: html
Type Specified: html
Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DB Id Inst Num DB Name Instance Host
------------ ---------- --------- ---------- ------
1610181442 1 ORCL orcl1 oel70-base1
1610181442 2 ORCL orcl2 oel70-base2
3184583509 1 ORCL orcls1 oel70-base3
3184583509 2 ORCL orcls2 oel70-base4
Enter value for dbid: 3184583509
Using 3184583509 for database Id
Enter value for inst_num: 1
Using 1 for instance number
Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed. Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days: 5
Listing the last 5 days of Completed Snapshots
Instance DB Name Snap Id Snap Started Snap Level
------------ ------------ ---------- ------------------ ----------
orcls1 ORCL 1 14 Sep 2021 23:44 1
2 15 Sep 2021 01:01 1
Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 1
Begin Snapshot Id specified: 1
Enter value for end_snap: 2
End Snapshot Id specified: 2
Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_1_1_2.html. To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: /home/oracle/awrrpt_orcls1.html
14. Execute for Instance-2:

SQL> @?/rdbms/admin/awrrpti.sql

Enter value for report_type: html
Type Specified: html
Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DB Id Inst Num DB Name Instance Host
------------ ---------- --------- ---------- ------
1610181442 1 ORCL orcl1 oel70-base1
1610181442 2 ORCL orcl2 oel70-base2
3184583509 1 ORCL orcls1 oel70-base3
3184583509 2 ORCL orcls2 oel70-base4
Enter value for dbid: 3184583509
Using 3184583509 for database Id
Enter value for inst_num: 2
Using 2 for instance number
Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed. Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days: 2
Listing the last 2 days of Completed Snapshots
Instance DB Name Snap Id Snap Started Snap Level
------------ ------------ ---------- ------------------ ----------
orcls2 ORCL 1 14 Sep 2021 23:47 1
2 15 Sep 2021 01:03 1
Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 1
Begin Snapshot Id specified: 1
Enter value for end_snap: 2
End Snapshot Id specified: 2
Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_2_1_2.html. To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: /home/oracle/awrrpt_orcls2.html
Using the report name /home/oracle/awrrpt_orcls2.html

Hope it helps....

Please refer for more details:
How to Generate AWRs in Active Data Guard Standby Databases (Doc ID 2409808.1)

Tuesday, September 14, 2021

Oracle 19c Restore Point Replication From Primary Database To Physical Standby Database

Oracle Active Data Guard and Flashback enhancements

1. Today, restore points are set on each Data Guard database individually

Requires multiple operations if the same restore point across the configuration is desired

2. With Oracle Database 19c, the primary restore points are automatically created on each standby

Identified by a suffix to the name of “_PRIMARY”

Environment:

Primary RAC: Oracle 19c (19.3) Grid Infrastructure and 19.12 RU

Oracle 19c (19.3) RDBMS and 19.12 RU

Physical Standby RAC: Oracle 19c (19.3) Grid Infrastructure and 19.12 RU
Oracle 19c (19.3) RDBMS and 19.12 RU

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status

from gv$database,gv$instance;

INSTANCE_N DATABASE_ROLE PROTECTION_MODE FLASHBACK_ON STATUS
---------- --------------- -------------------- -------------------- ----------
orcl2 PRIMARY MAXIMUM PERFORMANCE NO OPEN
orcl1 PRIMARY MAXIMUM PERFORMANCE NO OPEN
SQL>

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl stop database -d orcl

[oracle@oel70-base1 ~]$ srvctl start database -d orcl -startoption mount

[oracle@oel70-base1 ~]$ . oraenv
ORACLE_SID = [orcl1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:45:09 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database flashback on;
Database altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl stop database -d orcl
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ srvctl start database -d orcl
[oracle@oel70-base1 ~]$

[oracle@oel70-base1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:50:25 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> set lines 100 pages 100
col status format a10
col instance_name format a15
col database_role format a15
col protection_mode format a20
col flashback_on format a20

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status

from gv$database,gv$instance;

INSTANCE_NAME DATABASE_ROLE PROTECTION_MODE FLASHBACK_ON STATUS
--------------- --------------- -------------------- -------------------- ----------
orcl2 PRIMARY MAXIMUM PERFORMANCE YES OPEN
orcl1 PRIMARY MAXIMUM PERFORMANCE YES OPEN

SQL> alter system switch all logfile;
System altered.

SQL> create restore point grp_test guarantee flashback database;
Restore point created.

SQL> set lines 100 pages 100
SQL> col SCN format 99999999
SQL> col GUARANTEE_FLASHBACK_DATABASE format a10
SQL> col TIME format a20
SQL> col NAME format a10
SQL> col REPLICATED format a10
SQL> col TIME format a32

SQL> select SCN, GUARANTEE_FLASHBACK_DATABASE, TIME, NAME, REPLICATED

from v$restore_point;

SCN GUARANTEE_ TIME NAME REPLICATED
--------- ---------- -------------------------------- ---------- ----------
3190291 YES 14-SEP-21 04.07.09.000000000 PM GRP_TEST NO

Physical Standby Database:
-----------------------------------

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status

from gv$database,gv$instance;

INSTANCE_NAME DATABASE_ROLE PROTECTION_MODE FLASHBACK_ON STATUS
--------------- -------------------- -------------------- --------------- ----------
orcls2 PHYSICAL STANDBY MAXIMUM PERFORMANCE NO OPEN
orcls1 PHYSICAL STANDBY MAXIMUM PERFORMANCE NO OPEN

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl stop database -d orcls

[oracle@oel70-base3 ~]$ srvctl start database -d orcls -startoption mount

[oracle@oel70-base3 ~]$ . oraenv
ORACLE_SID = [orcls1] ?
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 15:58:11 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> alter database flashback on;
alter database flashback on
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

SQL> alter database recover managed standby database cancel;
Database altered.

SQL> alter database flashback on;
Database altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ srvctl stop database -d orcls

[oracle@oel70-base3 ~]$ srvctl start database -d orcls
[oracle@oel70-base3 ~]$

[oracle@oel70-base3 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 14 16:02:02 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> set lines 100 pages 100
col status format a10
col instance_name format a15
col database_role format a15
col protection_mode format a20
col flashback_on format a20

SQL> select distinct instance_name,database_role,protection_mode,flashback_on,status

from gv$database,gv$instance;

SQL> set lines 200
col DATABASE_HOST for a25;
col DB_UNIQUE_NAME for a25
col HOST_NAME for a15;
col DATABASE_ROLE for a10
col OPEN_MODE for a10
col STARTUP_TIME for a20

SQL> SELECT i.HOST_NAME "DATABASE_HOST" ,i.INSTANCE_NAME "DB_NAME",d.db_unique_name "DB_UNIQUE_NAME",
d.DATABASE_ROLE " DATABASE_ROLE", d.OPEN_MODE " OPEN_MODE ", STARTUP_TIME
from GV$DATABASE d, gv$instance i where i.INST_ID=d.INST_ID;
select inst_id,process, status, thread#, sequence#, block#, blocks from gv$managed_standby where process='MRP0';
select a.thread#, (select max (sequence#)
from v$archived_log
where archived='YES' and thread#=a.thread#) archived,max(a.sequence#) applied,
(select max(sequence#)
from v$archived_log
where archived='YES' and thread#=a.thread#)-max(a.sequence#)gap
from v$archived_log a
where a.applied='YES'
group by a.thread#
order by thread#

DATABASE_HOST DB_NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE STARTUP_TIME
------------------------- ---------------- ------------------------- ---------------- -------------------- -------------------
oel70-base3 orcls1 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 14-SEP-21
oel70-base4 orcls2 orcls PHYSICAL STANDBY READ ONLY WITH APPLY 14-SEP-21

SQL>
INST_ID PROCESS STATUS THREAD# SEQUENCE# BLOCK# BLOCKS
---------- --------- ---------- ---------- ---------- ---------- ----------
1 MRP0 APPLYING_L 1 41 7668 409600
OG

THREAD# ARCHIVED APPLIED GAP
---------- ---------- ---------- ----------
1 40 40 0
2 30 30 0
SQL>

SQL> set lines 100 pages 100
col SCN format 99999999
col GUARANTEE_FLASHBACK_DATABASE format a10
col TIME format a33
col NAME format a10
col REPLICATED format a10
SQL> col name format a20

SQL> select SCN, GUARANTEE_FLASHBACK_DATABASE, TIME, NAME, REPLICATED

from v$restore_point;

SCN GUARANTEE_ TIME NAME REPLICATED
--------- ---------- --------------------------------- -------------------- ----------
3190291 NO 14-SEP-21 04.07.09.000000000 PM GRP_TEST_PRIMARY YES
SQL>

DGMGRL> show configuration lag;
Configuration - dg_19c
Protection Mode: MaxPerformance
Members:
orcl - Primary database
orcls - Physical standby database
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 1 second ago)
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 31 seconds ago)
DGMGRL>

Saturday, September 11, 2021

Prepare a primary database for a Data Guard environment in Oracle 21c (21.3.0)

Prepare a primary database for a Data Guard environment in Oracle 21c (21.3.0)

Environment:
==========
First Container Database: orcl
Pluggable Databases: PDB & PDB1
GI Version: 21c (21.3.0)
RDBMS Version: 21c (21.3.0)

[oracle@rac1-21c ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-21c ~]$ asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 36852 23188 0 23188 0 N DATA/
MOUNTED EXTERN N 512 512 4096 4194304 24568 24172 0 24172 0 Y OCRVD/
MOUNTED EXTERN N 512 512 4096 4194304 36852 33244 0 33244 0 N RECO/
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ ps -ef | grep pmon
oracle 8272 1 0 14:33 ? 00:00:00 asm_pmon_+ASM1
oracle 11342 1 0 14:34 ? 00:00:00 ora_pmon_orcl1
oracle 69999 1 0 15:10 ? 00:00:00 ora_pmon_orcldb1
oracle 87499 19459 0 15:22 pts/0 00:00:00 grep --color=auto pmon
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcl1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ sqlplus / as sysdba
SQL*Plus: Release 21.0.0.0.0 - Production on Sat Sep 11 15:22:35 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDB READ WRITE NO
4 PDB2 READ WRITE NO

SQL> archive log list;
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 24
Current log sequence 25

SQL> select force_logging from gv$database;
FORCE_LOGGING
---------------------------------------
NO
NO
NO

SQL> select flashback_on from gv$database;
FLASHBACK_ON
------------------
NO
NO
NO

SQL> exit
Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ dgmgrl /
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Sat Sep 11 15:23:36 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "orcl"
Connected as SYSDG.
DGMGRL>

DGMGRL> help
The following commands are available:
@ Execute DGMGRL script file
! Host operating system command
/ Repeat the last command
-- Comment to be ignored by DGMGRL
add Adds a member to the broker configuration
connect Connects to an Oracle database instance
convert Converts a database from one type to another
create Creates a broker configuration or far sync instance
disable Disables a configuration, a member, or fast-start failover
edit Edits a configuration or a member
enable Enables a configuration, a member, or fast-start failover
exit Exits the program
export Export Data Guard Broker configuration to a file.
failover Changes a standby database to be the primary database
help Displays description and syntax for a command
host Host operating system command
import Import Data Guard Broker configuration from a file.
migrate Migrate a pluggable database from one configuration to another.
prepare Prepare a primary database for a Data Guard environment.
quit Exits the program
reinstate Changes a database marked for reinstatement into a viable standby
rem Comment to be ignored by DGMGRL
remove Removes a configuration or a member
set Set a DGMGRLI CLI property to a specified value
show Displays information about a configuration or a member
shutdown Shuts down a currently running Oracle database instance
spool store input and output of DGMGRL CLI in a file
sql Executes a SQL statement
start Starts the fast-start failover observer
startup Starts an Oracle database instance
stop Stops the fast-start failover observer
switchover Switches roles between a primary and standby database
validate Performs an exhaustive set of validations for a member
Use "help <command>" to see syntax for individual commands

DGMGRL> help prepare
Prepare a primary database for a Data Guard environment.
Syntax:
PREPARE DATABASE FOR DATA GUARD
[WITH [DB_UNIQUE_NAME IS <db-unique-name>]
[DB_RECOVERY_FILE_DEST IS <directory-location>]
[DB_RECOVERY_FILE_DEST_SIZE IS <size>]
[BROKER_CONFIG_FILE_1 IS <broker-config-file-1-location>]
[BROKER_CONFIG_FILE_2 IS <broker-config-file-2-location>]];

DGMGRL>

DGMGRL> PREPARE DATABASE FOR DATA GUARD
> WITH
> DB_UNIQUE_NAME IS ORCL
> DB_RECOVERY_FILE_DEST IS "+RECO"
> DB_RECOVERY_FILE_DEST_SIZE IS "10G"
> BROKER_CONFIG_FILE_1 IS "+DATA"
> BROKER_CONFIG_FILE_2 IS "+RECO";
Preparing database "orcl" for Data Guard.
Primary database must be restarted to enable archivelog mode.
Shutting down database "orcl".
Database stopped.
Starting database "orcl" to mounted mode.
Database started.
Initialization parameter DB_RECOVERY_FILE_DEST_SIZE set to '10G'.
Initialization parameter DB_RECOVERY_FILE_DEST set to '+RECO'.
Initialization parameter DG_BROKER_CONFIG_FILE1 set to '+DATA'.
Initialization parameter DG_BROKER_CONFIG_FILE2 set to '+RECO'.
LOG_ARCHIVE_DEST_n initialization parameter already set for local archival.
Initialization parameter LOG_ARCHIVE_DEST_2 set to 'location=use_db_recovery_file_dest valid_for=(all_logfiles, all_roles)'.
Initialization parameter LOG_ARCHIVE_DEST_STATE_2 set to 'Enable'.
Adding standby log group size 209715200 and assigning it to thread 3.
Adding standby log group size 209715200 and assigning it to thread 3.
Adding standby log group size 209715200 and assigning it to thread 2.
Adding standby log group size 209715200 and assigning it to thread 2.
Adding standby log group size 209715200 and assigning it to thread 1.
Adding standby log group size 209715200 and assigning it to thread 1.
Initialization parameter STANDBY_FILE_MANAGEMENT set to 'AUTO'.
Initialization parameter DG_BROKER_START set to TRUE.
Database set to FORCE LOGGING.
Database set to ARCHIVELOG.
Database set to FLASHBACK ON.
Shutting down database "orcl".
Database stopped.
Starting database "orcl".
Database started.
DGMGRL>

[oracle@rac1-21c ~]$ . oraenv

ORACLE_SID = [orcl1] ?

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@rac1-21c ~]$

[oracle@rac1-21c ~]$ sqlplus / as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on Sat Sep 11 15:42:43 2021

Version 21.3.0.0.0

Connected to:

Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production

Version 21.3.0.0.0

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED

---------- ------------------------------ ---------- ----------

2 PDB$SEED READ ONLY NO

3 PDB READ WRITE NO

4 PDB2 READ WRITE NO

SQL>

SQL> archive log list;

Database log mode Archive Mode

Automatic archival Enabled

Archive destination USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence 24

Next log sequence to archive 25

Current log sequence 25

SQL>

SQL> select force_logging from gv$database;

FORCE_LOGGING

---------------------------------------

YES

SQL> select flashback_on from gv$database;

FLASHBACK_ON

------------------

YES

SQL>

SQL> select group#,type,member from v$logfile where type like 'STAN%';

GROUP# TYPE MEMBER

------ --------------- ------------------------------------------------------------

7 STANDBY +DATA/ORCL/ONLINELOG/group_7.324.1082993621

7 STANDBY +RECO/ORCL/ONLINELOG/group_7.310.1082993621

8 STANDBY +DATA/ORCL/ONLINELOG/group_8.325.1082993621

8 STANDBY +RECO/ORCL/ONLINELOG/group_8.311.1082993623

9 STANDBY +DATA/ORCL/ONLINELOG/group_9.326.1082993623

9 STANDBY +RECO/ORCL/ONLINELOG/group_9.312.1082993625

10 STANDBY +DATA/ORCL/ONLINELOG/group_10.327.1082993625

10 STANDBY +RECO/ORCL/ONLINELOG/group_10.313.1082993625

11 STANDBY +DATA/ORCL/ONLINELOG/group_11.328.1082993627

11 STANDBY +RECO/ORCL/ONLINELOG/group_11.314.1082993627

12 STANDBY +DATA/ORCL/ONLINELOG/group_12.329.1082993629

12 STANDBY +RECO/ORCL/ONLINELOG/group_12.315.1082993629

12 rows selected.

SQL> select group#,type,member from v$logfile where type like 'ONLIN%';

GROUP# TYPE MEMBER

------ --------------- ------------------------------------------------------------

2 ONLINE +DATA/ORCL/ONLINELOG/group_2.263.1080574111

2 ONLINE +RECO/ORCL/ONLINELOG/group_2.258.1080574111

1 ONLINE +DATA/ORCL/ONLINELOG/group_1.262.1080574111

1 ONLINE +RECO/ORCL/ONLINELOG/group_1.257.1080574111

3 ONLINE +DATA/ORCL/ONLINELOG/group_3.271.1080574361

3 ONLINE +RECO/ORCL/ONLINELOG/group_3.259.1080574361

4 ONLINE +DATA/ORCL/ONLINELOG/group_4.272.1080574363

4 ONLINE +RECO/ORCL/ONLINELOG/group_4.260.1080574363

5 ONLINE +DATA/ORCL/ONLINELOG/group_5.273.1080574363

5 ONLINE +RECO/ORCL/ONLINELOG/group_5.261.1080574365

6 ONLINE +DATA/ORCL/ONLINELOG/group_6.274.1080574365

6 ONLINE +RECO/ORCL/ONLINELOG/group_6.262.1080574365

12 rows selected.

SQL>

Friday, September 3, 2021

PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current && PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old

After upgrading (using AutoUpgrade) from Oracle 12c (12.1.0.2) to Oracle 19c (19.3) and applied patch Oracle 19c (19.12) . Below are the messages in alert log.

PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current
PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old

Try to connect rman

[oracle@primary ~]$ rman target / nocatalog
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Sep 3 04:49:15 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
PL/SQL package SYS.DBMS_BACKUP_RESTORE version 19.03.00.00 in TARGET database is not current
PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 in TARGET database is not current
connected to target database: ORCLDB (DBID=2506522584)
using target database control file instead of recovery catalog
RMAN>

Connect to the target database and execute the following scripts:

[oracle@primary ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 3 04:50:07 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL> @$ORACLE_HOME/rdbms/admin/dbmsrman.sql
SQL> @$ORACLE_HOME/rdbms/admin/prvtrmns.plb
SQL> @$ORACLE_HOME/rdbms/admin/dbmsbkrs.sql
SQL> @$ORACLE_HOME/rdbms/admin/prvtbkrs.plb

[oracle@primary ~]$ rman target / nocatalog
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Sep 3 04:50:50 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCLDB (DBID=2506522584)
using target database control file instead of recovery catalog
RMAN>

Hope it helps...

Wednesday, September 1, 2021

CLSRSC-614: failed to get the list of configured diskgroups - Oracle 12cR2 (12.2.0) - Grid Installation

Environment:

Operating System: Oracle Linux 6.5
RDBMS Version: Oracle 12c (12.2.0)
RDBMS Patch: p32928749_122010_Linux-x86-64.zip (July 2021)
Instances: Two Node

Scenario: While executing 'root.sh' as a 'root' user for Oracle 12c R2 (12.2.0) Grid Installation on 2-node RAC.

Step-1: Node-1: While executing "root.sh" of Grid Installation got the following errors after "step 19 of 19"

2021/09/01 16:03:18 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/09/01 16:08:56 CLSRSC-614: failed to get the list of configured diskgroups
Died at /u01/app/12.2.0/grid/crs/install/oraasm.pm line 2069.
The command '/u01/app/12.2.0/grid/perl/bin/perl -I/u01/app/12.2.0/grid/perl/lib -I/u01/app/12.2.0/grid/crs/install /u01/app/12.2.0/grid/crs/install/rootcrs.pl ' execution failed

Step-2: Check 'asmcmd' login

[root@primary-rac1 ~]# ps -ef | grep pmon

oracle 10473 1 0 16:00 ? 00:00:00 asm_pmon_+ASM1

root 25874 9531 0 16:09 pts/1 00:00:00 grep pmon

[root@primary-rac1 ~]#

[root@primary-rac1 ~]# asmcmd

kgfnGetConnDetails requires 4 parameters at /u01/app/12.2.0/grid/lib/asmcmdbase.pm line 5704.

[root@primary-rac1 ~]#

Step-3: Execute the following in both the nodes one by one.

[root@primary-rac1 ~]# export ORACLE_HOME=/u01/app/12.2.0/grid

[root@primary-rac1 ~]# . oraenv

ORACLE_SID = [+ASM1] ?

ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[root@primary-rac1 ~]#

[root@primary-rac1 ~]# /usr/bin/make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk client_sharedlib libasmclntsh12.ohso libasmperl12.ohso ORACLE_HOME=/u01/app/12.2.0/grid

/u01/app/12.2.0/grid/bin/genclntsh

`if [ -d /u01/app/12.2.0/grid/lib32 ]; then rm -f /u01/app/12.2.0/grid/lib32/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib32/libclntsh.so.11.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; else rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; rm -f /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; fi `

`if [ -d /u01/app/12.2.0/grid/lib32 ]; then ln -s libclntsh.so /u01/app/12.2.0/grid/lib32/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib32/libclntsh.so.11.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; else ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.10.1; ln -s libclntsh.so /u01/app/12.2.0/grid/lib/libclntsh.so.11.1; fi `

/u01/app/12.2.0/grid/bin/genoccish

/u01/app/12.2.0/grid/bin/genagtsh /u01/app/12.2.0/grid/lib/libagtsh.so 1.0

make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ALWAYS=always /u01/app/12.2.0/grid/lib/libasmclntsh12.so

make[1]: Entering directory `/root'

rm -f /u01/app/12.2.0/grid/lib/libasmclntsh12.so

/u01/app/12.2.0/grid/bin/linkshlib /u01/app/12.2.0/grid/lib/libasmclntsh12.so /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk so ld_shlib LIBS

+ PATH=/bin:/usr/bin:/usr/ccs/bin

+ export PATH

+ lib=/u01/app/12.2.0/grid/lib/libasmclntsh12.so

+ makefile=/u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk

+ so_ext=so

+ target=ld_shlib

++ basename /u01/app/12.2.0/grid/lib/libasmclntsh12.so .so

+ libname=libasmclntsh12

++ dirname /u01/app/12.2.0/grid/lib/libasmclntsh12.so

+ sodir=/u01/app/12.2.0/grid/lib

+ ardir=/u01/app/12.2.0/grid/lib/

+ '[' var = ld_shlib ']'

+ suffix=LIBS

+ var=

+ '[' '!' -f /u01/app/12.2.0/grid/lib/libasmclntsh12.a ']'

+ '[' '' '!=' '' ']'

+ make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ld_shlib _FULL_LIBNAME=/u01/app/12.2.0/grid/lib/libasmclntsh12.so _LIBNAME=libasmclntsh12 _LIBDIR=/u01/app/12.2.0/grid/lib/ '_LIBNAME_LIBS=$(libasmclntsh12LIBS)' '_LIBNAME_EXTRALIBS=$(libasmclntsh12EXTRALIBS)'

make[2]: Entering directory `/root'

/u01/app/12.2.0/grid/bin/orald -o /u01/app/12.2.0/grid/lib/libasmclntsh12.so -shared -z noexecstack -Wl,--disable-new-dtags -L/tmp/bootstraplib/ -L/u01/app/12.2.0/grid/lib/ -L/u01/app/12.2.0/grid/rdbms/lib/ -L/u01/app/12.2.0/grid/lib/stubs/ -Wl,--version-script=/u01/app/12.2.0/grid/rdbms/admin/libasmclntsh12.def -Wl,--whole-archive /u01/app/12.2.0/grid/lib/libasmclntsh12.a -Wl,--no-whole-archive -lirc

make[2]: Leaving directory `/root'

make[1]: Leaving directory `/root'

make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk ALWAYS=always /u01/app/12.2.0/grid/lib/libasmperl12.so

make[1]: Entering directory `/root'

rm -f /u01/app/12.2.0/grid/lib/libasmperl12.so

/u01/app/12.2.0/grid/bin/linkshlib /u01/app/12.2.0/grid/lib/libasmperl12.so /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk so dlopenlib LIBS

+ PATH=/bin:/usr/bin:/usr/ccs/bin

+ export PATH

+ lib=/u01/app/12.2.0/grid/lib/libasmperl12.so

+ makefile=/u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk

+ so_ext=so

+ target=dlopenlib

++ basename /u01/app/12.2.0/grid/lib/libasmperl12.so .so

+ libname=libasmperl12

++ dirname /u01/app/12.2.0/grid/lib/libasmperl12.so

+ sodir=/u01/app/12.2.0/grid/lib

+ ardir=/u01/app/12.2.0/grid/lib/

+ '[' var = dlopenlib ']'

+ suffix=LIBS

+ var=

+ '[' '!' -f /u01/app/12.2.0/grid/lib/libasmperl12.a ']'

+ '[' '' '!=' '' ']'

+ make -f /u01/app/12.2.0/grid/rdbms/lib/ins_rdbms.mk dlopenlib _FULL_LIBNAME=/u01/app/12.2.0/grid/lib/libasmperl12.so _LIBNAME=libasmperl12 _LIBDIR=/u01/app/12.2.0/grid/lib/ '_LIBNAME_LIBS=$(libasmperl12LIBS)' '_LIBNAME_EXTRALIBS=$(libasmperl12EXTRALIBS)'

make[2]: Entering directory `/root'

/u01/app/12.2.0/grid/bin/orald -o /u01/app/12.2.0/grid/lib/libasmperl12.so -shared -z noexecstack -Wl,--disable-new-dtags -L/tmp/bootstraplib/ -L/u01/app/12.2.0/grid/lib/ -L/u01/app/12.2.0/grid/rdbms/lib/ -L/u01/app/12.2.0/grid/lib/stubs/ -Wl,--version-script=/u01/app/12.2.0/grid/rdbms/admin/libasmperl12.def -Wl,--whole-archive /u01/app/12.2.0/grid/lib/libasmperl12.a -Wl,--no-whole-archive -lclntsh -lclntshcore -lasmclntsh12 -lhasgen12 -locr12 -lirc

make[2]: Leaving directory `/root'

make[1]: Leaving directory `/root'

[root@primary-rac1 ~]#

Step-4: Executed again "root.sh" from Node-1

[root@primary-rac1 ~]# /u01/app/12.2.0/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params

The log of current session can be found at:

/u01/app/oracle/crsdata/primary-rac1/crsconfig/rootcrs_primary-rac1_2021-09-01_04-21-46PM.log

2021/09/01 16:21:52 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.

2021/09/01 16:21:52 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2021/09/01 16:25:27 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2021/09/01 16:25:27 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.

2021/09/01 16:25:37 CLSRSC-363: User ignored prerequisites during installation

2021/09/01 16:25:37 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.

2021/09/01 16:25:43 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.

2021/09/01 16:25:49 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.

2021/09/01 16:25:55 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.

2021/09/01 16:25:59 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.

2021/09/01 16:26:01 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.

2021/09/01 16:26:16 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.

2021/09/01 16:26:20 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.

2021/09/01 16:26:21 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.

2021/09/01 16:26:24 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.

2021/09/01 16:26:27 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.

2021/09/01 16:26:31 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.

2021/09/01 16:26:36 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.

2021/09/01 16:26:49 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.crsd' on 'primary-rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'primary-rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.primary-rac1.vip' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.cvu' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.qosmserver' on 'primary-rac1'

CRS-2677: Stop of 'ora.OCRVD.dg' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'primary-rac1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'primary-rac1'

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on 'primary-rac1'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on 'primary-rac1'

CRS-2677: Stop of 'ora.cvu' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac1'

CRS-2677: Stop of 'ora.primary-rac1.vip' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.scan3.vip' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.scan2.vip' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.qosmserver' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'primary-rac1'

CRS-2677: Stop of 'ora.ons' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'primary-rac1'

CRS-2677: Stop of 'ora.net1.network' on 'primary-rac1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'primary-rac1' has completed

CRS-2677: Stop of 'ora.crsd' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.storage' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.crf' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary-rac1'

CRS-2677: Stop of 'ora.storage' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'primary-rac1'

CRS-2677: Stop of 'ora.drivers.acfs' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.crf' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'primary-rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.evmd' on 'primary-rac1'

CRS-2677: Stop of 'ora.ctssd' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'primary-rac1'

CRS-2677: Stop of 'ora.cssd' on 'primary-rac1' succeeded

CRS-2673: Attempting to stop 'ora.driver.afd' on 'primary-rac1'

CRS-2673: Attempting to stop 'ora.gipcd' on 'primary-rac1'

CRS-2677: Stop of 'ora.driver.afd' on 'primary-rac1' succeeded

CRS-2677: Stop of 'ora.gipcd' on 'primary-rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

2021/09/01 16:27:18 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.evmd' on 'primary-rac1'

CRS-2676: Start of 'ora.mdnsd' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.evmd' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'primary-rac1'

CRS-2676: Start of 'ora.gpnpd' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'primary-rac1'

CRS-2676: Start of 'ora.gipcd' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary-rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'primary-rac1'

CRS-2676: Start of 'ora.diskmon' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.ctssd' on 'primary-rac1'

CRS-2676: Start of 'ora.ctssd' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'primary-rac1'

CRS-2676: Start of 'ora.asm' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'primary-rac1'

CRS-2676: Start of 'ora.storage' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'primary-rac1'

CRS-2676: Start of 'ora.crf' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'primary-rac1'

CRS-2676: Start of 'ora.crsd' on 'primary-rac1' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 01-SEP-2021 16:28:27

Connecting to (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=23.202.231.168)(PORT=1521)))

TNS-12535: TNS:operation timed out

TNS-12560: TNS:protocol adapter error

TNS-00505: Operation timed out

Connecting to (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=23.195.69.108)(PORT=1521)))

CRS-5014: Agent "ORAAGENT" timed out starting process "/u01/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/u01/app/oracle/diag/crs/primary-rac1/crs/trace/crsd_oraagent_oracle.trc"

CRS-5017: The resource action "ora.MGMTLSNR check" encountered the following error:

(:CLSN00009:)Command Aborted. For details refer to "(:CLSN00109:)" in "/u01/app/oracle/diag/crs/primary-rac1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2664: Resource 'ora.OCRVD.dg' is already running on 'primary-rac1'

CRS-6017: Processing resource auto-start for servers: primary-rac1

CRS-2672: Attempting to start 'ora.scan3.vip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.scan2.vip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.primary-rac1.vip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.ons' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.qosmserver' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.cvu' on 'primary-rac1'

CRS-2676: Start of 'ora.cvu' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.scan3.vip' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1'

CRS-2676: Start of 'ora.scan2.vip' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1'

CRS-2676: Start of 'ora.primary-rac1.vip' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'

CRS-2676: Start of 'ora.ons' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.qosmserver' on 'primary-rac1' succeeded

CRS-6016: Resource auto-start has completed for server primary-rac1

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2021/09/01 16:30:57 CLSRSC-343: Successfully started Oracle Clusterware stack

2021/09/01 16:30:57 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.

2021/09/01 16:31:03 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

2021/09/01 16:32:09 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@primary-rac1 ~]#

Step-5: Check the asmcmd as a 'oracle' user

[oracle@primary-rac1 ~]$ . oraenv

ORACLE_SID = [orcldb1] ? +ASM1

ORACLE_HOME = [/home/oracle] ? /u01/app/12.2.0/grid

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@primary-rac1 ~]$ asmcmd lsdg

State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 512 4096 4194304 40952 40660 0 40660 0 Y OCRVD/

[oracle@primary-rac1 ~]$

Step-6: Execute "root.sh" in Node-2

[root@primary-rac2 grid]# /u01/app/12.2.0/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /u01/app/12.2.0/grid/crs/install/crsconfig_params

The log of current session can be found at:

/u01/app/oracle/crsdata/primary-rac2/crsconfig/rootcrs_primary-rac2_2021-09-01_04-43-15PM.log

2021/09/01 16:43:18 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.

2021/09/01 16:43:18 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2021/09/01 16:48:16 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2021/09/01 16:48:16 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.

2021/09/01 16:48:19 CLSRSC-363: User ignored prerequisites during installation

2021/09/01 16:48:19 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.

2021/09/01 16:48:20 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.

2021/09/01 16:48:20 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.

2021/09/01 16:48:22 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.

2021/09/01 16:48:23 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.

2021/09/01 16:48:24 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.

2021/09/01 16:48:26 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.

2021/09/01 16:48:27 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.

2021/09/01 16:49:13 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.

2021/09/01 16:49:14 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.

2021/09/01 16:49:15 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.

2021/09/01 16:49:17 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

2021/09/01 16:49:34 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.

2021/09/01 16:49:36 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary-rac2'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary-rac2'

CRS-2677: Stop of 'ora.drivers.acfs' on 'primary-rac2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary-rac2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

2021/09/01 16:49:45 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'primary-rac2'

CRS-2672: Attempting to start 'ora.evmd' on 'primary-rac2'

CRS-2676: Start of 'ora.evmd' on 'primary-rac2' succeeded

CRS-2676: Start of 'ora.mdnsd' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'primary-rac2'

CRS-2676: Start of 'ora.gpnpd' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'primary-rac2'

CRS-2676: Start of 'ora.gipcd' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary-rac2'

CRS-2676: Start of 'ora.cssdmonitor' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'primary-rac2'

CRS-2672: Attempting to start 'ora.diskmon' on 'primary-rac2'

CRS-2676: Start of 'ora.diskmon' on 'primary-rac2' succeeded

CRS-2676: Start of 'ora.cssd' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary-rac2'

CRS-2672: Attempting to start 'ora.ctssd' on 'primary-rac2'

CRS-2676: Start of 'ora.ctssd' on 'primary-rac2' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'primary-rac2'

CRS-2676: Start of 'ora.asm' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'primary-rac2'

CRS-2676: Start of 'ora.storage' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'primary-rac2'

CRS-2676: Start of 'ora.crf' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'primary-rac2'

CRS-2676: Start of 'ora.crsd' on 'primary-rac2' succeeded

CRS-6017: Processing resource auto-start for servers: primary-rac2

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.net1.network' on 'primary-rac2'

CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac2'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac1' succeeded

CRS-2676: Start of 'ora.net1.network' on 'primary-rac2' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'primary-rac1'

CRS-2672: Attempting to start 'ora.ons' on 'primary-rac2'

CRS-2677: Stop of 'ora.scan1.vip' on 'primary-rac1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary-rac2'

CRS-2676: Start of 'ora.scan1.vip' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac2'

CRS-2676: Start of 'ora.ons' on 'primary-rac2' succeeded

CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'primary-rac2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'primary-rac2'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary-rac2' succeeded

CRS-2676: Start of 'ora.asm' on 'primary-rac2' succeeded

CRS-6016: Resource auto-start has completed for server primary-rac2

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2021/09/01 16:53:15 CLSRSC-343: Successfully started Oracle Clusterware stack

2021/09/01 16:53:15 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.

2021/09/01 16:53:29 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

2021/09/01 16:53:42 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

You have mail in /var/spool/mail/root

[root@primary-rac2 grid]#

Please refer for more details:

ASMCMD Failing With "KGFNGETCONNDETAILS Requires 4 Parameters at <GI_HOME>/lib/asmcmdbase.pm (Doc ID 2748316.1)

Hope it helps...

Thursday, August 26, 2021

Extending ROOT (/) volume in Oracle Linux 6.x

<pre>

[root@rac1 dev]# fdisk /dev/sdg

[root@rac1 dev]# fdisk -l

[root@rac1 dev]# pvcreate /dev/sdg1

[root@rac1 dev]# vgdisplay

[root@rac1 dev]# vgextend vg_rac1 /dev/sdg1

[root@rac1 dev]# vgdisplay

[root@rac1 dev]# df -h

[root@rac1 dev]# pvscan

[root@rac1 dev]# lvdisplay

[root@rac1 dev]# lvextend /dev/vg_rac1/lv_root /dev/sdg1

[root@rac1 dev]# df -h

[root@rac1 dev]# resize2fs /dev/vg_rac1/lv_root

[root@rac1 dev]# df -h

[root@rac1 dev]# init 6

Note: After formatting the disk, execute the above commands. Node by Node (If it is RAC).

</pre>