Reflection of my thoughts and technical expriences: Oracle 19c (19.12) - Oracle ASM Cluster File System (ACFS) Replication.

Oracle 19c - Oracle ASM Cluster File System (ACFS) Replication.

Primary RAC:
------------------
a. Operating System: Oracle Linux 8.2
b. Kernel: 5.4.17-2011.1.2.el8uek.x86_64
c. Grid Infra: Oracle 19c (19.3) + RU 19.12

Standby RAC:
-------------------
a. Operating System: Oracle Linux 8.2
b. Kernel: 5.4.17-2011.1.2.el8uek.x86_64
c. Grid Infra: Oracle 19c (19.3) + RU 19.12

Oracle ACFS, as part of Oracle Grid Infrastructure, is integrated with Oracle ASM, Oracle ADVM and Oracle Clusterware. Oracle ACFS will provide the following capabilities.

» Security
» Audit
» Compression
» Snapshots
» Replication:
» Tagging
» Plugins
» Accelerator Volumes
» NAS Maximum Availability eXtensions
» ACFS Remote Service

ACFS Replication:

Oracle ACFS provides the capability to replicate an entire file system from one location to a different location so that customers can provide for test and development and disaster recovery. Starting in Oracle 12cR2, Oracle ACFS replication evolved to a snapshot-based replication solution, enhancing its already existing capabilities and providing maximum efficiency and flexibility in its replication method. Starting in Oracle 18c, Oracle ACFS introduces role reversal, allowing a standby site to become an active site and the active site to become a standby site in case this is required.
The primary and standby sites can have differing configurations; i.e., primary can be multi-node cluster and standby can be a single-node cluster.

Oracle 19c (19.5) release, replication supports both switchover and failover. In these operations, the existing standby is converted to be the new primary, and the existing primary may either become the new standby, or may be replaced as the new standby by a different storage location.
In all cases, the standby file system is a read-only file system. One use-case for the standby file system is that it can be the source for backups.

Note that primary site running Linux, Solaris or AIX can replicate to a standby site running any of those operating systems

Configuring Password-less SSH login

In the password-less SSH login method, we’re going to exchange encrypted keys instead of entering the actual password while connecting to the remote systems using SSH. So, nobody can easily hack or guess your password, because we don’t use passwords any more to access the remote systems. And more importantly, hidden key-loggers and brute-force attacks do not work if we use Password-less SSH.

Primary RAC (oel82-rac1 & oel82-rac2)
------------------------------------------------------------
Step-1: If the public key file does not exist, generate a public / private key pair on the primary by running this command as “root”:

[root@oel82-rac1 ~]# cd /root/.ssh
-bash: cd: /root/.ssh: No such file or directory
[root@oel82-rac1 ~]# pwd
/root
[root@oel82-rac1 ~]# mkdir ~/.ssh
[root@oel82-rac1 ~]# cd .ssh/
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# pwd
/root/.ssh

[root@oel82-rac1 .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:ahbBDHCCaM7Jfr0rrcsnjfK4aZhzqm/qdANiHR25BVM root@oel82-rac1.localdomain
The key's randomart image is:
+---[RSA 3072]----+
|..o.o++E |
|o. o.=o. |
|= .. .* |
| =. .. . |
|oo .. . S |
|o... . o |
|.o.o+ = |
|=oB+.B |
|B%==*.. |
+----[SHA256]-----+
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# ls -la
total 12
drwxr-xr-x. 2 root root 38 Sep 28 14:31 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-------. 1 root root 2610 Sep 28 14:31 id_rsa
-rw-r--r--. 1 root root 581 Sep 28 14:31 id_rsa.pub
[root@oel82-rac1 .ssh]# cat id_rsa.pub >> authorized_keys
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# ls -la
total 16
drwxr-xr-x. 2 root root 61 Sep 28 14:32 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-r--r--. 1 root root 581 Sep 28 14:32 authorized_keys
-rw-------. 1 root root 2610 Sep 28 14:31 id_rsa
-rw-r--r--. 1 root root 581 Sep 28 14:31 id_rsa.pub

Step-2: Copy the "authorized_keys" from oel82-rac1 to oel82-rac2 node.

[root@oel82-rac1 .ssh]# scp authorized_keys root@oel82-rac2:/root/.ssh
The authenticity of host 'oel82-rac2 (192.168.2.20)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac2,192.168.2.20' (ECDSA) to the list of known hosts.
root@oel82-rac2's password:
authorized_keys 100% 581 0.5KB/s 00:01
[root@oel82-rac1 .ssh]#
[root@oel82-rac1 .ssh]# exit
logout
[oracle@oel82-rac1 ~]$

Step-3: If the public key file does not exist, generate a public / private key pair on the primary by running this command as “root”:

[oracle@oel82-rac2 ~]$ su - root
Password:
[root@oel82-rac2 ~]# pwd
/root
[root@oel82-rac2 ~]# mkdir ~/.ssh
[root@oel82-rac2 ~]# cd .ssh/
[root@oel82-rac2 .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:L6CIjvWl173vMYPckAobl1VWznW0Ez0omGvz2dHfX8c root@oel82-rac2
The key's randomart image is:
+---[RSA 3072]----+
| o o.oo=|
| o + + o=|
| o . +o.|
| * . . ..|
| + S = o ..o|
| . . . * + * . E|
|. o . o.o.+ = +|
|o. . o. ... + .|
|.. o. .+o |
+----[SHA256]-----+
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# ls -la
total 12
drwxr-xr-x. 2 root root 38 Sep 28 14:32 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-------. 1 root root 2602 Sep 28 14:32 id_rsa
-rw-r--r--. 1 root root 569 Sep 28 14:32 id_rsa.pub
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# ls -la
total 16
drwxr-xr-x. 2 root root 61 Sep 28 14:34 .
dr-xr-x---. 16 root root 4096 Sep 28 14:31 ..
-rw-r--r--. 1 root root 581 Sep 28 14:34 authorized_keys
-rw-------. 1 root root 2602 Sep 28 14:32 id_rsa
-rw-r--r--. 1 root root 569 Sep 28 14:32 id_rsa.pub
[root@oel82-rac2 .ssh]# cat id_rsa.pub >> authorized_keys
[root@oel82-rac2 .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2

Standby RAC (oel82-rac3 & oel82-rac4)
--------------------------------------------------
Step-4:

[root@oel82-rac2 .ssh]# scp authorized_keys oracle@oel82-rac3:/home/oracle/.ssh
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 1150 0.4KB/s 00:03
[root@oel82-rac2 .ssh]#
[root@oel82-rac2 .ssh]# cd
[root@oel82-rac2 ~]#
[root@oel82-rac2 ~]# vi /etc/fstab
[root@oel82-rac2 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac2 ~]#

[root@oel82-rac2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.7G 8.0K 4.7G 1% /dev
tmpfs 4.8G 845M 4.0G 18% /dev/shm
tmpfs 4.8G 18M 4.8G 1% /run
tmpfs 4.8G 0 4.8G 0% /sys/fs/cgroup
/dev/mapper/ol-root 50G 19G 32G 38% /
/dev/mapper/ol-home 65G 607M 64G 1% /home
/dev/sda1 1014M 295M 720M 30% /boot
tmpfs 968M 1.2M 967M 1% /run/user/42
tmpfs 968M 4.0K 968M 1% /run/user/54321
/dev/asm/acfs_vol-204 10G 570M 9.5G 6% /acfs_vol
[root@oel82-rac2 ~]#
[root@oel82-rac2 ~]# exit
logout
[oracle@oel82-rac2 ~]$ pwd
/home/oracle
[oracle@oel82-rac2 ~]$
[oracle@oel82-rac2 ~]$ cd .ssh/
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 10:29 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-r--r--. 1 oracle oinstall 1737 Sep 28 15:12 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 10:28 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 10:28 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 738 Sep 28 10:31 known_hosts
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac2 .ssh]$

Step-5:

-------------

[oracle@oel82-rac2 .ssh]$ scp authorized_keys oel82-rac3:/home/oracle/.ssh/
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 2308 0.2KB/s 00:12
[oracle@oel82-rac2 .ssh]$
Step-5:
[oracle@oel82-rac3 ~]$ su - root
Password:
pwd
[root@oel82-rac3 ~]# pwd
/root
[root@oel82-rac3 ~]# mkdir ~/.ssh
[root@oel82-rac3 ~]# exit
logout
[oracle@oel82-rac3 ~]$ clear
[oracle@oel82-rac3 ~]$ pwd
/home/oracle
[oracle@oel82-rac3 ~]$ cd .ssh/
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1142 Sep 28 13:19 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1150 Sep 28 14:36 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3

Step-6:

-----------

[oracle@oel82-rac3 .ssh]$ scp authorized_keys oracle@oel82-rac4:/home/oracle/.ssh

authorized_keys 100% 1721 0.4KB/s 00:03
[oracle@oel82-rac3 .ssh]$
Step-6
[oracle@oel82-rac4 ~]$ su - root
Password:
pwd
[root@oel82-rac4 ~]# pwd
/root
[root@oel82-rac4 ~]# mkdir ~/.ssh
[root@oel82-rac4 ~]# exit
logout
[oracle@oel82-rac4 ~]$ cd .ssh/
[oracle@oel82-rac4 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-------. 1 oracle oinstall 1142 Sep 28 13:19 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-------. 1 oracle oinstall 1694 Sep 28 13:19 known_hosts
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6wxULv13UOhUrRy/5dN5NiY48KAM/xLAsO4d0Ea+pdaLTWRt2hreGhE4eEuRZEes4BP5eoVSv/HvrS3xvBJcZVZLDx8wktVODqFH9P9ggLRpi/fLn20memLNz4EwFJt5DRv2UJAeJIQbycHUxt8N/R8h1z8H39V7KVbbUC+PZW6reVYgSSoyczwFj5tdzalwwoPPf4OwyYPb6EXhHWQZ5jFgwxlNfQaS7jGzcjgvKrNDVy+x9Lam7KrMB3IbhaCwZ0i0UrcFusF3PocONK0xbRThY4cztyVKyKVCkpG8/2VCv7yO2EUGc9P4f37DGj6D5WtBvCmWCqJ5S3AwRyyG7ZxWqSlvSjL6OGHFP2xatkr96jq456fbfejL8ZF7UHA886e/OZub1H6zwCJGnEe47oebhyBPrGEWukisZ5YttTuYzOOVWp5MclCaGxGRog10sJrutcX1+4dNqseFaZ11nubwJrLnFIL8Op+JndlewQotGjdgo4dKDF8OpMzuR0/s= oracle@oel82-rac4
[oracle@oel82-rac4 .ssh]$ cd
[oracle@oel82-rac4 ~]$ su - root
Password:
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# vi /etc/fstab
[root@oel82-rac4 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# exit
logout

Step-7: Create volume "acfs_vol" in Primary RAC
-------------------------------------------------------------
[oracle@oel82-rac1 ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 12284 12144 0 12144 0 N ACFSDG/
MOUNTED EXTERN N 512 512 4096 4194304 12284 11944 0 11944 0 Y OCRVD/
ASMCMD> volcreate -G ACFSDG -s 10G acfs_vol
ASMCMD> volinfo --all
Diskgroup Name: ACFSDG
Volume Name: ACFS_VOL
Volume Device: /dev/asm/acfs_vol-204
State: ENABLED
Size (MB): 10240
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
ASMCMD>
ASMCMD> exit
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ su - root
Password:
[root@oel82-rac1 ~]# . oraenv
ORACLE_SID = [/u01/app/19.3.0/grid] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been changed from /home/oracle to /u01/app/oracle
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# modprobe oracleacfs
[root@oel82-rac1 ~]# modprobe oracleadvm
[root@oel82-rac1 ~]# modprobe oracleoks
[root@oel82-rac1 ~]# lsmod | grep oracle
oracleacfs 5025792 1
oracleadvm 712704 7
oracleoks 733184 2 oracleadvm,oracleacfs
oracleafd 221184 1
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# cd /
[root@oel82-rac1 /]# mkdir acfs_vol
[root@oel82-rac1 /]# chown -R oracle:oinstall /acfs_vol
[root@oel82-rac1 /]# mkfs -t acfs /dev/asm/acfs_vol-204
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/acfs_vol-204
mkfs.acfs: volume size = 10737418240 ( 10.00 GB )
mkfs.acfs: Format complete.
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# acfsutil registry -a /dev/asm/acfs_vol-204 /acfs_vol/
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# acfsutil registry -c /dev/asm/acfs_vol-204 /acfs_vol/ -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-204'
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# mount -t acfs /dev/asm/acfs_vol-204 /acfs_vol/
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# mount | grep acfs
/dev/asm/acfs_vol-204 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# vi /etc/fstab
[root@oel82-rac1 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# srvctl config filesystem -d /dev/asm/acfs_vol-204
Volume device: /dev/asm/acfs_vol-204
Diskgroup name: acfsdg
Volume name: acfs_vol
Canonical volume device: /dev/asm/acfs_vol-204
Accelerator volume devices:
Mountpoint path: /acfs_vol
Mount point owner: root
Mount point group: root
Mount permissions: owner:root:rwx,pgrp:root:r-x,other::r-x
Mount users:
Type: ACFS
Mount options:
Description:
ACFS file system is enabled
ACFS file system is individually enabled on nodes:
ACFS file system is individually disabled on nodes:
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# srvctl status filesystem -d /dev/asm/acfs_vol-204
ACFS file system /acfs_vol is mounted on nodes oel82-rac1,oel82-rac2
[root@oel82-rac1 /]#

[root@oel82-rac1 /]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.7G 8.0K 4.7G 1% /dev
tmpfs 4.8G 845M 4.0G 18% /dev/shm
tmpfs 4.8G 18M 4.8G 1% /run
tmpfs 4.8G 0 4.8G 0% /sys/fs/cgroup
/dev/mapper/ol-root 50G 19G 32G 38% /
/dev/mapper/ol-home 65G 607M 64G 1% /home
/dev/sda1 1014M 295M 720M 30% /boot
tmpfs 968M 1.2M 967M 1% /run/user/42
tmpfs 968M 4.0K 968M 1% /run/user/54321
/dev/asm/acfs_vol-204 10G 570M 9.5G 6% /acfs_vol
[root@oel82-rac1 /]#

Step-8:
----------
[root@oel82-rac2 ~]# vi /etc/fstab
[root@oel82-rac2 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-204 /acfs_vol acfs defaults 0 0
[root@oel82-rac2 ~]#

Step-9: Create volume "acfs_vol" in Standby RAC
--------------------------------------------------------------------
[oracle@oel82-rac3 ~]$ . oraenv
ORACLE_SID = [orcl1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 12284 12144 0 12144 0 N ACFSDG/
MOUNTED EXTERN N 512 512 4096 4194304 12284 11944 0 11944 0 Y OCRVD/
ASMCMD>

ASMCMD> volcreate -G ACFSDG -s 10G acfs_vol
ASMCMD> volinfo --all
Diskgroup Name: ACFSDG
Volume Name: ACFS_VOL
Volume Device: /dev/asm/acfs_vol-211
State: ENABLED
Size (MB): 10240
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
ASMCMD>
ASMCMD> exit
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ su - root
Password:
[root@oel82-rac3 ~]#
[root@oel82-rac3 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base has been set to /u01/app/oracle
[root@oel82-rac3 ~]#

[root@oel82-rac3 ~]# modprobe oracleacfs
[root@oel82-rac3 ~]# modprobe oracleadvm
[root@oel82-rac3 ~]# modprobe oracleoks
[root@oel82-rac3 ~]# lsmod | grep oracle
oracleacfs 5025792 1
oracleadvm 712704 7
oracleoks 733184 2 oracleadvm,oracleacfs
oracleafd 221184 1
[root@oel82-rac3 ~]#
[root@oel82-rac3 ~]# cd /

[root@oel82-rac3 /]# mkdir acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# chown -R oracle:oinstall /acfs_vol
[root@oel82-rac3 /]# chmod -R 777 /acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# mkfs -t acfs /dev/asm/acfs_vol-211
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/acfs_vol-211
mkfs.acfs: volume size = 10737418240 ( 10.00 GB )
mkfs.acfs: Format complete.
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# acfsutil registry -a /dev/asm/acfs_vol-211 /acfs_vol/
acfsutil registry: mount point /acfs_vol successfully added to Oracle Registry
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# acfsutil registry -c /dev/asm/acfs_vol-211 /acfs_vol/ -u oracle
acfsutil registry: successfully modified ACFS registration for '/dev/asm/acfs_vol-211'
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# mount -t acfs /dev/asm/acfs_vol-211 /acfs_vol/
[root@oel82-rac3 /]#
[root@oel82-rac3 /]# mount | grep acfs
/dev/asm/acfs_vol-211 on /acfs_vol type acfs (rw,relatime,device,rootsuid,ordered)
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# vi /etc/fstab
[root@oel82-rac3 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl config filesystem -d /dev/asm/acfs_vol-211
Volume device: /dev/asm/acfs_vol-211
Diskgroup name: acfsdg
Volume name: acfs_vol
Canonical volume device: /dev/asm/acfs_vol-211
Accelerator volume devices:
Mountpoint path: /acfs_vol
Mount point owner: root
Mount point group: root
Mount permissions: owner:root:rwx,pgrp:root:r-x,other::r-x
Mount users:
Type: ACFS
Mount options:
Description:
ACFS file system is enabled
ACFS file system is individually enabled on nodes:
ACFS file system is individually disabled on nodes:
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl status filesystem -d /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3,oel82-rac4
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl stop filesystem -d /dev/asm/acfs_vol-211 -n oel82-rac4
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# srvctl status filesystem -d /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3
[root@oel82-rac3 /]#

Step-10:
-----------
[oracle@oel82-rac4 ~]$ su - root
Password:
[root@oel82-rac4 ~]#
[root@oel82-rac4 ~]# vi /etc/fstab
[root@oel82-rac4 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Sep 27 12:23:02 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=7979871f-05e0-4f7c-a838-e6e8addf8454 /boot xfs defaults 0 0
/dev/mapper/ol-home /home xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
/dev/asm/acfs_vol-211 /acfs_vol acfs defaults 0 0
[root@oel82-rac4 ~]#

Step-11: Initiate acfs replication in standby
-------------------------------------------------------
[root@oel82-rac3 /]# /sbin/acfsutil repl init standby -u oracle /acfs_vol
[root@oel82-rac3 /]#

[root@oel82-rac3 /]# /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:00:38 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:00:38 2021
Number of Events: 1
[root@oel82-rac3 /]#

Step-12: Start acfs replication in primary rac
--------------------------------------------------------------
[root@oel82-rac1 /]# /sbin/acfsutil repl info -c -v -u oracle oel82-rac3 /acfs_vol/
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
' -- retrying].
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
' -- retrying].
acfsutil repl info: ACFS-10581: Internal API error : [Command '/usr/bin/ssh -o Ciphers=aes128-ctr -o BatchMode=true -x oracle@oel82-rac3 "/bin/sh -c 'true'" 2>&1' failed with output 'Host key verification failed.
'].
acfsutil repl info: ACFS-05914: user root on node oel82-rac1 unable to log in via 'ssh' to the standby node oel82-rac3 as user oracle
[root@oel82-rac1 /]#
[root@oel82-rac1 /]# exit
logout
[oracle@oel82-rac1 ~]$

Note: I have got the above error and I have copied "keys" for Oracle user from primary rac to standby rac

[oracle@oel82-rac1 ~]$ pwd
/home/oracle
[oracle@oel82-rac1 ~]$ cd .ssh/
[oracle@oel82-rac1 .ssh]$
[oracle@oel82-rac1 .ssh]$ ls -la
total 28
drwx------. 2 oracle oinstall 134 Sep 28 10:35 .
drwx------. 7 oracle oinstall 4096 Sep 28 15:06 ..
-rw-r--r--. 1 oracle oinstall 1154 Sep 28 10:29 authorized_keys
-rw-------. 1 oracle oinstall 2622 Sep 28 10:26 id_rsa
-rw-r--r--. 1 oracle oinstall 583 Sep 28 10:26 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 1292 Sep 28 15:08 known_hosts
-rw-------. 1 oracle oinstall 2622 Sep 28 10:35 oel82-rac1.id_rsa
-rw-r--r--. 1 oracle oinstall 583 Sep 28 10:35 oel82-rac1.id_rsa.pub

[oracle@oel82-rac1 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
[oracle@oel82-rac1 .ssh]$

[oracle@oel82-rac1 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac1 .ssh]$

[oracle@oel82-rac1 .ssh]$ scp authorized_keys oel82-rac2:/home/oracle/.ssh/
authorized_keys 100% 1737 13.2KB/s 00:00
[oracle@oel82-rac1 .ssh]$
[oracle@oel82-rac2 ~]$ pwd
/home/oracle
[oracle@oel82-rac2 ~]$
[oracle@oel82-rac2 ~]$ cd .ssh/
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 10:29 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:30 ..
-rw-r--r--. 1 oracle oinstall 1737 Sep 28 15:12 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 10:28 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 10:28 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 738 Sep 28 10:31 known_hosts
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac2 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac2 .ssh]$

[oracle@oel82-rac2 .ssh]$ scp authorized_keys oel82-rac3:/home/oracle/.ssh/
The authenticity of host 'oel82-rac3 (192.168.2.40)' can't be established.
ECDSA key fingerprint is SHA256:id0Rgchdgz/5hmVmH1h40gc+T9QvJHbRAlH7qDerKh0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'oel82-rac3,192.168.2.40' (ECDSA) to the list of known hosts.
oracle@oel82-rac3's password:
authorized_keys 100% 2308 0.2KB/s 00:12
[oracle@oel82-rac2 .ssh]$
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ cd .ssh/
[oracle@oel82-rac3 .ssh]$ ls -l
total 16
-rw-------. 1 oracle oinstall 1721 Sep 28 14:38 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$

[oracle@oel82-rac3 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEEJLEhAR/akJ+oGb8XK3sCYMOMRkMsr3AToz3tAtEGWshGxlgJUF8QdKIQnDMJIXSsy/Ax6yYhmmC7UamxFR7VZnhfdi4J+jaRqFsfU98DKehENLFzxzc2MZq9W/mlpIOsxPJXxbbRGFxl+tko1rN9Fko83mhNizvPvpjlonOucHP8AtLtlxV47/h3ZLlIOaZxvGiWR7250WlZPhCxhiM2oOwg/JZVjuqn2xVTAElCxqq0nh3MUK28Z86+PPgw71OLLAs5urilM32A6c2WhH5dozdMbjOLVZwfQuKoOMt/i7p6+TXf1q1sNMli02sthoVWyO/Hb4M7sVQZqTFW+YGIf+rtIW+APtWuSNCHlnS4BZyjfWSU8nWt24ES+ug4O+mLani9G+dlNuwIHYq1zk09M0t0FbF+EJT86AFh2ZselWjIjcB2+ZWxHcXtgGkxV4TNjKphFJdCkhCB7EWo7UPN8bDPqW9IyZ4/hJVLKIH35++A73T14FmpO+goaRNOEc= root@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWGWFengAqAwu1zB4Kgsvx+OfLhdvzZfvOSD8dcgzeA2V1wqOewcaivDhdrBLSjTD3bD1q6tsMKt2bUza4b6gxTaEuyFyBfdGyq/ctRcNeJa2NZCaR9OoKgDVKQvexaCC87/LgrA7Gxrp2rNfbRUeki4oWOAvlBrh9MQQJJyElET0ZS5mSTKSGTrAAbHxOA0nkY8WIyOgZppa1LWlBEDVZ5DpshWxmYVIIyW3G/pAtX+a/fwNq99trR962PPqEMXXvyb+zFYckg150haPy+4vTbQLjhyN4RALt0DOLIj/wdO7tBo48NJiszcmE7RhI+1Ni/afHIV7njqflyw78RBJJl+m92josrusUilFQRbaOYhGUrm+W/9k8uCFmdq31ZaRaoDU0DHItn789SZ12+dYDcvHh0PbatYZplyCPk5txQ5gF0MVUIFFXRaYOJvusxMr054RpT+I5xnv/VNtQyCgSyg8iN++Mwz9NlqrPQErtBTWqWOphGTe3cWxcd7uGbH8= root@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2308 Sep 28 15:13 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ scp authorized_keys oel82-rac4:/home/oracle/.ssh/
authorized_keys 100% 2879 18.0KB/s 00:00
[oracle@oel82-rac3 .ssh]$
[oracle@oel82-rac3 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:54 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-r--r--. 1 oracle oinstall 2260 Sep 28 13:20 known_hosts
[oracle@oel82-rac3 .ssh]$

[oracle@oel82-rac4 ~]$ cd .ssh/
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ ls -la
total 20
drwx------. 2 oracle oinstall 80 Sep 28 13:19 .
drwx------. 7 oracle oinstall 4096 Sep 28 14:58 ..
-rw-------. 1 oracle oinstall 2879 Sep 28 15:14 authorized_keys
-rw-------. 1 oracle oinstall 2602 Sep 28 13:19 id_rsa
-rw-r--r--. 1 oracle oinstall 571 Sep 28 13:19 id_rsa.pub
-rw-------. 1 oracle oinstall 1694 Sep 28 13:19 known_hosts
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
[oracle@oel82-rac4 .ssh]$
[oracle@oel82-rac4 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@oel82-rac4 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYuikJ8T2Z3WP6Zkh3MJz4UbQuO0oe5sJf1OMNJMdH2ANtvVQgONuVo+qtDA3D7RyAPtiQRIXEdgyX6pTTD6bxw5wwglm7zFKPyBmyqGIr47qeBCUXDya2DcSvpe7gO9P+8VI2Ggi2UIPMx3jz+ODmysiwA425ech4/NAbrC54hyF70vL+3PyoDa2ook8iwo5insj8jgXJQnqlGdiPmKJXYnDERNJWgL5DTNb06qn4Hr1tTnJy5u2XJ+SalFyFNbRyAPwN/Hp2jtqFE73INCjGkZR5W9MnZM8nUlxy2vUoVXE5UJqCOhnXDsudpKD4wU77n3pF4WP1wTRfd+q5G6rgE3XOqPu3cRdGgM0wGYAh7yQC1xFj0bljGaHHw6aK2aS3d/SOhiwF1S4Lri4pd/dqg9ONkhjzAE9NeO2Vt+DJajwxbnLtRLaTkSe7uczLqyXxzicB4rlG1p/QXdfJpPUNuN+8eZ84NmmOvjnyArscAjm7LuPBeMFHyRFcg30yR9M= oracle@oel82-rac1.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaEosyGo9Gfv0TNKzqNQX+J90FdNmORdANx8U5D6o9zcRTmXRArIXK9+M1Wq9/+9sJFRe7HD4tEZOCqaNsMRSzQWbpMTCv78D/BnLg4gJDbOPDH9QBNbxuf1wMzOX2eGlZaNSbrFQ8IdzmWaDTBwsImFZWk6k8M+g9paeVOvCi0ihpirgmK+DkHaERqCm30xEgQ5abu0hOV6AvvjhuSLDbz3QWd36IQeXjJKKd0l44loeX4t+5So6dOTPU6+JxwEaJJ4KWOF5LipX+IDRG249qI81aBOUwhbApuBnumeOj/u+EW/8ns9IMoom5oFjvPNgwlXS5HiM46AmYl4dExaD+jg9YOnK5LrGyXyjt3YL0MVzbuww6za2DJUvNKxEwIX/Lv1+ZbZZooY9rYf0ZvFHBWCQSDp/5e3AS0tspyj+HWW4faEIFbL8KXZOAIwX5udTlVRfUYc9YGeMWFQRkqtHgWCMLr4e/ZssJKv+SNSJrF7ddW4ybHn5frJfKnTnFF/M= oracle@oel82-rac2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwrs6opgau+3lMQIGoO4ooLUXsnk4nA2Iu9HwRMbo2OxBFnRO3PZevIovdh9Ccclv9NtchHqXdUZoNi2UlQPXmJHm4LyG0uv0xz55LtTdrLUxaJYol8JRzOLUOPCad4yhc4uMfZPLLsTkiUMi6ucYDGmYWSgOObn7H4UacWW/exd+B6FpaCv5t/MryPf2WIvfYYvA7WOIgzh7i/NndrQSld0GzWyXBdItsTCGabUzcTvsoTFKZi0qk5WiPV2Be3iBrDNqNspQ58/3US4J3K/uSsj4OvkbrPSpvdjGOnHonSPMaFaB01UbZOcaQcCEiY8nNbM03EwCqMQnuh0w5LdHq2c4ne+Fa9aLtrrPAsOcXApMKfFwqRL3MobWGuBQXp21JysheU0JCvJs28fCSou7Mf7ZkbjC/E44R/eUJKYQAmc/mGwy8t89gbPLnYGI/aggt8usA+E38/TrnlQ56iqFkIdvhXW8lUZTKuVV8ve8le4SfohyWDQTzHoMTQJhHjIk= oracle@oel82-rac3
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6wxULv13UOhUrRy/5dN5NiY48KAM/xLAsO4d0Ea+pdaLTWRt2hreGhE4eEuRZEes4BP5eoVSv/HvrS3xvBJcZVZLDx8wktVODqFH9P9ggLRpi/fLn20memLNz4EwFJt5DRv2UJAeJIQbycHUxt8N/R8h1z8H39V7KVbbUC+PZW6reVYgSSoyczwFj5tdzalwwoPPf4OwyYPb6EXhHWQZ5jFgwxlNfQaS7jGzcjgvKrNDVy+x9Lam7KrMB3IbhaCwZ0i0UrcFusF3PocONK0xbRThY4cztyVKyKVCkpG8/2VCv7yO2EUGc9P4f37DGj6D5WtBvCmWCqJ5S3AwRyyG7ZxWqSlvSjL6OGHFP2xatkr96jq456fbfejL8ZF7UHA886e/OZub1H6zwCJGnEe47oebhyBPrGEWukisZ5YttTuYzOOVWp5MclCaGxGRog10sJrutcX1+4dNqseFaZ11nubwJrLnFIL8Op+JndlewQotGjdgo4dKDF8OpMzuR0/s= oracle@oel82-rac4
[oracle@oel82-rac4 .ssh]$

Step-13:
------------
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac2 date
Tue Sep 28 15:15:38 CDT 2021
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac3 date
Tue Sep 28 15:15:51 CDT 2021
[oracle@oel82-rac1 ~]$ ssh oracle@oel82-rac4 date
Tue Sep 28 15:16:02 CDT 2021
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c -v -u oracle oel82-rac3 /acfs_vol/
A valid 'ssh' connection was detected for standby node oel82-rac3 as user oracle.
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl init primary -C -s oracle@oel82-rac3 -m /acfs_vol/ /acfs_vol/

[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 3
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ echo "Test ACFS Replication" > /acfs_vol/replication_testing.txt
[oracle@oel82-rac1 ~]$ cd /acfs_vol/
[oracle@oel82-rac1 acfs_vol]$ ls -la
total 100
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:20 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:47 ..
drwxrwxrwx. 2 oracle oinstall 65536 Sep 28 14:48 lost+found
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac1 acfs_vol]$

[oracle@oel82-rac1 acfs_vol]$ cat replication_testing.txt

Test ACFS Replication

[oracle@oel82-rac1 acfs_vol]$

[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 20
[oracle@oel82-rac1 acfs_vol]$

Step-14:
-----------
[oracle@oel82-rac3 /]$ cd acfs_vol/
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$ ls -la
total 96
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:20 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:55 ..
drwx------. 2 root root 65536 Sep 28 14:56 lost+found
[oracle@oel82-rac3 acfs_vol]$ ls
lost+found
[oracle@oel82-rac3 acfs_vol]$ ls
lost+found
[oracle@oel82-rac3 acfs_vol]$
[oracle@oel82-rac3 acfs_vol]$ ls -la
total 100
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:21 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:55 ..
drwx------. 2 root root 65536 Sep 28 14:56 lost+found
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac3 acfs_vol]$

[oracle@oel82-rac3 acfs_vol]$ cat replication_testing.txt
Test ACFS Replication
[oracle@oel82-rac3 acfs_vol]$

[oracle@oel82-rac3 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:21:35 2021
Apply completed at: Tue Sep 28 15:21:35 2021
Time: Tue Sep 28 15:21:34 2021
Event: Clone applied
Session started at: Tue Sep 28 15:21:34 2021
Session ended at: Tue Sep 28 15:21:34 2021
Number of bytes applied: 23148
Time: Tue Sep 28 15:21:17 2021
Apply started at: Tue Sep 28 15:21:17 2021
Time: Tue Sep 28 15:20:45 2021
Apply completed at: Tue Sep 28 15:20:45 2021
Time: Tue Sep 28 15:20:44 2021
Event: Clone applied
Session started at: Tue Sep 28 15:20:44 2021
Session ended at: Tue Sep 28 15:20:44 2021
Number of bytes applied: 21114
Time: Tue Sep 28 15:20:44 2021
Apply started at: Tue Sep 28 15:20:44 2021
Time: Tue Sep 28 15:20:21 2021
Apply completed at: Tue Sep 28 15:20:21 2021
Time: Tue Sep 28 15:20:20 2021
Event: Clone applied
Session started at: Tue Sep 28 15:20:17 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes applied: 21114
Time: Tue Sep 28 15:20:17 2021
Apply started at: Tue Sep 28 15:20:17 2021
Time: Tue Sep 28 15:18:48 2021
Apply completed at: Tue Sep 28 15:18:48 2021
Time: Tue Sep 28 15:18:48 2021
Event: Clone applied
Session started at: Tue Sep 28 15:18:48 2021
Session ended at: Tue Sep 28 15:18:48 2021
Number of bytes applied: 21499
Time: Tue Sep 28 15:18:48 2021
Apply started at: Tue Sep 28 15:18:48 2021
Time: Tue Sep 28 15:00:38 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:00:38 2021
Number of Events: 13
[oracle@oel82-rac3 acfs_vol]$

Step-15: Testing replication:
-------------------------------------
[oracle@oel82-rac1 acfs_vol]$ pwd
/acfs_vol
[oracle@oel82-rac1 acfs_vol]$ cd
[oracle@oel82-rac1 ~]$ ls
p6880880_190000_Linux-x86-64.zip PatchSearch.xml
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ cp p6880880_190000_Linux-x86-64.zip /acfs_vol/
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ cd /acfs_vol/
[oracle@oel82-rac1 acfs_vol]$
[oracle@oel82-rac1 acfs_vol]$ ls -la
total 122960
drwxrwxrwx. 4 oracle oinstall 32768 Sep 28 15:25 .
dr-xr-xr-x. 19 root root 251 Sep 28 14:47 ..
drwxrwxrwx. 2 oracle oinstall 65536 Sep 28 14:48 lost+found
-rwxr-xr-x. 1 oracle oinstall 120998448 Sep 28 15:25 p6880880_190000_Linux-x86-64.zip
-rw-r--r--. 1 oracle oinstall 22 Sep 28 15:20 replication_testing.txt
[oracle@oel82-rac1 acfs_vol]$
[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:25:23 2021
Create started at: Tue Sep 28 15:25:23 2021
Time: Tue Sep 28 15:25:22 2021
Transfer started at: Tue Sep 28 15:25:22 2021
Time: Tue Sep 28 15:21:35 2021
Transfer completed at: Tue Sep 28 15:21:35 2021
Transfer status: 0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 23

[oracle@oel82-rac1 acfs_vol]$ /sbin/acfsutil repl info -v /acfs_vol/
Events Log: /acfs_vol/.ACFS/repl/logs/ReplicationEventsLog_0
Time: Tue Sep 28 15:25:23 2021
Create started at: Tue Sep 28 15:25:23 2021
Time: Tue Sep 28 15:25:22 2021
Transfer started at: Tue Sep 28 15:25:22 2021
Time: Tue Sep 28 15:21:35 2021
Transfer completed at: Tue Sep 28 15:21:35 2021
Transfer status: 0
Time: Tue Sep 28 15:21:02 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:55 2021
Session ended at: Tue Sep 28 15:21:02 2021
Number of bytes transmitted: 23532
Time: Tue Sep 28 15:21:02 2021
Create completed at: Tue Sep 28 15:21:02 2021
Create status: 0
Time: Tue Sep 28 15:20:55 2021
Create started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:55 2021
Transfer started at: Tue Sep 28 15:20:55 2021
Time: Tue Sep 28 15:20:46 2021
Transfer completed at: Tue Sep 28 15:20:46 2021
Transfer status: 0
Time: Tue Sep 28 15:20:37 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:31 2021
Session ended at: Tue Sep 28 15:20:37 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:37 2021
Create completed at: Tue Sep 28 15:20:37 2021
Create status: 0
Time: Tue Sep 28 15:20:31 2021
Create started at: Tue Sep 28 15:20:31 2021
Time: Tue Sep 28 15:20:30 2021
Transfer started at: Tue Sep 28 15:20:30 2021
Time: Tue Sep 28 15:20:21 2021
Transfer completed at: Tue Sep 28 15:20:21 2021
Transfer status: 0
Time: Tue Sep 28 15:20:20 2021
Event: Clone created
Session started at: Tue Sep 28 15:20:13 2021
Session ended at: Tue Sep 28 15:20:20 2021
Number of bytes transmitted: 21498
Time: Tue Sep 28 15:20:19 2021
Create completed at: Tue Sep 28 15:20:19 2021
Create status: 0
Time: Tue Sep 28 15:20:12 2021
Create started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:20:12 2021
Transfer started at: Tue Sep 28 15:20:12 2021
Time: Tue Sep 28 15:18:49 2021
Transfer completed at: Tue Sep 28 15:18:48 2021
Transfer status: 0
Time: Tue Sep 28 15:18:32 2021
Event: Clone created
Session started at: Tue Sep 28 15:18:25 2021
Session ended at: Tue Sep 28 15:18:32 2021
Number of bytes transmitted: 21883
Time: Tue Sep 28 15:18:31 2021
Create completed at: Tue Sep 28 15:18:31 2021
Create status: 0
Time: Tue Sep 28 15:18:25 2021
Create started at: Tue Sep 28 15:18:25 2021
Time: Tue Sep 28 15:18:25 2021
Transfer started at: Tue Sep 28 15:18:24 2021
Time: Tue Sep 28 15:18:25 2021
Event: Replication Initialization Started
Initialization Start Time: Tue Sep 28 15:18:24 2021
Number of Events: 23

[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Standby
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Last sync time with primary: Tue Sep 28 15:25:19 2021
Receiving primary as of: Tue Sep 28 15:25:17 2021
Status: Receive Completed
Last receive started at: Tue Sep 28 15:27:17 2021
Last receive completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last receive: 0 days, 0 hours, 0 minutes, 25 seconds
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac3 ~]$

Step-16:
------------
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -s /acfs_vol/
-------------------------------------------------------
Tue Sep 28 15:18:25 2021 - Tue Sep 28 15:27:42 2021
-------------------------------------------------------
Data replicated: 0.11GB
Avg. rate of data replication: 12.48MB/minute
Avg. time from capture to apply: 00:00:32
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl bg info /acfs_vol/
Resource: ora.repl.dupd.acfsdg.acfs_vol.acfs
Target State: ONLINE , OFFLINE
Current State: ONLINE on oel82-rac1, OFFLINE on oel82-rac2
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil info fs -s /acfs_vol/
/acfs_vol/
amount of change since mount: 115.53 MB
average rate of change since mount: 31.46 KB/s
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10066976768 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10066976768 ( 9.38 GB )
metadata read I/O count: 18287
metadata write I/O count: 768
total metadata bytes read: 78139392 ( 74.52 MB )
total metadata bytes written: 7852032 ( 7.49 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 393216 ( 384.00 KB )
replication status: primary
compression status: DISABLED
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl info -s /acfs_vol/
-------------------------------------------------------
Tue Sep 28 15:00:38 2021 - Tue Sep 28 15:27:42 2021
-------------------------------------------------------
Data replicated: 0.11GB
Avg. rate of data replication: 4.28MB/minute
Average time to apply snapshot duplicate stream: 00:00:05
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ /sbin/acfsutil repl bg info /acfs_vol/
acfsutil repl bg: ACFS-05058: command issued on the incorrect replication site
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac3 ~]$ /sbin/acfsutil info fs -s /acfs_vol/
/acfs_vol/
amount of change since mount: 115.48 MB
average rate of change since mount: 35.89 KB/s
[oracle@oel82-rac3 ~]$
[oracle@oel82-rac3 ~]$ /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:55:25 2021
mount time: Tue Sep 28 14:56:28 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 9999724544 ( 9.31 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-211
label:
state: Available
major, minor: 250, 108033
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 9999724544 ( 9.31 GB )
metadata read I/O count: 2343
metadata write I/O count: 731
total metadata bytes read: 32858112 ( 31.34 MB )
total metadata bytes written: 6725632 ( 6.41 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 393216 ( 384.00 KB )
replication status: standby
compression status: DISABLED
[oracle@oel82-rac3 ~]$
Step-17:
---------
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl pause /acfs_vol/
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Paused
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 15:25:17 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 15:25:22 2021
Last send completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last send: 0 days, 0 hours, 2 minutes, 20 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl resume /acfs_vol/
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Running
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 15:25:17 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 15:25:22 2021
Last send completed at: Tue Sep 28 15:27:42 2021
Elapsed time for last send: 0 days, 0 hours, 2 minutes, 20 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil compress on /acfs_vol/
acfsutil compress on: ACFS-09130: Root access required
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl sync apply /acfs_vol
[oracle@oel82-rac1 ~]$ /sbin/acfsutil repl info -c /acfs_vol/
Site: Primary
Primary hostname: oel82-rac1
Primary path: /acfs_vol
Primary status: Running
Background Resources: Active
Standby connect string: oracle@oel82-rac3
Standby path: /acfs_vol
Replication interval: 0 days, 0 hours, 0 minutes, 0 seconds
Sending primary as of: Tue Sep 28 16:01:44 2021
Status: Send Completed
Lag Time: 00:00:00
Retries made: 0
Last send started at: Tue Sep 28 16:01:44 2021
Last send completed at: Tue Sep 28 16:01:53 2021
Elapsed time for last send: 0 days, 0 hours, 0 minutes, 9 seconds
Next send starts at: now
Replicated tags:
Data transfer compression: Off
ssh strict host key checking: On
Debug log level: 3
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac1 ~]$ su - root
Password:
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil compress on /acfs_vol/
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info fs /acfs_vol/
/acfs_vol/
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10067369984 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10067369984 ( 9.38 GB )
metadata read I/O count: 22542
metadata write I/O count: 958
total metadata bytes read: 96133120 ( 91.68 MB )
total metadata bytes written: 9535488 ( 9.09 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 786432 ( 768.00 KB )
replication status: primary
compression status: ENABLED
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info fs
/acfs_vol
ACFS Version: 19.0.0.0.0
on-disk version: 49.0
compatible.advm: 19.0.0.0.0
ACFS compatibility: 19.0.0.0.0
flags: MountPoint,Available,Replication,KiloSnap
creation time: Tue Sep 28 14:47:47 2021
mount time: Tue Sep 28 14:48:36 2021
mount sequence number: 0
number of nodes: 2
allocation unit: 4096
metadata block size: 4096
volumes: 1
total size: 10737418240 ( 10.00 GB )
total free: 10067369984 ( 9.38 GB )
file entry table allocation: 393216
primary volume: /dev/asm/acfs_vol-204
label:
state: Available
major, minor: 250, 104449
logical sector size: 512
size: 10737418240 ( 10.00 GB )
free: 10067369984 ( 9.38 GB )
metadata read I/O count: 22554
metadata write I/O count: 958
total metadata bytes read: 96182272 ( 91.73 MB )
total metadata bytes written: 9535488 ( 9.09 MB )
ADVM diskgroup: ACFSDG
ADVM resize increment: 67108864
ADVM redundancy: unprotected
ADVM stripe columns: 8
ADVM stripe width: 1048576
number of snapshots: 1
snapshot space usage: 786432 ( 768.00 KB )
replication status: primary
compression status: ENABLED
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil meta -f /home/oracle/mymetafile /dev/asm/acfs_vol-204
Reading configuration information from device /dev/asm/acfs_vol-204 ...
Log file name: /home/oracle/mymetafile.log
Input device name: /dev/asm/acfs_vol-204
Output file name: /home/oracle/mymetafile
Copying first 10 megabytes from device /dev/asm/acfs_vol-204 ...
Continue scanning metadata on device /dev/asm/acfs_vol-204 ...
End of general metadata scan on device /dev/asm/acfs_vol-204 ...
Copying global bit map from device /dev/asm/acfs_vol-204 ...
Copying snap map bit map from device /dev/asm/acfs_vol-204 ...
Copying local bit map from device /dev/asm/acfs_vol-204 ...
Copying local bit map from device /dev/asm/acfs_vol-204 ...
acfsutil meta summary:
Log file name: /home/oracle/mymetafile.log
Input device name: /dev/asm/acfs_vol-204
Output file name: /home/oracle/mymetafile
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# /sbin/acfsutil info ftrace -s /acfs_vol/
Fileid: 2, Pathname: /acfs_vol
Fileid: 7, Pathname: /acfs_vol/.ACFS
Fileid: 9, Pathname: /acfs_vol/.ACFS/repl
Fileid: 11, Pathname: /acfs_vol/.ACFS/repl/ready
Fileid: 43, Pathname: /acfs_vol/.ACFS/repl/ready/DupdRunning_0
[root@oel82-rac1 ~]#

[oracle@oel82-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid/
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.acfsdg.acfs_vol.acfs
ONLINE ONLINE oel82-rac1 mounted on /acfs_vol
,STABLE
ONLINE ONLINE oel82-rac2 mounted on /acfs_vol
,STABLE
ora.chad
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.net1.network
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.ons
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
ora.proxy_advm
ONLINE ONLINE oel82-rac1 STABLE
ONLINE ONLINE oel82-rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel82-rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel82-rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel82-rac1 STABLE
ora.OCRVD.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 Started,STABLE
2 ONLINE ONLINE oel82-rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel82-rac1 STABLE
2 ONLINE ONLINE oel82-rac2 STABLE
ora.cvu
1 ONLINE ONLINE oel82-rac1 STABLE
ora.oel82-rac1.vip
1 ONLINE ONLINE oel82-rac1 STABLE
ora.oel82-rac2.vip
1 ONLINE ONLINE oel82-rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE oel82-rac1 STABLE
ora.repl.dupd.acfsdg.acfs_vol.acfs
1 ONLINE ONLINE oel82-rac1 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel82-rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE oel82-rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE oel82-rac1 STABLE
--------------------------------------------------------------------------------
[oracle@oel82-rac1 ~]$

[oracle@oel82-rac3 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSDG.ACFS_VOL.advm
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.acfsdg.acfs_vol.acfs
ONLINE ONLINE oel82-rac3 mounted on /acfs_vol
,STABLE
OFFLINE OFFLINE oel82-rac4 admin unmounted /acf
s_vol,STABLE
ora.chad
ONLINE OFFLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.net1.network
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.ons
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
ora.proxy_advm
ONLINE ONLINE oel82-rac3 STABLE
ONLINE ONLINE oel82-rac4 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ACFSDG.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel82-rac4 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel82-rac3 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel82-rac3 STABLE
ora.OCRVD.dg(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 Started,STABLE
2 ONLINE ONLINE oel82-rac4 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel82-rac3 STABLE
2 ONLINE ONLINE oel82-rac4 STABLE
ora.cvu
1 ONLINE ONLINE oel82-rac3 STABLE
ora.oel82-rac3.vip
1 ONLINE ONLINE oel82-rac3 STABLE
ora.oel82-rac4.vip
1 ONLINE ONLINE oel82-rac4 STABLE
ora.qosmserver
1 ONLINE ONLINE oel82-rac3 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel82-rac4 STABLE
ora.scan2.vip
1 ONLINE ONLINE oel82-rac3 STABLE
ora.scan3.vip
1 ONLINE ONLINE oel82-rac3 STABLE
--------------------------------------------------------------------------------
[oracle@oel82-rac3 ~]$

[oracle@oel82-rac1 ~]$ findmnt -t acfs
TARGET SOURCE FSTYPE OPTIONS
/acfs_vol /dev/asm/acfs_vol-204 acfs rw,relatime,device,rootsuid,ordered
[oracle@oel82-rac1 ~]$
[oracle@oel82-rac3 ~]$ findmnt -t acfs
TARGET SOURCE FSTYPE OPTIONS
/acfs_vol /dev/asm/acfs_vol-211 acfs rw,relatime,device,rootsuid,ordered
[oracle@oel82-rac3 ~]$

[root@oel82-rac1 ~]# . oraenv
ORACLE_SID = [+ASM] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid/
The Oracle base has been set to /u01/app/oracle
[root@oel82-rac1 ~]#
[root@oel82-rac1 ~]# srvctl status filesystem -device /dev/asm/acfs_vol-204
ACFS file system /acfs_vol is mounted on nodes oel82-rac1,oel82-rac2
[root@oel82-rac1 ~]#
[root@oel82-rac3 ~]# srvctl status filesystem -device /dev/asm/acfs_vol-211
ACFS file system /acfs_vol is mounted on nodes oel82-rac3
[root@oel82-rac3 ~]#

Reflection of my thoughts and technical expriences

Sunday, October 3, 2021

Oracle 19c (19.12) - Oracle ASM Cluster File System (ACFS) Replication.

3 comments: