Saturday, June 6, 2020

Configuring Kubernetes Cluster with Master Node, Two Worker Nodes on CENTOS7


Configuring Kubernetes Cluster With Master Node, Two Worker Nodes on CENTOS7

Master Node	: 192.168.2.130 (centos7)
Worker Node-1	: 192.168.2.131 (centos7-w1)
Worker Node-2	: 192.168.2.132 (centos7-w2)
Work Station	: 192.168.2.133 (centos7-ws)

Master Node	: 192.168.2.130 (centos7)
-----------------------------------------

1. Check the Status Of Docker:
-----------------------------

[root@centos7 ~]# systemctl status docker
? docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-31 12:08:08 CDT; 4h 4min ago
     Docs: https://docs.docker.com
 Main PID: 5561 (dockerd)
    Tasks: 18
   Memory: 82.9M
   CGroup: /system.slice/docker.service
           +-5561 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

May 31 16:04:31 centos7 dockerd[5561]: time="2020-05-31T16:04:31.921506028-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:05:32 centos7 dockerd[5561]: time="2020-05-31T16:05:32.234346689-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:06:32 centos7 dockerd[5561]: time="2020-05-31T16:06:32.553116712-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:07:32 centos7 dockerd[5561]: time="2020-05-31T16:07:32.875037752-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:08:33 centos7 dockerd[5561]: time="2020-05-31T16:08:33.192984415-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:09:28 centos7 dockerd[5561]: time="2020-05-31T16:09:28.727330062-05:00" level=info msg="NetworkDB stats centos7(9e86f33306e3) - netID:fhzqs13pt2x...etMsg/s:0"
May 31 16:09:33 centos7 dockerd[5561]: time="2020-05-31T16:09:33.499272211-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:10:33 centos7 dockerd[5561]: time="2020-05-31T16:10:33.847703789-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:11:34 centos7 dockerd[5561]: time="2020-05-31T16:11:34.182730856-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
May 31 16:12:34 centos7 dockerd[5561]: time="2020-05-31T16:12:34.488821680-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic...askDelete"
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos7 ~]#

2. Login to your kubernetes master node and set the hostname and disable selinux using following commands
---------------------------------------------------------------------------------------------------------
[root@centos7 ~]# hostnamectl set-hostname centos7

[root@centos7 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.130 centos7.localdomain centos7
192.168.2.131 centos7-w1.localdomain centos7-w1
192.168.2.132 centos7-w2.localdomain centos7-w2
192.168.2.133 centos7-ws.localdomain centos7-ws
[root@centos7 ~]#

[root@centos7 ~]# ping centos7-w1
PING centos7-w1.localdomain (192.168.2.131) 56(84) bytes of data.
64 bytes from centos7-w1.localdomain (192.168.2.131): icmp_seq=1 ttl=64 time=0.361 ms
64 bytes from centos7-w1.localdomain (192.168.2.131): icmp_seq=1 ttl=63 time=0.534 ms (DUP!)
64 bytes from centos7-w1.localdomain (192.168.2.131): icmp_seq=1 ttl=64 time=0.537 ms (DUP!)
64 bytes from centos7-w1.localdomain (192.168.2.131): icmp_seq=1 ttl=63 time=0.746 ms (DUP!)
^C
--- centos7-w1.localdomain ping statistics ---
1 packets transmitted, 1 received, +3 duplicates, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.361/0.544/0.746/0.138 ms
[root@centos7 ~]#

[root@centos7 ~]# ping centos7-w2
PING centos7-w2.localdomain (192.168.2.132) 56(84) bytes of data.
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=1 ttl=64 time=0.130 ms
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=1 ttl=64 time=0.188 ms (DUP!)
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=1 ttl=63 time=0.192 ms (DUP!)
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=1 ttl=63 time=0.298 ms (DUP!)
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=2 ttl=64 time=0.442 ms
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=2 ttl=63 time=0.566 ms (DUP!)
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=2 ttl=64 time=0.575 ms (DUP!)
64 bytes from centos7-w2.localdomain (192.168.2.132): icmp_seq=2 ttl=63 time=0.647 ms (DUP!)
^C
--- centos7-w2.localdomain ping statistics ---
2 packets transmitted, 2 received, +6 duplicates, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.130/0.379/0.647/0.191 ms
[root@centos7 ~]#

[root@centos7 ~]# setenforce 0
[root@centos7 ~]# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
[root@centos7 ~]# modprobe br_netfilter
[root@centos7 ~]# echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

3. Update latest Docker Binaries
--------------------------------
[root@master-node ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: repos-tx.psychz.net
 * extras: mirror.mobap.edu
 * updates: mirror.umd.edu
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-1.el7 will be updated
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-2.el7 will be an update
---> Package lvm2.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2 = 7:2.02.185-2.el7 for package: 7:lvm2-cluster-2.02.185-2.el7.x86_64
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: lvm2-libs = 7:2.02.186-7.el7_8.2 for package: 7:lvm2-2.02.186-7.el7_8.2.x86_64
---> Package yum-utils.noarch 0:1.1.31-52.el7 will be updated
---> Package yum-utils.noarch 0:1.1.31-54.el7_8 will be an update
--> Running transaction check
---> Package lvm2-cluster.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper = 7:1.02.164 for package: 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64
---> Package lvm2-libs.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2-libs = 7:2.02.185-2.el7 for package: 7:lvm2-python-libs-2.02.185-2.el7.x86_64
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event = 7:1.02.164-7.el7_8.2 for package: 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be updated
--> Processing Dependency: device-mapper = 7:1.02.158-2.el7 for package: 7:device-mapper-libs-1.02.158-2.el7.x86_64
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event-libs = 7:1.02.164-7.el7_8.2 for package: 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64
---> Package lvm2-python-libs.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Running transaction check
---> Package device-mapper-event-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                              Arch                          Version                                        Repository                      Size
========================================================================================================================================================================
Updating:
 device-mapper-persistent-data                        x86_64                        0.8.5-2.el7                                    base                           422 k
 lvm2                                                 x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.3 M
 yum-utils                                            noarch                        1.1.31-54.el7_8                                updates                        122 k
Updating for dependencies:
 device-mapper                                        x86_64                        7:1.02.164-7.el7_8.2                           updates                        295 k
 device-mapper-event                                  x86_64                        7:1.02.164-7.el7_8.2                           updates                        191 k
 device-mapper-event-libs                             x86_64                        7:1.02.164-7.el7_8.2                           updates                        190 k
 device-mapper-libs                                   x86_64                        7:1.02.164-7.el7_8.2                           updates                        324 k
 lvm2-cluster                                         x86_64                        7:2.02.186-7.el7_8.2                           updates                        750 k
 lvm2-libs                                            x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.1 M
 lvm2-python-libs                                     x86_64                        7:2.02.186-7.el7_8.2                           updates                        188 k

Transaction Summary
========================================================================================================================================================================
Upgrade  3 Packages (+7 Dependent packages)

Total size: 4.8 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      1/20
  Updating   : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           2/20
  Updating   : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                                3/20
  Updating   : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Updating   : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               5/20
  Updating   : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    6/20
  Updating   : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    7/20
  Updating   : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Updating   : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        9/20
  Updating   : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                   10/20
  Cleanup    : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               11/20
  Cleanup    : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       12/20
  Cleanup    : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           13/20
  Cleanup    : yum-utils-1.1.31-52.el7.noarch                                                                                                                     14/20
  Cleanup    : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  15/20
  Cleanup    : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        16/20
  Cleanup    : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   17/20
  Cleanup    : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Cleanup    : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              19/20
  Cleanup    : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   20/20
  Verifying  : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           1/20
  Verifying  : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    2/20
  Verifying  : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               3/20
  Verifying  : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Verifying  : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        5/20
  Verifying  : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    6/20
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      7/20
  Verifying  : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Verifying  : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                    9/20
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                               10/20
  Verifying  : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        11/20
  Verifying  : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   12/20
  Verifying  : yum-utils-1.1.31-52.el7.noarch                                                                                                                     13/20
  Verifying  : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           14/20
  Verifying  : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   15/20
  Verifying  : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               16/20
  Verifying  : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              17/20
  Verifying  : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Verifying  : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  19/20
  Verifying  : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       20/20

Updated:
  device-mapper-persistent-data.x86_64 0:0.8.5-2.el7                 lvm2.x86_64 7:2.02.186-7.el7_8.2                 yum-utils.noarch 0:1.1.31-54.el7_8

Dependency Updated:
  device-mapper.x86_64 7:1.02.164-7.el7_8.2            device-mapper-event.x86_64 7:1.02.164-7.el7_8.2       device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2       lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2              lvm2-libs.x86_64 7:2.02.186-7.el7_8.2
  lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2

Complete!
[root@master-node ~]#

[root@master-node ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror, langpacks
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@master-node ~]#

Install Docker Community Edition
----------------------------------
[root@master-node ~]# yum install -y docker-ce
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: repos-tx.psychz.net
 * extras: repos-lax.psychz.net
 * updates: mirror.arizona.edu
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 3:19.03.8-3.el7 will be updated
---> Package docker-ce.x86_64 3:19.03.10-3.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                              Arch                              Version                                       Repository                                   Size
========================================================================================================================================================================
Updating:
 docker-ce                            x86_64                            3:19.03.10-3.el7                              docker-ce-stable                             24 M

Transaction Summary
========================================================================================================================================================================
Upgrade  1 Package

Total download size: 24 M
Downloading packages:
No Presto metadata available for docker-ce-stable
docker-ce-19.03.10-3.el7.x86_64.rpm                                                                                                              |  24 MB  00:00:03
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Cleanup    : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2
  Verifying  : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Verifying  : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2

Updated:
  docker-ce.x86_64 3:19.03.10-3.el7

Complete!
[root@master-node ~]#

[root@master-node ~]# systemctl status docker
? docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-31 16:35:11 CDT; 10s ago
     Docs: https://docs.docker.com
 Main PID: 9801 (dockerd)
    Tasks: 18
   Memory: 72.6M
   CGroup: /system.slice/docker.service
           +-9801 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

May 31 16:35:11 master-node dockerd[9801]: time="2020-05-31T16:35:11.255973127-05:00" level=error msg="fatal task error" error="No such container: nginx.2....kp54sahn5y
May 31 16:35:11 master-node dockerd[9801]: time="2020-05-31T16:35:11.523002017-05:00" level=info msg="Daemon has completed initialization"
May 31 16:35:11 master-node dockerd[9801]: time="2020-05-31T16:35:11.540338030-05:00" level=info msg="API listen on /var/run/docker.sock"
May 31 16:35:11 master-node systemd[1]: Started Docker Application Container Engine.
May 31 16:35:12 master-node dockerd[9801]: time="2020-05-31T16:35:12.438487241-05:00" level=warning msg="failed to deactivate service binding for container...j25q2kesay
Hint: Some lines were ellipsized, use -l to show in full.
[root@master-node ~]#

[root@centos7 ~]# systemctl restart docker && systemctl enable docker

[root@master-node ~]# docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.10
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       9424aeaee9
  Built:            Thu May 28 22:16:43 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
[root@master-node ~]#

4. Kubernetes packages are not available in the default CentOS 7 & RHEL 7 repositories, Use below command to configure its package repositories.
------------------------------------------------------------------------------------------------------------------------------------------------
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Output:
-------
[root@master-node ~]# cat < /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=1
> repo_gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
>         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> EOF
[root@master-node ~]#

5. Once the package repositories are configured, run the beneath command to install kubeadm and docker packages.
===============================================================================================================
[root@master-node ~]# yum install -y kubelet kubeadm kubectl –disableexcludes=kubernetes
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: repos-tx.psychz.net
 * extras: mirror.mobap.edu
 * updates: repos-va.psychz.net
kubernetes/signature                                                                                                                             |  454 B  00:00:00
kubernetes/signature                                                                                                                             | 1.4 kB  00:00:00 !!!
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.18.3-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.18.3-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                          Arch                             Version                                   Repository                            Size
========================================================================================================================================================================
Installing:
 kubeadm                                          x86_64                           1.18.3-0                                  kubernetes                           8.8 M
 kubectl                                          x86_64                           1.18.3-0                                  kubernetes                           9.5 M
 kubelet                                          x86_64                           1.18.3-0                                  kubernetes                            21 M
Installing for dependencies:
 conntrack-tools                                  x86_64                           1.4.4-7.el7                               base                                 187 k
 cri-tools                                        x86_64                           1.13.0-0                                  kubernetes                           5.1 M
 kubernetes-cni                                   x86_64                           0.7.5-0                                   kubernetes                            10 M
 libnetfilter_cthelper                            x86_64                           1.0.0-11.el7                              base                                  18 k
 libnetfilter_cttimeout                           x86_64                           1.0.0-7.el7                               base                                  18 k
 libnetfilter_queue                               x86_64                           1.0.2-2.el7_2                             base                                  23 k
 socat                                            x86_64                           1.7.3.2-2.el7                             base                                 290 k

Transaction Summary
========================================================================================================================================================================
Install  3 Packages (+7 Dependent packages)

Total download size: 55 M
Installed size: 246 M
Downloading packages:
(1/10): conntrack-tools-1.4.4-7.el7.x86_64.rpm                                                                                                   | 187 kB  00:00:00
warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed
(2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm                                           | 5.1 MB  00:00:01
(3/10): cd5d6980c3e1b15de222db08729eff40f7031b7fa56c71ae3e28e420ba9678cd-kubectl-1.18.3-0.x86_64.rpm                                             | 9.5 MB  00:00:01
(4/10): a23839a743e789babb0ce912fa440f6e6ceb15bc5db42dd91aa0838c994b3452-kubeadm-1.18.3-0.x86_64.rpm                                             | 8.8 MB  00:00:03
(5/10): libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(6/10): libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(7/10): socat-1.7.3.2-2.el7.x86_64.rpm                                                                                                           | 290 kB  00:00:00
(8/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm                                                                                              |  23 kB  00:00:00
(9/10): d1a0216cfab2fb28e82be531327ebde9a554bb6d33e3c8313acc9bc728ba59d1-kubelet-1.18.3-0.x86_64.rpm                                             |  21 MB  00:00:03
(10/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm                                      |  10 MB  00:00:04
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   6.7 MB/s |  55 MB  00:00:08
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key "
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key "
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                           1/10
  Installing : socat-1.7.3.2-2.el7.x86_64                                                                                                                          2/10
  Installing : cri-tools-1.13.0-0.x86_64                                                                                                                           3/10
  Installing : kubectl-1.18.3-0.x86_64                                                                                                                             4/10
  Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             5/10
  Installing : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           6/10
  Installing : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  7/10
  Installing : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       8/10
  Installing : kubelet-1.18.3-0.x86_64                                                                                                                             9/10
  Installing : kubeadm-1.18.3-0.x86_64                                                                                                                            10/10
  Verifying  : kubelet-1.18.3-0.x86_64                                                                                                                             1/10
  Verifying  : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           2/10
  Verifying  : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  3/10
  Verifying  : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             4/10
  Verifying  : kubeadm-1.18.3-0.x86_64                                                                                                                             5/10
  Verifying  : kubectl-1.18.3-0.x86_64                                                                                                                             6/10
  Verifying  : cri-tools-1.13.0-0.x86_64                                                                                                                           7/10
  Verifying  : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       8/10
  Verifying  : socat-1.7.3.2-2.el7.x86_64                                                                                                                          9/10
  Verifying  : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                          10/10

Installed:
  kubeadm.x86_64 0:1.18.3-0                              kubectl.x86_64 0:1.18.3-0                              kubelet.x86_64 0:1.18.3-0

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-7.el7         cri-tools.x86_64 0:1.13.0-0                kubernetes-cni.x86_64 0:0.7.5-0  libnetfilter_cthelper.x86_64 0:1.0.0-11.el7
  libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7  libnetfilter_queue.x86_64 0:1.0.2-2.el7_2  socat.x86_64 0:1.7.3.2-2.el7

Complete!
[root@master-node ~]#

6. Start and enable kubectl and docker service
----------------------------------------------
[root@master-node ~]# systemctl restart docker && systemctl enable docker

[root@master-node ~]# systemctl restart kubelet && systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

7. Initialize Kubernetes Master with ‘kubeadm init’ && Run the beneath command to  initialize and setup kubernetes master.
--------------------------------------------------------------------------------------------------------------------------

[root@master-node ~]# swapoff -a
[root@master-node ~]#

[root@centos7 ~]# kubeadm init --apiserver-advertise-address=192.168.2.130 --pod-network-cidr=10.244.0.0/16
W0603 19:28:46.895852    3584 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.3
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [centos7 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.130]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [centos7 localhost] and IPs [192.168.2.130 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [centos7 localhost] and IPs [192.168.2.130 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0603 19:30:06.779785    3584 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0603 19:30:06.781008    3584 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.502516 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node centos7 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node centos7 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: a8a9vc.dk8krmn838is5gy6
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.130:6443 --token a8a9vc.dk8krmn838is5gy6 \
    --discovery-token-ca-cert-hash sha256:10c8310cd5dd8b6847bc8fa0404974051ea268bbb68b00020690c8003139975a
[root@centos7 ~]#


[root@master-node ~]# mkdir -p $HOME/.kube
[root@master-node ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-node ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

[root@master-node ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
[root@master-node ~]#

[root@centos7 ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE     VERSION
master-node   NotReady   master   3m38s   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
master-node   Ready    master   3m42s   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-d6cbx              1/1     Running   0          3m36s
kube-system   coredns-66bff467f8-x76j7              1/1     Running   0          3m36s
kube-system   etcd-master-node                      1/1     Running   0          3m44s
kube-system   kube-apiserver-master-node            1/1     Running   0          3m44s
kube-system   kube-controller-manager-master-node   1/1     Running   0          3m44s
kube-system   kube-flannel-ds-amd64-bmxjc           1/1     Running   0          25s
kube-system   kube-proxy-8ptqp                      1/1     Running   0          3m35s
kube-system   kube-scheduler-master-node            1/1     Running   0          3m44s
[root@centos7 ~]#

=========================================>
Worker Node-1
--------------

[root@centos7-w1 ~]# setenforce 0
[root@centos7-w1 ~]# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
[root@centos7-w1 ~]# modprobe br_netfilter
[root@centos7-w1 ~]# echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

[root@centos7-w1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.131 centos7-w1.localdomain centos7-w1
192.168.2.130 centos7.localdomain centos7
[root@centos7-w1 ~]#

[root@centos7-w1 ~]# firewall-cmd --state
not running

[root@centos7-w1 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.atl.genesisadaptive.com
 * extras: centos.host-engine.com
 * updates: mirror.steadfastnet.com
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-1.el7 will be updated
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-2.el7 will be an update
---> Package lvm2.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2 = 7:2.02.185-2.el7 for package: 7:lvm2-cluster-2.02.185-2.el7.x86_64
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: lvm2-libs = 7:2.02.186-7.el7_8.2 for package: 7:lvm2-2.02.186-7.el7_8.2.x86_64
---> Package yum-utils.noarch 0:1.1.31-52.el7 will be updated
---> Package yum-utils.noarch 0:1.1.31-54.el7_8 will be an update
--> Running transaction check
---> Package lvm2-cluster.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper = 7:1.02.164 for package: 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64
---> Package lvm2-libs.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2-libs = 7:2.02.185-2.el7 for package: 7:lvm2-python-libs-2.02.185-2.el7.x86_64
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event = 7:1.02.164-7.el7_8.2 for package: 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be updated
--> Processing Dependency: device-mapper = 7:1.02.158-2.el7 for package: 7:device-mapper-libs-1.02.158-2.el7.x86_64
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event-libs = 7:1.02.164-7.el7_8.2 for package: 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64
---> Package lvm2-python-libs.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Running transaction check
---> Package device-mapper-event-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                              Arch                          Version                                        Repository                      Size
========================================================================================================================================================================
Updating:
 device-mapper-persistent-data                        x86_64                        0.8.5-2.el7                                    base                           422 k
 lvm2                                                 x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.3 M
 yum-utils                                            noarch                        1.1.31-54.el7_8                                updates                        122 k
Updating for dependencies:
 device-mapper                                        x86_64                        7:1.02.164-7.el7_8.2                           updates                        295 k
 device-mapper-event                                  x86_64                        7:1.02.164-7.el7_8.2                           updates                        191 k
 device-mapper-event-libs                             x86_64                        7:1.02.164-7.el7_8.2                           updates                        190 k
 device-mapper-libs                                   x86_64                        7:1.02.164-7.el7_8.2                           updates                        324 k
 lvm2-cluster                                         x86_64                        7:2.02.186-7.el7_8.2                           updates                        750 k
 lvm2-libs                                            x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.1 M
 lvm2-python-libs                                     x86_64                        7:2.02.186-7.el7_8.2                           updates                        188 k

Transaction Summary
========================================================================================================================================================================
Upgrade  3 Packages (+7 Dependent packages)

Total size: 4.8 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      1/20
  Updating   : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           2/20
  Updating   : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                                3/20
  Updating   : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Updating   : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               5/20
  Updating   : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    6/20
  Updating   : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    7/20
  Updating   : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Updating   : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        9/20
  Updating   : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                   10/20
  Cleanup    : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               11/20
  Cleanup    : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       12/20
  Cleanup    : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           13/20
  Cleanup    : yum-utils-1.1.31-52.el7.noarch                                                                                                                     14/20
  Cleanup    : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  15/20
  Cleanup    : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        16/20
  Cleanup    : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   17/20
  Cleanup    : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Cleanup    : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              19/20
  Cleanup    : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   20/20
  Verifying  : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           1/20
  Verifying  : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    2/20
  Verifying  : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               3/20
  Verifying  : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Verifying  : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        5/20
  Verifying  : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    6/20
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      7/20
  Verifying  : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Verifying  : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                    9/20
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                               10/20
  Verifying  : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        11/20
  Verifying  : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   12/20
  Verifying  : yum-utils-1.1.31-52.el7.noarch                                                                                                                     13/20
  Verifying  : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           14/20
  Verifying  : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   15/20
  Verifying  : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               16/20
  Verifying  : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              17/20
  Verifying  : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Verifying  : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  19/20
  Verifying  : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       20/20

Updated:
  device-mapper-persistent-data.x86_64 0:0.8.5-2.el7                 lvm2.x86_64 7:2.02.186-7.el7_8.2                 yum-utils.noarch 0:1.1.31-54.el7_8

Dependency Updated:
  device-mapper.x86_64 7:1.02.164-7.el7_8.2            device-mapper-event.x86_64 7:1.02.164-7.el7_8.2       device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2       lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2              lvm2-libs.x86_64 7:2.02.186-7.el7_8.2
  lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2

Complete!
[root@centos7-w1 ~]#
[root@centos7-w1 ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror, langpacks
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo

[root@centos7-w1 ~]# yum install -y docker-ce
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.atl.genesisadaptive.com
 * extras: centos.host-engine.com
 * updates: mirror.steadfastnet.com
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 3:19.03.8-3.el7 will be updated
---> Package docker-ce.x86_64 3:19.03.10-3.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                              Arch                              Version                                       Repository                                   Size
========================================================================================================================================================================
Updating:
 docker-ce                            x86_64                            3:19.03.10-3.el7                              docker-ce-stable                             24 M

Transaction Summary
========================================================================================================================================================================
Upgrade  1 Package

Total download size: 24 M
Downloading packages:
No Presto metadata available for docker-ce-stable
docker-ce-19.03.10-3.el7.x86_64.rpm                                                                                                              |  24 MB  00:00:04
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Cleanup    : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2
  Verifying  : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Verifying  : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2

Updated:
  docker-ce.x86_64 3:19.03.10-3.el7

Complete!
[root@centos7-w1 ~]#

[root@centos7-w1 ~]# cat < /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=1
> repo_gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
>         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> EOF
[root@centos7-w1 ~]#

[root@centos7-w1 ~]# yum  install kubeadm docker -y --skip-broken
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.atl.genesisadaptive.com
 * extras: centos.host-engine.com
 * updates: mirrors.tummy.com
kubernetes/signature                                                                                                                             |  454 B  00:00:00
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key "
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
kubernetes/signature                                                                                                                             | 1.4 kB  00:00:00 !!!
kubernetes/primary                                                                                                                               |  69 kB  00:00:00
kubernetes                                                                                                                                                      505/505
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
--> Processing Dependency: docker-common = 2:1.13.1-161.git64e9980.el7_8 for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: docker-client = 2:1.13.1-161.git64e9980.el7_8 for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: subscription-manager-rhsm-certificates for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: kubelet >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Running transaction check
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package docker-client.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
---> Package docker-common.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
--> Processing Dependency: skopeo-containers >= 1:0.1.26-2 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-umount >= 2:2.3.3-3 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-systemd-hook >= 1:0.1.4-9 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-register-machine >= 1:0-5.13 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: container-storage-setup >= 0.9.0-1 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: atomic-registries for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.18.3-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.18.3-0.x86_64
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7.centos will be installed
--> Running transaction check
---> Package atomic-registries.x86_64 1:1.22.1-33.gitb507039.el7_8 will be installed
--> Processing Dependency: python-pytoml for package: 1:atomic-registries-1.22.1-33.gitb507039.el7_8.x86_64
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package container-storage-setup.noarch 0:0.11.0-2.git5eaf76c.el7 will be installed
---> Package containers-common.x86_64 1:0.1.40-7.el7_8 will be installed
--> Processing Dependency: subscription-manager for package: 1:containers-common-0.1.40-7.el7_8.x86_64
--> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-7.el7_8.x86_64
--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-7.el7_8.x86_64
---> Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be installed
---> Package oci-systemd-hook.x86_64 1:0.2.0-1.git05e6923.el7_6 will be installed
---> Package oci-umount.x86_64 2:2.5-3.el7 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package python-pytoml.noarch 0:0.1.14-1.git7dea353.el7 will be installed
---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
---> Package subscription-manager.x86_64 0:1.24.26-3.el7.centos will be installed
--> Processing Dependency: subscription-manager-rhsm = 1.24.26 for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Processing Dependency: python-dmidecode >= 3.12.2-2 for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Processing Dependency: python-syspurpose for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Running transaction check
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
---> Package python-dmidecode.x86_64 0:3.12.2-4.el7 will be installed
---> Package python-syspurpose.x86_64 0:1.24.26-3.el7.centos will be installed
---> Package subscription-manager-rhsm.x86_64 0:1.24.26-3.el7.centos will be installed
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker-io
--> Processing Conflict: 1:docker-ce-cli-19.03.8-3.el7.x86_64 conflicts docker
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package docker-ce-cli.x86_64 1:19.03.8-3.el7 will be updated
---> Package docker-ce-cli.x86_64 1:19.03.10-3.el7 will be an update
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker-io
--> Processing Conflict: 1:docker-ce-cli-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 1:docker-ce-cli-19.03.10-3.el7.x86_64 conflicts docker-io
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Finished Dependency Resolution

Packages skipped because of dependency problems:
    1:atomic-registries-1.22.1-33.gitb507039.el7_8.x86_64 from extras
    container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch from extras
    1:containers-common-0.1.40-7.el7_8.x86_64 from extras
    2:docker-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    1:docker-ce-cli-19.03.10-3.el7.x86_64 from docker-ce-stable
    2:docker-client-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    fuse-overlayfs-0.7.2-6.el7_8.x86_64 from extras
    fuse3-libs-3.6.1-4.el7.x86_64 from extras
    1:oci-register-machine-0-6.git2b44233.el7.x86_64 from extras
    1:oci-systemd-hook-0.2.0-1.git05e6923.el7_6.x86_64 from extras
    2:oci-umount-2.5-3.el7.x86_64 from extras
    python-dmidecode-3.12.2-4.el7.x86_64 from base
    python-pytoml-0.1.14-1.git7dea353.el7.noarch from extras
    python-syspurpose-1.24.26-3.el7.centos.x86_64 from updates
    slirp4netns-0.4.3-4.el7_8.x86_64 from extras
    subscription-manager-1.24.26-3.el7.centos.x86_64 from updates
    subscription-manager-rhsm-1.24.26-3.el7.centos.x86_64 from updates
    subscription-manager-rhsm-certificates-1.24.26-3.el7.centos.x86_64 from updates

Dependencies Resolved

========================================================================================================================================================================
 Package                                                 Arch                    Version                                        Repository                         Size
========================================================================================================================================================================
Installing:
 kubeadm                                                 x86_64                  1.18.3-0                                       kubernetes                        8.8 M
Installing for dependencies:
 conntrack-tools                                         x86_64                  1.4.4-7.el7                                    base                              187 k
 cri-tools                                               x86_64                  1.13.0-0                                       kubernetes                        5.1 M
 kubectl                                                 x86_64                  1.18.3-0                                       kubernetes                        9.5 M
 kubelet                                                 x86_64                  1.18.3-0                                       kubernetes                         21 M
 kubernetes-cni                                          x86_64                  0.7.5-0                                        kubernetes                         10 M
 libnetfilter_cthelper                                   x86_64                  1.0.0-11.el7                                   base                               18 k
 libnetfilter_cttimeout                                  x86_64                  1.0.0-7.el7                                    base                               18 k
 libnetfilter_queue                                      x86_64                  1.0.2-2.el7_2                                  base                               23 k
 socat                                                   x86_64                  1.7.3.2-2.el7                                  base                              290 k
Skipped (dependency problems):
 atomic-registries                                       x86_64                  1:1.22.1-33.gitb507039.el7_8                   extras                             36 k
 container-storage-setup                                 noarch                  0.11.0-2.git5eaf76c.el7                        extras                             35 k
 containers-common                                       x86_64                  1:0.1.40-7.el7_8                               extras                             42 k
 docker                                                  x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                             18 M
 docker-ce-cli                                           x86_64                  1:19.03.10-3.el7                               docker-ce-stable                   38 M
 docker-client                                           x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                            3.9 M
 docker-common                                           x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                             99 k
 fuse-overlayfs                                          x86_64                  0.7.2-6.el7_8                                  extras                             54 k
 fuse3-libs                                              x86_64                  3.6.1-4.el7                                    extras                             82 k
 oci-register-machine                                    x86_64                  1:0-6.git2b44233.el7                           extras                            1.1 M
 oci-systemd-hook                                        x86_64                  1:0.2.0-1.git05e6923.el7_6                     extras                             34 k
 oci-umount                                              x86_64                  2:2.5-3.el7                                    extras                             33 k
 python-dmidecode                                        x86_64                  3.12.2-4.el7                                   base                               83 k
 python-pytoml                                           noarch                  0.1.14-1.git7dea353.el7                        extras                             18 k
 python-syspurpose                                       x86_64                  1.24.26-3.el7.centos                           updates                           269 k
 slirp4netns                                             x86_64                  0.4.3-4.el7_8                                  extras                             81 k
 subscription-manager                                    x86_64                  1.24.26-3.el7.centos                           updates                           1.1 M
 subscription-manager-rhsm                               x86_64                  1.24.26-3.el7.centos                           updates                           327 k
 subscription-manager-rhsm-certificates                  x86_64                  1.24.26-3.el7.centos                           updates                           232 k

Transaction Summary
========================================================================================================================================================================
Install                         1 Package  (+9 Dependent packages)
Skipped (dependency problems)  19 Packages

Total download size: 55 M
Installed size: 246 M
Downloading packages:
(1/10): conntrack-tools-1.4.4-7.el7.x86_64.rpm                                                                                                   | 187 kB  00:00:00
warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed
(2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm                                           | 5.1 MB  00:00:01
(3/10): cd5d6980c3e1b15de222db08729eff40f7031b7fa56c71ae3e28e420ba9678cd-kubectl-1.18.3-0.x86_64.rpm                                             | 9.5 MB  00:00:02
(4/10): a23839a743e789babb0ce912fa440f6e6ceb15bc5db42dd91aa0838c994b3452-kubeadm-1.18.3-0.x86_64.rpm                                             | 8.8 MB  00:00:06
(5/10): libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm                                                                                              |  23 kB  00:00:00
(7/10): libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(8/10): socat-1.7.3.2-2.el7.x86_64.rpm                                                                                                           | 290 kB  00:00:00
(9/10): d1a0216cfab2fb28e82be531327ebde9a554bb6d33e3c8313acc9bc728ba59d1-kubelet-1.18.3-0.x86_64.rpm                                             |  21 MB  00:00:05
(10/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm                                      |  10 MB  00:00:06
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   4.3 MB/s |  55 MB  00:00:12
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key "
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key "
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : socat-1.7.3.2-2.el7.x86_64                                                                                                                          1/10
  Installing : cri-tools-1.13.0-0.x86_64                                                                                                                           2/10
  Installing : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           3/10
  Installing : kubectl-1.18.3-0.x86_64                                                                                                                             4/10
  Installing : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                           5/10
  Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             6/10
  Installing : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  7/10
  Installing : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       8/10
  Installing : kubelet-1.18.3-0.x86_64                                                                                                                             9/10
  Installing : kubeadm-1.18.3-0.x86_64                                                                                                                            10/10
  Verifying  : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             1/10
  Verifying  : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  2/10
  Verifying  : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                           3/10
  Verifying  : kubeadm-1.18.3-0.x86_64                                                                                                                             4/10
  Verifying  : kubectl-1.18.3-0.x86_64                                                                                                                             5/10
  Verifying  : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           6/10
  Verifying  : kubelet-1.18.3-0.x86_64                                                                                                                             7/10
  Verifying  : cri-tools-1.13.0-0.x86_64                                                                                                                           8/10
  Verifying  : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       9/10
  Verifying  : socat-1.7.3.2-2.el7.x86_64                                                                                                                         10/10

Installed:
  kubeadm.x86_64 0:1.18.3-0

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-7.el7                    cri-tools.x86_64 0:1.13.0-0                           kubectl.x86_64 0:1.18.3-0
  kubelet.x86_64 0:1.18.3-0                               kubernetes-cni.x86_64 0:0.7.5-0                       libnetfilter_cthelper.x86_64 0:1.0.0-11.el7
  libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7             libnetfilter_queue.x86_64 0:1.0.2-2.el7_2             socat.x86_64 0:1.7.3.2-2.el7

Skipped (dependency problems):
  atomic-registries.x86_64 1:1.22.1-33.gitb507039.el7_8                                    container-storage-setup.noarch 0:0.11.0-2.git5eaf76c.el7
  containers-common.x86_64 1:0.1.40-7.el7_8                                                docker.x86_64 2:1.13.1-161.git64e9980.el7_8
  docker-ce-cli.x86_64 1:19.03.10-3.el7                                                    docker-client.x86_64 2:1.13.1-161.git64e9980.el7_8
  docker-common.x86_64 2:1.13.1-161.git64e9980.el7_8                                       fuse-overlayfs.x86_64 0:0.7.2-6.el7_8
  fuse3-libs.x86_64 0:3.6.1-4.el7                                                          oci-register-machine.x86_64 1:0-6.git2b44233.el7
  oci-systemd-hook.x86_64 1:0.2.0-1.git05e6923.el7_6                                       oci-umount.x86_64 2:2.5-3.el7
  python-dmidecode.x86_64 0:3.12.2-4.el7                                                   python-pytoml.noarch 0:0.1.14-1.git7dea353.el7
  python-syspurpose.x86_64 0:1.24.26-3.el7.centos                                          slirp4netns.x86_64 0:0.4.3-4.el7_8
  subscription-manager.x86_64 0:1.24.26-3.el7.centos                                       subscription-manager-rhsm.x86_64 0:1.24.26-3.el7.centos
  subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7.centos

Complete!
[root@centos7-w1 ~]#

[root@centos7-w1 ~]# swapoff -a

[root@centos7-w1 ~]# systemctl restart docker && systemctl enable docker

[root@centos7-w1 ~]# systemctl restart kubelet && systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@centos7-w1 ~]#

[root@centos7-w1 ~]# kubeadm join 192.168.2.130:6443 --token a8a9vc.dk8krmn838is5gy6 \
    --discovery-token-ca-cert-hash sha256:10c8310cd5dd8b6847bc8fa0404974051ea268bbb68b00020690c8003139975a
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@centos7-w1 ~]#

Testing from Master Node
========================
[root@centos7 ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
centos7-w1    NotReady      40s   v1.18.3
master-node   Ready      master   18m   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE    VERSION
centos7-w1    Ready       100s   v1.18.3
master-node   Ready    master   19m    v1.18.3
[root@centos7 ~]#

========================================================================>
Worker Node-2
---------------

[root@centos7-w2 ~]# setenforce 0
[root@centos7-w2 ~]# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
[root@centos7-w2 ~]# modprobe br_netfilter
[root@centos7-w2 ~]# echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

[root@centos7-w2 ~]# firewall-cmd --state
not running
[root@centos7-w2 ~]#

[root@centos7-w2 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.dal.nexril.net
 * extras: mirror.hostduplex.com
 * updates: mirror.dal.nexril.net
base                                                                                                                                             | 3.6 kB  00:00:00
docker-ce-stable                                                                                                                                 | 3.5 kB  00:00:00
extras                                                                                                                                           | 2.9 kB  00:00:00
updates                                                                                                                                          | 2.9 kB  00:00:00
docker-ce-stable/x86_64/primary_db                                                                                                               |  43 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-1.el7 will be updated
---> Package device-mapper-persistent-data.x86_64 0:0.8.5-2.el7 will be an update
---> Package lvm2.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2 = 7:2.02.185-2.el7 for package: 7:lvm2-cluster-2.02.185-2.el7.x86_64
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: lvm2-libs = 7:2.02.186-7.el7_8.2 for package: 7:lvm2-2.02.186-7.el7_8.2.x86_64
---> Package yum-utils.noarch 0:1.1.31-52.el7 will be updated
---> Package yum-utils.noarch 0:1.1.31-54.el7_8 will be an update
--> Running transaction check
---> Package lvm2-cluster.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper = 7:1.02.164 for package: 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64
---> Package lvm2-libs.x86_64 7:2.02.185-2.el7 will be updated
--> Processing Dependency: lvm2-libs = 7:2.02.185-2.el7 for package: 7:lvm2-python-libs-2.02.185-2.el7.x86_64
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event = 7:1.02.164-7.el7_8.2 for package: 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be updated
--> Processing Dependency: device-mapper = 7:1.02.158-2.el7 for package: 7:device-mapper-libs-1.02.158-2.el7.x86_64
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Processing Dependency: device-mapper-event-libs = 7:1.02.164-7.el7_8.2 for package: 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64
---> Package lvm2-python-libs.x86_64 7:2.02.185-2.el7 will be updated
---> Package lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
--> Running transaction check
---> Package device-mapper-event-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                              Arch                          Version                                        Repository                      Size
========================================================================================================================================================================
Updating:
 device-mapper-persistent-data                        x86_64                        0.8.5-2.el7                                    base                           422 k
 lvm2                                                 x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.3 M
 yum-utils                                            noarch                        1.1.31-54.el7_8                                updates                        122 k
Updating for dependencies:
 device-mapper                                        x86_64                        7:1.02.164-7.el7_8.2                           updates                        295 k
 device-mapper-event                                  x86_64                        7:1.02.164-7.el7_8.2                           updates                        191 k
 device-mapper-event-libs                             x86_64                        7:1.02.164-7.el7_8.2                           updates                        190 k
 device-mapper-libs                                   x86_64                        7:1.02.164-7.el7_8.2                           updates                        324 k
 lvm2-cluster                                         x86_64                        7:2.02.186-7.el7_8.2                           updates                        750 k
 lvm2-libs                                            x86_64                        7:2.02.186-7.el7_8.2                           updates                        1.1 M
 lvm2-python-libs                                     x86_64                        7:2.02.186-7.el7_8.2                           updates                        188 k

Transaction Summary
========================================================================================================================================================================
Upgrade  3 Packages (+7 Dependent packages)

Total size: 4.8 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      1/20
  Updating   : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           2/20
  Updating   : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                                3/20
  Updating   : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Updating   : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               5/20
  Updating   : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    6/20
  Updating   : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    7/20
  Updating   : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Updating   : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        9/20
  Updating   : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                   10/20
  Cleanup    : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               11/20
  Cleanup    : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       12/20
  Cleanup    : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           13/20
  Cleanup    : yum-utils-1.1.31-52.el7.noarch                                                                                                                     14/20
  Cleanup    : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  15/20
  Cleanup    : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        16/20
  Cleanup    : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   17/20
  Cleanup    : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Cleanup    : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              19/20
  Cleanup    : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   20/20
  Verifying  : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                                                                           1/20
  Verifying  : device-mapper-persistent-data-0.8.5-2.el7.x86_64                                                                                                    2/20
  Verifying  : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                                                               3/20
  Verifying  : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                                                                     4/20
  Verifying  : 7:lvm2-python-libs-2.02.186-7.el7_8.2.x86_64                                                                                                        5/20
  Verifying  : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                                                                    6/20
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                                                                      7/20
  Verifying  : 7:lvm2-cluster-2.02.186-7.el7_8.2.x86_64                                                                                                            8/20
  Verifying  : yum-utils-1.1.31-54.el7_8.noarch                                                                                                                    9/20
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                                                               10/20
  Verifying  : 7:device-mapper-event-1.02.158-2.el7.x86_64                                                                                                        11/20
  Verifying  : device-mapper-persistent-data-0.8.5-1.el7.x86_64                                                                                                   12/20
  Verifying  : yum-utils-1.1.31-52.el7.noarch                                                                                                                     13/20
  Verifying  : 7:lvm2-python-libs-2.02.185-2.el7.x86_64                                                                                                           14/20
  Verifying  : 7:device-mapper-event-libs-1.02.158-2.el7.x86_64                                                                                                   15/20
  Verifying  : 7:lvm2-cluster-2.02.185-2.el7.x86_64                                                                                                               16/20
  Verifying  : 7:device-mapper-1.02.158-2.el7.x86_64                                                                                                              17/20
  Verifying  : 7:device-mapper-libs-1.02.158-2.el7.x86_64                                                                                                         18/20
  Verifying  : 7:lvm2-libs-2.02.185-2.el7.x86_64                                                                                                                  19/20
  Verifying  : 7:lvm2-2.02.185-2.el7.x86_64                                                                                                                       20/20

Updated:
  device-mapper-persistent-data.x86_64 0:0.8.5-2.el7                 lvm2.x86_64 7:2.02.186-7.el7_8.2                 yum-utils.noarch 0:1.1.31-54.el7_8

Dependency Updated:
  device-mapper.x86_64 7:1.02.164-7.el7_8.2            device-mapper-event.x86_64 7:1.02.164-7.el7_8.2       device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2       lvm2-cluster.x86_64 7:2.02.186-7.el7_8.2              lvm2-libs.x86_64 7:2.02.186-7.el7_8.2
  lvm2-python-libs.x86_64 7:2.02.186-7.el7_8.2

Complete!
[root@centos7-w2 ~]#

[root@centos7-w2 ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror, langpacks
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@centos7-w2 ~]#

[root@centos7-w2 ~]# yum install -y docker-ce
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.dal.nexril.net
 * extras: mirror.hostduplex.com
 * updates: mirror.dal.nexril.net
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 3:19.03.8-3.el7 will be updated
---> Package docker-ce.x86_64 3:19.03.10-3.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                              Arch                              Version                                       Repository                                   Size
========================================================================================================================================================================
Updating:
 docker-ce                            x86_64                            3:19.03.10-3.el7                              docker-ce-stable                             24 M

Transaction Summary
========================================================================================================================================================================
Upgrade  1 Package

Total download size: 24 M
Downloading packages:
No Presto metadata available for docker-ce-stable
docker-ce-19.03.10-3.el7.x86_64.rpm                                                                                                              |  24 MB  00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Cleanup    : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2
  Verifying  : 3:docker-ce-19.03.10-3.el7.x86_64                                                                                                                    1/2
  Verifying  : 3:docker-ce-19.03.8-3.el7.x86_64                                                                                                                     2/2

Updated:
  docker-ce.x86_64 3:19.03.10-3.el7

Complete!
[root@centos7-w2 ~]# 

cat < /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=1
> repo_gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
>         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> EOF
[root@centos7-w2 ~]#

[root@centos7-w2 ~]# yum  install kubeadm docker -y --skip-broken
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.dal.nexril.net
 * extras: mirror.steadfastnet.com
 * updates: mirror.dal.nexril.net
kubernetes/signature                                                                                                                             |  454 B  00:00:00
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key "
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
kubernetes/signature                                                                                                                             | 1.4 kB  00:00:00 !!!
kubernetes/primary                                                                                                                               |  69 kB  00:00:00
kubernetes                                                                                                                                                      505/505
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
--> Processing Dependency: docker-common = 2:1.13.1-161.git64e9980.el7_8 for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: docker-client = 2:1.13.1-161.git64e9980.el7_8 for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: subscription-manager-rhsm-certificates for package: 2:docker-1.13.1-161.git64e9980.el7_8.x86_64
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: kubelet >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.18.3-0.x86_64
--> Running transaction check
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package docker-client.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
---> Package docker-common.x86_64 2:1.13.1-161.git64e9980.el7_8 will be installed
--> Processing Dependency: skopeo-containers >= 1:0.1.26-2 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-umount >= 2:2.3.3-3 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-systemd-hook >= 1:0.1.4-9 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: oci-register-machine >= 1:0-5.13 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: container-storage-setup >= 0.9.0-1 for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
--> Processing Dependency: atomic-registries for package: 2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.18.3-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.18.3-0.x86_64
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7.centos will be installed
--> Running transaction check
---> Package atomic-registries.x86_64 1:1.22.1-33.gitb507039.el7_8 will be installed
--> Processing Dependency: python-pytoml for package: 1:atomic-registries-1.22.1-33.gitb507039.el7_8.x86_64
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package container-storage-setup.noarch 0:0.11.0-2.git5eaf76c.el7 will be installed
---> Package containers-common.x86_64 1:0.1.40-7.el7_8 will be installed
--> Processing Dependency: subscription-manager for package: 1:containers-common-0.1.40-7.el7_8.x86_64
--> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-7.el7_8.x86_64
--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-7.el7_8.x86_64
---> Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be installed
---> Package oci-systemd-hook.x86_64 1:0.2.0-1.git05e6923.el7_6 will be installed
---> Package oci-umount.x86_64 2:2.5-3.el7 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package python-pytoml.noarch 0:0.1.14-1.git7dea353.el7 will be installed
---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
---> Package subscription-manager.x86_64 0:1.24.26-3.el7.centos will be installed
--> Processing Dependency: subscription-manager-rhsm = 1.24.26 for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Processing Dependency: python-dmidecode >= 3.12.2-2 for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Processing Dependency: python-syspurpose for package: subscription-manager-1.24.26-3.el7.centos.x86_64
--> Running transaction check
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
---> Package python-dmidecode.x86_64 0:3.12.2-4.el7 will be installed
---> Package python-syspurpose.x86_64 0:1.24.26-3.el7.centos will be installed
---> Package subscription-manager-rhsm.x86_64 0:1.24.26-3.el7.centos will be installed
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker-io
--> Processing Conflict: 1:docker-ce-cli-19.03.8-3.el7.x86_64 conflicts docker
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package docker-ce-cli.x86_64 1:19.03.8-3.el7 will be updated
---> Package docker-ce-cli.x86_64 1:19.03.10-3.el7 will be an update
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 3:docker-ce-19.03.10-3.el7.x86_64 conflicts docker-io
--> Processing Conflict: 1:docker-ce-cli-19.03.10-3.el7.x86_64 conflicts docker
--> Processing Conflict: 1:docker-ce-cli-19.03.10-3.el7.x86_64 conflicts docker-io
docker-ce-stable/x86_64/filelists_db                                                                                                             |  20 kB  00:00:00
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubeadm.x86_64 0:1.18.3-0 will be installed
---> Package kubectl.x86_64 0:1.18.3-0 will be installed
---> Package kubelet.x86_64 0:1.18.3-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Finished Dependency Resolution

Packages skipped because of dependency problems:
    1:atomic-registries-1.22.1-33.gitb507039.el7_8.x86_64 from extras
    container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch from extras
    1:containers-common-0.1.40-7.el7_8.x86_64 from extras
    2:docker-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    1:docker-ce-cli-19.03.10-3.el7.x86_64 from docker-ce-stable
    2:docker-client-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    2:docker-common-1.13.1-161.git64e9980.el7_8.x86_64 from extras
    fuse-overlayfs-0.7.2-6.el7_8.x86_64 from extras
    fuse3-libs-3.6.1-4.el7.x86_64 from extras
    1:oci-register-machine-0-6.git2b44233.el7.x86_64 from extras
    1:oci-systemd-hook-0.2.0-1.git05e6923.el7_6.x86_64 from extras
    2:oci-umount-2.5-3.el7.x86_64 from extras
    python-dmidecode-3.12.2-4.el7.x86_64 from base
    python-pytoml-0.1.14-1.git7dea353.el7.noarch from extras
    python-syspurpose-1.24.26-3.el7.centos.x86_64 from updates
    slirp4netns-0.4.3-4.el7_8.x86_64 from extras
    subscription-manager-1.24.26-3.el7.centos.x86_64 from updates
    subscription-manager-rhsm-1.24.26-3.el7.centos.x86_64 from updates
    subscription-manager-rhsm-certificates-1.24.26-3.el7.centos.x86_64 from updates

Dependencies Resolved

========================================================================================================================================================================
 Package                                                 Arch                    Version                                        Repository                         Size
========================================================================================================================================================================
Installing:
 kubeadm                                                 x86_64                  1.18.3-0                                       kubernetes                        8.8 M
Installing for dependencies:
 conntrack-tools                                         x86_64                  1.4.4-7.el7                                    base                              187 k
 cri-tools                                               x86_64                  1.13.0-0                                       kubernetes                        5.1 M
 kubectl                                                 x86_64                  1.18.3-0                                       kubernetes                        9.5 M
 kubelet                                                 x86_64                  1.18.3-0                                       kubernetes                         21 M
 kubernetes-cni                                          x86_64                  0.7.5-0                                        kubernetes                         10 M
 libnetfilter_cthelper                                   x86_64                  1.0.0-11.el7                                   base                               18 k
 libnetfilter_cttimeout                                  x86_64                  1.0.0-7.el7                                    base                               18 k
 libnetfilter_queue                                      x86_64                  1.0.2-2.el7_2                                  base                               23 k
 socat                                                   x86_64                  1.7.3.2-2.el7                                  base                              290 k
Skipped (dependency problems):
 atomic-registries                                       x86_64                  1:1.22.1-33.gitb507039.el7_8                   extras                             36 k
 container-storage-setup                                 noarch                  0.11.0-2.git5eaf76c.el7                        extras                             35 k
 containers-common                                       x86_64                  1:0.1.40-7.el7_8                               extras                             42 k
 docker                                                  x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                             18 M
 docker-ce-cli                                           x86_64                  1:19.03.10-3.el7                               docker-ce-stable                   38 M
 docker-client                                           x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                            3.9 M
 docker-common                                           x86_64                  2:1.13.1-161.git64e9980.el7_8                  extras                             99 k
 fuse-overlayfs                                          x86_64                  0.7.2-6.el7_8                                  extras                             54 k
 fuse3-libs                                              x86_64                  3.6.1-4.el7                                    extras                             82 k
 oci-register-machine                                    x86_64                  1:0-6.git2b44233.el7                           extras                            1.1 M
 oci-systemd-hook                                        x86_64                  1:0.2.0-1.git05e6923.el7_6                     extras                             34 k
 oci-umount                                              x86_64                  2:2.5-3.el7                                    extras                             33 k
 python-dmidecode                                        x86_64                  3.12.2-4.el7                                   base                               83 k
 python-pytoml                                           noarch                  0.1.14-1.git7dea353.el7                        extras                             18 k
 python-syspurpose                                       x86_64                  1.24.26-3.el7.centos                           updates                           269 k
 slirp4netns                                             x86_64                  0.4.3-4.el7_8                                  extras                             81 k
 subscription-manager                                    x86_64                  1.24.26-3.el7.centos                           updates                           1.1 M
 subscription-manager-rhsm                               x86_64                  1.24.26-3.el7.centos                           updates                           327 k
 subscription-manager-rhsm-certificates                  x86_64                  1.24.26-3.el7.centos                           updates                           232 k

Transaction Summary
========================================================================================================================================================================
Install                         1 Package  (+9 Dependent packages)
Skipped (dependency problems)  19 Packages

Total download size: 55 M
Installed size: 246 M
Downloading packages:
(1/10): conntrack-tools-1.4.4-7.el7.x86_64.rpm                                                                                                   | 187 kB  00:00:00
warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed
(2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm                                           | 5.1 MB  00:00:01
(3/10): a23839a743e789babb0ce912fa440f6e6ceb15bc5db42dd91aa0838c994b3452-kubeadm-1.18.3-0.x86_64.rpm                                             | 8.8 MB  00:00:02
(4/10): cd5d6980c3e1b15de222db08729eff40f7031b7fa56c71ae3e28e420ba9678cd-kubectl-1.18.3-0.x86_64.rpm                                             | 9.5 MB  00:00:02
(5/10): libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm                                                                                              |  23 kB  00:00:00
(7/10): socat-1.7.3.2-2.el7.x86_64.rpm                                                                                                           | 290 kB  00:00:00
(8/10): libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm                                                                                            |  18 kB  00:00:00
(9/10): d1a0216cfab2fb28e82be531327ebde9a554bb6d33e3c8313acc9bc728ba59d1-kubelet-1.18.3-0.x86_64.rpm                                             |  21 MB  00:00:04
(10/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm                                      |  10 MB  00:00:02
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   8.2 MB/s |  55 MB  00:00:06
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key "
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key "
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : socat-1.7.3.2-2.el7.x86_64                                                                                                                          1/10
  Installing : cri-tools-1.13.0-0.x86_64                                                                                                                           2/10
  Installing : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           3/10
  Installing : kubectl-1.18.3-0.x86_64                                                                                                                             4/10
  Installing : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                           5/10
  Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             6/10
  Installing : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  7/10
  Installing : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       8/10
  Installing : kubelet-1.18.3-0.x86_64                                                                                                                             9/10
  Installing : kubeadm-1.18.3-0.x86_64                                                                                                                            10/10
  Verifying  : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                             1/10
  Verifying  : conntrack-tools-1.4.4-7.el7.x86_64                                                                                                                  2/10
  Verifying  : libnetfilter_cttimeout-1.0.0-7.el7.x86_64                                                                                                           3/10
  Verifying  : kubeadm-1.18.3-0.x86_64                                                                                                                             4/10
  Verifying  : kubectl-1.18.3-0.x86_64                                                                                                                             5/10
  Verifying  : libnetfilter_cthelper-1.0.0-11.el7.x86_64                                                                                                           6/10
  Verifying  : kubelet-1.18.3-0.x86_64                                                                                                                             7/10
  Verifying  : cri-tools-1.13.0-0.x86_64                                                                                                                           8/10
  Verifying  : kubernetes-cni-0.7.5-0.x86_64                                                                                                                       9/10
  Verifying  : socat-1.7.3.2-2.el7.x86_64                                                                                                                         10/10

Installed:
  kubeadm.x86_64 0:1.18.3-0

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-7.el7                    cri-tools.x86_64 0:1.13.0-0                           kubectl.x86_64 0:1.18.3-0
  kubelet.x86_64 0:1.18.3-0                               kubernetes-cni.x86_64 0:0.7.5-0                       libnetfilter_cthelper.x86_64 0:1.0.0-11.el7
  libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7             libnetfilter_queue.x86_64 0:1.0.2-2.el7_2             socat.x86_64 0:1.7.3.2-2.el7

Skipped (dependency problems):
  atomic-registries.x86_64 1:1.22.1-33.gitb507039.el7_8                                    container-storage-setup.noarch 0:0.11.0-2.git5eaf76c.el7
  containers-common.x86_64 1:0.1.40-7.el7_8                                                docker.x86_64 2:1.13.1-161.git64e9980.el7_8
  docker-ce-cli.x86_64 1:19.03.10-3.el7                                                    docker-client.x86_64 2:1.13.1-161.git64e9980.el7_8
  docker-common.x86_64 2:1.13.1-161.git64e9980.el7_8                                       fuse-overlayfs.x86_64 0:0.7.2-6.el7_8
  fuse3-libs.x86_64 0:3.6.1-4.el7                                                          oci-register-machine.x86_64 1:0-6.git2b44233.el7
  oci-systemd-hook.x86_64 1:0.2.0-1.git05e6923.el7_6                                       oci-umount.x86_64 2:2.5-3.el7
  python-dmidecode.x86_64 0:3.12.2-4.el7                                                   python-pytoml.noarch 0:0.1.14-1.git7dea353.el7
  python-syspurpose.x86_64 0:1.24.26-3.el7.centos                                          slirp4netns.x86_64 0:0.4.3-4.el7_8
  subscription-manager.x86_64 0:1.24.26-3.el7.centos                                       subscription-manager-rhsm.x86_64 0:1.24.26-3.el7.centos
  subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7.centos

Complete!
[root@centos7-w2 ~]#

[root@centos7-w2 ~]# swapoff -a

[root@centos7-w2 ~]# kubeadm join 192.168.2.130:6443 --token a8a9vc.dk8krmn838is5gy6 \
    --discovery-token-ca-cert-hash sha256:10c8310cd5dd8b6847bc8fa0404974051ea268bbb68b00020690c8003139975a

W0531 19:40:35.637237    5557 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@centos7-w2 ~]# systemctl restart docker && systemctl enable docker

============================================================================================>
[root@centos7 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
centos7-w1    Ready       22m   v1.18.3
centos7-w2    Ready       67s   v1.18.3
master-node   Ready    master   40m   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
centos7-w1    Ready       28m     v1.18.3
centos7-w2    Ready       6m42s   v1.18.3
master-node   Ready    master   45m     v1.18.3
[root@centos7 ~]#
============================================================================================>
Testing The Master Node With NGINX

[root@centos7 ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@centos7 ~]#

[root@centos7 ~]# kubectl describe deployment nginx
Name:                   nginx
Namespace:              default
CreationTimestamp:      Sun, 31 May 2020 20:03:57 -0500
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx
    Port:         
    Host Port:    
    Environment:  
    Mounts:       
  Volumes:        
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  
NewReplicaSet:   nginx-f89759699 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  8s    deployment-controller  Scaled up replica set nginx-f89759699 to 1
[root@centos7 ~]#

[root@centos7 ~]# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
[root@centos7 ~]#

[root@centos7 ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-jgcp4   1/1     Running   0          65s
[root@centos7 ~]#

[root@centos7 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
centos7-w1    Ready       45m   v1.18.3
centos7-w2    Ready       24m   v1.18.3
master-node   Ready    master   63m   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1                443/TCP        64m
nginx        NodePort    10.103.135.131           80:32701/TCP   40s
[root@centos7 ~]#

[root@centos7 ~]# kubectl edit deployment nginx
------------------------------------------------>
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2020-06-04T00:37:35Z"
  generation: 2
  labels:
    app: nginx
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
      f:spec:
        f:progressDeadlineSeconds: {}
        f:replicas: {}
        f:revisionHistoryLimit: {}
        f:selector:
          f:matchLabels:
            .: {}
            f:app: {}
        f:strategy:
          f:rollingUpdate:
            .: {}
            f:maxSurge: {}
            f:maxUnavailable: {}
          f:type: {}
        f:template:
          f:metadata:
            f:labels:
              .: {}
              f:app: {}
          f:spec:
            f:containers:
              k:{"name":"nginx"}:
                .: {}
                f:image: {}
                f:imagePullPolicy: {}
spec:
  progressDeadlineSeconds: 600
  replicas: 2 - - - - - --------------------------------->Change Here
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2020-06-04T00:37:35Z"
    lastUpdateTime: "2020-06-04T00:37:50Z"
    message: ReplicaSet "nginx-f89759699" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-06-04T00:45:31Z"
    lastUpdateTime: "2020-06-04T00:45:31Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
------------------------------------------------>
[root@centos7 ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
centos7      Ready    master   15m   v1.18.3
centos7-w1   Ready       12m   v1.18.3
centos7-w2   Ready       12m   v1.18.3
[root@centos7 ~]#

[root@centos7 ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-2st5d   1/1     Running   0          56s
nginx-f89759699-zlhhd   1/1     Running   0          8m39s
[root@centos7 ~]#

Testing the Setup:
-------------------

[root@centos7 ~]# curl centos7-w1:30668



Welcome to nginx!



Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

[root@centos7 ~]# [root@centos7 ~]# curl centos7-w2:30668 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

[root@centos7 ~]# [root@centos7 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-f89759699-2st5d 1/1 Running 0 52m 10.244.1.2 centos7-w1 nginx-f89759699-zlhhd 1/1 Running 0 59m 10.244.2.2 centos7-w2 [root@centos7 ~]#

Monday, May 18, 2020

Configuring 3-Node RAC - Oracle Database 19c (19.3) - Oracle Enterprise Linux 8.0


Configuring 3-Node RAC for Oracle Database 19c (19.3) with April 2020 RU on Oracle Enterprise Linux 8.0 

Oracle 19c - GI and RDBMS Softwares with April 2020 RU

1. Oracle 19c RDBMS: LINUX.X64_193000_db_home.zip
2. Oracle 19c APR 2020 RU:p30869156_190000_Linux-x86-64.zip
3. Oracle 19c GI: LINUX.X64_193000_grid_home.zip
4. Oracle 19c APR 2020 RU:p30899722_190000_Linux-x86-64.zip

I have shared in this blog post major steps.

1. Installed Oracle Linux 8.0 (Server with GUI) and configured 3 Network Interfaces

Go to “19c-rac1” VM Settings, 

Click on Network 

1. Select “Adapter 1” and Click on “Enable Network Adapter” and Select “NAT” as Attached to 
2. Select “Adapter 2” and Click on “Enable Network Adapter” and Select “Host-Only Adapter” as Attached to 
3. Select “Adapter 3” and Click on “Enable Network Adapter” and Select “Internal Network” as Attached to

a. NAT
b. “Host-Only Adapter” - Public IP: 192.168.2.10/255.255.255.0
c. “Internal Network”  - Private IP: 10.1.4.10/255.255.255.0

Repeat the step:1 for other two nodes and finally NAT, Public IP, Private IP and SCAN IPs 

19c-rac1:
---------
a. NAT
b. Public IP: 192.168.2.10/255.255.255.0
c. Private IP: 10.1.4.10/255.255.255.0

19c-rac2:
--------
a. NAT
b. Public IP: 192.168.2.20/255.255.255.0
c. Private IP: 10.1.4.20/255.255.255.0

19c-rac3:
---------
a. NAT
b. Public IP: 192.168.2.30/255.255.255.0
c. Private IP: 10.1.4.30/255.255.255.0

[root@19c-rac1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.10 19c-rac1.localdomain          19c-rac1
10.1.4.10    19c-rac1-priv.localdomain     19c-rac1-priv
192.168.2.11 19c-rac1-vip.localdomain      19c-rac1-vip

192.168.2.20 19c-rac2.localdomain          19c-rac2
10.1.4.20    19c-rac2-priv.localdomain     19c-rac2-priv
192.168.2.21 19c-rac2-vip.localdomain      19c-rac2-vip

192.168.2.30 19c-rac3.localdomain          19c-rac3
10.1.4.30    19c-rac3-priv.localdomain     19c-rac3-priv
192.168.2.31 19c-rac3-vip.localdomain      19c-rac3-vip

192.168.2.40 rac-scan.localdomain          rac-scan
192.168.2.50 rac-scan.localdomain          rac-scan
192.168.2.60 rac-scan.localdomain          rac-scan
[root@19c-rac1 ~]# 

Note-1: Information from /etc/hosts
Note-2: Same for 19c-rac2 and 19c-rac3 from /etc/hosts

How to install and activate Cockpit web console on OEL 8


[root@19c-rac1 ~]# systemctl enable --now cockpit.socket
Created symlink /etc/systemd/system/sockets.target.wants/cockpit.socket ? /usr/lib/systemd/system/cockpit.socket.

[root@19c-rac1 ~]# sudo ss -tulpn | grep :9090
tcp    LISTEN   0        128                     *:9090                 *:*      users:(("systemd",pid=1,fd=23))
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# sudo firewall-cmd --add-service=cockpit
Warning: ALREADY_ENABLED: 'cockpit' already in 'public'
success

[root@19c-rac1 ~]# sudo firewall-cmd --add-service=cockpit --permanent
Warning: ALREADY_ENABLED: cockpit
success
[root@19c-rac1 ~]#

Pre-installation tasks - - > Repeat 19c-rac2 and 19c-rac3 nodes also

[root@19c-rac1 ~]# dnf install -y https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm
Last metadata expiration check: 5:55:02 ago on Tue 12 May 2020 10:11:38 AM EDT.
oracle-database-preinstall-19c-1.0-1.el8.x86_64.rpm                                                                                   46 kB/s |  24 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================
 Package                                            Architecture               Version                                   Repository                             Size
=====================================================================================================================================================================
Installing:
 oracle-database-preinstall-19c                     x86_64                     1.0-1.el8                                 @commandline                           24 k
Installing dependencies:
 ksh                                                x86_64                     20120801-254.0.1.el8                      ol8_appstream                         927 k
 libaio-devel                                       x86_64                     0.3.112-1.el8                             ol8_baseos_latest                      19 k
 libnsl                                             x86_64                     2.28-101.0.1.el8                          ol8_baseos_latest                      97 k

Transaction Summary
=====================================================================================================================================================================
Install  4 Packages

Total size: 1.0 M
Total download size: 1.0 M
Installed size: 3.5 M
Downloading Packages:
(1/3): libaio-devel-0.3.112-1.el8.x86_64.rpm         107 kB/s |  19 kB     00:00
(2/3): libnsl-2.28-101.0.1.el8.x86_64.rpm            446 kB/s |  97 kB     00:00
(3/3): ksh-20120801-254.0.1.el8.x86_64.rpm           1.8 MB/s | 927 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                2.0 MB/s | 1.0 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                          1/1
  Installing       : ksh-20120801-254.0.1.el8.x86_64                          1/4
  Running scriptlet: ksh-20120801-254.0.1.el8.x86_64                          1/4
  Installing       : libaio-devel-0.3.112-1.el8.x86_64                        2/4
  Installing       : libnsl-2.28-101.0.1.el8.x86_64                           3/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Installing       : oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4
  Verifying        : libnsl-2.28-101.0.1.el8.x86_64                           1/4
  Verifying        : libaio-devel-0.3.112-1.el8.x86_64                        2/4
  Verifying        : ksh-20120801-254.0.1.el8.x86_64                          3/4
  Verifying        : oracle-database-preinstall-19c-1.0-1.el8.x86_64          4/4

Installed:
ksh-20120801-254.0.1.el8.x86_64            
libaio-devel-0.3.112-1.el8.x86_64            
libnsl-2.28-101.0.1.el8.x86_64            
oracle-database-preinstall-19c-1.0-1.el8.x86_64

Complete!
[root@19c-rac1 ~]#

Note: If you have not used the "oracle-database-preinstall-19c" package to perform all prerequisites, 
you will need to manually perform the setup tasks.


[root@rac2-19cr7-oel82 ~]# cat /etc/sysctl.conf
[root@19c-rac1 ~]# cat /etc/security/limits.d/oracle-database-preinstall-19c.conf
[root@19c-rac1 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

[root@19c-rac1 ~]# vi /etc/selinux/config
[root@19c-rac1 ~]# setenforce Permissive

[root@19c-rac1 ~]# systemctl stop firewalld
[root@19c-rac1 ~]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# mkdir -p /u01/app/19.3.0/grid
[root@19c-rac1 ~]# mkdir -p /u01/app/oracle/product/19.3.0/dbhome_1
[root@19c-rac1 ~]# chown -R oracle:oinstall /u01
[root@19c-rac1 ~]# chmod -R 775 /u01/

[oracle@19c-rac1 ~]$ vi grid_env
[oracle@19c-rac1 ~]$ cat grid_env
export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ vi db_env
[oracle@19c-rac1 ~]$ cat db_env
export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

[oracle@19c-rac1 ~]$
[oracle@19c-rac1 ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs

export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=19c-rac1
export ORACLE_UNQNAME=cdbrac
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/19.3.0/grid
export DB_HOME=$ORACLE_BASE/product/19.3.0/dbhome_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
[oracle@19c-rac1 ~]$

[root@19c-rac1 rpm]# pwd
/u01/app/19.3.0/grid/cv/rpm

[root@19c-rac1 rpm]# export CVUQDISK_GRP=dba
[root@19c-rac1 rpm]# rpm -ivh cvuqdisk-1.0.10-1.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:cvuqdisk-1.0.10-1                ################################# [100%]
[root@19c-rac1 rpm]#

[root@19c-rac1 rpm]# rpm -qa | grep cvuqdisk
cvuqdisk-1.0.10-1.x86_64
[root@19c-rac1 rpm]#

2. Create Shared Disks 

1. Choose “Fixed size” 
2. Best practice to Choose File location outside of existing VM directory and input “12 GB” as storage space 
3. Click “Create

3. Sharing Shared Disks 

1. Go to “File” > “Virtual Media Manager” 
2. Right click on Shared disk and make it Shareable. 
3. Repeat the task for all eight shared disk


4. Formatting Shared Disks Start the “19c-rac1” VM and login a root user. Now we have to configure the Shared disks.

1. Go to /dev file system and look for sd* using ls command. Since we have added four shared disks, 
you can see sdb,sdc,sdd,sde,sdf,sdg,sdh and sdi 

2. Format shared disks using “fdisk” command as shown below. 
3. Once format is completed, you can see sdb1,sdc1,sdd1,sde1,sdf1,sdg1,sdh1 and sdi1


[root@19c-rac1 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x3c82e2b8.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-25165823, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-25165823, default 25165823):

Created a new partition 1 of type 'Linux' and of size 12 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[root@19c-rac1 ~]#

Note: Repeat the same process for sdc,sde,sdf,sdg,sdh and sdi

5. Configure your UDEV rules 

1. Add “options=-g” to "/etc/scsi_id.config" file. 
2. Find SCSI ID of all eight disk using “scsi_id” command as shown above. 

[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VB422666b7-031705cb
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdd
1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sde
1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdf
1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdg
1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdh
1ATA_VBOX_HARDDISK_VBa855f734-9a09207e
[root@19c-rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi
1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d
[root@19c-rac1 ~]#

3. Using scsi_id, edit "/etc/udev/rules.d/99-oracle-asmdevices.rules" file as shown above. 

[root@19c-rac1~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
[root@19c-rac1~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"

4. Load updated block device partition tables.

Note: Clone 19c-rac1 node as 19c-rac2 and 19c-rac3.

19c-rac2

[oracle@19c-rac2 ~]$ su - root
Password:
[root@19c-rac2 ~]# cat /etc/scsi_id.config
options=-g

[root@19c-rac2 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"
[root@19c-rac2 ~]#

19c-rac3:

[oracle@19c-rac3 ~]$ su - root
Password:
[root@19c-rac3 ~]# cat /etc/scsi_id.config
options=-g

[root@19c-rac3 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBdeb8f461-889e4cff", SYMLINK+="oracleasm/19cr7-ocrvd1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB422666b7-031705cb", SYMLINK+="oracleasm/19cr7-ocrvd2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB00f5bad4-581cc9f7", SYMLINK+="oracleasm/19cr7-ocrvd3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB903058f7-d8ae2e4b", SYMLINK+="oracleasm/19cr7-ocrvd4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB99cca2d2-ca0d6275", SYMLINK+="oracleasm/19cr7-data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VB1403a516-bf0cd120", SYMLINK+="oracleasm/19cr7-data2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBa855f734-9a09207e", SYMLINK+="oracleasm/19cr7-reco1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", 
RESULT=="1ATA_VBOX_HARDDISK_VBd582c9b2-471bc81d", SYMLINK+="oracleasm/19cr7-reco2", OWNER="oracle", GROUP="dba", MODE="0660"
[root@19c-rac3 ~]#


Installation Oracle GRID Infrastructure 19c (19.3.0) and applying APRIL 2020 RU 19c (19.7.0) 

[oracle@19c-rac1 ~]$ export CV_ASSUME_DISTID=OEL7.6
[oracle@19c-rac1 grid]$ ./gridSetup.sh -applyPSU /u01/app/19.3.0/grid/30899722/

Preparing the home to patch...
Applying the patch /u01/app/19.3.0/grid/30899722/...
Successfully applied the patch.
The log can be found at: /tmp/GridSetupActions2020-05-16_08-47-40PM/installerPatchActions_2020-05-16_08-47-40PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at:
 /u01/app/19.3.0/grid/install/response/grid_2020-05-16_08-47-40PM.rsp

You can find the log of this install session at:
 /tmp/GridSetupActions2020-05-16_08-47-40PM/gridSetupActions2020-05-16_08-47-40PM.log
Moved the install session logs to:
 /u01/app/oraInventory/logs/GridSetupActions2020-05-16_08-47-40PM
[oracle@19c-rac1 grid]$









[root@19c-rac1 ~]# sh /tmp/GridSetupActions2020-05-18_01-13-22AM/CVU_19.0.0.0.0_oracle/runfixup.sh
All Fix-up operations were completed successfully.
[root@19c-rac1 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@19c-rac3 ~]#


[root@19c-rac1 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac1/crsconfig/rootcrs_19c-rac1_2020-05-18_01-45-03AM.log
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:45:14 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:45:14 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:45:16 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:45:17 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:45:46 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:45:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:46:06 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:46:12 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:46:12 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:46:17 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:46:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:46:37 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:46:37 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:46:41 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:46:46 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /u01/app/oracle/cfgtoollogs/asmca/asmca-200518AM014717.log for details.

2020/05/18 01:48:14 CLSRSC-482: Running command: '/u01/app/19.3.0/grid/bin/ocrconfig -upgrade oracle oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk 0f7b1a25f7bf4f32bf8fa0f50f003b75.
Successfully replaced voting disk group with +OCRVD.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0f7b1a25f7bf4f32bf8fa0f50f003b75 (/dev/sdb1) [OCRVD]
Located 1 voting disk(s).
2020/05/18 01:49:09 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:50:12 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:50:12 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:51:36 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:51:59 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac2/crsconfig/rootcrs_19c-rac2_2020-05-18_01-52-26AM.log
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:52:33 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:52:33 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:52:35 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:52:36 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:52:39 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:52:39 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:52:57 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:52:57 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:52:59 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:53:00 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:53:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:53:20 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:53:20 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:53:22 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:53:23 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service
2020/05/18 01:53:32 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:54:18 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:54:18 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:54:35 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:54:41 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/19c-rac3/crsconfig/rootcrs_19c-rac3_2020-05-18_01-55-10AM.log
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2020/05/18 01:55:17 CLSRSC-363: User ignored prerequisites during installation
2020/05/18 01:55:17 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2020/05/18 01:55:18 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2020/05/18 01:55:19 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2020/05/18 01:55:21 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2020/05/18 01:55:21 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2020/05/18 01:55:39 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2020/05/18 01:55:40 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2020/05/18 01:55:42 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2020/05/18 01:55:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2020/05/18 01:56:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/05/18 01:56:03 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2020/05/18 01:56:03 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2020/05/18 01:56:05 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2020/05/18 01:56:06 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
Redirecting to /bin/systemctl restart rsyslog.service
2020/05/18 01:56:15 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2020/05/18 01:57:01 CLSRSC-343: Successfully started Oracle Clusterware stack
2020/05/18 01:57:01 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2020/05/18 01:57:22 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2020/05/18 01:57:30 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@19c-rac3 ~]#

Configuring diskgroups DATADG and RECODG through ASMCA  





Check the status of ASM Instance from 19c-rac1, 19c-rac2 and 19c-rac3 nodes

[oracle@19c-rac1 ~]$ ps -ef | grep pmon
oracle    2071  2333  0 09:41 pts/0    00:00:00 grep --color=auto pmon
oracle    7081     1  0 01:52 ?        00:00:01 asm_pmon_+ASM1
oracle    5698     1  0 09:41 ?        00:00:00 mdb_pmon_-MGMTDB
[oracle@19c-rac1 ~]$

[oracle@19c-rac2 ~]$ ps -ef | grep pmon
oracle    5936  5895  0 09:43 pts/0    00:00:00 grep --color=auto pmon
oracle    8270     1  0 01:54 ?        00:00:01 asm_pmon_+ASM2
[oracle@19c-rac2 ~]$ 

[oracle@19c-rac3 ~]$ ps -ef | grep pmon
oracle    2371  2333  0 09:44 pts/0    00:00:00 grep --color=auto pmon
oracle    7981     1  0 01:57 ?        00:00:01 asm_pmon_+ASM3
[oracle@19c-rac3 ~]$

Installation of Oracle Database 19c (19.3.0) Software 

Note: Select "Setup Software Only" Option



[root@19c-rac1 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac1 ~]#

[root@19c-rac2 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@19c-rac3 ~]#

Creation Oracle Database 19c (Container Database with 2 PDBs) 









Check ASM Instances and RDBMS Instances

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [cdbrac1] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.3.0/grid
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@19c-rac1 ~]$ cat /etc/os-release | grep PRETTY
PRETTY_NAME="Oracle Linux Server 8.0"
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Mon May 18 09:48:51 2020
Version 19.7.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

SQL> select instance_name,instance_number from gv$instance;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
+ASM2                          2
+ASM3                          3
+ASM1                          1

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

[oracle@19c-rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? cdbrac
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/19.3.0/dbhome_1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ sqlplus sys@cdbrac as sysdba


SQL*Plus: Release 19.0.0.0.0 - Production on Mon May 18 09:50:31 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Enter password:
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 PDB2                           READ WRITE NO
SQL>

SQL> select instance_name,instance_number from gv$instance order by instance_number;

INSTANCE_NAME    INSTANCE_NUMBER
---------------- ---------------
cdbrac1                        1
cdbrac2                        2
cdbrac3                        3

SQL>

Applying APRIL 2020 RU (p30869156_190000_Linux-x86-64.zip) for Oracle 19c (19.3.0)

[root@19c-rac1 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 09:59:03 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_09-59-09AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_09-59-44AM.log
The id for this session is 7R66

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac1
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-02-12AM_1.log

OPatchauto session completed at Mon May 18 10:08:41 2020
Time taken to complete the session 9 minutes, 38 seconds
[root@19c-rac1 ~]#


[root@19c-rac2 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 10:10:57 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_10-11-04AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_10-11-22AM.log
The id for this session is YZ9L

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac2
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-13-50AM_1.log
OPatchauto session completed at Mon May 18 10:20:10 2020
Time taken to complete the session 9 minutes, 13 seconds
[root@19c-rac2 ~]#

[root@19c-rac3 ~]# sh /u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatchauto 
apply /u01/app/oracle/product/19.3.0/dbhome_1/30869156/ 
-oh /u01/app/oracle/product/19.3.0/dbhome_1/


OPatchauto session is initiated at Mon May 18 10:20:34 2020

System initialization log file is /u01/app/oracle/product/19.3.0/dbhome_1/
cfgtoollogs/opatchautodb/systemconfig2020-05-18_10-20-42AM.log.

Session log file is /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-18_10-21-03AM.log
The id for this session is 3BT9

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.3.0/dbhome_1 to bring down database service

Bringing down database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbrac
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing prepatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Start applying binary patch on home /u01/app/oracle/product/19.3.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/dbhome_1

Starting database service on home /u01/app/oracle/product/19.3.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/dbhome_1

Preparing home /u01/app/oracle/product/19.3.0/dbhome_1 after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/dbhome_1

SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/dbhome_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:19c-rac3
RAC Home:/u01/app/oracle/product/19.3.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/oracle/product/19.3.0/dbhome_1/30869156
Log: /u01/app/oracle/product/19.3.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-18_10-23-31AM_1.log
OPatchauto session completed at Mon May 18 11:04:59 2020
Time taken to complete the session 44 minutes, 26 seconds
[root@19c-rac3 ~]#


Check the status, stop and start cluster for all the nodes

[root@19c-rac1 ~]# crsctl check cluster -all
**************************************************************
19c-rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
19c-rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
19c-rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on '19c-rac3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.qosmserver' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server '19c-rac3'
CRS-2673: Attempting to stop 'ora.cdbrac.db' on '19c-rac3'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac2' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac2'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '19c-rac2'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac3' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac3'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac3'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.cdbrac.db' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on '19c-rac1'
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.19c-rac2.vip' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.scan1.vip' on '19c-rac2'
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac3.vip' on '19c-rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.19c-rac2.vip' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.scan3.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.cvu' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.19c-rac3.vip' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3'
CRS-2677: Stop of 'ora.scan1.vip' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac2'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac2' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac2' succeeded.
CRS-2677: Stop of 'ora.chad' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.chad' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac2'
CRS-2677: Stop of 'ora.ons' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac2'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac2'
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac2'
CRS-2677: Stop of 'ora.storage' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac3'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac3' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac3' succeeded.
CRS-2677: Stop of 'ora.ctssd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.chad' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac3'
CRS-2677: Stop of 'ora.ons' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac3'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac3' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac3'
CRS-2677: Stop of 'ora.storage' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac3'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac2'
CRS-2677: Stop of 'ora.cssd' on '19c-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac3'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac3'
CRS-2677: Stop of 'ora.evmd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac3'
CRS-2677: Stop of 'ora.cssd' on '19c-rac3' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on '19c-rac1'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server '19c-rac1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.RECODG.dg' on '19c-rac1'
CRS-2677: Stop of 'ora.OCRVD.dg' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.RECODG.dg' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1'
CRS-2677: Stop of 'ora.MGMTLSNR' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '19c-rac1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on '19c-rac1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on '19c-rac1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server '19c-rac1' succeeded.
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.19c-rac1.vip' on '19c-rac1'
CRS-2677: Stop of 'ora.19c-rac1.vip' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '19c-rac1'
CRS-2677: Stop of 'ora.ons' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '19c-rac1'
CRS-2677: Stop of 'ora.net1.network' on '19c-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '19c-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on '19c-rac1'
CRS-2677: Stop of 'ora.storage' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '19c-rac1'
CRS-2677: Stop of 'ora.asm' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '19c-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on '19c-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on '19c-rac1'
CRS-2677: Stop of 'ora.ctssd' on '19c-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on '19c-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '19c-rac1'
CRS-2677: Stop of 'ora.cssd' on '19c-rac1' succeeded
[root@19c-rac1 ~]#

[root@19c-rac1 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac3'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on '19c-rac2'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac3'
CRS-2672: Attempting to start 'ora.evmd' on '19c-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac3'
CRS-2672: Attempting to start 'ora.cssd' on '19c-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac1'
CRS-2676: Start of 'ora.diskmon' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.diskmon' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on '19c-rac3'
CRS-2676: Start of 'ora.evmd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.diskmon' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.evmd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.cssd' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac3'
CRS-2676: Start of 'ora.cssd' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac3'
CRS-2676: Start of 'ora.cssd' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '19c-rac1'
CRS-2672: Attempting to start 'ora.ctssd' on '19c-rac1'
CRS-2676: Start of 'ora.ctssd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.ctssd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.ctssd' on '19c-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac3'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac2'
CRS-2676: Start of 'ora.asm' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on '19c-rac1'
CRS-2672: Attempting to start 'ora.storage' on '19c-rac2'
CRS-2676: Start of 'ora.asm' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on '19c-rac1'
CRS-2676: Start of 'ora.asm' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on '19c-rac3'
CRS-2676: Start of 'ora.storage' on '19c-rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac3'
CRS-2676: Start of 'ora.crsd' on '19c-rac3' succeeded
CRS-2676: Start of 'ora.storage' on '19c-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac2'
CRS-2676: Start of 'ora.storage' on '19c-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on '19c-rac1'
CRS-2676: Start of 'ora.crsd' on '19c-rac2' succeeded
CRS-2676: Start of 'ora.crsd' on '19c-rac1' succeeded
[root@19c-rac1 ~]#

Check the Status as "Oracle" User

[oracle@19c-rac1 ~]$ srvctl status database -db cdbrac
Instance cdbrac1 is running on node 19c-rac1
Instance cdbrac2 is running on node 19c-rac2
Instance cdbrac3 is running on node 19c-rac3
[oracle@19c-rac1 ~]$

[oracle@19c-rac1 ~]$ srvctl config database -d cdbrac
Database unique name: cdbrac
Database name: cdbrac
Oracle home: /u01/app/oracle/product/19.3.0/dbhome_1
Oracle user: oracle
Spfile: +DATADG/CDBRAC/PARAMETERFILE/spfile.275.1040722631
Password file: +DATADG/CDBRAC/PASSWORD/pwdcdbrac.256.1040721721
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECODG,DATADG
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: cdbrac1,cdbrac2,cdbrac3
Configured nodes: 19c-rac1,19c-rac2,19c-rac3
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
[oracle@19c-rac1 ~]$

Hope it helps...........Thank You

Sunday, March 22, 2020

Oracle Certification Blog - Spotlight on Success


Oracle Certification Blog - Spotlight on Success 



Original URL: https://blogs.oracle.com/certification/spotlight-on-success%3a-ravi-kumar-y-v-%e2%80%9creal-time-experience-
along-with-certifications-opened-the-door-for-a-great-oracle-architect-position%e2%80%9d

Sunday, October 13, 2019

Oracle 19c (19.3.0) Installation on Oracle Enterprise Linux 8.0 (OEL 8.0)



Oracle 19c (19.3.0) Installation on Oracle Enterprise Linux 8.0

Error-1: 
==========

/u01/app/oracle/product/19.3.0/dbhome_1/perl/bin/perl: error while loading shared libraries: 
libnsl.so.1: cannot open shared object file: No such file or directory

Check the following
===================
[oracle@19c-19g-orcld ~]$ ldd /u01/app/oracle/product/19.3.0/dbhome_1/perl/bin/perl
        linux-vdso.so.1 (0x00007ffd0e9b1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f31d8706000)
        libnsl.so.1 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f31d82e9000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f31d7f67000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f31d7d3e000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f31d7b3a000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f31d7776000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f31d755e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f31d8926000)
[oracle@19c-19g-orcld ~]$

Installed required RPMs for OEL 8.0
===================================
# New for OL8
dnf install -y libnsl
dnf install -y libnsl.i686
dnf install -y libnsl2
dnf install -y libnsl2.i686

[root@19c-19g-orclp ~]# # New for OL8
[root@19c-19g-orclp ~]# dnf install -y libnsl
Last metadata expiration check: 0:01:18 ago on Sun 13 Oct 2019 11:02:26 AM EDT.
Dependencies resolved.
====================================================================================================================================
 Package                        Arch                           Version               Repository                                 Size
====================================================================================================================================
Installing:
 libnsl                         x86_64                         2.28-42.0.1.el8_0.1   ol8_baseos_latest                          87 k

Transaction Summary
=====================================================================================================================================
Install  1 Package

Total download size: 87 k
Installed size: 147 k
Downloading Packages:
libnsl-2.28-42.0.1.el8_0.1.x86_64.rpm                                                                                          71 kB/s |  87 kB     00:01
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                          71 kB/s |  87 kB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                 1/1
  Installing       : libnsl-2.28-42.0.1.el8_0.1.x86_64                                                               1/1
  Running scriptlet: libnsl-2.28-42.0.1.el8_0.1.x86_64                                                               1/1
  Verifying        : libnsl-2.28-42.0.1.el8_0.1.x86_64                                                               1/1

Installed:
  libnsl-2.28-42.0.1.el8_0.1.x86_64

Complete!
[root@19c-19g-orclp ~]# dnf install -y libnsl.i686
Last metadata expiration check: 0:01:21 ago on Sun 13 Oct 2019 11:02:26 AM EDT.
Dependencies resolved.
=========================================================================================================================================
 Package                         Arch                          Version                   Repository                                  Size
=========================================================================================================================================
Installing:
 libnsl                          i686                          2.28-42.0.1.el8_0.1       ol8_baseos_latest                           91 k
Installing dependencies:
 glibc                           i686                          2.28-42.0.1.el8_0.1       ol8_baseos_latest                          3.6 M

Transaction Summary
==========================================================================================================================================
Install  2 Packages

Total download size: 3.6 M
Installed size: 14 M
Downloading Packages:
(1/2): libnsl-2.28-42.0.1.el8_0.1.i686.rpm                                               84 kB/s |  91 kB     00:01
(2/2): glibc-2.28-42.0.1.el8_0.1.i686.rpm                                                2.8 MB/s | 3.6 MB     00:01
-------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                    2.9 MB/s | 3.6 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                     1/1
  Running scriptlet: glibc-2.28-42.0.1.el8_0.1.i686                                      1/2
  Installing       : glibc-2.28-42.0.1.el8_0.1.i686                                      1/2
  Running scriptlet: glibc-2.28-42.0.1.el8_0.1.i686                                      1/2
  Installing       : libnsl-2.28-42.0.1.el8_0.1.i686                                     2/2
  Running scriptlet: libnsl-2.28-42.0.1.el8_0.1.i686                                     2/2
  Verifying        : libnsl-2.28-42.0.1.el8_0.1.i686                                     1/2
  Verifying        : glibc-2.28-42.0.1.el8_0.1.i686                                      2/2

Installed:
  libnsl-2.28-42.0.1.el8_0.1.i686                                                glibc-2.28-42.0.1.el8_0.1.i686

Complete!
[root@19c-19g-orclp ~]# dnf install -y libnsl2
Last metadata expiration check: 0:01:26 ago on Sun 13 Oct 2019 11:02:26 AM EDT.
Package libnsl2-1.2.0-2.20180605git4a062cf.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@19c-19g-orclp ~]# dnf install -y libnsl2.i686
Last metadata expiration check: 0:02:28 ago on Sun 13 Oct 2019 11:02:26 AM EDT.
Dependencies resolved.
===========================================================================================================================================
 Package                           Arch                     Version                            Repository                             Size
===========================================================================================================================================
Installing:
 libnsl2                           i686                     1.2.0-2.20180605git4a062cf.el8     ol8_baseos_latest                      61 k
Installing dependencies:
 libsepol                          i686                     2.8-2.el8                          ol8_baseos_latest                     366 k
 libcom_err                        i686                     1.44.3-2.el8                       ol8_baseos_latest                      48 k
 pcre2                             i686                     10.32-1.el8                        ol8_baseos_latest                     246 k
 zlib                              i686                     1.2.11-10.el8                      ol8_baseos_latest                     103 k
 libverto                          i686                     0.3.0-5.el8                        ol8_baseos_latest                      25 k
 keyutils-libs                     i686                     1.5.10-6.el8                       ol8_baseos_latest                      34 k
 openssl-libs                      i686                     1:1.1.1-8.0.1.el8                  ol8_baseos_latest                     1.4 M
 krb5-libs                         i686                     1.16.1-22.el8                      ol8_baseos_latest                     906 k
 libselinux                        i686                     2.8-6.el8                          ol8_baseos_latest                     186 k
 libtirpc                          i686                     1.1.4-3.el8                        ol8_baseos_latest                     120 k

Transaction Summary
==========================================================================================================================================
Install  11 Packages

Total download size: 3.5 M
Installed size: 8.6 M
Downloading Packages:
(1/11): libcom_err-1.44.3-2.el8.i686.rpm                                                       42 kB/s |  48 kB     00:01
(2/11): libsepol-2.8-2.el8.i686.rpm                                                            312 kB/s | 366 kB     00:01
(3/11): pcre2-10.32-1.el8.i686.rpm                                                             205 kB/s | 246 kB     00:01
(4/11): libnsl2-1.2.0-2.20180605git4a062cf.el8.i686.rpm                                        839 kB/s |  61 kB     00:00
(5/11): zlib-1.2.11-10.el8.i686.rpm                                                            1.4 MB/s | 103 kB     00:00
(6/11): libverto-0.3.0-5.el8.i686.rpm                                                          300 kB/s |  25 kB     00:00
(7/11): keyutils-libs-1.5.10-6.el8.i686.rpm                                                    474 kB/s |  34 kB     00:00
(8/11): libselinux-2.8-6.el8.i686.rpm                                                          2.1 MB/s | 186 kB     00:00
(9/11): krb5-libs-1.16.1-22.el8.i686.rpm                                                       7.5 MB/s | 906 kB     00:00
(10/11): openssl-libs-1.1.1-8.0.1.el8.i686.rpm                                                 8.1 MB/s | 1.4 MB     00:00
(11/11): libtirpc-1.1.4-3.el8.i686.rpm                                                         2.3 MB/s | 120 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                          2.4 MB/s | 3.5 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                           1/1
  Installing       : libcom_err-1.44.3-2.el8.i686                                              1/11
  Running scriptlet: libcom_err-1.44.3-2.el8.i686                                              1/11
  Installing       : keyutils-libs-1.5.10-6.el8.i686                                           2/11
  Installing       : libverto-0.3.0-5.el8.i686                                                 3/11
  Installing       : zlib-1.2.11-10.el8.i686                                                   4/11
  Installing       : openssl-libs-1:1.1.1-8.0.1.el8.i686                                       5/11
  Running scriptlet: openssl-libs-1:1.1.1-8.0.1.el8.i686                                       5/11
  Installing       : pcre2-10.32-1.el8.i686                                                    6/11
  Installing       : libsepol-2.8-2.el8.i686                                                   7/11
  Running scriptlet: libsepol-2.8-2.el8.i686                                                   7/11
  Installing       : libselinux-2.8-6.el8.i686                                                 8/11
  Running scriptlet: libselinux-2.8-6.el8.i686                                                 8/11
  Installing       : krb5-libs-1.16.1-22.el8.i686                                              9/11
  Installing       : libtirpc-1.1.4-3.el8.i686                                                 10/11
  Running scriptlet: libtirpc-1.1.4-3.el8.i686                                                 10/11
  Installing       : libnsl2-1.2.0-2.20180605git4a062cf.el8.i686                               11/11
  Running scriptlet: libnsl2-1.2.0-2.20180605git4a062cf.el8.i686                               11/11
  Verifying        : libsepol-2.8-2.el8.i686                                                   1/11
  Verifying        : libcom_err-1.44.3-2.el8.i686                                              2/11
  Verifying        : pcre2-10.32-1.el8.i686                                                    3/11
  Verifying        : libnsl2-1.2.0-2.20180605git4a062cf.el8.i686                               4/11
  Verifying        : zlib-1.2.11-10.el8.i686                                                   5/11
  Verifying        : libverto-0.3.0-5.el8.i686                                                 6/11
  Verifying        : keyutils-libs-1.5.10-6.el8.i686                                           7/11
  Verifying        : openssl-libs-1:1.1.1-8.0.1.el8.i686                                       8/11
  Verifying        : krb5-libs-1.16.1-22.el8.i686                                              9/11
  Verifying        : libselinux-2.8-6.el8.i686                                                 10/11
  Verifying        : libtirpc-1.1.4-3.el8.i686                                                 11/11

Installed:
  libnsl2-1.2.0-2.20180605git4a062cf.el8.i686     libsepol-2.8-2.el8.i686       libcom_err-1.44.3-2.el8.i686        pcre2-10.32-1.el8.i686
  zlib-1.2.11-10.el8.i686                         libverto-0.3.0-5.el8.i686     keyutils-libs-1.5.10-6.el8.i686     openssl-libs-1:1.1.1-8.0.1.el8.i686
  krb5-libs-1.16.1-22.el8.i686                    libselinux-2.8-6.el8.i686     libtirpc-1.1.4-3.el8.i686

Complete!
[root@19c-19g-orclp ~]#

Checked the following:
======================
[oracle@19c-19g-orcld ~]$ ldd /u01/app/oracle/product/19.3.0/dbhome_1/perl/bin/perl
        linux-vdso.so.1 (0x00007ffd0e9b1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f31d8706000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f31d84ed000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f31d82e9000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f31d7f67000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f31d7d3e000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f31d7b3a000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f31d7776000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f31d755e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f31d8926000)
[oracle@19c-19g-orcld ~]$

Installed "xorg-x11-xauth" also 
==================================
[root@19c-19g-orclp ~]# dnf install -y xorg-x11-xauth
Oracle Linux 8 Application Stream (x86_64)                               2.7 kB/s | 2.9 kB     00:01
Oracle Linux 8 BaseOS Latest (x86_64)                                    2.5 kB/s | 2.7 kB     00:01
Package xorg-x11-xauth-1:1.0.9-12.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@19c-19g-orclp ~]#


Error-2: 
==========
[WARNING] [INS-08101] Unexpected error while executing the action at state: 'supportedOSCheck'
   CAUSE: No additional information available.
   ACTION: Contact Oracle Support Services or refer to the software manual.
   SUMMARY:
       - java.lang.NullPointerException




Set the following: It works. Please let me know if any work around

[oracle@19c-19g-orcld dbhome_1]$ export CV_ASSUME_DISTID=RHEL7.6
[oracle@19c-19g-orcld dbhome_1]$ ./runInstaller &




Note: Got warnings but the still the Oracle 19c Installation proceeds...... Thank you 

Please let me know if any other workaround.........