Saturday, July 14, 2018

Oracle has released BUG fix for "Oracle Exadata Storage Server Patch (12.2.1.1.7) encountered bugs - 'CELLSRV' Restarting On Exadata x4-2 and Exadata x6-2"


Two weeks back we have encountered BUG after patching PSU APR 2018 including Oracle Exadata Storage Server 12.2.1.1.7 version. 
So CELLSRV started giving some issues (Restarting frequently CELLSRV). I have posted the blog regarding the issue.

Oracle Exadata Storage Server Patch (12.2.1.1.7) encountered bugs - 'CELLSRV' Restarting On Exadata x4-2 and Exadata x6-2
http://yvrk1973.blogspot.com/2018/06/oracle-exadata-storage-server-patch.html

Thanks to Oracle Exadata Development Team. They developed Interim Patch for the fix for 
Oracle Exadata Storage Server Version 12.2.1.1.7 recently.


Patch: p28181789_122117_Linux-x86-64.zip

The zip file contains two rpms and this README.txt.

a.1) cell-12.2.1.1.7_LINUX.X64_180506-1-rpm.bin - 
     This is the base release rpm. This is included in case a rollback is needed.

a.2) cell-12.2.1.1.7.28181789V1_LINUX.X64_180706-1-rpm.bin - 
     This is the interim patch that contains the fix for the bug listed below.  
     This patch will replace existing storage server software.

a.3) ========== Bug fixes or Diagnostics included in this ONEOFF ===========

Bug Fixes: 28181789    ORA-07445: [0000000000000000+0] AFTER UPGRADING CELL TO 12.2.1.1.7


Non-Rolling
============================
Copy the Cell Oneoff RPM File(s) to Target Cell Nodes

   1.a) If the patch has already been downloaded to a database server and unzipped under /tmp, move to 1.b
        Download and unzip the p28181789_122117_Linux-x86_64.zip under /tmp/ on one of the database servers.
       # unzip p28181789_122117_Linux-x86_64.zip -d /tmp

   1.b) Change directory to the location where oneoff cell rpm is located
       # cd /tmp/RPM_patch_12.2.1.1.7

   1.c) Create temporary working directory /var/log/exadatatmp/SAVE_patch_28181789 on cells
       # dcli -l root -g cell_group mkdir -p /var/log/exadatatmp/SAVE_patch_28181789

   1.d) Copy new cell RPM bin file to /var/log/exadatatmp/SAVE_patch_28181789 on cells
       # dcli -l root -g cell_group -f /tmp/RPM_patch_12.2.1.1.7/cell-12.2.1.1.7.28181789V1_LINUX.X64_180706-1-rpm.bin 
        -d /var/log/exadatatmp/SAVE_patch_28181789

2. Shut Down the Cluster
   ------------------------
   If you want to stop cluster from one node, execute the following command from that node.
   [root@dbnode]# crsctl stop cluster -all

   If you want to stop crs on each node, execute following command on each dbnode.
   [root@dbnode]# crsctl stop crs

3. Install the RPM on Each Cell
------------------------------

   3.a) Check if the cell-12.2.1.1.7.28181789V1_LINUX.X64_180706-1-rpm.bin has execute permissions and enable them 
        if needed.

   3.b) Run the following command to apply the cell interim patch.
   
        [root@cell ~]# /var/log/exadatatmp/SAVE_patch_28181789/cell-12.2.1.1.7.28181789V1_LINUX.X64_180706-1-rpm.bin 
                       --doall --force
        ...
        [INFO] Upgrade was successful.
        "Upgrade was successful" is displayed on success.

Note: Refer to section (f) for any known issues during the installation

   3.c) Run the following command to verify the installation.

        [root@cell ~]# rpm -qa | grep ^cell-
        cell-12.2.1.1.7_LINUX.X64_180706-1.x86_64

   3.d) Logout and login the shell to reflect new command path.

       Then, run the following command to verify all services are running.
       [root@cell ~]# cellcli -e list cell attributes cellsrvStatus,msStatus,rsStatus

4. Restart the Rluster 
----------------------
   If you want to start cluster from one node, execute the following command from that node.
   [root@dbnode]# crsctl start cluster -all

   If you want to start crs on each node, execute following command from each node.
   [root@dbnode]# crsctl start crs


5. Remove the Patch File
   -------------------------
   After successful RPM installation, you can remove the temporary patch staging directory to save disk space.

[root@dbnode ~]# dcli -l root -g cell_group rm -rf /var/log/exadatatmp/SAVE_patch_28181789

==============================================================
e) Rolling Back This Interim Patch on Exadata Storage Servers
==============================================================
   To roll back, follow either the rolling or non-rolling steps above
   to install cell-12.2.1.1.7_LINUX.X64_180506-1-rpm.bin.
   This rpm is included in the zip file for this patch.

Note: We can apply the patch "ROLLING" manner also. Please refer for more details MOS with BUG number.

3 comments: