Monday 5 October 2020

GI July 2020 RU 19.8.0.0(31305339) patching Issues , resume failed patch with opatchauto

Had couple of issues while applying GI JULY RU 19.8.0.0 patch(31305339) on Cluster Node2  , Patch was successfully applied in node1

We can expect similar error for both grid and db home  while applying RU patch , Solution also same for both grid and db homes 

Error 1:
opatchauto apply failed with /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied) On node 2

grid patch apply on node2:
root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 02:52:32 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_02-52-36AM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_02-52-49AM.log
The id for this session is ST84
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start applying binary patch on home /u01/app/19.7.0.0/grid
Failed while applying binary patches on home /u01/app/19.7.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.
Command failed:  /u01/app/19.7.0.0/grid/OPatch/opatchauto  apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre -persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser -analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_02-55-30AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sun Oct  4 03:02:44 2020
Time taken to complete the session 10 minutes, 12 seconds
 opatchauto failed with error code 42


Cause : file permission issue, Compared file permission on both nodes ,they are different 
root@racdbnode02~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml
-rw-r--r-- 1 oracle oinstall 174 Oct  4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml

root@ racdbnode01~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml 
-rw-rw---- 1 oracle oinstall 174 Oct  4 02:38 /app/oraInventory/ContentsXML/oui-patch.xml

Solution for error 1: Change permissions to 660 
root@racdbnode02~ #chmod 660 /app/oraInventory/ContentsXML/oui-patch.xml
root@racdbnode02~ #ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 oracle oinstall 174 Oct  4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml

Restart patch apply with resume option: 
Opatchauto resume will resume previous patch session

root@racdbnode02~ # opatchauto resume 
OPatchauto session is initiated at Sun Oct  4 03:31:20 2020
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-31-23AM.log
Resuming existing session with id ST84

Start applying binary patch on home /u01/app/19.7.0.0/grid
Failed while applying binary patches on home /u01/app/19.7.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.
Command failed:  /u01/app/19.7.0.0/grid/OPatch/opatchauto  apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster 
-binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre 
-persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser 
-analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/app/19c_Software/31305339/31281355
Log: 
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: 
Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist. 

After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sun Oct  4 03:31:50 2020
Time taken to complete the session 0 minute, 30 seconds
opatchauto failed with error code 42

Error 2: patch apply again failed, 
Cause: Patch meta data files are missing 
Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist. 

Solution: we can continue patch apply with two option 
Solution 1 (Right way to Apply Patch): Patch was successfully applied on node1, Copy patch directory from node1
1.Take backup of 31281355 folder and copy to node 2 and untar file 
Node 1: tar -cvf 31281355_node1.tar /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355
Node2: 
cd /u01/app/19.7.0.0/grid/inventory/oneoffs/
tar -xvf 31281355_node1.tar

Rollback failed patch:
root@racdbnode02~ # opatchauto rollback /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 22:49:45 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_10-49-49PM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_10-50-04PM.log
The id for this session is QLI7

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start rolling back binary patch on home /u01/app/19.7.0.0/grid

Binary patch rolled back successfully on home /u01/app/19.7.0.0/grid

Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid

OPatchAuto successful.
--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:
Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY rolled back:
Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_22-51-53PM_1.log
OPatchauto session completed at Sun Oct  4 23:01:53 2020
Time taken to complete the session 12 minutes, 8 seconds

[grid@racdbnode02 ~]$ opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
30869156;Database Release Update : 19.7.0.0.200414 (30869156)
OPatch succeeded.

Apply Patch:
root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 23:03:34 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_11-03-38PM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_11-03-53PM.log
The id for this session is FEF8

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start applying binary patch on home /u01/app/19.7.0.0/grid
Binary patch applied successfully on home /u01/app/19.7.0.0/grid

Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_23-05-44PM_1.log
OPatchauto session completed at Sun Oct  4 23:20:59 2020
Time taken to complete the session 17 minutes, 25 seconds

[grid@racdbnode02 ~]$ opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
OPatch succeeded.

Solution 2 : 
Copy Missed meta datafile from patch staging folder and applly patch 
1.mkdir -p /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config

cd /u01/app/19c_Software/31305339/31281355/etc/config
[grid@racdbnode02 config]$ ls -ltr
total 2912
-rwxrwxrwx 1 grid oinstall  282218 Jul 10 05:23 inventory.xml
-rwxrwxrwx 1 grid oinstall 2697765 Jul 10 05:23 actions.xml

[grid@racdbnode02 config]$ cp * /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config

2.Restart patch apply with resume option: 
root@racdbnode02 ~ # opatchauto resume 

OPatchauto session is initiated at Sun Oct  4 03:47:42 2020
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-47-45AM.log
Resuming existing session with id ST84

Start applying binary patch on home /u01/app/19.7.0.0/grid
Binary patch applied successfully on home /u01/app/19.7.0.0/grid

Checking shared status of home.....

Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid

OPatchAuto successful
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:
Patch: /u01/app/19c_Software/31305339/31281355
Reason: This patch is already been applied, so not going to apply again.

==Following patches were SUCCESSFULLY applied:
Patch: /u01/app/19c_Software/31305339/31304218
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

Patch: /u01/app/19c_Software/31305339/31305087
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

Patch: /u01/app/19c_Software/31305339/31335188
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

OPatchauto session completed at Sun Oct  4 03:58:33 2020
Time taken to complete the session 10 minutes, 51 seconds
root@racdbnode02 ~ #


[grid@ racdbnode02 ~]$ opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
OPatch succeeded.


Patch apply will completed successfully by skipping previously failed patch( 31281355) 
it misleads us by showing patch details in Opatch lspatches and also opatch lsinventory 

When we check full version details from v$instance or while connecting to sqlplus we can observer it's still in previous version 

select INSTANCE_NAME,host_name,status,logins,version,VERSION_FULL, to_char(STARTUP_TIME,'DD/MM/YYYY HH24:MI:SS') "STARTUP_TIME" from gv$instance;
INSTANCE_NAME      HOST_NAME          STATUS       LOGINS     VERSION           VERSION_FULL      STARTUP_TIME
----------------  ----------------- ------------ ---------- ----------------- ----------------- -------------------
+ASM2             racdbnode02         OPEN         ALLOWED    19.0.0.0.0        19.7.0.0.0        04/10/2020 06:41:00
+ASM1             racdbnode01         OPEN         ALLOWED    19.0.0.0.0        19.8.0.0.0        04/10/2020 06:41:00


We can also observer it while connecting to db with sqlplus “/as sysdba”

[grid@ racdbnode02 config]$ sqlplus “/as sysdba”
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 5 21:56:36 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0

SQL>

Note: Right way to do is copy from successfully patch applied node and rollback patch and reapply 

No comments:

Post a Comment