Had couple of issues while applying GI JULY RU 19.8.0.0 patch(31305339) on Cluster Node2 , Patch was successfully applied in node1
We can expect similar error for both grid and db home while applying RU patch , Solution also same for both grid and db homes
Error 1:
opatchauto apply failed with /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied) On node 2
grid patch apply on node2:
root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid
OPatchauto session is initiated at Sun Oct 4 02:52:32 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_02-52-36AM.log.
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_02-52-49AM.log
The id for this session is ST84
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid
Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid
Start applying binary patch on home /u01/app/19.7.0.0/grid
Failed while applying binary patches on home /u01/app/19.7.0.0/grid
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.
Command failed: /u01/app/19.7.0.0/grid/OPatch/opatchauto apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre -persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser -analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_02-55-30AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Sun Oct 4 03:02:44 2020
Time taken to complete the session 10 minutes, 12 seconds
opatchauto failed with error code 42
Cause : file permission issue, Compared file permission on both nodes ,they are different
root@racdbnode02~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml
-rw-r--r-- 1 oracle oinstall 174 Oct 4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml
root@ racdbnode01~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 oracle oinstall 174 Oct 4 02:38 /app/oraInventory/ContentsXML/oui-patch.xml
Solution for error 1: Change permissions to 660
root@racdbnode02~ #chmod 660 /app/oraInventory/ContentsXML/oui-patch.xml
root@racdbnode02~ #ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 oracle oinstall 174 Oct 4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml
Restart patch apply with resume option:
Opatchauto resume will resume previous patch session
root@racdbnode02~ # opatchauto resume
OPatchauto session is initiated at Sun Oct 4 03:31:20 2020
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-31-23AM.log
Resuming existing session with id ST84
Start applying binary patch on home /u01/app/19.7.0.0/grid
Failed while applying binary patches on home /u01/app/19.7.0.0/grid
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.
Command failed: /u01/app/19.7.0.0/grid/OPatch/opatchauto apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster
-binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre
-persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser
-analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /u01/app/19c_Software/31305339/31281355
Log:
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor:
Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist.
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Sun Oct 4 03:31:50 2020
Time taken to complete the session 0 minute, 30 seconds
opatchauto failed with error code 42
Error 2: patch apply again failed,
Cause: Patch meta data files are missing
Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist.
Solution: we can continue patch apply with two option
Solution 1 (Right way to Apply Patch): Patch was successfully applied on node1, Copy patch directory from node1
1.Take backup of 31281355 folder and copy to node 2 and untar file
Node 1: tar -cvf 31281355_node1.tar /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355
Node2:
cd /u01/app/19.7.0.0/grid/inventory/oneoffs/
tar -xvf 31281355_node1.tar
Rollback failed patch:
root@racdbnode02~ # opatchauto rollback /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid
OPatchauto session is initiated at Sun Oct 4 22:49:45 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_10-49-49PM.log.
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_10-50-04PM.log
The id for this session is QLI7
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid
Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid
Start rolling back binary patch on home /u01/app/19.7.0.0/grid
Binary patch rolled back successfully on home /u01/app/19.7.0.0/grid
Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY rolled back:
Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_22-51-53PM_1.log
OPatchauto session completed at Sun Oct 4 23:01:53 2020
Time taken to complete the session 12 minutes, 8 seconds
[grid@racdbnode02 ~]$ opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
30869156;Database Release Update : 19.7.0.0.200414 (30869156)
OPatch succeeded.
Apply Patch:
root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid
OPatchauto session is initiated at Sun Oct 4 23:03:34 2020
System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_11-03-38PM.log.
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_11-03-53PM.log
The id for this session is FEF8
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid
Patch applicability verified successfully on home /u01/app/19.7.0.0/grid
Bringing down CRS service on home /u01/app/19.7.0.0/grid
CRS service brought down successfully on home /u01/app/19.7.0.0/grid
Start applying binary patch on home /u01/app/19.7.0.0/grid
Binary patch applied successfully on home /u01/app/19.7.0.0/grid
Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /u01/app/19c_Software/31305339/31281355
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_23-05-44PM_1.log
OPatchauto session completed at Sun Oct 4 23:20:59 2020
Time taken to complete the session 17 minutes, 25 seconds
[grid@racdbnode02 ~]$ opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
OPatch succeeded.
Solution 2 :
Copy Missed meta datafile from patch staging folder and applly patch
1.mkdir -p /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config
cd /u01/app/19c_Software/31305339/31281355/etc/config
[grid@racdbnode02 config]$ ls -ltr
total 2912
-rwxrwxrwx 1 grid oinstall 282218 Jul 10 05:23 inventory.xml
-rwxrwxrwx 1 grid oinstall 2697765 Jul 10 05:23 actions.xml
[grid@racdbnode02 config]$ cp * /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config
2.Restart patch apply with resume option:
root@racdbnode02 ~ # opatchauto resume
OPatchauto session is initiated at Sun Oct 4 03:47:42 2020
Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-47-45AM.log
Resuming existing session with id ST84
Start applying binary patch on home /u01/app/19.7.0.0/grid
Binary patch applied successfully on home /u01/app/19.7.0.0/grid
Checking shared status of home.....
Starting CRS service on home /u01/app/19.7.0.0/grid
CRS service started successfully on home /u01/app/19.7.0.0/grid
OPatchAuto successful
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:racdbnode02
CRS Home:/u01/app/19.7.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /u01/app/19c_Software/31305339/31281355
Reason: This patch is already been applied, so not going to apply again.
==Following patches were SUCCESSFULLY applied:
Patch: /u01/app/19c_Software/31305339/31304218
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log
Patch: /u01/app/19c_Software/31305339/31305087
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log
Patch: /u01/app/19c_Software/31305339/31335188
Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log
OPatchauto session completed at Sun Oct 4 03:58:33 2020
Time taken to complete the session 10 minutes, 51 seconds
root@racdbnode02 ~ #
[grid@ racdbnode02 ~]$ opatch lspatches
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
OPatch succeeded.
Patch apply will completed successfully by skipping previously failed patch( 31281355)
it misleads us by showing patch details in Opatch lspatches and also opatch lsinventory
When we check full version details from v$instance or while connecting to sqlplus we can observer it's still in previous version
select INSTANCE_NAME,host_name,status,logins,version,VERSION_FULL, to_char(STARTUP_TIME,'DD/MM/YYYY HH24:MI:SS') "STARTUP_TIME" from gv$instance;
INSTANCE_NAME HOST_NAME STATUS LOGINS VERSION VERSION_FULL STARTUP_TIME
---------------- ----------------- ------------ ---------- ----------------- ----------------- -------------------
+ASM2 racdbnode02 OPEN ALLOWED 19.0.0.0.0 19.7.0.0.0 04/10/2020 06:41:00
+ASM1 racdbnode01 OPEN ALLOWED 19.0.0.0.0 19.8.0.0.0 04/10/2020 06:41:00
We can also observer it while connecting to db with sqlplus “/as sysdba”
[grid@ racdbnode02 config]$ sqlplus “/as sysdba”
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 5 21:56:36 2020
Version 19.8.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0
SQL>
Note: Right way to do is copy from successfully patch applied node and rollback patch and reapply
