Permissions granted to the object storage service principal "" to this bucket are insufficient.

Error while copy object between tenancy in a region 

Permissions granted to the object storage service principal "objectstorage-ap-mumbai-1" to this bucket are insufficient.

we have to create a policy that authorizes the service in the specified region to manage Object Storage namespaces, buckets, and their associated objects in all compartments in the tenancy:

Create Policy:

Open the navigation menu,Under Governance and Administration. 

Click Identity and choose Policies 

Click on Create Policy

Policy Created now..let try object copy 

Click ok Copy Object

 now work request is submitted without error

Work Request is completed , now we can see object in target bucket 

ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage

rman backup failed with below error

Error:

channel c4: backup set complete, elapsed time: 00:00:11

RMAN-03009: failure of backup command on c2 channel at 10/20/2020 01:00:30

ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage

continuing other job steps, job failed will not be re-run

channel c1: finished piece 1 at Oct 20 1 01:04:35

piece handle=/backup/r/full/full_PRDDB31_1086397215_50020_1 tag=TAG2013401T10017 comment=NONE

channel c1: backup set complete, elapsed time: 00:04:20

released channel: c1

released channel: c2

released channel: c3

released channel: c4

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on c2 channel at 10/20/2020 01:00:30

ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage

Cause: Backup location not accessible from all cluster nodes 

Solution:

Mount  /backup  from all nodes and check permissions on backup folder 

Find master node in Oracle 19c Cluster

we can find Oracle cluster master node using below commands 

1.ocrconfig -manualbackup

2.grep "master node number" from ocssd.log 

3.oclumon manage -get master  

4.select MASTER_NODE from v$ges_resource;

AUTO_SAMPLE_SIZE in DBMS_STATS

SYS.DBMS_STATS.AUTO_SAMPLE_SIZE is a parameter that can be used in the Oracle DBMS_STATS package to specify that Oracle should automatically determine the appropriate sample size when gathering statistics on a database object.

When AUTO_SAMPLE_SIZE is used, Oracle analyzes the object being sampled and selects an appropriate sample size based on its characteristics. This can help to ensure that accurate statistics are gathered while minimizing the amount of time and system resources required to perform the operation.

It is highly recommended that from Oracle Database 11g on wards that the default AUTO_SAMPLE_SIZE is used for ESTIMATE_PRECENT. This is especially important because the newer histogram types (HYBRID and Top-Frequency) can only be created if an auto sample size is used.
 

Here is an example of how to use SYS.DBMS_STATS.AUTO_SAMPLE_SIZE with the DBMS_STATS.GATHER_TABLE_STATS procedure to collect statistics for a table named "EMPLOYEES":

BEGIN
   DBMS_STATS.GATHER_TABLE_STATS(
      ownname      => 'HR',
      tabname      => 'EMPLOYEES',
      estimate_percent => SYS.DBMS_STATS.AUTO_SAMPLE_SIZE,
      degree       => DBMS_STATS.AUTO_DEGREE,
      method_opt   => 'FOR ALL COLUMNS SIZE AUTO',
      cascade      => TRUE);
END;
/

In this example, the estimate_percent parameter is set to SYS.DBMS_STATS.AUTO_SAMPLE_SIZE, indicating that Oracle should automatically determine the appropriate sample size when gathering statistics for the "EMPLOYEES" table. The other parameters used in this example are similar to those used in the previous example I provided.

GI July 2020 RU 19.8.0.0(31305339) patching Issues , resume failed patch with opatchauto

Had couple of issues while applying GI JULY RU 19.8.0.0 patch(31305339) on Cluster Node2  , Patch was successfully applied in node1

We can expect similar error for both grid and db home  while applying RU patch , Solution also same for both grid and db homes 

Error 1:

opatchauto apply failed with /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied) On node 2

grid patch apply on node2:

root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 02:52:32 2020

System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_02-52-36AM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_02-52-49AM.log

The id for this session is ST84

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid

Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid

CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start applying binary patch on home /u01/app/19.7.0.0/grid

Failed while applying binary patches on home /u01/app/19.7.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.

Command failed:  /u01/app/19.7.0.0/grid/OPatch/opatchauto  apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre -persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser -analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser

Command failure output: 

==Following patches FAILED in apply:

Patch: /u01/app/19c_Software/31305339/31281355

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_02-55-30AM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sun Oct  4 03:02:44 2020

Time taken to complete the session 10 minutes, 12 seconds

 opatchauto failed with error code 42

Cause : file permission issue, Compared file permission on both nodes ,they are different 

root@racdbnode02~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml

-rw-r--r-- 1 oracle oinstall 174 Oct  4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml

root@ racdbnode01~ # ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml 

-rw-rw---- 1 oracle oinstall 174 Oct  4 02:38 /app/oraInventory/ContentsXML/oui-patch.xml

Solution for error 1: Change permissions to 660 

root@racdbnode02~ #chmod 660 /app/oraInventory/ContentsXML/oui-patch.xml

root@racdbnode02~ #ls -ltr /app/oraInventory/ContentsXML/oui-patch.xml

-rw-rw---- 1 oracle oinstall 174 Oct  4 03:52 /app/oraInventory/ContentsXML/oui-patch.xml

Restart patch apply with resume option: 

Opatchauto resume will resume previous patch session

root@racdbnode02~ # opatchauto resume 

OPatchauto session is initiated at Sun Oct  4 03:31:20 2020

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-31-23AM.log

Resuming existing session with id ST84

Start applying binary patch on home /u01/app/19.7.0.0/grid

Failed while applying binary patches on home /u01/app/19.7.0.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : racdbnode02->/u01/app/19.7.0.0/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/19.7.0.0/grid, host: racdbnode02.

Command failed:  /u01/app/19.7.0.0/grid/OPatch/opatchauto  apply /u01/app/19c_Software/31305339 -oh /u01/app/19.7.0.0/grid -target_type cluster 

-binary -invPtrLoc /u01/app/19.7.0.0/grid/oraInst.loc -jre /u01/app/19.7.0.0/grid/OPatch/jre 

-persistresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdbnode02_crs_4.ser 

-analyzedresult /u01/app/19.7.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdbnode02_crs_4.ser

Command failure output: 

==Following patches FAILED in apply:

Patch: /u01/app/19c_Software/31305339/31281355

Log: 

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject

Possible causes are:

   ORACLE_HOME/inventory/oneoffs/31281355 is corrupted. PatchObject constructor: 

Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist. 

After fixing the cause of failure Run opatchauto resume

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

OPatchauto session completed at Sun Oct  4 03:31:50 2020

Time taken to complete the session 0 minute, 30 seconds

opatchauto failed with error code 42

Error 2: patch apply again failed, 

Cause: Patch meta data files are missing 

Input file "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/actions" or "/u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config/inventory" does not exist. 

Solution: we can continue patch apply with two option 

Solution 1 (Right way to Apply Patch): Patch was successfully applied on node1, Copy patch directory from node1

1.Take backup of 31281355 folder and copy to node 2 and untar file 

Node 1: tar -cvf 31281355_node1.tar /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355

Node2: 

cd /u01/app/19.7.0.0/grid/inventory/oneoffs/

tar -xvf 31281355_node1.tar

Rollback failed patch:

root@racdbnode02~ # opatchauto rollback /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 22:49:45 2020

System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_10-49-49PM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_10-50-04PM.log

The id for this session is QLI7

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid

Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid

CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start rolling back binary patch on home /u01/app/19.7.0.0/grid

Binary patch rolled back successfully on home /u01/app/19.7.0.0/grid

Starting CRS service on home /u01/app/19.7.0.0/grid

CRS service started successfully on home /u01/app/19.7.0.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racdbnode02

CRS Home:/u01/app/19.7.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY rolled back:

Patch: /u01/app/19c_Software/31305339/31281355

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_22-51-53PM_1.log

OPatchauto session completed at Sun Oct  4 23:01:53 2020

Time taken to complete the session 12 minutes, 8 seconds

[grid@racdbnode02 ~]$ opatch lspatches

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

30869156;Database Release Update : 19.7.0.0.200414 (30869156)

OPatch succeeded.

Apply Patch:

root@racdbnode02~ # opatchauto apply /u01/app/19c_Software/31305339/31281355 -oh /u01/app/19.7.0.0/grid

OPatchauto session is initiated at Sun Oct  4 23:03:34 2020

System initialization log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-04_11-03-38PM.log.

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_11-03-53PM.log

The id for this session is FEF8

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.7.0.0/grid

Patch applicability verified successfully on home /u01/app/19.7.0.0/grid

Bringing down CRS service on home /u01/app/19.7.0.0/grid

CRS service brought down successfully on home /u01/app/19.7.0.0/grid

Start applying binary patch on home /u01/app/19.7.0.0/grid

Binary patch applied successfully on home /u01/app/19.7.0.0/grid

Starting CRS service on home /u01/app/19.7.0.0/grid

CRS service started successfully on home /u01/app/19.7.0.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racdbnode02

CRS Home:/u01/app/19.7.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/19c_Software/31305339/31281355

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_23-05-44PM_1.log

OPatchauto session completed at Sun Oct  4 23:20:59 2020

Time taken to complete the session 17 minutes, 25 seconds

[grid@racdbnode02 ~]$ opatch lspatches

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

OPatch succeeded.

Solution 2 : 

Copy Missed meta datafile from patch staging folder and applly patch 

1.mkdir -p /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config

cd /u01/app/19c_Software/31305339/31281355/etc/config

[grid@racdbnode02 config]$ ls -ltr

total 2912

-rwxrwxrwx 1 grid oinstall  282218 Jul 10 05:23 inventory.xml

-rwxrwxrwx 1 grid oinstall 2697765 Jul 10 05:23 actions.xml

[grid@racdbnode02 config]$ cp * /u01/app/19.7.0.0/grid/inventory/oneoffs/31281355/etc/config

2.Restart patch apply with resume option: 

root@racdbnode02 ~ # opatchauto resume 

OPatchauto session is initiated at Sun Oct  4 03:47:42 2020

Session log file is /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-04_03-47-45AM.log

Resuming existing session with id ST84

Start applying binary patch on home /u01/app/19.7.0.0/grid

Binary patch applied successfully on home /u01/app/19.7.0.0/grid

Checking shared status of home.....

Starting CRS service on home /u01/app/19.7.0.0/grid

CRS service started successfully on home /u01/app/19.7.0.0/grid

OPatchAuto successful

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racdbnode02

CRS Home:/u01/app/19.7.0.0/grid

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /u01/app/19c_Software/31305339/31281355

Reason: This patch is already been applied, so not going to apply again.

==Following patches were SUCCESSFULLY applied:

Patch: /u01/app/19c_Software/31305339/31304218

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

Patch: /u01/app/19c_Software/31305339/31305087

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

Patch: /u01/app/19c_Software/31305339/31335188

Log: /u01/app/19.7.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-04_03-48-08AM_1.log

OPatchauto session completed at Sun Oct  4 03:58:33 2020

Time taken to complete the session 10 minutes, 51 seconds

root@racdbnode02 ~ #

[grid@ racdbnode02 ~]$ opatch lspatches

31281355;Database Release Update : 19.8.0.0.200714 (31281355)

31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)

31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)

31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)

OPatch succeeded.

Patch apply will completed successfully by skipping previously failed patch( 31281355) 

it misleads us by showing patch details in Opatch lspatches and also opatch lsinventory 

When we check full version details from v$instance or while connecting to sqlplus we can observer it's still in previous version 

select INSTANCE_NAME,host_name,status,logins,version,VERSION_FULL, to_char(STARTUP_TIME,'DD/MM/YYYY HH24:MI:SS') "STARTUP_TIME" from gv$instance;

INSTANCE_NAME      HOST_NAME          STATUS       LOGINS     VERSION           VERSION_FULL      STARTUP_TIME

----------------  ----------------- ------------ ---------- ----------------- ----------------- -------------------

+ASM2             racdbnode02         OPEN         ALLOWED    19.0.0.0.0        19.7.0.0.0        04/10/2020 06:41:00

+ASM1             racdbnode01         OPEN         ALLOWED    19.0.0.0.0        19.8.0.0.0        04/10/2020 06:41:00

We can also observer it while connecting to db with sqlplus “/as sysdba”

[grid@ racdbnode02 config]$ sqlplus “/as sysdba”

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 5 21:56:36 2020

Version 19.8.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0

SQL>

Note: Right way to do is copy from successfully patch applied node and rollback patch and reapply 

You (oracle) are not allowed to use this program (crontab)

error while checking crontab details as oracle user

[oracle@dbracnode02 ~]$ crontab -l
You (oracle) are not allowed to use this program (crontab)
See crontab(1) for more information

Check entries in /etc/cron.allow

[root@dbracnode02 ~]# cat /etc/cron.allow


To solve this error you should add oracle or any other users that will use the Crontab into this file as follows.

Add oracle user and save /etc/cron.allow
[root@dbracnode02 ~]# vi /etc/cron.allow
root
oracle
grid

[root@dbracnode02 ~]# cat /etc/cron.allow
root
oracle
grid

Check now:
[oracle@dbracnode02 ~]$ crontab -l
no crontab for oracle