Production database backups which are running on standby side are failing with errors,
lets troubleshoot rman failure and fix
RMAN Backup Status:
START_TIME END_TIME OUTPUT_DEVICE_TYP INPUT_TYPE STATUS TIME_TAKEN_D INPUT_BYTES_ OUTPUT_BYTES
---------------- ---------------- ----------------- ------------ ------------ ------------ ------------ ------------
11/28/21 20:01 11/28/21 21:51 DISK DB INCR FAILED 01:50:12 643.48G 555.74G
Check rman log file:
Starting backup at Nov 28 2021 21:51:31
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=7524 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=8229 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=/DB_BACKUP/prddb211/incr/post_bkup_.bak tag=TAG20211128T215131 RECID=75 STAMP=1089841892
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at Nov 28 2021 21:51:32
Starting Control File and SPFILE Autobackup at Nov 28 2021 21:51:32
piece handle=/DB_BACKUP/prddb211/arch/c-1132445964-20211128-17 comment=NONE
Finished Control File and SPFILE Autobackup at Nov 28 2021 21:51:36
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of REFAF command on default channel at 11/28/2021 21:51:36
ORA-01804: failure to initialize timezone information
Note: as per log , error related to timezone information
Lets verify timezone version on both primary and standby databases
TZ_VERSION on Source and target:
SQL> SELECT tz_version FROM registry$database;
TZ_VERSION
----------
35
Timezone version is same across primary & standby databases,
Verify patch details on primary and standby servers
On Primary db:
$opatch lspatches
31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A
32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)
32904851;Database Release Update : 19.12.0.0.210720 (32904851)
OPatch succeeded.
On Standby db:
$ opatch lspatches
32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)
32904851;Database Release Update : 19.12.0.0.210720 (32904851)
Note: Timezone patch not applied on standby side, we have to match patch level on standby with primary
Apply timezone patch on standby
Patch apply steps on Standby:
Copy patch p31335037_190000_Linux-x86-64.zip
unzip p31335037_190000_Linux-x86-64.zip
$ cd 31335037/
Patch Precheck:
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.27
Copyright (c) 2021, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.8.0_64
Central Inventory : /app/oraInventory
from : /u01/app/oracle/product/19.8.0_64/oraInst.loc
OPatch version : 12.2.0.1.27
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-25PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
Shutdown standby db before patch apply and start db , mrp after patch
$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.27
Copyright (c) 2021, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.8.0_64
Central Inventory : /app/oraInventory
from : /u01/app/oracle/product/19.8.0_64/oraInst.loc
OPatch version : 12.2.0.1.27
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-55PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 31335037
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Backing up files...
Applying interim patch '31335037' to OH '/u01/app/oracle/product/19.8.0_64'
Patching component oracle.oracore.rsf, 19.0.0.0.0...
Patch 31335037 successfully applied.
Log file location: /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-55PM_1.log
OPatch succeeded.
Standby patch list matched with primary side now
$ opatch lspatches
31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A
32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)
32904851;Database Release Update : 19.12.0.0.210720 (32904851)
OPatch succeeded.
rman backup completed successfully after applying timezone patch on standby database
START_TIME END_TIME OUTPUT_DEVICE_TYP INPUT_TYPE STATUS TIME_TAKEN_D INPUT_BYTES_ OUTPUT_BYTES
---------------- ---------------- ----------------- ------------ ------------ ------------ ------------ ------------
11/28/21 20:01 11/28/21 21:51 DISK DB INCR FAILED 01:50:12 643.48G 555.74G
11/28/21 23:00 11/28/21 23:07 DISK DB INCR COMPLETED 00:07:14 49.67G 29.97G