Upgrade AHF to latest Version (Log4j Vulnerability )

Please find the below steps to upgrade AHF to latest version 

Download latest AHF version and copy to db host  

Unzip software:

[root@racnode01 ahf_stg]# unzip -oq  AHF-LINUX_v21.3.4.zip

run setup command to upgrade:

[root@racnode01 ahf_stg]# ./ahf_setup 

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_213400_50508_2021_12_28-20_47_27.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 21.3.4 Build Date: 202112151432

AHF is already installed at /opt/oracle.ahf

Installed AHF Version: 20.2.2 Build Date: 202008260643

Do you want to upgrade AHF [Y]|N : Y

Upgrading /opt/oracle.ahf

Shutting down AHF Services

Stopped OSWatcher

Nothing to do !

Shutting down TFA

Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.

Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.

. . . . . 

. . . 

Successfully shutdown TFA..

Starting AHF Services

Starting TFA..

Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.

Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.

Waiting up to 100 seconds for TFA to be started..

. . . . . 

Successfully started TFA Process..

. . . . . 

TFA Started and listening for commands

No new directories were added to TFA

Directory /app/grid/crsdata/racnode01/trace/chad was already added to TFA Directories.

INFO: Starting orachk scheduler in background. Details for the process can be found at /app/grid/oracle.ahf/data/racnode01/diag/orachk/compliance_start_281221_205029.log

Do you want AHF to store your My Oracle Support Credentials for Automatic Upload ? Y|[N] : N

AHF is successfully upgraded to latest version

.---------------------------------------------------------------------.

| Host          | TFA Version | TFA Build ID         | Upgrade Status |

+---------------+-------------+----------------------+----------------+

| racnode01 |  21.3.4.0.0 | 21340020211215143236 | UPGRADED       |

'---------------+-------------+----------------------+----------------'

Moving /tmp/ahf_install_213400_50508_2021_12_28-20_47_27.log to /app/grid/oracle.ahf/data/racnode01/diag/ahf/

[root@racnode01 ahf_stg]# tfactl status

.-----------------------------------------------------------------------------------------------------.

| Host          | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |

+---------------+---------------+-------+------+------------+----------------------+------------------+

| racnode01 | RUNNING       | 58382 | 5000 | 21.3.4.0.0 | 21340020211215143236 | COMPLETE         |

'---------------+---------------+-------+------+------------+----------------------+------------------'

Uninstalling Oracle Autonomous Health Framework

As a root user

#ahfctl uninstall -deleterepo -silent

Starting AHF Uninstall

NOTE : Uninstalling will delete the repository as well since Install type is GI

AHF will be uninstalled on: 

racnode02

Stopping AHF service on local node racnode02...

Stopping TFA Support Tools...

Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.

Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.

Removing AHF setup on racnode02:

Removing /etc/rc.d/rc0.d/K17init.tfa

Removing /etc/rc.d/rc1.d/K17init.tfa

Removing /etc/rc.d/rc2.d/K17init.tfa

Removing /etc/rc.d/rc4.d/K17init.tfa

Removing /etc/rc.d/rc6.d/K17init.tfa

Removing /etc/init.d/init.tfa...

Removing /etc/systemd/system/oracle-tfa.service...

Removing /u01/app/19.8.0.0/grid/bin/tfactl...

Removing /opt/oracle.ahf/rpms

Removing /opt/oracle.ahf/jre

Removing /opt/oracle.ahf/common

Removing /opt/oracle.ahf/bin

Removing /opt/oracle.ahf/python

Removing /opt/oracle.ahf/analyzer

Removing /opt/oracle.ahf/tfa

Removing /opt/oracle.ahf/orachk

Removing /opt/oracle.ahf/ahf

Removing /app/grid/oracle.ahf/data/racnode02

Removing /opt/oracle.ahf/install.properties

Removing /app/grid/oracle.ahf/data/repository

Removing /app/grid/oracle.ahf/data

Removing /app/grid/oracle.ahf

Removing AHF Home : /opt/oracle.ahf

Removing /sys/fs/cgroup/cpu/oratfagroup/

ERROR OGG-01117 Oracle GoldenGate Performance Metrics Server for Oracle: Received signal: Segmentation violation

Pmsrvr receives SIGSEGV and crashes after recreating datastore.

we can see below error on ggserr.log 

ERROR   OGG-01117  Oracle GoldenGate Performance Metrics Server for Oracle:  Received signal: Segmentation violation

 

Cause:

Empty dirbdb directory under GG_HOME existed when recreating datastore.

pmsrvr does not initialize the datastore completely if <GG_HOME>/dirbdb/ exists even if the directory is empty.

 

Solution:

stop jagent

stop *

stop manager

stop pmsrvr

GGSCI 14> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED                                           

JAGENT      STOPPED                                          

PMSRVR      STOPPED                                           

EXTRACT     STOPPED     PUMP1      00:00:00      00:00:53    

EXTRACT     STOPPED     EXT1       00:00:02      00:00:53    

EXTRACT     STOPPED     EXT2       00:00:00      00:00:51    

REPLICAT    STOPPED     REPLG1     00:00:00      00:00:46    

remove or mv dirbdb with backup name

mv <GG_HOME>/dirbdb/ <GG_HOME>/dirbdb_org

Start GG Process:

GGSCI 15> start pmsrvr -initds

PMSRVR started

GGSCI 16> start manager

Manager started.

GGSCI 17> start *

Sending START request to MANAGER ...

EXTRACT PUMP1 starting

Sending START request to MANAGER ...

EXTRACT EXT1 starting

Sending START request to MANAGER ...

EXTRACT EXT2 starting

Sending START request to MANAGER ...

REPLICAT REPLG1 starting

GGSCI 18> start jagent

Sending START request to MANAGER ...

GGCMD JAGENT starting

GGSCI 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           

JAGENT      RUNNING                                           

PMSRVR      RUNNING                                           

EXTRACT     RUNNING     PUMP1      00:00:00      00:00:09    

EXTRACT     RUNNING     EXT1      00:00:03      00:00:07    

EXTRACT     RUNNING     EXT2     00:00:02      00:00:07    

REPLICAT    RUNNING     REPLG1     00:00:03      00:00:00   

 

Recreate datastore after delete old one helped to start PMSRVR without errors

Jagent unable to start PMSRVR , Stop all polling tasks. Wait for 9 second(s) and retry again. Due to Connection refused

Unable to start PMSRVR in golden gate 19.1.0.0.4

Got below error;
Fail to get Web Service Port, java.lang.Exception: PM Server is down

[2021-12-06T23:48:36.361-08:00] [JAGENT] [ERROR] [OGGMON-20348] [com.goldengate.monitor.jagent.jmx.ManagerConnectionKeeperThread] [tid: ManagerConnectionKeeper] [ecid: 0000NqJr8243z0GayxaeMG1Xfl69000001,0] Failed to establish a connection to the OGG Core Web Services-Host:HOSTIP Port:9,004. Stop all polling tasks. Wait for 9 second(s) and retry again. Due to Connection refused (Connection refused)

Cause:

There are multiple GoldenGate installations on the same host.

Solution:

cd $OGG_HOME/dirprm

ls -ltr pmsrvr-config.dat

Backup pmsrvr-config.dat, then change port <Port> to a port available.

Check file detais:

/u01/gg/dirprm> head pmsrvr-config.dat

{"config":{"network":{"serviceListeningPort":9004,"ipACL":[{"permission":"allow","address":"ANY"}]},"legacyProtocolEnabled":false,"asynchronousOperationEnabled":true,"defaultSynchronousWait":30,"workerThreadCount":5,"taskManagerEnabled":true,"csrfTokenProtectionEnabled":true,"csrfHeaderProtectionEnabled":false,"authorizationEnabled":false,"authorizationDetails":{"sessionDurationSecs":3600,"useMovingExpirationWindow":false,"movingExpirationWindowSecs":900,"common":{"allow":["Digest","x-Cert"],"customAuthorizationEnabled":true}},"security":false,"securityDetails":{"network":{"common":{"id":"OracleSSL","fipsEnabled":false,"fipsModulePath":"<not-set>"},"inbound":{"role":"server","authMode":"clientOptional_server","wrl":"file:./etc/ssl/server","wrlPassword":"","protocolVersion":"1_2","cipherSuites":"^((?!anon|RC4|NULL|3DES).)*$","crlEnabled":false,"crlStore":"file:./etc/ssl/crls.pem","certACL":[{"permission":"allow","name":"ANY"}],"sessionCachingEnabled":true,"sessionCacheDetails":{"limit":20480,"timeoutSecs":1800}},"outbound":{"role":"client","authMode":"client_server","wrl":"file:./etc/ssl/client","wrlPassword":"","protocolVersion":"1_2_Or_1_1_Or_1_0_Or_3_0","cipherSuites":"^.*$","crlEnabled":false,"crlStore":"file:./etc/ssl/crls.pem","certACL":[{"permission":"allow","name":"ANY"}],"sessionCachingEnabled":false}}},"hstsEnabled":true,"hstsDetails":"max-age=31536000;includeSubDomains","contentUrlRewrite":[{"srcUrl":"/index.html","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/$","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/content$","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/content/$","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/content/","targetUrl":"/services/v2/content/"},{"srcUrl":"/services/v2/content$","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/services/v2/content/$","targetUrl":"/services/v2/content/index.html"},{"srcUrl":"/css$","targetUrl":"/services/v2/content/css/"},{"srcUrl":"/css/$","targetUrl":"/services/v2/content/css/"},{"srcUrl":"/css/","targetUrl":"/services/v2/content/css/"},{"srcUrl":"/js$","targetUrl":"/services/v2/content/js/"},{"srcUrl":"/js/$","targetUrl":"/services/v2/content/js/"},{"srcUrl":"/js/","targetUrl":"/services/v2/content/js/"},{"srcUrl":"/favicon.ico$","targetUrl":"/services/v2/content/css/images/favicon.ico"}],"loadExtractHandler":true,"loadReplicatHandler":true,"loadDSHandler":true,"UDPReceiveBufferSize":10485760,"CollectorWaitInitTime":100,"CollectorWaitIncrement":100,"CollectorWaitMax":1500,"CollectorThreadStack":1048576,"CollectorWorkerThreads":5,"CollectorWorkerQueueLimit":10000,"MonitorTaskHBTimeout":10000,"MonitorTaskFrequency":1000,"DataStoreMemoryModel":"PrivateMemory","DataStoreSharedKey":0,"DataStoreType":"BDB","DataStoreMaxAge":0,"DataStoreMaxDBs":5000,"RepoDatastorePath":"","MessageLimit":500,"StatusChangeLimit":500}}[oracle@prodhost] 

/u01/gg/dirprm> vi pmsrvr-config.dat

Modify port from 9004 to 5504

Started jagent, jagent able to start PMSRVR process without any issue after changing port 

GGSCI> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           

JAGENT      RUNNING                                           

PMSRVR      RUNNING    

Install OCI CLI tool on linux machine

Install oci cli using yum 

[ds4docker]# yum install python36-oci-cli

Loaded plugins: langpacks, ulninfo

Resolving Dependencies

--> Running transaction check

---> Package python36-oci-cli.noarch 0:2.26.3-1.el7 will be updated

---> Package python36-oci-cli.noarch 0:3.3.1-1.el7 will be an update

--> Processing Dependency: python36-oci-sdk = 2.51.0 for package: python36-oci-cli-3.3.1-1.el7.noarch

--> Processing Dependency: python36-arrow >= 1.0.0 for package: python36-oci-cli-3.3.1-1.el7.noarch

--> Running transaction check

---> Package python36-arrow.noarch 0:0.17.0-1.0.2.el7 will be updated

---> Package python36-arrow.noarch 0:1.1.1-1.el7 will be an update

--> Processing Dependency: python36-typing-extensions for package: python36-arrow-1.1.1-1.el7.noarch

---> Package python36-oci-sdk.x86_64 0:2.43.0-1.el7 will be updated

---> Package python36-oci-sdk.x86_64 0:2.51.0-1.el7 will be an update

--> Processing Dependency: python36-circuitbreaker < 2 for package: python36-oci-sdk-2.51.0-1.el7.x86_64

--> Processing Dependency: python36-circuitbreaker >= 1.3.1 for package: python36-oci-sdk-2.51.0-1.el7.x86_64

--> Running transaction check

---> Package python36-circuitbreaker.noarch 0:1.3.1-1.el7 will be installed

---> Package python36-typing-extensions.noarch 0:3.7.4.2-1.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================

 Package                    Arch      Version     Repository     Size

===========================================================================================================

Updating:

 python36-oci-cli           noarch     3.3.1-1.el7               ol7_oci_included     12 M

Installing for dependencies:

 python36-circuitbreaker    noarch    1.3.1-1.el7               ol7_oci_included     8.1 k

 python36-typing-extensions noarch    3.7.4.2-1.el7              ol7_oci_included    2 k

Updating for dependencies:

 python36-arrow             noarch    1.1.1-1.el7               ol7_oci_included     115 k

 python36-oci-sd            x86_64    2.51.0-1.el7              ol7_oci_included     18 M

Transaction Summary

============================================================================================================

Install             ( 2 Dependent packages)

Upgrade  1 Package  (+2 Dependent packages)

Total download size: 30 M

Is this ok [y/d/N]: y

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

(1/5): python36-circuitbreaker-1.3.1-1.el7.noarch.rpm                                                                                                               | 8.1 kB  00:00:00     

(2/5): python36-arrow-1.1.1-1.el7.noarch.rpm                                                                                                                        | 115 kB  00:00:00     

(3/5): python36-oci-cli-3.3.1-1.el7.noarch.rpm                                                                                                                      |  12 MB  00:00:00     

(4/5): python36-typing-extensions-3.7.4.2-1.el7.noarch.rpm                                                                                                          |  42 kB  00:00:00     

(5/5): python36-oci-sdk-2.51.0-1.el7.x86_64.rpm                                                                                                                     |  18 MB  00:00:00     

---------------------------------------------------------------------------------------------------------------

Total                               57 MB/s |  30 MB  00:00:00     

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : python36-circuitbreaker-1.3.1-1.el7.noarch  1/8 

  Updating   : python36-oci-sdk-2.51.0-1.el7.x86_64        2/8 

  Installing : python36-typing-extensions-3.7.4.2-1.el7.noarch 3/8 

  Updating   : python36-arrow-1.1.1-1.el7.noarch       4/8 

  Updating   : python36-oci-cli-3.3.1-1.el7.noarch  5/8 

  Cleanup    : python36-oci-cli-2.26.3-1.el7.noarch  6/8 

  Cleanup    : python36-arrow-0.17.0-1.0.2.el7.noarch 7/8 

  Cleanup    : python36-oci-sdk-2.43.0-1.el7.x86_64 8/8 

  Verifying  : python36-arrow-1.1.1-1.el7.noarch 1/8 

  Verifying  : python36-typing-extensions-3.7.4.2-1.el7.noarch 2/8 

  Verifying  : python36-oci-sdk-2.51.0-1.el7.x86_64  3/8 

  Verifying  : python36-circuitbreaker-1.3.1-1.el7.noarch 4/8 

  Verifying  : python36-oci-cli-3.3.1-1.el7.noarch   5/8 

  Verifying  : python36-arrow-0.17.0-1.0.2.el7.noarch 6/8 

  Verifying  : python36-oci-sdk-2.43.0-1.el7.x86_64  7/8 

  Verifying  : python36-oci-cli-2.26.3-1.el7.noarch  8/8 

Dependency Installed:

  python36-circuitbreaker.noarch 0:1.3.1-1.el7                    python36-typing-extensions.noarch 0:3.7.4.2-1.el7                                             

Updated:

  python36-oci-cli.noarch 0:3.3.1-1.el7                               

Dependency Updated:

  python36-arrow.noarch 0:1.1.1-1.el7                            python36-oci-sdk.x86_64 0:2.51.0-1.el7                                                       

Complete!

Set up oci config :

[root@oracle-zdmconfig .oci]# oci setup config

 This command provides a walkthrough of creating a valid CLI config file.

 The following links explain where to find the information required by this

    script:

    User API Signing Key, OCID and Tenancy OCID:

        https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#Other

    Region:      https://docs.cloud.oracle.com/Content/General/Concepts/regions.htm

General config documentation:  https://docs.cloud.oracle.com/Content/API/Concepts/sdkconfig.htm

Enter a location for your config [/root/.oci/config]: /root/.oci/config

Enter a user OCID: 

Enter a tenancy OCID: 

Enter a region by index or name(e.g.

1: ap-chiyoda-1, 2: ap-chuncheon-1, 3: ap-hyderabad-1, 4: ap-ibaraki-1, 5: ap-melbourne-1,

6: ap-mumbai-1, 7: ap-osaka-1, 8: ap-seoul-1, 9: ap-singapore-1, 10: ap-sydney-1,

11: ap-tokyo-1, 12: ca-montreal-1, 13: ca-toronto-1, 14: eu-amsterdam-1, 15: eu-frankfurt-1,

16: eu-marseille-1, 17: eu-zurich-1, 18: il-jerusalem-1, 19: me-dubai-1, 20: me-jeddah-1,

21: sa-santiago-1, 22: sa-saopaulo-1, 23: sa-vinhedo-1, 24: uk-cardiff-1, 25: uk-gov-cardiff-1,

26: uk-gov-london-1, 27: uk-london-1, 28: us-ashburn-1, 29: us-gov-ashburn-1, 30: us-gov-chicago-1,

31: us-gov-phoenix-1, 32: us-langley-1, 33: us-luke-1, 34: us-phoenix-1, 35: us-sanjose-1): us-sanjose-1

Do you want to generate a new API Signing RSA key pair? (If you decline you will be asked to supply the path to an existing key.) [Y/n]: n

Enter the location of your API Signing private key file: /opt/ds4docker/private_key

Fingerprint: 

Config written to /root/.oci/config

    If you haven't already uploaded your API Signing public key through the

    console, follow the instructions on the page linked below in the section

    'How to upload the public key':

        https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#How2

rman backup failed with error RMAN-03009: failure of REFAF command on 19c standby database

Production database backups which are running on standby side are failing with errors, 

lets troubleshoot rman failure and fix  

RMAN Backup Status:

START_TIME       END_TIME         OUTPUT_DEVICE_TYP INPUT_TYPE   STATUS       TIME_TAKEN_D INPUT_BYTES_ OUTPUT_BYTES

---------------- ---------------- ----------------- ------------ ------------ ------------ ------------ ------------

11/28/21 20:01   11/28/21 21:51   DISK              DB INCR      FAILED       01:50:12       643.48G      555.74G

Check rman log file:

Starting backup at Nov 28 2021 21:51:31

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=7524 device type=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: SID=8229 device type=DISK

channel ORA_DISK_1: starting datafile copy

copying current control file

output file name=/DB_BACKUP/prddb211/incr/post_bkup_.bak tag=TAG20211128T215131 RECID=75 STAMP=1089841892

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01

Finished backup at Nov 28 2021 21:51:32

Starting Control File and SPFILE Autobackup at Nov 28 2021 21:51:32

piece handle=/DB_BACKUP/prddb211/arch/c-1132445964-20211128-17 comment=NONE

Finished Control File and SPFILE Autobackup at Nov 28 2021 21:51:36

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of REFAF command on default channel at 11/28/2021 21:51:36

ORA-01804: failure to initialize timezone information

Note: as per log , error related to timezone information

Lets verify timezone version on both primary  and standby databases 

TZ_VERSION on Source and target:

SQL> SELECT tz_version FROM registry$database;

TZ_VERSION

----------

        35

Timezone version is same across primary & standby databases, 

Verify patch details on primary  and standby servers

On Primary db:

$opatch lspatches

31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

OPatch succeeded.

On Standby db:

$ opatch lspatches

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

Note: Timezone patch not applied on standby side, we have to match patch level on standby with primary 

Apply timezone patch on standby

Patch apply steps on Standby:

Copy patch p31335037_190000_Linux-x86-64.zip 

unzip p31335037_190000_Linux-x86-64.zip

$ cd 31335037/

Patch Precheck:

$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

Oracle Interim Patch Installer version 12.2.0.1.27

Copyright (c) 2021, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/oracle/product/19.8.0_64

Central Inventory : /app/oraInventory

   from           : /u01/app/oracle/product/19.8.0_64/oraInst.loc

OPatch version    : 12.2.0.1.27

OUI version       : 12.2.0.7.0

Log file location : /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-25PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

Shutdown standby db before patch apply and start db , mrp after patch 

$ opatch apply 

Oracle Interim Patch Installer version 12.2.0.1.27

Copyright (c) 2021, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/19.8.0_64

Central Inventory : /app/oraInventory

   from           : /u01/app/oracle/product/19.8.0_64/oraInst.loc

OPatch version    : 12.2.0.1.27

OUI version       : 12.2.0.7.0

Log file location : /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-55PM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   31335037  

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Backing up files...

Applying interim patch '31335037' to OH '/u01/app/oracle/product/19.8.0_64'

Patching component oracle.oracore.rsf, 19.0.0.0.0...

Patch 31335037 successfully applied.

Log file location: /u01/app/oracle/product/19.8.0_64/cfgtoollogs/opatch/opatch2021-11-28_22-49-55PM_1.log

OPatch succeeded.

Standby patch list matched with primary side now

$ opatch lspatches

31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

OPatch succeeded.

rman backup completed successfully after applying timezone patch on standby database

START_TIME       END_TIME         OUTPUT_DEVICE_TYP INPUT_TYPE   STATUS       TIME_TAKEN_D INPUT_BYTES_ OUTPUT_BYTES

---------------- ---------------- ----------------- ------------ ------------ ------------ ------------ ------------

11/28/21 20:01   11/28/21 21:51   DISK              DB INCR      FAILED       01:50:12       643.48G      555.74G

11/28/21 23:00   11/28/21 23:07   DISK              DB INCR      COMPLETED    00:07:14        49.67G       29.97G