Thursday 28 February 2019

RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process


Unable to delete archivelog from rman and got error RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
RMAN> delete archivelog sequence 510 thread 2
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=10299 instance=psrinc012 device type=DISK
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+RECOC1/psrinc01/archivelog/2019_02_28/thread_2_seq_510.790.1001376017 thread=2 sequence=510

Verified no standby configured and db cloned from production where golden gate configured,
OGG processes are in present db and they are disabled , Dropped ogg capture since they are not used

SQL> select CAPTURE_NAME, STATUS from dba_capture;
CAPTURE_NAME                   STATUS
------------------------------ --------
OGG$CAP_X4PBTR                 DISABLED
OGG$CAP_TST_PBTR               DISABLED
SQL> exec DBMS_CAPTURE_ADM.DROP_CAPTURE ('OGG$CAP_X4PBTR');
PL/SQL procedure successfully completed.

SQL> exec DBMS_CAPTURE_ADM.DROP_CAPTURE ('OGG$CAP_TST_PBTR');
PL/SQL procedure successfully completed.

SQL> select CAPTURE_NAME,STATUS from dba_capture;
no rows selected

Archive deletion will work fine now:
RMAN> delete archivelog sequence 510 thread 2;
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=10299 instance=psrinc012 device type=DISK
List of Archived Log Copies for database with db_unique_name PSRINC01
=====================================================================
Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
2434    2    510     A 27-FEB-19
        Name: +RECOC1/psrinc01/archivelog/2019_02_28/thread_2_seq_510.790.1001376017
Do you really want to delete the above objects (enter YES or NO)? yes
deleted archived log
archived log file name=+RECOC1/psrinc01/archivelog/2019_02_28/thread_2_seq_510.790.1001376017 RECID=2434 STAMP=1001376036
Deleted 1 objects

Monday 11 February 2019

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

CTSS is in Observer mode due to ntpd running on RAC hosts

[grid@racnode01~]$ crsctl check ctss

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

[grid@racnode02~]$  crsctl check ctss

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

 

Stop crs from both nodes:

root@racnode01:# ./crsctl stop crs

root@racnode02:# ./crsctl stop crs

 

 Stop and disable ntpd service, also remove conf file:

#service ntpd stop

#chkconfig ntpd off

 

# mv /etc/ntp.conf /etc/ntp.conf_bkp

mv: overwrite ‘/etc/ntp.conf_bkp’? y

 

# rm -rf /var/run/ntpd.pid

 

For Linux 7:

[root@~]# systemctl stop chronyd

[root@~]# systemctl disable chronyd

rm '/etc/systemd/system/multi-user.target.wants/chronyd.service'

--Delete or rename /etc/chrony.conf file

[root@~]# mv /etc/chrony.conf  /etc/chrony.conf_old

 

Start crs from both nodes:

root@racnode01:#./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

root@racnode02:#./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

 

Verify CTSS status now. It will be in active mode

[grid@racnode01 ~]$ crsctl check ctss

CRS-4701: The Cluster Time Synchronization Service is in Active mode.

CRS-4702: Offset (in msec): 0

[grid@racnode02 ~]$ crsctl check ctss

CRS-4701: The Cluster Time Synchronization Service is in Active mode.

CRS-4702: Offset (in msec): 0

Unable to start Instance on RAC Node2 with error ORA-01017: invalid username/password


Startup failed with below error:
/u01/app/oracle/product/11.2.0_64/dbs> sqlplus "/as sysdba"
SQL*Plus: Release 11.2.0.4.0 Production on Mon Feb 11 03:50:41 2019
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DB_REDO/PRD31/PARAMETERFILE/spfile.281.999919915'
ORA-17503: ksfdopn:2 Failed to open file +DB_REDO/PRD31/PARAMETERFILE/spfile.281.999919915
ORA-01017: invalid username/password; logon denied
SQL> exit

Note: Database fail to start with ORA-01017 due to wrong group setup (Doc ID 1922050.1)

Verify group setup for oracle user on both nodes:
[grid@dbhost1 bin]$ id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54324(asmdba)

Group setup is different from node1
[grid@dbhost2 bin]$ id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),26049(test)

Correct group id on node 2
root@dbhost2# /usr/sbin/usermod -G oinstall,dba,asmdba  oracle
root@dbhost2# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54324(asmdba)
  
Instance started without any issue after fixing group setup on node2


Wednesday 6 February 2019

Grid Installation Precheck runcluvfy failed Daemon "avahi-daemon" not configured and running

18.3 Cluster precheck failed due to avahi-daemon is running 
we need to stop daemon to proceed further

Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  host2    no                        passed
  host1    no                        passed
   Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  host2    yes                       failed
  host1    yes                       failed

Verifying Daemon "avahi-daemon" not configured and running ...FAILED (PRVG-1360)
#systemctl status avahi-daemon
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
   Loaded: loaded (/usr/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-02-05 13:23:55 PST; 9h ago
 Main PID: 10183 (avahi-daemon)
   Status: "avahi-daemon 0.6.31 starting up."
   CGroup: /system.slice/avahi-daemon.service
           ├─10183 avahi-daemon: running [host1.local]
           └─10187 avahi-daemon: chroot helper
Feb 05 13:24:00 host1.com avahi-daemon[10183]: Joining mDNS multicast group on interface eth0.IPv4 with add....41.
Feb 05 13:24:00 host1.com avahi-daemon[10183]: New relevant interface eth0.IPv4 for mDNS.
Feb 05 13:24:00 host1.com avahi-daemon[10183]: Registering new address record for  on eth0.IPv4.
Feb 05 13:24:02 host1.com avahi-daemon[10183]: Registering new address record for fe80::225:b5ff:fe00:45e o...1.*.
Feb 05 13:24:04 host1.com avahi-daemon[10183]: Joining mDNS multicast group on interface eth1.IPv4 with add....41.
Feb 05 13:24:04 host1.com avahi-daemon[10183]: New relevant interface eth1.IPv4 for mDNS.
Feb 05 13:24:04 host1.com avahi-daemon[10183]: Registering new address record for  on eth1.IPv4.
Feb 05 13:24:07 host1.com avahi-daemon[10183]: Joining mDNS multicast group on interface virbr0.IPv4 with a...2.1.
Feb 05 13:24:07 host1.com avahi-daemon[10183]: New relevant interface virbr0.IPv4 for mDNS.
Feb 05 13:24:07 host1.com avahi-daemon[10183]: Registering new address record for  on virbr0.IPv4.
Hint: Some lines were ellipsized, use -l to show in full.

root@host1:# systemctl stop avahi-daemon
Warning: Stopping avahi-daemon.service, but it can still be activated by:
  avahi-daemon.socket
 
root@host1:# systemctl disable avahi-daemon
Removed symlink /etc/systemd/system/sockets.target.wants/avahi-daemon.socket.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/avahi-daemon.service.
runcluvfy will pass after stopping daemon:
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  host2    no                        passed
  host1    no                        passed
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  host2    no                        passed
  host1    no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED