Thursday, 24 September 2020

Cluster Service with and without Domain Parameter

Service with Domain Parameter: 
SQL> show parameter domain
NAME      TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain      string corporation.local

Add Service with srvctl: (If you do not qualify the names in this parameter with a domain, Oracle qualifies them with the value of the DB_DOMAIN parameter) 

[oracle@racnode05 ~]$srvctl add service -d poc19c00 -r POC19C001,POC19C002 -s poc19c0_s1 -l PRIMARY -e SELECT -m BASIC -P BASIC
   
[oracle@racnode05 ~]$ srvctl start service -d poc19c00
[oracle@racnode05 ~]$ srvctl status service -d poc19c00
Service poc19c0_s1 is running on instance(s) POC19C001,POC19C002

Service details from listener status:
[oracle@racnode05 ~]$ lsnrctl status listener
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-SEP-2020 03:37:12
Copyright (c) 1991, 2020, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                23-SEP-2020 23:50:42
Uptime                    0 days 3 hr. 46 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/19.0.0/grid/network/admin/listener.ora
Listener Log File         /oracle/grid/diag/tnslsnr/racnode05/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.5)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.6)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP1" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP2" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_VOTEDISK" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "POC19C00" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this 
service...
Service "poc19c0_s1.corporation.local" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this service...
The command completed successfully

Service without  Domain Parameter: 
SQL> show parameter domain
NAME      TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain      string corporation.local

unset domain parameter and restart db:
SQL>  alter system set db_domain='' scope=spfile sid='*';
System altered.

[oracle@racnode05 ~]$ srvctl status database -d poc19c00
Instance POC19C001 is running on node racnode05
Instance POC19C002 is running on node racnode06

[oracle@racnode05 ~]$ srvctl stop database -d poc19c00
[oracle@racnode05 ~]$ srvctl start database -d poc19c00
[oracle@racnode05 ~]$ srvctl status database -d poc19c00
Instance POC19C001 is running on node racnode05
Instance POC19C002 is running on node racnode06

Check Service details:
[oracle@racnode05 ~]$ lsnrctl status listener

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-SEP-2020 03:41:03
Copyright (c) 1991, 2020, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                23-SEP-2020 23:50:42
Uptime                    0 days 3 hr. 50 min. 20 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/19.0.0/grid/network/admin/listener.ora
Listener Log File         /oracle/grid/diag/tnslsnr/racnode05/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.5)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.6)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP1" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP2" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_VOTEDISK" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "POC19C00" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this 
service...
Service "poc19c0_s1" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this service...
The command completed successfully

If DB_DOMAIN is not specified, then no domain will be applied to the non-qualified SERVICE_NAMES values

Wednesday, 23 September 2020

Find default ExaCS Storage Server Username and Password

On Exadata Cloud@Customer, the preconfigured user for Exadata Storage Server is cloud_user_clustername, where clustername is the name of the virtual machine (VM) cluster that is being used.


Check Clustername:

[grid@exanode1 ~]$ crsctl get cluster name

CRS-6724: Current cluster name is 'CLUSTER_NAME'


The password for cloud_user_clustername is initially set to a random value


Check default password:

[root@exanode1 ~]# /opt/exacloud/get_cs_data.py --data_file /opt/exacloud/cs_data.enc

**********************  PASSWORD




Thursday, 10 September 2020

Upload large file to Oracle SR

upload huge logs to oracle SR using curl from db server directly 

curl -T "filename" -u "supportloginid" https://transport.oracle.com/upload/issue/SR-Number/


Tuesday, 8 September 2020

failed to start asm on node4 with error CRS-2552: There are no available instances of resource 'ora.asm' to start.

Unable to start asm instance after adding 4 node to 3 node 19c cluster

[grid@racnode04 ~]$ srvctl start  asm -n racnode04
PRCR-1013 : Failed to start resource ora.asm
PRCR-1064 : Failed to start resource ora.asm on node racnode04
CRS-2552: There are no available instances of resource 'ora.asm' to start.

Verify asm status 
[grid@racnode03 ~]$ srvctl status asm
ASM is running on racnode02,racnode03,racnode01

Check asm instance count:
[grid@racnode04 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +VoteDisk/orapwASM
Backup of Password file: +VOTEDISK/orapwASM_backup
ASM listener: 
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

Check cardinality
[grid@racnode01 ]$ crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=3

Both instance count and cardinality set to 3 , it allows only 3 ams instance, asm on node4 can able to start after setting these number to 4 

Modify the Oracle ASM instance count, or cardinality 
[grid@racnode04 ~]$ srvctl modify asm -count 4
[grid@racnode04 ~]$ srvctl modify asm -count ALL

Verify Oracle ASM instance count, or cardinality
[grid@racnode04 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +VoteDisk/orapwASM
Backup of Password file: +VOTEDISK/orapwASM_backup
ASM listener: 
ASM instance count: 4
Cluster ASM listener: ASMNET1LSNR_ASM

[grid@racnode04 ~]$ crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=4

Start ASM instance on Node4:
[grid@racnode04 ~]$ srvctl start  asm -n racnode04

[grid@racnode04 ~]$ srvctl status asm
ASM is running on racnode02,racnode03,racnode01,racnode04


19.7 Grid upgrade: rootupgrade.sh fails on first node

 rootupgrade.sh failed with below error while upgrading Grid from 18.3 to 19.7.0.0

Error:

CRS-2676: Start of 'ora.cssdmonitor' on 'node01' succeeded

CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity;

details at (:CSSNM00086:) in /app/grid/diag/crs/node01/crs/trace/ocssd.trc.

CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.

CRS-4406: Oracle High Availability Services synchronous start failed.

CRS-41053: checking Oracle Grid Infrastructure for file permission issues

CRS-4000: Command Start failed, or completed with errors. 2020/09/07 09:08:46

CLSRSC-117: (Bad argc for has:clsrsc-117) Died at /u01/app/19.3.0.0/grid/crs/install/crsupgrade.pm line 1617.

 

We can get deviated with  “unable to communicate with other nodes”  errors in alert and trace files, Started looking communication between nodes

 

1.  Verified ssh connectivity between nodes , Its working fine

2.  Verified ping and traceroute , Looks good

From Node 1:

+ ping -s 9000 -c 4 -I <node1-private address> <node1-private address>

+ ping -s 9000 -c 4 -I <node1-private address> <node2-private address>

 

+ traceroute -s <node1-private address> -r -F <node1-private address> 8972

+ traceroute -s <node1-private address> -r -F <node2-private address> 8972

 

From Node 2:

+ ping -s 9000 -c 4 -I <node2-private address> <node1-private address>

+ ping -s 9000 -c 4 -I <node2-private address> <node2-private address>

 

+ traceroute -s <node2-private address> -r -F <node1-private address> 8972

+ traceroute -s <node2-private address> -r -F <node2-private address> 8972

 

While checking gipcd.trc found some failed errors:

020-09-07 08:21:27.483 : GIPCTLS:474797824:  gipcmodTlsAuthInit: tls context initialized successfully

2020-09-07 08:21:27.524 :GIPCXCPT:474797824:  gipcmodTlsLogErr: [NZOS], ssl_Handshake failed to perform operation on handshake with NZERROR [29024]

2020-09-07 08:21:27.524 :GIPCXCPT:474797824:  gipcmodTlsAuthStart: ssl_Handshake() failed with nzosErr : 29024, ret gipcretTlsErr (49)

 

As per bug id 2667217.1, Similar error reported in 19.6 upgrade

Workaround on 19.7:

1) Run rootupgrade.sh on node1

2) When it fails on Node1 with this error, then shutdown crs on node 2

  cd <18c_Gridhome/bin>

   ./crsctl stop crs

3)rerun rootupgrade.sh on node1

Sunday, 6 September 2020

Check CPU Clock Speed on Linux

we can find cpu processor details using below commands 


1. lscpu: 

It display information about the CPU architecture

# lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                96

On-line CPU(s) list:   0-95

Thread(s) per core:    2

Core(s) per socket:    24

Socket(s):             2

NUMA node(s):          1

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 85

Model name:            Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz

Stepping:              4

CPU MHz:               3374.652

CPU max MHz:           3700.0000

CPU min MHz:           1200.0000

BogoMIPS:              5400.00

Virtualization:        VT-x

L1d cache:             32K

L1i cache:             32K

L2 cache:              1024K

L3 cache:              33792K

NUMA node0 CPU(s):     0-95

2. dmidecode: print cpu speed

# dmidecode -t processor | grep "Speed"

        Max Speed: 4000 MHz

        Current Speed: 2700 MHz

        Max Speed: 4000 MHz

        Current Speed: 2700 MHz


3. Each CPU processor details from  /proc/cpuinfo file 

cat  /proc/cpuinfo

cat  /proc/cpuinfo | grep 

Saturday, 5 September 2020

Unable to delete archives in Standby db with rman

RMAN-08137: warning: archived log not deleted Error while deleting archive after backup using below command 

backup archivelog all format '/backup/$ORACLE_SID/arch/arch_flashblade_%d_%t_%s_%p' DELETE INPUT FILESPERSET 1;


Error:

piece handle=/backup/proddb01/arch/arch_flashblade_proddb0_1089597818_9469_1 tag=TAG20211126T020225 comment=NONE

channel c2: backup set complete, elapsed time: 00:00:51

channel c2: deleting archived log(s)

RMAN-08137: warning: archived log not deleted, needed for standby or upstream capture process

archived log file name=/san/proddb0_fra/1_7456_1085402700.arc thread=1 sequence=7456


Cause:

log_archive_dest_2 on standby which points to primary database was deferred


SQL> show parameter dest_2

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_create_online_log_dest_2          string

log_archive_dest_2                   string      SERVICE=proddb01 LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=proddb01



SQL> show parameter log_archive_dest_state_2

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_state_2             string      DEFER

log_archive_dest_state_20            string      enable

log_archive_dest_state_21            string      enable

log_archive_dest_state_22            string      enable

log_archive_dest_state_23            string      enable

log_archive_dest_state_24            string      enable

log_archive_dest_state_25            string      enable

log_archive_dest_state_26            string      enable

log_archive_dest_state_27            string      enable

log_archive_dest_state_28            string      enable

log_archive_dest_state_29            string      enable



Solution:

set null values to dest_2 and enable dest_state_2


SQL> alter system set log_archive_dest_2='' scope=both sid='*';

System altered.



SQL> alter system set log_archive_dest_state_2=enable scope=both sid='*';

System altered.

Wednesday, 2 September 2020

How to Install and Upgrade Terraform

Install Terraform on MAC:
1.Install Homebrew (The Missing Package Manager for macOS and Linux)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
2. Install terraform
$ brew install terraform
3. Check terraform Version 
$ terraform -v
Terraform v0.13.2

Install Terraform on Linux:
2.Copy Software to linux machine and unzip 
/home/oracle>unzip terraform_0.13.2_linux_amd64.zip
Archive:  terraform_0.13.2_linux_amd64.zip
  inflating: terraform      
         
/home/oracle> ls -ltr
total 117600
-rwxr-xr-x 1 oracle oinstall 85545348 Sep 1 07:39 terraform
-rw-r--r-- 1 oracle oinstall 34869122 Sep 1 08:16 terraform_0.13.2_linux_amd64.zip

3.Move Terraform to linux home path 
/home/oracle> mv terraform /usr/local/bin

/home/oracle> ls -ltr /usr/local/bin
total 83564
-rwxr-xr-x 1 oracle oinstall 85545348 Sep  1 07:39 terraform

4.Check terraform version 
/home/oracle> terraform -v
Terraform v0.13.2

Upgrade terraform to latest Version in MAC:
1.Present version 13.2 is out of date 
$terraform version 
Terraform v0.13.2
Your version of Terraform is out of date! The latest version
is 0.13.4. You can update by downloading from https://www.terraform.io/downloads.html

2.Upgrade to latest version (13.4)
$ brew upgrade terraform
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> New Formulae
asroute               flit                  leaf                  libnetfilter-queue    libxcomposite         libxkbfile            libxv                 server-go             xdpyinfo
blaze                 font-util             libaio                libnfnetlink          libxcursor            libxmu                libxvmc               shtools               xorgproto
box2d                 foreman               libdmx                libpciaccess          libxdamage            libxpm                libxxf86dga           structurizr-cli       xtrans
cloudformation-guard  fpart                 libdrm                libpthread-stubs      libxdmcp              libxrandr             libxxf86vm            tfsec                 zsh-you-should-use
commitizen            gcalcli               libfontenc            libsm                 libxext               libxrender            matplotplusplus       trunk
cvs-fast-export       git-hound             libfs                 libx11                libxfixes             libxres               or-tools              util-macros
device-mapper         gitql                 libgccjit             libxau                libxfont              libxscrnsaver         postgresql@12         vivid
dotnet                googletest            libhandy              libxaw                libxft                libxshmfence          protoc-gen-go-grpc    webify
envoy                 gost                  libice                libxaw3d              libxi                 libxt                 python@3.9            x86_64-elf-gdb
fleet-cli             halide                libmnl                libxcb                libxinerama           libxtst               rustscan              xcb-proto
==> Updated Formulae
Updated 1203 formulae.
==> Renamed Formulae
gst-validate -> gst-devtools
==> Deleted Formulae
meson-internal                                                                                       xu4

==> Upgrading 1 outdated package:
terraform 0.13.2 -> 0.13.4
==> Upgrading terraform 0.13.2 -> 0.13.4 
==> Downloading https://homebrew.bintray.com/bottles/terraform-0.13.4.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/05e7bf567f54c8396df8cf90470e573f5b08ebe8920bad1e638f531a54a152b6?response-content-disposition=attachment%3Bfilename%3D%22terraform-0.13.4.cat
######################################################################## 100.0%
==> Pouring terraform-0.13.4.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/terraform/0.13.4: 6 files, 67.6MB
==> `brew cleanup` has not been run in 30 days, running now...
Removing: /usr/local/Cellar/terraform/0.13.2... (6 files, 67.6MB)
Removing: /Users/jay/Library/Caches/Homebrew/terraform--0.13.2.catalina.bottle.tar.gz... (19.8MB)
Removing: /Users/jay/Library/Logs/Homebrew/terraform... (64B)

3.Verify terraform version after upgrade
$ terraform version 
Terraform v0.13.4