Thursday, 24 September 2020

Cluster Service with and without Domain Parameter

Service with Domain Parameter: 
SQL> show parameter domain
NAME      TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain      string corporation.local

Add Service with srvctl: (If you do not qualify the names in this parameter with a domain, Oracle qualifies them with the value of the DB_DOMAIN parameter) 

[oracle@racnode05 ~]$srvctl add service -d poc19c00 -r POC19C001,POC19C002 -s poc19c0_s1 -l PRIMARY -e SELECT -m BASIC -P BASIC
   
[oracle@racnode05 ~]$ srvctl start service -d poc19c00
[oracle@racnode05 ~]$ srvctl status service -d poc19c00
Service poc19c0_s1 is running on instance(s) POC19C001,POC19C002

Service details from listener status:
[oracle@racnode05 ~]$ lsnrctl status listener
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-SEP-2020 03:37:12
Copyright (c) 1991, 2020, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                23-SEP-2020 23:50:42
Uptime                    0 days 3 hr. 46 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/19.0.0/grid/network/admin/listener.ora
Listener Log File         /oracle/grid/diag/tnslsnr/racnode05/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.5)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.6)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP1" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP2" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_VOTEDISK" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "POC19C00" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this 
service...
Service "poc19c0_s1.corporation.local" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this service...
The command completed successfully

Service without  Domain Parameter: 
SQL> show parameter domain
NAME      TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain      string corporation.local

unset domain parameter and restart db:
SQL>  alter system set db_domain='' scope=spfile sid='*';
System altered.

[oracle@racnode05 ~]$ srvctl status database -d poc19c00
Instance POC19C001 is running on node racnode05
Instance POC19C002 is running on node racnode06

[oracle@racnode05 ~]$ srvctl stop database -d poc19c00
[oracle@racnode05 ~]$ srvctl start database -d poc19c00
[oracle@racnode05 ~]$ srvctl status database -d poc19c00
Instance POC19C001 is running on node racnode05
Instance POC19C002 is running on node racnode06

Check Service details:
[oracle@racnode05 ~]$ lsnrctl status listener

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-SEP-2020 03:41:03
Copyright (c) 1991, 2020, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                23-SEP-2020 23:50:42
Uptime                    0 days 3 hr. 50 min. 20 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/19.0.0/grid/network/admin/listener.ora
Listener Log File         /oracle/grid/diag/tnslsnr/racnode05/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.5)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=00.000.00.6)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP1" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_ORADSKGRP2" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_VOTEDISK" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "POC19C00" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this 
service...
Service "poc19c0_s1" has 1 instance(s).
  Instance "POC19C001", status READY, has 2 handler(s) for this service...
The command completed successfully

If DB_DOMAIN is not specified, then no domain will be applied to the non-qualified SERVICE_NAMES values

Wednesday, 23 September 2020

Find default ExaCS Storage Server Username and Password

On Exadata Cloud@Customer, the preconfigured user for Exadata Storage Server is cloud_user_clustername, where clustername is the name of the virtual machine (VM) cluster that is being used.


Check Clustername:

[grid@exanode1 ~]$ crsctl get cluster name

CRS-6724: Current cluster name is 'CLUSTER_NAME'


The password for cloud_user_clustername is initially set to a random value


Check default password:

[root@exanode1 ~]# /opt/exacloud/get_cs_data.py --data_file /opt/exacloud/cs_data.enc

**********************  PASSWORD




Tuesday, 8 September 2020

failed to start asm on node4 with error CRS-2552: There are no available instances of resource 'ora.asm' to start.

Unable to start asm instance after adding 4 node to 3 node 19c cluster

[grid@racnode04 ~]$ srvctl start  asm -n racnode04
PRCR-1013 : Failed to start resource ora.asm
PRCR-1064 : Failed to start resource ora.asm on node racnode04
CRS-2552: There are no available instances of resource 'ora.asm' to start.

Verify asm status 
[grid@racnode03 ~]$ srvctl status asm
ASM is running on racnode02,racnode03,racnode01

Check asm instance count:
[grid@racnode04 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +VoteDisk/orapwASM
Backup of Password file: +VOTEDISK/orapwASM_backup
ASM listener: 
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

Check cardinality
[grid@racnode01 ]$ crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=3

Both instance count and cardinality set to 3 , it allows only 3 ams instance, asm on node4 can able to start after setting these number to 4 

Modify the Oracle ASM instance count, or cardinality 
[grid@racnode04 ~]$ srvctl modify asm -count 4
[grid@racnode04 ~]$ srvctl modify asm -count ALL

Verify Oracle ASM instance count, or cardinality
[grid@racnode04 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +VoteDisk/orapwASM
Backup of Password file: +VOTEDISK/orapwASM_backup
ASM listener: 
ASM instance count: 4
Cluster ASM listener: ASMNET1LSNR_ASM

[grid@racnode04 ~]$ crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=4

Start ASM instance on Node4:
[grid@racnode04 ~]$ srvctl start  asm -n racnode04

[grid@racnode04 ~]$ srvctl status asm
ASM is running on racnode02,racnode03,racnode01,racnode04


19.7 Grid upgrade: rootupgrade.sh fails on first node

 rootupgrade.sh failed with below error while upgrading Grid from 18.3 to 19.7.0.0

Error:

CRS-2676: Start of 'ora.cssdmonitor' on 'node01' succeeded

CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity;

details at (:CSSNM00086:) in /app/grid/diag/crs/node01/crs/trace/ocssd.trc.

CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.

CRS-4406: Oracle High Availability Services synchronous start failed.

CRS-41053: checking Oracle Grid Infrastructure for file permission issues

CRS-4000: Command Start failed, or completed with errors. 2020/09/07 09:08:46

CLSRSC-117: (Bad argc for has:clsrsc-117) Died at /u01/app/19.3.0.0/grid/crs/install/crsupgrade.pm line 1617.

 

We can get deviated with  “unable to communicate with other nodes”  errors in alert and trace files, Started looking communication between nodes

 

1.  Verified ssh connectivity between nodes , Its working fine

2.  Verified ping and traceroute , Looks good

From Node 1:

+ ping -s 9000 -c 4 -I <node1-private address> <node1-private address>

+ ping -s 9000 -c 4 -I <node1-private address> <node2-private address>

 

+ traceroute -s <node1-private address> -r -F <node1-private address> 8972

+ traceroute -s <node1-private address> -r -F <node2-private address> 8972

 

From Node 2:

+ ping -s 9000 -c 4 -I <node2-private address> <node1-private address>

+ ping -s 9000 -c 4 -I <node2-private address> <node2-private address>

 

+ traceroute -s <node2-private address> -r -F <node1-private address> 8972

+ traceroute -s <node2-private address> -r -F <node2-private address> 8972

 

While checking gipcd.trc found some failed errors:

020-09-07 08:21:27.483 : GIPCTLS:474797824:  gipcmodTlsAuthInit: tls context initialized successfully

2020-09-07 08:21:27.524 :GIPCXCPT:474797824:  gipcmodTlsLogErr: [NZOS], ssl_Handshake failed to perform operation on handshake with NZERROR [29024]

2020-09-07 08:21:27.524 :GIPCXCPT:474797824:  gipcmodTlsAuthStart: ssl_Handshake() failed with nzosErr : 29024, ret gipcretTlsErr (49)

 

As per bug id 2667217.1, Similar error reported in 19.6 upgrade

Workaround on 19.7:

1) Run rootupgrade.sh on node1

2) When it fails on Node1 with this error, then shutdown crs on node 2

  cd <18c_Gridhome/bin>

   ./crsctl stop crs

3)rerun rootupgrade.sh on node1

Wednesday, 2 September 2020

How to Install and Upgrade Terraform

Install Terraform on MAC:
1.Install Homebrew (The Missing Package Manager for macOS and Linux)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
2. Install terraform
$ brew install terraform
3. Check terraform Version 
$ terraform -v
Terraform v0.13.2

Install Terraform on Linux:
2.Copy Software to linux machine and unzip 
/home/oracle>unzip terraform_0.13.2_linux_amd64.zip
Archive:  terraform_0.13.2_linux_amd64.zip
  inflating: terraform      
         
/home/oracle> ls -ltr
total 117600
-rwxr-xr-x 1 oracle oinstall 85545348 Sep 1 07:39 terraform
-rw-r--r-- 1 oracle oinstall 34869122 Sep 1 08:16 terraform_0.13.2_linux_amd64.zip

3.Move Terraform to linux home path 
/home/oracle> mv terraform /usr/local/bin

/home/oracle> ls -ltr /usr/local/bin
total 83564
-rwxr-xr-x 1 oracle oinstall 85545348 Sep  1 07:39 terraform

4.Check terraform version 
/home/oracle> terraform -v
Terraform v0.13.2

Upgrade terraform to latest Version in MAC:
1.Present version 13.2 is out of date 
$terraform version 
Terraform v0.13.2
Your version of Terraform is out of date! The latest version
is 0.13.4. You can update by downloading from https://www.terraform.io/downloads.html

2.Upgrade to latest version (13.4)
$ brew upgrade terraform
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> New Formulae
asroute               flit                  leaf                  libnetfilter-queue    libxcomposite         libxkbfile            libxv                 server-go             xdpyinfo
blaze                 font-util             libaio                libnfnetlink          libxcursor            libxmu                libxvmc               shtools               xorgproto
box2d                 foreman               libdmx                libpciaccess          libxdamage            libxpm                libxxf86dga           structurizr-cli       xtrans
cloudformation-guard  fpart                 libdrm                libpthread-stubs      libxdmcp              libxrandr             libxxf86vm            tfsec                 zsh-you-should-use
commitizen            gcalcli               libfontenc            libsm                 libxext               libxrender            matplotplusplus       trunk
cvs-fast-export       git-hound             libfs                 libx11                libxfixes             libxres               or-tools              util-macros
device-mapper         gitql                 libgccjit             libxau                libxfont              libxscrnsaver         postgresql@12         vivid
dotnet                googletest            libhandy              libxaw                libxft                libxshmfence          protoc-gen-go-grpc    webify
envoy                 gost                  libice                libxaw3d              libxi                 libxt                 python@3.9            x86_64-elf-gdb
fleet-cli             halide                libmnl                libxcb                libxinerama           libxtst               rustscan              xcb-proto
==> Updated Formulae
Updated 1203 formulae.
==> Renamed Formulae
gst-validate -> gst-devtools
==> Deleted Formulae
meson-internal                                                                                       xu4

==> Upgrading 1 outdated package:
terraform 0.13.2 -> 0.13.4
==> Upgrading terraform 0.13.2 -> 0.13.4 
==> Downloading https://homebrew.bintray.com/bottles/terraform-0.13.4.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/05e7bf567f54c8396df8cf90470e573f5b08ebe8920bad1e638f531a54a152b6?response-content-disposition=attachment%3Bfilename%3D%22terraform-0.13.4.cat
######################################################################## 100.0%
==> Pouring terraform-0.13.4.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/terraform/0.13.4: 6 files, 67.6MB
==> `brew cleanup` has not been run in 30 days, running now...
Removing: /usr/local/Cellar/terraform/0.13.2... (6 files, 67.6MB)
Removing: /Users/jay/Library/Caches/Homebrew/terraform--0.13.2.catalina.bottle.tar.gz... (19.8MB)
Removing: /Users/jay/Library/Logs/Homebrew/terraform... (64B)

3.Verify terraform version after upgrade
$ terraform version 
Terraform v0.13.4