Tuesday 26 March 2019

Create ASM DISKGROUP with EXADATA FLASHDISK

X7-2 half rack Exadata machine having 4 storage nodes and 24 TB allocated to flashcache on each cell
We are going allocate 12TB flashcache and 12TB to flashdisk

Check flashcache details:
CellCLI> list flashcache detail
         name:                   dbm0celadm01_FLASHCACHE
         cellDisk:               FD_00_dbm0celadm01,FD_02_dbm0celadm01,FD_01_dbm0celadm01,FD_03_dbm0celadm01
         creationTime:           2019-01-19T19:57:14-08:00
         degradedCelldisks:
         effectiveCacheSize:     23.28692626953125T
         id:                     1241246b-af5c-4636-b4a1-d23f21334c66
         size:                   23.28692626953125T
         status:                 normal
Will drop and recreate flashcache with 12TB:
CellCLI> drop flashcache
CELL-02769: Cannot complete flash cache drop on: FD_00_dbm0celadm01. Received error: CELL-02732: There may be data not synchronized to the grid disks (dirty data) in flash cache. Use ALTER FLASHCACHE to flush the dirty data and to stop caching, and then retry the command. Flash cache was not dropped on these cell disks: FD_00_dbm0celadm01 FD_02_dbm0celadm01 FD_01_dbm0celadm01 FD_03_dbm0celadm01 .
CellCLI> alter flashcache all flush

Monitor flush status from another session:
CellCLI>  list flashcache detail
         name:                   dbm0celadm01_FLASHCACHE
         cellDisk:               FD_00_dbm0celadm01,FD_02_dbm0celadm01,FD_01_dbm0celadm01,FD_03_dbm0celadm01
         creationTime:           2019-01-19T19:57:14-08:00
         degradedCelldisks:
         effectiveCacheSize:     23.28692626953125T
         id:                     1241246b-af5c-4636-b4a1-d23f21334c66
         size:                   23.28692626953125T
         status:                 normal - flushing
CellCLI> list flashcache detail
         name:                   dbm0celadm01_FLASHCACHE
         cellDisk:               FD_00_dbm0celadm01,FD_02_dbm0celadm01,FD_01_dbm0celadm01,FD_03_dbm0celadm01
         creationTime:           2019-01-19T19:57:14-08:00
         degradedCelldisks:
         effectiveCacheSize:     23.28692626953125T
         id:                     1241246b-af5c-4636-b4a1-d23f21334c66
         size:                   23.28692626953125T
         status:                 normal - FD_01_dbm0celadm01 (flushing), FD_03_dbm0celadm01 (flushing), FD_00_dbm0celadm01 (flushed), FD_02_dbm0celadm01 (flushed)
CellCLI> list flashcache detail
         name:                   dbm0celadm01_FLASHCACHE
         cellDisk:               FD_00_dbm0celadm01,FD_02_dbm0celadm01,FD_01_dbm0celadm01,FD_03_dbm0celadm01
         creationTime:           2019-01-19T19:57:14-08:00
         degradedCelldisks:
         effectiveCacheSize:     23.28692626953125T
         id:                     1241246b-af5c-4636-b4a1-d23f21334c66
         size:                   23.28692626953125T
         status:                 normal - flushed

Drop Existing flashcache:
CellCLI> drop flashcache
Flash cache dbm0celadm01_FLASHCACHE successfully dropped
CellCLI> list flashcache detail 

Create flashcache with 12TB:
CellCLI> create flashcache all size=12288G
Flash cache dbm0celadm03_FLASHCACHE successfully created
CellCLI> list flashcache detail
         name:                   dbm0celadm03_FLASHCACHE
         cellDisk:               FD_02_dbm0celadm03,FD_03_dbm0celadm03,FD_01_dbm0celadm03,FD_00_dbm0celadm03
         creationTime:           2019-03-21T22:56:17-07:00
         degradedCelldisks:
         effectiveCacheSize:     12T
         id:                     b14774c0-8d09-4cf3-ac15-51d88968e9cf
         size:                   12T
         status:                 normal

Create flashdisk:
CellCLI> CREATE GRIDDISK ALL FLASHDISK PREFIX='FDOM'
GridDisk FDOM_FD_00_dbm0celadm03 successfully created
GridDisk FDOM_FD_01_dbm0celadm03 successfully created
GridDisk FDOM_FD_02_dbm0celadm03 successfully created
GridDisk FDOM_FD_03_dbm0celadm03 successfully created
CellCLI> list griddisk attributes diskType, name where diskType=FLASHDISK
         FlashDisk       FDOM_FD_00_dbm0celadm03
         FlashDisk       FDOM_FD_01_dbm0celadm03
         FlashDisk       FDOM_FD_02_dbm0celadm03
         FlashDisk       FDOM_FD_03_dbm0celadm03
CellCLI> list griddisk  FDOM_FD_00_dbm0celadm01 detail
         name:                   FDOM_FD_00_dbm0celadm01
         asmDiskGroupName:
         asmDiskName:
         asmFailGroupName:
         availableTo:
         cachedBy:
         cachingPolicy:          default
         cellDisk:               FD_00_dbm0celadm01
         comment:
         creationTime:           2019-03-21T23:02:52-07:00
         diskType:               FlashDisk
         errorCount:             0
         id:                     a649aff8-20ec-4622-9315-8d877e6705ab
         size:                   2.8217315673828125T
         status:                 active

We need at least one failover group to create asm diskgroup in x7-2
Repeat same step on storage node 2 and create flashdisk
Set asm disk string parameter:
alter system set asm_diskstring='o/*/DATAC1_*','o/*/RECOC1_*','o/*/FDOM_*','/dev/exadata_quorum/*' scope=both sid='*';

Verify newly created flashdisk from cell nodes visible on asm or not:
SQL> select name, total_mb/1024, free_mb/1024, path, header_status from v$asm_disk where header_status='CANDIDATE';
NAME            TOTAL_MB/1024 FREE_MB/1024 PATH   HEADER_STATU
-------------- ------------- ------------ ----------------------------
                           0            0 o/000.00.00.15;000.00.00.16/FDOM_FD_00_dbm0celadm04                        CANDIDATE
                           0            0 o/000.00.00.9;000.00.00.10/FDOM_FD_03_dbm0celadm01                         CANDIDATE
                           0            0 o/000.00.00.15;000.00.00.16/FDOM_FD_01_dbm0celadm04                        CANDIDATE
                           0            0 o/000.00.00.9;000.00.00.10/FDOM_FD_00_dbm0celadm01                         CANDIDATE
                           0            0 o/000.00.00.13;000.00.00.14/FDOM_FD_03_dbm0celadm03                        CANDIDATE
                           0            0 o/000.00.00.15;000.00.00.16/FDOM_FD_02_dbm0celadm04                        CANDIDATE
                           0            0 o/000.00.00.9;000.00.00.10/FDOM_FD_02_dbm0celadm01                         CANDIDATE
                           0            0 o/000.00.00.15;000.00.00.16/FDOM_FD_03_dbm0celadm04                        CANDIDATE
                           0            0 o/000.00.00.11;000.00.00.12/FDOM_FD_01_dbm0celadm02                        CANDIDATE
                           0            0 o/000.00.00.11;000.00.00.12/FDOM_FD_03_dbm0celadm02                        CANDIDATE
                           0            0 o/000.00.00.13;000.00.00.14/FDOM_FD_01_dbm0celadm03                        CANDIDATE
                           0            0 o/000.00.00.9;000.00.00.10/FDOM_FD_01_dbm0celadm01                         CANDIDATE
                           0            0 o/000.00.00.13;000.00.00.14/FDOM_FD_02_dbm0celadm03                        CANDIDATE
                           0            0 o/000.00.00.11;000.00.00.12/FDOM_FD_02_dbm0celadm02                        CANDIDATE
                           0            0 o/000.00.00.13;000.00.00.14/FDOM_FD_00_dbm0celadm03                        CANDIDATE
                           0            0 o/000.00.00.11;000.00.00.12/FDOM_FD_00_dbm0celadm02                        CANDIDATE
16 rows selected.

Create asm diskgroup:
SQL> CREATE diskgroup FLASHDISK normal redundancy disk 'o/*/FDOM*'
 attribute 'compatible.rdbms'='11.2.0.4.0',
 'content.type' = 'data',
 'compatible.asm'='18.0.0.0.0',
 'cell.smart_scan_capable'='TRUE',
 'au_size'='4M';  2    3    4    5    6
Diskgroup created.

SQL> select name,state,total_mb/1024, free_mb/1024 from v$asm_diskgroup;
NAME           STATE       TOTAL_MB/1024 FREE_MB/1024
-------------- ----------- ------------- ------------
DATAC1         MOUNTED          61440.25    3746.0625
RECOC1         MOUNTED             16368   12160.2344
BACKUP         MOUNTED             30720   2719.21875
FLASHDISK      MOUNTED          46231.25   46230.3164

No comments:

Post a Comment