Sunday, 2 July 2017

Disk Scrubbing causing I/O Performance issue on Exadata Cell Server

Observing high I/O load and/or latency on Exadata rack. $CELLTRACE/alert.log of storage cells shows the following messages, which indicate that disk scrubbing is running:
Begin scrubbing CellDisk:CD_04_chx1cel04.
Begin scrubbing CellDisk:CD_03_chx1cel04.
Begin scrubbing CellDisk:CD_02_chx1cel04.
Begin scrubbing CellDisk:CD_00_chx1cel04.
Begin scrubbing CellDisk:CD_01_chx1cel04.

Issue is due to unpublished BUG 18508206 - EXADATA SCRUBBING IMPACTS USER WORKLOADS
Disk scrubbing is a new feature introduced in Oracle and Exadata software version. Its purpose is to periodically validate the integrity of the mirrored ASM extents and thus eliminate latent corruption. The scrubbing is designed to only run when average I/O utilization is under 25%. However, this can still cause spikes in utilization and latency and adversely affect database I/O. 
By default, the hard disk scrub runs every two weeks.
Parameters controlling the disk scrubbing:

hardDiskScrubInterval - sets the interval for proactive resilvering of latent bad sectors. Valid options are daily, weekly, biweekly and none. Using the none option stops all disk scrubbing.

hardDiskScrubStartTime command sets the start time for proactive resilvering of latent bad sectors. Valid options are a date/time combination or now.
Action Plan:
Stop disk scrubbing and reschedule it for non-peak hours’ time.
Cellcli -e list cell attributes name,hardDiskScrubInterval
CellCLI> ALTER CELL hardDiskScrubInterval=none
cellcli –e alter cell hardDiskScrubInterval=none
Please decide on hardDiskScrubStartTime to start over weekend/non-peak hours and set appropriately. Ex:
CellCLI> ALTER CELL hardDiskScrubStartTime='2014-02-05T05:45:46-05:00'
2. Change the interval to BIWEEKLY if the previous action plan was implemented to stop the disk scrub. Ex:
CellCLI> ALTER CELL hardDiskScrubInterval=biweekly
• hardDiskScrubInterval=daily
• hardDiskScrubInterval=weekly
• hardDiskScrubInterval=biweekly
Ref note: Doc ID 1684213.1

No comments:

Post a Comment