In a recent tuning activity it was necessary to change the Exadata Smart Flash Cache from “WriteThrough” to “WriteBack“. Because the system was used in a 24/7 environment we had to implement the change in a Rolling Upgrade Fashion.
Here below are described the different steps.
From one DB node using dcli check the currest status of the storage cells:
[root@efudbadm02 ~]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode" efuceladm01: WriteThrough efuceladm02: WriteThrough efuceladm03: WriteThrough efuceladm04: WriteThrough efuceladm05: WriteThrough efuceladm06: WriteThrough efuceladm07: WriteThrough efuceladm08: WriteThrough efuceladm09: WriteThrough efuceladm10: WriteThrough efuceladm11: WriteThrough
From one DB node using dcli check that the properties asmdeactivationoutcome and asmmodestatus of all griddisks are respectively “Yes” and “ONLINE” before continuing with the change.
[root@efudbadm02 ~]# dcli -g cell_group -l root cellcli -e list griddisk attributes asmdeactivationoutcome, asmmodestatus efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm01: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE efuceladm02: Yes ONLINE ... .. .
From one DB node using dcli check that all flashcache modules are in “normal” state and no flash disk is in degraded or critical state.
[root@efudbadm02 ~]# dcli -g cell_group -l root cellcli -e list flashcache detail efuceladm01: name: efuceladm01_FLASHCACHE efuceladm01: cellDisk: FD_00_efuceladm01,FD_07_efuceladm01,FD_06_efuceladm01,FD_03_efuceladm01,FD_05_efuceladm01,FD_01_efuceladm01,FD_02_efuceladm01,FD_04_efuceladm01 efuceladm01: creationTime: 2013-06-18T15:21:13+02:00 efuceladm01: degradedCelldisks: efuceladm01: effectiveCacheSize: 744.125G efuceladm01: id: 35b61001-438f-4d66-8ce9-40704f758d3f efuceladm01: size: 744.125G efuceladm01: status: normal efuceladm02: name: efuceladm02_FLASHCACHE efuceladm02: cellDisk: FD_06_efuceladm02,FD_05_efuceladm02,FD_00_efuceladm02,FD_02_efuceladm02,FD_01_efuceladm02,FD_07_efuceladm02,FD_03_efuceladm02,FD_04_efuceladm02 efuceladm02: creationTime: 2013-06-18T15:21:12+02:00 efuceladm02: degradedCelldisks: efuceladm02: effectiveCacheSize: 744.125G efuceladm02: id: 2f7eedd6-cda2-496e-98ec-417b94fb8ee7 efuceladm02: size: 744.125G efuceladm02: status: normal efuceladm03: name: efuceladm03_FLASHCACHE efuceladm03: cellDisk: FD_00_efuceladm03,FD_04_efuceladm03,FD_01_efuceladm03,FD_02_efuceladm03,FD_03_efuceladm03,FD_06_efuceladm03,FD_05_efuceladm03,FD_07_efuceladm03 efuceladm03: creationTime: 2013-06-18T15:21:10+02:00 efuceladm03: degradedCelldisks: efuceladm03: effectiveCacheSize: 744.125G efuceladm03: id: c271cdb8-dc70-4009-ba97-dfc4c26b00ef efuceladm03: size: 744.125G efuceladm03: status: normal ... .. .
Logon on the first Cell Storage and using CellCli interface perform the following procedure to enable the WriteBack Flash Cache in a rolling upgrade fashion.
Drop the existing flash cache
CellCLI> drop flashcache Flash cache efuceladm01_FLASHCACHE successfully dropped
Inactivate the griddisk on the cell
CellCLI> alter griddisk all inactive GridDisk DATA_CD_00_efuceladm01 successfully altered GridDisk DATA_CD_01_efuceladm01 successfully altered GridDisk DATA_CD_02_efuceladm01 successfully altered GridDisk DATA_CD_03_efuceladm01 successfully altered GridDisk DATA_CD_04_efuceladm01 successfully altered GridDisk DATA_CD_05_efuceladm01 successfully altered GridDisk DBFS_DG_CD_02_efuceladm01 successfully altered GridDisk DBFS_DG_CD_03_efuceladm01 successfully altered GridDisk DBFS_DG_CD_04_efuceladm01 successfully altered GridDisk DBFS_DG_CD_05_efuceladm01 successfully altered GridDisk RECO_CD_00_efuceladm01 successfully altered GridDisk RECO_CD_01_efuceladm01 successfully altered GridDisk RECO_CD_02_efuceladm01 successfully altered GridDisk RECO_CD_03_efuceladm01 successfully altered GridDisk RECO_CD_04_efuceladm01 successfully altered GridDisk RECO_CD_05_efuceladm01 successfully altered
Shut down cellsrv service
CellCLI> alter cell shutdown services cellsrv Stopping CELLSRV services... The SHUTDOWN of CELLSRV services was successful.
Enable the Smart Flash Cache WriteBack
CellCLI> alter cell flashCacheMode=writeback Cell efuceladm01 successfully altered
Restart the cellsrv service
CellCLI> alter cell startup services cellsrv Starting CELLSRV services... The STARTUP of CELLSRV services was successful.
Reactivate the griddisk on the cell
CellCLI> alter griddisk all active GridDisk DATA_CD_00_efuceladm01 successfully altered GridDisk DATA_CD_01_efuceladm01 successfully altered GridDisk DATA_CD_02_efuceladm01 successfully altered GridDisk DATA_CD_03_efuceladm01 successfully altered GridDisk DATA_CD_04_efuceladm01 successfully altered GridDisk DATA_CD_05_efuceladm01 successfully altered GridDisk DBFS_DG_CD_02_efuceladm01 successfully altered GridDisk DBFS_DG_CD_03_efuceladm01 successfully altered GridDisk DBFS_DG_CD_04_efuceladm01 successfully altered GridDisk DBFS_DG_CD_05_efuceladm01 successfully altered GridDisk RECO_CD_00_efuceladm01 successfully altered GridDisk RECO_CD_01_efuceladm01 successfully altered GridDisk RECO_CD_02_efuceladm01 successfully altered GridDisk RECO_CD_03_efuceladm01 successfully altered GridDisk RECO_CD_04_efuceladm01 successfully altered GridDisk RECO_CD_05_efuceladm01 successfully altered
Recreate the flash cache
CellCLI> create flashcache all Flash cache efuceladm01_FLASHCACHE successfully created
Verify that the Smart Flash Cache WriteBackWriteBack option is enabled
[root@efuceladm01 ~]# cellcli -e list cell detail | grep flashCacheMode flashCacheMode: writeback
Before applying the change to the next Exadata Storage Server wait that all griddisk are synronized and online.
[root@efuceladm01 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome DATA_CD_00_efuceladm01 SYNCING Yes DATA_CD_01_efuceladm01 SYNCING Yes DATA_CD_02_efuceladm01 SYNCING Yes DATA_CD_03_efuceladm01 SYNCING Yes DATA_CD_04_efuceladm01 SYNCING Yes DATA_CD_05_efuceladm01 SYNCING Yes DBFS_DG_CD_02_efuceladm01 ONLINE Yes DBFS_DG_CD_03_efuceladm01 ONLINE Yes DBFS_DG_CD_04_efuceladm01 ONLINE Yes DBFS_DG_CD_05_efuceladm01 ONLINE Yes RECO_CD_00_efuceladm01 OFFLINE Yes RECO_CD_01_efuceladm01 OFFLINE Yes RECO_CD_02_efuceladm01 OFFLINE Yes RECO_CD_03_efuceladm01 OFFLINE Yes RECO_CD_04_efuceladm01 OFFLINE Yes RECO_CD_05_efuceladm01 OFFLINE Yes
Once the asmmodestatus is ONLINE on all griddisks it is safe to move to the next Storage Server.
At the end of the procedure all Storage Servers are configured with Smart Flash Cache WriteBach option:
[root@efudbadm02 ~]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode" efuceladm01: writeback efuceladm02: writeback efuceladm03: writeback efuceladm04: writeback efuceladm05: writeback efuceladm06: writeback efuceladm07: writeback efuceladm08: writeback efuceladm09: writeback efuceladm10: writeback efuceladm11: writeback
My Oracle Notes 