起因
Oracle Rac集群已經(jīng)運行好幾年，近段時間一直出現(xiàn)節(jié)點無法啟動的情況留拾，再初步排查之后發(fā)現(xiàn)ASM中的ocrdg磁盤組dismount绎橘，再重新mount后恢復(fù)诈闺，但是過段時間還是會丟失
排查
排查crsd日志
查找crsd.log日志，發(fā)現(xiàn)在2:09分集群出現(xiàn)問題厂镇，ocrdg無法訪問纤壁，集群強制dismount磁盤組
2019-08-13 02:09:05.173: [UiServer][1386170112]{1:42831:329} Sending message to PE. ctx= 0x7f29e4009010, Client PID: 12737
2019-08-13 02:09:05.173: [ CRSPE][1388271360]{1:42831:329} Cmd : 0x7f29e0009580 : flags: EVENT_TAG | FORCE_TAG | QUEUE_TAG
2019-08-13 02:09:05.173: [ CRSPE][1388271360]{1:42831:329} Processing PE command id=381. Description: [Stop Resource : 0x7f29e0009580]
2019-08-13 02:09:05.174: [ CRSPE][1388271360]{1:42831:329} Expression Filter : (((NAME == ora.OCRDG.dg) AND (LAST_SERVER == db1)) AND (STATE != OFFLINE))
2019-08-13 02:09:05.174: [ CRSPE][1388271360]{1:42831:329} Expression Filter : (((NAME == ora.OCRDG.dg) AND (LAST_SERVER == db1)) AND (STATE != OFFLINE))
2019-08-13 02:09:05.175: [ CRSPE][1388271360]{1:42831:329} Attribute overrides for the command: USR_ORA_OPI = true;
2019-08-13 02:09:05.175: [ CRSPE][1388271360]{1:42831:329} Filtering duplicate ops: server [] state [OFFLINE]
2019-08-13 02:09:05.176: [ CRSPE][1388271360]{1:42831:329} Op 0x7f29e00e9410 has 5 WOs
2019-08-13 02:09:05.176: [ CRSPE][1388271360]{1:42831:329} RI [ora.OCRDG.dg db1 1] new target state: [OFFLINE] old value: [ONLINE] 'ocrdg狀態(tài)由Online變?yōu)镺fflne'
2019-08-13 02:09:05.176: [ CRSPE][1388271360]{1:42831:329} RI [ora.OCRDG.dg db1 1] new internal state: [STOPPING] old value: [STABLE] 'ocrdg狀態(tài)由STABLE可用變?yōu)镾TOPPING停止'
2019-08-13 02:09:05.176: [ CRSOCR][1396676352]{1:42831:329} Multi Write Batch processing...
2019-08-13 02:09:05.176: [ CRSPE][1388271360]{1:42831:329} Sending message to agfw: id = 20443
2019-08-13 02:09:05.177: [ CRSPE][1388271360]{1:42831:329} CRS-2673: Attempting to stop 'ora.OCRDG.dg' on 'db1'
'嘗試停止ora.OCRDG.dg'
排查ASM日志
上面看asm出現(xiàn)問題，再去查看asm日志捺信，發(fā)現(xiàn)：
磁盤組上的ASM磁盤被執(zhí)行延遲ASM PST心跳檢查酌媒，由于默認的超時時間為15秒導(dǎo)致檢查失敗，ASM實例會強制dismount磁盤組迄靠，導(dǎo)致crsd無法讀取ocrdg而掛掉
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2. "心跳監(jiān)測15秒超時"
WARNING: Waited 15 secs for write IO to PST disk 1 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 2.
NOTE: process _b000_+asm1 (15894) initiating offline of disk 0.3916004527 (OCR1) with mask 0x7e in group 2
NOTE: process _b000_+asm1 (15894) initiating offline of disk 1.3916004528 (OCR2) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
GMON checking disk modes for group 2 at 22 for pid 31, osid 15894
ERROR: no read quorum in group: required 2, found 0 disks "錯誤：依賴2秒咨，找到0"
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 0/0xe9697caf, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 2, dsk = 1/0xe9697cb0, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 23 for pid 31, osid 15894
ERROR: no read quorum in group: required 2, found 0 disks
Tue Aug 13 02:08:34 2019
NOTE: cache dismounting (not clean) group 2/0xE3398C73 (OCRDG)
WARNING: Offline for disk OCR1 in mode 0x7f failed.
WARNING: Offline for disk OCR2 in mode 0x7f failed.
NOTE: messaging CKPT to quiesce pins Unix process pid: 15896, image: oracle@db1 (B001)
Tue Aug 13 02:08:34 2019
NOTE: halting all I/Os to diskgroup 2 (OCRDG)
Tue Aug 13 02:08:34 2019
NOTE: LGWR doing non-clean dismount of group 2 (OCRDG)
NOTE: LGWR sync ABA=18.64 last written ABA 18.64
Tue Aug 13 02:08:34 2019
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
Tue Aug 13 02:08:35 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 2, cluster inc 44)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
15 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Tue Aug 13 02:08:35 2019
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0xE3398C73 (OCRDG)
SQL> alter diskgroup OCRDG dismount force /* ASM SERVER:3812199539 */ "強制dismount ocrdg磁盤組"
Tue Aug 13 02:08:35 2019
NOTE: cache deleting context for group OCRDG 2/0xe3398c73
嘗試解決
經(jīng)過網(wǎng)上查詢，有以下兩種方式：
1.確認操作系統(tǒng)和共享存儲之間是否存在無響應(yīng)并且響應(yīng)時間是否在15秒以下
2.如果無法保證可在ASM實例（在RAC的所有節(jié)點上）中設(shè)置以下參數(shù)：
_asm_hbeatiowait為120秒
As per internal bug 17274537 , based on internal testing the value should be increased to 120 secs, which is fixed in 12.1.0.2
此參數(shù)為Oracle 11.0.2.4中加入掌挚，默認為15秒雨席，oracle將會在12.1.0.2版本中修復(fù)為120秒
如何修改：
sqlplus / as sysdba
sql> alter system set "_asm_hbeatiowait" =120 scope=spfile sid='*';
修改后需要重啟crsd或者asm實例
后記
其實問題的根本原因還是在于節(jié)點跟共享存儲之間響應(yīng)時間過長，找出這個原因吠式，才能使集群更健康
