1、系统运行卡慢，磁盘IO能力不足
系统正常运行：
# iostat -x -d 1
Linux 2.6.32-431.29.2.el6.x86_64 (LancetServer) 2020年09月24日 _x86_64_ (32 CPU)
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.72 198.57 49.72 90.21 3602.41 2310.19 42.25 3.64 25.99 10.17 34.71 4.24 59.26
sdb 0.15 126.94 3.72 2.44 778.83 1035.08 294.40 0.46 74.24 0.77 186.22 1.28 0.79
dm-0 0.00 0.00 1.03 26.19 51.53 209.48 9.59 4.71 172.96 11.57 179.34 1.21 3.30
dm-1 0.00 0.00 0.00 0.01 0.01 0.06 8.00 0.00 42.88 4.15 47.14 2.49 0.00
dm-2 0.00 0.00 2.20 141.49 19.79 1131.92 8.02 0.31 2.15 5.11 2.10 2.74 39.36
dm-3 0.00 0.00 47.22 121.09 3531.06 968.70 26.73 5.08 30.20 10.44 37.90 2.40 40.40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 115.00 75.00 229.00 4264.00 4880.00 30.08 48.95 302.85 22.85 394.55 3.29 100.10
sdb 0.00 4.00 0.00 4.00 0.00 64.00 16.00 0.05 11.50 0.00 11.50 6.25 2.50
dm-0 0.00 0.00 0.00 11.00 0.00 88.00 8.00 0.97 87.82 0.00 87.82 42.09 46.30
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 1.00 178.00 8.00 1424.00 8.00 161.30 1423.55 1.00 1431.54 5.55 99.30
dm-3 0.00 0.00 74.00 8.00 4256.00 64.00 52.68 1.82 22.39 23.15 15.38 12.07 99.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 136.00 156.00 121.00 10016.00 2112.00 43.78 3.25 11.61 9.57 14.24 3.61 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.03 26.00 0.00 26.00 26.00 2.60
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 235.00 0.00 1880.00 8.00 3.48 14.88 0.00 14.88 4.17 98.10
dm-3 0.00 0.00 158.00 19.00 10080.00 152.00 57.81 1.92 10.81 9.54 21.37 5.56 98.50
%util 已经跑满
解压缩MySQL备份2.2GB 用了5分多钟：
[root@LancetServer test]# ls -shla
总用量 2.2G
4.0K drwxr-xr-x 3 root root 4.0K 9月 24 19:09 .
4.0K drwxr-xr-x 7 root root 4.0K 9月 24 01:41 ..
2.6M -rw-r--r-- 1 root root 2.6M 9月 23 17:35 ibm_utl_sraidmr_megacli-8.00.48_linux_32-64.zip
2.2G -rw-r--r-- 1 root root 2.2G 9月 23 16:44 lancet-icu-gx-20200923.tar.gz
4.0K drwxr-xr-x 2 root root 4.0K 9月 23 17:36 linux
[root@LancetServer test]# date; tar zxf lancet-icu-gx-20200923.tar.gz ; date
2020年 09月 24日 星期四 19:10:19 CST
2020年 09月 24日 星期四 19:15:46 CST
解压缩MySQL备份过程中，磁盘IO情况监控：
[root@LancetServer ~]# pidstat -d 1
19时10分42秒 PID kB_rd/s kB_wr/s kB_ccwr/s Command
19时10分43秒 1602 0.00 64.00 0.00 jbd2/dm-2-8
19时10分43秒 2379 0.00 4.00 0.00 mysqld
19时10分43秒 2767 12.00 8.00 4.00 node
19时10分43秒 2805 0.00 4.00 0.00 node
19时10分43秒 2813 0.00 4.00 0.00 node
19时10分43秒 3993 16.00 0.00 0.00 abs_monitor
19时10分43秒 7596 8.00 0.00 0.00 node
19时10分43秒 16432 0.00 96.00 0.00 sendmail
19时10分43秒 22724 0.00 135388.00 0.00 tar
19时10分43秒 22725 9472.00 0.00 0.00 gzip
19时10分43秒 22973 0.00 960.00 0.00 node
[root@LancetServer ~]# iostat -x -d 1
Linux 2.6.32-431.29.2.el6.x86_64 (LancetServer) 2020年09月24日 _x86_64_ (32 CPU)
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.72 198.64 49.71 90.24 3601.74 2310.98 42.25 3.64 25.99 10.17 34.71 4.24 59.27
sdb 0.15 128.14 3.72 2.46 779.68 1044.64 295.23 0.46 75.07 0.78 187.70 1.28 0.79
dm-0 0.00 0.00 1.03 26.20 51.53 209.57 9.59 4.71 172.96 11.57 179.34 1.21 3.30
dm-1 0.00 0.00 0.00 0.01 0.01 0.06 8.00 0.00 41.93 4.15 45.87 2.43 0.00
dm-2 0.00 0.00 2.20 141.56 19.79 1132.49 8.02 0.31 2.18 5.11 2.13 2.74 39.38
dm-3 0.00 0.00 47.21 121.10 3530.40 968.83 26.73 5.08 30.20 10.44 37.91 2.40 40.40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 322.00 4.00 181.00 56.00 4080.00 22.36 1.24 6.71 5.75 6.73 5.25 97.20
sdb 9.00 23818.00 11.00 318.00 4624.00 201176.00 625.53 141.92 429.33 77.27 441.51 3.04 100.00
dm-0 0.00 0.00 3.00 0.00 24.00 0.00 8.00 0.02 5.33 5.33 0.00 5.33 1.60
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 497.00 0.00 3976.00 8.00 3.58 7.24 0.00 7.24 1.95 97.10
dm-3 0.00 0.00 1.00 5.00 32.00 40.00 12.00 0.08 13.33 7.00 14.60 8.33 5.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 354.00 2.00 184.00 40.00 4248.00 23.05 1.46 7.70 6.50 7.71 5.27 98.10
sdb 12.00 25581.00 17.00 312.00 7176.00 199680.00 628.74 143.43 455.90 56.00 477.69 3.04 100.00
dm-0 0.00 0.00 1.00 42.00 8.00 336.00 8.00 1.50 34.79 6.00 35.48 1.84 7.90
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 493.00 0.00 3944.00 8.00 3.54 7.06 0.00 7.06 1.99 97.90
dm-3 0.00 0.00 4.00 4.00 328.00 32.00 45.00 0.09 8.88 1.75 16.00 7.75 6.20
2、查看本机RAID卡型号：
[root@LancetServer ~]# dmesg | grep -i raid
dracut: rd_NO_MD: removing MD RAID activation
megaraid_sas 0000:09:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
megaraid_sas 0000:09:00.0: setting latency timer to 64
megaraid_sas 0000:09:00.0: irq 86 for MSI/MSI-X
megaraid_sas 0000:09:00.0: [scsi0]: FW supports<0> MSIX vector,Online CPUs: <32>,Current MSIX <1>
megaraid_sas 0000:09:00.0: Controller type: MR,Memory size is: 512MB
scsi0 : LSI SAS based MegaRAID driver
scsi 0:2:0:0: Direct-Access IBM ServeRAID M5015 2.13 PQ: 0 ANSI: 5
scsi 0:2:1:0: Direct-Access IBM ServeRAID M5015 2.13 PQ: 0 ANSI: 5
3、服务器查raid卡电池情况
下载安装：ibm_utl_sraidmr_megacli-8.00.48_linux_32-64.zip
[root@LancetServer ~]# wget ftp://download2.boulder.ibm.com/ecc/sar/CMA/XSA/ibm_utl_sraidmr_megacli-8.00.48_linux_32-64.zip
[root@LancetServer MegaCli]# MegaCli64 -AdpBbuCmd -aAll
Adapter 0: Get BBU Status Failed.
Adapter 0: Get BBU Capacity Info Failed.
Adapter 0: Get BBU Status Info Failed.
Adapter 0: Get BBU Properties Failed.
Exit Code: 0x00
[root@LancetServer ~]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL
Adapter 0: Get BBU Status Failed.
Exit Code: 0x01
返回以上内容是：那raid卡的BBU电池模块就有问题，或者就没电池。
电池状态说明：
- Charging Status: 当前电池处于什么状态，有Charging, Discharging, None等值，分别代表电池充电，放电，及没有充放电操作的状态
- Learn Cycle Requested: Learn Cycle请求，当为Yes时，并且下面的Learn Cycle Active为No, 说明已经开始了Learn Cycle的第一阶段, 此时策略开始变为WriteThrough, 电池将经历一个放电后充电或者充电的过程
- Learn Cycle Active： 是否处于Learn Cycle的校准阶段，如果为Yes, 则进入了Learn Cycle的第二阶段，控制器开始校准电池.
- Battery Replacement required: 电池是否需要维修，如果为Yes, 请尽快更换电池
- Remaining Capacity Low: 剩余电容量低， 如果为Yes, 需要更换电池
4、查看缓存策略
[root@LancetServer ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 1.088 TB
State : Optimal
Strip Size : 128 KB
Number Of Drives : 3
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 557.861 GB
State : Optimal
Strip Size : 128 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
Exit Code: 0x00
- Default Cache Policy: 默认的缓存策略，针对每个RAID可以有不同的设置.
- Current Cache Policy: 当前生效的缓存策略.
策略说明
第一段: WriteBack, WriteThrough
- WriteBack：回写。进行写操作时，将数据写入RAID卡缓存，并直接返回，RAID卡控制器将在系统负载低或者Cache满了的情况下把数据写入硬盘。该设置会大大提升RAID卡写性能，绝大多数的情况下会降低系统IO负载。 数据的可靠性由RAID卡的BBU(Battery Backup Unit)进行保证。
- WriteThrough: 写通。数据写操作不使用缓存，数据直接写入磁盘。RAID卡写性能下降，在大多数情况下该设置会造成系统IO负载上升。
第二段: ReadAheadNone, ReadAdaptive, ReadAhead.
- ReadAheadNone: 不开启预读。这是默认的设置
- ReadAhead: 在读操作时，预先把后面顺序的数据加载入Cache，在顺序读取时，能提高性能，相反会降低随机读的性能。
- ReadAdaptive: 自适应预读，当Cache memory和IO空闲时，采取顺序预读，平衡了连续读性能及随机读的性能，需要消耗一定的计算能力。
第三段: Direct, Cached.
- Direct: Direct IO模式，读操作不缓存到cache memory中，数据将同时传输到cache中和应用，如果接下来要读取相同的数据块，则直接从Cache memory中获取. 这是默认的设置
- Cached: Cached IO模式，所有读操作都会缓存到cache memory中。
第四段: Write Cache OK if Bad BBU, No Write Cache if Bad BBU
- No Write Cache if Bad BBU: 如果BBU出问题，则关闭Write Cache。由WriteBack自动切换到WriteThrough模式。如果没有特殊要求，强烈建议采用该设置，以确保数据的安全。
- Write Cache OK if Bad BBU: 如果BBU出问题，依然启用Write Cache. 这是不推荐的设置，BBU出问题将无法保证断电情况下数据的正常，如果此时依然采用WriteBack模式，遇到断电将发生数据丢失。除非有UPS作额外保证，不然不推荐采用这个设置。
策略自动切换的问题
由于MegaSAS RAID卡默认采用No Write Cache if Bad BBU的设置，将可能发生Write Cache策略变更的情况(由WriteBack变成WriteThrough)，导致写性能下降，如果该自动变更发生在业务高峰且系统Io负载高的时候，可能会引发不可预测的问题，如卡机。以下原因将造成Write Cache策略的变更.
- RAID卡进入BBU Learn Cycle: 详细介绍见下面
- 检测到某些电池故障，如电池容量过低等，一般是电池老化带来的影响，IBM建议一年更换一次RAID卡电池
- 没有安装电池, 部分服务器购买时不带电池，导致被自动设置为WriteThrough
5、参考命令
命令使用：
#/opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL 查raid级别
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL 查raid卡信息
#/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL 查看硬盘信息
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -aAll 查看电池信息
#/opt/MegaRAID/MegaCli/MegaCli64 -FwTermLog -Dsply -aALL 查看raid卡日志
#/opt/MegaRAID/MegaCli/MegaCli64 -adpCount 【显示适配器个数】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpGetTime –aALL 【显示适配器时间】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aAll 【显示所有适配器信息】
#/opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -LALL -aAll 【显示所有逻辑磁盘组信息】
#/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aAll 【显示所有的物理信息】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL |grep ‘Charger Status’ 【查看充电状态】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL【显示BBU状态信息】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuCapacityInfo -aALL【显示BBU容量信息】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuDesignInfo -aALL 【显示BBU设计参数】
#/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -aALL 【显示当前BBU属性】
#/opt/MegaRAID/MegaCli/MegaCli64 -cfgdsply -aALL 【显示Raid卡型号，Raid设置，Disk相关信息】
3.磁带状态的变化，从拔盘，到插盘的过程中。
Device |Normal|Damage|Rebuild|Normal
Virtual Drive |Optimal|Degraded|Degraded|Optimal
Physical Drive |Online|Failed –> Unconfigured|Rebuild|Online
4.查看磁盘缓存策略
#/opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -Cache -L0 -a0
or
#/opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -Cache -L1 -a0
or
#/opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -Cache -LALL -a0
ro
#/opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -Cache -LALL -aALL
ro
#/opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -DskCache -LALL -aALL
5.设置磁盘缓存策略
缓存策略解释：
WT (Write through
WB (Write back)
NORA (No read ahead)
RA (Read ahead)
ADRA (Adaptive read ahead)
Cached
Direct
例子：
#/opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp WT|WB|NORA|RA|ADRA -L0 -a0
or
#/opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -Cached|-Direct -L0 -a0
or
enable / disable disk cache
#/opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -EnDskCache|-DisDskCache -L0 -a0
6.创建一个 raid5 阵列，由物理盘 2,3,4 构成，该阵列的热备盘是物理盘 5
#/opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r5 [1:2,1:3,1:4] WB Direct -Hsp[1:5] -a0
7.创建阵列，不指定热备
#/opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r5 [1:2,1:3,1:4] WB Direct -a0
8.删除阵列
#/opt/MegaRAID/MegaCli/MegaCli64 -CfgLdDel -L1 -a0
9.在线添加磁盘
#/opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -Start -r5 -Add -PhysDrv[1:4] -L1 -a0
10.阵列创建完后，会有一个初始化同步块的过程，可以看看其进度。
#/opt/MegaRAID/MegaCli/MegaCli64 -LDInit -ShowProg -LALL -aALL
或者以动态可视化文字界面显示
#/opt/MegaRAID/MegaCli/MegaCli64 -LDInit -ProgDsply -LALL -aALL
11.查看阵列后台初始化进度
#/opt/MegaRAID/MegaCli/MegaCli64 -LDBI -ShowProg -LALL -aALL
或者以动态可视化文字界面显示
#/opt/MegaRAID/MegaCli/MegaCli64 -LDBI -ProgDsply -LALL -aALL
12.指定第 5 块盘作为全局热备
#/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set [-EnclAffinity] [-nonRevertible] -PhysDrv[1:5] -a0
13.指定为某个阵列的专用热备
#/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set [-Dedicated [-Array1]] [-EnclAffinity] [-nonRevertible] -PhysDrv[1:5] -a0
14.删除全局热备
#/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Rmv -PhysDrv[1:5] -a0
15.将某块物理盘下线/上线
#/opt/MegaRAID/MegaCli/MegaCli64 -PDOffline -PhysDrv [1:4] -a0
#/opt/MegaRAID/MegaCli/MegaCli64 -PDOnline -PhysDrv [1:4] -a0
16.查看物理磁盘重建进度
#/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv [1:5] -a0
或者以动态可视化文字界面显示
#/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ProgDsply -PhysDrv [1:5] -a0
磁带状态的变化，从拔盘，到插盘的过程中：
Device |Normal|Damage|Rebuild|Normal
Virtual Drive |Optimal|Degraded|Degraded|Optimal
Physical Drive |Online|Failed –> Unconfigured|Rebuild|Online
6、查看磁盘信息：
[root@LancetServer MegaCli]# MegaCli64 -PDList -aALL
Adapter #0
Enclosure Device ID: 252
Slot Number: 0
Enclosure position: 0
Device Id: 15
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 557.861 GB [0x45bb9000 Sectors]
Firmware state: Online, Spun Up
SAS Address(0): 0x5000c5006c4ecf2d
SAS Address(1): 0x0
Connected Port Number: 3(path0)
Inquiry Data: IBM-ESXSST600MM0006 B56ES0M1D61F0328B56E
IBM FRU/CRU: 49Y2023
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive: Not Certified
Drive Temperature :34C (93.20 F)
Enclosure Device ID: 252
Slot Number: 1
Enclosure position: 0
Device Id: 14
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 557.861 GB [0x45bb9000 Sectors]
Firmware state: Online, Spun Up
SAS Address(0): 0x5000c5006c4eb775
SAS Address(1): 0x0
Connected Port Number: 2(path0)
Inquiry Data: IBM-ESXSST600MM0006 B56ES0M1DAFR0328B56E
IBM FRU/CRU: 49Y2023
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive: Not Certified
Drive Temperature :34C (93.20 F)
Enclosure Device ID: 252
Slot Number: 2
Enclosure position: 0
Device Id: 13
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 557.861 GB [0x45bb9000 Sectors]
Firmware state: Online, Spun Up
SAS Address(0): 0x5000c5006c4f2c79
SAS Address(1): 0x0
Connected Port Number: 1(path0)
Inquiry Data: IBM-ESXSST600MM0006 B56ES0M1D9DA0328B56E
IBM FRU/CRU: 49Y2023
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive: Not Certified
Drive Temperature :34C (93.20 F)
Enclosure Device ID: 252
Slot Number: 3
Enclosure position: 0
Device Id: 12
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 557.861 GB [0x45bb9000 Sectors]
Firmware state: Online, Spun Up
SAS Address(0): 0x5000c5006c4f1409
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: IBM-ESXSST600MM0006 B56ES0M1D9PX0328B56E
IBM FRU/CRU: 49Y2023
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive: Not Certified
Drive Temperature :30C (86.00 F)
Exit Code: 0x00
7、查看RAID卡日志
[root@LancetServer MegaCli]# MegaCli64 -FwTermLog -Dsply -aALL
Firmware Term Log Information on controller 0:
T0: LSI ROC firmware
T0: Copyright(C) LSI Corporation, 2010
T0: Firmware version 2.130.383-2854 built on Oct 10 2013 at 04:22:46
T0: *** HW Encryption Disabled : dcrReg=80000001
T0: setAdapterResetTime: CCR_MISC_CFG 8002ff2c
T0: DRAM_LOCAL_BASE: 80000000
T0: MEM_FIXED_SIZE: 5a00000
T0: FW_DRAM_REGION_START: 85a00000
T0: FW_DRAM_REGION_SIZE: 2200000
T0: MEM_POOL_BASE: 86df6348
T0: Initializing memory pool size=00E09CB8 bytes
T0: Press '!' within 3 seconds to enter debugger before INIT
T3: One Wire Device detected e1000005455ff233
T3: EepromInit: Family=33, SN=f25f45050000
T3: One Wire Device detected e1000005455ff233
T3: One Wire Device detected e1000005455ff233
T3: One Wire Device detected e1000005455ff233
T3: Authenticating RAID key: retryCount=0One Wire Device detected e1000005455ff233
T3: One Wire Device detected e1000005455ff233
T3: Done!
T3: CFGI firmware version 5.0.0.2
T3: Nvdata version 2.09.03-0046 built on Aug 08 2013 at 09:40:25
T3: LOAD section: src=fe560128, size=351e, dst=fe560058, mode=1...InflatNvdata - tempbuf = 87b6d7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b757c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b7d7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b857c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b8d7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b957c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87b9d7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87ba57c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bad7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bb57c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bbd7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bc57c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bcd7c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bd57c0, buf = 87bf57d0, len = =8000
T3: InflatNvdata - tempbuf = 87bdd7c0, buf = 87bf57d0, len = =4b08
T3: done
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=1
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=2
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=3
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=4
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=5
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=6
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=7
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=8
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=9
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=10
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=11
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=12
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=13
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=14
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=15
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=16
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=17
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=18
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=19
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=20
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=21
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=22
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=23
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=24
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=25
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=26
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=27
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=28
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=29
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=30
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=31
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=32
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=33
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=34
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=35
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=36
T3: DM_Cfg_InitConfigPages: Count of binaries found so far countNVDATA=37
T3: DM_Cfg_InitConfigPages: NVDATA page matched
T3: **********************************************************************
T3: iopiSODAlloc: size = 1a4, MemType = 1
T3: **********************************************************************
T3: iopiSODAlloc: size = 1ab0, MemType = 3
T4: Config Pages initialization completed. Persistent Nvdata version = a300f
T4: SRAM beginning and end for DM is xc0054400 xc0131400
T4: DM PL FRAME POOL INFO: plFrames=c0054400, plNonCritFrames=c00ca400 plCritFrames=c00d2400 iopNonCriticalFrames c00d4400
T4: Total memory allocated for DM PL Frames is : 84000
T4: PL MID #0: msgNm=00000000, addr=c0054400
T4: Address of dmIoCtx :: xc05e0880
T4: size of MFC Defaults structure in Bytes = 72
T4: Delaying POST by 60 seconds
T64: EVT#104468-T64: 0=Firmware initialization started (PCI ID 0079/1000/03b2/1014)
T64: EVT#104469-T64: 1=Firmware version 2.130.383-2854
T64: I2C 0 reset!
T64: I2Chandle obtained for MUX [0]0x0
T64: I2c 0 mux was reset!
T64: I2C 0 reset!
T64: I2C 1 reset!
T64: I2Chandle obtained for MUX [1]0x10
T64: I2c 0 mux was reset!
T64: I2C 0 reset!
T64: I2C 2 reset!
T64: I2Chandle obtained for MUX [2]0x20
T64: I2c 0 mux was reset!
T64: I2C 0 reset!
T64: SBR: I2Chandle obtained for SBR 21
T64: Raw SBR Image data:
T64: DRAM SIZE=512 MB
T64: Init flash timings : FlashTimeNs 78 NvsTimeNs 37
T64: setPhyMap: phy = 0 internalIndex = 0 externalIndex = ff
T64: setPhyMap: phy = 1 internalIndex = 0 externalIndex = ff
T64: setPhyMap: phy = 2 internalIndex = 0 externalIndex = ff
T64: setPhyMap: phy = 3 internalIndex = 0 externalIndex = ff
T64: setPhyMap: phy = 4 internalIndex = 1 externalIndex = ff
T64: setPhyMap: phy = 5 internalIndex = 1 externalIndex = ff
T64: setPhyMap: phy = 6 internalIndex = 1 externalIndex = ff
T64: setPhyMap: phy = 7 internalIndex = 1 externalIndex = ff
T64: setPhyMap: phy = 8 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 9 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = a internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = b internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = c internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = d internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = e internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = f internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 10 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 11 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 12 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 13 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 14 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 15 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 16 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 17 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 18 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 19 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1a internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1b internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1c internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1d internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1e internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 1f internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 20 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 21 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 22 internalIndex = ff externalIndex = ff
T64: setPhyMap: phy = 23 internalIndex = ff externalIndex = ff
T64: Environment data:
T64: onlineUpdateVersion=0000
T64: VERSIONS=APP_2.130.383-2854,BIOS_3.29.00_4.14.05.00_0x05290000,HIIM_01.38.00,PCLI_04.04-020:#%00009,BCON_6.0-52-e_48-Rel,NVDT_2.09.03-0046,BTBL_2.02.00.00-0000,BOOT_09.250.01.219
T64: PACKAGE=12.13.0-0179
T64: VALIDATION=GCA_10/10/13_05:08:31
T64: MFC data:
T64: vendorId/deviceId=1000/0079, subVendorId/subDeviceId=1014/03b2, OEM=9, SubOem=0, isRaidKeySecondary=0
T64: MFCF: clusterDisable=1, disableSAS=0, maxDisks=0, enableRaid6=0, disableWideCache=0
T64: disableRaid5=0, enableSecurity=0, enableReducedFeatureSet=0
T64: enableCTIO=0 enableSnapshot=0 enableSSC=0 enableSSCWB=0 enableCacheOffload=0
T64: MFCD: sasAddr=500605b00905bd10, phyPolarity=00 phyPolaritySplit 00
T64: backgroundRate=30, stripeSize=8(128K), flushTime=4
T64: writeBack=1, readAhead=0, cacheWhenBBUBad=0, cachedIo=0
T64: smartMode=0(6), alarmDisable=1, coercion=2(1G), zcrConfig=0(Undefined)
T64: dirtyLedShowsDriveActivity=0, biosContinueOnError=0, spindownMode=0(None)
T64: allowedDeviceTypes=0(SAS/SATA), allowMixInEnclosure=1, allowMixInLD=0, allowSataInCluster=0
T64: allowSSDMixInLD=0, allowMixSSDHDDInLD=0
T64: maxChainedEnclosures=16, disableCtrlR=1, enableWebBios=1, directPdMapping=0, biosEnumerateLds=1
T64: restoreHotSpareOnInsertion=0, exposeEnclosureDevices=1
T64: maintainPdFailHistory=1 disablePuncturing=1 zeroBasedEnclEnumeration=0 disableBootCLI=0
T64: quadPortConnectorMap=0 driveActivityLed=0 disableAutoDetectBackplane=0
T64: enableLedHeaders=0 useFdeOnly=1 delayPOST=4 enableCrashDump=0 enableLDBBM=0
T64: allowUnCertifiedHDDs=1 treatR1EAsR10=0, maxLdsPerArray=0, disableOnlineCtrlReset=0
T64: failPdOnSMARTer=0 nonRevertibleSpares=0,
T64: snapVDSpace=3, autoSnapVDSpace=8, viewSpace=4
T64: disablePowerSavings=16 spinDownTime=1e
T64: enableJBOD=0 ttyLogInFlash=0
T64: breakMirrorRAIDSupport=0, disableJoinMirror=0
T64: enableEmergencySpare=1 useGlobalSparesForEmergency=0 useUnconfGoodForEmergency=0
T64: enable512eSupport=0 disableHII=0
T64: MFC MFG data:
T64: date=06/24/14, sn="SV42622923", reworkDate=00/00/00, rev="79E"
T64: MFC MFG 2 data:
T64: Battery FRU=""
T64: disabledFeatures : 0
T64: can_flush = 0
T64: initFreeDDBs=xorPool c043c000-c044e000 genPool c0400000-c0418000
T64: initBladeSGL c0418000-c0438000
T64: C05F1350: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c20c0400
T64: C05F1370: c0038000 00000001 ffffffff ffffffff 00000000 00000000
T64: C05F1388: 00000001 00000000 00000000 00000000 00000000 00000000 00000000 c20c0800
T64: C05F13A8: c003a000 00000001 ffffffff ffffffff 00000000 00000000
T65: Controller Properties: (All values are in hex)
T65: seqNum=1, predFailPollInterval=12c, intThrottleCount=10
T65: intThrottleTimeUs=32, rebuildRate=1e, patrolReadRate=1e
T65: bgiRate=1e, ccRate=1e, reconRate=1e
T65: cacheFlushInterval=4, spinupDriveCount=4, spinupDelay=2
T65: clusterEnable=0, coercionMode=2, alarmEnable=0
T65: disableAutoRebuild=0, disableBatteryWarning=0, eccBucketSize=f
T65: eccBucketLeakRate=5a0, restoreHotSpareOnInsertion=0
T65: exposeEnclosureDevices=1, maintainPdFailHistory=1
T65: disallowHostRequestReordering=0
T65: abortCCOnError=0, copyBackDisabled=0, SMARTerEnabled=0
T65: prCorrectUnconfiguredAreas=1, useFdeOnly=1
T65: disableNCQ=0, SSDSMARTerEnabled=1, SSDPatrolReadEnabled=0
T65: enableSpinDownUnconfigured=0, disableSpinDownHS=1, spinDownTime=1e, autoEnhancedImport=0, enableSecretKeyControl=0, disableOnlineCtrlReset=0
T65: enableJBOD=0
T65: allowBootWithPinnedCache=0
T65: LoadBalanceEnabled=1, useDiskActivityForLocate=0
T65: useGlobalSparesForEmergency=0, useUnconfGoodForEmergency=0, useEmergencyForSMARTer=0
T65: forceSGPIOForQuadOnly=0
T65: OnOffProperties=0x0000102c autoSnapVDSpace=8, viewSpace=4, snapVDSpace=3
T65: defaultPSPolicy=fe autoPSPolicySupported=0
T65: maxPSPolicySupported=0 cachedWritesSupported=0
T65: disablePSTime=0 disablPSInterval=0
T65: spinupEnclDriveCount=4 spinupEnclDelay=6
T65: I2Chandle obtained for Ni-MH based BBU Charger 1
T65: I2Chandle obtained for Gas Guage 2
T65: I2Chandle obtained for Li-Ion based BBU Charger 3
T65: I2Chandle obtained for Li-Ion based BBU I2C Buffer 4
T65: I2Chandle obtained for impedance Tracking gas guage in iBBU08 5
T65: I2Chandle obtained for GPIO expander in iBBU08 6
T65: I2Chandle obtained for EEprom in expander in iBBU08 7
T65: I2Chandle obtained for I2C_SEP 8
T65: No Battery is Present
T65: EVT#104470-T65: 142=Battery Not Present
T65: PdInit: PD_INFO entries are 0x280 bytes
T65: powerSaveInit: Init structures
T65: EVT#104471-T65: 261=Package version 12.13.0-0179
T65: EVT#104472-T65: 266=Board Revision 79E
T65: EVT#104473-T65: 10=Controller cache discarded due to memory/battery problems
T65: ready_list=87c00000 (80000 elements)
T65: rebuild_list=88000000 (bc00 elements)
T65: lines_info=8805e000 (14d5ad elements)
T65: DISK_CACHE_ADDR=89f9e838
T65: MEM_END_ADDR=9ffffff0
T66: KM_KeyMgmtInitTRNG: security is not enabled
T66: Device Module Version 1.1 Date Oct 10 2013 Time 04:21:10
T66: phy = 0, connectorNo = 0, DASEPInfo[i].StartConnectorNo = 0
T66: phy = 4, connectorNo = 1, DASEPInfo[i].StartConnectorNo = 0
T66: QuadMapForDASEP[0] = 3 NoConnector = 2 StartConnectorNo = 0
T66: entry = 0 ConnectorIndex = 0 StartingPhy = 0 StartingSlot = 0
T66: entry = 1 ConnectorIndex = 1 StartingPhy = 0 StartingSlot = 4
T66: QuadMapForDASEP[1] = 0 NoConnector = 0 StartConnectorNo = 255
T66: QuadMapForDASEP[2] = 0 NoConnector = 0 StartConnectorNo = 255
T66: QuadMapForDASEP[3] = 0 NoConnector = 0 StartConnectorNo = 255
T66: QuadMapForDASEP[4] = 0 NoConnector = 0 StartConnectorNo = 255
T66: QuadMapForDASEP[5] = 0 NoConnector = 0 StartConnectorNo = 255
T66: PCI Device Info:
T66: vendorid = 0
T66: deviceid = 0
T66: bar0 = 1f01
T66: bar1 = 906bc004
T66: bar2 = 0
T66: bar3 = 906c0004
T66: bar4 = 0
T66: DM_ChipInit: Total chips=1
T66: Chip=0 vendorId 0000 deviceId 0000
T66: Pl firmware version 5.0.38.219
T66: PLMEM INFO : start address = 0xc00d8400 End=0xc0131400
T66: index 0x00, size 0x00004a80, Attrbute 0x11, Addr 0xc00d8400
T66: index 0x01, size 0x00000800, Attrbute 0x11, Addr 0xc00dce80
T66: index 0x02, size 0x00000010, Attrbute 0x11, Addr 0xc00dd680
T66: index 0x03, size 0x00005d20, Attrbute 0x11, Addr 0xc00dd6a0
T66: index 0x04, size 0x00001bf0, Attrbute 0x02, Addr 0xc04e33c0
T66: index 0x07, size 0x00006300, Attrbute 0x02, Addr 0xc04e4fc0
T66: index 0x08, size 0x00000254, Attrbute 0x12, Addr 0xc00eb2c0
T66: index 0x09, size 0x0000a500, Attrbute 0x12, Addr 0xc00eb520
T66: index 0x0a, size 0x00002100, Attrbute 0x12, Addr 0xc00f5a20
T66: index 0x0c, size 0x00000014, Attrbute 0x12, Addr 0xc00f7b20
T66: index 0x0d, size 0x00029400, Attrbute 0x12, Addr 0xc00f7b40
T66: index 0x0e, size 0x00005d20, Attrbute 0x03, Addr 0x87bf45e0
T66: index 0x0f, size 0x0000000c, Attrbute 0x04, Addr 0x87bf45a0
T66: index 0x10, size 0x000003b8, Attrbute 0x03, Addr 0x87bf41a0
T66: index 0x11, size 0x0000c940, Attrbute 0x03, Addr 0x87be7840
T66: index 0x12, size 0x000000cc, Attrbute 0x03, Addr 0x87be7740
T66: index 0x13, size 0x000000cc, Attrbute 0x03, Addr 0x87be7640
T66: index 0x14, size 0x000007bc, Attrbute 0x03, Addr 0x87be6e40
T66: index 0x15, size 0x00000042, Attrbute 0x03, Addr 0x87be6dc0
T66: index 0x16, size 0x00000400, Attrbute 0x03, Addr 0x87be69a0
T66: index 0x17, size 0x00000360, Attrbute 0x03, Addr 0x87be6600
T66: index 0x18, size 0x00000090, Attrbute 0x13, Addr 0x07be6540
T66: index 0x19, size 0x00000028, Attrbute 0x03, Addr 0x87be64e0
T66: index 0x1a, size 0x00000028, Attrbute 0x02, Addr 0xc0520f40
T66: index 0x1b, size 0x00000018, Attrbute 0x03, Addr 0x87be64a0
T66: index 0x1c, size 0x00000020, Attrbute 0x03, Addr 0x87be6440
T66: index 0x1d, size 0x00000074, Attrbute 0x12, Addr 0xc0120f80
T66: index 0x1e, size 0x00000334, Attrbute 0x14, Addr 0x07be60e0
T66: Total Memory allocated to PL: static = 00048bf4 Dyn=13eee total=5cae2
T66: Current head after allocating PL mem: c0120ff4
T66: SGL FRAME INFO: Start address in SRAM is : xc0121000
T66: DM_PL_AllocReqFrames: Max SGL frames in SRAM: 104
T66: From SRAM: SgInIo Frame xd SgInSg Frame x15 TotSg Frames x104 Frame Size x100 sgBaseAddr xc0521000
T66: Total memory allocated for SGL Frames in SRAM is : x10400
T66: Allocating left over x1efc SGL frames in DDR at address : x879f64a0
T66: From DDR: SgInIo Frame xd SgInSg Frame x15 TotSg Frames x1efc Frame Size x100 sgBaseAddr x879f64a0
T66: Waiting for First PL Discovery to complete
T67: Disabling UART for 120s due to discovery
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery started on Port 0
T67: iopiEvent: EVENT_SAS_ENCL_DEVICE_STATUS_CHANGE
T67: DM_HndlEnclStatusChgEvnt: EnclosureHandle x0001 Reason x01 PhysicalPort 0
T67: ELI 500605b0905bd10 NumSlots x0004 StartSlot x0000 PhyBits x00000000 IOCLogInfo x00000000 IOCStatus x0000
T67: Ctrl enclosure device added
T67: iopiEvent: MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST
T67: DM_HandleTopologyChgEvnt: PhysicalPort=0 NumberOfPhys=x08 NumEntries=x01 StartPhy=x0
T67: ExpStatus=x00 PhysicalPort=0 EnclosureHandle=x0001 Expander devHandle=x0000
T67: Device ADDED - phy x00 devHandle x0009 parentDevHdl x0001 phyNum x00 slot x03 deviceInfo=x00000c01 sasAddr=5000c5006c4f1409
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery started on Port 1
T67: iopiEvent: MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST
T67: DM_HandleTopologyChgEvnt: PhysicalPort=0 NumberOfPhys=x08 NumEntries=x01 StartPhy=x1
T67: ExpStatus=x00 PhysicalPort=0 EnclosureHandle=x0001 Expander devHandle=x0000
T67: Device ADDED - phy x01 devHandle x000a parentDevHdl x0002 phyNum x01 slot x02 deviceInfo=x00000c01 sasAddr=5000c5006c4f2c79
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery Completed on Port 1
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery started on Port 2
T67: iopiEvent: MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST
T67: DM_HandleTopologyChgEvnt: PhysicalPort=0 NumberOfPhys=x08 NumEntries=x01 StartPhy=x2
T67: ExpStatus=x00 PhysicalPort=0 EnclosureHandle=x0001 Expander devHandle=x0000
T67: Device ADDED - phy x02 devHandle x000b parentDevHdl x0003 phyNum x02 slot x01 deviceInfo=x00000c01 sasAddr=5000c5006c4eb775
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery Completed on Port 2
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery started on Port 3
T67: iopiEvent: MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST
T67: DM_HandleTopologyChgEvnt: PhysicalPort=0 NumberOfPhys=x08 NumEntries=x01 StartPhy=x3
T67: ExpStatus=x00 PhysicalPort=0 EnclosureHandle=x0001 Expander devHandle=x0000
T67: Device ADDED - phy x03 devHandle x000c parentDevHdl x0004 phyNum x03 slot x00 deviceInfo=x00000c01 sasAddr=5000c5006c4ecf2d
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery Completed on Port 3
T67: iopiEvent: EVENT_SAS_DISCOVERY
T67: DM_HandleDiscEvent: Discovery Completed on Port 0
T67: DM_DevMgrIsChipInit 0 State 1
T68: iopiEvent: EVENT_SAS_DISCOVERY
T68: DM_HandleDiscEvent: Discovery started on Port 0
T68: iopiEvent: MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST
T68: DM_HandleTopologyChgEvnt: PhysicalPort=0 NumberOfPhys=x08 NumEntries=x04 StartPhy=x4
T68: ExpStatus=x00 PhysicalPort=0 EnclosureHandle=x0001 Expander devHandle=x0000
T68: Phy changed - phy 04 devHandle 0000 linkRate 00 curLinkRate 0
T68: Phy changed - phy 05 devHandle 0000 linkRate 00 curLinkRate 0
T68: Phy changed - phy 06 devHandle 0000 linkRate 00 curLinkRate 0
T68: Phy changed - phy 07 devHandle 0000 linkRate 00 curLinkRate 0
T68: iopiDiscoveryComplete SubSystem 2 Count 0 InitState 0
T68: iopiEvent: EVENT_SAS_DISCOVERY
T68: DM_HandleDiscEvent: Discovery Completed on Port 0
T68: DM_DevMgrIsChipInit 0 State 1
T84: iopiDiscoveryComplete SubSystem 0 Count 0 InitState 0
T84: First discovery completed in 17 seconds
T84: MPT_SASIoUnitGetCurrentPage 0 :SASIoUnit = 87bfa620 size 150
T84: Hdr: ExtPageLength 0, ExtPageType 0 PageNumber 0
T84: PageType 0 PageVersion 0 NumPhys 8
T84: PHY0: Port=0, PortFlags=1, PhyFlags=0, NegRate=aa, CtrlPhyDevInf=71 AttDevHdl=9 CtrlDevHdl=1 DiscSts=0
T84: PHY1: Port=1, PortFlags=1, PhyFlags=0, NegRate=aa, CtrlPhyDevInf=71 AttDevHdl=a CtrlDevHdl=2 DiscSts=0
T84: PHY2: Port=2, PortFlags=1, PhyFlags=0, NegRate=aa, CtrlPhyDevInf=71 AttDevHdl=b CtrlDevHdl=3 DiscSts=0
T84: PHY3: Port=3, PortFlags=1, PhyFlags=0, NegRate=aa, CtrlPhyDevInf=71 AttDevHdl=c CtrlDevHdl=4 DiscSts=0
T84: PHY4: Port=0, PortFlags=1, PhyFlags=0, NegRate=00, CtrlPhyDevInf=71 AttDevHdl=0 CtrlDevHdl=1 DiscSts=0
T84: PHY5: Port=0, PortFlags=1, PhyFlags=0, NegRate=00, CtrlPhyDevInf=71 AttDevHdl=0 CtrlDevHdl=1 DiscSts=0
T84: PHY6: Port=0, PortFlags=1, PhyFlags=0, NegRate=00, CtrlPhyDevInf=71 AttDevHdl=0 CtrlDevHdl=1 DiscSts=0
T84: PHY7: Port=0, PortFlags=1, PhyFlags=0, NegRate=00, CtrlPhyDevInf=71 AttDevHdl=0 CtrlDevHdl=1 DiscSts=0
T84: DISM: Queued!
T84: DM_DevMgrIsChipInit 0 State 400
T84: DM_DevSSUCallback: DevId x000c errCount 0 Retry 0 Status 2
T84: DM_DevSSUCallback dev c Sense Data: Len 40 RespCode 70 senseKey 6 asc 29 ascq 1
T84: DM_DevSSUCallback: DevId x000e errCount 0 Retry 0 Status 2
T84: DM_DevSSUCallback dev e Sense Data: Len 40 RespCode 70 senseKey 6 asc 29 ascq 1
T84: DM_DevSSUCallback: DevId x000d errCount 0 Retry 0 Status 2
T84: DM_DevSSUCallback dev d Sense Data: Len 40 RespCode 70 senseKey 6 asc 29 ascq 1
T84: DM_DevSSUCallback: DevId x000f errCount 0 Retry 0 Status 2
T84: DM_DevSSUCallback dev f Sense Data: Len 40 RespCode 70 senseKey 6 asc 29 ascq 1
T84: devId 12 - slot num - 3
T84: devId 13 - slot num - 2
T84: devId 14 - slot num - 1
T84: devId 15 - slot num - 0
T84: DM_DevNotifyRAID: Notify Done. Check for Removal
T84: DISM Complete
T84: gDevInfo=86a047e0, size=160
T84: Total Device = 4
T84: PD Flags State Type Size S N Vendor Product Rev P C ID SAS Addr Port Phy DevH BFw BRev
T84: --- -------- ----- ---- -------- - - -------- ---------------- ---- - - -- ---------------- ---- --- ---- ---- ----
T84: c f1400005 00020 00 45dd2faf 0 0 0 IBM-ESXS ST600MM0006 B56E 0 0 09 5000c5006c4f1409 00 00 09 NA NA
T84: d f1400005 00020 00 45dd2faf 0 0 0 IBM-ESXS ST600MM0006 B56E 0 0 0a 5000c5006c4f2c79 01 01 0a NA NA
T84: e f1400005 00020 00 45dd2faf 0 0 0 IBM-ESXS ST600MM0006 B56E 0 0 0b 5000c5006c4eb775 02 02 0b NA NA
T84: f f1400005 00020 00 45dd2faf 0 0 0 IBM-ESXS ST600MM0006 B56E 0 0 0c 5000c5006c4ecf2d 03 03 0c NA NA
T84: 100 00400005 00020 03 0 0 0 0 LSI SMP/SGPIO/SEP 2854 0 0 ffff 0 00 ff 00 NA NA
T84: PhyId 0 Sas 5000c5006c4f1409 Type 1 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 5000c5006c4f2c79 Type 1 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 5000c5006c4eb775 Type 1 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 5000c5006c4ecf2d Type 1 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
T84: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
T84: Load Balance Statistics Path0PDs 4 Path1PDs 0
T84: DM_DevMgr become Ready- state 15
T84: EnclSgpioAdd: find ICO_SGPIO type Numslot=4, EnclHandle=1, StartSlot=0
T84: SGPIO slot mapping pd=c is mapped to slot=3
T84: EVT#104474-T84: 91=Inserted: PD 0c(e0xff/s3)
T84: EVT#104475-T84: 247=Inserted: PD 0c(e0xff/s3) Info: enclPd=ffff, scsiType=0, portMap=00, sasAddr=5000c5006c4f1409,0000000000000000
T84: EVT#104476-T84: 285=PD 0c(e0xff/s3) FRU is 49Y2023
T84: EVT#104477-T84: 91=Inserted: PD 0d(e0xff/s2)
T84: EVT#104478-T84: 247=Inserted: PD 0d(e0xff/s2) Info: enclPd=ffff, scsiType=0, portMap=01, sasAddr=5000c5006c4f2c79,0000000000000000
T84: EVT#104479-T84: 285=PD 0d(e0xff/s2) FRU is 49Y2023
T84: EVT#104480-T84: 91=Inserted: PD 0e(e0xff/s1)
T84: EVT#104481-T84: 247=Inserted: PD 0e(e0xff/s1) Info: enclPd=ffff, scsiType=0, portMap=02, sasAddr=5000c5006c4eb775,0000000000000000
T84: EVT#104482-T84: 285=PD 0e(e0xff/s1) FRU is 49Y2023
T84: EVT#104483-T84: 91=Inserted: PD 0f(e0xff/s0)
T84: EVT#104484-T84: 247=Inserted: PD 0f(e0xff/s0) Info: enclPd=ffff, scsiType=0, portMap=03, sasAddr=5000c5006c4ecf2d,0000000000000000
T84: EVT#104485-T84: 285=PD 0f(e0xff/s0) FRU is 49Y2023
T84: SGPIO slot mapping pd=d is mapped to slot=2
T84: SGPIO slot mapping pd=e is mapped to slot=1
T84: SGPIO slot mapping pd=f is mapped to slot=0
T84: StaggeredSpinUp complete
T84: Mfi init: MA dest FIFO: c004e000 firstFrame: 00400000 LastFrame: 00410000
T6532: KM_KeyMgmtInit: security is not enabled
T6532: KM_GetRandomNumber: security is not enabled
T6532: DDF_ConfigRead: perfect match found at pd=c, cod=87924080
T6532: DDF_ConfigRead: reference found at pd=c, merging config
T6532: DDF_Merge: copying bvd record for ld=1, span=0 from pd=c at partition=0
T6533: DDF_Merge: copying bvd record for ld=0, span=0 from pd=d at partition=0
T6533: missingSpan =0
T6533: missingSpan =0
T6533: ddfInitStatus: 0
T6533: configRaid: adding to raid config
T6533: configRaidAddSpan: adding ld=0 span=0
T6533: configRaidAddSpan: adding ld=1 span=0
T6533: ddfInitStatus after CheckForOfflinePRL11VDs : 0
T6533: DDF_PopulateCCRec cachecade lds=0
T6533: signature=10adc0ee crc=0 numCCLds=0
T6533: Missing src tid's=0 offline src tid's=0 missing cc tid's=0 offline cc tid's=0
T6533: No secondary cache found
T6533: wjMDInit: Initializing meta data write journal entries
T6533: rollBackActiveTargetIds=0 ,missingRepTargetIds=0
T6533: init_cache: recoverCache = 0 pinned_cache_present = 0
T6533: Memory End 9ffffff0
T6533: Total memory available for disk cache: 15f60c40
T6533: Total Number of Cache Lines 2792
T6533: SS 256: ss=8 mrs=3 lc=2792 ldc=2 ps=3 cm=ff ba=0 LDs: 0 1
T6533: Cache initialization complete...Config info
T6533: ARs configured: 2
T6533: Ar 0: size=45bb9000, activeLd=1, numPd=3, seq=0, powerSave=0 powerSaveInProgress=0 PDs: 0f(fc/0) 0e(fc/1) 0d(fc/2)
T6533: Ar 1: size=45bb9000, activeLd=2, numPd=1, seq=0, powerSave=0 powerSaveInProgress=0 PDs: 0c(fc/3)
T6533: LD 0: L=5 SS=256 Size=8b772000 NL=2792 Status=3 DT=1228 BT=1942, Encr=0, prop=[ID=00,dcp=00,ccp=00,ap=0,dc=2,dbgi=0,S=0|0,dps=fe,cps=fe]
T6533: span 0: sBlk=00000000, nBlk=45bb9000, ar=0
T6533: LD 1: L=0 SS=256 Size=45bb9000 NL=2792 Status=3 DT=1228 BT=1942, Encr=0, prop=[ID=01,dcp=00,ccp=00,ap=0,dc=2,dbgi=0,S=0|0,dps=fe,cps=fe]
T6533: span 0: sBlk=00000000, nBlk=45bb9000, ar=1
T6533: max-flush (Ar 0): 0x30
T6533: max-flush (Ar 1): 0x30
T6533: setup_cache_headers: ***** totalDirtyLines: 0 recoverCache: 0
T6533: pdFailHistInit: count=0
T6533: pdFailHistInit: count=0
T6533: ptrSscMissingAndOfflineVdInfo->offlineTargetIds = 0
T6533: ptrSscMissingAndOfflineVdInfo->missingTargetIds = 0
T6533: ptrSscMissingAndOfflineVdInfo->onlineTargetIds = 3
T6533: ssd_block_cacheable_vds_at_bootup_and_foreignimport: a
T6533: ssd_unblock_cacheable_vds: ptrSscMissingAndOfflineVdInfo->onlineTargetIds = 3raid_task:nvdimm:before flush enable the backup
T6533: Initializing write journal entries
T6533: NOT Flushing Cache
T6533: RMW: NVRAM structure valid - checking for active RMWs
T6533: total_current_windows = 0
T6533: EVT#104486-T6533: 195=BBU disabled; changing WB virtual disks to WT, Forced WB VDs are not affected
09/15/20 13:59:44: EVT#104487-09/15/20 13:59:44: 44=Time established as 09/15/20 13:59:44; (6533 seconds since power on)
09/15/20 14:00:29: Unable to get driver version my_timer_count=19b2
09/15/20 14:00:29: MSI-X Enabled: MSIX offset = 00002000
09/15/20 14:00:29: MSIX[0]: 00000000:fee00498 - 0
09/15/20 14:00:29: MSIX[1]: 00000000:00000000 - 0
09/15/20 14:00:29: MSIX[2]: 00000000:00000000 - 0
09/15/20 14:00:29: MSIX[3]: 00000000:00000000 - 0
09/15/20 14:00:29: findMsiState: Setting MSI Mode to 2 (0=Legacy, 1=MSI, 2=MSI-X)
09/15/20 14:00:29: EVT#104488-09/15/20 14:00:29: 389=Host driver is loaded and operational
09/15/20 14:00:29: MfiCmdInitQueue[0]: FW now OPERATIONAL
09/15/20 14:00:29: q.flags.mfaIs64Bits=0, q.flags.contextIs64Bits=0
09/15/20 14:00:29: q.responseQueueEntries=3f0, responseQueueStatr=755ab000
09/15/20 14:00:29: q.producerIndexPtr=75418000, q.consumerIndexPtr=75419000
09/15/20 14:00:29: producerIndex=0
09/19/20 3:00:00: prDiskStart: starting Patrol Read on PD=0c
09/19/20 3:00:00: prDiskStart: starting Patrol Read on PD=0d
09/19/20 3:00:00: prDiskStart: starting Patrol Read on PD=0e
09/19/20 3:00:00: prDiskStart: starting Patrol Read on PD=0f
09/19/20 3:00:00: EVT#104489-09/19/20 3:00:00: 39=Patrol Read started
09/19/20 3:00:00: EVT#104490-09/19/20 3:00:00: 66=Consistency Check started on VD 00/0
09/19/20 3:00:00: prDiskCheckOkToRun: PR cannot run on this pd=d init or CC running ld=0
09/19/20 3:00:00: prCallback: PR being stopped for pd=0d - state changed
09/19/20 3:00:00: prDiskCheckOkToRun: PR cannot run on this pd=e init or CC running ld=0
09/19/20 3:00:00: prCallback: PR being stopped for pd=0e - state changed
09/19/20 3:00:00: prDiskCheckOkToRun: PR cannot run on this pd=f init or CC running ld=0
09/19/20 3:00:00: prCallback: PR being stopped for pd=0f - state changed
09/19/20 3:00:18: EVT#104491-09/19/20 3:00:18: 65=Consistency Check progress on VD 00/0 is 0.24%(18s)
09/19/20 3:00:34: EVT#104492-09/19/20 3:00:34: 65=Consistency Check progress on VD 00/0 is 0.57%(34s)
09/19/20 3:00:50: EVT#104493-09/19/20 3:00:50: 65=Consistency Check progress on VD 00/0 is 0.78%(50s)
09/19/20 3:01:06: EVT#104494-09/19/20 3:01:06: 65=Consistency Check progress on VD 00/0 is 0.99%(66s)
09/19/20 3:01:22: EVT#104495-09/19/20 3:01:22: 65=Consistency Check progress on VD 00/0 is 1.30%(82s)
09/19/20 3:01:38: EVT#104496-09/19/20 3:01:38: 65=Consistency Check progress on VD 00/0 is 1.65%(98s)
09/19/20 3:01:54: EVT#104497-09/19/20 3:01:54: 65=Consistency Check progress on VD 00/0 is 2.00%(114s)
09/19/20 3:02:10: EVT#104498-09/19/20 3:02:10: 65=Consistency Check progress on VD 00/0 is 2.26%(130s)
09/19/20 3:02:26: EVT#104499-09/19/20 3:02:26: 65=Consistency Check progress on VD 00/0 is 2.63%(146s)
09/19/20 3:02:42: EVT#104500-09/19/20 3:02:42: 65=Consistency Check progress on VD 00/0 is 2.94%(162s)
09/19/20 3:02:58: EVT#104501-09/19/20 3:02:58: 65=Consistency Check progress on VD 00/0 is 3.26%(178s)
09/19/20 3:03:14: EVT#104502-09/19/20 3:03:14: 65=Consistency Check progress on VD 00/0 is 3.68%(194s)
09/19/20 3:03:30: EVT#104503-09/19/20 3:03:30: 65=Consistency Check progress on VD 00/0 is 4.16%(210s)
09/19/20 3:03:46: EVT#104504-09/19/20 3:03:46: 65=Consistency Check progress on VD 00/0 is 4.59%(226s)
09/19/20 3:04:02: EVT#104505-09/19/20 3:04:02: 65=Consistency Check progress on VD 00/0 is 4.99%(242s)
09/19/20 3:04:18: EVT#104506-09/19/20 3:04:18: 65=Consistency Check progress on VD 00/0 is 5.35%(258s)
09/19/20 3:04:34: EVT#104507-09/19/20 3:04:34: 65=Consistency Check progress on VD 00/0 is 5.76%(274s)
09/19/20 3:04:50: EVT#104508-09/19/20 3:04:50: 65=Consistency Check progress on VD 00/0 is 6.19%(290s)
09/19/20 3:06:13: EVT#104509-09/19/20 3:06:13: 65=Consistency Check progress on VD 00/0 is 6.21%(373s)
09/19/20 3:07:38: EVT#104510-09/19/20 3:07:38: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 10.00%(458s)
09/19/20 3:14:25: EVT#104511-09/19/20 3:14:25: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 20.00%(865s)
09/19/20 3:20:13: EVT#104512-09/19/20 3:20:13: 65=Consistency Check progress on VD 00/0 is 6.46%(1213s)
09/19/20 3:22:26: EVT#104513-09/19/20 3:22:26: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 30.00%(1346s)
09/19/20 3:23:02: EVT#104514-09/19/20 3:23:02: 65=Consistency Check progress on VD 00/0 is 6.71%(1382s)
09/19/20 3:23:18: EVT#104515-09/19/20 3:23:18: 65=Consistency Check progress on VD 00/0 is 7.17%(1398s)
09/19/20 3:23:34: EVT#104516-09/19/20 3:23:34: 65=Consistency Check progress on VD 00/0 is 7.63%(1414s)
09/19/20 3:23:50: EVT#104517-09/19/20 3:23:50: 65=Consistency Check progress on VD 00/0 is 8.07%(1430s)
09/19/20 3:24:06: EVT#104518-09/19/20 3:24:06: 65=Consistency Check progress on VD 00/0 is 8.49%(1446s)
09/19/20 3:24:22: EVT#104519-09/19/20 3:24:22: 65=Consistency Check progress on VD 00/0 is 8.97%(1462s)
09/19/20 3:24:38: EVT#104520-09/19/20 3:24:38: 65=Consistency Check progress on VD 00/0 is 9.43%(1478s)
09/19/20 3:24:54: EVT#104521-09/19/20 3:24:54: 65=Consistency Check progress on VD 00/0 is 9.88%(1494s)
09/19/20 3:25:10: EVT#104522-09/19/20 3:25:10: 65=Consistency Check progress on VD 00/0 is 10.30%(1510s)
09/19/20 3:25:26: EVT#104523-09/19/20 3:25:26: 65=Consistency Check progress on VD 00/0 is 10.78%(1526s)
09/19/20 3:25:42: EVT#104524-09/19/20 3:25:42: 65=Consistency Check progress on VD 00/0 is 11.25%(1542s)
09/19/20 3:25:58: EVT#104525-09/19/20 3:25:58: 65=Consistency Check progress on VD 00/0 is 11.70%(1558s)
09/19/20 3:26:14: EVT#104526-09/19/20 3:26:14: 65=Consistency Check progress on VD 00/0 is 12.18%(1574s)
09/19/20 3:26:30: EVT#104527-09/19/20 3:26:30: 65=Consistency Check progress on VD 00/0 is 12.64%(1590s)
09/19/20 3:26:46: EVT#104528-09/19/20 3:26:46: 65=Consistency Check progress on VD 00/0 is 13.12%(1606s)
09/19/20 3:27:02: EVT#104529-09/19/20 3:27:02: 65=Consistency Check progress on VD 00/0 is 13.56%(1622s)
09/19/20 3:27:18: EVT#104530-09/19/20 3:27:18: 65=Consistency Check progress on VD 00/0 is 14.01%(1638s)
09/19/20 3:27:34: EVT#104531-09/19/20 3:27:34: 65=Consistency Check progress on VD 00/0 is 14.48%(1654s)
09/19/20 3:27:50: EVT#104532-09/19/20 3:27:50: 65=Consistency Check progress on VD 00/0 is 14.87%(1670s)
09/19/20 3:28:06: EVT#104533-09/19/20 3:28:06: 65=Consistency Check progress on VD 00/0 is 15.28%(1686s)
09/19/20 3:28:22: EVT#104534-09/19/20 3:28:22: 65=Consistency Check progress on VD 00/0 is 15.73%(1702s)
09/19/20 3:28:38: EVT#104535-09/19/20 3:28:38: 65=Consistency Check progress on VD 00/0 is 16.00%(1718s)
09/19/20 3:28:54: EVT#104536-09/19/20 3:28:54: 65=Consistency Check progress on VD 00/0 is 16.17%(1734s)
09/19/20 3:29:10: EVT#104537-09/19/20 3:29:10: 65=Consistency Check progress on VD 00/0 is 16.49%(1750s)
09/19/20 3:29:26: EVT#104538-09/19/20 3:29:26: 65=Consistency Check progress on VD 00/0 is 16.98%(1766s)
09/19/20 3:29:42: EVT#104539-09/19/20 3:29:42: 65=Consistency Check progress on VD 00/0 is 17.38%(1782s)
09/19/20 3:29:58: EVT#104540-09/19/20 3:29:58: 65=Consistency Check progress on VD 00/0 is 17.81%(1798s)
09/19/20 3:30:10: EVT#104541-09/19/20 3:30:10: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 40.00%(1810s)
09/19/20 3:30:14: EVT#104542-09/19/20 3:30:14: 65=Consistency Check progress on VD 00/0 is 18.19%(1814s)
09/19/20 3:30:30: EVT#104543-09/19/20 3:30:30: 65=Consistency Check progress on VD 00/0 is 18.60%(1830s)
09/19/20 3:30:46: EVT#104544-09/19/20 3:30:46: 65=Consistency Check progress on VD 00/0 is 19.05%(1846s)
09/19/20 3:31:02: EVT#104545-09/19/20 3:31:02: 65=Consistency Check progress on VD 00/0 is 19.48%(1862s)
09/19/20 3:31:18: EVT#104546-09/19/20 3:31:18: 65=Consistency Check progress on VD 00/0 is 19.96%(1878s)
09/19/20 3:31:34: EVT#104547-09/19/20 3:31:34: 65=Consistency Check progress on VD 00/0 is 20.36%(1894s)
09/19/20 3:31:50: EVT#104548-09/19/20 3:31:50: 65=Consistency Check progress on VD 00/0 is 20.83%(1910s)
09/19/20 3:32:06: EVT#104549-09/19/20 3:32:06: 65=Consistency Check progress on VD 00/0 is 21.25%(1926s)
09/19/20 3:32:22: EVT#104550-09/19/20 3:32:22: 65=Consistency Check progress on VD 00/0 is 21.68%(1942s)
09/19/20 3:32:38: EVT#104551-09/19/20 3:32:38: 65=Consistency Check progress on VD 00/0 is 22.13%(1958s)
09/19/20 3:32:54: EVT#104552-09/19/20 3:32:54: 65=Consistency Check progress on VD 00/0 is 22.59%(1974s)
09/19/20 3:33:10: EVT#104553-09/19/20 3:33:10: 65=Consistency Check progress on VD 00/0 is 23.01%(1990s)
09/19/20 3:33:26: EVT#104554-09/19/20 3:33:26: 65=Consistency Check progress on VD 00/0 is 23.44%(2006s)
09/19/20 3:33:42: EVT#104555-09/19/20 3:33:42: 65=Consistency Check progress on VD 00/0 is 23.88%(2022s)
09/19/20 3:33:58: EVT#104556-09/19/20 3:33:58: 65=Consistency Check progress on VD 00/0 is 24.28%(2038s)
09/19/20 3:34:14: EVT#104557-09/19/20 3:34:14: 65=Consistency Check progress on VD 00/0 is 24.70%(2054s)
09/19/20 3:34:30: EVT#104558-09/19/20 3:34:30: 65=Consistency Check progress on VD 00/0 is 25.11%(2070s)
09/19/20 3:34:46: EVT#104559-09/19/20 3:34:46: 65=Consistency Check progress on VD 00/0 is 25.57%(2086s)
09/19/20 3:35:02: EVT#104560-09/19/20 3:35:02: 65=Consistency Check progress on VD 00/0 is 25.99%(2102s)
09/19/20 3:35:18: EVT#104561-09/19/20 3:35:18: 65=Consistency Check progress on VD 00/0 is 26.46%(2118s)
09/19/20 3:35:34: EVT#104562-09/19/20 3:35:34: 65=Consistency Check progress on VD 00/0 is 26.88%(2134s)
09/19/20 3:35:50: EVT#104563-09/19/20 3:35:50: 65=Consistency Check progress on VD 00/0 is 27.27%(2150s)
09/19/20 3:36:06: EVT#104564-09/19/20 3:36:06: 65=Consistency Check progress on VD 00/0 is 27.67%(2166s)
09/19/20 3:36:22: EVT#104565-09/19/20 3:36:22: 65=Consistency Check progress on VD 00/0 is 28.06%(2182s)
09/19/20 3:36:38: EVT#104566-09/19/20 3:36:38: 65=Consistency Check progress on VD 00/0 is 28.49%(2198s)
09/19/20 3:36:54: EVT#104567-09/19/20 3:36:54: 65=Consistency Check progress on VD 00/0 is 28.92%(2214s)
09/19/20 3:37:10: EVT#104568-09/19/20 3:37:10: 65=Consistency Check progress on VD 00/0 is 29.32%(2230s)
09/19/20 3:37:26: EVT#104569-09/19/20 3:37:26: 65=Consistency Check progress on VD 00/0 is 29.76%(2246s)
09/19/20 3:37:42: EVT#104570-09/19/20 3:37:42: 65=Consistency Check progress on VD 00/0 is 30.18%(2262s)
09/19/20 3:37:58: EVT#104571-09/19/20 3:37:58: 65=Consistency Check progress on VD 00/0 is 30.55%(2278s)
09/19/20 3:38:14: EVT#104572-09/19/20 3:38:14: 65=Consistency Check progress on VD 00/0 is 30.93%(2294s)
09/19/20 3:38:30: EVT#104573-09/19/20 3:38:30: 65=Consistency Check progress on VD 00/0 is 31.33%(2310s)
09/19/20 3:38:46: EVT#104574-09/19/20 3:38:46: 65=Consistency Check progress on VD 00/0 is 31.55%(2326s)
09/19/20 3:39:02: EVT#104575-09/19/20 3:39:02: 65=Consistency Check progress on VD 00/0 is 31.83%(2342s)
09/19/20 3:39:18: EVT#104576-09/19/20 3:39:18: 65=Consistency Check progress on VD 00/0 is 32.15%(2358s)
09/19/20 3:39:34: EVT#104577-09/19/20 3:39:34: 65=Consistency Check progress on VD 00/0 is 32.58%(2374s)
09/19/20 3:39:50: EVT#104578-09/19/20 3:39:50: 65=Consistency Check progress on VD 00/0 is 32.96%(2390s)
09/19/20 3:40:06: EVT#104579-09/19/20 3:40:06: 65=Consistency Check progress on VD 00/0 is 33.38%(2406s)
09/19/20 3:40:22: EVT#104580-09/19/20 3:40:22: 65=Consistency Check progress on VD 00/0 is 33.84%(2422s)
09/19/20 3:40:38: EVT#104581-09/19/20 3:40:38: 65=Consistency Check progress on VD 00/0 is 34.29%(2438s)
09/19/20 3:40:39: EVT#104582-09/19/20 3:40:39: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 49.99%(2439s)
09/19/20 3:40:55: EVT#104583-09/19/20 3:40:55: 65=Consistency Check progress on VD 00/0 is 34.72%(2454s)
09/19/20 3:41:11: EVT#104584-09/19/20 3:41:11: 65=Consistency Check progress on VD 00/0 is 35.14%(2470s)
09/19/20 3:41:27: EVT#104585-09/19/20 3:41:27: 65=Consistency Check progress on VD 00/0 is 35.59%(2486s)
09/19/20 3:41:43: EVT#104586-09/19/20 3:41:43: 65=Consistency Check progress on VD 00/0 is 35.98%(2502s)
09/19/20 3:41:59: EVT#104587-09/19/20 3:41:59: 65=Consistency Check progress on VD 00/0 is 36.36%(2518s)
09/19/20 3:42:15: EVT#104588-09/19/20 3:42:15: 65=Consistency Check progress on VD 00/0 is 36.79%(2534s)
09/19/20 3:42:31: EVT#104589-09/19/20 3:42:31: 65=Consistency Check progress on VD 00/0 is 37.21%(2550s)
09/19/20 3:42:47: EVT#104590-09/19/20 3:42:47: 65=Consistency Check progress on VD 00/0 is 37.65%(2566s)
09/19/20 3:43:03: EVT#104591-09/19/20 3:43:03: 65=Consistency Check progress on VD 00/0 is 38.06%(2582s)
09/19/20 3:43:19: EVT#104592-09/19/20 3:43:19: 65=Consistency Check progress on VD 00/0 is 38.42%(2598s)
09/19/20 3:43:35: EVT#104593-09/19/20 3:43:35: 65=Consistency Check progress on VD 00/0 is 38.86%(2614s)
09/19/20 3:43:51: EVT#104594-09/19/20 3:43:51: 65=Consistency Check progress on VD 00/0 is 39.26%(2630s)
09/19/20 3:44:07: EVT#104595-09/19/20 3:44:07: 65=Consistency Check progress on VD 00/0 is 39.64%(2646s)
09/19/20 3:44:23: EVT#104596-09/19/20 3:44:23: 65=Consistency Check progress on VD 00/0 is 40.08%(2662s)
09/19/20 3:44:39: EVT#104597-09/19/20 3:44:39: 65=Consistency Check progress on VD 00/0 is 40.48%(2678s)
09/19/20 3:44:55: EVT#104598-09/19/20 3:44:55: 65=Consistency Check progress on VD 00/0 is 40.89%(2694s)
09/19/20 3:45:11: EVT#104599-09/19/20 3:45:11: 65=Consistency Check progress on VD 00/0 is 41.29%(2710s)
09/19/20 3:45:27: EVT#104600-09/19/20 3:45:27: 65=Consistency Check progress on VD 00/0 is 41.73%(2726s)
09/19/20 3:45:43: EVT#104601-09/19/20 3:45:43: 65=Consistency Check progress on VD 00/0 is 42.11%(2742s)
09/19/20 3:45:59: EVT#104602-09/19/20 3:45:59: 65=Consistency Check progress on VD 00/0 is 42.52%(2758s)
09/19/20 3:46:15: EVT#104603-09/19/20 3:46:15: 65=Consistency Check progress on VD 00/0 is 42.95%(2774s)
09/19/20 3:46:31: EVT#104604-09/19/20 3:46:31: 65=Consistency Check progress on VD 00/0 is 43.37%(2790s)
09/19/20 3:46:47: EVT#104605-09/19/20 3:46:47: 65=Consistency Check progress on VD 00/0 is 43.79%(2806s)
09/19/20 3:47:03: EVT#104606-09/19/20 3:47:03: 65=Consistency Check progress on VD 00/0 is 44.18%(2822s)
09/19/20 3:47:19: EVT#104607-09/19/20 3:47:19: 65=Consistency Check progress on VD 00/0 is 44.56%(2838s)
09/19/20 3:47:35: EVT#104608-09/19/20 3:47:35: 65=Consistency Check progress on VD 00/0 is 44.99%(2854s)
09/19/20 3:47:51: EVT#104609-09/19/20 3:47:51: 65=Consistency Check progress on VD 00/0 is 45.34%(2870s)
09/19/20 3:48:07: EVT#104610-09/19/20 3:48:07: 65=Consistency Check progress on VD 00/0 is 45.72%(2886s)
09/19/20 3:48:23: EVT#104611-09/19/20 3:48:23: 65=Consistency Check progress on VD 00/0 is 46.11%(2902s)
09/19/20 3:48:39: EVT#104612-09/19/20 3:48:39: 65=Consistency Check progress on VD 00/0 is 46.46%(2918s)
09/19/20 3:48:55: EVT#104613-09/19/20 3:48:55: 65=Consistency Check progress on VD 00/0 is 46.66%(2934s)
09/19/20 3:49:11: EVT#104614-09/19/20 3:49:11: 65=Consistency Check progress on VD 00/0 is 46.94%(2950s)
09/19/20 3:49:27: EVT#104615-09/19/20 3:49:27: 65=Consistency Check progress on VD 00/0 is 47.35%(2966s)
09/19/20 3:49:43: EVT#104616-09/19/20 3:49:43: 65=Consistency Check progress on VD 00/0 is 47.75%(2982s)
09/19/20 3:49:59: EVT#104617-09/19/20 3:49:59: 65=Consistency Check progress on VD 00/0 is 48.15%(2998s)
09/19/20 3:50:15: EVT#104618-09/19/20 3:50:15: 65=Consistency Check progress on VD 00/0 is 48.53%(3014s)
09/19/20 3:50:31: EVT#104619-09/19/20 3:50:31: 65=Consistency Check progress on VD 00/0 is 48.94%(3030s)
09/19/20 3:50:47: EVT#104620-09/19/20 3:50:47: 65=Consistency Check progress on VD 00/0 is 49.29%(3046s)
09/19/20 3:50:52: EVT#104621-09/19/20 3:50:52: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 60.00%(3052s)
09/19/20 3:51:03: EVT#104622-09/19/20 3:51:03: 65=Consistency Check progress on VD 00/0 is 49.65%(3062s)
09/19/20 3:51:19: EVT#104623-09/19/20 3:51:19: 65=Consistency Check progress on VD 00/0 is 50.02%(3078s)
09/19/20 3:51:35: EVT#104624-09/19/20 3:51:35: 65=Consistency Check progress on VD 00/0 is 50.41%(3094s)
09/19/20 3:51:51: EVT#104625-09/19/20 3:51:51: 65=Consistency Check progress on VD 00/0 is 50.80%(3110s)
09/19/20 3:52:07: EVT#104626-09/19/20 3:52:07: 65=Consistency Check progress on VD 00/0 is 51.16%(3126s)
09/19/20 3:52:23: EVT#104627-09/19/20 3:52:23: 65=Consistency Check progress on VD 00/0 is 51.55%(3142s)
09/19/20 3:52:39: EVT#104628-09/19/20 3:52:39: 65=Consistency Check progress on VD 00/0 is 51.90%(3158s)
09/19/20 3:52:55: EVT#104629-09/19/20 3:52:55: 65=Consistency Check progress on VD 00/0 is 52.28%(3174s)
09/19/20 3:53:11: EVT#104630-09/19/20 3:53:11: 65=Consistency Check progress on VD 00/0 is 52.68%(3190s)
09/19/20 3:53:27: EVT#104631-09/19/20 3:53:27: 65=Consistency Check progress on VD 00/0 is 53.09%(3206s)
09/19/20 3:53:43: EVT#104632-09/19/20 3:53:43: 65=Consistency Check progress on VD 00/0 is 53.51%(3222s)
09/19/20 3:53:59: EVT#104633-09/19/20 3:53:59: 65=Consistency Check progress on VD 00/0 is 53.89%(3238s)
09/19/20 3:54:15: EVT#104634-09/19/20 3:54:15: 65=Consistency Check progress on VD 00/0 is 54.31%(3254s)
09/19/20 3:54:31: EVT#104635-09/19/20 3:54:31: 65=Consistency Check progress on VD 00/0 is 54.72%(3270s)
09/19/20 3:54:47: EVT#104636-09/19/20 3:54:47: 65=Consistency Check progress on VD 00/0 is 55.13%(3286s)
09/19/20 3:55:03: EVT#104637-09/19/20 3:55:03: 65=Consistency Check progress on VD 00/0 is 55.49%(3302s)
09/19/20 3:55:19: EVT#104638-09/19/20 3:55:19: 65=Consistency Check progress on VD 00/0 is 55.90%(3318s)
09/19/20 3:55:35: EVT#104639-09/19/20 3:55:35: 65=Consistency Check progress on VD 00/0 is 56.31%(3334s)
09/19/20 3:55:51: EVT#104640-09/19/20 3:55:51: 65=Consistency Check progress on VD 00/0 is 56.68%(3350s)
09/19/20 3:56:07: EVT#104641-09/19/20 3:56:07: 65=Consistency Check progress on VD 00/0 is 57.07%(3366s)
09/19/20 3:56:23: EVT#104642-09/19/20 3:56:23: 65=Consistency Check progress on VD 00/0 is 57.47%(3382s)
09/19/20 3:56:39: EVT#104643-09/19/20 3:56:39: 65=Consistency Check progress on VD 00/0 is 57.88%(3398s)
09/19/20 3:56:55: EVT#104644-09/19/20 3:56:55: 65=Consistency Check progress on VD 00/0 is 58.27%(3414s)
09/19/20 3:57:11: EVT#104645-09/19/20 3:57:11: 65=Consistency Check progress on VD 00/0 is 58.65%(3430s)
09/19/20 3:57:27: EVT#104646-09/19/20 3:57:27: 65=Consistency Check progress on VD 00/0 is 59.06%(3446s)
09/19/20 3:57:43: EVT#104647-09/19/20 3:57:43: 65=Consistency Check progress on VD 00/0 is 59.44%(3462s)
09/19/20 3:57:59: EVT#104648-09/19/20 3:57:59: 65=Consistency Check progress on VD 00/0 is 59.76%(3478s)
09/19/20 3:58:15: EVT#104649-09/19/20 3:58:15: 65=Consistency Check progress on VD 00/0 is 60.12%(3494s)
09/19/20 3:58:31: EVT#104650-09/19/20 3:58:31: 65=Consistency Check progress on VD 00/0 is 60.52%(3510s)
09/19/20 3:58:47: EVT#104651-09/19/20 3:58:47: 65=Consistency Check progress on VD 00/0 is 60.78%(3526s)
09/19/20 3:59:04: EVT#104652-09/19/20 3:59:04: 65=Consistency Check progress on VD 00/0 is 60.93%(3543s)
09/19/20 3:59:20: EVT#104653-09/19/20 3:59:20: 65=Consistency Check progress on VD 00/0 is 61.29%(3559s)
09/19/20 3:59:36: EVT#104654-09/19/20 3:59:36: 65=Consistency Check progress on VD 00/0 is 61.67%(3575s)
09/19/20 3:59:52: EVT#104655-09/19/20 3:59:52: 65=Consistency Check progress on VD 00/0 is 62.06%(3591s)
09/19/20 4:00:08: EVT#104656-09/19/20 4:00:08: 65=Consistency Check progress on VD 00/0 is 62.39%(3607s)
09/19/20 4:00:18: EVT#104657-09/19/20 4:00:18: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 69.99%(3618s)
09/19/20 4:00:24: EVT#104658-09/19/20 4:00:24: 65=Consistency Check progress on VD 00/0 is 62.79%(3623s)
09/19/20 4:00:40: EVT#104659-09/19/20 4:00:40: 65=Consistency Check progress on VD 00/0 is 63.11%(3639s)
09/19/20 4:00:56: EVT#104660-09/19/20 4:00:56: 65=Consistency Check progress on VD 00/0 is 63.48%(3655s)
09/19/20 4:01:12: EVT#104661-09/19/20 4:01:12: 65=Consistency Check progress on VD 00/0 is 63.84%(3671s)
09/19/20 4:01:28: EVT#104662-09/19/20 4:01:28: 65=Consistency Check progress on VD 00/0 is 64.22%(3687s)
09/19/20 4:01:44: EVT#104663-09/19/20 4:01:44: 65=Consistency Check progress on VD 00/0 is 64.57%(3703s)
09/19/20 4:02:00: EVT#104664-09/19/20 4:02:00: 65=Consistency Check progress on VD 00/0 is 64.89%(3719s)
09/19/20 4:02:16: EVT#104665-09/19/20 4:02:16: 65=Consistency Check progress on VD 00/0 is 65.25%(3735s)
09/19/20 4:02:32: EVT#104666-09/19/20 4:02:32: 65=Consistency Check progress on VD 00/0 is 65.62%(3751s)
09/19/20 4:02:48: EVT#104667-09/19/20 4:02:48: 65=Consistency Check progress on VD 00/0 is 65.99%(3767s)
09/19/20 4:03:04: EVT#104668-09/19/20 4:03:04: 65=Consistency Check progress on VD 00/0 is 66.30%(3783s)
09/19/20 4:03:20: EVT#104669-09/19/20 4:03:20: 65=Consistency Check progress on VD 00/0 is 66.65%(3799s)
09/19/20 4:03:36: EVT#104670-09/19/20 4:03:36: 65=Consistency Check progress on VD 00/0 is 67.00%(3815s)
09/19/20 4:03:52: EVT#104671-09/19/20 4:03:52: 65=Consistency Check progress on VD 00/0 is 67.30%(3831s)
09/19/20 4:04:08: EVT#104672-09/19/20 4:04:08: 65=Consistency Check progress on VD 00/0 is 67.62%(3847s)
09/19/20 4:04:24: EVT#104673-09/19/20 4:04:24: 65=Consistency Check progress on VD 00/0 is 67.99%(3863s)
09/19/20 4:04:40: EVT#104674-09/19/20 4:04:40: 65=Consistency Check progress on VD 00/0 is 68.38%(3879s)
09/19/20 4:04:56: EVT#104675-09/19/20 4:04:56: 65=Consistency Check progress on VD 00/0 is 68.62%(3895s)
09/19/20 4:05:56: EVT#104676-09/19/20 4:05:56: 65=Consistency Check progress on VD 00/0 is 68.64%(3955s)
09/19/20 4:11:19: EVT#104677-09/19/20 4:11:19: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 79.99%(4279s)
09/19/20 4:20:16: EVT#104678-09/19/20 4:20:16: 65=Consistency Check progress on VD 00/0 is 68.89%(4815s)
09/19/20 4:22:53: EVT#104679-09/19/20 4:22:53: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 89.99%(4973s)
09/19/20 4:26:28: EVT#104680-09/19/20 4:26:28: 65=Consistency Check progress on VD 00/0 is 69.14%(5187s)
09/19/20 4:26:44: EVT#104681-09/19/20 4:26:44: 65=Consistency Check progress on VD 00/0 is 69.39%(5203s)
09/19/20 4:27:04: EVT#104682-09/19/20 4:27:04: 65=Consistency Check progress on VD 00/0 is 69.64%(5223s)
09/19/20 4:27:20: EVT#104683-09/19/20 4:27:20: 65=Consistency Check progress on VD 00/0 is 69.89%(5239s)
09/19/20 4:27:36: EVT#104684-09/19/20 4:27:36: 65=Consistency Check progress on VD 00/0 is 70.17%(5255s)
09/19/20 4:27:54: EVT#104685-09/19/20 4:27:54: 65=Consistency Check progress on VD 00/0 is 70.38%(5273s)
09/19/20 4:28:10: EVT#104686-09/19/20 4:28:10: 65=Consistency Check progress on VD 00/0 is 70.69%(5289s)
09/19/20 4:28:26: EVT#104687-09/19/20 4:28:26: 65=Consistency Check progress on VD 00/0 is 71.03%(5305s)
09/19/20 4:28:42: EVT#104688-09/19/20 4:28:42: 65=Consistency Check progress on VD 00/0 is 71.28%(5321s)
09/19/20 4:28:58: EVT#104689-09/19/20 4:28:58: 65=Consistency Check progress on VD 00/0 is 71.60%(5337s)
09/19/20 4:29:14: EVT#104690-09/19/20 4:29:14: 65=Consistency Check progress on VD 00/0 is 71.90%(5353s)
09/19/20 4:29:30: EVT#104691-09/19/20 4:29:30: 65=Consistency Check progress on VD 00/0 is 72.14%(5369s)
09/19/20 4:29:48: EVT#104692-09/19/20 4:29:48: 65=Consistency Check progress on VD 00/0 is 72.37%(5387s)
09/19/20 4:30:05: EVT#104693-09/19/20 4:30:05: 65=Consistency Check progress on VD 00/0 is 72.62%(5404s)
09/19/20 4:30:34: EVT#104694-09/19/20 4:30:34: 65=Consistency Check progress on VD 00/0 is 72.87%(5433s)
09/19/20 4:30:50: EVT#104695-09/19/20 4:30:50: 65=Consistency Check progress on VD 00/0 is 73.20%(5449s)
09/19/20 4:31:06: EVT#104696-09/19/20 4:31:06: 65=Consistency Check progress on VD 00/0 is 73.44%(5465s)
09/19/20 4:31:22: EVT#104697-09/19/20 4:31:22: 65=Consistency Check progress on VD 00/0 is 73.73%(5481s)
09/19/20 4:31:38: EVT#104698-09/19/20 4:31:38: 65=Consistency Check progress on VD 00/0 is 74.02%(5497s)
09/19/20 4:31:54: EVT#104699-09/19/20 4:31:54: 65=Consistency Check progress on VD 00/0 is 74.35%(5513s)
09/19/20 4:32:10: EVT#104700-09/19/20 4:32:10: 65=Consistency Check progress on VD 00/0 is 74.64%(5529s)
09/19/20 4:32:26: EVT#104701-09/19/20 4:32:26: 65=Consistency Check progress on VD 00/0 is 74.99%(5545s)
09/19/20 4:32:42: EVT#104702-09/19/20 4:32:42: 65=Consistency Check progress on VD 00/0 is 75.34%(5561s)
09/19/20 4:32:58: EVT#104703-09/19/20 4:32:58: 65=Consistency Check progress on VD 00/0 is 75.70%(5577s)
09/19/20 4:33:14: EVT#104704-09/19/20 4:33:14: 65=Consistency Check progress on VD 00/0 is 76.06%(5593s)
09/19/20 4:33:30: EVT#104705-09/19/20 4:33:30: 65=Consistency Check progress on VD 00/0 is 76.41%(5609s)
09/19/20 4:33:46: EVT#104706-09/19/20 4:33:46: 65=Consistency Check progress on VD 00/0 is 76.72%(5625s)
09/19/20 4:34:02: EVT#104707-09/19/20 4:34:02: 65=Consistency Check progress on VD 00/0 is 77.05%(5641s)
09/19/20 4:34:18: EVT#104708-09/19/20 4:34:18: 65=Consistency Check progress on VD 00/0 is 77.38%(5657s)
09/19/20 4:34:34: EVT#104709-09/19/20 4:34:34: 65=Consistency Check progress on VD 00/0 is 77.72%(5673s)
09/19/20 4:34:50: EVT#104710-09/19/20 4:34:50: 65=Consistency Check progress on VD 00/0 is 78.05%(5689s)
09/19/20 4:35:06: EVT#104711-09/19/20 4:35:06: 65=Consistency Check progress on VD 00/0 is 78.35%(5705s)
09/19/20 4:35:22: EVT#104712-09/19/20 4:35:22: 65=Consistency Check progress on VD 00/0 is 78.68%(5721s)
09/19/20 4:35:38: EVT#104713-09/19/20 4:35:38: 65=Consistency Check progress on VD 00/0 is 79.00%(5737s)
09/19/20 4:35:54: EVT#104714-09/19/20 4:35:54: 65=Consistency Check progress on VD 00/0 is 79.31%(5753s)
09/19/20 4:36:10: EVT#104715-09/19/20 4:36:10: 65=Consistency Check progress on VD 00/0 is 79.60%(5769s)
09/19/20 4:36:26: EVT#104716-09/19/20 4:36:26: 65=Consistency Check progress on VD 00/0 is 79.90%(5785s)
09/19/20 4:36:26: EVT#104717-09/19/20 4:36:26: 94=Patrol Read progress on PD 0c(e0xfc/s3) is 99.99%(5786s)
09/19/20 4:36:26: prCallback: PR completed for pd=0c
09/19/20 4:36:26: PR cycle complete
09/19/20 4:36:26: EVT#104718-09/19/20 4:36:26: 35=Patrol Read complete
09/19/20 4:36:26: Next PR scheduled to start at 09/26/20 3:00:00
09/19/20 4:36:42: EVT#104719-09/19/20 4:36:42: 65=Consistency Check progress on VD 00/0 is 80.20%(5801s)
09/19/20 4:36:58: EVT#104720-09/19/20 4:36:58: 65=Consistency Check progress on VD 00/0 is 80.48%(5817s)
09/19/20 4:37:14: EVT#104721-09/19/20 4:37:14: 65=Consistency Check progress on VD 00/0 is 80.80%(5833s)
09/19/20 4:37:30: EVT#104722-09/19/20 4:37:30: 65=Consistency Check progress on VD 00/0 is 81.11%(5849s)
09/19/20 4:37:46: EVT#104723-09/19/20 4:37:46: 65=Consistency Check progress on VD 00/0 is 81.37%(5865s)
09/19/20 4:38:02: EVT#104724-09/19/20 4:38:02: 65=Consistency Check progress on VD 00/0 is 81.65%(5881s)
09/19/20 4:38:18: EVT#104725-09/19/20 4:38:18: 65=Consistency Check progress on VD 00/0 is 81.96%(5897s)
09/19/20 4:38:34: EVT#104726-09/19/20 4:38:34: 65=Consistency Check progress on VD 00/0 is 82.25%(5913s)
09/19/20 4:38:50: EVT#104727-09/19/20 4:38:50: 65=Consistency Check progress on VD 00/0 is 82.45%(5929s)
09/19/20 4:39:06: EVT#104728-09/19/20 4:39:06: 65=Consistency Check progress on VD 00/0 is 82.57%(5945s)
09/19/20 4:39:23: EVT#104729-09/19/20 4:39:23: 65=Consistency Check progress on VD 00/0 is 82.82%(5962s)
09/19/20 4:39:39: EVT#104730-09/19/20 4:39:39: 65=Consistency Check progress on VD 00/0 is 83.10%(5978s)
09/19/20 4:39:55: EVT#104731-09/19/20 4:39:55: 65=Consistency Check progress on VD 00/0 is 83.41%(5994s)
09/19/20 4:40:11: EVT#104732-09/19/20 4:40:11: 65=Consistency Check progress on VD 00/0 is 83.68%(6010s)
09/19/20 4:40:27: EVT#104733-09/19/20 4:40:27: 65=Consistency Check progress on VD 00/0 is 84.01%(6026s)
09/19/20 4:40:43: EVT#104734-09/19/20 4:40:43: 65=Consistency Check progress on VD 00/0 is 84.29%(6042s)
09/19/20 4:40:59: EVT#104735-09/19/20 4:40:59: 65=Consistency Check progress on VD 00/0 is 84.58%(6058s)
09/19/20 4:41:15: EVT#104736-09/19/20 4:41:15: 65=Consistency Check progress on VD 00/0 is 84.89%(6074s)
09/19/20 4:41:31: EVT#104737-09/19/20 4:41:31: 65=Consistency Check progress on VD 00/0 is 85.22%(6090s)
09/19/20 4:41:47: EVT#104738-09/19/20 4:41:47: 65=Consistency Check progress on VD 00/0 is 85.55%(6106s)
09/19/20 4:42:03: EVT#104739-09/19/20 4:42:03: 65=Consistency Check progress on VD 00/0 is 85.84%(6122s)
09/19/20 4:42:19: EVT#104740-09/19/20 4:42:19: 65=Consistency Check progress on VD 00/0 is 86.18%(6138s)
09/19/20 4:42:39: EVT#104741-09/19/20 4:42:39: 65=Consistency Check progress on VD 00/0 is 86.30%(6158s)
09/19/20 4:42:55: EVT#104742-09/19/20 4:42:55: 65=Consistency Check progress on VD 00/0 is 86.59%(6174s)
09/19/20 4:43:11: EVT#104743-09/19/20 4:43:11: 65=Consistency Check progress on VD 00/0 is 86.89%(6190s)
09/19/20 4:43:27: EVT#104744-09/19/20 4:43:27: 65=Consistency Check progress on VD 00/0 is 87.19%(6206s)
09/19/20 4:43:43: EVT#104745-09/19/20 4:43:43: 65=Consistency Check progress on VD 00/0 is 87.50%(6222s)
09/19/20 4:43:59: EVT#104746-09/19/20 4:43:59: 65=Consistency Check progress on VD 00/0 is 87.79%(6238s)
09/19/20 4:44:15: EVT#104747-09/19/20 4:44:15: 65=Consistency Check progress on VD 00/0 is 88.06%(6254s)
09/19/20 4:44:31: EVT#104748-09/19/20 4:44:31: 65=Consistency Check progress on VD 00/0 is 88.32%(6270s)
09/19/20 4:44:47: EVT#104749-09/19/20 4:44:47: 65=Consistency Check progress on VD 00/0 is 88.63%(6286s)
09/19/20 4:45:03: EVT#104750-09/19/20 4:45:03: 65=Consistency Check progress on VD 00/0 is 88.89%(6302s)
09/19/20 4:45:19: EVT#104751-09/19/20 4:45:19: 65=Consistency Check progress on VD 00/0 is 89.19%(6318s)
09/19/20 4:45:35: EVT#104752-09/19/20 4:45:35: 65=Consistency Check progress on VD 00/0 is 89.52%(6334s)
09/19/20 4:45:51: EVT#104753-09/19/20 4:45:51: 65=Consistency Check progress on VD 00/0 is 89.83%(6350s)
09/19/20 4:46:07: EVT#104754-09/19/20 4:46:07: 65=Consistency Check progress on VD 00/0 is 90.11%(6366s)
09/19/20 4:46:23: EVT#104755-09/19/20 4:46:23: 65=Consistency Check progress on VD 00/0 is 90.39%(6382s)
09/19/20 4:46:39: EVT#104756-09/19/20 4:46:39: 65=Consistency Check progress on VD 00/0 is 90.70%(6398s)
09/19/20 4:46:55: EVT#104757-09/19/20 4:46:55: 65=Consistency Check progress on VD 00/0 is 91.01%(6414s)
09/19/20 4:47:11: EVT#104758-09/19/20 4:47:11: 65=Consistency Check progress on VD 00/0 is 91.29%(6430s)
09/19/20 4:47:27: EVT#104759-09/19/20 4:47:27: 65=Consistency Check progress on VD 00/0 is 91.58%(6446s)
09/19/20 4:47:43: EVT#104760-09/19/20 4:47:43: 65=Consistency Check progress on VD 00/0 is 91.85%(6462s)
09/19/20 4:48:04: EVT#104761-09/19/20 4:48:04: 65=Consistency Check progress on VD 00/0 is 92.02%(6483s)
09/19/20 4:48:29: EVT#104762-09/19/20 4:48:29: 65=Consistency Check progress on VD 00/0 is 92.27%(6508s)
09/19/20 4:48:58: EVT#104763-09/19/20 4:48:58: 65=Consistency Check progress on VD 00/0 is 92.52%(6537s)
09/19/20 4:49:25: EVT#104764-09/19/20 4:49:25: 65=Consistency Check progress on VD 00/0 is 92.77%(6564s)
09/19/20 4:49:52: EVT#104765-09/19/20 4:49:52: 65=Consistency Check progress on VD 00/0 is 93.02%(6591s)
09/19/20 4:50:17: EVT#104766-09/19/20 4:50:17: 65=Consistency Check progress on VD 00/0 is 93.27%(6616s)
09/19/20 4:50:40: EVT#104767-09/19/20 4:50:40: 65=Consistency Check progress on VD 00/0 is 93.51%(6639s)
09/19/20 4:51:07: EVT#104768-09/19/20 4:51:07: 65=Consistency Check progress on VD 00/0 is 93.76%(6666s)
09/19/20 4:51:34: EVT#104769-09/19/20 4:51:34: 65=Consistency Check progress on VD 00/0 is 94.01%(6693s)
09/19/20 4:52:04: EVT#104770-09/19/20 4:52:04: 65=Consistency Check progress on VD 00/0 is 94.26%(6723s)
09/19/20 4:52:35: EVT#104771-09/19/20 4:52:35: 65=Consistency Check progress on VD 00/0 is 94.51%(6754s)
09/19/20 4:52:58: EVT#104772-09/19/20 4:52:58: 65=Consistency Check progress on VD 00/0 is 94.76%(6777s)
09/19/20 4:53:15: EVT#104773-09/19/20 4:53:15: 65=Consistency Check progress on VD 00/0 is 95.01%(6794s)
09/19/20 4:53:31: EVT#104774-09/19/20 4:53:31: 65=Consistency Check progress on VD 00/0 is 95.28%(6810s)
09/19/20 4:53:47: EVT#104775-09/19/20 4:53:47: 65=Consistency Check progress on VD 00/0 is 95.57%(6826s)
09/19/20 4:54:03: EVT#104776-09/19/20 4:54:03: 65=Consistency Check progress on VD 00/0 is 95.77%(6842s)
09/19/20 4:54:19: EVT#104777-09/19/20 4:54:19: 65=Consistency Check progress on VD 00/0 is 96.00%(6858s)
09/19/20 4:54:35: EVT#104778-09/19/20 4:54:35: 65=Consistency Check progress on VD 00/0 is 96.28%(6874s)
09/19/20 4:54:54: EVT#104779-09/19/20 4:54:54: 65=Consistency Check progress on VD 00/0 is 96.50%(6893s)
09/19/20 4:55:12: EVT#104780-09/19/20 4:55:12: 65=Consistency Check progress on VD 00/0 is 96.75%(6911s)
09/19/20 4:55:28: EVT#104781-09/19/20 4:55:28: 65=Consistency Check progress on VD 00/0 is 97.05%(6927s)
09/19/20 4:55:44: EVT#104782-09/19/20 4:55:44: 65=Consistency Check progress on VD 00/0 is 97.25%(6943s)
09/19/20 4:56:10: EVT#104783-09/19/20 4:56:10: 65=Consistency Check progress on VD 00/0 is 97.49%(6969s)
09/19/20 4:56:29: EVT#104784-09/19/20 4:56:29: 65=Consistency Check progress on VD 00/0 is 97.74%(6988s)
09/19/20 4:56:59: EVT#104785-09/19/20 4:56:59: 65=Consistency Check progress on VD 00/0 is 97.99%(7018s)
09/19/20 4:57:20: EVT#104786-09/19/20 4:57:20: 65=Consistency Check progress on VD 00/0 is 98.24%(7039s)
09/19/20 4:57:42: EVT#104787-09/19/20 4:57:42: 65=Consistency Check progress on VD 00/0 is 98.49%(7061s)
09/19/20 4:58:08: EVT#104788-09/19/20 4:58:08: 65=Consistency Check progress on VD 00/0 is 98.74%(7087s)
09/19/20 4:58:33: EVT#104789-09/19/20 4:58:33: 65=Consistency Check progress on VD 00/0 is 98.99%(7112s)
09/19/20 4:58:49: EVT#104790-09/19/20 4:58:49: 65=Consistency Check progress on VD 00/0 is 99.24%(7128s)
09/19/20 4:59:15: EVT#104791-09/19/20 4:59:15: 65=Consistency Check progress on VD 00/0 is 99.48%(7154s)
09/19/20 4:59:42: EVT#104792-09/19/20 4:59:42: 65=Consistency Check progress on VD 00/0 is 99.73%(7181s)
09/19/20 5:00:11: EVT#104793-09/19/20 5:00:11: 65=Consistency Check progress on VD 00/0 is 99.98%(7210s)
09/19/20 5:00:12: EVT#104794-09/19/20 5:00:12: 58=Consistency Check done on VD 00/0
09/19/20 5:00:12: ccScheduleSetNextStartTime: RTC_TimeStamp=26f84cdc, nextStartTime=27016b30
09/19/20 5:00:12: Next cc scheduled to start at 09/26/20 3:00:00
09/19/20 5:00:12: CC Schedule cycle complete
Exit Code: 0x00
以上有一条告警：
T6533: EVT#104486-T6533: 195=BBU disabled; changing WB virtual disks to WT, Forced WB VDs are not affected
- 明显是磁盘raid 卡BBU电池挂了将磁盘的写策略有write back 修改为write through ，io性能急剧下降导致应用层的线程等待问题。
- RAID卡都有写cache(Battery Backed Write Cache),写cache对IO性能的提升非常明显,因为掉电会丢失数据,所以必须由电池提供支持。电池会定期充放电,一般为90天左右,当发现电量低于某个阀值时,会将写cache策略从writeback置为writethrough,相当于写cache会失效,这时如果系统有大量的IO操作,可能会明显感觉到IO响应速度变慢,cpu 队列堆积系统load 飙高。
8、显示所有逻辑磁盘组信息
t@LancetServer MegaCli]#
[root@LancetServer MegaCli]# MegaCli64 -LDInfo -LALL -aAll
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 1.088 TB
State : Optimal
Strip Size : 128 KB
Number Of Drives : 3
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 557.861 GB
State : Optimal
Strip Size : 128 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
BBU修好后：
[root@LancetServer disk2]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL
BBU status for Adapter: 0
BatteryType: iBBU08
Voltage: 3973 mV
Current: 0 mA
Temperature: 34 C
BBU Firmware Status:
Charging Status : None
Voltage : OK
Temperature : OK
Learn Cycle Requested : No
Learn Cycle Active : No
Learn Cycle Status : OK
Learn Cycle Timeout : No
I2c Errors Detected : No
Battery Pack Missing : No
Battery Replacement required : No
Remaining Capacity Low : No
Periodic Learn Required : No
Transparent Learn : No
No space to cache offload : No
Pack is about to fail & should be replaced : No
Cache Offload premium feature required : No
Module microcode update required : No
Battery state:
GasGuageStatus:
Fully Discharged : No
Fully Charged : No
Discharging : No
Initialized : Yes
Remaining Time Alarm : Yes
Remaining Capacity Alarm: No
Discharge Terminated : No
Over Temperature : No
Charging Terminated : No
Over Charged : No
Relative State of Charge: 83 %
Charger System State: 1
Charger System Ctrl: 0
Charging current: 0 mA
Absolute state of charge: 73 %
Max Error: 0 %
Exit Code: 0x00
[root@LancetServer disk2]# MegaCli64 -LDSetProp -WB -Lall -aAll
Set Write Policy to WriteBack on Adapter 0, VD 0 (target id: 0) success
Set Write Policy to WriteBack on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
文章评论