Hi
For a customer I just installed a vSphere 5.1 essentials environment. Two IBM x3650M4 hosts connected over ONE path to an IBM DS3512 storage. Soon a second HBA will be added to the configuration and we'll have two paths.
Before installing I checked the BIOS versions of the ESXi hosts. According to the VMware HCL, the BIOS version should be IBM-[VVE112H]. According to the IMM, the hosts are running
UEFI (Active) 1.21 VVE120EUS 11 Oct 2012
The problem we're experiencing is that all disk actions are very slow. In the vmkernel.log I can see a lot of path failovers that would explain why the system is slow. (See logs at end of this post).
When searching for these errors, I found this community post: http://communities.vmware.com/thread/341512
They refer to this VMware KB http://kb.vmware.com/kb/1030265
In that KB there is the following note:
"Note: This issue only applies if you see this specific alert in the vmkernel/messages log files: ALERT: APIC: 1823: APICID 0x00000000 - ESR = 0x40. If you do not see this message, you are not experiencing this issue."
Questions:
- We can't find that message (ALERT: APIC: 1823: APICID 0x00000000 - ESR = 0x40) in the vmkernel logs. Is this enough to state this KB does not apply?
- What is the impact if we DO apply the recommended solution on the ESXi 5.1 hosts?
Regards
Gabrie
Below is the vmkernel.log:
2013-02-01T17:05:30.373Z cpu22:8214)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.60080e50002fb706000002635107aa63"
2013-02-01T17:05:30.796Z cpu15:8374)NMP: nmp_DeviceUpdatePathStates:615: Activated path "vmhba2:C0:T0:L1" for NMP device "naa.60080e50002f7b320000028c5107a9ff".
2013-02-01T17:05:30.797Z cpu0:10554)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60080e50002f7b320000028c5107a9ff" - issuing command 0x412401f91300
2013-02-01T17:05:30.797Z cpu10:13686)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.60080e50002f7b320000028c5107a9ff"
2013-02-01T17:05:32.520Z cpu20:217782)VMW_SATP_LSI: satp_lsi_pathFailure:1120: Command 0x8a to naa.60080e50002fb706000002635107aa63 (fcf 0) failed with NOT_READY (0x2/0x4/0x1), on path vmhba2:C0:T0:L2 (pnr 1, iet 0xac848a2)
2013-02-01T17:05:32.520Z cpu20:217782)ScsiDeviceIO: 2303: Cmd(0x4124403dcb00) 0x8a, CmdSN 0x8000002c from world 217778 to dev "naa.60080e50002fb706000002635107aa63" failed H:0x0 D:0x2 P:0x4 Possible sense data: 0x2 0x4 0x1.
2013-02-01T17:05:32.520Z cpu22:8214)ScsiDeviceIO: 2303: Cmd(0x4124403f8200) 0x8a, CmdSN 0x80000041 from world 217778 to dev "naa.60080e50002fb706000002635107aa63" failed H:0x0 D:0x2 P:0x4 Possible sense data: 0x2 0x4 0x1.
2013-02-01T17:05:33.039Z cpu22:8214)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x8a (0x4124403dcb00, 217778) to dev "naa.60080e50002fb706000002635107aa63" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x94 0x1. Act:FAILOVER
2013-02-01T17:05:33.039Z cpu22:8214)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60080e50002fb706000002635107aa63": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2013-02-01T17:05:33.039Z cpu22:8214)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x8a (0x4124403f8200, 217778) to dev "naa.60080e50002fb706000002635107aa63" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x94 0x1. Act:FAILOVER
2013-02-01T17:05:33.039Z cpu22:8214)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x8a (0x4124425c4e40, 217778) to dev "naa.60080e50002fb706000002635107aa63" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x94 0x1. Act:FAILOVER
2013-02-01T17:05:33.306Z cpu3:8195)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x412402b82c00, 11035) to dev "naa.60080e50002f7b320000028c5107a9ff" on path "vmhba2:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x94 0x1. Act:FAILOVER
2013-02-01T17:05:33.306Z cpu3:8195)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60080e50002f7b320000028c5107a9ff": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2013-02-01T17:05:34.496Z cpu3:8375)NMP: nmp_DeviceUpdatePathStates:615: Activated path "vmhba2:C0:T0:L2" for NMP device "naa.60080e50002fb706000002635107aa63".
2013-02-01T17:05:34.497Z cpu1:8786)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60080e50002fb706000002635107aa63" - issuing command 0x4124403dcb00
2013-02-01T17:05:34.506Z cpu22:8214)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.60080e50002fb706000002635107aa63"
2013-02-01T17:05:34.795Z cpu2:8369)NMP: nmp_DeviceUpdatePathStates:615: Activated path "vmhba2:C0:T0:L1" for NMP device "naa.60080e50002f7b320000028c5107a9ff".
2013-02-01T17:05:34.796Z cpu0:10554)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60080e50002f7b320000028c5107a9ff" - issuing command 0x412402b82c00
2013-02-01T17:05:34.796Z cpu4:8634)NMP: nmpCompleteRetryForPath:321: Retry world recovered device "naa.60080e50002f7b320000028c5107a9ff"
~ #