I totally meant to get this out a few days ago but am now finally catching up. During the beta testing period, I had all kinds of problems getting 5.1 working. Come to find out, Synology DSM is not compatible with vSphere 5.1 with BOTH iSCSI and NFS. I HAVE NOT had a chance to test the GA version, but build 732143 which is a beta build along with the beta of DSM 4.1, experienced this issue. This has also been confirmed by Jason Nash and Jeremiah Dooley, so I know it's not just me.
It's possible to mount both NFS and iSCSI datastores, but when you try to build or power on a VM, everything begins to halt when it needs to write to disk. If everything runs in memory (like mounting an ISO and beginning an OS installation) it looks like it's working, but when it comes to install the OS to disk, it begins to crawl. I checked the vmkernel log and here is what is shown. You can see that after I power on the VM, it gets binded to a port but when it tries to read from disk, there are all sorts of errors:
2012-08-03T11:45:59.441Z cpu1:3958)Net: 1858: connected New Virtual Machine eth0 to VM Network, portID 0x2000007 2012-08-03T11:45:59.441Z cpu1:3958)NetPort: 1380: enabled port 0x2000007 with mac 00:00:00:00:00:00 2012-08-03T11:45:59.523Z cpu1:3044)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0) 2012-08-03T11:46:00.251Z cpu0:3926)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 134193 (0 outstanding commands) 2012-08-03T11:46:00.251Z cpu3:2122)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-08-03T11:46:00.251Z cpu3:2122)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-08-03T11:46:03.077Z cpu3:3023)BC: 3115: Blocking due to no free buffers. nDirty = 3 nWaiters = 1 2012-08-03T11:46:28.426Z cpu3:3926)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 134193 (0 outstanding commands) 2012-08-03T11:46:28.426Z cpu0:2122)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-08-03T11:46:28.426Z cpu0:2122)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-08-03T11:48:16.212Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:16.498Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:16.904Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:17.310Z cpu3:3930)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:17.701Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:18.107Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:18.496Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:18.902Z cpu1:2049)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:19.309Z cpu0:2052)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:19.698Z cpu0:2052)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 ... ... ... 2012-08-03T11:48:51.710Z cpu3:2051)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:52.099Z cpu1:3153)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:52.505Z cpu1:2049)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:52.910Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:53.300Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:53.706Z cpu2:2854)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:54.096Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:54.501Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:54.907Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:55.297Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:55.703Z cpu2:2854)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:56.109Z cpu2:2050)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:56.499Z cpu0:2052)WARNING: NFS: 3813: Short read for object b00f 44 9b513ae6 f2182b00 1070001 3804f9 0 a4272cd1 ef53929e 0 bc2ffa00000000 c41389f3 0 0 offset: 0xc63be000 requested: 0x100000 read: 0x20000 2012-08-03T11:48:57.732Z cpu1:2080)NetPort: 1574: disabled port 0x2000007 2012-08-03T11:48:57.732Z cpu1:2080)Net: 2475: disconnected client from port 0x2000007 2012-08-03T11:48:57.813Z cpu1:3925)WARNING: Net: vm 3926: 4477: port 0x2000007 not found 2012-08-03T11:48:57.813Z cpu1:3925)Net: 2451: 0x2000007: can't access port 2012-08-03T16:42:46.065Z cpu0:16624)WARNING: UserLinux: 1331: unsupported: (void) 2012-08-03T16:43:56.889Z cpu0:16661)WARNING: UserLinux: 1331: unsupported: (void)
I've made the Synology team well aware of this bug over a month ago. They have been lacking to get a 5.1 environment up and running to test, but if you want to help fan the flame, please open a Synology Support ticket so they know this is actually critical.
Stay Tuned to this site and I will let you know when it is safe to upgrade to vSphere 5.1. I plan to test the GA version of both 5.1 and DSM 4.1 this Friday 9/14.
Updated 9/12/2012 07:45AM EST
Thanks to two of my readers below (Steve and J) in the comment section, it seems that I don't need to worry about trying the GA version of both vSphere 5.1 and DSM 4.1. It's completely broke. Now it's up to YOU, Synology owners, to take a stance. Submit a Synology Support ticket if you are a Synology owner. The more tickets they receive, the higher priority this will receive. If you would like, you can mention Support #137146, which was submitted by me on 8/3/12 detailing this severe bug.
Updated 9/13/2012 11:28PM EST
I received an email from Synology and they are testing 5.1 (that's the good news). Here is the email:
Hi Kendrick,
Our engineer was not able to reproduce the iSCSI issue in ESX 5.1-799733 using both DS1512+ and RS3412RPxs running 4.1-2636. However, we were able to reproduce the NFS issue. Our developers are working hard to resolve this specific issue with VMWARE and NFS. However, please note that we are currently not certified to operate with VMWARE over NFS protocol so we do not recommend to use NFS in a production ESX environment.
If you are able to reproduce this issue over iSCSI using ESX 5.1-799733, please provide the reproduce steps and environment details.
So I went ahead and tried testing. I had another USB stick sitting around so I loaded ESXi 5.1 build 799733 and booted up. On the Synology, I created a new iSCSI target and associated 2 LUNs with it. I ran some tests and I think their engineer is running into some false positives. It may seem like it's working, but it's not. The performance is severely degraded. I have 2 servers, 1 running 5.1, the other running 5.0 Update 1. I created a new virtual machine on each server connecting them both to different iSCSI luns. The test was to see which server would install Windows 2008 R2 first. I began by creating the VM on the 5.1 host first, mounted the ISO, then began the installation. About 3 minutes later, I had the VM created on the 5.0 host and the installation started. After about 2 minutes later, the 5.0 host was further along in the installation.
If we look at the vmkernel logs, we can clearly see the point of powering on the VM ("starting the world" which is the first line) and where the storage issues start happening. where ever it says "ScsiDeviceIO:1191 Device naa.34lotsofcharacters43" is an error on the storage side. If you look in the 5.0 log, that error never appears. So this issue is still with both iSCSI and NFS. I am also running the latest DSM 4.1 (upgraded today).
If you have a DS1512+, please comment if it works for you or if it doesn't. Thank you!
Here is the vmkernel log from the host running vSphere 5.0 Update 1
2012-09-14T03:02:49.180Z cpu3:2527192)World: vm 2519001: 1267: Starting world vmm0:New_Virtual_Machine with flags 4008 2012-09-14T03:02:49.180Z cpu3:2527192)Sched: vm 2519001: 6207: Adding world 'vmm0:New_Virtual_Machine', group 'host/user', cpu: shares=-1 min=-1 minLimit=-1 max=-1, mem: shares=-1 min=-1 minLimit=-1 max=-1 2012-09-14T03:02:49.180Z cpu3:2527192)Sched: vm 2519001: 6222: renamed group 2000982 to vm.2527192 2012-09-14T03:02:49.180Z cpu3:2527192)Sched: vm 2519001: 6239: group 2000982 is located under group 4 2012-09-14T03:02:49.183Z cpu3:2527192)MemSched: vm 2527192: 7453: extended swap to 12291 pgs 2012-09-14T03:02:49.340Z cpu2:2527192)VSCSI: 3683: handle 8242(vscsi0:0):Creating Virtual Device for world 2519001 (FSS handle 24271960) 2012-09-14T03:02:49.393Z cpu1:2519001)VMMVMKCall: 194: Received INIT from world 2519001 2012-09-14T03:02:49.395Z cpu1:2519001)PVSCSI: 3054: scsi0: wdt=1 intrCoalescingMode=2 flags=0xf 2012-09-14T03:02:49.395Z cpu1:2529245)Net: 1652: connected New Virtual Machine eth0 to VM Network, portID 0x1000003 2012-09-14T03:02:49.477Z cpu3:18988)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0) 2012-09-14T03:05:15.321Z cpu1:2484695)ScsiDeviceIO: 2322: Cmd(0x412400772280) 0x93, CmdSN 0x445 from world 2519176 to dev "naa.60014054bd26d6ddccfdd3335da191dc" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. 2012-09-14T03:09:18.958Z cpu0:2529559)WARNING: UserObj: 675: Failed to crossdup fd 8, /dev/ptyp0 type CHAR: Busy 2012-09-14T03:09:18.958Z cpu0:2529559)WARNING: UserObj: 675: Failed to crossdup fd 10, /dev/ptyp0 type CHAR: Busy 2012-09-14T03:09:18.959Z cpu0:2531609)WARNING: UserLinux: 1340: unsupported: (void)
Here is the vmkernel log from the host running vSphere 5.1
2012-09-14T02:55:13.056Z cpu0:10692)World: vm 10693: 1421: Starting world vmm0:New_Virtual_Machine with flags 8 2012-09-14T02:55:13.056Z cpu0:10692)Sched: vm 10693: 6406: Adding world 'vmm0:New_Virtual_Machine', group 'host/user', cpu: shares=-1 min=-1 minLimit=-1 max=-1, mem: shares=-1 min=-1 minLimit=-1 max=-1 2012-09-14T02:55:13.056Z cpu0:10692)Sched: vm 10693: 6421: renamed group 11712 to vm.10692 2012-09-14T02:55:13.056Z cpu0:10692)Sched: vm 10693: 6438: group 11712 is located under group 4 2012-09-14T02:55:13.062Z cpu0:10692)MemSched: vm 10692: 7756: extended swap to 27336 pgs 2012-09-14T02:55:13.175Z cpu0:10692)WARNING: MemSched: 6105: Psharing is disabled but balooning is not. 2012-09-14T02:55:13.270Z cpu2:10692)VSCSI: 3775: handle 8192(vscsi0:0):Creating Virtual Device for world 10693 (FSS handle 144991) 2012-09-14T02:55:13.366Z cpu0:10693)VMMVMKCall: 208: Received INIT from world 10693 2012-09-14T02:55:13.368Z cpu0:10693)PVSCSI: 3070: scsi0: wdt=1 intrCoalescingMode=2 flags=0xf 2012-09-14T02:55:13.369Z cpu0:10699)Net: 1858: connected New Virtual Machine eth0 to VM Network, portID 0x200000a 2012-09-14T02:55:13.481Z cpu3:5028)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0) 2012-09-14T02:57:13.819Z cpu2:4098)ScsiDeviceIO: 2316: Cmd(0x4124007d8e40) 0x93, CmdSN 0x1865 from world 4151 to dev "naa.60014057c686360dc26ad3f9dd9387d1" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. 2012-09-14T02:58:20.051Z cpu1:4989)BC: 3115: Blocking due to no free buffers. nDirty = 3 nWaiters = 1 2012-09-14T02:58:40.064Z cpu3:5008)BC: 3115: Blocking due to no free buffers. nDirty = 1 nWaiters = 1 2012-09-14T02:59:40.040Z cpu2:7402)BC: 3115: Blocking due to no free buffers. nDirty = 1 nWaiters = 1 2012-09-14T02:59:40.058Z cpu1:7402)BC: 3115: Blocking due to no free buffers. nDirty = 1 nWaiters = 1 2012-09-14T03:00:35.035Z cpu0:4100)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 18058 microseconds to 689452 microseconds. 2012-09-14T03:00:40.574Z cpu2:4098)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 689452 microseconds to 135482 microseconds. 2012-09-14T03:00:48.834Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 18479 microseconds to 451434 microseconds. 2012-09-14T03:00:52.357Z cpu3:4802)ScsiDeviceIO: 2316: Cmd(0x4124007f2d00) 0x93, CmdSN 0x3c32 from world 10895 to dev "naa.60014057c686360dc26ad3f9dd9387d1" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. 2012-09-14T03:01:20.062Z cpu3:5027)BC: 3115: Blocking due to no free buffers. nDirty = 1 nWaiters = 1 2012-09-14T03:01:20.088Z cpu3:5027)BC: 3115: Blocking due to no free buffers. nDirty = 1 nWaiters = 1 2012-09-14T03:01:25.840Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 19369 microseconds to 940709 microseconds. 2012-09-14T03:01:34.508Z cpu1:4097)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 940709 microseconds to 183475 microseconds. 2012-09-14T03:01:49.614Z cpu2:5236)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 19825 microseconds to 597476 microseconds. 2012-09-14T03:02:09.682Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 20310 microseconds to 1642669 microseconds. 2012-09-14T03:02:19.570Z cpu1:10893)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 1642669 microseconds to 326246 microseconds. 2012-09-14T03:02:20.052Z cpu0:4971)BC: 3115: Blocking due to no free buffers. nDirty = 6 nWaiters = 1 2012-09-14T03:02:22.283Z cpu0:4100)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 20588 microseconds to 734351 microseconds. 2012-09-14T03:02:37.988Z cpu3:10693)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 734351 microseconds to 143217 microseconds. 2012-09-14T03:02:42.081Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21001 microseconds to 562387 microseconds. 2012-09-14T03:02:42.087Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21063 microseconds to 1249678 microseconds. 2012-09-14T03:02:45.302Z cpu0:4100)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 1249678 microseconds to 249146 microseconds. 2012-09-14T03:02:48.537Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21255 microseconds to 537728 microseconds. 2012-09-14T03:02:49.664Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21329 microseconds to 1259841 microseconds. 2012-09-14T03:02:50.172Z cpu3:4099)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 1259841 microseconds to 247279 microseconds. 2012-09-14T03:02:56.636Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21451 microseconds to 505935 microseconds. ....
Update 9/18/2012 11:30am EST
More updates and comments are rolling in. Jason Nash was able to test his DS3612XS which is the high end system Synology sells (close to $4k). Synology thought their higher end models were supported, but according to Jason Nash
"Oh yeah. I'm not getting errors in logs but VMs are behaving BADLY even on the DS3611xs w/ vSphere 5.1. Be warned. No go. /cc @Synology" - Tweet
In addition, thanks to kfingerlos in the comments section, his 1512+ is down for the count.
Thanks everyone for your continued testing and support. Please remember to open up a support ticket if you haven't already.
Update 9/18/2012 3:00pm EST
I got an email from Synology that asked:
"Hi Kendrick,
We found that 5.1 uses VAAI by default. If the target does not have VAAI enabled, then there will be performance issues. This option was introduced in DSM 4.1 and VAAI support has to be enabled when creating the LUN. Please try creating a new LUN, enable the VAAI option during creation, and please let me know if you experience the same issue."
I tried to do this and perhaps ran into some more bugs. There is a new dropdown box that enables or disables VAAI on the iSCSI Lun. What does this mean? Cool if we need to add new LUNs, but bad because we not have to migrate VMs from one LUN to another VAAI supported LUN because all my other LUNs that have been created previous of DSM 4.1 say "VMware VAAI Support....No".
When i go to add a new iSCSI LUN to an existing iSCSI target, the LUN never gets created. It goes through the entire process, then I click Apply and it says "saving", but then the new LUN is never added. Yet, I can add a new LUN when it's not associated with an existing iSCSI target. Another problem is that when that new LUN is created, I cannot delete that LUN.
I tried adding 51-LUN3 and 51-LUN-4 associated with the iSCSI Target "51", but it never gets created
I created a new LUN called TESTLUN-1, not associated with any iSCSI Target, it gets created, but then I cannot delete it
I created a new iSCSI Target and LUN with the wizard. The iSCSI Target "51-TEST2-TARGET" gets created but the LUN 51-TEST-01 never gets created. I can delete this iSCSI Target though.
If I take the newly created "TESTLUN-1", and try to map it to the existing "51" iSCSI Target, I get an error that the operation failed and the iSCSI Target goes offline.
So I had the idea of disabling VAAI on the ESXi side by following KB: 1033665 and then writing to the LUNs that said "VMware VAAI support...No" and it still came up with all the errors as before:
2012-09-14T03:02:50.172Z cpu3:4099)ScsiDeviceIO: 1191: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has improved. I/O latency reduced from 1259841 microseconds to 247279 microseconds. 2012-09-14T03:02:56.636Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.60014057c686360dc26ad3f9dd9387d1 performance has deteriorated. I/O latency increased from average value of 21451 microseconds to 505935 microseconds.
If you see the same issues, please update your existing support ticket.
Update 9/19/2012 3:30pm EST
Synology is starting to send out patches to fix the NFS bug. Be aware, some of the patches they are sending out are MODEL SPECIFIC.
DS1512+, DS1812+ you will be getting a patch called cedarview_2636_nfsd.pat that will fix NFS
DS411+, DS1511+ you will be getting a patch called x64_2636_nfsd.pat
DS3611XS, DS3612SX you will be getting a different patch.
All other models please stay tuned or update in the comments section.
iSCSI is a different story. For iSCSI, you MUST create new LUNs and make sure the VAAI option is set to enabled (picture shown above) and present the new LUNs to your ESXi host to no longer get errors. LUNs created previous to DSM 4.1 CANNOT be migrated because of something they are doing with the VAAI flags (thanks to Mark Achtemichuk for sharing this). The workaround I'm suggesting and which should work is that you:
1. create a new iSCSI Target
2. create new iSCSI LUNs to attach present with that target
3. present those LUNsto your vSphere 5.0 Update 1 hosts
4. start performing Storage vMotions
5. Once everything has been moved over, remove the existing LUNs and iSCSI targets from vSphere, then from DSM 4.1. Upgrade a single ESXi host to 5.1 and test.
Be advised, with the current problems, it's going to be impossible to Storage vMotion a running vCenter VM running on 5.1 from one datastore to another without experiencing issues. Stay tuned for more updates
Update 9/19/2012 4:33pm EST
NFS Patches are being rolled out. If you want your patch, please open a support ticket as I am not yet able to link the patch for download. In addition, please keep in mind this still needs to change your iSCSI upgrade strategy.
Update 9/21/2012 10:04am EST **NFS CONFIRMED FIX**
As many of you are aware, if you have an open support ticket with Synology, there are patches available to fix the NFS issue. I have tested this with 5.1 and IT WORKS. I cannot distribute the NFS patch, Synology has asked that you need to open a support ticket to receive it because there are different patches for different models. So the great news is that NFS has been fixed. Others are reporting that iSCSI has been fixed. In my case, it's still broken. It's been reported to have iSCSI work with DSM 4.1 and ESXi 5.1, you have to create new VAAI LUNs. You CANNOT migrate old existing LUNs (see my advised walkthrough on the 9/19 update). I cannot mount a newly created iSCSI Lun to my ESXi 5.1 host. To reproduce my iSCSI issue here is what is happening.
In DSM 4.1, create a new iSCSI target, this should be pretty straight forward. Then click the Disable button to take the iSCSI target offline. I have found out that you cannot add a VAAI enabled LUN to an iSCSI target when it is online. I have talked to support and this issue is being fixed for their next patch release because at this point you CANNOT dynamically add new VAAI enabled LUNs.
Create a new iSCSI LUN, make sure it is VAAI Enabled (note that you MUST use Thin Provisioning on VAAI enabled LUNs. You cannot choose to have a thick LUN with VAAI enabled).
Go back to your iSCSI target and enable it.
The problem I'm seeing is that when I mount an iSCSI target to my host, the LUNs are not showing up with the correct amount of storage and are showing only 32KB. In addition, it will only show LUN 0, even though I have 4 iSCSI LUNs attached to this iSCSI Target. Therefore, it means I cannot mount and format my datastores. Again, this is a problem *I* have been seeing and not everyone, so YMMV. If I mount the same iSCSI Target to my ESXi 5.0 Update 1 host, I still see the same error, but there are some hints. As you can see the, there is a difference in the types of iSCSI LUNs depicted in the identifier with t10 vs naa. This is leading me to believe I may have an LIO vs IET iSCSI Driver issue as I had pointed out back in Fixing Synology DS411+ iSCSI Connection Drops for VMware ESX.
Update 9/21/2012 2:08pm EST
As some comments have rolled in, not all NFS issues have been fixed. In addition, some more people are saying that for iSCSI to work, you need to completely wipe any and all traces of existing iSCSI targets and LUNs in the DSM and re-create them from scratch. This will fix some issues. In addition, here is another email I received from Synology support relating to the iSCSI issue noted from this morning:
Hi Kendrick,
Like I said in my last email, you need to wait for the hotfix or do a fresh install of the DSM using 4.1-2636. Doing anything else is just spinning tires, you have unfortunately hit a limitation. It is not a LIO vs IET issue, please refrain from spreading this disinformation.
I will update you as soon as I know when the hotfix will be released, but it’s still tentatively scheduled for early next month.
For the time being, I going to refrain from updating to vSphere 5.1 and will wait for the hotfix patch.
Update 10/02/2012 11:50am EST
Synology has released an update DSM 4.1-2647 that has fixed many of the bugs we have been seeing. This morning I updated my Synology, let it reboot, then continued with my 5.1 tests. Mounting a NFS volume works flawlessly. Running a VM from the NFS volume proved no problem. I then created a new iSCSI Target and created a new VAAI enable iSCSI Lun. I was successfully able to mount the iSCSI Target to my 5.1 host and complete an installation of Windows 2008 R2. I did recieve a few of the errors that we saw originally but only a handful compared to the hundred before. This log was for the entire Win2k8R2 installation and then deleting the VM. In addition, the bugs that I had seen previously when trying to remove VAAI Luns and iSCSI drops has been addressed as I was told by Synology support in my last update.
~ # tail -f /var/log/vmkernel.log 2012-10-02T14:04:38.821Z cpu2:628449)MemSched: vm 628449: 7756: extended swap to 27336 pgs 2012-10-02T14:04:38.938Z cpu2:628449)VSCSI: 3775: handle 8193(vscsi0:0):Creating Virtual Device for world 628450 (FSS handle 7140055) 2012-10-02T14:04:39.005Z cpu1:628450)VMMVMKCall: 208: Received INIT from world 628450 2012-10-02T14:04:39.007Z cpu1:628450)PVSCSI: 3070: scsi0: wdt=1 intrCoalescingMode=2 flags=0xf 2012-10-02T14:04:39.008Z cpu1:628456)Net: 1858: connected New Virtual Machine eth0 to VM Network, portID 0x2000009 2012-10-02T14:04:39.045Z cpu3:4940)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0) 2012-10-02T14:04:41.074Z cpu1:628450)PVSCSI: 2390: Failed to issue sync i/o : Busy (btstat=0x0 sdstat=0x8) 2012-10-02T14:06:15.545Z cpu1:4457)BC: 3115: Blocking due to no free buffers. nDirty = 3 nWaiters = 1 2012-10-02T14:06:20.137Z cpu2:4947)BC: 3115: Blocking due to no free buffers. nDirty = 2 nWaiters = 1 2012-10-02T14:06:33.815Z cpu3:628553)WARNING: UserLinux: 1331: unsupported: (void) 2012-10-02T14:12:51.639Z cpu1:5540)BC: 3115: Blocking due to no free buffers. nDirty = 0 nWaiters = 1 2012-10-02T14:16:25.872Z cpu3:628456)NetPort: 1380: enabled port 0x2000009 with mac 00:0c:29:69:a8:99 2012-10-02T14:17:41.759Z cpu0:628456)NetPort: 1574: disabled port 0x2000009 2012-10-02T14:17:59.225Z cpu3:628456)NetPort: 1380: enabled port 0x2000009 with mac 00:0c:29:69:a8:99 2012-10-02T14:18:25.841Z cpu0:628454)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 21589 microseconds to 692493 microseconds. 2012-10-02T14:18:26.229Z cpu1:4097)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 692493 microseconds to 134071 microseconds. 2012-10-02T14:18:33.582Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 21802 microseconds to 448306 microseconds. 2012-10-02T14:18:33.997Z cpu3:4099)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 21802 microseconds to 951832 microseconds. 2012-10-02T14:18:37.096Z cpu1:628454)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 951832 microseconds to 186875 microseconds. 2012-10-02T14:18:42.062Z cpu2:4098)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 186875 microseconds to 43614 microseconds. 2012-10-02T14:19:06.217Z cpu1:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:19:06.217Z cpu0:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:19:06.217Z cpu0:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:19:32.967Z cpu1:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:19:32.967Z cpu0:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:19:32.967Z cpu0:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:19:41.839Z cpu1:628456)NetPort: 1574: disabled port 0x2000009 2012-10-02T14:19:42.131Z cpu0:628449)VSCSI: 6335: handle 8193(vscsi0:0):Destroying Device for world 628450 (pendCom 0) 2012-10-02T14:19:42.168Z cpu1:4128)Net: 2475: disconnected client from port 0x2000009 2012-10-02T14:21:45.491Z cpu2:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:21:45.491Z cpu0:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:21:45.491Z cpu0:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:21:58.134Z cpu2:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:21:58.134Z cpu0:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:21:58.134Z cpu0:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:33:38.744Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 15899 microseconds to 333734 microseconds. 2012-10-02T14:33:42.381Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 15982 microseconds to 1070728 microseconds. 2012-10-02T14:33:44.432Z cpu2:4098)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 1070728 microseconds to 211824 microseconds. 2012-10-02T14:33:45.893Z cpu3:4099)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16023 microseconds to 440025 microseconds. 2012-10-02T14:33:46.084Z cpu2:4098)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16031 microseconds to 1335644 microseconds. 2012-10-02T14:33:46.385Z cpu1:4097)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 1335644 microseconds to 266140 microseconds. 2012-10-02T14:33:47.570Z cpu3:4099)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16091 microseconds to 325984 microseconds. 2012-10-02T14:33:48.065Z cpu2:628438)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16091 microseconds to 711278 microseconds. 2012-10-02T14:33:48.684Z cpu0:4100)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16091 microseconds to 1505027 microseconds. 2012-10-02T14:33:49.627Z cpu1:4097)WARNING: ScsiDeviceIO: 1211: Device naa.6001405086e0688db161d3c71da451da performance has deteriorated. I/O latency increased from average value of 16200 microseconds to 3155224 microseconds. 2012-10-02T14:33:53.519Z cpu0:4100)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 3155224 microseconds to 614815 microseconds. 2012-10-02T14:33:53.908Z cpu0:4100)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 614815 microseconds to 121099 microseconds. 2012-10-02T14:33:54.849Z cpu2:4098)ScsiDeviceIO: 1191: Device naa.6001405086e0688db161d3c71da451da performance has improved. I/O latency reduced from 121099 microseconds to 32688 microseconds. 2012-10-02T14:34:01.263Z cpu1:628443)NetPort: 1380: enabled port 0x2000008 with mac 00:0c:29:58:f6:72 2012-10-02T14:39:37.892Z cpu1:628443)NetPort: 1574: disabled port 0x2000008 2012-10-02T14:39:37.921Z cpu1:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:39:37.921Z cpu2:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:39:37.921Z cpu2:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:39:48.504Z cpu3:628438)VSCSI: 2370: handle 8192(vscsi0:0):Reset request on FSS handle 5976751 (0 outstanding commands) 2012-10-02T14:39:48.504Z cpu2:4170)VSCSI: 2648: handle 8192(vscsi0:0):Reset [Retries: 0/0] 2012-10-02T14:39:48.504Z cpu2:4170)VSCSI: 2446: handle 8192(vscsi0:0):Completing reset (0 outstanding commands) 2012-10-02T14:39:53.233Z cpu3:628443)NetPort: 1380: enabled port 0x2000008 with mac 00:0c:29:58:f6:72 2012-10-02T14:40:33.993Z cpu0:628437)NetPort: 1574: disabled port 0x2000008 2012-10-02T14:40:33.997Z cpu0:628437)VSCSI: 6335: handle 8192(vscsi0:0):Destroying Device for world 628438 (pendCom 0) 2012-10-02T14:40:34.045Z cpu2:4127)Net: 2475: disconnected client from port 0x2000008 2012-10-02T14:40:49.062Z cpu0:4155)WARNING: VFAT: 4346: Failed to flush file times: Stale file handle
Here is what you need to know. Old iSCSI CANNOT be updated to the new VAAI format. This is what I suggest to do before migrating to 5.1. Create a brand new iSCSI Target and associate some new iSCSI VAAI LUNs to associate with it.
1. Present this new iSCSI Target to your vSphere 5.0 hosts.
2. Create the datastores, change multipathing to Round Robin, enable Storage I/o Control, and create a datastore cluster
3. Go to your datastore cluster view tab, select all VMs in the old LUNs
4. Right Click, select Migrate
5. Select to change the datastore
6. Make sure that the Format is the same as the Source, then select your newly created iSCSI Datastore Cluster as your target in the Incompatible section. You will see some warning, but the SvMotions will happen without problems. Click OK
7. Storage vMotions will begin to take place.
now you can upgrade your hosts to 5.1. I'm still going to hold off for a bit until View is supported because that's how I access my lab externally when I'm on the road.