With vSphere 5.5 now out, it was time to refresh my lab.  I have ESXi 5.5 installed on a DL160 G6, with 1 SSD and 3 SATA drives, so I decided to make a single node VSAN instance. (3 nodes is a minimum for a supported configuration).

The plan was to spin up some nested ESXi instances, like my usual lab setups, but I ran into a problem.  I could not get ESXi 5.5 to install – it kept failing.  Here is what I was receiving:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3’ exited with status 1048320

Thinking my ISO download was bad, I tried again, this time with the download manager, and even checked MD5 sums.  Everything seemed fine, but still the error.

After rebuilding the VM a couple times, I decided to put the VM on my NAS instead of the VSAN – success! After much testing:

You cannot install nested ESXi 5.5 on the VSAN datastore!

Now, I had no problem using Storage vMotion to move the VM over to the VSAN datastore, and it has had no issues. So, long story short – you will have to pick a different datastore for the initial install of ESXi.

Why does it fail during install?  So far, I have not found a reason.  Again, this is all for lab purposes, and completely unsupported.

** Update**

William Lam reached out to the VSAN team at VMware and was able to find out why this was happening.  See his post here. What he found was basically, when ESXi is installing, it tries to create a local VMFS volume, and when it does it uses a SCSI-2 reservation.  VMFS5 however, doesn’t use SCSI reservations anymore – so with VSAN, they were removed. Here’s the fix to emulate the SCSI reservation:

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

This needs to be run on each host in the VSAN cluster – a reboot is not required.