Recover an ESXi if the USB boot device has been enabled for VMDirectPath


You have activated VMDirectPath an all USB interfaces to find the right USB Port ?
Unfortunately, your system was booting from an USB device... (like me... )


Symptoms after the reboot (needed to enable VMDirectPath):

- The configuration is "read-only" / configuration modifications are lost after reboot
- the local console is frozen.
- ESXi is booting slowly.
- ESXi can't find the VMtools when you try to install it.
- You can't disable VMDirectPath.


The problem is obvious, ESXi makes available the USB booting interfaces for VMDirectPath.
The USB boot drive behind this interface is no more available for the system.
Surprisingly, ESXi don't crash, VM's start as usual (try to remove system drive of Hyper-V).


Which operations have no effect ?

- Any modification through vSphere Client (all settings are stored on the USB flash drive).
- Trying to connect in SSH on host to modify something (there is no more files to edit).
- ...


What can we do ?

Manually edit the configuration for VMDirectPath on the USB Flash drive through a virtual machine.
Let's go:

- Download the last version of Ubuntu (http://www.ubuntu.com/desktop/get-ubuntu/download)
- Load the ISO on hypervisor datastore
- Create a new VM for Ubuntu (no harddrive, no network, 1 vCPU and 512MB is enough).
- Attach the Ubunutu ISO file to virtual CD drive
- Add the VMDirectPath USB interface (which have the USB boot device)
- Boot on Ubuntu, don't install, just start as live CD
- Open the Hypervisor1 partition
- Copy local.tgz on Desktop
- Untar the copy of local.tgz through these commands:

- Open the extracted folder etc on desktop, then go in vmware folder and finaly right click on esx.conf, edit in text editor.
- find the line looks like

- Delete these lines, there is two lines for one VMDirectPath enabled device.
- Save this file on Desktop and close editor
- Replace the old config, compact the new local.tgz through these commands

- If the new tgz as been made without error, replace local.tgz on Hypervisor1 partition with the new one on Desktop.
- Reboot ESXi
- Now, you are root, a read-write root


Does it work for you ?


(VMware ESXi 4.1 / Ubuntu 10.10 x86)

I haven't tried this, but it's a slick method. I had posted something similar here - http://www.vm-help.com/esx40i/VMDirectP ... issues.php but using a VM saves a reboot. Nice job.

Thank you.
During troubleshooting, I have read your interesting post. It gave me a lot of usefull information.

I just made this mistake. I was able to boot a Ubuntu CD and mount the USB partition with the local.tgz file but it appears empty as does the oem.tgz file. Any thoughts?

I'm trying to FIX it to learn rather than just reinstall. Besides, I don't want to reconfigure all the switches again

Thanks for any help. This website is a great help and wonderful learning place.

The idea behind this worked great for me. My "empty" file was because the first tar command was incorrect. I didn't really pay attention. Once I was able to successfully extract the file and modified it then the next tar command worked fine.

I used a Ubuntu CD and booted up into it. I did my changes and then copied the modified files off to a different USB memory stick as well.

This will teach me to just enable things without thoroughly understanding them.

Thanks again for all the good information.

Thanks for feedback !

This looked like a great write up, but unfortunatly in ESXi 5 there appeared to be no local.tgz file.

It's almost cretain that I had caused the problem by accidentally setting the USB Boot device for passthough on the host.

However it's pretty hard to know which is the correct device to enable for passthough from the information given in Vspheare, so it's a pretty risky option to try to enable.

On Rebooting I got 'two file systems with same uuid have been detected' which I could bypass as a one of using Shift + O then entering "overrideDuplicateImageDetection" but could not dig myself out of this hole without doing a Host Reinstall.

You'd want to change state.tgz. In the past ESXi would use local.tgz on flash devices and state.tgz on hd installs.