I have been able to successfully install ESXi 5.0u1 on an Intel DQ77KB motherboard with firmware 0043, although there are a couple of things to be aware of.

The first thing to be aware of is that you have to install ESXi 5.0u1 using the install option of "formatwithmbr". You do this on the install screen by hitting shift-O and then entering that option without the quotes. If you don't do that ESXi is installed on a GPT partition, and apparently the DQ77KB motherboard has problems booting to a GPT partition. You get an error saying no bootable device was found, although if you interrupt the normal boot sequence by hitting F10 and manually select the device then it will boot that one time to the GPT partition; however it won't automatically boot to a GPT partition without manual intervention each time. Installing ESXi on a MBR partition completely solves this, and I did not need to enable UEFI.

ESXi recognizes all the devices on the motherboard except the second ethernet port (i.e. the one used by Intel AMT). I used a 1tb hard drive connected to one of the 6gb SATA ports and that worked fine for the install and the datastore.

From my initial testing the Intel AMT and KVM support works fine and I can see and control the ESXi console from VNC Viewer, and I can turn off, reboot, or go into the bios from there.

I've also done some initial testing of the vt-d capabilities of the motherboard and while they do work there are some unusual results. I've done my tests with the USB controllers and an Intel 5300 wifi mini-pci express card. The mini-pci express card seems to work fine and it is recognized inside a VM, although I haven't actually connected to anything from it yet. The USB controllers though are exhibiting some strange behavior. The board has 6 usb ports and ESXi shows two different USB controllers and one item labeled "unknown" which can be assigned to VMs. It looks like if I assign any USB controller to a VM all the USB ports get assigned to that VM no matter which port I pick. Also, the USB 3 ports are identified within the VM as USB 2 ports, so I'm unable to get USB 3 speeds. I did verify that I can get near native USB 2 speeds on a hard drive whose USB controller is assigned directly to the VM, whereas if you pass through the USB device only via ESXi, the throughput is more like USB 1.1.

I'll post more updates as I do more testing.