I haven't seen a comprehensive public guide but here are some notes from a sourceforge project that was started a while back to create extra drivers for ESXi (these notes are for ESXi 3.5 but should be Ok with 4.0).
2009-08-07 02:34:31 UTC
The wiki pages are now gone. Here is the text from them, which I got from Google's cache.
This project is an effort to expand hardware support for the ESX and ESXi platforms, especially ESXi due to the fact that VMware is now licensing it for free. This will allow wider use of it, especially on commodity and Whitebox hardware - hardware not typically supported by VMware.
Software Development, especially Linux Kernel development, is fairly new to me, so this project could definitely use some knowledable and talented developers to help support the effort.
Getting Started with VMK Driver Development
You'll need the following the get started developing drivers for the VMK environment:
* GCC 3.2 or 3.3 - I know this is a little outdated since most modern distros ship GCC 4 or better nowadays, but this is specifically stated in the GPL Sources from VMware, and, trust me, I've tried it with GCC 4 and ran into issues. I've actually found it easiest to create a Virtual Machine running CentOS3 (latest update is fine) as this pretty well matches the environment in which the drivers actually run.
* VMware GPL Sources for ESX and/or ESXi - These are available in the Open Source section of the VI Download page. When you download this package, you'll get a tarball that has the VMware Public Sources (including the vmklinux emulation layer, busybox, glibc, and a few other utilities used in ESX/ESXi. The package also includes the VMware Driver Sources, which has source code and a build script for the drivers currently included in ESX/ESXi. Also, don't worry too much about the build version on the drivers - I'm currently building using the 123629 build of sources, but I've loaded the drivers in earlier and later versions of ESXi.
Once you have your development system set up with the correct version of GCC, download the VMware GPL Sources and unpack the tarballs. You should end up with a couple of key directories - a VMware-esx-public-sources-<build> directory and a VMware-esx-drivers-public-sources-<build> directory. The very first step is to go into the drivers-public-sources directory, then into src/include. In this directory, you need to create a symbolic link up to the VMware-esx-public-sources-<build>/include/lkheaders/asm-i386 directory. I didn't find this documented anywhere in the GPL sources, but it is necessary for the drivers to build correctly. Once you've done this, go into the root of the driver sources directory and you should be able to execute the build-vmkdrivers.sh script and have all of the current drivers build correctly.
Assuming the drivers build correctly, you're now on the road to being able to develop drivers for ESX and ESXi! If you want to check out the current SVN code and build the drivers in that code, do the following (in addition to the steps above):
* Check out the svn code using something like this:code
# svn co https://open-vdrivers.svn.sourceforge.n ... vers/trunk
* Go into the open-vdrivers/src directory and create a symlink to the VMware-esx-drivers-public-source-3.5.0-<build>/src/include directory:code
# ln -s ../../VMware-esx-drivers-public-source-3.5.0-123629/src/include
* Execute one or more of the build scripts inside the SVN tree to build the drivers.
Another couple of notes about this, though...
* By default, the flags in the build script for all of the drivers include the -DVMX86_DEVEL and -DVMX86_DEBUG. The DEBUG flags seems to cripple the drivers in some form or another - for example, if you build the drivers with this flag and then try to load a network driver, the driver will load, capture the PCI network device, but you'll get a warning in the VMKernel log file about a missing open function for the driver, and the driver will not present vmnic interface(s) for the card. If you remove the VMX86_DEBUG flag, the driver loads completely correctly, including setting up the correct interfaces. I'm not certain about the effects of the DEVEL flag, but I've removed that one, too, from my builds.
* The script to build the drivers is less than ideal (again, sorry VMware, no offense). Each line of the script specifies it's own build flags, instead of using variables. We're going to work on creating some Makefiles for the GPL Sources and use some of the features of that system to make this easier, but just be aware that if you want to compile a set of working drivers, you're going to have to go through each line that compiles one of those drivers and remove the DEVEL and DEBUG flags. This can be done in a pretty smart fashion using some search and replace and regular expressions present in most editors, but it's still a pain. You've been warned.
* The script also removes all build output by default before compiling, so be aware that if you create a separate script for a driver you're working, you may see the binary output disappear when you run one script or the other. Toward the top of the script you can remove the rm command that accomplishes this.
This page describes the steps necessary to generate the .tgz files that will allow your driver to load in ESXi. There are a couple of ways to go about this, both have pros and cons. The first method involves creating a separate .tgz file for the new drivers and updated simple.map file. I consider this slightly easier, as you don't need to remember all of the paths that go into the existing .tgz files if you decide to update those. Here's how to create and use your own .tgz file.
1. First, you'll need to make sure you have ESXi service console (aka "Unsupport Console" or "Tech Support Console") and/or SSH access to the machine.
2. Once you get logged in, copy your files into the appropriate place and/or edit the /etc/vmware/simple.map file. New or updated drivers should be placed in the /mod directory. Binaries should go in either /bin or sbin.
3. To create a tarball that includes the files, make sure you're in the root (/) directory and then execute the following command: tar czf mymods.tgz <path/to/file1> <path/to/file2>...Using absolute paths is pointless, as tar removes the leading slash. I also don't recommend doing entire directories - best to just include the files your modifying (unless you're modifying a lot of them).
4. You now need to place the mymods.tgz file in the correct location for boot. I boot off PXE, so I copy the tarball off to a PXE server and edit my PXE boot configuration. If you're using a local disk and/or USB device, copy it to the /bootbank and /bootbank2 directories.
5. Now you need to edit the boot configuration. With a PXE boot system this will be where you configured it; with a local drive it will be called boot.cfg in both /bootbank and /bootbank2. Then you'll want to find the line that has the modules in it (PXE: append; Local: modules=) and add the mymods.tgz file into the line. Note that you'll also need to add an additional "---" to separate it from the other modules. So, a boot.cfg file looking like this:
modules=binmod.tgz --- environ.tgz --- cim.tgz --- oem.tgz --- license.tgz
would become this:code
modules=binmod.tgz --- mymods.tgz --- environ.tgz --- cim.tgz --- oem.tgz --- license.tgz
6. That's it - you should be able to reboot and have the system come up with the modifications you've made. Note that if you do something incorrectly, you have a bad driver, etc., you may get the PSOD - make sure you test things before creating this boot archive! Also, if you're running locally and you run into issues, try pressing Ctrl+O (letter "o") right as ESXi is beginning to boot to get the boot menu - you should be able to interrupt the boot and remove the mymods.tgz from the file.
As mentioned before, the second method for adding or changing files in ESXi is to modify the files and then regenerate the default .tgz files. The advantage of this method is that you don't have another .tgz to worry about and you use the limited /bootbank space more efficiently, but this is a little more difficult and has more risk associated with it in terms of messing the system up to the point where it won't boot. Here's how to regenerate the original binmod.tgz file:
1. Gain command-line access to the system.
2. Make any modifications necessary, including adding your own drivers, modifying the simple.map file, etc.
3. Next, you'll need to list out and note the files that are in the default binmod.tgz file. To do this, run:code
# zcat binmod.tgz |tar tf -
4. Note all of the files listed. Now you'll need to run the following command to regenerate the binmod.tgz file (might want to back up the original, first):code
tar czf binmod.tgz <list of files>
5. where <list of files> is all of the files output from step 3 plus any additional ones you've added. Once you've done this you should be able to reboot without any configuration changes. Of course, if you mess something up and your system won't boot anymore, you're going to have to interrupt the boot process with Ctrl-O and attempt to change the boot sequence to use the backup copy. You did make a backup, right