View unanswered posts | View active topics It is currently Sat Dec 16, 2017 12:44 pm



Reply to topic  [ 20 posts ]  Go to page 1, 2  Next
 ESXI 5.x Drivers Part 1: Making a Build Environment 
Author Message

Joined: Mon Jan 28, 2013 5:09 pm
Posts: 100
Reply with quote
Post ESXI 5.x Drivers Part 1: Making a Build Environment
This is a write-up from the rough notes I made when I made my build environment. Hopefully it'll help you to get going too. These instructions assume you are building either a physical or virtual Linux system dedicated to this task. If you are sufficiently experienced with installing, compiling and configuring software for Linux systems then these notes should also give you what you need to adapt an existing Linux system for the same purpose.

What is the best system to build with? If you download the ESXI open source bundle, the embedded documentation will advise you to build using a CentOS 5.3 system. If you're just looking to build new drivers to add hardware compatibility to ESXI, then that is not in fact necessary. The source bundle contains the toolchain which cleanly builds the drivers, so therefore you can build the toolchain (and subsequently the drivers) using any modern linux system.

For my build environment, despite being a CentOS person, I selected Ubuntu 12.10 simply because it had a fairly small .iso file and I wasn't patient enough to wait for the latest CentOS iso.

The first step is to install Ubuntu Server. Conducting an Ubuntu Server installation is well documented on the Ubuntu website, and beyond the scope of this post, but here are relevant details from my installation which later parts of these instructions depends upon. During installation:

1) Create your user account with the username set as "build". Password to your liking.
2) During server type, make sure "OpenSSH Server" is selected a role.
3) Make sure there's at least 10GB of space for the system.

Easy huh? Once you have an installed system, the next step is to prepare the system for building software. Boot it up and login as the "build" user. All of these instructions assume you are operating as the build user ID; do not do any of this as root. Confirm network connectivity is working, and then complete the following sequence of commands:

Code:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
sudo apt-get install rpm2cpio
sudo apt-get install ncurses-dev
sudo apt-get install flex
sudo apt-get install bison


This installs most of the packages that will be needed during the build of the toolchain with the exception of the texinfo software. For that you need a specific version to be installed, and simply doing apt-get install texinfo is going to get you the wrong version. Instead we have to download the version we need and compile it ourselves:

Code:

mkdir /home/build/texinfo
cd /home/build/texinfo
wget http://ftp.gnu.org/gnu/texinfo/texinfo-4.9.tar.gz
tar zxvf texinfo-4.9.tar.gz
cd texinfo-4.9
./configure && make
sudo make install


Why do we need a special texinfo? There is a problem with one of the build components for the toolchain. It looked like the automake environment wasn't considering the makeinfo binary as being available because the reply to --version had 4.xx in it rather than 4.x. So the quick and dirty solution was to use a version that didn't break the build.

With this done we need to do a little more house-keeping and then get a quick reboot out of the way.

Code:
mkdir /home/build/vsphere
mkdir /home/build/toolchain
sudo reboot


While the system is rebooting, go on over https://my.vmware.com/web/vmware/details?productId=229&downloadGroup=VSPHERE50U2_OSS and download VMware-esx-open-source-5.0.0u2.oss.tgz. When the build host comes back, scp the .tgz file over to the /home/build/vsphere directory. Login as the build user, and then:

Code:
cd vsphere
tar zxvf VMware-esx-open-source-5.0.0u2.oss.tgz
cd vmkdrivers-gpl


If you're tempted to look at the BUILD.txt you'll see it suggests you "must" be root in order to do most of this, but that is not my experience. The build environment and instructions contained within, for the sake or clarity I guess, contains paths such as "/build" which would require root, but a simple bit of tweaking to the build scripts and they work just fine without root. So that's the direction I went in. You shouldn't really be running anything as root unless it's crucial.

Now we get to building the toolchain:

Code:
cd glibc-2.3.2-95.44
vi BUILD.txt


Change "TROOT=/build/toolchain" to "TROOT=/home/build/toolchain" and save BUILD.txt. You may need to use :w! and force the save since the file is read only.

Code:
bash BUILD.txt
cd ..


glibc should not be built and you should be back in ~/vsphere/vmkdrivers-gpl.

Code:
cd binutils-2.17.50.0.15-modcall
vi BUILD.txt


Again, change "TROOT=/build/toolchain" to "TROOT=/home/build/toolchain" and save BUILD.txt.

Code:
export CFLAGS="-Wno-error=unused-but-set-variable -Wno-error=format-security"
bash BUILD.txt
cd ..


This was a little different. Without these CFLAGS set, binutils fails to build because somehow a compile setting is activated which makes a warning level error turn in to a full error that halts the compilation. Similarly, any warning for format errors also causes a failure in this setup. I couldn't quite figure out how and where those compile options were getting set, so in the interest of getting on with things, I chose to set CFLAGS at the shell with the options to override the previous settings. This works because those flags are appended to the compilation string and they negate the previous settings. Next it's on to gcc, and you should be back in ~/vsphere/vmkdrivers-gpl:

Code:
cd gcc-4.1.2-9
vi BUILD.txt


Again, change "TROOT=/build/toolchain" to "TROOT=/home/build/toolchain" and save BUILD.txt. Yes you've done this 3 times now because I don't want you doing this as root. Sorry :-)

Code:
export CFLAGS=""
bash BUILD.txt
cd ..


Sit back and relax while gcc builds, it will take a while. We cleared the CFLAGS setting before commencing the gcc build because it wasn't necessary.

Once gcc is done, the toolchain is now built and ready for use. The next step is to validate that your toolchain and build environment is capable of producing modules that will load in to ESXI. You should be back in the ~/vsphere/vmkdrivers-gpl directory.

First we'll unpack the driver source and then we'll modify the driver build script:

Code:
tar zxvf vmkdrivers-gpl.tgz
vi build-vmkdrivers.sh


Change line 6 to:

Code:
CC=/home/build/toolchain/lin32/gcc-4.1.2-9/bin/x86_64-linux-gcc


Change line 11 to:

Code:
LD=/home/build/toolchain/lin32/binutils-2.17.50.0.15-modcall/bin/x86_64-linux-ld


Save the file, and then let's try building all of the drivers that VMWare supplies:

Code:
./build-vmkdrivers.sh


A successful output will look like this:

Code:
build@esx-build:~/vsphere/vmkdrivers-gpl$ ./build-vmkdrivers.sh
In file included from vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_init.c:12:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h: In function 'pci_free_consistent_esx':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h:1413: warning: assignment makes integer from pointer without a cast
vmkdrivers/src_9/drivers/char/random/random.c: In function 'current_fs_time':
vmkdrivers/src_9/drivers/char/random/random.c:311: warning: 'notime.tv_nsec' is used uninitialized in this function
vmkdrivers/src_9/drivers/char/random/random.c:311: warning: 'notime.tv_sec' is used uninitialized in this function
In file included from vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c:12:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h: In function 'pci_free_consistent_esx':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h:1413: warning: assignment makes integer from pointer without a cast
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c: In function 'bnx2i_cm_remote_close':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c:2578: warning: format '%d' expects type 'int', but argument 3 has type 'char *'
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c:2578: warning: too many arguments for format
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c: In function 'bnx2i_cm_remote_abort':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c:2616: warning: format '%d' expects type 'int', but argument 3 has type 'char *'
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_hwi.c:2616: warning: too many arguments for format
In file included from vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_sysfs.c:12:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h: In function 'pci_free_consistent_esx':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h:1413: warning: assignment makes integer from pointer without a cast
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_sysfs.c: At top level:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_sysfs.c:230: warning: initialization makes integer from pointer without a cast
In file included from vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c:12:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h: In function 'pci_free_consistent_esx':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i.h:1413: warning: assignment makes integer from pointer without a cast
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c: In function 'bnx2i_process_scsi_resp':
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c:2854: warning: 'return' with a value, in function returning void
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c:2859: warning: 'return' with a value, in function returning void
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c:2864: warning: 'return' with a value, in function returning void
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c: At top level:
vmkdrivers/src_9/drivers/scsi/bnx2i/bnx2i_iscsi.c:2294: warning: 'bnx2i_session_recovery_start' defined but not used
vmkdrivers/src_9/drivers/char/openipmi/ipmi_si_drv/ipmi_si_intf.c: In function 'init_ipmi_si':
vmkdrivers/src_9/drivers/char/openipmi/ipmi_si_drv/ipmi_si_intf.c:3133: warning: implicit declaration of function 'register_reboot_notifier'
vmkdrivers/src_9/drivers/char/openipmi/ipmi_si_drv/ipmi_si_intf.c: At top level:
vmkdrivers/src_9/drivers/char/openipmi/ipmi_si_drv/ipmi_si_intf.c:1652: warning: 'hotmod_handler' defined but not used
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c: In function 'megaraid_reset':
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:2222: warning: unused variable 'mc'
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c: In function 'megadev_ioctl':
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:3934: warning: cast to pointer from integer of different size
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:3986: warning: cast to pointer from integer of different size
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:4012: warning: cast to pointer from integer of different size
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:4065: warning: cast to pointer from integer of different size
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:4105: warning: cast to pointer from integer of different size
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c: At top level:
vmkdrivers/src_9/drivers/scsi/megaraid2/megaraid.c:4675: warning: 'mega_support_cluster' defined but not used
vmkdrivers/src_9/drivers/net/forcedeth/forcedeth.c: In function 'nv_probe':
vmkdrivers/src_9/drivers/net/forcedeth/forcedeth.c:6306: warning: assignment discards qualifiers from pointer target type
vmkdrivers/src_9/drivers/net/forcedeth/forcedeth.c: At top level:
vmkdrivers/src_9/drivers/net/forcedeth/forcedeth.c:4781: warning: 'nv_poll_controller' defined but not used
vmkdrivers/src_9/drivers/net/sky2/sky2.c: In function 'sky2_init_netdev':
vmkdrivers/src_9/drivers/net/sky2/sky2.c:3976: warning: assignment discards qualifiers from pointer target type
vmkdrivers/src_9/drivers/net/sky2/sky2.c: In function 'sky2_mhz':
vmkdrivers/src_9/drivers/net/sky2/sky2.c:2685: warning: control reaches end of non-void function
vmkdrivers/src_9/drivers/net/tg3/tg3_vmware.h:28: warning: 'tg3_netq_index' defined but not used
vmkdrivers/src_9/drivers/net/igb/igb_main.c:3541: warning: 'igb_update_itr' defined but not used
build@esx-build:~/vsphere/vmkdrivers-gpl$


It does not compile cleanly, but there were no build errors and just warnings.

So now the build host is running, the source is installed, the toolchain is prepared, and the provided source is compiled. All that remains is to check that your compiled modules are actually usable. To do that we need to try and load a module you just produced in to a running kernel.

First, pick a module you think you probably do not have hardware for, because in that case the module isn't likely to already be loaded.

Code:
ls -l BLD/build


You'll see all the provided drivers. I chose the sky2 NIC driver. Now we need to get the newly compiled module over to the target ESXI system. If you have an operating target ESXI server, you can simply scp the file over, otherwise if you're working with a system that you can't even get installed because of the lack of the driver you are hoping to make, you'll need to copy the module over to the system using the USB flash drive technique I describe in another post. For the sake of simplicity I will assume you have a running system you can simply scp the module to:

Code:
scp BLD/build/vmkdriver-sky2-CUR/release/vmkernel64/sky2 root@YOURHOST:/tmp


Next ssh in to your ESXI host and try to insert the new module you just copied:

Code:
build@esx-build:~/vsphere/vmkdrivers-gpl$ ssh root@YOURHOST
Password:
The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # vmkload_mod /tmp/sky2
Module /tmp/sky2 loaded successfully


Looks good, and to further confirm it's working as expected, you can:

Code:
~ # grep sky2 /scratch/log/vmkernel.log
VMB: 212:    name: /net_sky2.v00
VMB: 1359: Relocating module /net_sky2.v00 to high memory
2013-02-27T13:32:05.492Z cpu0:4096)VisorFSTar: 1743: net_sky2.v00 for 0x1c08e bytes
2013-03-03T16:28:01.204Z cpu3:246450)Loading module sky2 ...
2013-03-03T16:28:01.204Z cpu3:246450)Elf: 1852: module sky2 has license GPL
2013-03-03T16:28:01.205Z cpu3:246450)skb_mem_info mempool for module sky2 created - max size 23068672
2013-03-03T16:28:01.205Z cpu3:246450)module heap vmklnx_sky2: creation succeeded. id = 0x41001dc40000
2013-03-03T16:28:01.205Z cpu3:246450)PCI: driver sky2 is looking for devices
2013-03-03T16:28:01.205Z cpu3:246450)PCI: driver sky2 claimed 0 device
2013-03-03T16:28:01.205Z cpu3:246450)Mod: 4485: Initialization of sky2 succeeded with module ID 68.
2013-03-03T16:28:01.205Z cpu3:246450)sky2 loaded successfully.


You can see here that the driver loaded and attempted to locate a PCI device it could support but it couldn't find one.

Congratulations. You now have a build environment that can produce modules for ESXI 5.x.


Last edited by trickstarter on Wed Mar 13, 2013 12:54 pm, edited 1 time in total.



Sun Mar 03, 2013 5:12 pm
Profile

Joined: Tue Jan 22, 2013 11:31 am
Posts: 137
Reply with quote
Post Re: Creating a build environment for ESXI 5.x
Great write up! I'm gonna try this out first thing tomorrow.

Quite the contribution to this site. I hope Dave sees fit to make it a sticky.

We all owe you a great deal of gratitude. I for one, am immensely grateful.

Thanks so much, for this....


Sun Mar 03, 2013 7:48 pm
Profile

Joined: Tue Feb 19, 2013 4:04 am
Posts: 14
Reply with quote
Post Re: Creating a build environment for ESXI 5.x
Hello,

I followed this tutorial to add to my driver for my wireless card.
AR-9380 atheros chipset, I have atheros driver source but I do not see how to add this driver.
Is there manipulation of doing?

Thank you very much.
Julien


Fri Mar 08, 2013 2:42 pm
Profile

Joined: Mon Jan 28, 2013 5:09 pm
Posts: 100
Reply with quote
Post Re: Creating a build environment for ESXI 5.x
I will try to write Part 2 of this guide some time this weekend.


Fri Mar 08, 2013 5:31 pm
Profile

Joined: Tue Feb 19, 2013 4:04 am
Posts: 14
Reply with quote
Post Re: Creating a build environment for ESXI 5.x
trickstarter wrote:
I will try to write Part 2 of this guide some time this weekend.


Where will be after the tutorial when it is ready?


Tue Mar 12, 2013 10:16 am
Profile

Joined: Wed May 29, 2013 7:10 pm
Posts: 4
Reply with quote
Post Re: ESXI 5.x Drivers Part 1: Making a Build Environment
Great tutorial!!! Thank you very much!!!

I compiled as the build user as well, but added a symlink as root from /build to /home/build, so I didn't have to modify the build scripts.
Code:
ln -s /home/build /build

To avoid installing the older version of texinfo, I saved a patchfile named binutils-2.17.50.0.15-makeinfo.patch in the binutils-2.17.50.0.15-modcall directory with the following content:
Code:
diff -Naur binutils-2.17.50.0.15.orig/configure binutils-2.17.50.0.15/configure
--- binutils-2.17.50.0.15.orig/configure 2013-05-30 01:29:36.135301795 -0300
+++ binutils-2.17.50.0.15/configure      2013-05-30 01:29:51.430301919 -0300
@@ -6114,7 +6114,7 @@
     # For an installed makeinfo, we require it to be from texinfo 4.4 or
     # higher, else we use the "missing" dummy.
     if ${MAKEINFO} --version \
-       | egrep 'texinfo[^0-9]*([1-3][0-9]|4\.[4-9]|[5-9])' >/dev/null 2>&1; then
+           | egrep 'texinfo[^0-9]*([1-3][0-9]|4\.[4-9]|4\.[1-9][0-9]|[5-9])' >/dev/null 2>&1; then
       :
     else
       MAKEINFO="$MISSING makeinfo"

and added the line
Code:
patch -p1 < ../$NAME-$VERSION-makeinfo.patch

after "patch -p1 < ../$NAME-$VERSION-modcall.patch" on BUILD.txt


Wed May 29, 2013 8:41 pm
Profile

Joined: Sun Aug 11, 2013 12:01 pm
Posts: 2
Reply with quote
Post Re: ESXI 5.x Drivers Part 1: Making a Build Environment
Hi trickstarter,

Great series of posts.... U must be a esxi wizard.... I tried ur steps and was successfully able to create a module and insert in esxi... ;)

Now, i have this problem, in my real module, i deal with pages and the flags inside that.
like lock_page, unlock_page etc.
How can i port that to esxi.....
Because, from the headers i can see that esxi has defined page as
struct page {}

There are no member variables inside it......

Can u help me out with ur knowledge as to how i can get access to the page flags .....


Wed Sep 11, 2013 6:28 am
Profile

Joined: Thu Dec 30, 2010 7:55 am
Posts: 104
Reply with quote
Post Re: ESXI 5.x Drivers Part 1: Making a Build Environment
SO I just tried this, and it seems some things have changed since you wrote this awesome tutorial.

First, it looks like the glibc folder is now glibc-2.5, and the BUILD.txt in that folder does not have a TROOT declaration in it.

Since I'm just blinding following orders, maybe you'd be willing to give us an update? :)

Thanks!


Sun Oct 06, 2013 9:28 am
Profile

Joined: Wed Oct 09, 2013 11:13 am
Posts: 6
Reply with quote
Post Re: ESXI 5.x Drivers Part 1: Making a Build Environment
I actually created a script to setup your build environment. You can get it from here: https://www.dropbox.com/s/vl4pcpmf1gg5t ... v_setup.sh . In order to make it work for apt-based linux distros, just update the script to use YUM instead of apt-get.

Hope this helps.


Sat Nov 09, 2013 12:19 pm
Profile

Joined: Sat Nov 09, 2013 6:38 am
Posts: 4
Reply with quote
Post Re: ESXI 5.x Drivers Part 1: Making a Build Environment
robertgraham1906 wrote:
I actually created a script to setup your build environment. You can get it from here: https://www.dropbox.com/s/vl4pcpmf1gg5t ... v_setup.sh . In order to make it work for apt-based linux distros, just update the script to use YUM instead of apt-get.

Hope this helps.


Thanks, worked a treat (as far as i can see so far)


Tue Nov 12, 2013 5:24 am
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 20 posts ]  Go to page 1, 2  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.