HI Vm'ers

Im having some ISCSI problems with my whitebox. In short with any type of load
the ISCSI connection drops, as do all other network connections on
any nic with any protocol forcing me to reboot my esx.

My Specs
esx 4.1 (have tried 3.5, 4.0, same problem)
Mobo - GA-MA770-UD3
Processor - AMD Phenom II x4
Ram - 8G
1-Intel Pro 1000 MT Desktop
1- Dual Port Dell OEM - Intel Pro 1000 MT chipset.
SAAS Sata Raid card for internal storage
Shared Storage -
QNAP TS 439 w ISCSI
Homebrew Openfiler 2.3 w ISCSI


this server has run 6vm's faithfully for over a year with local storage, but when I tried to move to iscsi things went awry.
ISCSI connection to LUN is there, I can copy small files back and forth to the shared storage.

With any load, not only do the ISCSI connections drop, but the ip connection on a separate nic to the mgmt network also drop.
With no other Vm's running on the server, and a direct crossover cable to the shared storage, I cannot even create a single new vm,
the connection will drop during the installation of windows in the guest OS.

This happens regardless of which of my Intel Pro 1000 MT nics I use as the connection to the shared storage.

The real kicker is I have also tried installing Hyper V R2 as my hypervisor SO and the exact same thing happens with this hardware.
I have confirmed my shared storage hardware is good, by connecting other devices and stressing the connections.


Any idea's? at this point I was thinking some type of motherboard issue, since this problem persists across both
esx and hyper v, and across 2 separate nics, but this is ONLY with ISCSI. High load standard TCP/IP network connections run fine.

Thoughts, suggestions, something blatant I'm missing?
Whitebox hardware troubleshooting is difficult for me. so I'm thinking of just dumping this hardware but I can't afford to replace it atm.

Appreciate any thoughts.

-MC

Have you taken a look at /var/log/messags or ALT+F12 when this happens?

I have, but did not save it. I will post it when I get home, and perhaps it will give an indicator.

Log attached.

You can see the Nic barf, and the path to ISCSI being lost. At this point I'd replace the nic, which I've done. The nic that generated this log was also in a separate ESX whitebox I have and works fine.

this load is really minimal its just the creation of one vm with nothing else running on the server or the network.

All my nics do have the intel 1000 mt chipset, so I may try and dig up something else that works.

Im currently googling everything in the log to see if that helps..

The bizarre thing is that you had the some issue with Hyper-V. When I first started to read that I was thinking it was going to be that it worked fine.

Have you tried setting the port to 100 MBps? Are you just testing this with a cross-over cable or via a switch as well?

Hi MC,

My opinion: irq troubles!

My suggestion: copy the native iperf tool to your ESXi and try to stress your NIC, meanwhile in the ESXi console run "esxtop" and see the interrupts (press <i>).

Regards.

I have tried both crossover cable and switch.
Granted my switch is VERY low end consumer grade switch.

I haven't tried setting to 100mbps.

I found some links on VMware related to interrupt problems and the e1000 driver, so I'm going to keep moving in that direction.

I have tried this, but it doesnt seem to fix things.
http://kb.vmware.com/selfservice/micros ... Id=1010313

Thanks for the suggestions thus far, I will keep acting on them.

Update:

I got lazy and swapped in a PCI-E nic - Intel PT.
All other nics that were causing problems were pci (not -e), same driver different nic, different slot type.

Everything works fine now.

I think it was an IRQ issue, that makes sense given the symptoms.

You guys rule, thanks for the responses.


MC

Hi MC,

Gigabit speeds are difficult to achieve, and commodity PCI buses (32bit/33MHz) can't handle high loads (like iSCSI) without good IRQ coalescence techniques!

My recommendation: use PCI cards only for management or external connectivity; iSCSI and Vmotion really needs PCIex!

And remember: 1 gigabit card in a PCI bus consumes all bandwidth; 2 gigabit cards requires 32bit/66 (or PCI-X 64bit).

Regards.