Okay, so I have been roaming around here for at least 6 months.
I need to replace my current "server" setup which is running VirtualBox, on top of opensolaris.

I have decided to give ESXi a try.
Problem is, I am not even a bit close to choose hardware..


NOTE:
Since Im running it at home in my closet , I can only handle 1 machine. Everywhere I read, all I see is recommendations like:
Run 1 machine for you VM-hosts, and 1 physical machine for your storage.

Problem is, I cant "afford" to have 2 machines due to several reasons.

== Basic setup ==
A. I need to have my storage in a VM. Is this totally insane? I _NEED_ it to be run on Solaris, or at least a system that supports ZFS, since i already have an 8 TB ZFS array.

A(1), Whhyyy is the storage options in ESXI so limited? It would rooock if EXSI for example would have native support for ZFS...

Also, I store almost _everything_ on my server, so I really need as close as possible to GIGABIT throughput... Will this be possible with the storage in a VM?
Comment on this one:
My old 1,2 ghz CELERON, with debian, was able to give me like 850 megabits...... So if a new nice and shiny Sandy Bridge with an VM running from ESXi cant give me gigabit today, I must say I'm really disappointed in all this "OMG virtualization is the way to go today" talk....

B. Due to (A), im thinking of a chipset that supports passthrough in order get the harddrives directly inside my storage VM.

C. My worst problem, WHY OOOOH WHY are the most new motherboards mini/micro atx?
I need a motherboard with as many as possible:
PCI
PCIe
S-ATA
NIC´s ( INTEL... )

I have checked out the Q67 chipset, but for example, INTEL only seems to have micro/mini ITX.
http://www.intel.com/products/desktop/m ... /index.htm
Also, ASUS only seem to have one Q67 board, and its also a small form factor:
http://commercial.asus.com/product/detail/103

D. I really would like to have some lights out management, therefore Q67 also seems like a good choice with the vPRO.

E. I want to build something that lasts. I could even go for a DUAL-CPU board. But most dual-cpu boards seems to be quite "old" and "only" support older (2-3 years old?) xeon series and such.
I have heard good things about SuperMicro, for example.

F. Can you recommend me some hot-swap S-ata drive bays/backplane?
I have checked out SNT-3051 for example, but I have read several reviews of those more "end user" backplanes that said that they broke down after a while
I will have between 8-10 disks.


G. Almost forgot, I guess I will be needing a dedicated S-ATA card of some sort. Everywhere I read, everyone recommends different things...
I also have a hard time understanding how cards are specced for up to 64-128 drives, but only have like 1-4 physical "ports"...
Do I need some gigantic backplane with 1 connector to the raid card, and than a lot of harddrive connectors?


== VM:s ==
1-4x lab-machines

1x router os, e.g pfsense (this one should be able to handle my 100/100 Mbit line)

1x Storage os, ZFS, passthru/raw import of the disks (as close as possible to gigabit thruput, really (!) important)
The storage OS will also serve ISCSI shares from which I will boot my HTPC's from...

2x linux "live" servers with ftp/http/voice-comm (mumble)/irc/....
Those will probably increase over time..


Can you please give som help and point me in a good direction?
Nothing is written in stone, I just want to end up with a good system.

I have a couple of X8DLT-3F MBs for my lab and I really like them. They're not small and have a fair number of slots. The PCI bus setup is such that I can push through a number of cards into VMs (if I needed to).



They would need to support a few other file systems as well . You can do this in two ways
1) Raw device mappings - see the home page for SATA RDMs. Note that you'll be limited to 2 TB drives. Also take a look at this - viewtopic.php?f=16&t=3067&p=11676&hilit=RDM#p11676
2) Use VMDirectpath to pass through a storage controller into the VM. In that case the VM would have exclusive access to the controller (and attached disks).



You'll be able to push a gigabit easily over the network, but disk performance will be a different issue. For the best performance go with a RAID controller with battery backed write cache.

I beleive the 3051 has individual SATA connectors so it may not be the best option for a back plane.

Thanks for the input.
The problem is I do not know how to find a good RAID-card that works well with VMDirectpath, and how to find a backplane that works well with that controller.
And since all hardware in this segment is kind of pricy, I dont have the luxuary to just buy&try

Also, it feels like my requirements are quite high, way over the "standard" whitebox user.
The few ppl I have talked to only refers to "high end" enterprise stuff, I.e dell and HP blade servers, 16 bay's rack s-ata backplanes and such

Of course, with an unlimited budget, I would have no problem at all building a system that works and gets me satisfied.


The problem is I need to build a whitebox good enough to fulfill my requirements, at the same time I cant afford to buy "wrong" hardware that doesnt work good enough, and also, to the lowest possible cost.... mission impossible?


And to make my performance worries more clear:
I need to, in some way, calculate if the box will be able to deliver gigabit-speeds to/from the harddrives, at the same time handling the 100/100Mbit internet nat:ing, while running the "live" debian boxes with voice-comm without problem.

If this is not possible, then I have to take another strategy, and there's no meaning to choke up alot of $$$ for a whitebox that in the end wont give me the performance I need.
So:
How do you calculate how much harwrare you need?
Again, the ppl I have talked with, goes like "well, we buy a blade rack, and install enough blades until we have enough computing power for our needs"....

How many drives do you plan to support? If you're looking at 8 or less then you could skip the backplane.

Performance wise what sort of I/O do you expect to support? Overall there's a bit of a hit, but with BBWC you should get very good performance.

Really appreciate your help and input!

I will probably not go over 8 drives, I mostly want the backplane for easy switching/hot swap of the drives in case of a failure.


Performance:
I guess there are two issues here:

1. I dont want the network activity and disk write/read at gigabit speeds to make the other VM's suffer, example given:

Internal network, gigabit speeds:
One machine is storing files to VM #1 (file server)
One machine is reading files from VM #1

External network:
* Some http-load ( VM #2) ("some" but expect around 10-20 concurrent HTTP users at prime time)
* ~20-30 people connected to voice comm (mumble) (VM #2)
* A couple of ftp file transfers (expect my 100/100 Mbit to be fully utilized) (VM #3)


In this scenario, I dont want the MUMBLE performance/ping (response time) to flip out due to the fact that the gigabit file transfers on the internal network eats upp all resources (cpu, network bandwidth).

I also want VM #3 to be able to handle 100/100 Mbit (~10 megs / sec) ftp-transfers, while utilizing full gigabit speeds internally.

All the external traffic will also probably be run thru VM #4 which will run PFSENSE or similar, since I will need to QoS the traffic (lowest prio on file transfers, highest prio on voicecomm and gaming).


2. I want the disk writes to be able to get as close as possible to gigabit speeds. When I used my "single debian" setup on the old Celeron, I got around 80 megabytes/sec over the gigabit network to my workstations, and that's what I aim at getting.

So in other words, 80 megs/sec, and at the same time still be able to serve the other VM's with enough disk-performance to continue to keep business as usual.




Trying to summarize:
Disk activity:
1: Read (as close to 125 meg/sec as possible)
2: Write (as close to 125 meg/sec as possible)
3-6: Around 3 reads (ftp users) of a maximum total of 100 Mbit ~= 12 meg/sec
7>: All the other read/writes the other VM:s need to do for "normal" operation


CPU usage:
VM#1 Storage, needs to be able to handle all the reads + network traffic
VM#2 HTTP + voice vomm
VM#3 FTP sessions
VM#4 NAT "router", needs to be able to handle the QoS inspections



I realize this is a way complicated question...
The more I write and think about this, the more I start to doubt that it will be possible on consumer hardware

> 1. I dont want the network activity and disk write/read at gigabit speeds to make the other VM's suffer, example given:

I thought you were thinking of doing storage within a VM. I.e. no network storage.

> In this scenario, I dont want the MUMBLE performance/ping (response time) to flip out due to the fact that the gigabit file transfers on the internal network eats upp all resources (cpu, network bandwidth).

In this case you could seperate the traffic using different vSwitches (and physical NICs).

> Trying to summarize:
Disk activity:
1: Read (as close to 125 meg/sec as possible)
2: Write (as close to 125 meg/sec as possible)
3-6: Around 3 reads (ftp users) of a maximum total of 100 Mbit ~= 12 meg/sec

In regards to the I/O and bandwidth needs, is your Internet connection able to handle this? Also is the I/O typically sequential reads and writes or random. If it's random it would be a challenge to push the network as the disks will be the bottle neck.

The #1 VM, file server, will share it's different ZFS pools to the network.
The other computers have either:
* 0 harddrives, they network boot over iscsi, from the server.
* 1 SSD disk, for OS and Applications, but store all other data over the network to the fileserver.

This is why i really need the #1 VM (file server) to be able to handle BOTH gigabit network speeds, and good harddrive performance (80 megs/sec+)

At the same time, those shares will also be mounted in some other VM's, where they will be "shared" with http/ftp

I use this setup since it's convenient, I have all data at one place. Easy to back up, and it's available from every machine, and at the same time also remote available via Internet/VPN.
Also, If I run out of space, I just add one extra drive at one place.
And I minimize noise/heat in all other machines.


Hm, okay. But I reason like this:
If my storage VM, and the other VM's, use too much CPU time, it will not matter if I have 1 or 10 vSwtiches, since CPU time is a limited resource.
(ESXi still needs a bit of processing power to handle everything "below the hood" i guess, and if my VM's use 100% of the CPU, I guess I will start to see problems. Either ESXi will use the processing power necessary, but this will choke the VM's, or the VM's take what they need, and then in turn choke ESXi)

To make it really clear, lets say I add 100 VM's (guess this is not even possible in ESXI), and then start to run lets say http://en.wikipedia.org/wiki/SETI@home on each VM.... well then I will have a problem All those VM's will kill my one CPU.

So, 100 VM's clearly wont work...
How can I be sure that my setup will work on a given hardware?

At some point I _will_ run out of raw CPU processing power, I/O power and/or memory (guess this will be my least problem).

This is why I started to think about:
A. Use a dedicated machine for storage
B. Buy a dual cpu motherboard



My Internet connection can handle 100/100 Mbit
The gigabit speeds will be internally, between the server and other PHYSICAL machines on the network. (see top of this post)
Guess it will be mostly sequential read and writes.
But since I run 2 HTPC's that boot from, and uses the network storage as its primary mount point /, I will have sequential data too.

On the CPU front are you looking at a quad or hex core? How many virtual CPUs would you need to allocate. Most of your VMs would do fine with a single I would think so with a hex core even if the VMs were loaded for CPU you might still have free cycles.



You could, but you would run out of memory before CPU. Depending on your CPU load you can run 5 - 8 vCPUs per physical core so even with a quad core / single CPU you can run 20 - 32 VMs. That adds up to a lot of memory pretty quickly. Storage and network wise I think you'll be OK. If the 4 VMs are very CPU intensive and use multiple cores today, then that could be an issue. But typically those roles might not be actually that busy CPU wise so you'll be able to consolidate them without concerns.

Thanks for all your help.

Just one final question:
I was just about to order, when I accidentially read about the new "Amd bulldozer" which seems to be scheduled for release in August/September.

The top level of those will have 8 cores... seems kind of sweet?
But it seems like they dont have HT?

Question:
A intel sandy bridge i7 cpu will give me 4 cores with HT = 8 threads..
Will those 8 treads == a bulldozer 8 core cpu?
(in terms of usable CPU's and performance in ESXi)

If you're still up in the air, or open to suggestions, about storage, this might be up your alley (just a suggestion):

http://www.overclock.net/raid-controlle ... -tips.html

That thread has been going for a few years now, but most of the pertinent info has been kept up to date on the first page by the OP.

Bottom line from it is that if you're fine with RAID0, RAID1, RAID5, or RAID10, a PERC5/i should work for you. If it's not, and you want RAID6, then a PERC6/i should work. The important parts I've gleaned from what I've read in that thread so far, is:

- If you get one, definitely get one with the most cache it supports,
- If you get one, definitely get it with a battery backup unit,
- do not flash the adapter with the LSI firmware (too many people have experienced trouble after doing so ... benefit does not outweigh the risk, imo.)
- pay close attention to posts from people that have gotten the card to work with no modification of the PCIe pins on the card. People have been posting their motherboards that they've gotten it to work in without modding it. Personally, I prefer not to modify hardware if I can. Modding is a last resort, and even then, I do it grudgingly.

This is all under the assumption that you can't afford a new RAID controller without a BBU that would cost several hundred dollars.