EDIT: 11/29/2011
I have no updated these to reflect vSphere 5 changes!
VMware vSphere 5 Host Network Design Layout and Configuration
Another free consulting gig from yours truly. I was asked for some help through VMTN Forums and LinkedIn on planning a host NIC design. The design incorporated 6 NICs and it was going to be used as a proof of concept based on all vSphere features. Well, I couldn't just do a design based on that alone so I figured I would diagram out a few different solutions for the masses.
So here we go. If you want to do just a proof of concept and don't care about doing it "right", you can always design it as pictured below. This design gives you the ability to incorporate all of vSphere's features and should have plenty of bandwidth to take care of everything. Of course, this is proof of concept so I didn't take NIC redundancy into account. This design is solely to give you the ability to test out all of vSphere's features. One thing I constantly see mistaken on VMTN forums about NIC design is the Fault Tolerance network. To my understanding, when you enable FT on a VM, there is going to be tons of traffic flowing through that particular NIC and VLAN so you want to have it segregated from everything else. So if you are designing a vSphere environment to use Fault Tolerance, I would think about adding a few more NIC cards and checking out my blog post on vSphere Host NIC Design - 10 NICs. **UPDATED 5/28/2010** Check the bottom of the page for a layout to use FT.
NOTE: vmnic0, vmnic2, vmnic3 must all be configured as trunk ports on your physical switch, use VLAN tagging on your vSwitch Port Groups to allow traffic to flow.