vSphere Host NIC Design - 12 NICs

EDIT: 11/29/2011

I have no updated these to reflect vSphere 5 changes!

VMware vSphere 5 Host Network Design Layout and Configuration

I was approached by someone via LinkedIn and the VMTN Community Forums this past week about possible ways to design a vSphere Host NIC layout. The customer has 12 NICs for each server (talk about making it easy), has Enterprise Plus licensing, but wasn't planning on using features such as Fault Tolerance or Virtual Distributed Switches. With that said, here is how I came up with a few designs.

Table Key
2 on-board NICs = vmnic0, vmnic1
4 expansion NICs = vmnic2, vmnic3, vmnic4, vmnic5
4 expansion NICs = vmnic6, vmnic7, vmnic8, vmnic9
2 expansion NICs = vmnic10, vmnic11


Option 1 easier and less configuration:
vSwitch0

• Service Console
  vmnic0 Active
  vmnic6 Stand-by
  • VMKernel for Service Console= IP Address in VLAN 50
• vMotion
  vmnic2 Active
  vmnic8 Stand-by
  • VMKernel for vMotion = IP Address in VLAN 51
This is dedicating 2 NICs to each option. I do not combine Active/Active because you don't gain any additional benefits unless LACP is implemented

• iSCSI network
  vmnic5,9,11 all Active
  • VMKernel for iSCSI = IP Address in VLAN 52
  • VMKernel for Service Console = IP Address in VLAN 52 (redundant service console is needed for HA)
If you are in charge of the network or have a network team available, use Jumbo Frames. You will see a huge performance boost.
How to on vDS - http://blog.scottlowe.org/2009/05/21/vmware-vsphere-vds-vmkernel-ports-and-jumbo-frames/
How to on vSwitch - http://blog.scottlowe.org/2008/04/22/esx-server-ip-storage-and-jumbo-frames/

• FT Network
  • vmnic11 Active
  • vmnic4 Standby
  • VMKernel for FT = IP Address in VLAN 53

Option 2
vSwitch0

• Service Console
  vmnic0 Active
  vmnic6 Stand-by
  • VMKernel for Service Console= IP Address in VLAN 50

vSwitch1

• vMotion
  • vmnic2 Active
  • vmnic8 Active
  • VMKernel for vMotion = IP Address in VLAN 51
enable LACP (Etherchannel) on your switches and use Route based on IP Hash for vSwitch
How to: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

• iSCSI Network
  vmnic5,9,11 all Active
  • VMKernel for iSCSI = IP Address in VLAN 52
  • VMKernel for Service Console = IP Address in VLAN 52 (redundant service console is needed for HA)
Use Jumbo Frames and LACP trunking.
Both how-to's are shown from Scott Lowe (Jumbo Frames) and the VMware KB (LACP).
Troublshooting: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001109

Option 2a
vSwitch4

• FT Network
  • vmnic11 Active
  • vmnic4 Standby
  • VMKernel for FT = IP Address in VLAN 53

Networking (use your own VLANs, this is just an example)
Service Console - VLAN 50
vMotion - VLAN 51
iSCSI - VLAN 52
FT - VLAN 53
Virtual Machine Network (any vlans needed by your VMs)

You can tag VLANs in 2 places. On the physical switch itself, or on the port group within a vSwitch inside of VMware.
You can tag VLANs within VMware for any of these, but you have to set your switches up to use 801.2q trunking on each port. I would also suggest specifying the VLANs on the configuration to only allow those specific VLANs to pass traffic.

You can also tag vlans on the physical switch itself for everything above (except the Virtual Machine Network if you have more than 1 VLAN or a DMZ too), in lieu of Port Group VLAN tagging.

I would personally tag Service Console, vMotion, iSCSI, and FT all at the physical switch. Then use 801.2q trunking for my Virtual Machine Network. More often than not, you will have to add new VLANs for a VM Network and nothing else. Of course, this is your own cup of tea, so if you want to set every single switch port up to be trunked and specify the VLAN on the port group, you can do that as well.

It goes without saying that you can use a Virtual Distributed Switch instead of a vSwitch in any scenario. I would keep Service Console and vMotion on vSwitches and move everything else to a vDS. That's your call.

The design and mismatch of vmnics was done with a purpose. To alleviate NIC failure, NIC expansion card failure, or On-board NIC failure. If you end up losing a 4 port expansion card, you don't want to have all your iSCSI traffic flowing over 2 NICs from the same card.

I would personally do configuration Option 2a. It's a alot more configuration, but it should yield optimum results. They might not want to use Fault Tolerance, but an extra NIC in the Virtual Machine Network and the iSCSI vSwitches aren't going to give you any benefits besides another layer of redundancy. I setup the FT network here at my work, but we don't use any FT VMs. Having it all designed out correctly from the beginning so you're not having to backtrack will make your life easier. Initial design matters most.

Please take a look at my other posts for more information and diagrams:

vSphere Host NIC Design - 6 NICs

vSphere Host NIC Design - 10 NICs