vCloud Director networking is one of the trickiest pieces of the stack. Put it this way, if you can understand vCloud Networking, then understanding everything else is vCloud should be fairly simple. I don't want to get into the concepts of "what is an external network?" or "what is a fenced network?" because a lot of the information is already available. I would encourage you to watch Mike DiPetrillos VMworld 2010 & 2011 talks called "vCloud Networking Finally Explained." Or check out Massimo's blog post vCloud Director Networking for Dummies. This is geared towards making you understand the layers of Organizational Networks and how it relates to External networks, etc.
This is going to be mainly focused on things that need to be setup that you don't normally find in the install guides.
First thing is first, you must think about design.
How many external networks do you need? An external network should be a VLAN that A) already exists within your corporate network that you want connections attached to such as a dev/test network, connections to IP based storage, etc or B) VLANs for vCloud external access to the internet. The amount of external networks is completely dependent on your environment, and will solely rely on where the VMs inside the vCloud need connections. These external networks can be Layer 2 or Layer 3 networks.
How do you want to accomplish multi-tenant networking with network pools? This is where I will briefly explain the options.
- Port-group backed = A port group must be configured manually and created as a network in vCD
- VLAN backed = specify a range of VLANs that you will use to separate tenants
- vCD-NI = VMware's mac-in-mac encapsulation method to put multiple tenants on a single VLAN
I will not touch on port-group backed or VLAN backed because let's be honest, you probably shouldn't be buying vCloud Director unless you plan on using vCD-NI Networking. If you want to do it in the standard fashion of separating tenants based on VLANs, then perhaps you should be looking at OpSource or NewScale because their portals are far more advanced than vCloud Director and they can talk natively to vCenter to accomplish the multi-tenant isolation with VLAN based port-group creation.
Moving on... now that we are choosing vCD-NI backed networks, we must choose how many VLANs we want to dedicate to vCD-NI. From my findings, you can create up to 1000 logical networks on a single vCD-NI VLAN. So what does that mean? Think of a logical network as how many organizational and vApp networks you plan on creating. So if you plan on creating over a 1000 logical networks, you will want more than 1 VLAN. The vCloud 1.5 maximum is 7500 logical networks. Another design concept is do you really want to assign 1000 logical networks to a single VLAN? In essence, it all travels on the same pipes to the same switches, so spanning them out to multiple VLANs doesn't make much sense. Another design consideration is how many different organization vDCs do you want sharing the same vCD-NI backed network. In an enterprise scenario, we know that Marketing, HR, Finance, and QA can all be coupled on a single vCD-NI VLAN, but perhaps engineering/development needs their own because they will be deploying more networks than all of those departments.
Another thing to mention is that the VLAN(s) you assign to vCD-NI need to be a Layer 3 routed VLAN. The actual vCD-NI traffic is all Layer 2, but if you want a vApp to access the outside world on that external port-group we created, inter-vlan routing must be enabled. At this time, go ahead and create the VLANs you plan on using on your switches in your infrastructure so they are ready to be deployed later on.
NOTE: Your external networks and network pools CANNOT share the same VLAN. For instance, my external network is VLAN 6, and my vCD-NI network is VLAN 8. Both cannot use VLAN 6. In addition, assign a VLAN to these networks because it's best practice to keep everything off of default VLANs for predictable behavior.
Now that we have a very simple design, let's get started with the setup of what needs to happen even before you start installing vCloud Director. Go into your sphere client and we need to make a change to the vNetwork Distributed Switch. Right click on your dvSwitch and click Edit Settings. Change the MTU of your dvSwitch to >1600. For ease, 9000 is the maximum and there is no penalty by setting it that high.
At this time, it's best to mention why we are configuring jumbo frames. vCD-NI networking within vCloud 1.5 requires 1600MTU packet sizes to avoid fragmentation (Taken from Architecting a VMware vCloud). DON'T FORGET: Every device within this Layer 2 transport MUST be set at an MTU >1600. That includes any upstream or downstream switches that this packet can traverse. You do not need to worry about any device outside of the L2 domain. As said before, 9000 is just the easiest to set and forget.
We have to create a new port group on your vNetwork Distributed Switch for external network connections. I called mine External-VLAN-6. You can create as many external networks as you wish, but remember that the purpose these serve is to access resources outside of the vCloud such as the internet, a test/dev VLAN, est. In my case, it's for vApps to access the internet. This VLAN must be L3 and have a default gateway assigned to it as well.
When I took my vCloud Design course based on vCloud 1.0.x, it was best practice to set the port binding on this port group to ephemeral due to limits on vDS ports. A restricted number of port groups can cause issues because the number of connected machines is unpredictable. Every NAT-routed organization will need an available port. So to play it safe, ephemeral binding can be used, but static binding has better security. Hopefully, one of my vCloud buds that know more than I, can comment if this practice has changed with vCloud 1.5. For now, I'm going to set this as ephemeral because once the type of binding it set, it cannot be changed unless there are no ports being occupied. Therefore, it's best to make this decision 1 time because it's hard to change it without having to move or delete everything and start over on the networking setup. I would also suggest that you set your proper Teaming and Failover settings as Route Based on Physical NIC Load and Failback set to NO.
I also created an additional portgroup called "TemplateTest" which is the port group I use for testing the functionality of the templates/VMs before importing them from the vSphere environment into vCloud Director.
Go ahead and install vCloud Director. I'm not going to go through the steps because that is already documented in plenty of other places. How To Install VMware vCloud Director 1.5 From Beginning to End Go through the process of adding a vCenter server and creating your first provider vDC.
Now we need to create an external network. Every external network must be pre-created in vSphere before vCloud Director can take control of it. This is where we will assign my "External-VLAN-6" port group.
We must also assign networking properties to this VLAN. This is why I said this VLAN must be L3 capable and have a gateway assign to it.
Give the network an identifiable name
and click finish.
And now we have a new external network created within vCloud Director. Note: Nothing has changed in the vSphere Networking view as of yet
Now comes the fun part of creating network pools. We are going to want to create a Network Isolation backed network pool.
Now we type in the maximum amount of configurable pools which is 1000 and type in the VLAN number that should be used. Remember: External Networks and Network Pools cannot share the same VLAN. VLAN(s) assigned to vCD-NI should be configured on all switches that traverse the L2 network and should be given L3 capabilities to route out to the external network portgroup we provided earlier.
Give it a unique name
and click finish
We now have a network pool setup
Before proceeding to the next step, we need to configure the MTU properties of this network pool. Right click the network pool and hit properties.
Change the MTU settings to 1600. This will prevent fragmentation of the vCD-NI frames.
Next you can create a few organizations
Now we can create organization vDCs and assign the newly created network pool to them. This is where you must make sure that you ration out vCD-NI networks effectively as a Cloud Admin. You have 1000 networks available to in a single vCD-NI backed network. You can give every Org vDC the ability to create up to 1000 networks, but a limit may hit. Think of it like oversubscription of resources within vSphere. You know what thresholds there are and how much you can oversubscribe, just be conscious of your decisions.
Make sure you are descriptive in creating Org vDCs because you can create multiple Org vDCs for a single Organization. For Instance, If Dr. Pepper wants a Silver only tier, then I can create a DrPepper-Silver Org vDC. If Pepsi wants a Gold and Bronze Tier, I can create Pepsi-Gold and Pepsi-Bronze Org vDCs and have those all be assigned under a single organization of Pepsi.
Last thing to do is create Organization Networks. I select the organization or department that needs a network.
On the next screen I really have multiple options available to me. I can create an internal network with a routed external network. This routed external network will create a vShield Edge device that will take care of NATing to the outside world. This will allow VMs on vApps to be placed on either A) internal networks and allow it to talk with other VMs on that internal network or B) on an external network and let it talk to the outside world and have it be assigned an IP via NAT from vShield Edge, the VMs are also assigned unique IPs via DHCP from vShield Edge. This is the default behavior when creating an Organizational Network
I can create a similar type of network but remove the vShield Edge appliance by selecting a direct connection to the external network. This removes the NATing ability from vShield Edge, but also removes scale at the same. This will allow you to create an internal network that will have a unique IP scheme, and an IP address is used from the Scope for the external network for outside access. In my case, an IP address from 192.168.60.50-200 will be used. In For any VM that is placed on the external network, an IP will be used from that range as well.
I can create an internal network only. This internal network allows VMs within a vApp to talk ONLY to each other. There is no access to the outside world.
Lastly, I can create a direct connection to the external network. Think of this like placing a VM on a vSphere port group. It's about as simple as it can get.
I will choose the default setting and move on.
The internal network will be configured to use the vCD-NI network.
I can assign any IP range I want to it. These IP ranges can be used across Org vDCs as well so you can have multiple Org vDCs using the same subnets but keeping multi-tenancy through the use of vShield Edge appliances.
Give it a unique name
Now we are going to proceed with our external organization configuration. choose the external network this Organizational Network needs access to and a network pool that can be utilized.
Enter in our IP address settings. Again, since this is utilizing a vShield Edge device, we can use any IP addressing scheme we want and it can also be used by other external orgs sitting behind a vShield Edge appliance.
Give it a unique name
and click finish
We will see the networks being created in vCD.
If we look behind the scenes inside of vSphere we can see new portgroups being created
In addition a vShield Edge device was deployed for the external network connection.
There you have it, that's how you setup networking inside vCloud Director from start to finish. Now you can begin playing with vApp creation and fenced networking.