This is a bonehead move. I mean, seriously boneheaded. I spent probably 2 weeks trying to wrap my head around what was going on. I was tasked with getting vCloud Director and Nexus 1000v working hand-in-hand on a Vblock for documentation and testing purposes. I had 2 vCenter servers built, one used the vDS and the other used the 1000v. I had followed the setup on Tom Fojta's Cisco Nexus 1000V and vCloud Director 5.1 guide. My setup was using the latest GA of vCloud 5.1.1 and Nexus 1000v 4.2.1.SV2.1.1 which was a bit newer than Tom's, but the process didn't change.
During the configuration, I could create Provider vDCs and the VXLAN pieces would populate without issue. VXLAN Virtual Wires were being created by the Nexus 1000v and that could be seen in both the vSphere Client as well as seeing the port-profiles on the VSM. So I was certain I had configured everything correctly.
So I went to go and deploy my first vApp and it ended up failing.
I checked both links and each pointed out errors but no actual symptom of the error itself. The details just said "Error". Fantastic.
After doing a little digging, you can see that the VM actually does get cloned successfully.
When I looked in the vSphere Client, I can see the VM actually does get cloned, but it never got added to the virtual wire. The VM got added to the external network and just failed. This had me thinking it was a problem with the 1000v.
Here is what a correct deployment of a vApp should look like. This is a screenshot taken from the vCenter with the vDS. As you can see, the deployed vApp successfully gets attached to the VXLAN virtual wire.
So I thought it was a problem during the copy process of the catalog VM from the other vCenter to the vCenter with the N1KV. So I figured I would add a catalog item that is local to this vCenter. When I tired uploading an item to a catalog with an Organization vDC backed by a Provider vDC using the Nexus 1000v, it failed again! Again, no real details of an error
No joke. I spent the next week and half doing tails of the vcloud logs, ESX logs, vCenter logs, doing debugging of the 1000v. Literally nothing could point out why this was happening. It got to the point that I started calling for favors from the big dogs from both Cisco and VMware to come together and help troubleshoot this. I had VMware support on the phone for 4 hours and they couldn't find a trace of anything. I then had Cisco 1000v engineering on the phone for an hour and they couldn't see a single problem. Then yesterday, I had Cisco support on the line to see if someone in the support organization had any ideas.
We completely started from scratch. removed the organization vDCs, networks, and provider vDCs. When I went back to add in the Provider vDC I noticed that I couldn't choose any datastores via Storage Profiles. This is weird because I didn't have a problem the first time adding a Provider vDC with Storage Profiles. I went back into the vSphere Client and made sure that all datastores were given a datastore capability. Then I went into Storage Profiles and the Storage Profiles were still there. I didn't understand it. I got a little curious and backtracked a bit further. I went down a few rabbit holes. I even went to the point of removing a host from the 1000v, removing it from the cluster, creating a new cluster, and creating a vDS. Added it back into vCloud and everything worked as normal. I was convinced it had to be a 1000v problem Then it struck me, I know I'm licensed to do this, but did I enable it?
There was the issue. I had accidentally enabled Storage Profiles for the wrong cluster! The cluster that actually had been enabled was for the vDS test cluster.
I enabled the profile and all was well in vCD land.
After this, I was able to deploy vApps into vCD as well as add VMs to the Organization vDC Catalog backed by a Provider vDC utilizing the Nexus 1000v. Storage Profiles changed a lot of things within vCD 5.1, and be sure to Enable them for all clusters taking part in vCD. The logical explanation for this is that vCloud has a dependency on Storage Profiles. During the provisioning process of vApps into Organization vDCs, a Storage Profile must be selected and API calls are utilized to make that hapoen. If Storage Profiles aren't enabled for that cluster, then it will fail.