Using Global Load Balancing to access Multisite vCloud Director (vCD) has been possible since vCD 9.0 but only worked if the tenants using it had services in each site. If a Provider had, say five sites, but a tenant only had a presence in two of them, connecting to a site in which the tenant did not have service would result in a failed login. All this has changed in vCloud Director 10.0.

A couple of years ago, in the Architecting Multisite vCloud Director white paper I wrote about the way we allowed vCloud Director sites to be federated (we call it “associated”) with each other. Back in vCD 9.0 this didn’t do much, but we had big plans for it. How things have moved on across two years and four major releases.

The back-story

TL;DR - Just take me to the interesting new stuff!

In vCloud Director 9.0 we introduced the idea of “Multisite”, but its apparent usefulness was limited. At least on the surface, but more on that in a moment. It allowed Cloud Providers to associate their vCD instances together. This allowed a level of secured communications between different sites. With this in place, the Provider or Tenant could then associate their Organizations (Orgs) across those sites. Once this was complete, magic happened! Well, some magic. Behind the scenes lots of cool stuff was in motion, but at the GUI level, the users who logged into an associated Org (we call it a “member of the association”) would see a menu item with an icon of some “Datacenters” which allowed them to easily switch their view between each of their member sites.

Here’s what it looked like in my two-site lab. We’re currently connected to my test Org in Site-A, so the only option is to “switch to” my TestOrg in Site-B.

The hero of the piece

The clever part which enabled this was, and still is, hidden within those association steps. Associating sites allows vCD instances to securely communicate with each other at the System level. Associating Organizations allows vCD instances to securely communicate with each other at the Org / User level. We’ve used this over the versions since it was introduced to bring more and more functionality. Here’s an example Tenant Admin view from vCD9.7.

Here you can see (click the image for the larger version if this is too blurry), that the Tenant UI creates a consolidated view of the resources from a number of sites (although, just one in this lab), showing both Org VDCs (T1-OVDC and T1-K8S) and dedicated vCenters through a feature called “Central Point of Management”, or CPoM for short.

In the early Multisite releases, other than pre-authenticating between sites, neither vCD or the client GUI needed to do much heavy lifting so we didn’t anticipate a limit to the number of vCD instances which could be associated. As the GUI has grown to provide much richer multisite information, this is no longer the case. When the GUI is launched in a user’s browser, there’s a flurry of traffic across the associated sites to collect this information for display. The more sites the user has services in, the longer this takes. Unfortunately, this has led to a limit of ten associated Sites / Orgs.

One of the ways in which the Multisite capability was added to vCloud Director without breaking lots of existing stuff was in the area of user authentication. Clearly this is an important part of any web service, so the inter-site authentication relied upon the existing user authentication mechanisms already present. When your connection arrived on a vCD instance, other that an accept header which included version=29.0;multisite=global everything else was pretty much the same, which was good. v29.0 was the new multisite capable API version when we released vCD 9.0. It’s just been deprecated in vCD 10.0 which introduced API v33.0 and you should definitely check the vCloud API Programming Guide for Service Providers for the currently supported version(s) after reading this.

Now a user can authenticate at any site they connect to, and (originally switch to, but now), view content from all their locations, a Provider can offer a more resilient service. Imagine a Provider with three locations and vCD in each. Using a simple Round Robin, or more complex monitor/healthcheck based intelligent DNS service, the Provider can give their customers a single URL to connect to, rather than three separate ones. Here’s the diagram from the Multisite white paper to illustrate.

The stages from the diagram which the connection process goes through are.

1. Even though a user connects to https://portal.cloud.example.com/tenant/<org-name> the DNS lookup only cares about the Fully Qualified Domain Name (FQDN) part ‘portal.cloud.example.com’.
2. The intelligent DNS system, often called a Global Load Balancer (GLB) or Global Traffic Manager (GTM), looks up the requested FQDN in its database, checks which answers (DNS results) are valid for the request, and then uses the load balancing algorithm, specified for that FQDN, to choose the best answer.
3. This is returned to the client in the form the site specific FQDN (or actual IP address) of one of the (in this case) three possible sites.

You can read more about this in the white paper here

The villain of the story

So, where’s the catch? Well, as long as the Customer had an Organization in each of the sites, everything worked well. There were/are a few caveats around the type of account (see the white paper here for more details) but basically a user needed the same account type (local, LDAP or External IDP) in all sites, with the same username and password. The key there was the “all sites” part. If the site selection mechanism deposited a user at a site where they did not have a presence, the login would simply fail. That’s a less than stellar user experience I’m sure you’ll agree.

Could we work around the limitation? Well, yes, we could. It wasn’t simple, but there’s a generalized version of the solution described in the white paper here. You can see how it would work in the diagram (again, stolen from the WP) below.

This is similar to the sequence of events we followed in the example above. This time however, we have to get a bit more clever. The DNS system doesn’t care about the full URI path, just the FQDN, but in this case, we do. We need to terminate the user’s connection request in such a way that we can check the Org name at the end of the URI ("/tenant-1" in the diagram ). Here’s the sequence of events in this connection process.

1. When a user connects to https://portal.cloud.example.com/tenant/<org-name> the connection is passed to a ‘service’ which can examine the full path to the tenant Org.
2. The service uses the Org name to query a data source for valid vCD site locations where this tenant has service.
3. The initial connection is then returned with an HTTP 3xx status code identifying the vCD location the user should be redirected to.
4. The user’s browser makes a connection to the vCD site specified in the earlier redirect response and, as we know the Org is present in this location, the login process should complete successfully.

While this works (although this example would need to be coded into a real-world application), even at a generic level the design is, what an old colleague of mine would describe as, “sub-optimal”. For one thing, you have to have hardware in some (ideally, multiple) location to run the service. Again, ideally, you really need a Global Load Balancer of some sort to direct traffic, intelligently to one/other of those locations. You then need to bounce the connection back to DNS again to finally resolve the site connection with, ideally, some monitor or health-check to make sure the site selected is operational.

Oh, and we’d better not forget the operational overhead involved in keeping the site-selection database up to date in all the locations where the service runs!

If, as a Provider, you only offer this single URL, Load-Balanced login across a small group of vCD sites, let’s call that a region, and your customers always get service delivered with an Organization / OrgVDC in every site in that region, this works fine. Clearly, there would be a different URL for each region, but within each of those, this model could simply be repeated. Unfortunately, if a customer wants geographic separation for resilience, or more often as a BCDR requirement, this approach forces them to log into each region separately. Again, a sub-optimal user experience.

We’re fixing a problem that, in an ideal world, we wouldn’t actually have. If the vCloud Director development team cared about us, we’d just be able to throw the connection at any vCD site and, if the tenant didn’t exist there, vCD would magically sort things out! Oh, if only there was a new version of vCD with this magic built it…

What’s new in vCD 10.0

My friend Daniel Paluszek wrote a series of great posts here covering many of the new features in the 10.0 release of vCloud Director, and there were a lot of them! However, one small addition made it into the new release without any fanfare either in Daniel’s posts or the product release notes.

Over the last few releases, as the multisite functionality has grown, the Engineering team have developed a library of tools to find, get and set, information across vCD instances. Here’s the really cool part… As of version 10.0, using these, vCD will now step in to help if a user tries to connect to a vCD instance in which they do not have a Organization. Instead of a failed login, vCD will use its list of other association member sites to find one where the Org name requested by the user, does exist. As long as vCD can find a site where the requested Org exists, the connection will be redirected to that site. As if by magic!

Behind the scenes, if a connection arrives for an Organization <org-name> which isn’t present in that site, vCD will, in parallel, make a call to each of the other associated sites using the vCD /query API with a parameter of type=organization&name=<org-name> to see if that Org exists on another site. If it gets a positive response back, vCD will first check that the URL it’s going to redirect to is valid, before sending the user over to that site. We’re fans of Darwin around here, so whichever site responds first gets the prize! If we get other, slower responses, too bad, you snooze, you lose. Although, to be fair, the fastest response is probably the most able site to handle the request at that point in time.

Here’s what happens when a connection lands on a vCloud Director 10.0 site which is associated with other members.

As you can see, we have most bases covered, but will only forward the connection to a valid alternate site.

There is a configuration element which can be used to control this behavior using the manage-config sub-command of the vCD cell-management-tool (CMT). The element is com.vmware.multisite.ui.redirect which is a boolean field and defaults to true. Setting it to false disables this behavior and may be useful if a Provider has implemented their own alternative mechanism, or simply wants to stick to direct, per-site, URLs.

So what does this mean?

What seems an eternity ago, (see Global Access High Level Overview and The villain of the story if you missed it earlier) we looked at a simple Global Load Balancing model which took a single URI and let users connect through that to any one of the associated vCD sites beneath it. It looked ideal. We had DNS based load balancing, it’s well understood, resilient and doesn’t need any special client-side support. Except, then we saw how it had a flaw. If the user attempted to log in to a vCD site where they didn’t have an Organization, the login would fail, even if they had a presence at another associated site. In earlier versions of vCD, this led to all sorts of complex malarkey to protect the user experience.

Now, of course, with vCD 10.0 that’s no longer the case. Thanks to the caring folks in the vCD development team, Providers can use the simple Global Load Balancing model, knowing that if a user is “balanced” to a site which doesn’t host one of their Orgs, vCD will spin around in a nearby phone-box and step out to save the day. If you like graphical explanations, here’s a picture to illustrate the heroism at work.

You probably figured all this out by now, but for completeness, here’s the sequence of events depicted above.

1. The user tries to connect to their Org using the Provider’s “single” portal URI.
2. Their DNS client strips the /tenant/<tenant-1> part off and requests a lookup for the FQDN only.
3. The Provider’s GLB solution returns one of the three sites using it’s load balancing algorithm.
4. The user is directed to Site-B which happens to be a site which doesn’t host either of the user’s Organizations / OrgVDCs
5. Site-B queries sites [A] and [C] to see if either hosts <tenant-1>.
6. Site-A is quickest to respond with a positive answer.
7. Site-B redirects the user to Site-A where their Org does exist.

Hopefully, if you made it this far (firstly thank you!), you now understand how vCD 10.0 simplifies resilient access to a multisite association, even if customers don’t have Orgs at all the sites. If you have questions or comments about this post, please feel free to leave a comment below. If you have a specific question about your own vCD solution, or that of your Provider, please get in touch via the contact page or via your own local VMware representative and we’ll do our best to help.

Further reading

Closing Credits

Special thanks to (Abdullah)^2 whose original question on Slack prompted this post, and my (caring) colleagues in the vCD development team for help in keeping my mistakes in this post to an absolute minimum!

Feel free to share this post...

See also

• Cross OrgVDC Networking in vCD 9.5