Integrating DNS between Oracle Cloud and Oracle Cloud VMware Solution.

If in doubt, you blame the Firewall. But, if you know your stuff, and particularly if you manage a firewall, you’ll know that the real culprit is always DNS… Yes, DNS is often the villain of the piece. Seeming to be innocently minding its own business, until it does that thing it does when it doesn’t do what you think it should! Something you typically discover after hours of exasperated hair-pulling and teeth gnashing. In the interests of restoring some balance, here’s a short post exploring how we can get the best out of DNS within an OCI/OCVS environment.

The inevitable NT.B back-story…

TL;DR - Just take me to the how-to!

Recently, a colleague and I were deploying VMware Horizon into an OCVS SDDC ready for some testing. We wanted to connect some of the components within the SDDC to the SDDC’s vCenter. We have our own domain within the environment, and the Active Directory Domain Controller inside the OCVS SDDC was playing the role of DNS server. As we deploy native OCI components, Oracle Cloud creates DNS records for them, and provides a “DNS server” which will answer queries for those records.

It would be pretty useful if we could somehow connect those two DNS worlds together, to avoid having to create duplicate records in the/each other DNS for everything. Something which would look, in a cool infographic, kind of, like this.

Let’s take a look into how DNS works in OCI, then see if we can’t hack something together with damp string and zip-ties to connect that to our SDDC DNS.

DNS configuration on OCI devices on older VCNs

If we take a quick look at a Linux VM in OCI, and check its DNS server (which in Linux is usually found in the /etc/resolv.conf file), we see this.

Asking the VM to list its resolv.conf file, we see some notes about how/where the actual address is set from, and then, finally the IP address of the VM’s DNS server. In this example it is 169.254.169.254. If we check out the DNS server which a Windows VM in OCI picked up from DHCP, we see this.

Again, we see the DNS Server (and the DHCP server) is/are 169.254.169.254. You may recognize the IP address, or at least the range it’s from, particularly if you forget to connect a device to the network, and it has to make up its own IP address. The Link-Local (sometimes called Stateless Autoconfig or Automatic Private IP Allocation - APIPA) address shouldn’t ever be routed anywhere, hence Link-Local. So that’s odd.. It gives us a clue to the fact that not all is as it seems with this DNS/DHCP server.

Network devices are not supposed to route traffic for the 169.254/16 range (see rfc3927), but as the networks in OCI are all virtualized on top of Oracle’s own datacenter networks, some magical bending of the rules is used to make this work. What’s good about this rule bending is that those particular IP addresses shouldn’t ever clash with something a customer is using in their own network, or at least, is routing within their own network.

DNS configuration on OCI devices in newer VCNs

While the APIPA trick works, it isn’t really within the spirit of “link-local”, so newer OCI subnets and VLANs have DNS forwarders hidden inside their gateway interfaces. Let’s take a look at a vCenter in one of our newer lab environments to see this in practice. Here’s the /etc/resolv.conf entry from the VCSA shell.

As you can see, this vCenter is using 10.76.1.17 (and its own localhost loopback address) as its DNS server (not the 169.254.169.254 address we saw earlier). If we take a look at its routing table using the shell’s ip route command…

…we can see that the address we are using for DNS resolution is actually the next-hop gateway on our local IP subnet. There’s clearly more OCI networking magic going on here!

This is much neater as the addresses the customer chooses for their VCN really shouldn’t clash with on-prem addresses (), but does mean that each device needs to be have its DNS set based upon its default gateway as it is deployed or moved.

The main downside of this for our SDDC workloads is that inside the NSX overlay, they don’t live on OCI networks and so can’t reach either version of the magical OCI DNS resolver. But, all is not lost.

OCI DNS to the rescue

Hidden away in the OCI DNS world (which you can read about in the official docs here) are a couple of tools to help. The problem is two-fold. Firstly, if we want our SDDC devices to be able to resolve hosts in the OCI VCN we need a way to forward queries to the OCI DNS “server”. Secondly, and, conversely, if we want devices in native Oracle Cloud using their OCI provided DNS to be able resolve hosts in the SDDC AD/DNS world, we need a way to forward queries to our DC/DNS servers. While the native DNS service is somewhat obscured behind the scenes, OCI has a way to push it through the curtain and into the spotlight in the form of DNS “Endpoints” which the Console describes as;

“The private endpoints used for forwarding and listening to DNS queries to or from another private DNS system such as a peered VCN or an on-premises network.”

The two endpoints types are configured in a very similar way, as we’ll see shortly, but in brief…

• The Listener, as you’ve probably guessed, listens for queries from our SDDC DNS for queries about hosts inside the OCI address space of our VCN.
• The Forwarder allows OCI DNS to forward queries, which match specific rules, to another DNS server somewhere. In our example, that would be our AD/DNS servers inside the SDDC.

In picture form, it looks like this;

As we can’t use those weird IP addresses which the OCI hosts do, we need another way to reach these endpoints, and we do this by giving each an interface (in the form of an IP address) on a subnet in our VCN. In our example, we’ll create a dedicated “Network Services Subnet” but in practice you could use an existing subnet if you want, as long as our preparatory steps below are, or have already been, completed.

Configuring the Listener and Forwarder

Enough, jibber-jabber, let’s get on and set things up! First we’ll give the new Endpoints a home by preparing the Network Services Subnet, then we’ll deploy and configure them.

Preparation

This post isn’t an introduction to OCI, so we’ll skip through this stage quite quickly, but in summary, we’ll;

1. Create a new OCI Subnet for which we’ll need a name and a suitably sized IP address subnet from within (one of) the VNC IPv4 CIDR block(s).
2. Attach a suitable Route Table with Route Rules to the new Subnet.
3. Attach a suitable Security List with Security Rules to the new Subnet.

In practice, we have to create things in reverse order, as we have to have the Route Table and Security List before we can create the Subnet (unless we’re using existing ones, or select existing ones until we create/swap our new ones). In this example, we’ll create new so that we can see the specific details we need to add more easily.

Creating the Route Table

We can use an existing Route Table or create a new one specifically for our new Subnet. What is important about the table is that it should have the ability to reach the sources of queries to our Listener, and the destinations of queries from our Forwarder. Typically, this means a Route Rule to our SDDC overlay network addresses, and, if not learned automatically from something like FastConnect, a Route Rule towards any on-net DNS server we might need to talk to.

Here’s a shared Route Table which would work fine for our Network Services Subnet. If you click on it, you’ll get a larger, hopefully less-blurry, version in a new window.

From the bottom upwards, the rules in this simple example route traffic back to the SDDC via a Private IP route target, our On-Prem networks via a DRG route target, and everything else to the Internet via a NAT-Gateway target.

Creating the Security List

While sharing a Route Table might make sense, we should probably have a dedicated Security List for our Network Services. Visiting the Security List page in our VCN and selecting the blue “Create Security List” button presents us with this:

How granular you need to be in configuring your Ingress/Egress rules will depend upon your own Security Policies for things like this. DNS queries typically target UDP port 53, but will sometimes switch to TCP port 53. To be safe, you should allow appropriate sources Ingress permission to a Destination port range of 53 in both TCP and UDP “IP Protocol”, and, appropriate destinations Egress permission, again, to a Destination port range of 53 in both TCP and UDP “IP Protocol”. If you add other network services to this subnet, you will of course need to modify these rules, but they should get us up and running in our DNS endeavors!

Creating the Subnet

Heading to the Subnets list in our VCN and clicking the blue “Create Subnet” button allows us to define our Network Services Subnet, attach a suitable Route Table and our shiny new, bespoke, Security List so that we end up with something like this.

The next few steps require that we locate the VCN DNS Resolver which isn’t a straightforward as you might guess. If you rushed off to the Networking -> DNS Management section of the Console you will find details of the Private Zone created for your VCN, but you won’t find the stuff we’re looking for. To find that, go to the “details” page for your VCN and in the VCN Information pane, up on the right, you’ll find a secret link to the DNS Resolver, like this… (again, click to enlarge the image)

This takes us to the VCN’s Private Resolver details screen…

As we’re going to create Endpoints, we can select that link from the Resources list on the lower left of the page. Creating the two endpoints is very similar. For each, we need to tell OCI which subnet we want the endpoint creating on, which direction it should operate on, and if we want, the IP address we want to allocate. If we ignore the last bit, OCI will allocate one for us.

The Listener

Firstly we’ll create the Listener Endpoint. We click the blue “Create Endpoint” button and fill in the form. We’ll give the Endpoint a helpfully descriptive name, choose the Subnet we want the Endpoint connecting to, select “Listening” for the Type and give the Endpoint an IP address from the Subnet’s IP addresses.

The Forwarder

In a Rinse and repeat moment, we’ll do essentially the same for the Forwarder except we’ll select “Forwarding” for the Type and choose a different address to the one we used for the Listener…

Completing those two steps we can return to the Resolver details screen where we can see our two endpoints listed.

Don’t cross the streams

The final pieces of the puzzle are to implement the two “Queries” arrows from that diagram we saw earlier. We want the OCI Resolver to query our SDDC DNS for its authoritative domains, and vice-versa. As we’re in the OCI console, in almost the right place, we’ll look at that first.

Forwarding queries from the VCN Resolver

Switching to the Rules view in the Resources list, we see that there aren’t any forwarding rules configured yet. We can fix that with the blue “Manage Rules” button.

Once in the “Manage Rules” dialogue, we can add up to ten rules. Each forwarding rule is comprised of a number of attributes as we see in the examples below.

Let’s look at each attribute in turn.

• Order - Rule matching order, from top (1) downwards. Use the arrows on the right to move a rule up or down.
• Condition - There are three matching Conditions to choose from.
  • “Domains” - Used to forward domain queries to the remote resolver. This field supports multiple entries so you have to press [Enter] after each domain to add them to the list.
  • “CIDR Blocks” - Used to forward IP PTR queries to the remote resolver. This field also supports multiple entries, so remember to hit [Enter] after each subnet you specify!
  • “None” - Catch-all. This condition will forward everything which gets to this point in the list to the specified remote resolver.
• Domains/CIDR Blocks - The next field is a list of strings containing either domain names or CIDR blocks to match depending upon the condition we chose. In the example above, we’re forwarding queries about sddc-domain.local and reverse lookups for hosts in 10.76.120.0/24 to our remote resolver.
• Action - At the moment, we can only specify Forwarding rules here so this field doesn’t help much.
• Source Endpoint - Here we can select from any Forwarding Endpoints we created. In our example one to choose from, but in a more complex solution, we may have multiple Endpoints in different Subnets. Firewall filtering rules will see queries originate from the IP address allocated to the forwarding endpoint we specify here.
• Destination IP Address - Per-rule we can specify the remote resolver to forward to. We may have some rules sending queries to the SDDC resolver and others forwarding domain and IP lookup queries to our on-prem DNS servers. Under the hood, this field is also a “List”, but, at the time of writing, is limited to a single remote resolver. Being able to specify multiple resolvers to try would offer greater resilience. If you’re reading this from the future, you may be able to add more than one.
• Buttons - At the right hand end we have the up and down arrows to re-order the rules, and an ominous red “-” button whose role I’m sure we can all guess.

Forwarding queries to the VCN Resolver

We don’t need to do anything much here in the OCI console to receive queries for the VCN Resolver. The IP address we selected for the Listening Endpoint will be the target used in the forwarding rules on our SDDC or on-prem DNS servers, and the Security List rules, together with any other filtering rules in the NSX firewalls in the SDDC, or between the Listener and a remote, on-prem, DNS Server will play their part of course.

The icon in the Private Resolver Information panel next to the Default Private View: link tells us:

So the VCN Resolver will check any custom private views, then the default OCI provisioned records, then it will follow the Forwarding rules, then, crucially, public Internet DNS resolvers. This means that while we can use a full featured DNS server inside the SDDC (such as the DNS role on Windows Server) and conditionally forward queries to the VCN listener, we can also just configure hosts in the SDDC to use the Listener as their “DNS Server”.

Which is the right approach will depend upon a number of factors. Hosts which only need public Internet resolution might be fine just using the VCN Listener. Hosts which need to resolve a small number of bespoke private records might be fine with those records in a custom Private View in the OCI resolver. If you have a large number of Active Directory clients in the SDDC, which have forward and PTR records created automatically as part of the AD DHCP/DNS processes, they would be best using the AD DNS server. That AD DNS server itself might be best configured to use the VCN Listener as its upstream resolver. Doing that will ensure clients have access to AD, OCI and public Internet name resolution, without the need for conditional forwarding of specific domains or CIDR ranges.

As always though, OCI and OCVS provide the customer with choice, and don’t enforce a single course of action or configuration model onto us. This is very powerful, but, to quote somebody quoting Spiderman, with great power comes great responsibility. If we don’t want those Firewall folks laughing at us because it was the DNS at fault, we should probably take great care with the design of our DNS model, whichever model we choose.

Conclusions

In the last quarter of an hour or so, we learned that OCI’s DNS can be reached from our SDDC workloads and our on-prem DNS servers. We learned that OCI’s DNS can also use our SDDC or on-prem DNS servers to resolve queries for the records they know about. We saw the steps to set up this two-way cooperation and for once in an NT.B post, there was absolutely no mention of NAT! (until just then aaaargh!).

As always, if you made it this far, thank you! And, if you have any questions about the content of this post, or about Oracle Cloud VMware Solution, please drop them in the comments below.

Feel free to share this post...

See also

• OCVS - It came from the Internet...
• OCVS - To the Internet and beyond...
• OCVS Networking Reference Architecture Poster
• Creating VLAN backed Port Groups in Oracle Cloud VMware Solution