MULTI-TENANT WAN ACCESS on NotThe.Blog

Multi-Tenant WAN Access to Shared Resources: Part 1 - The Problem

rss@notthe.blog (Steve Dockar) — Tue, 19 Nov 2019 09:01:00 +0000

We have a thing on the Internet, lets connect it to our WAN customers! What could possibly go wrong!

A number of Cloud Services require the transfer of large volumes of data. In the Cloud Provider world, that could be uploading VMs in the form of OVA files, ISO disk images, or sending backup data to a DRaaS service. Customers can connect their on-prem SDDCs to the Cloud Provider with only an Internet connection, and make use of these great services. How easy is that! In locations where the Internet is readily available, fast and reliable, this is great. But what about locations where that’s not the case? Well, that’s where the trusty WAN steps in. Using a Communication Provider’s services, customers can get direct network links to their Cloud Provider’s datacenters which, while more costly, may offer the speed and reliability which local Internet services lack.

Excellent, problem solved! We’ll just connect our services to our customers’ WANs and go back to watching Netflix, right? Hang on, surely it can’t be that simple? Of course not, for one thing, if the Internet is poor, how will we watch Netflix????

In this post, we’ll explore the nature of the problem in connecting a service designed to face the Internet, to the Wide Area Networks of multiple customers. In the second post in the series we’ll explore a possible solution. Before we start, let’s clear up a couple of points. First, I used “WAN” in a fairly general way. For the purposes of this post it doesn’t matter if the WAN is a L3 MPLS/IP-VPN, L2 Ethernet Private Line or even a legacy Frame Relay or X.25 network. Actually, a Frame Relay or X.25 network may present their own challenges, but, I digress. Secondly, by direct I meant that a customer’s on-prem network is extended into a Cloud Provider’s facility in such a way that the Provider’s DC looks like a site on the customer’s WAN. The fact that short of dark fiber/fibre between locations, the network links are unlikely to actually be direct isn’t the point in this context.

Okay, pedantic explanations aside on with the post…

If you don’t need all the background to the issue, you can jump straight past the tl;dr intro, to the solution by clicking here.

Introduction - The good, the bad and the ugly.. complex

From a security point of view, services or portals, API gateways etc. do have a tough time on the Internet. But, what’s good about the Internet is that everybody on it has to use unique IP addresses. Okay, you can spoof a source address as part of a DoS attack, but to use the Internet you have to play by the rules and use an IP address (or addresses) that nobody else is using. If you’re reading this post I’m guessing you know, that other than a few, typically large, and typically long established companies, most organizations don’t use public Internet addresses inside their networks. Instead, they all use the same set of addresses reserved for use inside private networks and detailed in rfc1918.

As a slight aside, not all organizations do. Some, often unknowingly, use public Internet addresses which they think are private addresses. That’s another story, and one which we can’t fix in a blog post… sadly…

And there, in a nutshell is the problem. If you place your service on the Internet and get a connection from a user or another computer making an API call, it will come from a unique public address which you can then answer, and reply to, without difficulty. But, let’s say we also want to connect that service, maybe it’s some kind of portal, to a customer over their private network too. This presents a couple of problems and we’ll look at them from the perspective of the Customer and then the Service Provider.

The Customer is always right…

As the customer, I want to connect over my WAN. That means I need to connect to an IP address that works on my network, even though the service lives in my Provider’s network. If I use its public Internet address, my network will send the connection, well… over the Internet, and that kind of defeats the object. Here’s what I want. Don’t worry, I’m sure this diagram will fill up as we go…

I might be able use the same private address that the provider uses inside their network, but there is a good chance they might be using addresses that are already in use on my network so that might not always work… But let’s say the address doesn’t clash. Now I need to work with the Provider to teach my WAN that those particular addresses need to be routed to their network. We call this injecting or learning routes, and that means I’m going to need access to some Networking skills (and a good Project Manager) to get things aligned, just so. In the diagram you can see that I use 10.1.x.x, 10.2.x.x and 10.3.x.x in my network and the Provider uses 10.4.x.x. So, all I have to do is teach my network where 10.4.x.x lives and we’re in business!

But, even if I can use the provider’s real address, the Provider, as we’ll see next, might still have some challenges with that.

The Service Provider is bigger, so should win in a brawl…

As the Provider, we will have deployed the service before trying to connect it to any customers, and as Public addresses are in short supply, it’s likely that the service will have been configured with those same RFC1918 addresses that we mentioned earlier, let’s say ‘10.1.x.x’. So when we try and offer the service over a customer’s network, they just have to connect to the addresses we used.

To do this, those networking skills (and that good Project Manager) come in to play to add routes to the customer’s WAN to tell it to send traffic destined for ‘10.1.x.x’ to a WAN router in our Provider datacenter, where it breaks out into the network and heads for the service portal, or API endpoint or whatever.

The first problem comes (as we noted in the last section) if the customer is already using those same ‘10.1.x.x’ addresses somewhere in their network as you can see in the image below. We’ll look at solutions in the next post, but I guess for now it just serves us right for not using something more obscure like ‘10.113.x.x’! In simple terms, when this happens each network will route traffic to the nearest ‘10.1.x.x’ addresses as the have no way to know which ‘10.1.x.x’ addresses we meant them to route to. When that happens, the Provider’s service just seems like it’s not working, when in actual fact, the issue is that the customer is sending their requests to their ‘10.1.x.x’ addresses and not ours.

Let’s say though, that for now, we got lucky and the customer isn’t using our addresses anywhere (we’ll use the ‘10.4.x.x’ addresses from the first example), and can get to work using their networking skills (and that good PM) to route their traffic to our service. When they do connect to our service, it would only be polite for us to reply to them, right. So we just send our reply back to the address they connected from, let’s say Alice in Accounts at “Customer A”. Alice’s computer has an IP address of ‘10.100.100.1’ so we try and send our reply back to that address. But, unfortunately, the computer used by Peter in Personnel (HR doesn’t start with a “P”) in the Provider also has an address of ‘10.100.100.1’ so our network (the purple “router” in the diagram below) sends the reply we wanted to go back to Alice, to Peter instead.

So problem two is that if the customer connects to the provider from any address on their network which is in use anywhere in our Provider network, the service can’t reply so the whole thing fails. We better add that to the list of things to fix!

Again, for now, let’s pretend we don’t use Alice’s ‘10.100.100.1’ address in our Provider network, we then need to use our networking skills (and another good PM) to make sure we can route traffic from our service platform to Alice’s address, back to the “Customer A” router so it can find it’s way back to Alice. Here’s what that looks like in a picture, where our Provider network knows that 10.1.x.x. 10.2.x.x, 10.3.x.x and 10.100.x.x all live behind the blue Customer-A router in our datacenter.

So, as long as our customer doesn’t use an address we are using, it’s okay. Woohoo!

Now, if the service is successful, we’ll hopefully have more than one customer (which in the Provider world is known as “a good thing”), so let’s introduce Bill from Billing (where else!) in “Customer B”. He uses his shiny new computer to connect to the Provider’s portal to check something and (miraculously) his company doesn’t use, so can also route to, those ‘10.4.x.x’ addresses that the Provider uses. Great, obviously we want to reply to Bill (as we’re nothing if not polite) so we carefully craft our reply and send it off to his ‘10.100.100.1’ address. Hang on, that address seems familiar, right? It’s the same one as Alice was using earlier. Oh no! Things were going well until this point. Now, how will our awesome service know which ‘10.100.100.1’ address it should reply to?

Examining this problem more closely, we can extend our earlier problem two statement to “if the customer connects to the provider from any address on their network which is in use anywhere in our Provider network or, anywhere in any of our (connected) customers’ networks, the service can’t reply so the whole thing fails.

Summary

Connecting a Provider service, portal, API gateway etc. to the Internet is dangerous, but fairly straightforward. Connecting that same service to one or more Customer Wide Area Networks is not as dangerous because we know who our customers are, and, have a business relationship with them. That means they’re less likely (unless their network has been attacked or compromised ) to attack our network. But, the simple fact that most of our customers and, potentially, our own networks, all use the same bunch of IP addresses makes the picture below much more difficult to actually build than this (heavily simplified) drawing suggests.

Okay, with the problems faced by both the Customer/Tenant and the Provider highlighted, in Part 2 of this series we’ll look at the building blocks of a possible solution, and how they address each of the issues we identified in this post. See you there!

Posts in this series:

Multi-Tenant WAN Access to Shared Resources: Part 2 - A Solution

rss@notthe.blog (Steve Dockar) — Thu, 19 Dec 2019 02:00:00 +0000

We have a thing on the Internet, lets connect it to our WAN customers! It went wrong! How can we fix it?

In Part 1 of this series we looked at the problem Cloud Providers and their customers face when accessing provider services over their Wide Area Networks (WANs). In this second post in the series we’ll explore a number of possible solutions. In some instances, services were designed from the beginning to face multiple tenant networks, each with possible overlapping address space. Where this is the case, we don’t really need to “design” a solution, just ensure that our service, and the connectivity model which connects our tenant WAN environments to it, follows the way the service’s designer envisioned it.

In some cases of course, this isn’t the case, and we need to hide the complexity of those overlapping customer networks from our service, so that we don’t confuse or scare it. We’ll look at both scenarios in this post.

Before we start, in Part 1, we carefully engineered our customers’ and the Provider’s networks so the service either worked, or, only exhibited one of the possible problems. This was done to introduce the issues in manageable bite-sized chunks. If you’ve read that post, you’re now, officially, an expert, so in this post, we’re (as Emeril used to say) kickin’ it up a notch! In the examples in which we fix the various problems we identified in Part 1, we’ll take a worse case scenario where each of our tenants and the Provider networks all use exactly the same addresses. Buckle up, things are gonna get bumpy!

Don’t and say we did

Sometimes, the best way to fix a problem is not to have it in the first place. Some software/hardware services are designed to face this issue without breaking sweat. The problem we’re going to fix exists because the services in question only have one, single, network stack. That means they have no way to differentiate between multiple addressing schemas which might also overlap. Where services are designed to cope, architecturally, the model is usually something like the one in the diagram below.

For clarity, we’ll move the “service” from that big stack of servers top left, to its own space in the middle of the diagram.

In the diagram, the dark box represents the natively multi-tenant service, for simplicity let’s call it an “appliance”. On its left hand side there’s a purple “router” which connects to the Provider’s network. This is an independent Routing and Forwarding base within the appliance, just handling traffic to and from the Provider. On the other side, there’s a similar setup, but in this case, one for each customer. In between the two, and here’s where the magic happens, the application itself runs in a network space which is independent of all of the connected networks. This part of the appliance can send and receive traffic to or from all of them, without being tied to the network addressing of any of the connected networks. Pretty clever stuff!

An alternative approach is to fix the issue at Layer 2 instead of Layer 3. Instead of using the packet’s source IP address to figure out how to route a reply, devices use the physical port that the connection came in through as the sign-post to the direction they should switch the reply back through. Usually in the Provider world this would be a L2 (datalink) interface with a VLAN ID to allow multiple connections on the same physical interface. A packet with source IP of 10.1.1.1 on VLAN 101 would be returned over VLAN 101 whereas one with the same source which came in on VLAN 102 would be returned over that VLAN and so on. F5’s BigIP “Auto Last Hop” feature is an example of this (although F5 KC Article K55225090 suggests support for this, at least in VE is something you should check against the version you’re running).

The main challenge we’re working so hard to overcome is when more than one network uses the same addresses as another. In Part 1 we saw that, although a bit more dangerous, the Internet doesn’t add to this overlapping address malarkey. This means that we don’t necessarily need another “Internet router” inside the appliance, instead, we can bring in the Internet through the Provider side as we saw in Part 1. To save you flicking back to that, here’s what that would look like if we add it to the previous diagram.

Internet access and the Provider networks don’t clash, so we can deliver them both to the appliance on the same interface. In practice, this would be the Provider’s networking specifically to do with delivering these types of services to the Internet, complete with firewalls, IDS, IPS, etc. and not all of their corporate IT networks or other sensitive parts of their infrastructure.

Okay, so that’s what the problem looks like in a solution where we err… don’t actually err… have the problem. Say what? Actually, although seemingly a bit weird, that’s no bad thing, as we now have a model that we can use to build our own solution against.

Not so lucky

It’s no help knowing that some services don’t have the problem if the service we have does. Actually, more often than not, services designed for the Internet or to be deployed to a single network will need help to operate in our multi-tenant Provider world. Typically, this means we need to connect the service to a network which doesn’t suffer from this problem, and then, do the same for each of the connected customers WAN networks. Easy! Here’s a diagram representing the solution (and I bet you were expecting another looooong NT.B post right?).

Inside the black box

When we need to fix the problem ourselves, Network Address Translation (NAT) is the usual go-to tool we reach for when we need to hide one network from another. “Just use NAT!” is a phrase which you can use to identify a real network engineer. Say that to them, and if they look like they just swallowed something really unpleasant, they’re a network engineer! There’s a maxim that holds true most of the time which is “Route if you can, NAT if you must”. Sadly, in this case we must. There are many explanations of NAT on the Internet (like the one in the link above), but for our purposes, NAT takes an IP packet with a Source and Destination address and port number, and lets us swap some, or all of them, for new values, usually following some rules that ensure we can keep track of what’s going on. We can swap the source, destination or, in some cases, both, addresses and/or ports.

Let’s take a look at the flow from the first tenant, over their WAN, to the Provider’s service. Fair warned, this time, we’re going to change the addresses so that the customer is using the same addresses as the Provider. As the song says, if we can make it [work] here, we can make it [work] anywhere! Alice from Customer-A has been promoted, she’s now in err.. Another Department and has a new PC whose IP address is ‘10.1.1.1’. Over in the Provider network, the new portal is live and, on the Provider’s network its IP address is also ‘10.1.1.1’. If we were to color-code them for clarity, Alice has an indigo ‘10.1.1.1’ whereas the Provider portal is a plum ‘10.1.1.1’ .

Colors help us, but sadly IPv4 is color-blind and can’t tell them apart. That means, Alice can’t target the portal’s real address. She has to connect to an address which makes sense on her network and doesn’t clash with her or anyone else’s address on her network. We’ll need to ask the Customer A network admins to allocate some addresses which we can use to represent the Provider services in the customer network. Let’s say they give us ‘10.2.2.x’ (‘10.2.2.0/24’) and from that we choose ‘10.2.2.2’ for the portal. So now, Alice can connect to that and, (as we learned in Part 1, with the help of network engineers and a good Project Manager) we’ll then route her traffic to the Customer-A router within the Provider’s DC. As each packet flows through that router, we’ll translate the “destination” address field (DNAT) in the packet from ‘10.2.2.2’ to ‘10.1.1.1’ . We’re not finished yet, but let’s see what that looks like so far.

So far, so good! We’ve managed to route a packet from Alice to the portal, swapping colors.. I mean addresses (using ‘NAT’) as we go. We’re currently NATing on the Customer router in the Provider’s DC, as we need to translate the packets while they’re still in the context of that customer’s WAN. If we try and manage the NAT on the single plum-colored Provider router we’d have probably lost the customer context before we got around to NATing anything. However, if you read Part 1 and are now an expert, you may spot a slight flaw in our otherwise awesome plan. Currently, although we’ve made it possible for Alice to send a packet (or more than one) to the Portal, when the packet gets there and we want to try and reply to it, the Source address the packet came from is still Alice’s indigo ‘10.1.1.1’ . we’ll look to fix that in a moment, but before we do, there’s another, slightly more hidden problem in what we’ve got so far. This sneaky issue is going to try and bite us in a few places, so it’s worth beating it into submission with a stick now, before it gets out of hand…

Orders are orders

Most network devices perform a mixture of roles. Routers are optimized for routing packets and managing the protocols that allow learning of network routing topologies, but, they can also filter packets (like a firewall) and NAT them too. Firewalls may be optimized for complex filtering rules or traffic analysis, but can also route and NAT too. What’s common to all of these devices is that they will each have a specific sequence or ‘order’ in which the steps are carried out. Take our NSX-T Edges for example. Traffic flowing from outside to inside is first NAT’d, then subject to firewall rules and finally forwarded (routed) towards its next hop/destination. But, on the way back, the packet is first filtered (firewall rules), then NAT’d and finally forwarded. Different direction, different order!

Let’s see that NSX-T order of operation laid out more clearly.

[Outside] NAT Filter (Firewall) Forward (Route) [Inside]
[Outside] Forward (Route) NAT Filter (Firewall) [Inside]

Taking this as an example, if we were using one of our NSX-T T0/T1 Edges to perform the job of the tenant WAN router in the diagram, it would take the packet, translate (NAT) the destination from ‘10.2.2.2’ to ‘10.1.1.1’ then it would check its firewall rules to see if the packet is allowed (we’ll assume it is) and then try and route the packet to ‘10.1.1.1’. But, err… which ‘10.1.1.1’? It now has one on each side!

If the order of operation where route first and then NAT, we would decide the packet was heading (being routed) out towards the Provider Router (which we’re still pretending is ‘10.2.2.2’ ) first, then, once we’d decided which interface to send the packet out of, we’d swap the destination to ‘10.1.1.1’ and away it would go towards the portal. As we’re working on a worse case scenario here, we’re going with the router in our diagram NATing before it routes, and that means we can’t simply NAT to the Portal’s ‘10.1.1.1’ address because that address (without the pretty color) also exists on Alice’s PC, which the router knows is in the opposite direction to the Portal. Aaargh!

So NAT is an awesome tool, but it works best where the addresses used on either side of the NAT-ing device can’t overlap. Like when we swap an rfc1918 address for a public Internet address when we connect to the some website on the Internet. In that case, we know that when we’ve swapped our address the new one will only exist on the inside or the Internet but not both. Here, NATing between potentially overlapping rfc1918 addresses is fraught with danger!

We’ll come back to that in a moment, but first let’s pretend for now that it’s not a problem, and look at the Provider trying to reply.

The right to reply

As we’ve engineered this to be the worse case scenario, when the Provider portal tries to reply to Alice, it will send from its address of ‘10.1.1.1’ to Alice’s… oh, wait, no! It can’t send to Alice’s ‘10.1.1.1’ because colors don’t count! We need to temporarily assign Alice an alternative address which is unique across the Provider’s whole network. We also need to have the routing devices know that whatever this address is, we should route it towards the Customer-A WAN router. Let’s just use some letters for now and call Alice ‘a.b.c.1’ . It’s okay for us to give Alice a temporary address so we can talk to her, but we need her to talk to us from this address first or we won’t know it’s her. This means we need to arrange to fix her incoming packets so they appear to come from her temporary address. Looks like we’ll need more NATs! Let’s take a look at that in a picture…

In the picture above, we’ve swapped Alice’s ‘10.2.2.2’ destination to the Provider’s real ‘10.1.1.1’ address, and we’ve also swapped her real ‘10.1.1.1’ source for the Provider-allocated ‘a.b.c.1’ address. We’ve also added a route to the Provider’s network which says that ‘a.b.c.x’ can be found through the Customer-A WAN router. Now that we’ve hidden Alice’s addresses from the Portal and vice-versa, we’ll get a flow which should look like this.

Let’s get real

It works on this diagram, but, because of the order of operation challenge, it won’t work in the real world. Let’s examine why. Here’s the diagram again, but this time with some handy-dandy numbers on (as we’ve spared no expense in this post!), to help guide us. Even when things fail, we’ll ignore that and carry on, as that will give us a list of things to fix.

Here’s the flow, step by step. Remember, even when things break, we’ll carry on as if they didn’t.

Step	Action	Result
1	Alice connects to her WAN’s address for the portal - ‘10.2.2.2’.	The packet with the original addresses in is sent over the WAN.
2	The packet, still with its original source and destination addresses arrives at the Customer-A WAN router	The packet is passed to the NAT engine in the router
3	On it’s way through the router we swap (NAT) Alice’s source and the Portal’s destination addresses	The translated packet needs routing to the new 10.1.1.1 destination, but that exists on both sides of the router.
4	The translated packet is forwarded to the Provider router with it’s new source and destination addresses	The packet is routed to the portal.
5	The portal builds a reply packet using its own address (source) and the Alice’s ‘a.b.c.1’ address from the received packet (destination).	The packet is forwarded to the Customer-A WAN router because of the route we added to the Provider router
6	On it’s way back through the Customer-A router, the packet has its source swapped to the ‘10.2.2.2’ address Alice knows the Portal by, and its destination to Alice’s real ‘10.1.1.1’ address.	Like in [3] we need to route to ‘10.1.1.1’ but again, we have one on each side of the router.
7	The packet is delivered back to Alice’s ‘10.1.1.1’ address, apparently from ‘10.2.2.2’	The rest of Alice’s connection follows the same sequence

On reflection, we did pretty well and almost got it to work. Let’s look at what’s still on the glitch-list.

[3.] We can’t NAT Alice’s packet’s destination address to the real address of the Portal because that address exists on both sides of the NAT-ing router.
[6.] We can’t NAT Alice’s temporary destination address to her real address for the same reason as [3.] above.
[5.] (and [.3]) we need some sort of schema for allocating temporary addresses, potentially to lots of devices from lots of customers and to multiple Provider devices too.
(Either that or wait for IPv7 and see if it supports Hex, Alphabetic or even colored addresses!)

We have a dog, a cat, a mouse and a rowing boat

Remember here where we said that NAT was awesome as long as the same address(es) didn’t exist on more than one side of a NAT device ? Well, therein lies the secret to fixing this! A bit like the per connection routers in the black box example, we need each of the networks to be completely independent of the others. We need a sort of “no mans land” in the middle (like the lake in Harry Potter VI but without the lurking nasties), and we need to know how to cross it safely and get back. Kind of like this.

You see here, packets arriving at the lake’s shore are more worried about getting their bits wet than what’s on the other side. While they’re in a state of panic, we can NAT them to some “lake” addresses (don’t worry, I’ll get bored of the water metaphor soon). Just like the earlier case where we gave Alice a temporary (‘a.b.c.1’) address to ensure she didn’t clash with anything on the provider network, here, we’ll assign her (and anything else from the Customer-A WAN) a “Customer-A lake” address like ’la.ke.A.1’, so she doesn’t clash with anything else. We can also allocate the Provider a bunch of “lake” addresses to use for each of their components or services which customers will need to reach.

When a packet leaves its origin, say Alice’s PC, it will still need source and destination addresses which make sense on the Customer-A WAN. Let’s say Alice’s ‘10.1.1.1’ and the Portal’s ‘10.2.2.2’ . We shouldn’t need Alice or her network engineer’s (or their PM) to know about the scary lake, that nightmare lives in the Provider space but we do need to Source and Destination NAT (SNAT/DNAT) the packet somewhere, so we’ll need a Provider-side Customer-A NAT device of some sort. There, we’ll hide Alice’s address behind a “lake” address and, we’ll see which of our services she wants (oh, it’s the Portal again) and swap her ‘10.2.2.2’ destination for the portal’s “lake” address. Now the packet is safely wrapped in “lake” addresses we can forward it across to the Provider’s side of the lake.

When the packet reaches the Provider’s side, we’re going to need more NATs, and, something to do the NAT-ing. This something will take the destination (“lake”) address and swap that for the portal’s real address in the provider network ( ‘10.1.1.1’ ). We can then decide to either leave Alice’s “lake” source address and have the Provider network route all lake addresses back to the err… well lake. Or, we can also NAT Alice’s lake source to a Provider-allocated rfc1918 address. Why go through this additional, and seemingly pointless step? Well, it depends on the range of addresses we choose to fill our lake from. We’ll look at that choice shortly. In the meantime, here’s a schematic of our solution so far.

We still need a WAN “Customer Edge” (CE) router for each customer (that’s part of their WAN service) and we might be able to use that for some of the NATs, but erring on the side of caution, we’ve got a WAN <-> Lake NAT device for each customer, and one for the Provider, all managed by the provider. Here’s our Packet walk again, this time with the lake in it.

Here’s the flow again, step by step. Fingers X’d that nothing breaks this time!

Step	Action	Result
1	Alice connects to her WAN’s address for the portal - ‘10.2.2.2’.	The packet with the original addresses in is sent over the WAN.
2	The packet, still with its original source and destination addresses arrives at the Customer-A WAN router	The packet is passed to the Provider managed “Customer-A” NAT engine
3	On it’s way through, we swap (NAT) Alice’s source (to ’la.ke.a.1’) and the Portal’s destination (to ’la.ke.p.1’) addresses	The translated packet needs routing to Provider portal’s “lake” address which only exists on the other side of the lake.
4	The packet has its destination NAT’d to the real ( ‘10.1.1.1’) address of the portal	The packet is forwarded towards the portal.
5	The packet’s source and destination are unique within the Provider network	The packet routes correctly to the portal
6	The portal builds a reply packet using its own address (source) and the Alice’s ’la.ke.a.1’ address from the received packet (destination).	The packet is forwarded to the Provider NAT engine because of the route we added to the Provider router
7	On it’s way back through the Provider NAT engine, the packet has its source swapped to the portal’s ’la.ke.p.1’ address.	As in [3] Alice’s ’lake’ address now only exists on one side of the lake so the packet routes back correctly.
8	Before we send the packet back to the Customer-A WAN we need to get rid of all this ’lake’ stuff, so we NAT both source and destination addresses	Once the packet has Customer-A WAN addresses again, it can be sent to the Customer-A WAN router in the Provider DC
9	As the packet has Customer-A WAN addresses again, it can be forwarded back to Alice	With Alice’s real destination address, the packets route correctly to her site on the WAN
10	The packet is delivered back to Alice’s ‘10.1.1.1’ address, apparently from ‘10.2.2.2’	The rest of Alice’s connection follows the same sequence

So, this time we made it all the way to the end of the flow without a single ‘’ - Woohoo! If we can find some addresses to replace the stupid lake metaphor, we’ll be able to build a real solution. When we do that, we should also stress test it with Bill over at Customer-B who, as you might have guessed also has a new computer and his IP address is, of course, ‘10.1.1.1’ !

If you read that and it all sounds familiar (or if you skipped it and just want the TL;DR), that’s because it is. If Alice connected to the Provider’s portal over the Internet we’d see exactly the same process. Don’t connect two rfc1918 networks together, that’s like crossing the streams. Instead, we connect from Alice’s private network to the Internet at one end, and back to the Provider’s private network on the other. The subtle difference is that in this case, Alice simply targets the Internet address (via DNS lookup of the portal domain name of course) directly. She doesn’t have to connect to a fake internal address to force the connection over the Customer-A WAN. But, other than that, it’s a very similar model.

It’s a wrap

Go us! - We made it work. Let’s recap where we’ve got to and take a moment to let our headaches calm down before we head off to Part 3. In this post, we’ve finally managed to get a packet from a customer device (Alice’s PC) all the way to the Provider portal even though they both had the same IP address. We worked out that although NAT was part of the solution, you can’t safely NAT between two networks connected to the same device unless (as it the case with your network and the public Internet), the two networks do not have any overlapping addresses. We learned that this was in part because there’s the potential for a device to NAT first, and then route/forward, and this breaks when the new address exists somewhere on both sides.

Finally, we were subjected to that stupid lake metaphor to illustrate that NAT will work as long as we go from Customer to unique addresses first, then unique addresses to Provider (or vice-versa). In this way, isolating both Customer and Provider networks from each other (with the ’lake’ addresses) stops NAT from trying to bite us. All we need now is a practical example that we can use to do this in the real world, and to see if it will work when: No_of_Customers > 1.

Let’s keep our fingers crossed that all of that turns up in Part 3!

Posts in this series:

Multi-Tenant WAN Access to Shared Resources: Part 3 - Getting Real

rss@notthe.blog (Steve Dockar) — Thu, 19 Dec 2019 09:03:00 +0000

We have a thing on the Internet, lets connect it to our WAN customers! It went wrong! Theory is fine, but how can we really fix it?

In the first two posts of this series we looked at what could possibly go wrong when we tried to connect a service designed to face the Internet, to not one, not many, but lots of customers’ wide area networks (WANs). We learned that NAT was a great tool, but that it was only part of the solution. We also learned that to connect two networks which use the same addresses, we need an intermediate set of addresses to hide the two networks from each other. In this post, we’ll find some suitable addresses for that intermediate stage, and look at the scaling of this model for multiple customers connecting to multiple services.

Let’s just recap a couple of the points from that last paragraph in pictures, as it will set us up nicely to design a working solution. First, in the picture below, we’re using a single routing device / Gateway to both NAT packets and interconnect the various networks. This would be convenient for a Provider as they could manage all this complex networking malarkey in one place, except we learned in the last post that this won’t work. The reason was that, although NAT swaps from (potentially overlapping) customer networks to something coordinated by the Provider, on the way back, when we (un)NAT back to real addresses, that single routing device needs to send the packets back to the correct customer network which it can’t do when faced with multiple ‘10.1.1.1s’ in different customer networks! The red circle is the gateway’s view into all the connected networks and their potentially overlapping address ranges.

In order to overcome that problem, we need to make sure that, by the time we’re un-NATing the packet back to its real addresses, the routing/(un)NATing device only sees one pesky ‘10.1.1.1’. In practice, this means we have to route to a per-customer NATing device before we lose track of which customer we’re trying to get to. That means we need a solution like this, with that stupid lake from the last post replaced by some real networking which for now we’ll color orange and call intermediate network (or “lake” for short ).

In the diagram above, whichever way we’re crossing the intermediate network, we hide our real addresses behind temporary intermediate addresses (lake was way shorter to type.. just sayin’…) and make our routing decision by sending the traffic to the intermediate network address of the destination NAT device. Once we’re safely in the NAT device for the destination network, we can dump ( un-NAT) our temporary intermediate addresses and set off across the destination network as if nothing happened. This time, where the red circles cross networks, each NAT device can see only its connected network and the shared intermediate one.

Think of the network packet like Tom Cruise leaving the IMF safe house and running through a crowded eastern European city. Dashing into the (equally crowded) railway station, pausing just long enough to rip off his false rubber nose and “I’m in disguise” glasses, before nonchalantly boarding the 14:10 to Paris Gare du Nord and heading to the restaurant car to order a fine Chablis. All, while the MI(longINT) theme music starts to play. Anyhoo, I digress (and wine would make the packets all soggy), so back to the network stuff…

In that last diagram, even if multiple customers and/or the Provider use the same addresses anywhere in their networks, no single device will see more than one of them. The only other thing the NAT devices will need to connect to, is the intermediate network and its addresses. In the first post when we contrived to make the solution work (impractically) by not having any addresses overlap between the various networks, we saw how choosing ‘10.1.1.x’ would be a bad idea because everybody else was likely to choose it somewhere in their network too. In all seriousness (okay, just briefly), where do we get addresses that will work for our intermediate network and not clash with any customer network..?

To bravely choose what nobody chose before

There are three subnets reserved in our old friend rfc1918. There’s ‘192.168.0.0/16’ (or ‘192.168.x.x’) which we often find in consumer level networking devices. Then there’s the more industrial strength ‘10.0.0.0/8’ (or ‘10.x.x.x’) which we often see in enterprise and Service Provider networks. The third subnet it slightly different and not so easy to simply drop ‘x’ into. It’s ‘172.16.0.0/12’. As the slash-twelve doesn’t fall nicely on an octet boundary like slash-eight and slash-sixteen, this range is ‘172.16.x.x’, ‘172.17.x.x’ all the way up to ‘172.31.x.x’. The subnets from ‘172.32.x.x’ upwards aren’t included in the reserved range so shouldn’t be used in private networks. According to the ARIN search below, ‘172.32(.x.x)-172.63’ belongs to T-Mobile US, and if we bothered to search further, we’d find out who the rest of the range belongs to.

That said, there are corporate networks out there I’m sure, who allocated ‘172.(16-31).x.x’ to their remote locations and carried on through ‘172.32.x.x’ and beyond. We should keep this in mind whenever we plan for addresses customers might use.

We could choose some addresses from RFC1918 and hope nobody else is using them, but that’s not exactly something we want to put our name against in the project Risk Register! What we need are some IP addresses that nobody is (or at least should be) using in their network and that’s big enough to scale to allow for many, or even lots, of customers. What we really need are some special IPv4 addresses.

Checking out the IANA table in (conveniently) that link, it turns out that there are a bunch of special ranges, included in which, are a couple of likely candidates we can potentially use.

Let’s take a look at a couple from the IANA IPv4 Special Registery!

Address Block	Name	RFC	Term’n Date	SRC	DST	Fwd-able	Globally Reachable	Reserved-by-Protocol
169.254.0.0/16	Link Local	RFC3927	N/A	True	True	False	False	True
100.64.0.0/10	Shared Address Space	RFC6598	N/A	True	True	True	False	False

Let’s take a look at each in turn.

Those weird 169.254.0.0/16 addresses

‘169.254.0.0/16’, Link Local or sometimes “Automatic Private IP Allocation” (APIPA) is the range of addresses which a device will fall back to when it had neither a statically configured IP address on an interface or a reachable DHCP server from which to request one. You may have seen them in action when you’re enjoying the feeling that comes from forgetting to plug the Ethernet cable in, or not clicking “Connect”, on that new Windows virtual machine. (Just mentioning that for a friend you understand…)

The RFC defines how a device should choose its own Link Local address and what to do if another device is already using it. The things which make this suitable for our use (at least at this point in our assessment) is that those addresses are not globally reachable (just like the RFC1918 addresses), or forwardable. This means that even if they are in use in a Customer’s network somewhere, we should never see them being sent over the WAN to our interconnection.

“Forwardable = False” might be an issue though, as strictly speaking these addresses shouldn’t be forwarded across any routing device, but contained within a single physical or logical network. Arguably, we’re not forwarding them. They’re being allocated to the (translated packets of the) NAT device and then sent on to a local link to another (NAT) device on that same network. With a good legal team, we could probably get away with that in court. However, section 1.6 of the RFC states that “addresses in the 169.254/16 prefix SHOULD NOT be configured manually or by a DHCP server” Wow, SHOULD NOT. It’s not as definitive as MUST NOT, but it is pretty close. Again, we’re not manually configuring or using DHCP, we’re using NAT, so we might get away with that defense.

Back in Part 2 we left a hanging “We can then decide to either leave Alice’s “lake” source address and have the Provider network route all lake addresses back”. Looking at the diagram below, we can see that, if we stick (moderately) closely to the RFC3927, the extent of the Link Local addresses doesn’t get us out of the NAT devices on the interface opposite the intermediate networks. If we use the ‘169.254.x.x’ addresses, we can’t route them past the NAT devices.

On the customer side, this isn’t an issue, because whatever addresses we use in the intermediate network, as packets leave on the customer side, we have to swap back the customer’s real destination (so we can reach their device), and the source address back to the one from their network which they used to represent our service.

However at the Provider end, using those ‘169.254.x.x’ addresses means the “decision” above is kind of made for us. As packets leave the NAT device for the Provider network, we already have to DNAT the real address of the Portal or service in order to reach it. If we leave the ‘169.254.x.x’ source address we used to cross the intermediate network, any router in the Provider network should refuse to forward the packet. I say should, because in some cases it is possible to amend the default behavior of a Linux-based routing device to bend this rule a little.

Possible is not the same as advisable though. We should probably stay away from rule-bending and just NAT both source and destination addresses of the packet as it crosses the Provider network. This does mean that as well as coordinating the NATs in and out of the intermediate network (so which customer WAN gets which 169.254.x.x addresses), we now have to have some RFC1918 addresses on hand to temporarily assign to the customers’ packets as they cross the provider network.

We do need to make sure these temporary addresses are unique within the Provider network, but we don’t have to care about any of the customer networks as these temporary addresses will never be seen beyond the Provider network. Similarly, if the same addresses happen to be in use in a customer network, we’ll never see those beyond the NAT device at the customer side of the intermediate network, so we’re all good on that score too.

Let’s take a look at a packet trace using 169.254.x.x’ addresses. We will need an address in that range for every possible address in the Provider network that a customer will connect to. This could be discrete devices, load balanced VIPs, secondary addresses on devices or similar. We’ll grab the top end of the range for the Provider network. In the intermediate network, ‘169.254.[255].x’ will route to the Provider network NAT device, and there, we’ll assign individual NATs to each endpoint that a customer will connect to.

In this example:

We’ll make the portal’s real address ‘10.1.1.1’ and hide it behind ’169.254.255.1’ from the ‘169.254.255.x’ range on the intermediate network.
Customer A is the first customer to ride this roller coaster, so we’ll give them ‘169.254.[1].x’ from the intermediate network address pool.
Alice from Customer A is back (Yay!) and she’s still ‘10.1.1.1’. She’s connecting to what she thinks of as the Provider’s portal on ‘10.2.2.2’ within the Customer A WAN, that the network engineers (and long suffering PM) have routed to the WAN CE router in the Provider’s DC.
When Alice is crossing the intermediate network, we’ll swap her ‘10.1.1.1’ source address for ‘169.254.1.1’ . At the same time, we’ll swap the portal’s ‘10.2.2.2’ destination address for its ’169.254.255.1’ intermediate network address.
As we can’t route these temporary intermediate network addresses across the Provider network, we’ll allocate some Provider addresses to hide them. Let’s stick with the upper end, and allocate ‘10.255.x.x’ and from that ‘10.255.[1].x’ to Customer A and ‘10.255.1.1’ to Alice.
In the Provider network, we route these new ‘10.255.x.x’ addresses back to our intermediate network NAT device.
Everything starts with Alice, so she’s on the left, and the Provider is on the right.

Here’s the flow again, step by step. Fingers X’d that everything still works!

Step	Action	Result
1	Alice connects to her WAN’s address for the portal - ‘10.2.2.2’.	The packet with the original addresses in is sent over the WAN.
2	The packet, still with its original source and destination addresses arrives at the Customer-A WAN router	The packet is passed to the Provider managed “Customer-A” NAT engine
3	On it’s way through, we swap (NAT) Alice’s source (to ‘169.254.1.1’) and the Portal’s destination (to ‘169.254.255.1’) intermediate addresses	The translated packet needs routing to the Provider portal’s intermediate address which only exists on the other side of the intermediate network.
4	The packet has its destination NAT’d to the real ( ‘10.1.1.1’) address of the portal and it’s source NAT’d to Alice’s temporary Provider network address (‘10.255.1.1’).	The packet is forwarded towards the portal.
5	The packet’s source and destination are unique within the Provider network	The packet routes correctly to the portal
6	The portal builds a reply packet using its own address (source) and the Alice’s ‘10.254.1.1’ temporary Provider network address from the received packet (destination).	The packet is forwarded to the Provider NAT engine because of the route we added to the Provider router
7	On it’s way back through the Provider NAT engine, the packet has its source swapped to the portal’s ‘169.254.255.1’ and destination to Alice’s ‘169.254.1.1’ intermediate addresses.	As in [3] Alice’s ‘169.254.1.x’ address now only exists on one side of the intermediate network so the packet routes back correctly.
8	Before we send the packet back to the Customer-A WAN we need to get rid of all this ‘~~lake~~ intermediate network’ stuff, so we NAT both source and destination addresses	Once the packet has Customer-A WAN addresses again, it can be sent to the Customer-A WAN router in the Provider DC
9	As the packet has Customer-A WAN addresses again, it can be forwarded back to Alice	With Alice’s real destination address, the packets route correctly to her site on the WAN
10	The packet is delivered back to Alice’s ‘10.1.1.1’ address, apparently from ‘10.2.2.2’	The rest of Alice’s connection follows the same sequence

Woohoo! - Wow, that was an edge of the seat ride for sure!

Okay, so that worked, but there were a couple of sub-optimal elements in there. First, although we didn’t exactly break any rules, we’re not really using those addresses as they were intended in the RFC document. Secondly, we couldn’t route those intermediate addresses across the Provider network, so we had to do yet more NATs. If you remember back to Part 1 all those coffees (or stronger beverages) ago, we said something like “and NAT if you must”. We still must here, but if NATing isn’t great, NATing multiple times is probably worse right? So, having used these RFC3927 addresses as a learning exercise, let’s move on and try find a better solution.

What the heck is a 100.64.0.0 addresses and just how big is a /10

What would be really cool would be if somebody had realized that with the advent of mass IPv4 adoption, there would be times when different networks would crash into each other, and worked out a way to make it less painful. Enter RFC6598 stage-left (or is it just Tom Cruise with another disguise?). Shared Address Space or, “Carrier Grade NAT” (CGN) as it’s often known, is a reservation of IPv4 addresses put aside for Carriers (or Providers in our case) to slip in between large networks whose native addresses can’t be used directly. Sounds familiar right? Well, once again, we’re not quite sticking to the letter of the ~~law~~ RFC here. There’s a note in the Introduction of RFC6598 which says:

In particular, Shared Address Space can only be used in Service Provider networks or on routing equipment that is able to do address translation across router interfaces when the addresses are identical on two different interfaces.

Now we might have that (see Don’t and say we did in Part 2), but in these examples, we’re working on the basis that we can’t do all our NATing in one place because of that order of operation gotcha that we found. However, our issue is still pretty much the same, so we’ll roll with it for now, work on the assumption that that note was a capability constraint rather than a functional requirement and hope nobody notices…

Okay, so what are the advantages of using these addresses instead of the Link Local ones in the last section? Well, for one, there are lots of them, err.. addresses that is. To answer the “just how big is a /10” question, the range goes from ‘100.64.x.x’ right up to ‘100.127.x.x’! Whichever way you slice and dice that, it’s a lot of addresses. We’ll look at allocating chunks of address space in more detail soon, but for now, you could allocate an entire /16 to over 60 different sites, which would give space for more than 60,000 individual Alices, Bills and Peters, in each site!

The second advantage is that, as per the table above, although not “Globally Reachable” these addresses are “Forwardable”. That makes them more like RFC1918 addresses than those ‘169.254’ addresses, in that the Provider can route them around their network, but they can’t be used over the Internet. Or, at least we’d hope, coming in from a customer’s WAN. That means that where, in the last example, we had to take customer traffic, NAT it to Intermediate Network addresses, then NAT it to Provider temporary addresses (because Link Local addresses shouldn’t be forwarded), using these ‘100.64’ addresses means we can skip that last NAT. If we sed s/169.254/100.64/ that last packet flow example, we’ll end up with this.

We’ll make the portal’s real address ‘10.1.1.1’ and hide it behind ’100.64.255.1’ from the ‘100.64.255.x’ range on the intermediate network.
Customer A is the first customer to ride this roller coaster, so we’ll give them ‘100.64.[1].x’ from the intermediate network address pool.
Alice from Customer A is still here, and she’s still ‘10.1.1.1’. She’s (still valiantly) connecting to what she still thinks of as the Provider’s portal on ‘10.2.2.2’ within the Customer A WAN, that the network engineers (and long suffering PM) had long ago routed to the WAN CE router in the Provider’s DC.
When Alice is crossing the intermediate network, we’ll swap her ‘10.1.1.1’ source address for ‘100.64.1.1’ . At the same time, we’ll swap the portal’s ‘10.2.2.2’ destination address for its ’100.64.255.1’ intermediate network address.
~~As we can’t route temporary intermediate network addresses across the Provider network, we’ll allocate some Provider addresses to hide them.~~ We don’t need to do this as we can forward CGN addresses.
In the Provider network, we route these ‘100.64.x.x/10’ addresses back to our intermediate network NAT device.
Everything, as usual starts with Alice, so she’s still on the left, and the Provider is still on the right.

Okay, let’s take a look at the new packet walk…

Here’s the flow again, step by step, hopefully for the last time! Fingers X’d that everything still, still works!

Step	Action	Result
1	Alice connects to her WAN’s address for the portal - ‘10.2.2.2’.	The packet with the original addresses in is sent over the WAN.
2	The packet, still with its original source and destination addresses arrives at the Customer-A WAN router	The packet is passed to the Provider managed “Customer-A” NAT engine
3	On it’s way through, we swap (NAT) Alice’s source (to ‘100.64.1.1’) and the Portal’s destination (to ‘100.64.255.1’) intermediate addresses	The translated packet needs routing to the Provider portal’s intermediate address which only exists on the other side of the intermediate network.
4	The packet has its destination NAT’d to the real ( ‘10.1.1.1’) address of the portal ~~and it’s source NAT’d to Alice’s temporary Provider network address~~. (Oh no it doesn’t!)	The packet is forwarded towards the portal.
5	The packet’s source and destination are unique within the Provider network	The packet routes correctly to the portal
6	The portal builds a reply packet using its own address (source) and the Alice’s ‘100.64.1.1’ intermediate network address from the received packet (destination).	The packet is forwarded to the Provider NAT engine because of the route we added to the Provider router
7	On it’s way back through the Provider NAT engine, the packet has its source swapped to the portal’s ‘100.64.255.1’ intermediate address, but its destination is already/still Alice’s ‘100.64.1.1’ intermediate address.	As in [3] Alice’s ‘100.64.1.x’ address now only exists on one side of the intermediate network so the packet routes back correctly.
8	Before we send the packet back to the Customer-A WAN we need to get rid of all this ‘intermediate network’ stuff, so we NAT both source and destination addresses	Once the packet has Customer-A WAN addresses again, it can be sent to the Customer-A WAN router in the Provider DC
9	As the packet has Customer-A WAN addresses again, it can be forwarded back to Alice	With Alice’s real destination address, the packets route correctly to her site on the WAN
10	The packet is delivered back to Alice’s ‘10.1.1.1’ address, apparently from ‘10.2.2.2’	The rest of Alice’s connection follows the same sequence

Woohoo! - Wow, err.. actually, I’m not sure that’s quite as exciting twenty-seventh time around. But at least it still worked!

Industrialize and scale

Both the Link Local and Carrier Grade NAT (Shared Address Space) methods seem to work. The CGN one is slightly slicker as we don’t need to allocate, coordinate and configure the second set of NAT addresses but hopefully it’s clear how they both work in essentially the same way. In the Provider world, half the battle is getting something to work, the other half is getting it to work repeatedly. The third half of the battle[sic] is getting it to bill correctly, but that’s a (half a) battle for another day. Repeatedly involves things like hardware and software scaling, creating a configuration model that lets us easily create identifiers like names and IP addresses that take the “design” out of adding extra customers. Things like:

Device names will be [device-type]-<pod-no>-<device-no> so each time we add a new one we just bump <device-no> by one.
Each tenant will be allocated a <some-size> range of IP addresses for their [whatever] network connection from this [pre-allocated/reserved] range/pool, so each time we do this, we get the next <some-size> block from the pool.
Each tenant will be allocated <some-number> of VLANs for L2 connectivity from this [pre-allocate/reserved] range, so again, each time we onboard a new tenant, we know what their VLANs will be.

If you’re really dedicated/lazy you can create a Slack bot (or a spreadsheet) that will take a tenant “number”, and use it to calculate all those per-tenant configuration elements. Or, take a tenant number as an input to a vRealize Orchestrator workflow that will do the hard work for you. Anyway, more digressing. Let’s try and scale that last example and see if it still works.

Remember, now we’re experts, every customer uses the same IP addresses, and we want to use the same model over and over again (because lazy err… repeatable, is good)! Okay, without the aid of a safety net (and slightly regretting using orange for the intermediate network as it’s very bright ), here’s the multi-tenant version of the packet flow(s).

Although we show per-tenant flows on the right hand side of the intermediate network, there’s only a single Provider NAT device (stretched here to show it affects all customers). We can get away with just having one because it’s only responsible for NATing between the Provider’s real and intermediate addresses. It doesn’t need to do anything with Customer/Tenant addresses as they get taken care of on the per-Tenant NAT devices on the other side of the lake… err intermediate network.

Hopefully this makes sense if you slogged through all the earlier examples. Let’s look at what we had to change to make this multi-tenant ready, and what we managed to leave alone.

We’ve allocated each customer a /24 of CGN NAT addresses. As before, Customer A (Tenant ID: 1) gets ‘100.64.[1].x’ or, ‘100.64.1.0/24 if you prefer. Depending upon the number of connections from a given customer network and the specifics of the NAT (or PAT) device this may be way overkill. But, operationally, knowing which customer a connection is from just by looking at one octet in the source IP address can be really useful.
Customer B (Tenant ID: 2) gets ‘100.64.[2].x’ or ‘100.64.2.0/24. The last octet (the “dot 1”) would be allocated pseudo-randomly depending on the configuration of the NAT device and the number of other connections from Customer B, but we’ve cheated a little for the sake of clarity here, so we see Bill (from Customer B) as ‘100.64.2.1’.
At the bottom of the diagram we see… Oh, somebody new. Let’s call him Nigel from Tenant ’ninety-nine’ (You can go with Xavier from Customer ‘xx’ if you prefer). Just to show how we scale, Nigel’s WAN is allocated, you’ve guessed it: ‘100.64.[99].0/24’. Again, for clarity we’ve cheated and given Nigel the “dot 1” so he becomes ‘100.64.99.1’.

And, that’s about it. As our successive customers get allocated higher and higher ID numbers, we still go through the same process for each one. The portal or service sees all its WAN connections from neatly managed ‘100.64.some.thing’ addresses, and simply routes all of that range back to the Intermediate network. That third octet goes all the way from ‘1’ (well, zero really, but if you have to use that, make sure any “algorithm” you create for naming or numbering stuff doesn’t break in the presence of a zero value) to ‘254’, because we used ‘100.64.255.0/24’ for the Provider’s devices and services. This might be enough to reach the design limit/maximum in which case, Go Us! But it might not be if we have bigger plans (Muhahahaha!).

Divide and Conquer

We can easily scale to higher numbers of customers/tenants if we have to meet that requirement. As we saw earlier, there are a lot of addresses in the CGN range. We could treat each ‘site’ as an isolated island (oh no, more lake analogies) and never route the CGN addresses out of a given location (on our Provider internal WAN). If that is the case, we can happily use the same address model with exactly the same addresses in every site. That does make it kind of awkward if anyone changes their mind and decides to route between customers and portals in different sites.

A safer model (as far as the Architect’s ability to say “I thought you’d want to do that, so I designed it in already”) would be to plan to (be able to) route between sites even if you don’t think you need to, at least not to begin with. Depending upon how many connections you expect per tenant, how many addresses you want to reserve for each tenant, and how many tenants you want to scale to, will tell you how many addresses you need per “site”. How many sites you want to design for will then give you the building blocks of a formula to carve up your chosen intermediate address space. Using the same address space, but distributed across Provider sites allows us to route between them if we want to.

Let’s take a look at how we could slice’n’dice the CGN address space. Here’s the range again, with some extra details for clarity.

We could, for example, decide that one /16 (entire second octet) is big enough for a site, and that the 60-ish /16s in the ‘100.64.0.0/10’ subnet is also enough for the number of sites we’ll need. Then, our site allocation model would be something like:

Site number: 1 to 64 [site]
Customer number: 1 to 254 [ID] (per site)
Let’s take Site number and bump it by 63 for the >=64, 2nd octet range
Provider Subnet: 100.(63+[site]).255.0/24 (per site)
Customer Subnet: 100.(63+[site]).[ID].0/24 (per customer, per site)

At the other end of the scale, we could decide that each customer only needs a single IP address and that Port Address Translation (AKA Dynamic PAT or Overload / Hide NAT depending on your NAT device vendor) will do the job. If that’s the case, the formula above looks like this. The single intermediate network address could be the one assigned to the interface of the NAT device or it could be a separate address used just for NAT.

Site number: 1 to 64 [site]
Customer number: 1 to 254 [ID] (per site)
Again we need to bump the Site number by 63
Provider Subnet: 100.(63+[site]).255.0/24 (per site)
Customer Subnet: 100.(63+[site]).[0,1,2 etc.][ID]/24 (per customer, per site)

The likelihood is, that a /24 for each customer is overkill. Similarly, a single address will probably work, but might, on a bad day, lead to ‘port exhaustion’. We will probably end up with something in between the two extremes. Some network devices make it easy to specify a pool or range of addresses for things like NAT, so we could allocate an arbitrary number of addresses to each customer. Some prefer subnet sized blocks, so we can only assign binary powers so 1,2,4,18,16 etc. NAT addresses per customer. The device doing the NATing will determine what we have to work with here.

Knowing how many individual addresses we need for each customer will then define the minimum size we should allocate, but we may choose to allocate more for operational simplicity. Exactly how this should be done is something for the detailed design stage of a project.

Name names for goodness sake

All throughout this looooong post there have been references to “NAT device”, “NATing device”, “Provider NATing device”, “per-Customer/Tenant NAT device” and similar, but what the heck are these really? Are they firewalls? Are they routers? Load balancers? What? Well, the answer is “it depends”.

Yes, sorry, I know…

The thing is, it does depend. There are solutions doing this kind of thing using big switches to create a VRF per tenant and then handling the NAT there. That’s great if you have the licence for the feature and don’t have to pay per tenant for a separate context or virtual appliance licence. You could do this with a Linux VM (per tenant/provider of course) using the native kernel routing capabilities or a ready-made router virtual appliance. You could use NSX-V Edges or any one of a myriad network devices that allow some sort of virtual routing and forwarding separation or similar.

Hopefully though, the models explored in this series of posts will help whatever network device you have at your disposal. Often. as with many “which is best” technology-based questions, the answer is “the one with which you’re most familiar”.

Summary

To pass as a network engineer remember, Route if you can, NAT if you must!
NAT is not safe to use if the same addresses might appear on either side of the NATing device.
As per RFC6598, some NATing devices can cope if they see the same addresses on either side (but it’s best not to rely on that unless you know for sure yours can).
Filter, NAT, Route or NAT, Filter Route? The order of operation within a network device can bite you if you’re not careful.
Even if the order of operation works in your favor, what would happen if somewhere down the line, somebody upgrades your solution with a device that has a different order of operation?
When connecting networks whose administration lives within different groups and cannot be guaranteed to play nicely, design for the worse case and you’ll look like a hero when it still works!
If you’ve read all three posts, you’re now carrying, in total, over 10,000 words of NAT stuff in your head, and probably need a drink and a lie down.

If you made it all the way down here, thanks for persevering! Hopefully you’ve found this series of posts useful, or at least mildly interesting. If you have any comments or questions, please feel free to use the comments section below.

I’d also like to give a huge shout out to my good friends Daniel Paluszek, John Marrone and Grant Markham for reviewing these posts and, The Inti Shah for teaching me to use sub-optimal instead of “you could have done that better!”